2024-09-19T07:42:32.960 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-19T07:42:32.987 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-19T07:42:33.091 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911829 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/iozone}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7911829' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon warn on pool no app: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request sha1: 2f599a46ad10e37c929e5c43da2f6907ddcd1961 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 2f599a46ad10e37c929e5c43da2f6907ddcd1961 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 6fad5dd73c93545e52f57f681e140a9c3529c11e owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1747 sha1: 2f599a46ad10e37c929e5c43da2f6907ddcd1961 sleep_before_teardown: 0 subset: 176/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 6fad5dd73c93545e52f57f681e140a9c3529c11e syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBClN3S2AyqboNhkHB3e0PqG08YinL8xwG58R2RRFPe6PYntcblhlLaLKsa7mMjZk9yLvOcO9UaC/9XPBXDK3mAE= smithi139.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNwAeVNznZuGyTxzxihdmdGv2BzV95kx8F+62LbNpXOjIesFzC8iZgyeEyg4bJQsTj4gJ1MQhkbqVLtVvvNLSu4= smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPZmh6uRw2QM4HvfIonb5JwTQw72NhdNqST2KX2eYkYVvrFfMoex1tHGmxDqOvWA3VCl+UtkRmP1he1gMgjFWYI= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - quiescer: null - workunit: clients: all: - suites/iozone.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_21:24:17 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-19T07:42:33.091 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa; will attempt to use it 2024-09-19T07:42:33.092 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa/tasks 2024-09-19T07:42:33.092 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-19T07:42:33.094 INFO:teuthology.task.internal:Checking packages... 2024-09-19T07:42:33.118 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '2f599a46ad10e37c929e5c43da2f6907ddcd1961' 2024-09-19T07:42:33.119 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-19T07:42:33.119 INFO:teuthology.packaging:ref: None 2024-09-19T07:42:33.119 INFO:teuthology.packaging:tag: None 2024-09-19T07:42:33.119 INFO:teuthology.packaging:branch: squid 2024-09-19T07:42:33.119 INFO:teuthology.packaging:sha1: 2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:42:33.119 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-19T07:42:33.360 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-363.g6fad5dd7 2024-09-19T07:42:33.361 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-19T07:42:33.703 INFO:teuthology.task.internal:no buildpackages task found 2024-09-19T07:42:33.703 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-19T07:42:33.843 INFO:teuthology.task.internal:Saving configuration 2024-09-19T07:42:33.864 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-19T07:42:33.962 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-19T07:42:33.982 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911829', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-19 07:37:19.714639', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBClN3S2AyqboNhkHB3e0PqG08YinL8xwG58R2RRFPe6PYntcblhlLaLKsa7mMjZk9yLvOcO9UaC/9XPBXDK3mAE='} 2024-09-19T07:42:34.002 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911829', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-19 07:37:19.716185', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNwAeVNznZuGyTxzxihdmdGv2BzV95kx8F+62LbNpXOjIesFzC8iZgyeEyg4bJQsTj4gJ1MQhkbqVLtVvvNLSu4='} 2024-09-19T07:42:34.018 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911829', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-19 07:37:19.717312', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPZmh6uRw2QM4HvfIonb5JwTQw72NhdNqST2KX2eYkYVvrFfMoex1tHGmxDqOvWA3VCl+UtkRmP1he1gMgjFWYI='} 2024-09-19T07:42:34.018 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-19T07:42:34.127 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-09-19T07:42:34.127 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-09-19T07:42:34.128 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-09-19T07:42:34.128 INFO:teuthology.run_tasks:Running task console_log... 2024-09-19T07:42:34.367 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efce2583520>, signals=[15]) 2024-09-19T07:42:34.367 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-19T07:42:34.373 INFO:teuthology.task.internal:Opening connections... 2024-09-19T07:42:34.373 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-09-19T07:42:34.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:42:34.449 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi139.front.sepia.ceph.com 2024-09-19T07:42:34.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:42:34.524 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-09-19T07:42:34.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:42:34.602 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-19T07:42:34.609 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-09-19T07:42:34.628 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-09-19T07:42:34.628 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-09-19T07:42:34.686 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-09-19T07:42:34.686 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-09-19T07:42:34.686 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-09-19T07:42:34.686 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-09-19T07:42:34.686 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-09-19T07:42:34.686 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-09-19T07:42:34.686 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-19T07:42:34.686 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-09-19T07:42:34.686 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-09-19T07:42:34.686 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-19T07:42:34.687 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-09-19T07:42:34.687 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-19T07:42:34.687 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-19T07:42:34.687 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-19T07:42:34.687 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-09-19T07:42:34.705 DEBUG:teuthology.orchestra.run.smithi139:> uname -m 2024-09-19T07:42:34.724 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2024-09-19T07:42:34.724 DEBUG:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2024-09-19T07:42:34.782 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="CentOS Stream" 2024-09-19T07:42:34.782 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="9" 2024-09-19T07:42:34.782 INFO:teuthology.orchestra.run.smithi139.stdout:ID="centos" 2024-09-19T07:42:34.782 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="rhel fedora" 2024-09-19T07:42:34.782 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="9" 2024-09-19T07:42:34.783 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el9" 2024-09-19T07:42:34.783 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-19T07:42:34.783 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31" 2024-09-19T07:42:34.783 INFO:teuthology.orchestra.run.smithi139.stdout:LOGO="fedora-logo-icon" 2024-09-19T07:42:34.783 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-19T07:42:34.783 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://centos.org/" 2024-09-19T07:42:34.783 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-19T07:42:34.783 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-19T07:42:34.783 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-19T07:42:34.783 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2024-09-19T07:42:34.805 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-09-19T07:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-09-19T07:42:34.825 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-09-19T07:42:34.883 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-09-19T07:42:34.884 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="9" 2024-09-19T07:42:34.884 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-09-19T07:42:34.884 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-09-19T07:42:34.884 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="9" 2024-09-19T07:42:34.884 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el9" 2024-09-19T07:42:34.884 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-19T07:42:34.884 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-09-19T07:42:34.884 INFO:teuthology.orchestra.run.smithi162.stdout:LOGO="fedora-logo-icon" 2024-09-19T07:42:34.884 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-19T07:42:34.884 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-09-19T07:42:34.884 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-19T07:42:34.884 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-19T07:42:34.885 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-19T07:42:34.885 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-09-19T07:42:34.905 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-19T07:42:34.916 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-19T07:42:35.015 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-19T07:42:35.016 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-09-19T07:42:35.018 DEBUG:teuthology.orchestra.run.smithi139:> test '!' -e /home/ubuntu/cephtest 2024-09-19T07:42:35.021 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-09-19T07:42:35.038 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-19T07:42:35.160 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-19T07:42:35.160 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-09-19T07:42:35.162 DEBUG:teuthology.orchestra.run.smithi139:> test -z $(ls -A /var/lib/ceph) 2024-09-19T07:42:35.165 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-09-19T07:42:35.268 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-19T07:42:35.341 INFO:teuthology.run_tasks:Running task kernel... 2024-09-19T07:42:35.404 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-19T07:42:35.404 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-09-19T07:42:35.404 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-09-19T07:42:35.405 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-19T07:42:35.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:42:35.422 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-09-19T07:42:35.480 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-503.el9.x86_64 2024-09-19T07:42:35.481 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-503.el9.x86_64 2024-09-19T07:42:35.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-09-19T07:42:38.025 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.3 MB 00:00 2024-09-19T07:42:39.743 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-19T07:42:46.994 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 6.5 MB 00:03 2024-09-19T07:42:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 19 kB 00:01 2024-09-19T07:42:50.541 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-19T07:42:55.938 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-19T07:42:57.803 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-19T07:42:57.803 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-19T07:42:57.883 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-19T07:42:57.893 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-19T07:42:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-09-19T07:42:57.893 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-19T07:42:57.893 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-19T07:42:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-19T07:42:57.893 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-19T07:42:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-19T07:42:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-19T07:42:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-19T07:42:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:42:57.894 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-19T07:42:57.894 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-19T07:42:57.894 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-09-19T07:42:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:42:57.894 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 79 M 2024-09-19T07:42:57.895 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 126 M 2024-09-19T07:42:57.895 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-19T07:42:58.240 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 120 kB/s | 36 kB 00:00 2024-09-19T07:43:14.114 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 991 kB/s | 16 MB 00:16 2024-09-19T07:43:30.779 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 1.1 MB/s | 35 MB 00:32 2024-09-19T07:43:34.637 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 804 kB/s | 29 MB 00:36 2024-09-19T07:43:34.637 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-19T07:43:34.638 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.1 MB/s | 79 MB 00:36 2024-09-19T07:43:35.320 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-19T07:43:35.682 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-19T07:43:35.682 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-19T07:43:36.644 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-19T07:43:36.645 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-19T07:43:39.089 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-19T07:43:39.825 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-19T07:43:39.914 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-19T07:43:42.660 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-19T07:43:42.793 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-19T07:43:47.606 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-19T07:43:47.643 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-19T07:43:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-19T07:44:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-19T07:44:30.478 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-19T07:44:30.901 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-19T07:44:30.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-19T07:44:30.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-19T07:44:30.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-19T07:44:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-19T07:44:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:44:31.754 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-19T07:44:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-19T07:44:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-19T07:44:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-19T07:44:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-19T07:44:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:44:31.755 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-19T07:44:32.110 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-09-19T07:44:32.184 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-19T07:44:32.184 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-19T07:44:32.184 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-19T07:44:32.185 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-19T07:44:32.185 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-19T07:44:32.185 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-19T07:44:32.185 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-09-19T07:44:32.185 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-09-19T07:44:32.244 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-19T07:44:32.244 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-19T07:44:32.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-09-19T07:44:32.276 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-09-19T07:44:32.276 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-09-19T07:44:32.276 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-09-19T07:44:32.276 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 82.el9 2024-09-19T07:44:32.276 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-09-19T07:44:32.277 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-19T07:44:32.277 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-09-19T07:44:32.277 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8274670 2024-09-19T07:44:32.277 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-09-19T07:44:32.277 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-19T07:44:32.277 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-19T07:44:32.277 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-19T07:44:32.277 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-19T07:44:32.277 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-09-19T07:44:32.277 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-09-19T07:44:32.277 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-19T07:44:32.277 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-09-19T07:44:32.277 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-09-19T07:44:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:44:32.278 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-19T07:44:32.278 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-19T07:44:32.278 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-19T07:44:32.278 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-09-19T07:44:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:44:32.278 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-09-19T07:44:32.279 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-19T07:44:32.279 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-508.el9.x86_64 2024-09-19T07:44:32.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-19T07:44:32.906 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-09-19T07:44:34.189 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-19T07:44:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-09-19T07:44:34.230 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-09-19T07:44:34.246 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.ifddWtsms7 2024-09-19T07:44:34.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ifddWtsms7 2024-09-19T07:44:34.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.ifddWtsms7 2024-09-19T07:44:34.471 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.ifddWtsms7 is 6KB 2024-09-19T07:44:34.520 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.ifddWtsms7 2024-09-19T07:44:34.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-09-19T07:44:34.598 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-19T07:44:34.598 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-19T07:44:34.598 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-19T07:44:34.598 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-19T07:44:34.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-19T07:44:34.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-09-19T07:45:04.742 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-19T07:45:04.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-19T07:45:04.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:45:23.261 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-09-19T07:45:32.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-19T07:45:32.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:45:35.325 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-09-19T07:45:47.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-19T07:45:47.327 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:46:19.147 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-09-19T07:46:19.580 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-19T07:46:19.580 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-508.el9.x86_64"... 2024-09-19T07:46:19.580 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-09-19T07:46:19.595 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-508.el9.x86_64 2024-09-19T07:46:19.595 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-19T07:46:19.596 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-19T07:46:19.596 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-19T07:46:20.597 DEBUG:teuthology.parallel:result is None 2024-09-19T07:46:20.597 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-19T07:46:20.607 INFO:teuthology.task.internal:Creating test directory... 2024-09-19T07:46:20.608 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-19T07:46:20.610 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-19T07:46:20.613 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-19T07:46:20.632 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-19T07:46:20.737 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-19T07:46:20.806 INFO:teuthology.task.internal:Creating archive directory... 2024-09-19T07:46:20.807 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-19T07:46:20.809 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-19T07:46:20.811 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-19T07:46:20.917 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-19T07:46:20.925 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-19T07:46:20.925 DEBUG:teuthology.orchestra.run.smithi139:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-19T07:46:20.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:46:20.943 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-19T07:46:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:46:20.960 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-19T07:46:20.962 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-19T07:46:20.985 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-19T07:46:21.027 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T07:46:21.033 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T07:46:21.046 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T07:46:21.058 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T07:46:21.068 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T07:46:21.074 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T07:46:21.076 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-19T07:46:21.084 INFO:teuthology.task.internal:Configuring sudo... 2024-09-19T07:46:21.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-19T07:46:21.087 DEBUG:teuthology.orchestra.run.smithi139:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-19T07:46:21.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-19T07:46:21.118 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-19T07:46:21.163 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-19T07:46:21.163 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-19T07:46:21.165 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-19T07:46:21.167 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-19T07:46:21.184 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-19T07:46:21.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-19T07:46:21.308 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-19T07:46:21.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-19T07:46:21.429 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T07:46:21.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-19T07:46:21.494 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-19T07:46:21.538 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-19T07:46:21.589 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-19T07:46:21.673 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-19T07:46:21.718 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-19T07:46:21.719 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-19T07:46:21.786 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-19T07:46:21.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-19T07:46:21.884 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-19T07:46:21.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-19T07:46:22.012 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-19T07:46:22.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-19T07:46:22.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-09-19T07:46:22.083 DEBUG:teuthology.orchestra.run.smithi139:> sudo service rsyslog restart 2024-09-19T07:46:22.086 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-09-19T07:46:22.126 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-19T07:46:22.130 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-19T07:46:22.166 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-19T07:46:22.665 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-19T07:46:22.672 INFO:teuthology.task.internal:Starting timer... 2024-09-19T07:46:22.672 INFO:teuthology.run_tasks:Running task pcp... 2024-09-19T07:46:22.752 INFO:teuthology.run_tasks:Running task selinux... 2024-09-19T07:46:22.914 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-19T07:46:22.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-09-19T07:46:22.965 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-09-19T07:46:22.967 DEBUG:teuthology.orchestra.run.smithi139:> sudo service auditd rotate 2024-09-19T07:46:23.015 INFO:teuthology.orchestra.run.smithi139.stdout:Rotating logs: 2024-09-19T07:46:23.017 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-09-19T07:46:23.071 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-09-19T07:46:23.073 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-19T07:46:23.073 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-09-19T07:46:23.110 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-09-19T07:46:23.111 DEBUG:teuthology.orchestra.run.smithi139:> /usr/sbin/getenforce 2024-09-19T07:46:23.129 INFO:teuthology.orchestra.run.smithi139.stdout:Permissive 2024-09-19T07:46:23.130 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-09-19T07:46:23.153 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-09-19T07:46:23.153 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi139.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi162.front.sepia.ceph.com': 'permissive'} 2024-09-19T07:46:23.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-19T07:46:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:46:23.181 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-19T07:46:23.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:46:23.209 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-19T07:46:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:46:23.238 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-19T07:46:23.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-09-19T07:46:23.270 DEBUG:teuthology.orchestra.run.smithi139:> sudo /usr/sbin/setenforce permissive 2024-09-19T07:46:23.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-09-19T07:46:23.333 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-19T07:46:23.346 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-19T07:46:23.350 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-19T07:46:23.492 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-19T07:46:23.506 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-19T07:46:23.507 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi139.front.sepia.ceph.com,smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-19T07:50:26.295 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi139.front.sepia.ceph.com'), Remote(name='ubuntu@smithi162.front.sepia.ceph.com')] 2024-09-19T07:50:26.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-19T07:50:26.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:50:26.375 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-09-19T07:50:26.455 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-19T07:50:26.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-19T07:50:26.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:50:26.532 DEBUG:teuthology.orchestra.run.smithi139:> true 2024-09-19T07:50:26.613 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-19T07:50:26.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-19T07:50:26.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:50:26.694 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-09-19T07:50:26.779 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-19T07:50:26.780 INFO:teuthology.run_tasks:Running task clock... 2024-09-19T07:50:26.791 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-19T07:50:26.792 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-19T07:50:26.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-19T07:50:26.795 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-19T07:50:26.795 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-19T07:50:26.798 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-19T07:50:26.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-19T07:50:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-19T07:50:26.827 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-19T07:50:26.837 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-19T07:50:26.843 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-19T07:50:26.852 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-19T07:50:26.862 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-09-19T07:50:26.868 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-19T07:50:26.869 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: ntpd: command not found 2024-09-19T07:50:26.873 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-09-19T07:50:26.880 INFO:teuthology.orchestra.run.smithi139.stdout:506 Cannot talk to daemon 2024-09-19T07:50:26.887 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-19T07:50:26.895 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-19T07:50:26.895 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-09-19T07:50:26.901 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-19T07:50:26.907 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-09-19T07:50:26.909 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-19T07:50:26.923 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-19T07:50:26.937 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-19T07:50:26.954 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-09-19T07:50:26.958 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-19T07:50:26.958 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-09-19T07:50:26.958 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T07:50:26.959 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T07:50:26.959 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T07:50:26.959 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T07:50:26.968 INFO:teuthology.orchestra.run.smithi139.stderr:bash: line 1: ntpq: command not found 2024-09-19T07:50:26.971 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-19T07:50:26.971 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2024-09-19T07:50:26.972 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T07:50:26.972 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T07:50:26.972 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T07:50:26.972 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T07:50:26.996 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-09-19T07:50:27.000 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-19T07:50:27.000 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-09-19T07:50:27.000 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T07:50:27.000 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T07:50:27.001 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T07:50:27.001 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T07:50:27.002 INFO:teuthology.run_tasks:Running task install... 2024-09-19T07:50:27.010 DEBUG:teuthology.task.install:project ceph 2024-09-19T07:50:27.010 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961'}} 2024-09-19T07:50:27.010 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961'} 2024-09-19T07:50:27.010 INFO:teuthology.task.install:Using flavor: default 2024-09-19T07:50:27.016 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-19T07:50:27.016 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-19T07:50:27.017 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961', 'tag': None, 'wait_for_package': False} 2024-09-19T07:50:27.018 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:50:27.020 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961', 'tag': None, 'wait_for_package': False} 2024-09-19T07:50:27.020 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:50:27.021 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961', 'tag': None, 'wait_for_package': False} 2024-09-19T07:50:27.021 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:50:27.169 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/ 2024-09-19T07:50:27.170 INFO:teuthology.task.install.rpm:Package version is 19.1.1-361.g2f599a46 2024-09-19T07:50:27.174 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/ 2024-09-19T07:50:27.174 INFO:teuthology.task.install.rpm:Package version is 19.1.1-361.g2f599a46 2024-09-19T07:50:27.185 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/ 2024-09-19T07:50:27.185 INFO:teuthology.task.install.rpm:Package version is 19.1.1-361.g2f599a46 2024-09-19T07:50:27.282 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-19T07:50:27.283 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T07:50:27.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-19T07:50:27.295 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-19T07:50:27.295 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-19T07:50:27.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-19T07:50:27.302 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-19T07:50:27.303 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-19T07:50:27.303 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-19T07:50:27.308 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-19T07:50:27.309 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2f599a46ad10e37c929e5c43da2f6907ddcd1961/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-19T07:50:27.322 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-19T07:50:27.322 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2f599a46ad10e37c929e5c43da2f6907ddcd1961/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-19T07:50:27.329 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-19T07:50:27.329 DEBUG:teuthology.orchestra.run.smithi139:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2f599a46ad10e37c929e5c43da2f6907ddcd1961/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-19T07:50:27.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-19T07:50:27.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-19T07:50:27.395 DEBUG:teuthology.orchestra.run.smithi139:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-19T07:50:27.450 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-19T07:50:27.466 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-19T07:50:27.470 DEBUG:teuthology.orchestra.run.smithi139:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-19T07:50:27.515 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-09-19T07:50:27.516 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-09-19T07:50:27.530 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-09-19T07:50:27.532 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-09-19T07:50:27.534 INFO:teuthology.orchestra.run.smithi139.stdout:check_obsoletes = 1 2024-09-19T07:50:27.535 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum clean all 2024-09-19T07:50:27.756 INFO:teuthology.orchestra.run.smithi098.stdout:45 files removed 2024-09-19T07:50:27.759 INFO:teuthology.orchestra.run.smithi162.stdout:45 files removed 2024-09-19T07:50:27.760 INFO:teuthology.orchestra.run.smithi139.stdout:45 files removed 2024-09-19T07:50:27.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-19T07:50:27.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-19T07:50:27.788 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-19T07:50:28.332 INFO:teuthology.orchestra.run.smithi139.stdout:ceph packages for x86_64 315 kB/s | 84 kB 00:00 2024-09-19T07:50:28.338 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 303 kB/s | 84 kB 00:00 2024-09-19T07:50:28.346 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 301 kB/s | 84 kB 00:00 2024-09-19T07:50:28.623 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 48 kB/s | 12 kB 00:00 2024-09-19T07:50:28.624 INFO:teuthology.orchestra.run.smithi139.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-09-19T07:50:28.630 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-19T07:50:28.898 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 8.3 kB/s | 1.9 kB 00:00 2024-09-19T07:50:28.898 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-19T07:50:28.899 INFO:teuthology.orchestra.run.smithi139.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-19T07:50:29.440 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.3 MB 00:00 2024-09-19T07:50:29.450 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.3 MB 00:00 2024-09-19T07:50:29.534 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-19T07:50:31.441 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-19T07:50:31.601 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - AppStream 19 MB/s | 20 MB 00:01 2024-09-19T07:50:31.857 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 15 MB/s | 20 MB 00:01 2024-09-19T07:50:35.708 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-19T07:50:36.158 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-19T07:50:37.492 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-19T07:50:37.667 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 19 kB 00:00 2024-09-19T07:50:37.976 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.5 MB 00:02 2024-09-19T07:50:39.097 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-19T07:50:39.405 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 13 MB/s | 23 MB 00:01 2024-09-19T07:50:39.552 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 19 kB 00:00 2024-09-19T07:50:40.898 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-19T07:50:44.586 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-19T07:50:44.903 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-19T07:50:46.253 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-19T07:50:46.749 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-19T07:50:46.750 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-19T07:50:46.762 INFO:teuthology.orchestra.run.smithi162.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-19T07:50:46.763 INFO:teuthology.orchestra.run.smithi162.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-19T07:50:46.764 INFO:teuthology.orchestra.run.smithi162.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-19T07:50:46.765 INFO:teuthology.orchestra.run.smithi162.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-19T07:50:46.767 INFO:teuthology.orchestra.run.smithi162.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-19T07:50:46.770 INFO:teuthology.orchestra.run.smithi162.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-19T07:50:46.770 INFO:teuthology.orchestra.run.smithi162.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-19T07:50:46.834 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-19T07:50:46.846 INFO:teuthology.orchestra.run.smithi162.stdout:====================================================================================== 2024-09-19T07:50:46.846 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-19T07:50:46.846 INFO:teuthology.orchestra.run.smithi162.stdout:====================================================================================== 2024-09-19T07:50:46.846 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-19T07:50:46.846 INFO:teuthology.orchestra.run.smithi162.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-19T07:50:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:19.1.1-361.g2f599a46.el9 ceph 6.6 k 2024-09-19T07:50:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:19.1.1-361.g2f599a46.el9 ceph 5.4 M 2024-09-19T07:50:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 896 k 2024-09-19T07:50:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-361.g2f599a46.el9 ceph 145 k 2024-09-19T07:50:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:19.1.1-361.g2f599a46.el9 ceph 1.5 M 2024-09-19T07:50:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 146 k 2024-09-19T07:50:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 3.6 M 2024-09-19T07:50:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 7.4 M 2024-09-19T07:50:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 49 k 2024-09-19T07:50:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:19.1.1-361.g2f599a46.el9 ceph 11 M 2024-09-19T07:50:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:19.1.1-361.g2f599a46.el9 ceph 49 M 2024-09-19T07:50:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 285 k 2024-09-19T07:50:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 765 k 2024-09-19T07:50:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-mirror x86_64 2:19.1.1-361.g2f599a46.el9 ceph 228 k 2024-09-19T07:50:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-shell x86_64 2:19.1.1-361.g2f599a46.el9 ceph 27 k 2024-09-19T07:50:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-top noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 18 k 2024-09-19T07:50:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-19T07:50:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-19T07:50:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-19T07:50:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:19.1.1-361.g2f599a46.el9 ceph 32 k 2024-09-19T07:50:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 751 k 2024-09-19T07:50:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:19.1.1-361.g2f599a46.el9 ceph 127 k 2024-09-19T07:50:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-19T07:50:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-19T07:50:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-19T07:50:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-19T07:50:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-19T07:50:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-19T07:50:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:19.1.1-361.g2f599a46.el9 ceph 161 k 2024-09-19T07:50:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:19.1.1-361.g2f599a46.el9 ceph 320 k 2024-09-19T07:50:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 300 k 2024-09-19T07:50:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:19.1.1-361.g2f599a46.el9 ceph 99 k 2024-09-19T07:50:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 85 k 2024-09-19T07:50:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.1 M 2024-09-19T07:50:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 171 k 2024-09-19T07:50:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-19T07:50:46.850 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-09-19T07:50:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-19T07:50:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-19T07:50:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-19T07:50:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-19T07:50:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.4 M 2024-09-19T07:50:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.2 M 2024-09-19T07:50:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-19T07:50:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-19T07:50:46.850 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-19T07:50:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-19T07:50:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-19T07:50:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-19T07:50:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:19.1.1-361.g2f599a46.el9 ceph 21 M 2024-09-19T07:50:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 32 k 2024-09-19T07:50:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:19.1.1-361.g2f599a46.el9 ceph 2.4 M 2024-09-19T07:50:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 249 k 2024-09-19T07:50:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:19.1.1-361.g2f599a46.el9 ceph 4.7 M 2024-09-19T07:50:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 17 M 2024-09-19T07:50:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 16 k 2024-09-19T07:50:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:19.1.1-361.g2f599a46.el9 ceph 25 k 2024-09-19T07:50:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-19T07:50:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-19T07:50:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-19T07:50:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-19T07:50:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-19T07:50:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-19T07:50:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-19T07:50:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-19T07:50:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:19.1.1-361.g2f599a46.el9 ceph 163 k 2024-09-19T07:50:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-19T07:50:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-19T07:50:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-19T07:50:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-19T07:50:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-19T07:50:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 506 k 2024-09-19T07:50:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-19T07:50:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 5.3 M 2024-09-19T07:50:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-19T07:50:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-19T07:50:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-19T07:50:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-19T07:50:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-19T07:50:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-19T07:50:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-19T07:50:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-19T07:50:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-19T07:50:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-19T07:50:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-19T07:50:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-19T07:50:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-19T07:50:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-19T07:50:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-19T07:50:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-19T07:50:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-19T07:50:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 45 k 2024-09-19T07:50:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:19.1.1-361.g2f599a46.el9 ceph 136 k 2024-09-19T07:50:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-19T07:50:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-19T07:50:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-19T07:50:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-19T07:50:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-19T07:50:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-19T07:50:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-19T07:50:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-19T07:50:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-19T07:50:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-19T07:50:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-19T07:50:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-19T07:50:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-19T07:50:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-19T07:50:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-19T07:50:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-19T07:50:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-19T07:50:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-19T07:50:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-19T07:50:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-19T07:50:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-19T07:50:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-19T07:50:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-19T07:50:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-19T07:50:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-19T07:50:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-19T07:50:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-19T07:50:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-19T07:50:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-19T07:50:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-19T07:50:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-19T07:50:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-19T07:50:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-19T07:50:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-19T07:50:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-19T07:50:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-19T07:50:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-19T07:50:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-19T07:50:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-19T07:50:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-19T07:50:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-19T07:50:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-19T07:50:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-19T07:50:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-19T07:50:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-19T07:50:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-19T07:50:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-19T07:50:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-19T07:50:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-19T07:50:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-19T07:50:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-19T07:50:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-19T07:50:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-19T07:50:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-19T07:50:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-19T07:50:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-19T07:50:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-19T07:50:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-19T07:50:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-19T07:50:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-19T07:50:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-19T07:50:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-19T07:50:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-19T07:50:46.859 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2024-09-19T07:50:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-19T07:50:46.860 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:50:46.860 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-19T07:50:46.860 INFO:teuthology.orchestra.run.smithi162.stdout:====================================================================================== 2024-09-19T07:50:46.860 INFO:teuthology.orchestra.run.smithi162.stdout:Install 148 Packages 2024-09-19T07:50:46.860 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 8 Packages 2024-09-19T07:50:46.860 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:50:46.861 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 226 M 2024-09-19T07:50:46.861 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-19T07:50:47.057 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-19T07:50:47.058 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-19T07:50:47.071 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-19T07:50:47.072 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-19T07:50:47.073 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-19T07:50:47.074 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-19T07:50:47.076 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-19T07:50:47.079 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-19T07:50:47.080 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-19T07:50:47.142 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-19T07:50:47.159 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-09-19T07:50:47.159 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-19T07:50:47.159 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-09-19T07:50:47.159 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-19T07:50:47.159 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-19T07:50:47.159 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.1.1-361.g2f599a46.el9 ceph 6.6 k 2024-09-19T07:50:47.159 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.1.1-361.g2f599a46.el9 ceph 5.4 M 2024-09-19T07:50:47.159 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 896 k 2024-09-19T07:50:47.159 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-361.g2f599a46.el9 ceph 145 k 2024-09-19T07:50:47.159 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.1.1-361.g2f599a46.el9 ceph 1.5 M 2024-09-19T07:50:47.160 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 146 k 2024-09-19T07:50:47.160 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 3.6 M 2024-09-19T07:50:47.160 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 7.4 M 2024-09-19T07:50:47.160 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 49 k 2024-09-19T07:50:47.160 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.1.1-361.g2f599a46.el9 ceph 11 M 2024-09-19T07:50:47.160 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.1.1-361.g2f599a46.el9 ceph 49 M 2024-09-19T07:50:47.160 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 285 k 2024-09-19T07:50:47.160 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 765 k 2024-09-19T07:50:47.160 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:19.1.1-361.g2f599a46.el9 ceph 228 k 2024-09-19T07:50:47.160 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell x86_64 2:19.1.1-361.g2f599a46.el9 ceph 27 k 2024-09-19T07:50:47.160 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 18 k 2024-09-19T07:50:47.161 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-19T07:50:47.161 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-19T07:50:47.161 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-19T07:50:47.161 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.1.1-361.g2f599a46.el9 ceph 32 k 2024-09-19T07:50:47.161 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 751 k 2024-09-19T07:50:47.161 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.1.1-361.g2f599a46.el9 ceph 127 k 2024-09-19T07:50:47.161 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-19T07:50:47.161 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-19T07:50:47.161 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-19T07:50:47.161 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-19T07:50:47.161 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-19T07:50:47.162 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-19T07:50:47.162 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.1.1-361.g2f599a46.el9 ceph 161 k 2024-09-19T07:50:47.162 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.1.1-361.g2f599a46.el9 ceph 320 k 2024-09-19T07:50:47.162 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 300 k 2024-09-19T07:50:47.162 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.1.1-361.g2f599a46.el9 ceph 99 k 2024-09-19T07:50:47.162 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 85 k 2024-09-19T07:50:47.162 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.1 M 2024-09-19T07:50:47.162 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 171 k 2024-09-19T07:50:47.162 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-19T07:50:47.162 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-09-19T07:50:47.162 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-19T07:50:47.162 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-19T07:50:47.163 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-19T07:50:47.163 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-19T07:50:47.163 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.4 M 2024-09-19T07:50:47.163 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.2 M 2024-09-19T07:50:47.163 INFO:teuthology.orchestra.run.smithi098.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-19T07:50:47.163 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-19T07:50:47.163 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-19T07:50:47.163 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-19T07:50:47.163 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-19T07:50:47.163 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-19T07:50:47.163 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.1.1-361.g2f599a46.el9 ceph 21 M 2024-09-19T07:50:47.163 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 32 k 2024-09-19T07:50:47.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.1.1-361.g2f599a46.el9 ceph 2.4 M 2024-09-19T07:50:47.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 249 k 2024-09-19T07:50:47.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.1.1-361.g2f599a46.el9 ceph 4.7 M 2024-09-19T07:50:47.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 17 M 2024-09-19T07:50:47.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 16 k 2024-09-19T07:50:47.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.1.1-361.g2f599a46.el9 ceph 25 k 2024-09-19T07:50:47.164 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-19T07:50:47.164 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-19T07:50:47.164 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-19T07:50:47.164 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-19T07:50:47.165 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-19T07:50:47.165 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-19T07:50:47.165 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-19T07:50:47.165 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-19T07:50:47.165 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.1.1-361.g2f599a46.el9 ceph 163 k 2024-09-19T07:50:47.165 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-19T07:50:47.165 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-19T07:50:47.165 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-19T07:50:47.165 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-19T07:50:47.165 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-19T07:50:47.165 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 506 k 2024-09-19T07:50:47.166 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-19T07:50:47.166 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 5.3 M 2024-09-19T07:50:47.166 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-19T07:50:47.166 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-19T07:50:47.166 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-19T07:50:47.166 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-19T07:50:47.166 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-19T07:50:47.166 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-19T07:50:47.166 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-19T07:50:47.166 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-19T07:50:47.166 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-19T07:50:47.166 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-19T07:50:47.167 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-19T07:50:47.167 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-19T07:50:47.167 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-19T07:50:47.167 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-19T07:50:47.167 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-19T07:50:47.167 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-19T07:50:47.167 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-19T07:50:47.167 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 45 k 2024-09-19T07:50:47.167 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.1.1-361.g2f599a46.el9 ceph 136 k 2024-09-19T07:50:47.167 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-19T07:50:47.167 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-19T07:50:47.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-19T07:50:47.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-19T07:50:47.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-19T07:50:47.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-19T07:50:47.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-19T07:50:47.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-19T07:50:47.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-19T07:50:47.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-19T07:50:47.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-19T07:50:47.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-19T07:50:47.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-19T07:50:47.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-19T07:50:47.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-19T07:50:47.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-19T07:50:47.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-19T07:50:47.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-19T07:50:47.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-19T07:50:47.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-19T07:50:47.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-19T07:50:47.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-19T07:50:47.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-19T07:50:47.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-19T07:50:47.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-19T07:50:47.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-19T07:50:47.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-19T07:50:47.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-19T07:50:47.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-19T07:50:47.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-19T07:50:47.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-19T07:50:47.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-19T07:50:47.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-19T07:50:47.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-19T07:50:47.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-19T07:50:47.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-19T07:50:47.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-19T07:50:47.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-19T07:50:47.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-19T07:50:47.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-19T07:50:47.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-19T07:50:47.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-19T07:50:47.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-19T07:50:47.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-19T07:50:47.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-19T07:50:47.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-19T07:50:47.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-19T07:50:47.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-19T07:50:47.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-19T07:50:47.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-19T07:50:47.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-19T07:50:47.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-19T07:50:47.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-19T07:50:47.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-19T07:50:47.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-19T07:50:47.172 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-19T07:50:47.172 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-19T07:50:47.172 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-19T07:50:47.172 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-19T07:50:47.172 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-19T07:50:47.172 INFO:teuthology.orchestra.run.smithi098.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-19T07:50:47.173 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-19T07:50:47.173 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-19T07:50:47.173 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-09-19T07:50:47.173 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-19T07:50:47.173 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:50:47.173 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-19T07:50:47.173 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-09-19T07:50:47.173 INFO:teuthology.orchestra.run.smithi098.stdout:Install 148 Packages 2024-09-19T07:50:47.173 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 8 Packages 2024-09-19T07:50:47.173 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:50:47.174 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 226 M 2024-09-19T07:50:47.174 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-19T07:50:47.629 INFO:teuthology.orchestra.run.smithi162.stdout:(1/156): ceph-19.1.1-361.g2f599a46.el9.x86_64.r 51 kB/s | 6.6 kB 00:00 2024-09-19T07:50:47.646 INFO:teuthology.orchestra.run.smithi098.stdout:(1/156): ceph-19.1.1-361.g2f599a46.el9.x86_64.r 46 kB/s | 6.6 kB 00:00 2024-09-19T07:50:47.895 INFO:teuthology.orchestra.run.smithi162.stdout:(2/156): ceph-fuse-19.1.1-361.g2f599a46.el9.x86 3.3 MB/s | 896 kB 00:00 2024-09-19T07:50:48.004 INFO:teuthology.orchestra.run.smithi162.stdout:(3/156): ceph-immutable-object-cache-19.1.1-361 1.3 MB/s | 145 kB 00:00 2024-09-19T07:50:48.055 INFO:teuthology.orchestra.run.smithi098.stdout:(2/156): ceph-base-19.1.1-361.g2f599a46.el9.x86 9.8 MB/s | 5.4 MB 00:00 2024-09-19T07:50:48.097 INFO:teuthology.orchestra.run.smithi098.stdout:(3/156): ceph-fuse-19.1.1-361.g2f599a46.el9.x86 1.9 MB/s | 896 kB 00:00 2024-09-19T07:50:48.139 INFO:teuthology.orchestra.run.smithi098.stdout:(4/156): ceph-immutable-object-cache-19.1.1-361 1.7 MB/s | 145 kB 00:00 2024-09-19T07:50:48.188 INFO:teuthology.orchestra.run.smithi162.stdout:(4/156): ceph-base-19.1.1-361.g2f599a46.el9.x86 7.9 MB/s | 5.4 MB 00:00 2024-09-19T07:50:48.239 INFO:teuthology.orchestra.run.smithi098.stdout:(5/156): ceph-mgr-19.1.1-361.g2f599a46.el9.x86_ 15 MB/s | 1.5 MB 00:00 2024-09-19T07:50:48.405 INFO:teuthology.orchestra.run.smithi162.stdout:(5/156): ceph-mds-19.1.1-361.g2f599a46.el9.x86_ 6.0 MB/s | 2.4 MB 00:00 2024-09-19T07:50:48.423 INFO:teuthology.orchestra.run.smithi139.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-19T07:50:48.424 INFO:teuthology.orchestra.run.smithi139.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-19T07:50:48.437 INFO:teuthology.orchestra.run.smithi139.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-19T07:50:48.438 INFO:teuthology.orchestra.run.smithi139.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-19T07:50:48.439 INFO:teuthology.orchestra.run.smithi139.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-19T07:50:48.441 INFO:teuthology.orchestra.run.smithi139.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-19T07:50:48.442 INFO:teuthology.orchestra.run.smithi139.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-19T07:50:48.445 INFO:teuthology.orchestra.run.smithi139.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-19T07:50:48.446 INFO:teuthology.orchestra.run.smithi139.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-19T07:50:48.455 INFO:teuthology.orchestra.run.smithi162.stdout:(6/156): ceph-mgr-19.1.1-361.g2f599a46.el9.x86_ 5.7 MB/s | 1.5 MB 00:00 2024-09-19T07:50:48.508 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-19T07:50:48.528 INFO:teuthology.orchestra.run.smithi139.stdout:====================================================================================== 2024-09-19T07:50:48.528 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-09-19T07:50:48.528 INFO:teuthology.orchestra.run.smithi139.stdout:====================================================================================== 2024-09-19T07:50:48.528 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-19T07:50:48.528 INFO:teuthology.orchestra.run.smithi139.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-19T07:50:48.528 INFO:teuthology.orchestra.run.smithi139.stdout: ceph x86_64 2:19.1.1-361.g2f599a46.el9 ceph 6.6 k 2024-09-19T07:50:48.528 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base x86_64 2:19.1.1-361.g2f599a46.el9 ceph 5.4 M 2024-09-19T07:50:48.528 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 896 k 2024-09-19T07:50:48.528 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-361.g2f599a46.el9 ceph 145 k 2024-09-19T07:50:48.529 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr x86_64 2:19.1.1-361.g2f599a46.el9 ceph 1.5 M 2024-09-19T07:50:48.529 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 146 k 2024-09-19T07:50:48.529 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 3.6 M 2024-09-19T07:50:48.529 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 7.4 M 2024-09-19T07:50:48.529 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 49 k 2024-09-19T07:50:48.529 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw x86_64 2:19.1.1-361.g2f599a46.el9 ceph 11 M 2024-09-19T07:50:48.529 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test x86_64 2:19.1.1-361.g2f599a46.el9 ceph 49 M 2024-09-19T07:50:48.529 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-volume noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 285 k 2024-09-19T07:50:48.529 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 765 k 2024-09-19T07:50:48.529 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-mirror x86_64 2:19.1.1-361.g2f599a46.el9 ceph 228 k 2024-09-19T07:50:48.529 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-shell x86_64 2:19.1.1-361.g2f599a46.el9 ceph 27 k 2024-09-19T07:50:48.529 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-top noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 18 k 2024-09-19T07:50:48.530 INFO:teuthology.orchestra.run.smithi139.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-19T07:50:48.530 INFO:teuthology.orchestra.run.smithi139.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-19T07:50:48.530 INFO:teuthology.orchestra.run.smithi139.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-19T07:50:48.530 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel x86_64 2:19.1.1-361.g2f599a46.el9 ceph 32 k 2024-09-19T07:50:48.530 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 751 k 2024-09-19T07:50:48.530 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel x86_64 2:19.1.1-361.g2f599a46.el9 ceph 127 k 2024-09-19T07:50:48.530 INFO:teuthology.orchestra.run.smithi139.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-19T07:50:48.530 INFO:teuthology.orchestra.run.smithi139.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-19T07:50:48.530 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-19T07:50:48.530 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-19T07:50:48.530 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-19T07:50:48.530 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-19T07:50:48.530 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs x86_64 2:19.1.1-361.g2f599a46.el9 ceph 161 k 2024-09-19T07:50:48.531 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados x86_64 2:19.1.1-361.g2f599a46.el9 ceph 320 k 2024-09-19T07:50:48.531 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 300 k 2024-09-19T07:50:48.531 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw x86_64 2:19.1.1-361.g2f599a46.el9 ceph 99 k 2024-09-19T07:50:48.531 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 85 k 2024-09-19T07:50:48.531 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.1 M 2024-09-19T07:50:48.531 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 171 k 2024-09-19T07:50:48.531 INFO:teuthology.orchestra.run.smithi139.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-19T07:50:48.531 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrading: 2024-09-19T07:50:48.531 INFO:teuthology.orchestra.run.smithi139.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-19T07:50:48.531 INFO:teuthology.orchestra.run.smithi139.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-19T07:50:48.531 INFO:teuthology.orchestra.run.smithi139.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-19T07:50:48.532 INFO:teuthology.orchestra.run.smithi139.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-19T07:50:48.532 INFO:teuthology.orchestra.run.smithi139.stdout: librados2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.4 M 2024-09-19T07:50:48.532 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.2 M 2024-09-19T07:50:48.532 INFO:teuthology.orchestra.run.smithi139.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-19T07:50:48.532 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-19T07:50:48.532 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-09-19T07:50:48.532 INFO:teuthology.orchestra.run.smithi139.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-19T07:50:48.532 INFO:teuthology.orchestra.run.smithi139.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-19T07:50:48.532 INFO:teuthology.orchestra.run.smithi139.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-19T07:50:48.532 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common x86_64 2:19.1.1-361.g2f599a46.el9 ceph 21 M 2024-09-19T07:50:48.532 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 32 k 2024-09-19T07:50:48.532 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds x86_64 2:19.1.1-361.g2f599a46.el9 ceph 2.4 M 2024-09-19T07:50:48.532 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 249 k 2024-09-19T07:50:48.533 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon x86_64 2:19.1.1-361.g2f599a46.el9 ceph 4.7 M 2024-09-19T07:50:48.533 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 17 M 2024-09-19T07:50:48.533 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 16 k 2024-09-19T07:50:48.533 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux x86_64 2:19.1.1-361.g2f599a46.el9 ceph 25 k 2024-09-19T07:50:48.533 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-19T07:50:48.533 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-19T07:50:48.533 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-19T07:50:48.533 INFO:teuthology.orchestra.run.smithi139.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-19T07:50:48.533 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-19T07:50:48.533 INFO:teuthology.orchestra.run.smithi139.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-19T07:50:48.533 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-19T07:50:48.534 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-19T07:50:48.534 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite x86_64 2:19.1.1-361.g2f599a46.el9 ceph 163 k 2024-09-19T07:50:48.534 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-19T07:50:48.534 INFO:teuthology.orchestra.run.smithi139.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-19T07:50:48.534 INFO:teuthology.orchestra.run.smithi139.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-19T07:50:48.534 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-19T07:50:48.534 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-19T07:50:48.534 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 506 k 2024-09-19T07:50:48.534 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-19T07:50:48.534 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 5.3 M 2024-09-19T07:50:48.534 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-19T07:50:48.534 INFO:teuthology.orchestra.run.smithi139.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-19T07:50:48.535 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-19T07:50:48.535 INFO:teuthology.orchestra.run.smithi139.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-19T07:50:48.535 INFO:teuthology.orchestra.run.smithi139.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-19T07:50:48.535 INFO:teuthology.orchestra.run.smithi139.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-19T07:50:48.535 INFO:teuthology.orchestra.run.smithi139.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-19T07:50:48.535 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-19T07:50:48.535 INFO:teuthology.orchestra.run.smithi139.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-19T07:50:48.535 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-19T07:50:48.536 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-19T07:50:48.536 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-19T07:50:48.536 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-19T07:50:48.536 INFO:teuthology.orchestra.run.smithi139.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-19T07:50:48.536 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-19T07:50:48.536 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-19T07:50:48.536 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-19T07:50:48.536 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 45 k 2024-09-19T07:50:48.536 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common x86_64 2:19.1.1-361.g2f599a46.el9 ceph 136 k 2024-09-19T07:50:48.536 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-19T07:50:48.536 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-19T07:50:48.536 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-19T07:50:48.537 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-19T07:50:48.537 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-19T07:50:48.537 INFO:teuthology.orchestra.run.smithi139.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-19T07:50:48.537 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-19T07:50:48.537 INFO:teuthology.orchestra.run.smithi139.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-19T07:50:48.537 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-19T07:50:48.537 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-19T07:50:48.537 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-19T07:50:48.537 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-19T07:50:48.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-19T07:50:48.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-19T07:50:48.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-19T07:50:48.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-19T07:50:48.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-19T07:50:48.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-19T07:50:48.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-19T07:50:48.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-19T07:50:48.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-19T07:50:48.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-19T07:50:48.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-19T07:50:48.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-19T07:50:48.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-19T07:50:48.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-19T07:50:48.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-19T07:50:48.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-19T07:50:48.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-19T07:50:48.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-19T07:50:48.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-19T07:50:48.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-19T07:50:48.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-19T07:50:48.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-19T07:50:48.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-19T07:50:48.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-19T07:50:48.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-19T07:50:48.540 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-19T07:50:48.540 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-19T07:50:48.540 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-19T07:50:48.540 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-19T07:50:48.540 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-19T07:50:48.540 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-19T07:50:48.540 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-19T07:50:48.540 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-19T07:50:48.540 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-19T07:50:48.540 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-19T07:50:48.540 INFO:teuthology.orchestra.run.smithi139.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-19T07:50:48.541 INFO:teuthology.orchestra.run.smithi139.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-19T07:50:48.541 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-19T07:50:48.541 INFO:teuthology.orchestra.run.smithi139.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-19T07:50:48.541 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-19T07:50:48.541 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-19T07:50:48.541 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-19T07:50:48.541 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-19T07:50:48.541 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-19T07:50:48.541 INFO:teuthology.orchestra.run.smithi139.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-19T07:50:48.541 INFO:teuthology.orchestra.run.smithi139.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-19T07:50:48.541 INFO:teuthology.orchestra.run.smithi139.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-19T07:50:48.541 INFO:teuthology.orchestra.run.smithi139.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-19T07:50:48.541 INFO:teuthology.orchestra.run.smithi139.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-19T07:50:48.541 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-19T07:50:48.542 INFO:teuthology.orchestra.run.smithi139.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-19T07:50:48.542 INFO:teuthology.orchestra.run.smithi139.stdout:Installing weak dependencies: 2024-09-19T07:50:48.542 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-19T07:50:48.542 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:50:48.542 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-19T07:50:48.542 INFO:teuthology.orchestra.run.smithi139.stdout:====================================================================================== 2024-09-19T07:50:48.542 INFO:teuthology.orchestra.run.smithi139.stdout:Install 148 Packages 2024-09-19T07:50:48.542 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrade 8 Packages 2024-09-19T07:50:48.542 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:50:48.543 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 226 M 2024-09-19T07:50:48.543 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-19T07:50:48.647 INFO:teuthology.orchestra.run.smithi098.stdout:(6/156): ceph-mon-19.1.1-361.g2f599a46.el9.x86_ 12 MB/s | 4.7 MB 00:00 2024-09-19T07:50:48.979 INFO:teuthology.orchestra.run.smithi098.stdout:(7/156): ceph-common-19.1.1-361.g2f599a46.el9.x 14 MB/s | 21 MB 00:01 2024-09-19T07:50:49.022 INFO:teuthology.orchestra.run.smithi162.stdout:(7/156): ceph-mon-19.1.1-361.g2f599a46.el9.x86_ 7.6 MB/s | 4.7 MB 00:00 2024-09-19T07:50:49.071 INFO:teuthology.orchestra.run.smithi139.stdout:(1/156): ceph-19.1.1-361.g2f599a46.el9.x86_64.r 41 kB/s | 6.6 kB 00:00 2024-09-19T07:50:49.140 INFO:teuthology.orchestra.run.smithi098.stdout:(8/156): ceph-mds-19.1.1-361.g2f599a46.el9.x86_ 2.3 MB/s | 2.4 MB 00:01 2024-09-19T07:50:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:(9/156): ceph-selinux-19.1.1-361.g2f599a46.el9. 378 kB/s | 25 kB 00:00 2024-09-19T07:50:49.690 INFO:teuthology.orchestra.run.smithi098.stdout:(10/156): ceph-radosgw-19.1.1-361.g2f599a46.el9 15 MB/s | 11 MB 00:00 2024-09-19T07:50:49.774 INFO:teuthology.orchestra.run.smithi098.stdout:(11/156): cephfs-mirror-19.1.1-361.g2f599a46.el 2.7 MB/s | 228 kB 00:00 2024-09-19T07:50:49.813 INFO:teuthology.orchestra.run.smithi139.stdout:(2/156): ceph-fuse-19.1.1-361.g2f599a46.el9.x86 1.2 MB/s | 896 kB 00:00 2024-09-19T07:50:49.857 INFO:teuthology.orchestra.run.smithi098.stdout:(12/156): cephfs-shell-19.1.1-361.g2f599a46.el9 324 kB/s | 27 kB 00:00 2024-09-19T07:50:49.938 INFO:teuthology.orchestra.run.smithi139.stdout:(3/156): ceph-immutable-object-cache-19.1.1-361 1.1 MB/s | 145 kB 00:00 2024-09-19T07:50:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:(13/156): ceph-osd-19.1.1-361.g2f599a46.el9.x86 12 MB/s | 17 MB 00:01 2024-09-19T07:50:50.100 INFO:teuthology.orchestra.run.smithi098.stdout:(14/156): libcephfs-devel-19.1.1-361.g2f599a46. 131 kB/s | 32 kB 00:00 2024-09-19T07:50:50.139 INFO:teuthology.orchestra.run.smithi162.stdout:(8/156): ceph-osd-19.1.1-361.g2f599a46.el9.x86_ 10 MB/s | 17 MB 00:01 2024-09-19T07:50:50.142 INFO:teuthology.orchestra.run.smithi098.stdout:(15/156): libcephfs2-19.1.1-361.g2f599a46.el9.x 11 MB/s | 751 kB 00:00 2024-09-19T07:50:50.167 INFO:teuthology.orchestra.run.smithi098.stdout:(16/156): libcephsqlite-19.1.1-361.g2f599a46.el 2.4 MB/s | 163 kB 00:00 2024-09-19T07:50:50.192 INFO:teuthology.orchestra.run.smithi098.stdout:(17/156): librados-devel-19.1.1-361.g2f599a46.e 2.5 MB/s | 127 kB 00:00 2024-09-19T07:50:50.226 INFO:teuthology.orchestra.run.smithi098.stdout:(18/156): libradosstriper1-19.1.1-361.g2f599a46 8.4 MB/s | 506 kB 00:00 2024-09-19T07:50:50.293 INFO:teuthology.orchestra.run.smithi098.stdout:(19/156): python3-ceph-argparse-19.1.1-361.g2f5 677 kB/s | 45 kB 00:00 2024-09-19T07:50:50.347 INFO:teuthology.orchestra.run.smithi162.stdout:(9/156): ceph-common-19.1.1-361.g2f599a46.el9.x 7.4 MB/s | 21 MB 00:02 2024-09-19T07:50:50.360 INFO:teuthology.orchestra.run.smithi098.stdout:(20/156): python3-ceph-common-19.1.1-361.g2f599 2.0 MB/s | 136 kB 00:00 2024-09-19T07:50:50.371 INFO:teuthology.orchestra.run.smithi139.stdout:(4/156): ceph-mds-19.1.1-361.g2f599a46.el9.x86_ 5.5 MB/s | 2.4 MB 00:00 2024-09-19T07:50:50.372 INFO:teuthology.orchestra.run.smithi162.stdout:(10/156): ceph-selinux-19.1.1-361.g2f599a46.el9 108 kB/s | 25 kB 00:00 2024-09-19T07:50:50.427 INFO:teuthology.orchestra.run.smithi098.stdout:(21/156): python3-cephfs-19.1.1-361.g2f599a46.e 2.4 MB/s | 161 kB 00:00 2024-09-19T07:50:50.447 INFO:teuthology.orchestra.run.smithi162.stdout:(11/156): cephfs-mirror-19.1.1-361.g2f599a46.el 3.0 MB/s | 228 kB 00:00 2024-09-19T07:50:50.455 INFO:teuthology.orchestra.run.smithi139.stdout:(5/156): ceph-base-19.1.1-361.g2f599a46.el9.x86 3.5 MB/s | 5.4 MB 00:01 2024-09-19T07:50:50.506 INFO:teuthology.orchestra.run.smithi162.stdout:(12/156): cephfs-shell-19.1.1-361.g2f599a46.el9 463 kB/s | 27 kB 00:00 2024-09-19T07:50:50.536 INFO:teuthology.orchestra.run.smithi098.stdout:(22/156): python3-rados-19.1.1-361.g2f599a46.el 2.9 MB/s | 320 kB 00:00 2024-09-19T07:50:50.557 INFO:teuthology.orchestra.run.smithi162.stdout:(13/156): libcephfs-devel-19.1.1-361.g2f599a46. 629 kB/s | 32 kB 00:00 2024-09-19T07:50:50.611 INFO:teuthology.orchestra.run.smithi098.stdout:(23/156): librgw2-19.1.1-361.g2f599a46.el9.x86_ 13 MB/s | 5.3 MB 00:00 2024-09-19T07:50:50.644 INFO:teuthology.orchestra.run.smithi098.stdout:(24/156): python3-rbd-19.1.1-361.g2f599a46.el9. 2.7 MB/s | 300 kB 00:00 2024-09-19T07:50:50.682 INFO:teuthology.orchestra.run.smithi162.stdout:(14/156): libcephfs2-19.1.1-361.g2f599a46.el9.x 5.9 MB/s | 751 kB 00:00 2024-09-19T07:50:50.695 INFO:teuthology.orchestra.run.smithi098.stdout:(25/156): python3-rgw-19.1.1-361.g2f599a46.el9. 1.2 MB/s | 99 kB 00:00 2024-09-19T07:50:50.714 INFO:teuthology.orchestra.run.smithi139.stdout:(6/156): ceph-mgr-19.1.1-361.g2f599a46.el9.x86_ 4.5 MB/s | 1.5 MB 00:00 2024-09-19T07:50:50.720 INFO:teuthology.orchestra.run.smithi098.stdout:(26/156): rbd-fuse-19.1.1-361.g2f599a46.el9.x86 1.1 MB/s | 85 kB 00:00 2024-09-19T07:50:50.804 INFO:teuthology.orchestra.run.smithi098.stdout:(27/156): rbd-nbd-19.1.1-361.g2f599a46.el9.x86_ 2.0 MB/s | 171 kB 00:00 2024-09-19T07:50:50.816 INFO:teuthology.orchestra.run.smithi162.stdout:(15/156): ceph-radosgw-19.1.1-361.g2f599a46.el9 5.9 MB/s | 11 MB 00:01 2024-09-19T07:50:50.841 INFO:teuthology.orchestra.run.smithi162.stdout:(16/156): libcephsqlite-19.1.1-361.g2f599a46.el 1.0 MB/s | 163 kB 00:00 2024-09-19T07:50:50.896 INFO:teuthology.orchestra.run.smithi098.stdout:(28/156): ceph-grafana-dashboards-19.1.1-361.g2 347 kB/s | 32 kB 00:00 2024-09-19T07:50:50.908 INFO:teuthology.orchestra.run.smithi162.stdout:(17/156): librados-devel-19.1.1-361.g2f599a46.e 1.3 MB/s | 127 kB 00:00 2024-09-19T07:50:50.950 INFO:teuthology.orchestra.run.smithi162.stdout:(18/156): libradosstriper1-19.1.1-361.g2f599a46 4.5 MB/s | 506 kB 00:00 2024-09-19T07:50:50.988 INFO:teuthology.orchestra.run.smithi098.stdout:(29/156): rbd-mirror-19.1.1-361.g2f599a46.el9.x 11 MB/s | 3.1 MB 00:00 2024-09-19T07:50:51.021 INFO:teuthology.orchestra.run.smithi098.stdout:(30/156): ceph-mgr-cephadm-19.1.1-361.g2f599a46 1.1 MB/s | 146 kB 00:00 2024-09-19T07:50:51.034 INFO:teuthology.orchestra.run.smithi162.stdout:(19/156): python3-ceph-argparse-19.1.1-361.g2f5 541 kB/s | 45 kB 00:00 2024-09-19T07:50:51.131 INFO:teuthology.orchestra.run.smithi139.stdout:(7/156): ceph-mon-19.1.1-361.g2f599a46.el9.x86_ 7.0 MB/s | 4.7 MB 00:00 2024-09-19T07:50:51.430 INFO:teuthology.orchestra.run.smithi098.stdout:(31/156): ceph-mgr-dashboard-19.1.1-361.g2f599a 8.1 MB/s | 3.6 MB 00:00 2024-09-19T07:50:51.530 INFO:teuthology.orchestra.run.smithi098.stdout:(32/156): ceph-mgr-modules-core-19.1.1-361.g2f5 2.4 MB/s | 249 kB 00:00 2024-09-19T07:50:51.631 INFO:teuthology.orchestra.run.smithi098.stdout:(33/156): ceph-mgr-rook-19.1.1-361.g2f599a46.el 493 kB/s | 49 kB 00:00 2024-09-19T07:50:51.731 INFO:teuthology.orchestra.run.smithi098.stdout:(34/156): ceph-prometheus-alerts-19.1.1-361.g2f 164 kB/s | 16 kB 00:00 2024-09-19T07:50:51.809 INFO:teuthology.orchestra.run.smithi162.stdout:(20/156): python3-ceph-common-19.1.1-361.g2f599 176 kB/s | 136 kB 00:00 2024-09-19T07:50:51.881 INFO:teuthology.orchestra.run.smithi098.stdout:(35/156): ceph-mgr-diskprediction-local-19.1.1- 8.6 MB/s | 7.4 MB 00:00 2024-09-19T07:50:51.901 INFO:teuthology.orchestra.run.smithi162.stdout:(21/156): python3-cephfs-19.1.1-361.g2f599a46.e 1.7 MB/s | 161 kB 00:00 2024-09-19T07:50:51.906 INFO:teuthology.orchestra.run.smithi098.stdout:(36/156): ceph-volume-19.1.1-361.g2f599a46.el9. 1.6 MB/s | 285 kB 00:00 2024-09-19T07:50:51.982 INFO:teuthology.orchestra.run.smithi098.stdout:(37/156): cephadm-19.1.1-361.g2f599a46.el9.noar 7.5 MB/s | 765 kB 00:00 2024-09-19T07:50:51.993 INFO:teuthology.orchestra.run.smithi162.stdout:(22/156): python3-rados-19.1.1-361.g2f599a46.el 3.4 MB/s | 320 kB 00:00 2024-09-19T07:50:52.007 INFO:teuthology.orchestra.run.smithi098.stdout:(38/156): cephfs-top-19.1.1-361.g2f599a46.el9.n 182 kB/s | 18 kB 00:00 2024-09-19T07:50:52.068 INFO:teuthology.orchestra.run.smithi162.stdout:(23/156): librgw2-19.1.1-361.g2f599a46.el9.x86_ 4.6 MB/s | 5.3 MB 00:01 2024-09-19T07:50:52.093 INFO:teuthology.orchestra.run.smithi162.stdout:(24/156): python3-rbd-19.1.1-361.g2f599a46.el9. 2.9 MB/s | 300 kB 00:00 2024-09-19T07:50:52.116 INFO:teuthology.orchestra.run.smithi098.stdout:(39/156): fuse-2.9.9-16.el9.x86_64.rpm 597 kB/s | 80 kB 00:00 2024-09-19T07:50:52.119 INFO:teuthology.orchestra.run.smithi162.stdout:(25/156): python3-rgw-19.1.1-361.g2f599a46.el9. 1.9 MB/s | 99 kB 00:00 2024-09-19T07:50:52.144 INFO:teuthology.orchestra.run.smithi162.stdout:(26/156): rbd-fuse-19.1.1-361.g2f599a46.el9.x86 1.7 MB/s | 85 kB 00:00 2024-09-19T07:50:52.156 INFO:teuthology.orchestra.run.smithi139.stdout:(8/156): ceph-common-19.1.1-361.g2f599a46.el9.x 6.5 MB/s | 21 MB 00:03 2024-09-19T07:50:52.183 INFO:teuthology.orchestra.run.smithi098.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 2.8 MB/s | 191 kB 00:00 2024-09-19T07:50:52.211 INFO:teuthology.orchestra.run.smithi162.stdout:(27/156): rbd-nbd-19.1.1-361.g2f599a46.el9.x86_ 2.5 MB/s | 171 kB 00:00 2024-09-19T07:50:52.217 INFO:teuthology.orchestra.run.smithi098.stdout:(41/156): libgfortran-11.5.0-2.el9.x86_64.rpm 3.7 MB/s | 801 kB 00:00 2024-09-19T07:50:52.231 INFO:teuthology.orchestra.run.smithi139.stdout:(9/156): ceph-selinux-19.1.1-361.g2f599a46.el9. 337 kB/s | 25 kB 00:00 2024-09-19T07:50:52.242 INFO:teuthology.orchestra.run.smithi098.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-09-19T07:50:52.278 INFO:teuthology.orchestra.run.smithi162.stdout:(28/156): ceph-grafana-dashboards-19.1.1-361.g2 478 kB/s | 32 kB 00:00 2024-09-19T07:50:52.292 INFO:teuthology.orchestra.run.smithi098.stdout:(43/156): python3-cryptography-36.0.1-4.el9.x86 16 MB/s | 1.2 MB 00:00 2024-09-19T07:50:52.318 INFO:teuthology.orchestra.run.smithi098.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-09-19T07:50:52.343 INFO:teuthology.orchestra.run.smithi098.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-09-19T07:50:52.361 INFO:teuthology.orchestra.run.smithi162.stdout:(29/156): ceph-mgr-cephadm-19.1.1-361.g2f599a46 1.7 MB/s | 146 kB 00:00 2024-09-19T07:50:52.368 INFO:teuthology.orchestra.run.smithi098.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-09-19T07:50:52.394 INFO:teuthology.orchestra.run.smithi098.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-09-19T07:50:52.670 INFO:teuthology.orchestra.run.smithi162.stdout:(30/156): rbd-mirror-19.1.1-361.g2f599a46.el9.x 5.6 MB/s | 3.1 MB 00:00 2024-09-19T07:50:52.690 INFO:teuthology.orchestra.run.smithi139.stdout:(10/156): ceph-radosgw-19.1.1-361.g2f599a46.el9 6.8 MB/s | 11 MB 00:01 2024-09-19T07:50:52.790 INFO:teuthology.orchestra.run.smithi139.stdout:(11/156): cephfs-mirror-19.1.1-361.g2f599a46.el 2.2 MB/s | 228 kB 00:00 2024-09-19T07:50:52.882 INFO:teuthology.orchestra.run.smithi139.stdout:(12/156): cephfs-shell-19.1.1-361.g2f599a46.el9 295 kB/s | 27 kB 00:00 2024-09-19T07:50:52.974 INFO:teuthology.orchestra.run.smithi139.stdout:(13/156): libcephfs-devel-19.1.1-361.g2f599a46. 346 kB/s | 32 kB 00:00 2024-09-19T07:50:53.012 INFO:teuthology.orchestra.run.smithi162.stdout:(31/156): ceph-mgr-dashboard-19.1.1-361.g2f599a 5.5 MB/s | 3.6 MB 00:00 2024-09-19T07:50:53.094 INFO:teuthology.orchestra.run.smithi098.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 950 kB/s | 689 kB 00:00 2024-09-19T07:50:53.112 INFO:teuthology.orchestra.run.smithi162.stdout:(32/156): ceph-mgr-modules-core-19.1.1-361.g2f5 2.4 MB/s | 249 kB 00:00 2024-09-19T07:50:53.141 INFO:teuthology.orchestra.run.smithi139.stdout:(14/156): libcephfs2-19.1.1-361.g2f599a46.el9.x 4.4 MB/s | 751 kB 00:00 2024-09-19T07:50:53.213 INFO:teuthology.orchestra.run.smithi162.stdout:(33/156): ceph-mgr-rook-19.1.1-361.g2f599a46.el 493 kB/s | 49 kB 00:00 2024-09-19T07:50:53.232 INFO:teuthology.orchestra.run.smithi139.stdout:(15/156): libcephsqlite-19.1.1-361.g2f599a46.el 1.7 MB/s | 163 kB 00:00 2024-09-19T07:50:53.280 INFO:teuthology.orchestra.run.smithi162.stdout:(34/156): ceph-prometheus-alerts-19.1.1-361.g2f 247 kB/s | 16 kB 00:00 2024-09-19T07:50:53.316 INFO:teuthology.orchestra.run.smithi139.stdout:(16/156): librados-devel-19.1.1-361.g2f599a46.e 1.5 MB/s | 127 kB 00:00 2024-09-19T07:50:53.363 INFO:teuthology.orchestra.run.smithi162.stdout:(35/156): ceph-volume-19.1.1-361.g2f599a46.el9. 3.3 MB/s | 285 kB 00:00 2024-09-19T07:50:53.475 INFO:teuthology.orchestra.run.smithi139.stdout:(17/156): libradosstriper1-19.1.1-361.g2f599a46 3.1 MB/s | 506 kB 00:00 2024-09-19T07:50:53.530 INFO:teuthology.orchestra.run.smithi162.stdout:(36/156): cephadm-19.1.1-361.g2f599a46.el9.noar 4.5 MB/s | 765 kB 00:00 2024-09-19T07:50:53.603 INFO:teuthology.orchestra.run.smithi098.stdout:(49/156): ceph-test-19.1.1-361.g2f599a46.el9.x8 11 MB/s | 49 MB 00:04 2024-09-19T07:50:53.628 INFO:teuthology.orchestra.run.smithi098.stdout:(50/156): boost-program-options-1.75.0-8.el9.x8 200 kB/s | 107 kB 00:00 2024-09-19T07:50:53.630 INFO:teuthology.orchestra.run.smithi162.stdout:(37/156): cephfs-top-19.1.1-361.g2f599a46.el9.n 182 kB/s | 18 kB 00:00 2024-09-19T07:50:53.670 INFO:teuthology.orchestra.run.smithi098.stdout:(51/156): bison-3.7.4-5.el9.x86_64.rpm 739 kB/s | 943 kB 00:01 2024-09-19T07:50:53.787 INFO:teuthology.orchestra.run.smithi098.stdout:(52/156): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-09-19T07:50:53.829 INFO:teuthology.orchestra.run.smithi098.stdout:(53/156): flex-2.6.4-9.el9.x86_64.rpm 1.5 MB/s | 313 kB 00:00 2024-09-19T07:50:53.888 INFO:teuthology.orchestra.run.smithi098.stdout:(54/156): elfutils-libelf-devel-0.191-4.el9.x86 76 kB/s | 22 kB 00:00 2024-09-19T07:50:53.938 INFO:teuthology.orchestra.run.smithi098.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 167 kB/s | 18 kB 00:00 2024-09-19T07:50:53.996 INFO:teuthology.orchestra.run.smithi098.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 103 kB/s | 11 kB 00:00 2024-09-19T07:50:54.006 INFO:teuthology.orchestra.run.smithi162.stdout:(38/156): fuse-2.9.9-16.el9.x86_64.rpm 213 kB/s | 80 kB 00:00 2024-09-19T07:50:54.055 INFO:teuthology.orchestra.run.smithi098.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-09-19T07:50:54.083 INFO:teuthology.orchestra.run.smithi139.stdout:(18/156): ceph-osd-19.1.1-361.g2f599a46.el9.x86 5.0 MB/s | 17 MB 00:03 2024-09-19T07:50:54.098 INFO:teuthology.orchestra.run.smithi162.stdout:(39/156): ceph-mgr-diskprediction-local-19.1.1- 5.2 MB/s | 7.4 MB 00:01 2024-09-19T07:50:54.133 INFO:teuthology.orchestra.run.smithi139.stdout:(19/156): python3-ceph-argparse-19.1.1-361.g2f5 908 kB/s | 45 kB 00:00 2024-09-19T07:50:54.206 INFO:teuthology.orchestra.run.smithi098.stdout:(58/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.2 MB/s | 3.0 MB 00:00 2024-09-19T07:50:54.231 INFO:teuthology.orchestra.run.smithi098.stdout:(59/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 193 kB/s | 45 kB 00:00 2024-09-19T07:50:54.234 INFO:teuthology.orchestra.run.smithi139.stdout:(20/156): librgw2-19.1.1-361.g2f599a46.el9.x86_ 7.0 MB/s | 5.3 MB 00:00 2024-09-19T07:50:54.259 INFO:teuthology.orchestra.run.smithi139.stdout:(21/156): python3-ceph-common-19.1.1-361.g2f599 1.1 MB/s | 136 kB 00:00 2024-09-19T07:50:54.273 INFO:teuthology.orchestra.run.smithi098.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-09-19T07:50:54.284 INFO:teuthology.orchestra.run.smithi139.stdout:(22/156): python3-cephfs-19.1.1-361.g2f599a46.e 3.2 MB/s | 161 kB 00:00 2024-09-19T07:50:54.310 INFO:teuthology.orchestra.run.smithi139.stdout:(23/156): python3-rados-19.1.1-361.g2f599a46.el 6.2 MB/s | 320 kB 00:00 2024-09-19T07:50:54.323 INFO:teuthology.orchestra.run.smithi098.stdout:(61/156): libtool-2.4.6-46.el9.x86_64.rpm 4.9 MB/s | 581 kB 00:00 2024-09-19T07:50:54.335 INFO:teuthology.orchestra.run.smithi139.stdout:(24/156): python3-rbd-19.1.1-361.g2f599a46.el9. 5.8 MB/s | 300 kB 00:00 2024-09-19T07:50:54.348 INFO:teuthology.orchestra.run.smithi098.stdout:(62/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 208 kB/s | 24 kB 00:00 2024-09-19T07:50:54.360 INFO:teuthology.orchestra.run.smithi139.stdout:(25/156): python3-rgw-19.1.1-361.g2f599a46.el9. 1.9 MB/s | 99 kB 00:00 2024-09-19T07:50:54.382 INFO:teuthology.orchestra.run.smithi098.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 435 kB/s | 47 kB 00:00 2024-09-19T07:50:54.385 INFO:teuthology.orchestra.run.smithi139.stdout:(26/156): rbd-fuse-19.1.1-361.g2f599a46.el9.x86 1.7 MB/s | 85 kB 00:00 2024-09-19T07:50:54.441 INFO:teuthology.orchestra.run.smithi098.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-19T07:50:54.491 INFO:teuthology.orchestra.run.smithi098.stdout:(65/156): openblas-0.3.26-2.el9.x86_64.rpm 355 kB/s | 38 kB 00:00 2024-09-19T07:50:54.582 INFO:teuthology.orchestra.run.smithi162.stdout:(40/156): libgfortran-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 801 kB 00:00 2024-09-19T07:50:54.641 INFO:teuthology.orchestra.run.smithi098.stdout:(66/156): lua-5.4.4-4.el9.x86_64.rpm 644 kB/s | 188 kB 00:00 2024-09-19T07:50:54.673 INFO:teuthology.orchestra.run.smithi162.stdout:(41/156): libquadmath-11.5.0-2.el9.x86_64.rpm 332 kB/s | 191 kB 00:00 2024-09-19T07:50:54.699 INFO:teuthology.orchestra.run.smithi162.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-09-19T07:50:54.808 INFO:teuthology.orchestra.run.smithi162.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 983 kB/s | 106 kB 00:00 2024-09-19T07:50:54.819 INFO:teuthology.orchestra.run.smithi139.stdout:(27/156): ceph-test-19.1.1-361.g2f599a46.el9.x8 19 MB/s | 49 MB 00:02 2024-09-19T07:50:54.844 INFO:teuthology.orchestra.run.smithi139.stdout:(28/156): rbd-nbd-19.1.1-361.g2f599a46.el9.x86_ 372 kB/s | 171 kB 00:00 2024-09-19T07:50:54.869 INFO:teuthology.orchestra.run.smithi139.stdout:(29/156): ceph-grafana-dashboards-19.1.1-361.g2 638 kB/s | 32 kB 00:00 2024-09-19T07:50:54.895 INFO:teuthology.orchestra.run.smithi139.stdout:(30/156): ceph-mgr-cephadm-19.1.1-361.g2f599a46 2.9 MB/s | 146 kB 00:00 2024-09-19T07:50:54.924 INFO:teuthology.orchestra.run.smithi162.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-09-19T07:50:54.953 INFO:teuthology.orchestra.run.smithi139.stdout:(31/156): rbd-mirror-19.1.1-361.g2f599a46.el9.x 5.2 MB/s | 3.1 MB 00:00 2024-09-19T07:50:55.033 INFO:teuthology.orchestra.run.smithi162.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-09-19T07:50:55.037 INFO:teuthology.orchestra.run.smithi139.stdout:(32/156): ceph-mgr-dashboard-19.1.1-361.g2f599a 21 MB/s | 3.6 MB 00:00 2024-09-19T07:50:55.062 INFO:teuthology.orchestra.run.smithi139.stdout:(33/156): ceph-mgr-modules-core-19.1.1-361.g2f5 2.2 MB/s | 249 kB 00:00 2024-09-19T07:50:55.087 INFO:teuthology.orchestra.run.smithi139.stdout:(34/156): ceph-mgr-rook-19.1.1-361.g2f599a46.el 989 kB/s | 49 kB 00:00 2024-09-19T07:50:55.108 INFO:teuthology.orchestra.run.smithi098.stdout:(67/156): postgresql-13.16-1.el9.x86_64.rpm 3.4 MB/s | 1.6 MB 00:00 2024-09-19T07:50:55.113 INFO:teuthology.orchestra.run.smithi139.stdout:(35/156): ceph-prometheus-alerts-19.1.1-361.g2f 328 kB/s | 16 kB 00:00 2024-09-19T07:50:55.138 INFO:teuthology.orchestra.run.smithi139.stdout:(36/156): ceph-volume-19.1.1-361.g2f599a46.el9. 5.6 MB/s | 285 kB 00:00 2024-09-19T07:50:55.188 INFO:teuthology.orchestra.run.smithi139.stdout:(37/156): cephadm-19.1.1-361.g2f599a46.el9.noar 9.9 MB/s | 765 kB 00:00 2024-09-19T07:50:55.214 INFO:teuthology.orchestra.run.smithi139.stdout:(38/156): cephfs-top-19.1.1-361.g2f599a46.el9.n 242 kB/s | 18 kB 00:00 2024-09-19T07:50:55.242 INFO:teuthology.orchestra.run.smithi098.stdout:(68/156): postgresql-contrib-13.16-1.el9.x86_64 6.7 MB/s | 910 kB 00:00 2024-09-19T07:50:55.314 INFO:teuthology.orchestra.run.smithi139.stdout:(39/156): fuse-2.9.9-16.el9.x86_64.rpm 637 kB/s | 80 kB 00:00 2024-09-19T07:50:55.359 INFO:teuthology.orchestra.run.smithi098.stdout:(69/156): postgresql-private-libs-13.16-1.el9.x 1.1 MB/s | 137 kB 00:00 2024-09-19T07:50:55.439 INFO:teuthology.orchestra.run.smithi139.stdout:(40/156): ceph-mgr-diskprediction-local-19.1.1- 14 MB/s | 7.4 MB 00:00 2024-09-19T07:50:55.464 INFO:teuthology.orchestra.run.smithi139.stdout:(41/156): libquadmath-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-19T07:50:55.515 INFO:teuthology.orchestra.run.smithi139.stdout:(42/156): libgfortran-11.5.0-2.el9.x86_64.rpm 2.6 MB/s | 801 kB 00:00 2024-09-19T07:50:55.540 INFO:teuthology.orchestra.run.smithi139.stdout:(43/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2024-09-19T07:50:55.565 INFO:teuthology.orchestra.run.smithi139.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-19T07:50:55.567 INFO:teuthology.orchestra.run.smithi162.stdout:(46/156): ceph-test-19.1.1-361.g2f599a46.el9.x8 9.3 MB/s | 49 MB 00:05 2024-09-19T07:50:55.590 INFO:teuthology.orchestra.run.smithi139.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-09-19T07:50:55.600 INFO:teuthology.orchestra.run.smithi162.stdout:(47/156): python3-cryptography-36.0.1-4.el9.x86 1.3 MB/s | 1.2 MB 00:00 2024-09-19T07:50:55.624 INFO:teuthology.orchestra.run.smithi139.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-09-19T07:50:55.625 INFO:teuthology.orchestra.run.smithi162.stdout:(48/156): python3-urllib3-1.26.5-6.el9.noarch.r 363 kB/s | 215 kB 00:00 2024-09-19T07:50:55.649 INFO:teuthology.orchestra.run.smithi139.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-09-19T07:50:55.733 INFO:teuthology.orchestra.run.smithi139.stdout:(48/156): python3-cryptography-36.0.1-4.el9.x86 4.6 MB/s | 1.2 MB 00:00 2024-09-19T07:50:55.959 INFO:teuthology.orchestra.run.smithi098.stdout:(70/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 3.0 MB/s | 4.4 MB 00:01 2024-09-19T07:50:56.200 INFO:teuthology.orchestra.run.smithi139.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 229 kB/s | 107 kB 00:00 2024-09-19T07:50:56.308 INFO:teuthology.orchestra.run.smithi139.stdout:(50/156): elfutils-libelf-devel-0.191-4.el9.x86 200 kB/s | 22 kB 00:00 2024-09-19T07:50:56.367 INFO:teuthology.orchestra.run.smithi139.stdout:(51/156): automake-1.16.2-8.el9.noarch.rpm 928 kB/s | 689 kB 00:00 2024-09-19T07:50:56.384 INFO:teuthology.orchestra.run.smithi162.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 141 kB/s | 107 kB 00:00 2024-09-19T07:50:56.459 INFO:teuthology.orchestra.run.smithi162.stdout:(50/156): automake-1.16.2-8.el9.noarch.rpm 772 kB/s | 689 kB 00:00 2024-09-19T07:50:56.483 INFO:teuthology.orchestra.run.smithi139.stdout:(52/156): flexiblas-3.0.4-8.el9.x86_64.rpm 281 kB/s | 33 kB 00:00 2024-09-19T07:50:56.526 INFO:teuthology.orchestra.run.smithi162.stdout:(51/156): bison-3.7.4-5.el9.x86_64.rpm 1.0 MB/s | 943 kB 00:00 2024-09-19T07:50:56.584 INFO:teuthology.orchestra.run.smithi139.stdout:(53/156): bison-3.7.4-5.el9.x86_64.rpm 1.0 MB/s | 943 kB 00:00 2024-09-19T07:50:56.609 INFO:teuthology.orchestra.run.smithi139.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 1.0 MB/s | 313 kB 00:00 2024-09-19T07:50:56.701 INFO:teuthology.orchestra.run.smithi139.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 156 kB/s | 18 kB 00:00 2024-09-19T07:50:56.726 INFO:teuthology.orchestra.run.smithi139.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 95 kB/s | 11 kB 00:00 2024-09-19T07:50:56.818 INFO:teuthology.orchestra.run.smithi139.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-09-19T07:50:56.844 INFO:teuthology.orchestra.run.smithi139.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-09-19T07:50:56.918 INFO:teuthology.orchestra.run.smithi162.stdout:(52/156): elfutils-libelf-devel-0.191-4.el9.x86 41 kB/s | 22 kB 00:00 2024-09-19T07:50:56.919 INFO:teuthology.orchestra.run.smithi139.stdout:(59/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.9 MB/s | 3.0 MB 00:00 2024-09-19T07:50:57.036 INFO:teuthology.orchestra.run.smithi139.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-09-19T07:50:57.061 INFO:teuthology.orchestra.run.smithi139.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 172 kB/s | 24 kB 00:00 2024-09-19T07:50:57.127 INFO:teuthology.orchestra.run.smithi162.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 54 kB/s | 33 kB 00:00 2024-09-19T07:50:57.145 INFO:teuthology.orchestra.run.smithi139.stdout:(62/156): libtool-2.4.6-46.el9.x86_64.rpm 1.9 MB/s | 581 kB 00:00 2024-09-19T07:50:57.170 INFO:teuthology.orchestra.run.smithi139.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 355 kB/s | 47 kB 00:00 2024-09-19T07:50:57.195 INFO:teuthology.orchestra.run.smithi139.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-19T07:50:57.261 INFO:teuthology.orchestra.run.smithi162.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 392 kB/s | 313 kB 00:00 2024-09-19T07:50:57.262 INFO:teuthology.orchestra.run.smithi139.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-09-19T07:50:57.287 INFO:teuthology.orchestra.run.smithi139.stdout:(66/156): openblas-0.3.26-2.el9.x86_64.rpm 328 kB/s | 38 kB 00:00 2024-09-19T07:50:57.429 INFO:teuthology.orchestra.run.smithi139.stdout:(67/156): postgresql-13.16-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-19T07:50:57.493 INFO:teuthology.orchestra.run.smithi098.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 674 kB/s | 1.0 MB 00:01 2024-09-19T07:50:57.570 INFO:teuthology.orchestra.run.smithi139.stdout:(68/156): postgresql-contrib-13.16-1.el9.x86_64 6.3 MB/s | 910 kB 00:00 2024-09-19T07:50:57.652 INFO:teuthology.orchestra.run.smithi162.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 35 kB/s | 18 kB 00:00 2024-09-19T07:50:57.696 INFO:teuthology.orchestra.run.smithi139.stdout:(69/156): postgresql-private-libs-13.16-1.el9.x 1.1 MB/s | 137 kB 00:00 2024-09-19T07:50:57.709 INFO:teuthology.orchestra.run.smithi098.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 404 kB/s | 87 kB 00:00 2024-09-19T07:50:57.761 INFO:teuthology.orchestra.run.smithi162.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 22 kB/s | 11 kB 00:00 2024-09-19T07:50:57.762 INFO:teuthology.orchestra.run.smithi139.stdout:(70/156): openblas-openmp-0.3.26-2.el9.x86_64.r 8.7 MB/s | 4.9 MB 00:00 2024-09-19T07:50:57.963 INFO:teuthology.orchestra.run.smithi139.stdout:(71/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 6.3 MB/s | 4.4 MB 00:00 2024-09-19T07:50:58.028 INFO:teuthology.orchestra.run.smithi162.stdout:(57/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.7 MB/s | 3.0 MB 00:01 2024-09-19T07:50:58.088 INFO:teuthology.orchestra.run.smithi139.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 701 kB/s | 87 kB 00:00 2024-09-19T07:50:58.378 INFO:teuthology.orchestra.run.smithi162.stdout:(58/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 221 kB/s | 160 kB 00:00 2024-09-19T07:50:58.403 INFO:teuthology.orchestra.run.smithi162.stdout:(59/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 70 kB/s | 45 kB 00:00 2024-09-19T07:50:58.576 INFO:teuthology.orchestra.run.smithi098.stdout:(73/156): openblas-openmp-0.3.26-2.el9.x86_64.r 1.2 MB/s | 4.9 MB 00:04 2024-09-19T07:50:58.720 INFO:teuthology.orchestra.run.smithi098.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 433 kB/s | 61 kB 00:00 2024-09-19T07:50:58.844 INFO:teuthology.orchestra.run.smithi098.stdout:(75/156): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-19T07:50:58.905 INFO:teuthology.orchestra.run.smithi139.stdout:(73/156): postgresql-server-13.16-1.el9.x86_64. 4.8 MB/s | 5.8 MB 00:01 2024-09-19T07:50:58.945 INFO:teuthology.orchestra.run.smithi162.stdout:(60/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 45 kB/s | 24 kB 00:00 2024-09-19T07:50:58.970 INFO:teuthology.orchestra.run.smithi162.stdout:(61/156): librdkafka-1.6.1-102.el9.x86_64.rpm 709 kB/s | 662 kB 00:00 2024-09-19T07:50:59.030 INFO:teuthology.orchestra.run.smithi139.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 491 kB/s | 61 kB 00:00 2024-09-19T07:50:59.136 INFO:teuthology.orchestra.run.smithi098.stdout:(76/156): python3-babel-2.9.1-2.el9.noarch.rpm 4.2 MB/s | 6.0 MB 00:01 2024-09-19T07:50:59.221 INFO:teuthology.orchestra.run.smithi162.stdout:(62/156): libtool-2.4.6-46.el9.x86_64.rpm 691 kB/s | 581 kB 00:00 2024-09-19T07:50:59.261 INFO:teuthology.orchestra.run.smithi098.stdout:(77/156): python3-jinja2-2.11.3-6.el9.noarch.rp 595 kB/s | 248 kB 00:00 2024-09-19T07:50:59.286 INFO:teuthology.orchestra.run.smithi098.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 171 kB/s | 26 kB 00:00 2024-09-19T07:50:59.370 INFO:teuthology.orchestra.run.smithi098.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 177 kB/s | 19 kB 00:00 2024-09-19T07:50:59.403 INFO:teuthology.orchestra.run.smithi098.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-19T07:50:59.413 INFO:teuthology.orchestra.run.smithi139.stdout:(75/156): python3-devel-3.9.19-8.el9.x86_64.rpm 639 kB/s | 245 kB 00:00 2024-09-19T07:50:59.487 INFO:teuthology.orchestra.run.smithi098.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 297 kB/s | 35 kB 00:00 2024-09-19T07:50:59.521 INFO:teuthology.orchestra.run.smithi162.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 82 kB/s | 47 kB 00:00 2024-09-19T07:50:59.746 INFO:teuthology.orchestra.run.smithi162.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 377 kB/s | 292 kB 00:00 2024-09-19T07:50:59.755 INFO:teuthology.orchestra.run.smithi139.stdout:(76/156): protobuf-3.14.0-13.el9.x86_64.rpm 518 kB/s | 1.0 MB 00:01 2024-09-19T07:50:59.823 INFO:teuthology.orchestra.run.smithi139.stdout:(77/156): python3-jinja2-2.11.3-6.el9.noarch.rp 607 kB/s | 248 kB 00:00 2024-09-19T07:50:59.865 INFO:teuthology.orchestra.run.smithi139.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 235 kB/s | 26 kB 00:00 2024-09-19T07:50:59.870 INFO:teuthology.orchestra.run.smithi098.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 654 kB/s | 250 kB 00:00 2024-09-19T07:50:59.946 INFO:teuthology.orchestra.run.smithi098.stdout:(83/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.4 MB/s | 5.1 MB 00:00 2024-09-19T07:50:59.948 INFO:teuthology.orchestra.run.smithi139.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 153 kB/s | 19 kB 00:00 2024-09-19T07:50:59.990 INFO:teuthology.orchestra.run.smithi139.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-19T07:51:00.060 INFO:teuthology.orchestra.run.smithi162.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 225 kB/s | 188 kB 00:00 2024-09-19T07:51:00.074 INFO:teuthology.orchestra.run.smithi139.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 278 kB/s | 35 kB 00:00 2024-09-19T07:51:00.079 INFO:teuthology.orchestra.run.smithi098.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 579 kB/s | 77 kB 00:00 2024-09-19T07:51:00.127 INFO:teuthology.orchestra.run.smithi162.stdout:(66/156): openblas-0.3.26-2.el9.x86_64.rpm 64 kB/s | 38 kB 00:00 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi098.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 338 kB/s | 42 kB 00:00 2024-09-19T07:51:00.322 INFO:teuthology.orchestra.run.smithi098.stdout:(86/156): python3-oauthlib-3.1.1-5.el9.noarch.r 493 kB/s | 222 kB 00:00 2024-09-19T07:51:00.355 INFO:teuthology.orchestra.run.smithi098.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 1.7 MB/s | 269 kB 00:00 2024-09-19T07:51:00.489 INFO:teuthology.orchestra.run.smithi098.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 2.0 MB/s | 279 kB 00:00 2024-09-19T07:51:00.499 INFO:teuthology.orchestra.run.smithi139.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 590 kB/s | 250 kB 00:00 2024-09-19T07:51:00.623 INFO:teuthology.orchestra.run.smithi098.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 382 kB/s | 51 kB 00:00 2024-09-19T07:51:00.665 INFO:teuthology.orchestra.run.smithi098.stdout:(90/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 463 kB/s | 159 kB 00:00 2024-09-19T07:51:00.682 INFO:teuthology.orchestra.run.smithi139.stdout:(83/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.4 MB/s | 5.1 MB 00:00 2024-09-19T07:51:00.757 INFO:teuthology.orchestra.run.smithi098.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 402 kB/s | 54 kB 00:00 2024-09-19T07:51:00.824 INFO:teuthology.orchestra.run.smithi139.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 546 kB/s | 77 kB 00:00 2024-09-19T07:51:00.877 INFO:teuthology.orchestra.run.smithi162.stdout:(67/156): openblas-openmp-0.3.26-2.el9.x86_64.r 4.4 MB/s | 4.9 MB 00:01 2024-09-19T07:51:00.907 INFO:teuthology.orchestra.run.smithi098.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 278 kB/s | 42 kB 00:00 2024-09-19T07:51:00.950 INFO:teuthology.orchestra.run.smithi139.stdout:(85/156): python3-oauthlib-3.1.1-5.el9.noarch.r 493 kB/s | 222 kB 00:00 2024-09-19T07:51:00.975 INFO:teuthology.orchestra.run.smithi139.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 281 kB/s | 42 kB 00:00 2024-09-19T07:51:01.032 INFO:teuthology.orchestra.run.smithi098.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 353 kB/s | 44 kB 00:00 2024-09-19T07:51:01.100 INFO:teuthology.orchestra.run.smithi139.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-19T07:51:01.102 INFO:teuthology.orchestra.run.smithi162.stdout:(68/156): postgresql-13.16-1.el9.x86_64.rpm 1.6 MB/s | 1.6 MB 00:00 2024-09-19T07:51:01.166 INFO:teuthology.orchestra.run.smithi098.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-09-19T07:51:01.234 INFO:teuthology.orchestra.run.smithi139.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 2.0 MB/s | 279 kB 00:00 2024-09-19T07:51:01.269 INFO:teuthology.orchestra.run.smithi162.stdout:(69/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 3.7 MB/s | 4.4 MB 00:01 2024-09-19T07:51:01.300 INFO:teuthology.orchestra.run.smithi098.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 269 kB/s | 36 kB 00:00 2024-09-19T07:51:01.367 INFO:teuthology.orchestra.run.smithi139.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 383 kB/s | 51 kB 00:00 2024-09-19T07:51:01.433 INFO:teuthology.orchestra.run.smithi098.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 356 kB/s | 47 kB 00:00 2024-09-19T07:51:01.501 INFO:teuthology.orchestra.run.smithi139.stdout:(90/156): python3-protobuf-3.14.0-13.el9.noarch 489 kB/s | 269 kB 00:00 2024-09-19T07:51:01.526 INFO:teuthology.orchestra.run.smithi139.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 338 kB/s | 54 kB 00:00 2024-09-19T07:51:01.567 INFO:teuthology.orchestra.run.smithi098.stdout:(97/156): socat-1.7.4.1-6.el9.x86_64.rpm 2.2 MB/s | 304 kB 00:00 2024-09-19T07:51:01.651 INFO:teuthology.orchestra.run.smithi139.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 333 kB/s | 42 kB 00:00 2024-09-19T07:51:01.692 INFO:teuthology.orchestra.run.smithi098.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 462 kB/s | 58 kB 00:00 2024-09-19T07:51:01.777 INFO:teuthology.orchestra.run.smithi139.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 353 kB/s | 44 kB 00:00 2024-09-19T07:51:01.811 INFO:teuthology.orchestra.run.smithi162.stdout:(70/156): postgresql-contrib-13.16-1.el9.x86_64 975 kB/s | 910 kB 00:00 2024-09-19T07:51:01.817 INFO:teuthology.orchestra.run.smithi098.stdout:(99/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rpm 766 kB/s | 96 kB 00:00 2024-09-19T07:51:01.836 INFO:teuthology.orchestra.run.smithi162.stdout:(71/156): postgresql-private-libs-13.16-1.el9.x 189 kB/s | 137 kB 00:00 2024-09-19T07:51:01.910 INFO:teuthology.orchestra.run.smithi139.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-09-19T07:51:01.951 INFO:teuthology.orchestra.run.smithi098.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 478 kB/s | 64 kB 00:00 2024-09-19T07:51:02.044 INFO:teuthology.orchestra.run.smithi139.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 269 kB/s | 36 kB 00:00 2024-09-19T07:51:02.093 INFO:teuthology.orchestra.run.smithi098.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 316 kB/s | 45 kB 00:00 2024-09-19T07:51:02.177 INFO:teuthology.orchestra.run.smithi139.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 355 kB/s | 47 kB 00:00 2024-09-19T07:51:02.268 INFO:teuthology.orchestra.run.smithi098.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-09-19T07:51:02.311 INFO:teuthology.orchestra.run.smithi139.stdout:(97/156): socat-1.7.4.1-6.el9.x86_64.rpm 2.2 MB/s | 304 kB 00:00 2024-09-19T07:51:02.387 INFO:teuthology.orchestra.run.smithi162.stdout:(72/156): postgresql-server-13.16-1.el9.x86_64. 5.2 MB/s | 5.8 MB 00:01 2024-09-19T07:51:02.453 INFO:teuthology.orchestra.run.smithi139.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 408 kB/s | 58 kB 00:00 2024-09-19T07:51:02.503 INFO:teuthology.orchestra.run.smithi162.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 131 kB/s | 87 kB 00:00 2024-09-19T07:51:02.535 INFO:teuthology.orchestra.run.smithi098.stdout:(103/156): protobuf-compiler-3.14.0-13.el9.x86_ 3.2 MB/s | 863 kB 00:00 2024-09-19T07:51:02.578 INFO:teuthology.orchestra.run.smithi139.stdout:(99/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rpm 767 kB/s | 96 kB 00:00 2024-09-19T07:51:02.636 INFO:teuthology.orchestra.run.smithi098.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 5.4 MB/s | 548 kB 00:00 2024-09-19T07:51:02.686 INFO:teuthology.orchestra.run.smithi098.stdout:(105/156): gperftools-libs-2.9.1-3.el9.x86_64.r 6.0 MB/s | 308 kB 00:00 2024-09-19T07:51:02.695 INFO:teuthology.orchestra.run.smithi139.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 546 kB/s | 64 kB 00:00 2024-09-19T07:51:02.711 INFO:teuthology.orchestra.run.smithi098.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 781 kB/s | 19 kB 00:00 2024-09-19T07:51:02.729 INFO:teuthology.orchestra.run.smithi162.stdout:(74/156): protobuf-3.14.0-13.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2024-09-19T07:51:02.812 INFO:teuthology.orchestra.run.smithi139.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 384 kB/s | 45 kB 00:00 2024-09-19T07:51:03.012 INFO:teuthology.orchestra.run.smithi098.stdout:(107/156): libarrow-9.0.0-11.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-09-19T07:51:03.028 INFO:teuthology.orchestra.run.smithi098.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.5 MB/s | 26 kB 00:00 2024-09-19T07:51:03.054 INFO:teuthology.orchestra.run.smithi098.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-19T07:51:03.088 INFO:teuthology.orchestra.run.smithi098.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-19T07:51:03.121 INFO:teuthology.orchestra.run.smithi098.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 4.5 MB/s | 151 kB 00:00 2024-09-19T07:51:03.197 INFO:teuthology.orchestra.run.smithi098.stdout:(112/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-09-19T07:51:03.215 INFO:teuthology.orchestra.run.smithi162.stdout:(75/156): python3-colorama-0.4.6-3.el9.noarch.r 86 kB/s | 61 kB 00:00 2024-09-19T07:51:03.230 INFO:teuthology.orchestra.run.smithi098.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 16 MB/s | 548 kB 00:00 2024-09-19T07:51:03.264 INFO:teuthology.orchestra.run.smithi098.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-09-19T07:51:03.346 INFO:teuthology.orchestra.run.smithi139.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 42 kB/s | 22 kB 00:00 2024-09-19T07:51:03.375 INFO:teuthology.orchestra.run.smithi098.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 289 kB/s | 32 kB 00:00 2024-09-19T07:51:03.414 INFO:teuthology.orchestra.run.smithi098.stdout:(116/156): python3-certifi-2023.05.07-4.el9.noa 365 kB/s | 14 kB 00:00 2024-09-19T07:51:03.448 INFO:teuthology.orchestra.run.smithi098.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 5.0 MB/s | 172 kB 00:00 2024-09-19T07:51:03.490 INFO:teuthology.orchestra.run.smithi098.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 8.4 MB/s | 358 kB 00:00 2024-09-19T07:51:03.521 INFO:teuthology.orchestra.run.smithi162.stdout:(76/156): python3-devel-3.9.19-8.el9.x86_64.rpm 312 kB/s | 245 kB 00:00 2024-09-19T07:51:03.532 INFO:teuthology.orchestra.run.smithi098.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 7.1 MB/s | 303 kB 00:00 2024-09-19T07:51:03.558 INFO:teuthology.orchestra.run.smithi098.stdout:(120/156): python3-google-auth-2.34.0-1.el9.noa 8.5 MB/s | 220 kB 00:00 2024-09-19T07:51:03.605 INFO:teuthology.orchestra.run.smithi162.stdout:(77/156): python3-babel-2.9.1-2.el9.noarch.rpm 4.9 MB/s | 6.0 MB 00:01 2024-09-19T07:51:03.658 INFO:teuthology.orchestra.run.smithi098.stdout:(121/156): python3-grpcio-1.46.7-10.el9.x86_64. 20 MB/s | 2.0 MB 00:00 2024-09-19T07:51:03.683 INFO:teuthology.orchestra.run.smithi098.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 5.7 MB/s | 144 kB 00:00 2024-09-19T07:51:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 428 kB/s | 11 kB 00:00 2024-09-19T07:51:03.734 INFO:teuthology.orchestra.run.smithi098.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 705 kB/s | 18 kB 00:00 2024-09-19T07:51:03.759 INFO:teuthology.orchestra.run.smithi098.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 930 kB/s | 23 kB 00:00 2024-09-19T07:51:03.784 INFO:teuthology.orchestra.run.smithi098.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 778 kB/s | 19 kB 00:00 2024-09-19T07:51:03.809 INFO:teuthology.orchestra.run.smithi098.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 787 kB/s | 20 kB 00:00 2024-09-19T07:51:03.877 INFO:teuthology.orchestra.run.smithi098.stdout:(128/156): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-09-19T07:51:03.902 INFO:teuthology.orchestra.run.smithi098.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-19T07:51:03.936 INFO:teuthology.orchestra.run.smithi098.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-09-19T07:51:03.969 INFO:teuthology.orchestra.run.smithi098.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-09-19T07:51:03.996 INFO:teuthology.orchestra.run.smithi162.stdout:(78/156): python3-jinja2-2.11.3-6.el9.noarch.rp 318 kB/s | 248 kB 00:00 2024-09-19T07:51:04.011 INFO:teuthology.orchestra.run.smithi098.stdout:(132/156): python3-pecan-1.4.2-3.el9.noarch.rpm 6.4 MB/s | 272 kB 00:00 2024-09-19T07:51:04.036 INFO:teuthology.orchestra.run.smithi098.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 661 kB/s | 16 kB 00:00 2024-09-19T07:51:04.063 INFO:teuthology.orchestra.run.smithi162.stdout:(79/156): python3-jsonpatch-1.21-16.el9.noarch. 47 kB/s | 26 kB 00:00 2024-09-19T07:51:04.070 INFO:teuthology.orchestra.run.smithi098.stdout:(134/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.6 MB/s | 90 kB 00:00 2024-09-19T07:51:04.095 INFO:teuthology.orchestra.run.smithi098.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-09-19T07:51:04.130 INFO:teuthology.orchestra.run.smithi162.stdout:(80/156): python3-jsonpointer-2.0-4.el9.noarch. 36 kB/s | 19 kB 00:00 2024-09-19T07:51:04.150 INFO:teuthology.orchestra.run.smithi098.stdout:(136/156): python3-repoze-lru-0.7-16.el9.noarch 571 kB/s | 31 kB 00:00 2024-09-19T07:51:04.188 INFO:teuthology.orchestra.run.smithi098.stdout:(137/156): python3-routes-2.5.1-5.el9.noarch.rp 4.9 MB/s | 188 kB 00:00 2024-09-19T07:51:04.222 INFO:teuthology.orchestra.run.smithi098.stdout:(138/156): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-19T07:51:04.238 INFO:teuthology.orchestra.run.smithi139.stdout:(103/156): protobuf-compiler-3.14.0-13.el9.x86_ 968 kB/s | 863 kB 00:00 2024-09-19T07:51:04.247 INFO:teuthology.orchestra.run.smithi098.stdout:(139/156): python3-tempora-5.0.0-2.el9.noarch.r 1.4 MB/s | 36 kB 00:00 2024-09-19T07:51:04.272 INFO:teuthology.orchestra.run.smithi098.stdout:(140/156): python3-typing-extensions-4.12.2-2.e 2.9 MB/s | 74 kB 00:00 2024-09-19T07:51:04.296 INFO:teuthology.orchestra.run.smithi139.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 9.3 MB/s | 548 kB 00:00 2024-09-19T07:51:04.297 INFO:teuthology.orchestra.run.smithi098.stdout:(141/156): python3-webob-1.8.8-2.el9.noarch.rpm 9.0 MB/s | 230 kB 00:00 2024-09-19T07:51:04.330 INFO:teuthology.orchestra.run.smithi139.stdout:(105/156): gperftools-libs-2.9.1-3.el9.x86_64.r 9.1 MB/s | 308 kB 00:00 2024-09-19T07:51:04.347 INFO:teuthology.orchestra.run.smithi139.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-19T07:51:04.385 INFO:teuthology.orchestra.run.smithi098.stdout:(142/156): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-09-19T07:51:04.448 INFO:teuthology.orchestra.run.smithi098.stdout:(143/156): python3-werkzeug-2.0.3-3.el9.1.noarc 6.6 MB/s | 427 kB 00:00 2024-09-19T07:51:04.473 INFO:teuthology.orchestra.run.smithi098.stdout:(144/156): python3-zc-lockfile-2.0-10.el9.noarc 798 kB/s | 20 kB 00:00 2024-09-19T07:51:04.515 INFO:teuthology.orchestra.run.smithi098.stdout:(145/156): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-19T07:51:04.581 INFO:teuthology.orchestra.run.smithi139.stdout:(107/156): libarrow-9.0.0-11.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2024-09-19T07:51:04.597 INFO:teuthology.orchestra.run.smithi139.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.5 MB/s | 26 kB 00:00 2024-09-19T07:51:04.614 INFO:teuthology.orchestra.run.smithi162.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 63 kB/s | 35 kB 00:00 2024-09-19T07:51:04.614 INFO:teuthology.orchestra.run.smithi139.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 2.9 MB/s | 49 kB 00:00 2024-09-19T07:51:04.619 INFO:teuthology.orchestra.run.smithi098.stdout:(146/156): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-19T07:51:04.648 INFO:teuthology.orchestra.run.smithi139.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-19T07:51:04.673 INFO:teuthology.orchestra.run.smithi139.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 5.9 MB/s | 151 kB 00:00 2024-09-19T07:51:04.733 INFO:teuthology.orchestra.run.smithi139.stdout:(112/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 14 MB/s | 839 kB 00:00 2024-09-19T07:51:04.769 INFO:teuthology.orchestra.run.smithi139.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 16 MB/s | 548 kB 00:00 2024-09-19T07:51:04.789 INFO:teuthology.orchestra.run.smithi162.stdout:(82/156): python3-mako-1.1.4-6.el9.noarch.rpm 217 kB/s | 172 kB 00:00 2024-09-19T07:51:04.791 INFO:teuthology.orchestra.run.smithi139.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-19T07:51:04.816 INFO:teuthology.orchestra.run.smithi139.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-19T07:51:04.841 INFO:teuthology.orchestra.run.smithi139.stdout:(116/156): python3-certifi-2023.05.07-4.el9.noa 563 kB/s | 14 kB 00:00 2024-09-19T07:51:04.867 INFO:teuthology.orchestra.run.smithi139.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 6.7 MB/s | 172 kB 00:00 2024-09-19T07:51:04.870 INFO:teuthology.orchestra.run.smithi098.stdout:(147/156): librados2-19.1.1-361.g2f599a46.el9.x 14 MB/s | 3.4 MB 00:00 2024-09-19T07:51:04.900 INFO:teuthology.orchestra.run.smithi139.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 11 MB/s | 358 kB 00:00 2024-09-19T07:51:04.926 INFO:teuthology.orchestra.run.smithi139.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 12 MB/s | 303 kB 00:00 2024-09-19T07:51:04.951 INFO:teuthology.orchestra.run.smithi139.stdout:(120/156): python3-google-auth-2.34.0-1.el9.noa 8.6 MB/s | 220 kB 00:00 2024-09-19T07:51:05.012 INFO:teuthology.orchestra.run.smithi098.stdout:(148/156): librbd1-19.1.1-361.g2f599a46.el9.x86 22 MB/s | 3.2 MB 00:00 2024-09-19T07:51:05.018 INFO:teuthology.orchestra.run.smithi139.stdout:(121/156): python3-grpcio-1.46.7-10.el9.x86_64. 31 MB/s | 2.0 MB 00:00 2024-09-19T07:51:05.043 INFO:teuthology.orchestra.run.smithi139.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 5.7 MB/s | 144 kB 00:00 2024-09-19T07:51:05.068 INFO:teuthology.orchestra.run.smithi139.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 427 kB/s | 11 kB 00:00 2024-09-19T07:51:05.093 INFO:teuthology.orchestra.run.smithi139.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 707 kB/s | 18 kB 00:00 2024-09-19T07:51:05.118 INFO:teuthology.orchestra.run.smithi139.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 927 kB/s | 23 kB 00:00 2024-09-19T07:51:05.144 INFO:teuthology.orchestra.run.smithi139.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 776 kB/s | 19 kB 00:00 2024-09-19T07:51:05.169 INFO:teuthology.orchestra.run.smithi139.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 784 kB/s | 20 kB 00:00 2024-09-19T07:51:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:(149/156): NetworkManager-1.51.0-1.el9.x86_64.r 13 MB/s | 2.3 MB 00:00 2024-09-19T07:51:05.211 INFO:teuthology.orchestra.run.smithi139.stdout:(128/156): python3-kubernetes-26.1.0-2.el9.noar 25 MB/s | 1.0 MB 00:00 2024-09-19T07:51:05.236 INFO:teuthology.orchestra.run.smithi139.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-19T07:51:05.256 INFO:teuthology.orchestra.run.smithi162.stdout:(83/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.5 MB/s | 5.1 MB 00:01 2024-09-19T07:51:05.261 INFO:teuthology.orchestra.run.smithi139.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-19T07:51:05.262 INFO:teuthology.orchestra.run.smithi098.stdout:(150/156): NetworkManager-libnm-1.51.0-1.el9.x8 27 MB/s | 1.8 MB 00:00 2024-09-19T07:51:05.287 INFO:teuthology.orchestra.run.smithi139.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-19T07:51:05.304 INFO:teuthology.orchestra.run.smithi098.stdout:(151/156): NetworkManager-team-1.51.0-1.el9.x86 982 kB/s | 41 kB 00:00 2024-09-19T07:51:05.321 INFO:teuthology.orchestra.run.smithi139.stdout:(132/156): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-09-19T07:51:05.346 INFO:teuthology.orchestra.run.smithi139.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 660 kB/s | 16 kB 00:00 2024-09-19T07:51:05.346 INFO:teuthology.orchestra.run.smithi098.stdout:(152/156): NetworkManager-tui-1.51.0-1.el9.x86_ 5.8 MB/s | 249 kB 00:00 2024-09-19T07:51:05.371 INFO:teuthology.orchestra.run.smithi139.stdout:(134/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-09-19T07:51:05.396 INFO:teuthology.orchestra.run.smithi139.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-09-19T07:51:05.397 INFO:teuthology.orchestra.run.smithi098.stdout:(153/156): openssl-3.2.2-6.el9.x86_64.rpm 27 MB/s | 1.4 MB 00:00 2024-09-19T07:51:05.421 INFO:teuthology.orchestra.run.smithi139.stdout:(136/156): python3-repoze-lru-0.7-16.el9.noarch 1.2 MB/s | 31 kB 00:00 2024-09-19T07:51:05.431 INFO:teuthology.orchestra.run.smithi162.stdout:(84/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 307 kB/s | 250 kB 00:00 2024-09-19T07:51:05.447 INFO:teuthology.orchestra.run.smithi139.stdout:(137/156): python3-routes-2.5.1-5.el9.noarch.rp 7.4 MB/s | 188 kB 00:00 2024-09-19T07:51:05.464 INFO:teuthology.orchestra.run.smithi139.stdout:(138/156): python3-rsa-4.9-2.el9.noarch.rpm 3.5 MB/s | 59 kB 00:00 2024-09-19T07:51:05.471 INFO:teuthology.orchestra.run.smithi098.stdout:(154/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 32 MB/s | 2.4 MB 00:00 2024-09-19T07:51:05.481 INFO:teuthology.orchestra.run.smithi139.stdout:(139/156): python3-tempora-5.0.0-2.el9.noarch.r 2.1 MB/s | 36 kB 00:00 2024-09-19T07:51:05.506 INFO:teuthology.orchestra.run.smithi139.stdout:(140/156): python3-typing-extensions-4.12.2-2.e 2.9 MB/s | 74 kB 00:00 2024-09-19T07:51:05.531 INFO:teuthology.orchestra.run.smithi139.stdout:(141/156): python3-webob-1.8.8-2.el9.noarch.rpm 9.0 MB/s | 230 kB 00:00 2024-09-19T07:51:05.556 INFO:teuthology.orchestra.run.smithi139.stdout:(142/156): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-09-19T07:51:05.557 INFO:teuthology.orchestra.run.smithi162.stdout:(85/156): python3-oauthlib-3.1.1-5.el9.noarch.r 290 kB/s | 222 kB 00:00 2024-09-19T07:51:05.590 INFO:teuthology.orchestra.run.smithi139.stdout:(143/156): python3-werkzeug-2.0.3-3.el9.1.noarc 13 MB/s | 427 kB 00:00 2024-09-19T07:51:05.615 INFO:teuthology.orchestra.run.smithi139.stdout:(144/156): python3-zc-lockfile-2.0-10.el9.noarc 800 kB/s | 20 kB 00:00 2024-09-19T07:51:05.641 INFO:teuthology.orchestra.run.smithi139.stdout:(145/156): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-19T07:51:05.708 INFO:teuthology.orchestra.run.smithi139.stdout:(146/156): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-09-19T07:51:05.883 INFO:teuthology.orchestra.run.smithi139.stdout:(147/156): python3-babel-2.9.1-2.el9.noarch.rpm 783 kB/s | 6.0 MB 00:07 2024-09-19T07:51:05.932 INFO:teuthology.orchestra.run.smithi162.stdout:(86/156): python3-packaging-20.9-5.el9.noarch.r 114 kB/s | 77 kB 00:00 2024-09-19T07:51:06.016 INFO:teuthology.orchestra.run.smithi162.stdout:(87/156): python3-prettytable-0.7.2-27.el9.noar 72 kB/s | 42 kB 00:00 2024-09-19T07:51:06.083 INFO:teuthology.orchestra.run.smithi139.stdout:(148/156): librados2-19.1.1-361.g2f599a46.el9.x 9.1 MB/s | 3.4 MB 00:00 2024-09-19T07:51:06.200 INFO:teuthology.orchestra.run.smithi139.stdout:(149/156): librbd1-19.1.1-361.g2f599a46.el9.x86 10 MB/s | 3.2 MB 00:00 2024-09-19T07:51:06.350 INFO:teuthology.orchestra.run.smithi162.stdout:(88/156): python3-protobuf-3.14.0-13.el9.noarch 340 kB/s | 269 kB 00:00 2024-09-19T07:51:06.659 INFO:teuthology.orchestra.run.smithi139.stdout:(150/156): NetworkManager-libnm-1.51.0-1.el9.x8 3.9 MB/s | 1.8 MB 00:00 2024-09-19T07:51:06.666 INFO:teuthology.orchestra.run.smithi162.stdout:(89/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 216 kB/s | 159 kB 00:00 2024-09-19T07:51:06.701 INFO:teuthology.orchestra.run.smithi139.stdout:(151/156): NetworkManager-1.51.0-1.el9.x86_64.r 3.8 MB/s | 2.3 MB 00:00 2024-09-19T07:51:06.726 INFO:teuthology.orchestra.run.smithi139.stdout:(152/156): NetworkManager-team-1.51.0-1.el9.x86 614 kB/s | 41 kB 00:00 2024-09-19T07:51:06.739 INFO:teuthology.orchestra.run.smithi098.stdout:(155/156): postgresql-server-13.16-1.el9.x86_64 520 kB/s | 5.8 MB 00:11 2024-09-19T07:51:06.783 INFO:teuthology.orchestra.run.smithi162.stdout:(90/156): python3-pyasn1-modules-0.4.8-6.el9.no 364 kB/s | 279 kB 00:00 2024-09-19T07:51:06.834 INFO:teuthology.orchestra.run.smithi139.stdout:(153/156): NetworkManager-tui-1.51.0-1.el9.x86_ 1.8 MB/s | 249 kB 00:00 2024-09-19T07:51:06.975 INFO:teuthology.orchestra.run.smithi162.stdout:(91/156): python3-pytz-2021.1-5.el9.noarch.rpm 82 kB/s | 51 kB 00:00 2024-09-19T07:51:07.251 INFO:teuthology.orchestra.run.smithi139.stdout:(154/156): openssl-3.2.2-6.el9.x86_64.rpm 2.6 MB/s | 1.4 MB 00:00 2024-09-19T07:51:07.283 INFO:teuthology.orchestra.run.smithi162.stdout:(92/156): python3-requests-oauthlib-1.3.0-12.el 87 kB/s | 54 kB 00:00 2024-09-19T07:51:07.567 INFO:teuthology.orchestra.run.smithi162.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 70 kB/s | 42 kB 00:00 2024-09-19T07:51:07.760 INFO:teuthology.orchestra.run.smithi139.stdout:(155/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.6 MB/s | 16 MB 00:06 2024-09-19T07:51:07.835 INFO:teuthology.orchestra.run.smithi139.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 2.4 MB/s | 2.4 MB 00:01 2024-09-19T07:51:07.839 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-19T07:51:07.839 INFO:teuthology.orchestra.run.smithi139.stdout:Total 12 MB/s | 226 MB 00:19 2024-09-19T07:51:07.867 INFO:teuthology.orchestra.run.smithi162.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 76 kB/s | 44 kB 00:00 2024-09-19T07:51:08.419 INFO:teuthology.orchestra.run.smithi162.stdout:(95/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.9 MB/s | 16 MB 00:01 2024-09-19T07:51:08.560 INFO:teuthology.orchestra.run.smithi162.stdout:(96/156): qatlib-24.02.0-1.el9.x86_64.rpm 222 kB/s | 221 kB 00:00 2024-09-19T07:51:08.752 INFO:teuthology.orchestra.run.smithi162.stdout:(97/156): qatlib-service-24.02.0-1.el9.x86_64.r 41 kB/s | 36 kB 00:00 2024-09-19T07:51:08.994 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-19T07:51:09.068 INFO:teuthology.orchestra.run.smithi162.stdout:(98/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 73 kB/s | 47 kB 00:00 2024-09-19T07:51:09.143 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-19T07:51:09.143 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-19T07:51:09.335 INFO:teuthology.orchestra.run.smithi162.stdout:(99/156): socat-1.7.4.1-6.el9.x86_64.rpm 392 kB/s | 304 kB 00:00 2024-09-19T07:51:09.386 INFO:teuthology.orchestra.run.smithi162.stdout:(100/156): uuid-1.6.2-55.el9.x86_64.rpm 91 kB/s | 58 kB 00:00 2024-09-19T07:51:09.786 INFO:teuthology.orchestra.run.smithi162.stdout:(101/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 134 kB/s | 96 kB 00:00 2024-09-19T07:51:09.870 INFO:teuthology.orchestra.run.smithi162.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 269 kB/s | 22 kB 00:00 2024-09-19T07:51:09.953 INFO:teuthology.orchestra.run.smithi162.stdout:(103/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 103 kB/s | 64 kB 00:00 2024-09-19T07:51:09.978 INFO:teuthology.orchestra.run.smithi162.stdout:(104/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 76 kB/s | 45 kB 00:00 2024-09-19T07:51:10.029 INFO:teuthology.orchestra.run.smithi162.stdout:(105/156): protobuf-compiler-3.14.0-13.el9.x86_ 5.3 MB/s | 863 kB 00:00 2024-09-19T07:51:10.070 INFO:teuthology.orchestra.run.smithi162.stdout:(106/156): gperftools-libs-2.9.1-3.el9.x86_64.r 3.3 MB/s | 308 kB 00:00 2024-09-19T07:51:10.104 INFO:teuthology.orchestra.run.smithi162.stdout:(107/156): abseil-cpp-20211102.0-3.el9.x86_64.r 3.6 MB/s | 548 kB 00:00 2024-09-19T07:51:10.129 INFO:teuthology.orchestra.run.smithi162.stdout:(108/156): grpc-data-1.46.7-10.el9.noarch.rpm 197 kB/s | 19 kB 00:00 2024-09-19T07:51:10.163 INFO:teuthology.orchestra.run.smithi162.stdout:(109/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 444 kB/s | 26 kB 00:00 2024-09-19T07:51:10.197 INFO:teuthology.orchestra.run.smithi162.stdout:(110/156): liboath-2.6.7-2.el9.x86_64.rpm 724 kB/s | 49 kB 00:00 2024-09-19T07:51:10.222 INFO:teuthology.orchestra.run.smithi162.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-09-19T07:51:10.247 INFO:teuthology.orchestra.run.smithi162.stdout:(112/156): luarocks-3.9.2-5.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-09-19T07:51:10.289 INFO:teuthology.orchestra.run.smithi162.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-09-19T07:51:10.315 INFO:teuthology.orchestra.run.smithi162.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-19T07:51:10.357 INFO:teuthology.orchestra.run.smithi162.stdout:(115/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.1 MB/s | 839 kB 00:00 2024-09-19T07:51:10.374 INFO:teuthology.orchestra.run.smithi162.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 550 kB/s | 32 kB 00:00 2024-09-19T07:51:10.391 INFO:teuthology.orchestra.run.smithi162.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 419 kB/s | 14 kB 00:00 2024-09-19T07:51:10.416 INFO:teuthology.orchestra.run.smithi162.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-09-19T07:51:10.458 INFO:teuthology.orchestra.run.smithi162.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-09-19T07:51:10.483 INFO:teuthology.orchestra.run.smithi162.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-09-19T07:51:10.517 INFO:teuthology.orchestra.run.smithi162.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 3.7 MB/s | 220 kB 00:00 2024-09-19T07:51:10.550 INFO:teuthology.orchestra.run.smithi162.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-09-19T07:51:10.576 INFO:teuthology.orchestra.run.smithi162.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 429 kB/s | 11 kB 00:00 2024-09-19T07:51:10.601 INFO:teuthology.orchestra.run.smithi162.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 708 kB/s | 18 kB 00:00 2024-09-19T07:51:10.626 INFO:teuthology.orchestra.run.smithi162.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 934 kB/s | 23 kB 00:00 2024-09-19T07:51:10.652 INFO:teuthology.orchestra.run.smithi162.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 780 kB/s | 19 kB 00:00 2024-09-19T07:51:10.677 INFO:teuthology.orchestra.run.smithi162.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 790 kB/s | 20 kB 00:00 2024-09-19T07:51:10.761 INFO:teuthology.orchestra.run.smithi162.stdout:(128/156): libarrow-9.0.0-11.el9.x86_64.rpm 6.4 MB/s | 4.4 MB 00:00 2024-09-19T07:51:10.786 INFO:teuthology.orchestra.run.smithi162.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-19T07:51:10.811 INFO:teuthology.orchestra.run.smithi162.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-19T07:51:10.862 INFO:teuthology.orchestra.run.smithi162.stdout:(131/156): python3-grpcio-1.46.7-10.el9.x86_64. 5.4 MB/s | 2.0 MB 00:00 2024-09-19T07:51:10.895 INFO:teuthology.orchestra.run.smithi162.stdout:(132/156): python3-kubernetes-26.1.0-2.el9.noar 4.7 MB/s | 1.0 MB 00:00 2024-09-19T07:51:10.920 INFO:teuthology.orchestra.run.smithi162.stdout:(133/156): python3-natsort-7.1.1-5.el9.noarch.r 529 kB/s | 58 kB 00:00 2024-09-19T07:51:10.940 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-19T07:51:10.940 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-19T07:51:10.946 INFO:teuthology.orchestra.run.smithi162.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 326 kB/s | 16 kB 00:00 2024-09-19T07:51:10.971 INFO:teuthology.orchestra.run.smithi162.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-09-19T07:51:11.005 INFO:teuthology.orchestra.run.smithi162.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-09-19T07:51:11.030 INFO:teuthology.orchestra.run.smithi162.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 523 kB/s | 31 kB 00:00 2024-09-19T07:51:11.055 INFO:teuthology.orchestra.run.smithi162.stdout:(138/156): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-09-19T07:51:11.072 INFO:teuthology.orchestra.run.smithi162.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-09-19T07:51:11.098 INFO:teuthology.orchestra.run.smithi162.stdout:(140/156): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-09-19T07:51:11.114 INFO:teuthology.orchestra.run.smithi162.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 609 kB/s | 36 kB 00:00 2024-09-19T07:51:11.140 INFO:teuthology.orchestra.run.smithi162.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 1.1 MB/s | 74 kB 00:00 2024-09-19T07:51:11.165 INFO:teuthology.orchestra.run.smithi162.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-19T07:51:11.191 INFO:teuthology.orchestra.run.smithi162.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-19T07:51:11.207 INFO:teuthology.orchestra.run.smithi162.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 478 kB/s | 20 kB 00:00 2024-09-19T07:51:11.241 INFO:teuthology.orchestra.run.smithi162.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-19T07:51:11.275 INFO:teuthology.orchestra.run.smithi162.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-09-19T07:51:11.325 INFO:teuthology.orchestra.run.smithi162.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-19T07:51:11.584 INFO:teuthology.orchestra.run.smithi162.stdout:(149/156): librbd1-19.1.1-361.g2f599a46.el9.x86 10 MB/s | 3.2 MB 00:00 2024-09-19T07:51:11.651 INFO:teuthology.orchestra.run.smithi162.stdout:(150/156): librados2-19.1.1-361.g2f599a46.el9.x 8.4 MB/s | 3.4 MB 00:00 2024-09-19T07:51:12.084 INFO:teuthology.orchestra.run.smithi162.stdout:(151/156): NetworkManager-team-1.51.0-1.el9.x86 94 kB/s | 41 kB 00:00 2024-09-19T07:51:12.493 INFO:teuthology.orchestra.run.smithi162.stdout:(152/156): NetworkManager-1.51.0-1.el9.x86_64.r 2.0 MB/s | 2.3 MB 00:01 2024-09-19T07:51:12.651 INFO:teuthology.orchestra.run.smithi162.stdout:(153/156): openssl-3.2.2-6.el9.x86_64.rpm 8.6 MB/s | 1.4 MB 00:00 2024-09-19T07:51:12.676 INFO:teuthology.orchestra.run.smithi162.stdout:(154/156): NetworkManager-tui-1.51.0-1.el9.x86_ 420 kB/s | 249 kB 00:00 2024-09-19T07:51:12.943 INFO:teuthology.orchestra.run.smithi162.stdout:(155/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 8.2 MB/s | 2.4 MB 00:00 2024-09-19T07:51:13.019 INFO:teuthology.orchestra.run.smithi162.stdout:(156/156): NetworkManager-libnm-1.51.0-1.el9.x8 1.3 MB/s | 1.8 MB 00:01 2024-09-19T07:51:13.026 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-19T07:51:13.026 INFO:teuthology.orchestra.run.smithi162.stdout:Total 8.6 MB/s | 226 MB 00:26 2024-09-19T07:51:14.207 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-19T07:51:14.359 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-19T07:51:14.359 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-19T07:51:16.167 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-19T07:51:16.168 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-19T07:51:17.263 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-19T07:51:17.380 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-19T07:51:17.418 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-19T07:51:17.470 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-19T07:51:17.505 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-19T07:51:17.539 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-19T07:51:17.581 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-19T07:51:17.902 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-19T07:51:18.007 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librados2-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T07:51:18.129 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T07:51:18.158 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T07:51:18.249 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T07:51:18.279 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 10/164 2024-09-19T07:51:18.425 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-19T07:51:18.484 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-19T07:51:18.512 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T07:51:18.615 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T07:51:18.638 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 14/164 2024-09-19T07:51:18.700 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 14/164 2024-09-19T07:51:18.722 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-19T07:51:18.748 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-19T07:51:18.871 INFO:teuthology.orchestra.run.smithi098.stdout:(156/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 914 kB/s | 16 MB 00:18 2024-09-19T07:51:18.874 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-19T07:51:18.875 INFO:teuthology.orchestra.run.smithi098.stdout:Total 7.1 MB/s | 226 MB 00:31 2024-09-19T07:51:18.898 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-19T07:51:19.025 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-19T07:51:19.145 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 17/164 2024-09-19T07:51:19.248 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T07:51:19.313 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T07:51:19.517 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T07:51:19.625 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-19T07:51:19.713 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-19T07:51:19.743 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-19T07:51:19.778 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-19T07:51:19.843 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-19T07:51:19.895 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-19T07:51:19.927 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-19T07:51:19.971 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-19T07:51:20.003 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-19T07:51:20.031 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-19T07:51:20.100 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-19T07:51:20.120 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-19T07:51:20.153 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-19T07:51:20.184 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-19T07:51:20.227 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-19T07:51:20.270 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-19T07:51:20.270 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-19T07:51:20.294 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-19T07:51:20.330 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x 34/164 2024-09-19T07:51:20.373 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9 35/164 2024-09-19T07:51:20.402 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 36/164 2024-09-19T07:51:20.472 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-19T07:51:20.517 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-19T07:51:20.595 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-19T07:51:20.725 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-19T07:51:20.780 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-19T07:51:21.301 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-19T07:51:21.385 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-19T07:51:21.416 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-19T07:51:21.456 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-19T07:51:21.494 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 46/164 2024-09-19T07:51:21.536 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-19T07:51:21.706 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-19T07:51:21.773 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-19T07:51:21.827 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-19T07:51:21.908 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-19T07:51:21.940 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-19T07:51:21.976 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-19T07:51:22.022 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-19T07:51:22.059 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-19T07:51:22.102 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-19T07:51:22.143 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-19T07:51:22.286 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-19T07:51:22.354 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-19T07:51:22.395 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-19T07:51:22.466 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-19T07:51:22.466 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-19T07:51:22.497 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-19T07:51:22.537 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-19T07:51:22.576 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-19T07:51:22.602 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-19T07:51:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-19T07:51:22.626 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-19T07:51:22.702 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-19T07:51:22.755 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-19T07:51:22.793 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-19T07:51:22.827 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-19T07:51:22.855 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-19T07:51:22.889 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-19T07:51:22.915 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-19T07:51:22.952 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-19T07:51:22.958 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-19T07:51:23.095 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-19T07:51:23.130 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-19T07:51:23.169 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-19T07:51:23.209 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-19T07:51:23.266 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-19T07:51:23.273 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-19T07:51:23.291 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-19T07:51:23.306 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-19T07:51:23.348 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-19T07:51:23.382 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-19T07:51:23.407 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T07:51:23.471 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-19T07:51:23.512 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-19T07:51:23.530 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T07:51:23.543 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-19T07:51:23.550 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T07:51:23.641 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T07:51:23.645 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-19T07:51:23.671 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 10/164 2024-09-19T07:51:23.694 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-19T07:51:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-19T07:51:23.876 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-19T07:51:23.905 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T07:51:24.008 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T07:51:24.031 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 14/164 2024-09-19T07:51:24.093 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 14/164 2024-09-19T07:51:24.115 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-19T07:51:24.140 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-19T07:51:24.333 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-19T07:51:24.418 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-19T07:51:24.466 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-19T07:51:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-19T07:51:24.500 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-19T07:51:24.596 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-19T07:51:24.597 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 17/164 2024-09-19T07:51:24.647 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-19T07:51:24.691 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-19T07:51:24.694 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T07:51:24.766 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T07:51:24.779 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-19T07:51:24.819 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-19T07:51:25.002 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-19T07:51:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T07:51:25.034 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-19T07:51:25.111 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-19T07:51:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-19T07:51:25.218 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-19T07:51:25.254 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-19T07:51:25.289 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-19T07:51:25.362 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-19T07:51:25.415 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-19T07:51:25.455 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-19T07:51:25.500 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-19T07:51:25.531 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-19T07:51:25.559 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-19T07:51:25.629 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-19T07:51:25.682 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-19T07:51:25.712 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-19T07:51:25.747 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-19T07:51:25.750 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-19T07:51:25.815 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-19T07:51:25.850 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x 34/164 2024-09-19T07:51:25.865 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-19T07:51:25.893 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9 35/164 2024-09-19T07:51:25.922 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 36/164 2024-09-19T07:51:26.000 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-19T07:51:26.045 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-19T07:51:26.119 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-19T07:51:26.253 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-19T07:51:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-19T07:51:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-19T07:51:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-19T07:51:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-19T07:51:26.977 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-19T07:51:27.016 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 46/164 2024-09-19T07:51:27.058 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-19T07:51:27.083 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-19T07:51:27.155 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-19T07:51:27.183 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-19T07:51:27.216 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-19T07:51:27.222 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-19T07:51:27.285 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-19T07:51:27.341 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-19T07:51:27.414 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-19T07:51:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-19T07:51:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-19T07:51:27.526 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-19T07:51:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-19T07:51:27.564 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-19T07:51:27.570 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 101/164 2024-09-19T07:51:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-19T07:51:27.644 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 101/164 2024-09-19T07:51:27.649 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-19T07:51:27.675 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 102/164 2024-09-19T07:51:27.711 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-19T07:51:27.794 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-19T07:51:27.843 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-19T07:51:27.884 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-19T07:51:27.987 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-19T07:51:28.018 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-19T07:51:28.057 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-19T07:51:28.088 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-19T07:51:28.113 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-19T07:51:28.184 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-19T07:51:28.188 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-19T07:51:28.223 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 105/164 2024-09-19T07:51:28.294 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 105/164 2024-09-19T07:51:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-19T07:51:28.325 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 106/164 2024-09-19T07:51:28.346 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-19T07:51:28.374 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-19T07:51:28.408 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-19T07:51:28.456 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-19T07:51:28.487 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-19T07:51:28.517 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-19T07:51:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-19T07:51:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-19T07:51:28.627 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-19T07:51:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-19T07:51:28.748 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-19T07:51:28.792 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-19T07:51:28.823 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-19T07:51:28.844 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-19T07:51:28.927 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-19T07:51:28.982 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-19T07:51:28.984 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-19T07:51:29.024 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-19T07:51:29.089 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-19T07:51:29.134 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-19T07:51:29.160 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-19T07:51:29.217 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-19T07:51:29.543 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-19T07:51:29.653 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T07:51:29.701 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-19T07:51:29.746 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-19T07:51:29.775 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T07:51:29.797 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-19T07:51:29.803 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T07:51:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-19T07:51:29.895 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T07:51:29.925 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 10/164 2024-09-19T07:51:29.952 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-19T07:51:29.997 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-19T07:51:30.087 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-19T07:51:30.091 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-19T07:51:30.125 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-19T07:51:30.147 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-19T07:51:30.175 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T07:51:30.280 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T07:51:30.309 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 14/164 2024-09-19T07:51:30.313 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-19T07:51:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-19T07:51:30.372 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 14/164 2024-09-19T07:51:30.393 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-19T07:51:30.417 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-19T07:51:30.417 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-19T07:51:30.580 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T07:51:30.613 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-19T07:51:30.621 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T07:51:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-19T07:51:30.909 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 17/164 2024-09-19T07:51:31.027 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T07:51:31.058 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-19T07:51:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T07:51:31.162 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T07:51:31.172 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-19T07:51:31.207 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 108/164 2024-09-19T07:51:31.299 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 108/164 2024-09-19T07:51:31.300 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-19T07:51:31.300 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-19T07:51:31.300 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:51:31.342 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 109/164 2024-09-19T07:51:31.365 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T07:51:31.474 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-19T07:51:31.568 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-19T07:51:31.599 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-19T07:51:31.650 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-19T07:51:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-19T07:51:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-19T07:51:31.808 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-19T07:51:31.851 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-19T07:51:31.884 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-19T07:51:31.912 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-19T07:51:31.984 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-19T07:51:32.229 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-19T07:51:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-19T07:51:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-19T07:51:32.404 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-19T07:51:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-19T07:51:32.478 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x 34/164 2024-09-19T07:51:32.495 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-19T07:51:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9 35/164 2024-09-19T07:51:32.522 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-19T07:51:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 36/164 2024-09-19T07:51:32.556 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-19T07:51:32.620 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-19T07:51:32.665 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-19T07:51:32.748 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-19T07:51:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-19T07:51:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-19T07:51:32.910 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 101/164 2024-09-19T07:51:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-19T07:51:32.984 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 101/164 2024-09-19T07:51:33.015 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 102/164 2024-09-19T07:51:33.051 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-19T07:51:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-19T07:51:33.527 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-19T07:51:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-19T07:51:33.563 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 105/164 2024-09-19T07:51:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-19T07:51:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 105/164 2024-09-19T07:51:33.643 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-19T07:51:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 106/164 2024-09-19T07:51:33.681 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 46/164 2024-09-19T07:51:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-19T07:51:33.886 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-19T07:51:33.935 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-19T07:51:33.989 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-19T07:51:34.062 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-19T07:51:34.102 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-19T07:51:34.138 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-19T07:51:34.186 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-19T07:51:34.221 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-19T07:51:34.264 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-19T07:51:34.307 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-19T07:51:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-19T07:51:34.483 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-19T07:51:34.533 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-19T07:51:34.644 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-19T07:51:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-19T07:51:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-19T07:51:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-19T07:51:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-19T07:51:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-19T07:51:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-19T07:51:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-19T07:51:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-19T07:51:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-19T07:51:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-19T07:51:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-19T07:51:35.222 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-19T07:51:35.258 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-19T07:51:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-19T07:51:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-19T07:51:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-19T07:51:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-19T07:51:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-19T07:51:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-19T07:51:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-19T07:51:35.713 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-19T07:51:35.979 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T07:51:36.012 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T07:51:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-19T07:51:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-19T07:51:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-19T07:51:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T07:51:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-19T07:51:36.617 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 108/164 2024-09-19T07:51:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-19T07:51:36.701 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 108/164 2024-09-19T07:51:36.701 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-19T07:51:36.701 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-19T07:51:36.702 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:51:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-19T07:51:36.735 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 109/164 2024-09-19T07:51:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-19T07:51:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-19T07:51:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-19T07:51:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-19T07:51:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-19T07:51:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-19T07:51:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-19T07:51:39.162 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-19T07:51:39.246 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-19T07:51:39.273 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-19T07:51:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-19T07:51:39.616 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-19T07:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 101/164 2024-09-19T07:51:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 101/164 2024-09-19T07:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 102/164 2024-09-19T07:51:39.827 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-19T07:51:40.315 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-19T07:51:40.348 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 105/164 2024-09-19T07:51:40.419 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 105/164 2024-09-19T07:51:40.458 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 106/164 2024-09-19T07:51:42.776 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T07:51:43.055 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T07:51:43.193 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 109/164 2024-09-19T07:51:43.193 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /sys 2024-09-19T07:51:43.193 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /proc 2024-09-19T07:51:43.194 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /mnt 2024-09-19T07:51:43.194 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /var/tmp 2024-09-19T07:51:43.194 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /home 2024-09-19T07:51:43.194 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /root 2024-09-19T07:51:43.194 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /tmp 2024-09-19T07:51:43.194 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:51:43.377 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 110/164 2024-09-19T07:51:43.393 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 110/164 2024-09-19T07:51:43.393 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:43.393 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-19T07:51:43.393 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:51:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T07:51:43.658 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 108/164 2024-09-19T07:51:43.741 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 108/164 2024-09-19T07:51:43.741 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-19T07:51:43.742 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-19T07:51:43.744 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:51:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 109/164 2024-09-19T07:51:44.871 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 111/164 2024-09-19T07:51:44.897 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 111/164 2024-09-19T07:51:44.898 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:44.898 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-19T07:51:44.898 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-19T07:51:44.898 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-19T07:51:44.898 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:51:45.130 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 112/164 2024-09-19T07:51:45.153 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 112/164 2024-09-19T07:51:45.153 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:45.153 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-19T07:51:45.153 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-19T07:51:45.153 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-19T07:51:45.153 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:51:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 113/164 2024-09-19T07:51:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 113/164 2024-09-19T07:51:45.612 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:45.612 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-19T07:51:45.612 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-19T07:51:45.612 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-19T07:51:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:51:45.719 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-19T07:51:45.776 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-19T07:51:45.839 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-19T07:51:46.025 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-19T07:51:46.073 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-19T07:51:46.151 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-19T07:51:46.240 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-19T07:51:46.378 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-19T07:51:46.431 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-19T07:51:46.463 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-19T07:51:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-19T07:51:46.919 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:19.1.1-361.g2f599a46.el9.noarch 125/164 2024-09-19T07:51:46.948 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephadm-2:19.1.1-361.g2f599a46.el9.noarch 125/164 2024-09-19T07:51:46.979 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el 126/164 2024-09-19T07:51:47.042 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.e 127/164 2024-09-19T07:51:47.071 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 128/164 2024-09-19T07:51:48.112 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 128/164 2024-09-19T07:51:48.147 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 129/164 2024-09-19T07:51:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 109/164 2024-09-19T07:51:48.341 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-09-19T07:51:48.341 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-09-19T07:51:48.341 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-09-19T07:51:48.341 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-09-19T07:51:48.341 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-09-19T07:51:48.341 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /root 2024-09-19T07:51:48.341 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-09-19T07:51:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:51:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 110/164 2024-09-19T07:51:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 110/164 2024-09-19T07:51:48.522 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:48.522 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-19T07:51:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:51:49.105 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 129/164 2024-09-19T07:51:49.131 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 130/164 2024-09-19T07:51:49.232 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 130/164 2024-09-19T07:51:49.401 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9 131/164 2024-09-19T07:51:49.441 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 132/164 2024-09-19T07:51:49.465 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 132/164 2024-09-19T07:51:49.465 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:49.465 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-19T07:51:49.465 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-19T07:51:49.465 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-19T07:51:49.465 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:51:49.517 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 133/164 2024-09-19T07:51:49.528 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 133/164 2024-09-19T07:51:49.627 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-2:19.1.1-361.g2f599a46.el9.x86_64 134/164 2024-09-19T07:51:49.673 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 135/164 2024-09-19T07:51:50.002 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 111/164 2024-09-19T07:51:50.029 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 111/164 2024-09-19T07:51:50.030 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:50.030 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-19T07:51:50.030 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-19T07:51:50.030 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-19T07:51:50.030 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:51:50.262 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 112/164 2024-09-19T07:51:50.285 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 112/164 2024-09-19T07:51:50.285 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:50.285 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-19T07:51:50.285 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-19T07:51:50.285 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-19T07:51:50.286 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:51:50.595 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-19T07:51:50.628 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 137/164 2024-09-19T07:51:50.651 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 137/164 2024-09-19T07:51:50.651 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:50.651 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-19T07:51:50.651 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-19T07:51:50.651 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-19T07:51:50.651 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:51:50.695 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-361.g2f599a 138/164 2024-09-19T07:51:50.717 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-361.g2f599a 138/164 2024-09-19T07:51:50.717 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:50.717 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-19T07:51:50.717 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:51:50.721 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 113/164 2024-09-19T07:51:50.744 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 113/164 2024-09-19T07:51:50.744 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:50.744 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-19T07:51:50.745 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-19T07:51:50.745 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-19T07:51:50.745 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:51:50.762 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 139/164 2024-09-19T07:51:50.785 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 139/164 2024-09-19T07:51:50.785 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:50.785 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-19T07:51:50.785 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-19T07:51:50.785 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-19T07:51:50.785 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:51:50.859 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-19T07:51:50.916 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-19T07:51:50.971 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-19T07:51:51.088 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 140/164 2024-09-19T07:51:51.110 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 140/164 2024-09-19T07:51:51.111 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:51.111 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-19T07:51:51.111 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-19T07:51:51.111 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-19T07:51:51.111 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:51:51.157 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-19T07:51:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-19T07:51:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-19T07:51:51.373 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-19T07:51:51.512 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-19T07:51:51.556 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-19T07:51:51.588 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-19T07:51:51.616 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-19T07:51:52.062 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.1.1-361.g2f599a46.el9.noarch 125/164 2024-09-19T07:51:52.090 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:19.1.1-361.g2f599a46.el9.noarch 125/164 2024-09-19T07:51:52.121 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el 126/164 2024-09-19T07:51:52.184 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.e 127/164 2024-09-19T07:51:52.213 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 128/164 2024-09-19T07:51:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 128/164 2024-09-19T07:51:53.316 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 129/164 2024-09-19T07:51:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 129/164 2024-09-19T07:51:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 130/164 2024-09-19T07:51:54.425 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 130/164 2024-09-19T07:51:54.579 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9 131/164 2024-09-19T07:51:54.609 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 132/164 2024-09-19T07:51:54.633 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 132/164 2024-09-19T07:51:54.633 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:54.633 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-19T07:51:54.633 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-19T07:51:54.633 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-19T07:51:54.633 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:51:54.684 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 133/164 2024-09-19T07:51:54.696 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 133/164 2024-09-19T07:51:54.794 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:19.1.1-361.g2f599a46.el9.x86_64 134/164 2024-09-19T07:51:54.850 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 135/164 2024-09-19T07:51:55.650 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 141/164 2024-09-19T07:51:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 109/164 2024-09-19T07:51:55.691 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-09-19T07:51:55.691 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-09-19T07:51:55.691 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-09-19T07:51:55.691 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-09-19T07:51:55.691 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-09-19T07:51:55.691 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-09-19T07:51:55.691 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-09-19T07:51:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:51:55.692 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 142/164 2024-09-19T07:51:55.732 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 143/164 2024-09-19T07:51:55.764 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-19T07:51:55.785 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-19T07:51:55.791 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-19T07:51:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 137/164 2024-09-19T07:51:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 137/164 2024-09-19T07:51:55.836 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:55.836 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-19T07:51:55.836 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-19T07:51:55.836 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-19T07:51:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:51:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 110/164 2024-09-19T07:51:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-361.g2f599a 138/164 2024-09-19T07:51:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 110/164 2024-09-19T07:51:55.890 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:55.890 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-19T07:51:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:51:55.902 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-361.g2f599a 138/164 2024-09-19T07:51:55.902 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:55.902 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-19T07:51:55.902 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:51:55.947 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 139/164 2024-09-19T07:51:55.970 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 139/164 2024-09-19T07:51:55.970 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:55.970 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-19T07:51:55.971 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-19T07:51:55.971 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-19T07:51:55.971 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:51:55.974 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 146/164 2024-09-19T07:51:56.013 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-19T07:51:56.043 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_6 148/164 2024-09-19T07:51:56.073 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 149/164 2024-09-19T07:51:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 150/164 2024-09-19T07:51:56.273 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 140/164 2024-09-19T07:51:56.296 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 140/164 2024-09-19T07:51:56.297 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:56.297 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-19T07:51:56.297 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-19T07:51:56.297 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-19T07:51:56.297 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:51:56.698 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T07:51:56.733 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T07:51:56.753 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T07:51:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 111/164 2024-09-19T07:51:57.403 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 111/164 2024-09-19T07:51:57.403 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:57.403 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-19T07:51:57.403 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-19T07:51:57.403 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-19T07:51:57.404 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:51:57.583 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 152/164 2024-09-19T07:51:57.627 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 112/164 2024-09-19T07:51:57.650 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 112/164 2024-09-19T07:51:57.650 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:57.650 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-19T07:51:57.651 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-19T07:51:57.651 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-19T07:51:57.651 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:51:57.679 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-19T07:51:57.721 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-19T07:51:57.809 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-19T07:51:57.840 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-19T07:51:57.866 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-19T07:51:57.892 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-19T07:51:57.892 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-19T07:51:57.976 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-19T07:51:57.976 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-19T07:51:58.060 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-19T07:51:58.085 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-19T07:51:58.089 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T07:51:58.090 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T07:51:58.096 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 113/164 2024-09-19T07:51:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 113/164 2024-09-19T07:51:58.118 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:51:58.118 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-19T07:51:58.118 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-19T07:51:58.118 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-19T07:51:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:51:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-19T07:51:58.270 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T07:51:58.270 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-19T07:51:58.272 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-19T07:51:58.329 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-19T07:51:58.353 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-19T07:51:58.416 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T07:51:58.513 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-19T07:51:58.570 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-19T07:51:58.649 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-19T07:51:58.737 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-19T07:51:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-19T07:51:58.925 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-19T07:51:58.969 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-19T07:51:59.005 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-19T07:51:59.392 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.1.1-361.g2f599a46.el9.noarch 125/164 2024-09-19T07:51:59.420 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.1.1-361.g2f599a46.el9.noarch 125/164 2024-09-19T07:51:59.452 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el 126/164 2024-09-19T07:51:59.514 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.e 127/164 2024-09-19T07:51:59.544 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 128/164 2024-09-19T07:52:00.589 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 128/164 2024-09-19T07:52:00.629 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 129/164 2024-09-19T07:52:00.885 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 141/164 2024-09-19T07:52:00.926 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 142/164 2024-09-19T07:52:00.968 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 143/164 2024-09-19T07:52:00.999 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-19T07:52:01.026 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-19T07:52:01.220 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 146/164 2024-09-19T07:52:01.265 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-19T07:52:01.295 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_6 148/164 2024-09-19T07:52:01.325 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 149/164 2024-09-19T07:52:01.372 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 150/164 2024-09-19T07:52:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 129/164 2024-09-19T07:52:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 130/164 2024-09-19T07:52:01.726 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 130/164 2024-09-19T07:52:01.884 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9 131/164 2024-09-19T07:52:01.915 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 132/164 2024-09-19T07:52:01.940 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 132/164 2024-09-19T07:52:01.940 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:52:01.940 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-19T07:52:01.940 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-19T07:52:01.940 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-19T07:52:01.940 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:52:01.963 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T07:52:01.991 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 133/164 2024-09-19T07:52:02.003 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 133/164 2024-09-19T07:52:02.012 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T07:52:02.032 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T07:52:02.101 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.1.1-361.g2f599a46.el9.x86_64 134/164 2024-09-19T07:52:02.167 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 135/164 2024-09-19T07:52:02.883 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 152/164 2024-09-19T07:52:02.976 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-19T07:52:03.017 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-19T07:52:03.086 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-19T07:52:03.099 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-19T07:52:03.127 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 137/164 2024-09-19T07:52:03.129 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-19T07:52:03.151 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 137/164 2024-09-19T07:52:03.151 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:52:03.151 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-19T07:52:03.151 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-19T07:52:03.152 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-19T07:52:03.152 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:52:03.163 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-19T07:52:03.189 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-19T07:52:03.189 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-19T07:52:03.195 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-361.g2f599a 138/164 2024-09-19T07:52:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-361.g2f599a 138/164 2024-09-19T07:52:03.218 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:52:03.218 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-19T07:52:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:52:03.265 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-19T07:52:03.265 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-19T07:52:03.270 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 139/164 2024-09-19T07:52:03.294 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 139/164 2024-09-19T07:52:03.294 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:52:03.294 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-19T07:52:03.294 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-19T07:52:03.294 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-19T07:52:03.294 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:52:03.357 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-19T07:52:03.382 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-19T07:52:03.387 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T07:52:03.387 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T07:52:03.594 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T07:52:03.594 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-19T07:52:03.596 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 140/164 2024-09-19T07:52:03.620 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 140/164 2024-09-19T07:52:03.620 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T07:52:03.620 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-19T07:52:03.620 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-19T07:52:03.620 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-19T07:52:03.620 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:52:03.685 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-19T07:52:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T07:52:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T07:52:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-2:19.1.1-361.g2f599a46.el9.x86_64 1/164 2024-09-19T07:52:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 2/164 2024-09-19T07:52:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 3/164 2024-09-19T07:52:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 4/164 2024-09-19T07:52:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-361.g2f599a 5/164 2024-09-19T07:52:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 6/164 2024-09-19T07:52:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 7/164 2024-09-19T07:52:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T07:52:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T07:52:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 10/164 2024-09-19T07:52:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 11/164 2024-09-19T07:52:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 12/164 2024-09-19T07:52:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T07:52:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 14/164 2024-09-19T07:52:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_6 15/164 2024-09-19T07:52:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 16/164 2024-09-19T07:52:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 17/164 2024-09-19T07:52:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 18/164 2024-09-19T07:52:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 19/164 2024-09-19T07:52:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 20/164 2024-09-19T07:52:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9 21/164 2024-09-19T07:52:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x 22/164 2024-09-19T07:52:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 23/164 2024-09-19T07:52:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 24/164 2024-09-19T07:52:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 25/164 2024-09-19T07:52:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 26/164 2024-09-19T07:52:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 27/164 2024-09-19T07:52:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 28/164 2024-09-19T07:52:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 29/164 2024-09-19T07:52:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.e 30/164 2024-09-19T07:52:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 31/164 2024-09-19T07:52:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 32/164 2024-09-19T07:52:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 33/164 2024-09-19T07:52:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9 34/164 2024-09-19T07:52:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 35/164 2024-09-19T07:52:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el 36/164 2024-09-19T07:52:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 37/164 2024-09-19T07:52:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephadm-2:19.1.1-361.g2f599a46.el9.noarch 38/164 2024-09-19T07:52:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 39/164 2024-09-19T07:52:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-19T07:52:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-19T07:52:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-19T07:52:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-19T07:52:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-19T07:52:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-19T07:52:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-19T07:52:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-19T07:52:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-19T07:52:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-19T07:52:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-19T07:52:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-19T07:52:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-19T07:52:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-19T07:52:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-19T07:52:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-19T07:52:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-19T07:52:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-19T07:52:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-19T07:52:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-19T07:52:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-19T07:52:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-19T07:52:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-19T07:52:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-19T07:52:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-19T07:52:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-19T07:52:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-19T07:52:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-19T07:52:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-19T07:52:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-19T07:52:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-19T07:52:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-19T07:52:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-19T07:52:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-19T07:52:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-19T07:52:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-19T07:52:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-19T07:52:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-19T07:52:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-19T07:52:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-19T07:52:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-19T07:52:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-19T07:52:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-19T07:52:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-19T07:52:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-19T07:52:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-19T07:52:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-19T07:52:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-19T07:52:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-19T07:52:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-19T07:52:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-19T07:52:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-19T07:52:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-19T07:52:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-19T07:52:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-19T07:52:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-19T07:52:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-19T07:52:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-19T07:52:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-19T07:52:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-19T07:52:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-19T07:52:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-19T07:52:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-19T07:52:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-19T07:52:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-19T07:52:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-19T07:52:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-19T07:52:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-19T07:52:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-19T07:52:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-19T07:52:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-19T07:52:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-19T07:52:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-19T07:52:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-19T07:52:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-19T07:52:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-19T07:52:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-19T07:52:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-19T07:52:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-19T07:52:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-19T07:52:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-19T07:52:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-19T07:52:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-19T07:52:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-19T07:52:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-19T07:52:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-19T07:52:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-19T07:52:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-19T07:52:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-19T07:52:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-19T07:52:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-19T07:52:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-19T07:52:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-19T07:52:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-19T07:52:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-19T07:52:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:19.1.1-361.g2f599a46.el9.x86_64 149/164 2024-09-19T07:52:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-19T07:52:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 151/164 2024-09-19T07:52:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-19T07:52:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 153/164 2024-09-19T07:52:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-19T07:52:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 155/164 2024-09-19T07:52:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-19T07:52:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 157/164 2024-09-19T07:52:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-19T07:52:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 159/164 2024-09-19T07:52:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-19T07:52:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-19T07:52:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-19T07:52:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-19T07:52:06.447 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T07:52:06.447 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:52:06.447 INFO:teuthology.orchestra.run.smithi139.stdout:Upgraded: 2024-09-19T07:52:06.447 INFO:teuthology.orchestra.run.smithi139.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-19T07:52:06.447 INFO:teuthology.orchestra.run.smithi139.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-19T07:52:06.448 INFO:teuthology.orchestra.run.smithi139.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-19T07:52:06.448 INFO:teuthology.orchestra.run.smithi139.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-19T07:52:06.448 INFO:teuthology.orchestra.run.smithi139.stdout: librados2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.448 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.448 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-19T07:52:06.448 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-19T07:52:06.448 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-19T07:52:06.448 INFO:teuthology.orchestra.run.smithi139.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-19T07:52:06.448 INFO:teuthology.orchestra.run.smithi139.stdout: automake-1.16.2-8.el9.noarch 2024-09-19T07:52:06.448 INFO:teuthology.orchestra.run.smithi139.stdout: bison-3.7.4-5.el9.x86_64 2024-09-19T07:52:06.448 INFO:teuthology.orchestra.run.smithi139.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-19T07:52:06.448 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.449 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.449 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.449 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.449 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:06.449 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.449 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.449 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.449 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:06.449 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:06.449 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:06.449 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:06.449 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:06.450 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.450 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.450 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:06.450 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.450 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.450 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.450 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:06.450 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:06.450 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.450 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.450 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:06.450 INFO:teuthology.orchestra.run.smithi139.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-19T07:52:06.450 INFO:teuthology.orchestra.run.smithi139.stdout: flex-2.6.4-9.el9.x86_64 2024-09-19T07:52:06.450 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-19T07:52:06.451 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-19T07:52:06.451 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-19T07:52:06.451 INFO:teuthology.orchestra.run.smithi139.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-19T07:52:06.451 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-19T07:52:06.451 INFO:teuthology.orchestra.run.smithi139.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-19T07:52:06.451 INFO:teuthology.orchestra.run.smithi139.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-19T07:52:06.451 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-19T07:52:06.451 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-19T07:52:06.451 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.451 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.451 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.451 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-19T07:52:06.452 INFO:teuthology.orchestra.run.smithi139.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-19T07:52:06.452 INFO:teuthology.orchestra.run.smithi139.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-19T07:52:06.452 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-19T07:52:06.452 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-19T07:52:06.452 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.452 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.452 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-19T07:52:06.452 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.452 INFO:teuthology.orchestra.run.smithi139.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-19T07:52:06.452 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-19T07:52:06.452 INFO:teuthology.orchestra.run.smithi139.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-19T07:52:06.452 INFO:teuthology.orchestra.run.smithi139.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-19T07:52:06.452 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-19T07:52:06.453 INFO:teuthology.orchestra.run.smithi139.stdout: lua-5.4.4-4.el9.x86_64 2024-09-19T07:52:06.453 INFO:teuthology.orchestra.run.smithi139.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-19T07:52:06.453 INFO:teuthology.orchestra.run.smithi139.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-19T07:52:06.453 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-19T07:52:06.453 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-19T07:52:06.453 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-19T07:52:06.453 INFO:teuthology.orchestra.run.smithi139.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-19T07:52:06.453 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-13.16-1.el9.x86_64 2024-09-19T07:52:06.453 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-19T07:52:06.453 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-19T07:52:06.453 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-19T07:52:06.453 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-19T07:52:06.453 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-19T07:52:06.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-19T07:52:06.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-19T07:52:06.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-19T07:52:06.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-19T07:52:06.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-19T07:52:06.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-19T07:52:06.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-19T07:52:06.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-19T07:52:06.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-19T07:52:06.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-19T07:52:06.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-19T07:52:06.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-19T07:52:06.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-19T07:52:06.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-19T07:52:06.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-19T07:52:06.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-19T07:52:06.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-19T07:52:06.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-19T07:52:06.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-19T07:52:06.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-19T07:52:06.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-19T07:52:06.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-19T07:52:06.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-19T07:52:06.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-19T07:52:06.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-19T07:52:06.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-19T07:52:06.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-19T07:52:06.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-19T07:52:06.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-19T07:52:06.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-19T07:52:06.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-19T07:52:06.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-19T07:52:06.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-19T07:52:06.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-19T07:52:06.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-19T07:52:06.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ply-3.11-14.el9.noarch 2024-09-19T07:52:06.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-19T07:52:06.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-19T07:52:06.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-19T07:52:06.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-19T07:52:06.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-19T07:52:06.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-19T07:52:06.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-19T07:52:06.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-19T07:52:06.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-19T07:52:06.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-19T07:52:06.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-19T07:52:06.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-19T07:52:06.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-19T07:52:06.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-19T07:52:06.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-19T07:52:06.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-19T07:52:06.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-19T07:52:06.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-19T07:52:06.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-19T07:52:06.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-19T07:52:06.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-19T07:52:06.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-19T07:52:06.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-19T07:52:06.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-19T07:52:06.459 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-19T07:52:06.459 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-19T07:52:06.459 INFO:teuthology.orchestra.run.smithi139.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-19T07:52:06.459 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.459 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.459 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:06.459 INFO:teuthology.orchestra.run.smithi139.stdout: re2-1:20211101-20.el9.x86_64 2024-09-19T07:52:06.459 INFO:teuthology.orchestra.run.smithi139.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-19T07:52:06.459 INFO:teuthology.orchestra.run.smithi139.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-19T07:52:06.459 INFO:teuthology.orchestra.run.smithi139.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-19T07:52:06.459 INFO:teuthology.orchestra.run.smithi139.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-19T07:52:06.459 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-19T07:52:06.460 INFO:teuthology.orchestra.run.smithi139.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-19T07:52:06.460 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:52:06.460 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-19T07:52:06.851 DEBUG:teuthology.parallel:result is None 2024-09-19T07:52:08.200 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 141/164 2024-09-19T07:52:08.241 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 142/164 2024-09-19T07:52:08.281 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 143/164 2024-09-19T07:52:08.315 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-19T07:52:08.342 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-19T07:52:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 146/164 2024-09-19T07:52:08.589 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-19T07:52:08.619 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_6 148/164 2024-09-19T07:52:08.649 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 149/164 2024-09-19T07:52:08.695 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 150/164 2024-09-19T07:52:09.257 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T07:52:09.288 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T07:52:09.628 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:19.1.1-361.g2f599a46.el9.x86_64 1/164 2024-09-19T07:52:09.628 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 2/164 2024-09-19T07:52:09.628 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 3/164 2024-09-19T07:52:09.628 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 4/164 2024-09-19T07:52:09.628 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-361.g2f599a 5/164 2024-09-19T07:52:09.628 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 6/164 2024-09-19T07:52:09.628 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 7/164 2024-09-19T07:52:09.628 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T07:52:09.628 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T07:52:09.628 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 10/164 2024-09-19T07:52:09.628 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 11/164 2024-09-19T07:52:09.629 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 12/164 2024-09-19T07:52:09.629 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T07:52:09.629 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 14/164 2024-09-19T07:52:09.629 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_6 15/164 2024-09-19T07:52:09.629 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 16/164 2024-09-19T07:52:09.629 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 17/164 2024-09-19T07:52:09.629 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 18/164 2024-09-19T07:52:09.629 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 19/164 2024-09-19T07:52:09.629 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 20/164 2024-09-19T07:52:09.629 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9 21/164 2024-09-19T07:52:09.629 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x 22/164 2024-09-19T07:52:09.629 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 23/164 2024-09-19T07:52:09.629 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 24/164 2024-09-19T07:52:09.630 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 25/164 2024-09-19T07:52:09.630 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 26/164 2024-09-19T07:52:09.630 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 27/164 2024-09-19T07:52:09.630 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 28/164 2024-09-19T07:52:09.630 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 29/164 2024-09-19T07:52:09.630 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.e 30/164 2024-09-19T07:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T07:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T07:52:09.642 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 31/164 2024-09-19T07:52:09.642 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 32/164 2024-09-19T07:52:09.642 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 33/164 2024-09-19T07:52:09.642 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9 34/164 2024-09-19T07:52:09.642 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 35/164 2024-09-19T07:52:09.642 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el 36/164 2024-09-19T07:52:09.642 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 37/164 2024-09-19T07:52:09.642 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:19.1.1-361.g2f599a46.el9.noarch 38/164 2024-09-19T07:52:09.643 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 39/164 2024-09-19T07:52:09.643 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-19T07:52:09.643 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-19T07:52:09.643 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-19T07:52:09.643 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-19T07:52:09.643 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-19T07:52:09.643 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-19T07:52:09.643 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-19T07:52:09.643 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-19T07:52:09.643 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-19T07:52:09.643 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-19T07:52:09.643 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-19T07:52:09.643 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-19T07:52:09.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-19T07:52:09.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-19T07:52:09.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-19T07:52:09.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-19T07:52:09.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-19T07:52:09.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-19T07:52:09.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-19T07:52:09.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-19T07:52:09.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-19T07:52:09.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-19T07:52:09.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-19T07:52:09.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-19T07:52:09.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-19T07:52:09.645 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-19T07:52:09.645 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-19T07:52:09.645 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-19T07:52:09.645 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-19T07:52:09.645 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-19T07:52:09.645 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-19T07:52:09.645 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-19T07:52:09.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-19T07:52:09.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-19T07:52:09.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-19T07:52:09.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-19T07:52:09.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-19T07:52:09.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-19T07:52:09.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-19T07:52:09.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-19T07:52:09.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-19T07:52:09.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-19T07:52:09.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-19T07:52:09.647 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-19T07:52:09.647 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-19T07:52:09.647 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-19T07:52:09.647 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-19T07:52:09.647 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-19T07:52:09.647 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-19T07:52:09.647 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-19T07:52:09.647 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-19T07:52:09.647 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-19T07:52:09.647 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-19T07:52:09.647 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-19T07:52:09.647 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-19T07:52:09.648 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-19T07:52:09.648 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-19T07:52:09.648 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-19T07:52:09.648 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-19T07:52:09.648 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-19T07:52:09.648 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-19T07:52:09.648 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-19T07:52:09.648 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-19T07:52:09.648 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-19T07:52:09.648 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-19T07:52:09.648 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-19T07:52:09.649 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-19T07:52:09.649 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-19T07:52:09.649 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-19T07:52:09.649 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-19T07:52:09.649 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-19T07:52:09.649 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-19T07:52:09.649 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-19T07:52:09.649 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-19T07:52:09.649 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-19T07:52:09.649 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-19T07:52:09.649 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-19T07:52:09.650 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-19T07:52:09.650 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-19T07:52:09.650 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-19T07:52:09.650 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-19T07:52:09.650 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-19T07:52:09.650 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-19T07:52:09.650 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-19T07:52:09.650 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-19T07:52:09.650 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-19T07:52:09.650 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-19T07:52:09.650 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-19T07:52:09.650 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-19T07:52:09.651 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-19T07:52:09.651 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-19T07:52:09.651 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-19T07:52:09.651 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-19T07:52:09.651 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-19T07:52:09.651 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-19T07:52:09.651 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-19T07:52:09.651 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-19T07:52:09.651 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-19T07:52:09.651 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-19T07:52:09.651 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-19T07:52:09.651 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-19T07:52:09.651 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-19T07:52:09.652 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-19T07:52:09.652 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-19T07:52:09.652 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-19T07:52:09.652 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-19T07:52:09.652 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-19T07:52:09.652 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-19T07:52:09.652 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-19T07:52:09.652 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:19.1.1-361.g2f599a46.el9.x86_64 149/164 2024-09-19T07:52:09.652 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-19T07:52:09.652 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 151/164 2024-09-19T07:52:09.652 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-19T07:52:09.653 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 153/164 2024-09-19T07:52:09.653 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-19T07:52:09.653 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 155/164 2024-09-19T07:52:09.653 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-19T07:52:09.653 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 157/164 2024-09-19T07:52:09.653 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-19T07:52:09.653 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 159/164 2024-09-19T07:52:09.653 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-19T07:52:09.653 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-19T07:52:09.653 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-19T07:52:09.653 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-19T07:52:10.142 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 152/164 2024-09-19T07:52:10.240 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-19T07:52:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-19T07:52:10.386 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-19T07:52:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-19T07:52:10.453 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-19T07:52:10.487 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-19T07:52:10.488 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-19T07:52:10.571 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-19T07:52:10.571 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-19T07:52:10.656 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-19T07:52:10.681 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-19T07:52:10.686 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T07:52:10.686 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T07:52:10.866 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T07:52:10.866 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-19T07:52:10.950 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-19T07:52:11.015 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T07:52:12.039 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T07:52:12.040 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:52:12.040 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-09-19T07:52:12.040 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-19T07:52:12.040 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-19T07:52:12.040 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-19T07:52:12.040 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-19T07:52:12.040 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.040 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.040 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-19T07:52:12.041 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-19T07:52:12.041 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-19T07:52:12.041 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-19T07:52:12.041 INFO:teuthology.orchestra.run.smithi162.stdout: automake-1.16.2-8.el9.noarch 2024-09-19T07:52:12.041 INFO:teuthology.orchestra.run.smithi162.stdout: bison-3.7.4-5.el9.x86_64 2024-09-19T07:52:12.041 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-19T07:52:12.041 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.041 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.041 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.041 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.042 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:12.042 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.042 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.042 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.042 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:12.042 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:12.042 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:12.042 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:12.042 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:12.042 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.042 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.042 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:12.042 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.043 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.043 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.043 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:12.043 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:12.043 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.043 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.043 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:12.043 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-19T07:52:12.044 INFO:teuthology.orchestra.run.smithi162.stdout: flex-2.6.4-9.el9.x86_64 2024-09-19T07:52:12.044 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-19T07:52:12.044 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-19T07:52:12.044 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-19T07:52:12.044 INFO:teuthology.orchestra.run.smithi162.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-19T07:52:12.044 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-19T07:52:12.044 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-19T07:52:12.044 INFO:teuthology.orchestra.run.smithi162.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-19T07:52:12.044 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-19T07:52:12.044 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-19T07:52:12.044 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.044 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.045 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.045 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-19T07:52:12.045 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-19T07:52:12.045 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-19T07:52:12.045 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-19T07:52:12.045 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-19T07:52:12.045 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.045 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.045 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-19T07:52:12.045 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.045 INFO:teuthology.orchestra.run.smithi162.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-19T07:52:12.045 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-19T07:52:12.045 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-19T07:52:12.046 INFO:teuthology.orchestra.run.smithi162.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-19T07:52:12.046 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-19T07:52:12.046 INFO:teuthology.orchestra.run.smithi162.stdout: lua-5.4.4-4.el9.x86_64 2024-09-19T07:52:12.046 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-19T07:52:12.046 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-19T07:52:12.046 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-19T07:52:12.046 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-19T07:52:12.046 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-19T07:52:12.046 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-19T07:52:12.046 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-13.16-1.el9.x86_64 2024-09-19T07:52:12.046 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-19T07:52:12.046 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-19T07:52:12.047 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-19T07:52:12.047 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-19T07:52:12.047 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-19T07:52:12.047 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-19T07:52:12.047 INFO:teuthology.orchestra.run.smithi162.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-19T07:52:12.047 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-19T07:52:12.047 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-19T07:52:12.047 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-19T07:52:12.047 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.047 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.047 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.047 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-19T07:52:12.047 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-19T07:52:12.048 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-19T07:52:12.048 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-19T07:52:12.048 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-19T07:52:12.048 INFO:teuthology.orchestra.run.smithi162.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-19T07:52:12.048 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-19T07:52:12.048 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-19T07:52:12.048 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-19T07:52:12.048 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-19T07:52:12.048 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-19T07:52:12.048 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-19T07:52:12.048 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-19T07:52:12.048 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-19T07:52:12.049 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-19T07:52:12.049 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-19T07:52:12.049 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-19T07:52:12.049 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-19T07:52:12.049 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-19T07:52:12.049 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-19T07:52:12.049 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-19T07:52:12.049 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-19T07:52:12.049 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-19T07:52:12.049 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-19T07:52:12.049 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-19T07:52:12.049 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-19T07:52:12.049 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-19T07:52:12.050 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-19T07:52:12.050 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-19T07:52:12.050 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-19T07:52:12.050 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply-3.11-14.el9.noarch 2024-09-19T07:52:12.050 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-19T07:52:12.050 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-19T07:52:12.050 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-19T07:52:12.050 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-19T07:52:12.050 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-19T07:52:12.050 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-19T07:52:12.050 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-19T07:52:12.051 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-19T07:52:12.051 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-19T07:52:12.051 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.051 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.051 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-19T07:52:12.051 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-19T07:52:12.051 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-19T07:52:12.051 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.051 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-19T07:52:12.051 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-19T07:52:12.051 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-19T07:52:12.051 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-19T07:52:12.051 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-19T07:52:12.052 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-19T07:52:12.052 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-19T07:52:12.052 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-19T07:52:12.052 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-19T07:52:12.052 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-19T07:52:12.052 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-19T07:52:12.052 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-19T07:52:12.052 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-19T07:52:12.052 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-19T07:52:12.052 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-19T07:52:12.052 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.052 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.053 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:12.053 INFO:teuthology.orchestra.run.smithi162.stdout: re2-1:20211101-20.el9.x86_64 2024-09-19T07:52:12.053 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-19T07:52:12.053 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-19T07:52:12.053 INFO:teuthology.orchestra.run.smithi162.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-19T07:52:12.053 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-19T07:52:12.053 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-19T07:52:12.053 INFO:teuthology.orchestra.run.smithi162.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-19T07:52:12.054 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:52:12.054 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-19T07:52:12.453 DEBUG:teuthology.parallel:result is None 2024-09-19T07:52:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T07:52:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.1.1-361.g2f599a46.el9.x86_64 1/164 2024-09-19T07:52:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 2/164 2024-09-19T07:52:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 3/164 2024-09-19T07:52:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 4/164 2024-09-19T07:52:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-361.g2f599a 5/164 2024-09-19T07:52:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 6/164 2024-09-19T07:52:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 7/164 2024-09-19T07:52:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T07:52:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T07:52:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 10/164 2024-09-19T07:52:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 11/164 2024-09-19T07:52:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 12/164 2024-09-19T07:52:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T07:52:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 14/164 2024-09-19T07:52:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_6 15/164 2024-09-19T07:52:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 16/164 2024-09-19T07:52:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 17/164 2024-09-19T07:52:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 18/164 2024-09-19T07:52:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 19/164 2024-09-19T07:52:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 20/164 2024-09-19T07:52:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9 21/164 2024-09-19T07:52:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x 22/164 2024-09-19T07:52:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 23/164 2024-09-19T07:52:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 24/164 2024-09-19T07:52:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 25/164 2024-09-19T07:52:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 26/164 2024-09-19T07:52:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 27/164 2024-09-19T07:52:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 28/164 2024-09-19T07:52:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 29/164 2024-09-19T07:52:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.e 30/164 2024-09-19T07:52:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 31/164 2024-09-19T07:52:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 32/164 2024-09-19T07:52:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 33/164 2024-09-19T07:52:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9 34/164 2024-09-19T07:52:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 35/164 2024-09-19T07:52:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el 36/164 2024-09-19T07:52:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 37/164 2024-09-19T07:52:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.1.1-361.g2f599a46.el9.noarch 38/164 2024-09-19T07:52:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 39/164 2024-09-19T07:52:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-19T07:52:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-19T07:52:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-19T07:52:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-19T07:52:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-19T07:52:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-19T07:52:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-19T07:52:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-19T07:52:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-19T07:52:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-19T07:52:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-19T07:52:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-19T07:52:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-19T07:52:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-19T07:52:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-19T07:52:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-19T07:52:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-19T07:52:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-19T07:52:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-19T07:52:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-19T07:52:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-19T07:52:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-19T07:52:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-19T07:52:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-19T07:52:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-19T07:52:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-19T07:52:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-19T07:52:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-19T07:52:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-19T07:52:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-19T07:52:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-19T07:52:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-19T07:52:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-19T07:52:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-19T07:52:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-19T07:52:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-19T07:52:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-19T07:52:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-19T07:52:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-19T07:52:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-19T07:52:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-19T07:52:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-19T07:52:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-19T07:52:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-19T07:52:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-19T07:52:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-19T07:52:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-19T07:52:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-19T07:52:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-19T07:52:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-19T07:52:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-19T07:52:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-19T07:52:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-19T07:52:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-19T07:52:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-19T07:52:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-19T07:52:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-19T07:52:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-19T07:52:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-19T07:52:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-19T07:52:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-19T07:52:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-19T07:52:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-19T07:52:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-19T07:52:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-19T07:52:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-19T07:52:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-19T07:52:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-19T07:52:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-19T07:52:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-19T07:52:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-19T07:52:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-19T07:52:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-19T07:52:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-19T07:52:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-19T07:52:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-19T07:52:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-19T07:52:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-19T07:52:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-19T07:52:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-19T07:52:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-19T07:52:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-19T07:52:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-19T07:52:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-19T07:52:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-19T07:52:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-19T07:52:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-19T07:52:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-19T07:52:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-19T07:52:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-19T07:52:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-19T07:52:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-19T07:52:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-19T07:52:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-19T07:52:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-19T07:52:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-19T07:52:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-19T07:52:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-19T07:52:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-19T07:52:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-19T07:52:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-19T07:52:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-19T07:52:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-19T07:52:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-19T07:52:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-19T07:52:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-19T07:52:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-19T07:52:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-19T07:52:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-19T07:52:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.1.1-361.g2f599a46.el9.x86_64 149/164 2024-09-19T07:52:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-19T07:52:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 151/164 2024-09-19T07:52:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-19T07:52:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 153/164 2024-09-19T07:52:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-19T07:52:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 155/164 2024-09-19T07:52:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-19T07:52:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 157/164 2024-09-19T07:52:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-19T07:52:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 159/164 2024-09-19T07:52:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-19T07:52:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-19T07:52:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-19T07:52:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-19T07:52:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T07:52:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:52:19.734 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-09-19T07:52:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-19T07:52:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-19T07:52:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-19T07:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-19T07:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-19T07:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-19T07:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-19T07:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-19T07:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.2-8.el9.noarch 2024-09-19T07:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.7.4-5.el9.x86_64 2024-09-19T07:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-19T07:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T07:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-19T07:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.4-9.el9.x86_64 2024-09-19T07:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-19T07:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-19T07:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-19T07:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-19T07:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-19T07:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-19T07:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-19T07:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-19T07:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-19T07:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-19T07:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-19T07:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-19T07:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-19T07:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-19T07:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-19T07:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-19T07:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-19T07:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-19T07:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-19T07:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-19T07:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-09-19T07:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-19T07:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-19T07:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-19T07:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-19T07:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-19T07:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-19T07:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-13.16-1.el9.x86_64 2024-09-19T07:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-19T07:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-19T07:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-19T07:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-19T07:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-19T07:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-19T07:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-19T07:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-19T07:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-19T07:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-19T07:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-19T07:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-19T07:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-19T07:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-19T07:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-19T07:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-19T07:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-19T07:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-19T07:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-19T07:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-19T07:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-19T07:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-19T07:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-19T07:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-19T07:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-19T07:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-19T07:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-19T07:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-19T07:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-19T07:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-19T07:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-19T07:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-19T07:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-19T07:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-19T07:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-19T07:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-19T07:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-19T07:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-19T07:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-19T07:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-19T07:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-09-19T07:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-19T07:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-19T07:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-19T07:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-19T07:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-19T07:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-19T07:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-19T07:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-19T07:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-19T07:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-19T07:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-19T07:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-19T07:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-19T07:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-19T07:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-19T07:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-19T07:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-19T07:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-19T07:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-19T07:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-19T07:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-19T07:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-19T07:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-19T07:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-19T07:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-19T07:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-19T07:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-19T07:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T07:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-20.el9.x86_64 2024-09-19T07:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-19T07:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-19T07:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-19T07:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-19T07:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-19T07:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-19T07:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-19T07:52:20.169 DEBUG:teuthology.parallel:result is None 2024-09-19T07:52:20.170 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:52:20.318 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-19T07:52:20.341 INFO:teuthology.orchestra.run.smithi098.stdout:19.1.1-361.g2f599a46.el9 2024-09-19T07:52:20.341 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-361.g2f599a46.el9 2024-09-19T07:52:20.341 INFO:teuthology.task.install:The correct ceph version 19.1.1-361.g2f599a46 is installed. 2024-09-19T07:52:20.343 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:52:20.481 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-19T07:52:20.508 INFO:teuthology.orchestra.run.smithi139.stdout:19.1.1-361.g2f599a46.el9 2024-09-19T07:52:20.509 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-361.g2f599a46.el9 2024-09-19T07:52:20.509 INFO:teuthology.task.install:The correct ceph version 19.1.1-361.g2f599a46 is installed. 2024-09-19T07:52:20.510 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:52:20.645 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-19T07:52:20.672 INFO:teuthology.orchestra.run.smithi162.stdout:19.1.1-361.g2f599a46.el9 2024-09-19T07:52:20.673 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-361.g2f599a46.el9 2024-09-19T07:52:20.673 INFO:teuthology.task.install:The correct ceph version 19.1.1-361.g2f599a46 is installed. 2024-09-19T07:52:20.675 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-19T07:52:20.675 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T07:52:20.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-19T07:52:20.704 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-19T07:52:20.704 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-19T07:52:20.738 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-19T07:52:20.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-19T07:52:20.770 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-19T07:52:20.771 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T07:52:20.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-09-19T07:52:20.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-19T07:52:20.863 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-19T07:52:20.864 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/daemon-helper 2024-09-19T07:52:20.894 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-19T07:52:20.963 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-19T07:52:20.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-09-19T07:52:20.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-19T07:52:21.061 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-19T07:52:21.061 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T07:52:21.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-19T07:52:21.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-19T07:52:21.152 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-19T07:52:21.152 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-19T07:52:21.183 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-19T07:52:21.252 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-19T07:52:21.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-19T07:52:21.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-19T07:52:21.347 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-19T07:52:21.348 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T07:52:21.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-09-19T07:52:21.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-19T07:52:21.437 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-19T07:52:21.437 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/stdin-killer 2024-09-19T07:52:21.462 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-19T07:52:21.528 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-19T07:52:21.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-09-19T07:52:21.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-19T07:52:21.625 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-19T07:52:21.719 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-09-19T07:52:21.719 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:52:21.720 INFO:tasks.cephadm:Cluster fsid is 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:52:21.720 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-19T07:52:21.720 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.98', 'mon.b': '172.21.15.139', 'mon.c': '172.21.15.162'} 2024-09-19T07:52:21.720 INFO:tasks.cephadm:First mon is mon.a on smithi098 2024-09-19T07:52:21.720 INFO:tasks.cephadm:First mgr is x 2024-09-19T07:52:21.721 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-19T07:52:21.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-09-19T07:52:21.746 DEBUG:teuthology.orchestra.run.smithi139:> sudo hostname $(hostname -s) 2024-09-19T07:52:21.772 DEBUG:teuthology.orchestra.run.smithi162:> sudo hostname $(hostname -s) 2024-09-19T07:52:21.800 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-19T07:52:21.800 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:52:21.947 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56106/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.1.1-361-g2f599a46', 'node_name': '172.21.2.7+braggi07', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.1.1-361.g2f599a46'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-18 17:52:24.287651', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-19T07:52:22.086 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 2f599a46ad10e37c929e5c43da2f6907ddcd1961 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:52:22.088 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/x86_64/flavors/default/cephadm 2024-09-19T07:52:22.088 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/x86_64/flavors/default/cephadm 2024-09-19T07:52:22.088 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-19T07:52:22.466 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r--. 1 ubuntu ubuntu 784313 Sep 19 07:52 /home/ubuntu/cephtest/cephadm 2024-09-19T07:52:22.467 DEBUG:teuthology.orchestra.run.smithi139:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-19T07:52:22.860 INFO:teuthology.orchestra.run.smithi139.stdout:-rw-r--r--. 1 ubuntu ubuntu 784313 Sep 19 07:52 /home/ubuntu/cephtest/cephadm 2024-09-19T07:52:22.860 DEBUG:teuthology.orchestra.run.smithi162:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-19T07:52:23.246 INFO:teuthology.orchestra.run.smithi162.stdout:-rw-r--r--. 1 ubuntu ubuntu 784313 Sep 19 07:52 /home/ubuntu/cephtest/cephadm 2024-09-19T07:52:23.247 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-19T07:52:23.266 DEBUG:teuthology.orchestra.run.smithi139:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-19T07:52:23.285 DEBUG:teuthology.orchestra.run.smithi162:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-19T07:52:23.320 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 on all hosts... 2024-09-19T07:52:23.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 pull 2024-09-19T07:52:23.322 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 pull 2024-09-19T07:52:23.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 pull 2024-09-19T07:52:24.910 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961... 2024-09-19T07:52:24.956 INFO:teuthology.orchestra.run.smithi139.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961... 2024-09-19T07:52:24.968 INFO:teuthology.orchestra.run.smithi162.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961... 2024-09-19T07:52:57.305 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-19T07:52:57.306 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_version": "ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable)", 2024-09-19T07:52:57.306 INFO:teuthology.orchestra.run.smithi139.stdout: "image_id": "d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d", 2024-09-19T07:52:57.306 INFO:teuthology.orchestra.run.smithi139.stdout: "repo_digests": [ 2024-09-19T07:52:57.306 INFO:teuthology.orchestra.run.smithi139.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e" 2024-09-19T07:52:57.306 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-19T07:52:57.306 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-19T07:53:17.497 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-19T07:53:17.498 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_version": "ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable)", 2024-09-19T07:53:17.498 INFO:teuthology.orchestra.run.smithi162.stdout: "image_id": "d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d", 2024-09-19T07:53:17.498 INFO:teuthology.orchestra.run.smithi162.stdout: "repo_digests": [ 2024-09-19T07:53:17.498 INFO:teuthology.orchestra.run.smithi162.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e" 2024-09-19T07:53:17.498 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-19T07:53:17.498 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-19T07:53:20.058 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T07:53:20.058 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable)", 2024-09-19T07:53:20.058 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d", 2024-09-19T07:53:20.058 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-09-19T07:53:20.058 INFO:teuthology.orchestra.run.smithi098.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e" 2024-09-19T07:53:20.058 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-19T07:53:20.058 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T07:53:20.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-09-19T07:53:20.140 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph 2024-09-19T07:53:20.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph 2024-09-19T07:53:20.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-09-19T07:53:20.231 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 777 /etc/ceph 2024-09-19T07:53:20.260 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /etc/ceph 2024-09-19T07:53:20.288 INFO:tasks.cephadm:Writing seed config... 2024-09-19T07:53:20.290 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-09-19T07:53:20.290 INFO:tasks.cephadm: override: [client] debug client = 20 2024-09-19T07:53:20.290 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-09-19T07:53:20.290 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-09-19T07:53:20.290 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-09-19T07:53:20.290 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-09-19T07:53:20.290 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-09-19T07:53:20.290 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-09-19T07:53:20.291 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-09-19T07:53:20.291 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-09-19T07:53:20.291 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-09-19T07:53:20.291 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-09-19T07:53:20.291 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-09-19T07:53:20.291 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-09-19T07:53:20.291 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-09-19T07:53:20.291 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-09-19T07:53:20.291 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-09-19T07:53:20.291 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-09-19T07:53:20.291 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-09-19T07:53:20.291 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-09-19T07:53:20.291 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-09-19T07:53:20.291 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-09-19T07:53:20.292 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-09-19T07:53:20.292 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-09-19T07:53:20.292 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-09-19T07:53:20.292 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-19T07:53:20.292 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-19T07:53:20.292 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-19T07:53:20.292 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-19T07:53:20.292 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-19T07:53:20.292 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-19T07:53:20.293 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-09-19T07:53:20.293 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-09-19T07:53:20.293 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-09-19T07:53:20.293 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-09-19T07:53:20.293 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-09-19T07:53:20.293 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-09-19T07:53:20.293 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-09-19T07:53:20.293 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-09-19T07:53:20.293 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-09-19T07:53:20.293 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-19T07:53:20.293 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-19T07:53:20.293 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-09-19T07:53:20.293 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-09-19T07:53:20.294 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-09-19T07:53:20.294 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-09-19T07:53:20.294 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-09-19T07:53:20.294 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-09-19T07:53:20.294 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-09-19T07:53:20.294 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-19T07:53:20.294 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-09-19T07:53:20.295 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T07:53:20.295 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-19T07:53:20.311 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 1a02525a-765c-11ef-bceb-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-09-19T07:53:20.311 DEBUG:teuthology.orchestra.run.smithi098:mon.a> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mon.a.service 2024-09-19T07:53:20.353 DEBUG:teuthology.orchestra.run.smithi098:mgr.x> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mgr.x.service 2024-09-19T07:53:20.397 INFO:tasks.cephadm:Bootstrapping... 2024-09-19T07:53:20.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 -v bootstrap --fsid 1a02525a-765c-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.98 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-19T07:53:20.599 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-19T07:53:20.599 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961', '-v', 'bootstrap', '--fsid', '1a02525a-765c-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.98', '--skip-admin-label'] 2024-09-19T07:53:20.600 INFO:teuthology.orchestra.run.smithi098.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-19T07:53:20.600 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying podman|docker is present... 2024-09-19T07:53:20.623 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 5.2.2 2024-09-19T07:53:20.623 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying lvm2 is present... 2024-09-19T07:53:20.623 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying time synchronization is in place... 2024-09-19T07:53:20.631 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-19T07:53:20.631 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-19T07:53:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-19T07:53:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-09-19T07:53:20.644 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-09-19T07:53:20.650 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-09-19T07:53:20.650 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-09-19T07:53:20.650 INFO:teuthology.orchestra.run.smithi098.stdout:Repeating the final host check... 2024-09-19T07:53:20.672 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 5.2.2 2024-09-19T07:53:20.672 INFO:teuthology.orchestra.run.smithi098.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-19T07:53:20.672 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl is present 2024-09-19T07:53:20.673 INFO:teuthology.orchestra.run.smithi098.stdout:lvcreate is present 2024-09-19T07:53:20.679 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-19T07:53:20.679 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-19T07:53:20.685 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-19T07:53:20.685 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-09-19T07:53:20.692 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-09-19T07:53:20.698 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-09-19T07:53:20.698 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-09-19T07:53:20.698 INFO:teuthology.orchestra.run.smithi098.stdout:Host looks OK 2024-09-19T07:53:20.698 INFO:teuthology.orchestra.run.smithi098.stdout:Cluster fsid: 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:53:20.698 INFO:teuthology.orchestra.run.smithi098.stdout:Acquiring lock 139769703625728 on /run/cephadm/1a02525a-765c-11ef-bceb-c7b262605968.lock 2024-09-19T07:53:20.698 INFO:teuthology.orchestra.run.smithi098.stdout:Lock 139769703625728 acquired on /run/cephadm/1a02525a-765c-11ef-bceb-c7b262605968.lock 2024-09-19T07:53:20.699 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 3300 ... 2024-09-19T07:53:20.699 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 6789 ... 2024-09-19T07:53:20.699 INFO:teuthology.orchestra.run.smithi098.stdout:Base mon IP(s) is [172.21.15.98:3300, 172.21.15.98:6789], mon addrv is [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-09-19T07:53:20.702 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.98 metric 100 2024-09-19T07:53:20.702 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.98 metric 100 2024-09-19T07:53:20.705 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-19T07:53:20.706 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-19T07:53:20.706 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-19T07:53:20.709 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-19T07:53:20.709 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-19T07:53:20.710 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-19T07:53:20.710 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-19T07:53:20.710 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link noprefixroute 2024-09-19T07:53:20.710 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-19T07:53:20.710 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-09-19T07:53:20.710 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-09-19T07:53:20.711 INFO:teuthology.orchestra.run.smithi098.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-19T07:53:20.711 INFO:teuthology.orchestra.run.smithi098.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-19T07:53:20.711 INFO:teuthology.orchestra.run.smithi098.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961... 2024-09-19T07:53:21.178 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d 2024-09-19T07:53:21.178 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961... 2024-09-19T07:53:21.178 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Getting image source signatures 2024-09-19T07:53:21.179 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:c4221751d16d26d2a4b7031ebd71ecffffff13c8cf7401cf95ce80dab3b6d67a 2024-09-19T07:53:21.179 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-19T07:53:21.179 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying config sha256:d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d 2024-09-19T07:53:21.179 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-19T07:53:22.383 INFO:teuthology.orchestra.run.smithi098.stdout:ceph: stdout ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable) 2024-09-19T07:53:22.383 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph version: ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable) 2024-09-19T07:53:22.383 INFO:teuthology.orchestra.run.smithi098.stdout:Extracting ceph user uid/gid from container image... 2024-09-19T07:53:23.432 INFO:teuthology.orchestra.run.smithi098.stdout:stat: stdout 167 167 2024-09-19T07:53:23.432 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial keys... 2024-09-19T07:53:24.312 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQBz2Otm1gCmLxAAopE7Wa1cMgJ5dDcKeKFbpQ== 2024-09-19T07:53:31.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQB12OtmqQBzARAAOudnEzJmHY5Pd7ir+M8bNA== 2024-09-19T07:53:35.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQB+2Otmi1m/IRAAqOzZ2APnsNpkQLaghjf43w== 2024-09-19T07:53:35.531 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial monmap... 2024-09-19T07:53:36.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-19T07:53:36.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-19T07:53:36.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:53:36.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-19T07:53:36.479 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool for a [v2:172.21.15.98:3300,v1:172.21.15.98:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-19T07:53:36.479 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = quincy 2024-09-19T07:53:36.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: set fsid to 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:53:36.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-19T07:53:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:53:36.479 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mon... 2024-09-19T07:53:37.669 INFO:teuthology.orchestra.run.smithi098.stdout:create mon.a on 2024-09-19T07:53:37.885 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-19T07:53:38.040 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-19T07:53:38.210 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-1a02525a-765c-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-1a02525a-765c-11ef-bceb-c7b262605968.target. 2024-09-19T07:53:38.210 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-1a02525a-765c-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-1a02525a-765c-11ef-bceb-c7b262605968.target. 2024-09-19T07:53:38.459 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1a02525a-765c-11ef-bceb-c7b262605968@mon.a 2024-09-19T07:53:38.459 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1a02525a-765c-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-1a02525a-765c-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-19T07:53:38.621 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1a02525a-765c-11ef-bceb-c7b262605968.target.wants/ceph-1a02525a-765c-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-1a02525a-765c-11ef-bceb-c7b262605968@.service. 2024-09-19T07:53:39.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:39 smithi098 podman[30950]: 2024-09-19 07:53:39.090904105 +0000 UTC m=+0.356212520 container start b0d6b6e52d9be464fc88058d066c1327a8402c7ea903bb2e31ca820d65bb2a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-19T07:53:39.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:39 smithi098 ceph-mon[30969]: mkfs 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:53:39.184 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-19T07:53:39.184 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-09-19T07:53:39.192 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-19T07:53:39.192 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-09-19T07:53:39.192 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-09-19T07:53:39.192 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-09-19T07:53:39.192 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon to start... 2024-09-19T07:53:39.193 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon... 2024-09-19T07:53:39.415 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:39 smithi098 ceph-mon[30969]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-19T07:53:39.415 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:39 smithi098 bash[30950]: b0d6b6e52d9be464fc88058d066c1327a8402c7ea903bb2e31ca820d65bb2a31 2024-09-19T07:53:39.415 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:39 smithi098 systemd[1]: Started Ceph mon.a for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T07:53:40.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:40 smithi098 ceph-mon[30969]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-19T07:53:40.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:40 smithi098 ceph-mon[30969]: monmap epoch 1 2024-09-19T07:53:40.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:40 smithi098 ceph-mon[30969]: fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:53:40.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:40 smithi098 ceph-mon[30969]: last_changed 2024-09-19T07:53:35.879566+0000 2024-09-19T07:53:40.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:40 smithi098 ceph-mon[30969]: created 2024-09-19T07:53:35.879566+0000 2024-09-19T07:53:40.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:40 smithi098 ceph-mon[30969]: min_mon_release 19 (squid) 2024-09-19T07:53:40.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:40 smithi098 ceph-mon[30969]: election_strategy: 1 2024-09-19T07:53:40.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:40 smithi098 ceph-mon[30969]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-19T07:53:40.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:40 smithi098 ceph-mon[30969]: fsmap 2024-09-19T07:53:40.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:40 smithi098 ceph-mon[30969]: osdmap e1: 0 total, 0 up, 0 in 2024-09-19T07:53:40.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:40 smithi098 ceph-mon[30969]: mgrmap e1: no daemons active 2024-09-19T07:53:40.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:40 smithi098 ceph-mon[30969]: from='client.? 172.21.15.98:0/1523071780' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-19T07:53:41.803 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout cluster: 2024-09-19T07:53:41.803 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout id: 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:53:41.803 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-19T07:53:41.803 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-19T07:53:41.803 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout services: 2024-09-19T07:53:41.803 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.550158s) 2024-09-19T07:53:41.803 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-19T07:53:41.803 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-19T07:53:41.804 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-19T07:53:41.804 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout data: 2024-09-19T07:53:41.804 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-19T07:53:41.804 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-19T07:53:41.804 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-19T07:53:41.804 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pgs: 2024-09-19T07:53:41.804 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-19T07:53:41.804 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.608+0000 7f22e58e4640 1 Processor -- start 2024-09-19T07:53:41.804 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.608+0000 7f22e58e4640 1 -- start start 2024-09-19T07:53:41.804 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.608+0000 7f22e58e4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22e0107900 0x7f22e0107d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:41.804 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.608+0000 7f22e58e4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f22e01082d0 con 0x7f22e0107900 2024-09-19T07:53:41.804 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.609+0000 7f22deffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22e0107900 0x7f22e0107d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:41.805 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.609+0000 7f22deffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22e0107900 0x7f22e0107d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57392/0 (socket says 172.21.15.98:57392) 2024-09-19T07:53:41.805 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.609+0000 7f22deffd640 1 -- 172.21.15.98:0/2810753908 learned_addr learned my addr 172.21.15.98:0/2810753908 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:41.805 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.609+0000 7f22deffd640 1 -- 172.21.15.98:0/2810753908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e0108b00 con 0x7f22e0107900 2024-09-19T07:53:41.805 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.611+0000 7f22deffd640 1 --2- 172.21.15.98:0/2810753908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22e0107900 0x7f22e0107d00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f22d4009820 tx=0x7f22d402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=521069799eb90465 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:41.805 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.611+0000 7f22ddffb640 1 -- 172.21.15.98:0/2810753908 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22d4037070 con 0x7f22e0107900 2024-09-19T07:53:41.805 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.611+0000 7f22ddffb640 1 -- 172.21.15.98:0/2810753908 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22d4032440 con 0x7f22e0107900 2024-09-19T07:53:41.805 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.611+0000 7f22ddffb640 1 -- 172.21.15.98:0/2810753908 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22d4032740 con 0x7f22e0107900 2024-09-19T07:53:41.805 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.611+0000 7f22e58e4640 1 -- 172.21.15.98:0/2810753908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22e0107900 msgr2=0x7f22e0107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:41.805 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.611+0000 7f22e58e4640 1 --2- 172.21.15.98:0/2810753908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22e0107900 0x7f22e0107d00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f22d4009820 tx=0x7f22d402a270 comp rx=0 tx=0).stop 2024-09-19T07:53:41.805 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.612+0000 7f22e58e4640 1 -- 172.21.15.98:0/2810753908 shutdown_connections 2024-09-19T07:53:41.805 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.612+0000 7f22e58e4640 1 --2- 172.21.15.98:0/2810753908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22e0107900 0x7f22e0107d00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:41.805 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.612+0000 7f22e58e4640 1 -- 172.21.15.98:0/2810753908 >> 172.21.15.98:0/2810753908 conn(0x7f22e01030b0 msgr2=0x7f22e01054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:41.806 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.612+0000 7f22e58e4640 1 -- 172.21.15.98:0/2810753908 shutdown_connections 2024-09-19T07:53:41.806 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.612+0000 7f22e58e4640 1 -- 172.21.15.98:0/2810753908 wait complete. 2024-09-19T07:53:41.806 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.612+0000 7f22e58e4640 1 Processor -- start 2024-09-19T07:53:41.806 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.612+0000 7f22e58e4640 1 -- start start 2024-09-19T07:53:41.806 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.613+0000 7f22e58e4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22e0107900 0x7f22e019b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:41.806 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.613+0000 7f22e58e4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f22e0109030 con 0x7f22e0107900 2024-09-19T07:53:41.806 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.613+0000 7f22deffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22e0107900 0x7f22e019b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:41.806 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.613+0000 7f22deffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22e0107900 0x7f22e019b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57402/0 (socket says 172.21.15.98:57402) 2024-09-19T07:53:41.806 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.613+0000 7f22deffd640 1 -- 172.21.15.98:0/1523071780 learned_addr learned my addr 172.21.15.98:0/1523071780 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:41.806 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.613+0000 7f22deffd640 1 -- 172.21.15.98:0/1523071780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e019b560 con 0x7f22e0107900 2024-09-19T07:53:41.806 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.613+0000 7f22deffd640 1 --2- 172.21.15.98:0/1523071780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22e0107900 0x7f22e019b020 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f22d40090b0 tx=0x7f22d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:41.806 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.613+0000 7f22e48e2640 1 -- 172.21.15.98:0/1523071780 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22d4041070 con 0x7f22e0107900 2024-09-19T07:53:41.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.613+0000 7f22e48e2640 1 -- 172.21.15.98:0/1523071780 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22d403c7a0 con 0x7f22e0107900 2024-09-19T07:53:41.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.613+0000 7f22e48e2640 1 -- 172.21.15.98:0/1523071780 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22d4037040 con 0x7f22e0107900 2024-09-19T07:53:41.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.613+0000 7f22e58e4640 1 -- 172.21.15.98:0/1523071780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22e019b7f0 con 0x7f22e0107900 2024-09-19T07:53:41.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.613+0000 7f22e58e4640 1 -- 172.21.15.98:0/1523071780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22e019e4e0 con 0x7f22e0107900 2024-09-19T07:53:41.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.614+0000 7f22e48e2640 1 -- 172.21.15.98:0/1523071780 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f22d4044070 con 0x7f22e0107900 2024-09-19T07:53:41.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.614+0000 7f22e48e2640 1 -- 172.21.15.98:0/1523071780 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f22d403d3f0 con 0x7f22e0107900 2024-09-19T07:53:41.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.614+0000 7f22e58e4640 1 -- 172.21.15.98:0/1523071780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22e0109680 con 0x7f22e0107900 2024-09-19T07:53:41.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.617+0000 7f22e48e2640 1 -- 172.21.15.98:0/1523071780 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f22d4046450 con 0x7f22e0107900 2024-09-19T07:53:41.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.675+0000 7f22e58e4640 1 -- 172.21.15.98:0/1523071780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f22e002d900 con 0x7f22e0107900 2024-09-19T07:53:41.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.676+0000 7f22e48e2640 1 -- 172.21.15.98:0/1523071780 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f22d4046630 con 0x7f22e0107900 2024-09-19T07:53:41.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.677+0000 7f22e58e4640 1 -- 172.21.15.98:0/1523071780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22e0107900 msgr2=0x7f22e019b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:41.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.677+0000 7f22e58e4640 1 --2- 172.21.15.98:0/1523071780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22e0107900 0x7f22e019b020 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f22d40090b0 tx=0x7f22d40046c0 comp rx=0 tx=0).stop 2024-09-19T07:53:41.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.677+0000 7f22e58e4640 1 -- 172.21.15.98:0/1523071780 shutdown_connections 2024-09-19T07:53:41.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.677+0000 7f22e58e4640 1 --2- 172.21.15.98:0/1523071780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22e0107900 0x7f22e019b020 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:41.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.677+0000 7f22e58e4640 1 -- 172.21.15.98:0/1523071780 >> 172.21.15.98:0/1523071780 conn(0x7f22e01030b0 msgr2=0x7f22e018f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:41.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.677+0000 7f22e58e4640 1 -- 172.21.15.98:0/1523071780 shutdown_connections 2024-09-19T07:53:41.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:39.677+0000 7f22e58e4640 1 -- 172.21.15.98:0/1523071780 wait complete. 2024-09-19T07:53:41.808 INFO:teuthology.orchestra.run.smithi098.stdout:mon is available 2024-09-19T07:53:41.808 INFO:teuthology.orchestra.run.smithi098.stdout:Assimilating anything we can from ceph.conf... 2024-09-19T07:53:42.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:42 smithi098 ceph-mon[30969]: from='client.? 172.21.15.98:0/4076554560' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-19T07:53:42.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:42 smithi098 ceph-mon[30969]: from='client.? 172.21.15.98:0/4076554560' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-19T07:53:43.018 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-19T07:53:43.018 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-09-19T07:53:43.018 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:53:43.018 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-09-19T07:53:43.019 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-19T07:53:43.019 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-19T07:53:43.019 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-19T07:53:43.019 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-19T07:53:43.019 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-19T07:53:43.019 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-19T07:53:43.019 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-19T07:53:43.019 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-19T07:53:43.019 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-09-19T07:53:43.019 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-19T07:53:43.019 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-19T07:53:43.019 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-19T07:53:43.020 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-19T07:53:43.020 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-19T07:53:43.020 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.213+0000 7f87692cc640 1 Processor -- start 2024-09-19T07:53:43.020 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.213+0000 7f87692cc640 1 -- start start 2024-09-19T07:53:43.020 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.213+0000 7f87692cc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87641067b0 0x7f8764106bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:43.020 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.213+0000 7f87692cc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8764107180 con 0x7f87641067b0 2024-09-19T07:53:43.020 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.214+0000 7f8762ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87641067b0 0x7f8764106bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:43.020 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.214+0000 7f8762ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87641067b0 0x7f8764106bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57414/0 (socket says 172.21.15.98:57414) 2024-09-19T07:53:43.020 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.214+0000 7f8762ffd640 1 -- 172.21.15.98:0/1416422943 learned_addr learned my addr 172.21.15.98:0/1416422943 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:43.020 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.214+0000 7f8762ffd640 1 -- 172.21.15.98:0/1416422943 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87641079b0 con 0x7f87641067b0 2024-09-19T07:53:43.020 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.215+0000 7f8762ffd640 1 --2- 172.21.15.98:0/1416422943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87641067b0 0x7f8764106bb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8754009e80 tx=0x7f875402a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4a71a251ad7ab1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:43.020 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.215+0000 7f8761ffb640 1 -- 172.21.15.98:0/1416422943 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8754036070 con 0x7f87641067b0 2024-09-19T07:53:43.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.215+0000 7f8761ffb640 1 -- 172.21.15.98:0/1416422943 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8754031400 con 0x7f87641067b0 2024-09-19T07:53:43.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.215+0000 7f8761ffb640 1 -- 172.21.15.98:0/1416422943 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8754031700 con 0x7f87641067b0 2024-09-19T07:53:43.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.215+0000 7f87692cc640 1 -- 172.21.15.98:0/1416422943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87641067b0 msgr2=0x7f8764106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:43.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.215+0000 7f87692cc640 1 --2- 172.21.15.98:0/1416422943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87641067b0 0x7f8764106bb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8754009e80 tx=0x7f875402a7a0 comp rx=0 tx=0).stop 2024-09-19T07:53:43.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.215+0000 7f87692cc640 1 -- 172.21.15.98:0/1416422943 shutdown_connections 2024-09-19T07:53:43.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.215+0000 7f87692cc640 1 --2- 172.21.15.98:0/1416422943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87641067b0 0x7f8764106bb0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:43.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.215+0000 7f87692cc640 1 -- 172.21.15.98:0/1416422943 >> 172.21.15.98:0/1416422943 conn(0x7f8764101f40 msgr2=0x7f8764104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:43.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.215+0000 7f87692cc640 1 -- 172.21.15.98:0/1416422943 shutdown_connections 2024-09-19T07:53:43.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.215+0000 7f87692cc640 1 -- 172.21.15.98:0/1416422943 wait complete. 2024-09-19T07:53:43.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.216+0000 7f87692cc640 1 Processor -- start 2024-09-19T07:53:43.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.216+0000 7f87692cc640 1 -- start start 2024-09-19T07:53:43.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.216+0000 7f87692cc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87641067b0 0x7f876407ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:43.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.216+0000 7f87692cc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8764107ee0 con 0x7f87641067b0 2024-09-19T07:53:43.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.217+0000 7f8762ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87641067b0 0x7f876407ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:43.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.217+0000 7f8762ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87641067b0 0x7f876407ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57428/0 (socket says 172.21.15.98:57428) 2024-09-19T07:53:43.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.217+0000 7f8762ffd640 1 -- 172.21.15.98:0/4076554560 learned_addr learned my addr 172.21.15.98:0/4076554560 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:43.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.217+0000 7f8762ffd640 1 -- 172.21.15.98:0/4076554560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876419ddf0 con 0x7f87641067b0 2024-09-19T07:53:43.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.217+0000 7f8762ffd640 1 --2- 172.21.15.98:0/4076554560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87641067b0 0x7f876407ba70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8754034040 tx=0x7f8754004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:43.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.217+0000 7f875bfff640 1 -- 172.21.15.98:0/4076554560 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8754041070 con 0x7f87641067b0 2024-09-19T07:53:43.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.217+0000 7f875bfff640 1 -- 172.21.15.98:0/4076554560 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f875403c650 con 0x7f87641067b0 2024-09-19T07:53:43.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.217+0000 7f87692cc640 1 -- 172.21.15.98:0/4076554560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f876419dfc0 con 0x7f87641067b0 2024-09-19T07:53:43.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.217+0000 7f875bfff640 1 -- 172.21.15.98:0/4076554560 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8754036040 con 0x7f87641067b0 2024-09-19T07:53:43.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.217+0000 7f87692cc640 1 -- 172.21.15.98:0/4076554560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f876419e440 con 0x7f87641067b0 2024-09-19T07:53:43.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.217+0000 7f875bfff640 1 -- 172.21.15.98:0/4076554560 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8754044070 con 0x7f87641067b0 2024-09-19T07:53:43.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.217+0000 7f875bfff640 1 -- 172.21.15.98:0/4076554560 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8754046c10 con 0x7f87641067b0 2024-09-19T07:53:43.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.218+0000 7f87692cc640 1 -- 172.21.15.98:0/4076554560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8764108530 con 0x7f87641067b0 2024-09-19T07:53:43.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.220+0000 7f875bfff640 1 -- 172.21.15.98:0/4076554560 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f875403c950 con 0x7f87641067b0 2024-09-19T07:53:43.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.277+0000 7f87692cc640 1 -- 172.21.15.98:0/4076554560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f876419e730 con 0x7f87641067b0 2024-09-19T07:53:43.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.280+0000 7f875bfff640 1 -- 172.21.15.98:0/4076554560 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f875403cb30 con 0x7f87641067b0 2024-09-19T07:53:43.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.281+0000 7f875bfff640 1 -- 172.21.15.98:0/4076554560 <== mon.0 v2:172.21.15.98:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8754056070 con 0x7f87641067b0 2024-09-19T07:53:43.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.282+0000 7f87692cc640 1 -- 172.21.15.98:0/4076554560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87641067b0 msgr2=0x7f876407ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:43.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.282+0000 7f87692cc640 1 --2- 172.21.15.98:0/4076554560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87641067b0 0x7f876407ba70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8754034040 tx=0x7f8754004a40 comp rx=0 tx=0).stop 2024-09-19T07:53:43.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.282+0000 7f87692cc640 1 -- 172.21.15.98:0/4076554560 shutdown_connections 2024-09-19T07:53:43.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.282+0000 7f87692cc640 1 --2- 172.21.15.98:0/4076554560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87641067b0 0x7f876407ba70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:43.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.282+0000 7f87692cc640 1 -- 172.21.15.98:0/4076554560 >> 172.21.15.98:0/4076554560 conn(0x7f8764101f40 msgr2=0x7f876418e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:43.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.282+0000 7f87692cc640 1 -- 172.21.15.98:0/4076554560 shutdown_connections 2024-09-19T07:53:43.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:42.282+0000 7f87692cc640 1 -- 172.21.15.98:0/4076554560 wait complete. 2024-09-19T07:53:43.023 INFO:teuthology.orchestra.run.smithi098.stdout:Generating new minimal ceph.conf... 2024-09-19T07:53:43.914 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:43 smithi098 ceph-mon[30969]: from='client.? 172.21.15.98:0/1580989488' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:53:44.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.404+0000 7f125c704640 1 Processor -- start 2024-09-19T07:53:44.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.404+0000 7f125c704640 1 -- start start 2024-09-19T07:53:44.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.405+0000 7f125c704640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254108360 0x7f125410a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:44.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.405+0000 7f125c704640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f125407be50 con 0x7f1254108360 2024-09-19T07:53:44.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.405+0000 7f125a4a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254108360 0x7f125410a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:44.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.405+0000 7f125a4a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254108360 0x7f125410a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57442/0 (socket says 172.21.15.98:57442) 2024-09-19T07:53:44.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.405+0000 7f125a4a0640 1 -- 172.21.15.98:0/2017817643 learned_addr learned my addr 172.21.15.98:0/2017817643 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:44.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.405+0000 7f125a4a0640 1 -- 172.21.15.98:0/2017817643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125410ac90 con 0x7f1254108360 2024-09-19T07:53:44.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.406+0000 7f125a4a0640 1 --2- 172.21.15.98:0/2017817643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254108360 0x7f125410a750 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1250009820 tx=0x7f125002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18a53919ac25fdc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:44.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.406+0000 7f125949e640 1 -- 172.21.15.98:0/2017817643 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1250037070 con 0x7f1254108360 2024-09-19T07:53:44.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.406+0000 7f125949e640 1 -- 172.21.15.98:0/2017817643 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1250032900 con 0x7f1254108360 2024-09-19T07:53:44.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.406+0000 7f125949e640 1 -- 172.21.15.98:0/2017817643 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1250030850 con 0x7f1254108360 2024-09-19T07:53:44.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.406+0000 7f125c704640 1 -- 172.21.15.98:0/2017817643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254108360 msgr2=0x7f125410a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:44.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.406+0000 7f125c704640 1 --2- 172.21.15.98:0/2017817643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254108360 0x7f125410a750 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1250009820 tx=0x7f125002a270 comp rx=0 tx=0).stop 2024-09-19T07:53:44.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.407+0000 7f125c704640 1 -- 172.21.15.98:0/2017817643 shutdown_connections 2024-09-19T07:53:44.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.407+0000 7f125c704640 1 --2- 172.21.15.98:0/2017817643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254108360 0x7f125410a750 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:44.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.407+0000 7f125c704640 1 -- 172.21.15.98:0/2017817643 >> 172.21.15.98:0/2017817643 conn(0x7f1254101f40 msgr2=0x7f1254104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:44.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.407+0000 7f125c704640 1 -- 172.21.15.98:0/2017817643 shutdown_connections 2024-09-19T07:53:44.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.407+0000 7f125c704640 1 -- 172.21.15.98:0/2017817643 wait complete. 2024-09-19T07:53:44.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.407+0000 7f125c704640 1 Processor -- start 2024-09-19T07:53:44.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.407+0000 7f125c704640 1 -- start start 2024-09-19T07:53:44.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.407+0000 7f125c704640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254108360 0x7f125419b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:44.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.407+0000 7f125c704640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f125410bc60 con 0x7f1254108360 2024-09-19T07:53:44.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.407+0000 7f125a4a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254108360 0x7f125419b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:44.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.407+0000 7f125a4a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254108360 0x7f125419b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57446/0 (socket says 172.21.15.98:57446) 2024-09-19T07:53:44.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.407+0000 7f125a4a0640 1 -- 172.21.15.98:0/1580989488 learned_addr learned my addr 172.21.15.98:0/1580989488 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:44.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.408+0000 7f125a4a0640 1 -- 172.21.15.98:0/1580989488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125419bdc0 con 0x7f1254108360 2024-09-19T07:53:44.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.408+0000 7f125a4a0640 1 --2- 172.21.15.98:0/1580989488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254108360 0x7f125419b880 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1250032ce0 tx=0x7f1250032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:44.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.408+0000 7f124b7fe640 1 -- 172.21.15.98:0/1580989488 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1250042070 con 0x7f1254108360 2024-09-19T07:53:44.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.408+0000 7f124b7fe640 1 -- 172.21.15.98:0/1580989488 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f12500308c0 con 0x7f1254108360 2024-09-19T07:53:44.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.408+0000 7f124b7fe640 1 -- 172.21.15.98:0/1580989488 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1250037040 con 0x7f1254108360 2024-09-19T07:53:44.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.408+0000 7f125c704640 1 -- 172.21.15.98:0/1580989488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125419c050 con 0x7f1254108360 2024-09-19T07:53:44.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.408+0000 7f125c704640 1 -- 172.21.15.98:0/1580989488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f125419ed40 con 0x7f1254108360 2024-09-19T07:53:44.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.408+0000 7f124b7fe640 1 -- 172.21.15.98:0/1580989488 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1250045070 con 0x7f1254108360 2024-09-19T07:53:44.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.408+0000 7f124b7fe640 1 -- 172.21.15.98:0/1580989488 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1250047bd0 con 0x7f1254108360 2024-09-19T07:53:44.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.409+0000 7f125c704640 1 -- 172.21.15.98:0/1580989488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12541087c0 con 0x7f1254108360 2024-09-19T07:53:44.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.411+0000 7f124b7fe640 1 -- 172.21.15.98:0/1580989488 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f125003c830 con 0x7f1254108360 2024-09-19T07:53:44.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.469+0000 7f125c704640 1 -- 172.21.15.98:0/1580989488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f125419ef10 con 0x7f1254108360 2024-09-19T07:53:44.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.470+0000 7f124b7fe640 1 -- 172.21.15.98:0/1580989488 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f125003ca10 con 0x7f1254108360 2024-09-19T07:53:44.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.470+0000 7f125c704640 1 -- 172.21.15.98:0/1580989488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254108360 msgr2=0x7f125419b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:44.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.470+0000 7f125c704640 1 --2- 172.21.15.98:0/1580989488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254108360 0x7f125419b880 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1250032ce0 tx=0x7f1250032d10 comp rx=0 tx=0).stop 2024-09-19T07:53:44.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.470+0000 7f125c704640 1 -- 172.21.15.98:0/1580989488 shutdown_connections 2024-09-19T07:53:44.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.470+0000 7f125c704640 1 --2- 172.21.15.98:0/1580989488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254108360 0x7f125419b880 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:44.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.470+0000 7f125c704640 1 -- 172.21.15.98:0/1580989488 >> 172.21.15.98:0/1580989488 conn(0x7f1254101f40 msgr2=0x7f1254102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:44.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.471+0000 7f125c704640 1 -- 172.21.15.98:0/1580989488 shutdown_connections 2024-09-19T07:53:44.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:43.471+0000 7f125c704640 1 -- 172.21.15.98:0/1580989488 wait complete. 2024-09-19T07:53:44.026 INFO:teuthology.orchestra.run.smithi098.stdout:Restarting the monitor... 2024-09-19T07:53:44.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:44 smithi098 systemd[1]: Stopping Ceph mon.a for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T07:53:44.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:44 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a[30965]: 2024-09-19T07:53:44.302+0000 7f6077f39640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T07:53:44.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:44 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a[30965]: 2024-09-19T07:53:44.302+0000 7f6077f39640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-19T07:53:44.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:44 smithi098 podman[31213]: 2024-09-19 07:53:44.446086432 +0000 UTC m=+0.262272951 container died b0d6b6e52d9be464fc88058d066c1327a8402c7ea903bb2e31ca820d65bb2a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-19T07:53:44.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:44 smithi098 podman[31213]: 2024-09-19 07:53:44.777894772 +0000 UTC m=+0.594081291 container cleanup b0d6b6e52d9be464fc88058d066c1327a8402c7ea903bb2e31ca820d65bb2a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=squid-2f599a4, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-19T07:53:44.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:44 smithi098 bash[31213]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a 2024-09-19T07:53:45.387 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:44 smithi098 podman[31225]: 2024-09-19 07:53:44.970855834 +0000 UTC m=+0.527477353 container remove b0d6b6e52d9be464fc88058d066c1327a8402c7ea903bb2e31ca820d65bb2a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-19T07:53:45.387 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 systemd[1]: ceph-1a02525a-765c-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-19T07:53:45.388 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 systemd[1]: Stopped Ceph mon.a for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T07:53:45.388 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 systemd[1]: Starting Ceph mon.a for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T07:53:45.636 INFO:teuthology.orchestra.run.smithi098.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-19T07:53:45.649 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 podman[31314]: 2024-09-19 07:53:45.388299123 +0000 UTC m=+0.106941450 container create 44a037c430ef182d56e33e0bb4c723bc2fcb078a0e5746f498f035526d354886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, RELEASE=squid-2f599a4, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-19T07:53:45.649 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 podman[31314]: 2024-09-19 07:53:45.297407837 +0000 UTC m=+0.016050180 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:53:45.650 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 podman[31314]: 2024-09-19 07:53:45.557328962 +0000 UTC m=+0.275971300 container init 44a037c430ef182d56e33e0bb4c723bc2fcb078a0e5746f498f035526d354886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True) 2024-09-19T07:53:45.650 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 podman[31314]: 2024-09-19 07:53:45.560313529 +0000 UTC m=+0.278955852 container start 44a037c430ef182d56e33e0bb4c723bc2fcb078a0e5746f498f035526d354886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-19T07:53:45.650 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: set uid:gid to 167:167 (ceph:ceph) 2024-09-19T07:53:45.650 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable), process ceph-mon, pid 2 2024-09-19T07:53:45.650 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: pidfile_write: ignore empty --pid-file 2024-09-19T07:53:45.650 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: load: jerasure load: lrc 2024-09-19T07:53:45.650 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: RocksDB version: 7.9.2 2024-09-19T07:53:45.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Git sha 0 2024-09-19T07:53:45.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Compile date 2024-09-18 16:29:48 2024-09-19T07:53:45.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: DB SUMMARY 2024-09-19T07:53:45.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: DB Session ID: S5Z8AAE1TXHME8BWSV46 2024-09-19T07:53:45.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: CURRENT file: CURRENT 2024-09-19T07:53:45.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: IDENTITY file: IDENTITY 2024-09-19T07:53:45.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-19T07:53:45.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-19T07:53:45.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 107224 ; 2024-09-19T07:53:45.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.error_if_exists: 0 2024-09-19T07:53:45.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.create_if_missing: 0 2024-09-19T07:53:45.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.paranoid_checks: 1 2024-09-19T07:53:45.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-19T07:53:45.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-19T07:53:45.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-19T07:53:45.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.env: 0x5610ca6fb940 2024-09-19T07:53:45.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.fs: PosixFileSystem 2024-09-19T07:53:45.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.info_log: 0x5610cb10f6e0 2024-09-19T07:53:45.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_file_opening_threads: 16 2024-09-19T07:53:45.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.statistics: (nil) 2024-09-19T07:53:45.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.use_fsync: 0 2024-09-19T07:53:45.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_log_file_size: 0 2024-09-19T07:53:45.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-19T07:53:45.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-19T07:53:45.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.keep_log_file_num: 1000 2024-09-19T07:53:45.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.recycle_log_file_num: 0 2024-09-19T07:53:45.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.allow_fallocate: 1 2024-09-19T07:53:45.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.allow_mmap_reads: 0 2024-09-19T07:53:45.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.allow_mmap_writes: 0 2024-09-19T07:53:45.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.use_direct_reads: 0 2024-09-19T07:53:45.655 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-19T07:53:45.655 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.create_missing_column_families: 0 2024-09-19T07:53:45.655 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.db_log_dir: 2024-09-19T07:53:45.655 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.wal_dir: 2024-09-19T07:53:45.655 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-19T07:53:45.655 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-19T07:53:45.655 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-19T07:53:45.655 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-19T07:53:45.655 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-19T07:53:45.655 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-19T07:53:45.655 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.advise_random_on_open: 1 2024-09-19T07:53:45.655 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.db_write_buffer_size: 0 2024-09-19T07:53:45.655 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.write_buffer_manager: 0x5610cb11edc0 2024-09-19T07:53:45.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-19T07:53:45.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-19T07:53:45.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-19T07:53:45.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.rate_limiter: (nil) 2024-09-19T07:53:45.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-19T07:53:45.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.wal_recovery_mode: 2 2024-09-19T07:53:45.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.enable_thread_tracking: 0 2024-09-19T07:53:45.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.enable_pipelined_write: 0 2024-09-19T07:53:45.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.unordered_write: 0 2024-09-19T07:53:45.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-19T07:53:45.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-19T07:53:45.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-19T07:53:45.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-19T07:53:45.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.row_cache: None 2024-09-19T07:53:45.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.wal_filter: None 2024-09-19T07:53:45.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-19T07:53:45.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.allow_ingest_behind: 0 2024-09-19T07:53:45.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.two_write_queues: 0 2024-09-19T07:53:45.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.manual_wal_flush: 0 2024-09-19T07:53:45.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.wal_compression: 0 2024-09-19T07:53:45.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.atomic_flush: 0 2024-09-19T07:53:45.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-19T07:53:45.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-19T07:53:45.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-19T07:53:45.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.log_readahead_size: 0 2024-09-19T07:53:45.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-19T07:53:45.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.best_efforts_recovery: 0 2024-09-19T07:53:45.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-19T07:53:45.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-19T07:53:45.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.allow_data_in_errors: 0 2024-09-19T07:53:45.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.db_host_id: __hostname__ 2024-09-19T07:53:45.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-19T07:53:45.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_background_jobs: 2 2024-09-19T07:53:45.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_background_compactions: -1 2024-09-19T07:53:45.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_subcompactions: 1 2024-09-19T07:53:45.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-19T07:53:45.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-19T07:53:45.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-19T07:53:45.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_total_wal_size: 0 2024-09-19T07:53:45.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-19T07:53:45.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-19T07:53:45.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-19T07:53:45.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-19T07:53:45.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_open_files: -1 2024-09-19T07:53:45.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.bytes_per_sync: 0 2024-09-19T07:53:45.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-19T07:53:45.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-19T07:53:45.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compaction_readahead_size: 0 2024-09-19T07:53:45.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_background_flushes: -1 2024-09-19T07:53:45.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Compression algorithms supported: 2024-09-19T07:53:45.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: kZSTD supported: 0 2024-09-19T07:53:45.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: kXpressCompression supported: 0 2024-09-19T07:53:45.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: kBZip2Compression supported: 0 2024-09-19T07:53:45.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-19T07:53:45.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: kLZ4Compression supported: 1 2024-09-19T07:53:45.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: kZlibCompression supported: 1 2024-09-19T07:53:45.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: kLZ4HCCompression supported: 1 2024-09-19T07:53:45.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: kSnappyCompression supported: 1 2024-09-19T07:53:45.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-19T07:53:45.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-19T07:53:45.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-19T07:53:45.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-19T07:53:45.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-19T07:53:45.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.merge_operator: 2024-09-19T07:53:45.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compaction_filter: None 2024-09-19T07:53:45.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compaction_filter_factory: None 2024-09-19T07:53:45.661 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.sst_partitioner_factory: None 2024-09-19T07:53:45.661 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-19T07:53:45.661 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-19T07:53:45.661 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5610cb10f320) 2024-09-19T07:53:45.661 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-09-19T07:53:45.661 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-19T07:53:45.661 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-19T07:53:45.661 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-09-19T07:53:45.661 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_type: 0 2024-09-19T07:53:45.661 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_index_type: 0 2024-09-19T07:53:45.661 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_shortening: 1 2024-09-19T07:53:45.661 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-19T07:53:45.661 INFO:journalctl@ceph.mon.a.smithi098.stdout: checksum: 4 2024-09-19T07:53:45.662 INFO:journalctl@ceph.mon.a.smithi098.stdout: no_block_cache: 0 2024-09-19T07:53:45.662 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache: 0x5610cb1031f0 2024-09-19T07:53:45.662 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-09-19T07:53:45.662 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_options: 2024-09-19T07:53:45.662 INFO:journalctl@ceph.mon.a.smithi098.stdout: capacity : 536870912 2024-09-19T07:53:45.662 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_shard_bits : 4 2024-09-19T07:53:45.662 INFO:journalctl@ceph.mon.a.smithi098.stdout: strict_capacity_limit : 0 2024-09-19T07:53:45.662 INFO:journalctl@ceph.mon.a.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-09-19T07:53:45.662 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_compressed: (nil) 2024-09-19T07:53:45.662 INFO:journalctl@ceph.mon.a.smithi098.stdout: persistent_cache: (nil) 2024-09-19T07:53:45.662 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size: 4096 2024-09-19T07:53:45.662 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size_deviation: 10 2024-09-19T07:53:45.662 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_restart_interval: 16 2024-09-19T07:53:45.662 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_block_restart_interval: 1 2024-09-19T07:53:45.663 INFO:journalctl@ceph.mon.a.smithi098.stdout: metadata_block_size: 4096 2024-09-19T07:53:45.663 INFO:journalctl@ceph.mon.a.smithi098.stdout: partition_filters: 0 2024-09-19T07:53:45.663 INFO:journalctl@ceph.mon.a.smithi098.stdout: use_delta_encoding: 1 2024-09-19T07:53:45.663 INFO:journalctl@ceph.mon.a.smithi098.stdout: filter_policy: bloomfilter 2024-09-19T07:53:45.663 INFO:journalctl@ceph.mon.a.smithi098.stdout: whole_key_filtering: 1 2024-09-19T07:53:45.663 INFO:journalctl@ceph.mon.a.smithi098.stdout: verify_compression: 0 2024-09-19T07:53:45.663 INFO:journalctl@ceph.mon.a.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-09-19T07:53:45.663 INFO:journalctl@ceph.mon.a.smithi098.stdout: format_version: 5 2024-09-19T07:53:45.663 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_index_compression: 1 2024-09-19T07:53:45.663 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_align: 0 2024-09-19T07:53:45.663 INFO:journalctl@ceph.mon.a.smithi098.stdout: max_auto_readahead_size: 262144 2024-09-19T07:53:45.663 INFO:journalctl@ceph.mon.a.smithi098.stdout: prepopulate_block_cache: 0 2024-09-19T07:53:45.663 INFO:journalctl@ceph.mon.a.smithi098.stdout: initial_auto_readahead_size: 8192 2024-09-19T07:53:45.663 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2024-09-19T07:53:45.664 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.write_buffer_size: 33554432 2024-09-19T07:53:45.664 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_write_buffer_number: 2 2024-09-19T07:53:45.664 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compression: NoCompression 2024-09-19T07:53:45.664 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.bottommost_compression: Disabled 2024-09-19T07:53:45.664 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.prefix_extractor: nullptr 2024-09-19T07:53:45.664 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-19T07:53:45.664 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.num_levels: 7 2024-09-19T07:53:45.664 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-19T07:53:45.664 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-19T07:53:45.664 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-19T07:53:45.664 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-19T07:53:45.664 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-19T07:53:45.664 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-19T07:53:45.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-19T07:53:45.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-19T07:53:45.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-19T07:53:45.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-19T07:53:45.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-19T07:53:45.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-19T07:53:45.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-19T07:53:45.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compression_opts.level: 32767 2024-09-19T07:53:45.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compression_opts.strategy: 0 2024-09-19T07:53:45.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-19T07:53:45.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-19T07:53:45.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-19T07:53:45.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-19T07:53:45.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compression_opts.enabled: false 2024-09-19T07:53:45.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-19T07:53:45.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-19T07:53:45.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-19T07:53:45.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-19T07:53:45.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.target_file_size_base: 67108864 2024-09-19T07:53:45.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-19T07:53:45.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-19T07:53:45.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-19T07:53:45.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-19T07:53:45.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-19T07:53:45.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-19T07:53:45.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-19T07:53:45.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-19T07:53:45.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-19T07:53:45.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-19T07:53:45.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-19T07:53:45.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-19T07:53:45.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-19T07:53:45.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-19T07:53:45.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.arena_block_size: 1048576 2024-09-19T07:53:45.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-19T07:53:45.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-19T07:53:45.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.disable_auto_compactions: 0 2024-09-19T07:53:45.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-19T07:53:45.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-19T07:53:45.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-19T07:53:45.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-19T07:53:45.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-19T07:53:45.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-19T07:53:45.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-19T07:53:45.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-19T07:53:45.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-19T07:53:45.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-19T07:53:45.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-19T07:53:45.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.inplace_update_support: 0 2024-09-19T07:53:45.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-19T07:53:45.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-19T07:53:45.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-19T07:53:45.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-19T07:53:45.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.bloom_locality: 0 2024-09-19T07:53:45.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.max_successive_merges: 0 2024-09-19T07:53:45.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-19T07:53:45.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.paranoid_file_checks: 0 2024-09-19T07:53:45.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.force_consistency_checks: 1 2024-09-19T07:53:45.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.report_bg_io_stats: 0 2024-09-19T07:53:45.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.ttl: 2592000 2024-09-19T07:53:45.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-19T07:53:45.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-19T07:53:45.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-19T07:53:45.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.enable_blob_files: false 2024-09-19T07:53:45.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.min_blob_size: 0 2024-09-19T07:53:45.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.blob_file_size: 268435456 2024-09-19T07:53:45.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-19T07:53:45.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-19T07:53:45.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-19T07:53:45.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-19T07:53:45.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-19T07:53:45.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.blob_file_starting_level: 0 2024-09-19T07:53:45.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-19T07:53:45.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-19T07:53:45.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-19T07:53:45.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 01a0978e-dd64-4b56-98ad-ee22b1fc66ac 2024-09-19T07:53:45.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726732425587081, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-19T07:53:45.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-19T07:53:45.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726732425588725, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 102107, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 336, "table_properties": {"data_size": 99995, "index_size": 304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14782, "raw_average_key_size": 50, "raw_value_size": 91997, "raw_average_value_size": 311, "num_data_blocks": 12, "num_entries": 295, "num_filter_entries": 295, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726732425, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "01a0978e-dd64-4b56-98ad-ee22b1fc66ac", "db_session_id": "S5Z8AAE1TXHME8BWSV46", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-19T07:53:45.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726732425588862, "job": 1, "event": "recovery_finished"} 2024-09-19T07:53:45.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-19T07:53:45.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-19T07:53:45.671 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5610cb130e00 2024-09-19T07:53:45.671 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: DB pointer 0x5610cb22c000 2024-09-19T07:53:45.671 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-19T07:53:45.671 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-19T07:53:45.671 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** DB Stats ** 2024-09-19T07:53:45.671 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-19T07:53:45.671 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-19T07:53:45.671 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-19T07:53:45.671 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-19T07:53:45.671 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-19T07:53:45.671 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-19T07:53:45.671 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-19T07:53:45.671 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-19T07:53:45.672 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-09-19T07:53:45.672 INFO:journalctl@ceph.mon.a.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-19T07:53:45.672 INFO:journalctl@ceph.mon.a.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-19T07:53:45.672 INFO:journalctl@ceph.mon.a.smithi098.stdout: L0 2/0 101.57 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 91.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-19T07:53:45.672 INFO:journalctl@ceph.mon.a.smithi098.stdout: Sum 2/0 101.57 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 91.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-19T07:53:45.672 INFO:journalctl@ceph.mon.a.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 91.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-19T07:53:45.672 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-19T07:53:45.672 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-09-19T07:53:45.672 INFO:journalctl@ceph.mon.a.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-19T07:53:45.672 INFO:journalctl@ceph.mon.a.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-19T07:53:45.672 INFO:journalctl@ceph.mon.a.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 91.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-19T07:53:45.672 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-19T07:53:45.672 INFO:journalctl@ceph.mon.a.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-19T07:53:45.673 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-19T07:53:45.673 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-19T07:53:45.673 INFO:journalctl@ceph.mon.a.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-19T07:53:45.673 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-19T07:53:45.673 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-19T07:53:45.673 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-19T07:53:45.673 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-19T07:53:45.673 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative compaction: 0.00 GB write, 21.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-19T07:53:45.673 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval compaction: 0.00 GB write, 21.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-19T07:53:45.673 INFO:journalctl@ceph.mon.a.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-19T07:53:45.673 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache BinnedLRUCache@0x5610cb1031f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-19T07:53:45.673 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-09-19T07:53:45.673 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-19T07:53:45.673 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-19T07:53:45.674 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: starting mon.a rank 0 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:53:45.674 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: mon.a@-1(???) e1 preinit fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:53:45.674 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: mon.a@-1(???).mds e1 new map 2024-09-19T07:53:45.674 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: mon.a@-1(???).mds e1 print_map 2024-09-19T07:53:45.674 INFO:journalctl@ceph.mon.a.smithi098.stdout: e1 2024-09-19T07:53:45.674 INFO:journalctl@ceph.mon.a.smithi098.stdout: btime 2024-09-19T07:53:39:127203+0000 2024-09-19T07:53:45.674 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-19T07:53:45.674 INFO:journalctl@ceph.mon.a.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-19T07:53:45.674 INFO:journalctl@ceph.mon.a.smithi098.stdout: legacy client fscid: -1 2024-09-19T07:53:45.674 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-19T07:53:45.674 INFO:journalctl@ceph.mon.a.smithi098.stdout: No filesystems configured 2024-09-19T07:53:45.674 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-19T07:53:45.674 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-19T07:53:45.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-19T07:53:45.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-19T07:53:45.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-19T07:53:45.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 bash[31314]: 44a037c430ef182d56e33e0bb4c723bc2fcb078a0e5746f498f035526d354886 2024-09-19T07:53:45.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 systemd[1]: Started Ceph mon.a for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T07:53:45.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-19T07:53:45.676 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: monmap epoch 1 2024-09-19T07:53:45.676 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:53:45.676 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: last_changed 2024-09-19T07:53:35.879566+0000 2024-09-19T07:53:45.676 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: created 2024-09-19T07:53:35.879566+0000 2024-09-19T07:53:45.676 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: min_mon_release 19 (squid) 2024-09-19T07:53:45.676 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: election_strategy: 1 2024-09-19T07:53:45.676 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-19T07:53:45.676 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: fsmap 2024-09-19T07:53:45.676 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: osdmap e1: 0 total, 0 up, 0 in 2024-09-19T07:53:45.676 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:45 smithi098 ceph-mon[31333]: mgrmap e1: no daemons active 2024-09-19T07:53:46.720 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.022+0000 7fbb7fcc0640 1 Processor -- start 2024-09-19T07:53:46.720 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.023+0000 7fbb7fcc0640 1 -- start start 2024-09-19T07:53:46.720 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.023+0000 7fbb7fcc0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb78106210 0x7fbb78106610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:46.720 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.023+0000 7fbb7fcc0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7807be10 con 0x7fbb78106210 2024-09-19T07:53:46.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.023+0000 7fbb7da5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb78106210 0x7fbb78106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:46.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.023+0000 7fbb7da5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb78106210 0x7fbb78106610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57462/0 (socket says 172.21.15.98:57462) 2024-09-19T07:53:46.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.023+0000 7fbb7da5c640 1 -- 172.21.15.98:0/68318895 learned_addr learned my addr 172.21.15.98:0/68318895 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:46.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.024+0000 7fbb7da5c640 1 -- 172.21.15.98:0/68318895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb78106b50 con 0x7fbb78106210 2024-09-19T07:53:46.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.025+0000 7fbb7da5c640 1 --2- 172.21.15.98:0/68318895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb78106210 0x7fbb78106610 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009820 tx=0x7fbb7402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8468b877dcaf5cf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:46.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.025+0000 7fbb7ca5a640 1 -- 172.21.15.98:0/68318895 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb74037070 con 0x7fbb78106210 2024-09-19T07:53:46.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.025+0000 7fbb7ca5a640 1 -- 172.21.15.98:0/68318895 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbb74032900 con 0x7fbb78106210 2024-09-19T07:53:46.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.025+0000 7fbb7ca5a640 1 -- 172.21.15.98:0/68318895 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb74030850 con 0x7fbb78106210 2024-09-19T07:53:46.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.026+0000 7fbb7fcc0640 1 -- 172.21.15.98:0/68318895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb78106210 msgr2=0x7fbb78106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:46.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.026+0000 7fbb7fcc0640 1 --2- 172.21.15.98:0/68318895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb78106210 0x7fbb78106610 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009820 tx=0x7fbb7402a270 comp rx=0 tx=0).stop 2024-09-19T07:53:46.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.026+0000 7fbb7fcc0640 1 -- 172.21.15.98:0/68318895 shutdown_connections 2024-09-19T07:53:46.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.026+0000 7fbb7fcc0640 1 --2- 172.21.15.98:0/68318895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb78106210 0x7fbb78106610 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:46.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.026+0000 7fbb7fcc0640 1 -- 172.21.15.98:0/68318895 >> 172.21.15.98:0/68318895 conn(0x7fbb78102020 msgr2=0x7fbb78104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:46.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.026+0000 7fbb7fcc0640 1 -- 172.21.15.98:0/68318895 shutdown_connections 2024-09-19T07:53:46.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.026+0000 7fbb7fcc0640 1 -- 172.21.15.98:0/68318895 wait complete. 2024-09-19T07:53:46.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.027+0000 7fbb7fcc0640 1 Processor -- start 2024-09-19T07:53:46.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.027+0000 7fbb7fcc0640 1 -- start start 2024-09-19T07:53:46.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.027+0000 7fbb7fcc0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb78106210 0x7fbb7819dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:46.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.027+0000 7fbb7fcc0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbb78107be0 con 0x7fbb78106210 2024-09-19T07:53:46.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.027+0000 7fbb7da5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb78106210 0x7fbb7819dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:46.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.027+0000 7fbb7da5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb78106210 0x7fbb7819dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57470/0 (socket says 172.21.15.98:57470) 2024-09-19T07:53:46.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.027+0000 7fbb7da5c640 1 -- 172.21.15.98:0/241442504 learned_addr learned my addr 172.21.15.98:0/241442504 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:46.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.027+0000 7fbb7da5c640 1 -- 172.21.15.98:0/241442504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7819e010 con 0x7fbb78106210 2024-09-19T07:53:46.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.027+0000 7fbb7da5c640 1 --2- 172.21.15.98:0/241442504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb78106210 0x7fbb7819dad0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbb740097f0 tx=0x7fbb74030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:46.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.028+0000 7fbb6effd640 1 -- 172.21.15.98:0/241442504 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb74042070 con 0x7fbb78106210 2024-09-19T07:53:46.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.028+0000 7fbb6effd640 1 -- 172.21.15.98:0/241442504 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbb74032ca0 con 0x7fbb78106210 2024-09-19T07:53:46.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.028+0000 7fbb7fcc0640 1 -- 172.21.15.98:0/241442504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb7819e2a0 con 0x7fbb78106210 2024-09-19T07:53:46.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.028+0000 7fbb6effd640 1 -- 172.21.15.98:0/241442504 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb74037040 con 0x7fbb78106210 2024-09-19T07:53:46.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.028+0000 7fbb7fcc0640 1 -- 172.21.15.98:0/241442504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb781a0f90 con 0x7fbb78106210 2024-09-19T07:53:46.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.028+0000 7fbb6effd640 1 -- 172.21.15.98:0/241442504 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbb74045070 con 0x7fbb78106210 2024-09-19T07:53:46.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.028+0000 7fbb6effd640 1 -- 172.21.15.98:0/241442504 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbb74047a90 con 0x7fbb78106210 2024-09-19T07:53:46.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.029+0000 7fbb7fcc0640 1 -- 172.21.15.98:0/241442504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb78106610 con 0x7fbb78106210 2024-09-19T07:53:46.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.031+0000 7fbb6effd640 1 -- 172.21.15.98:0/241442504 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fbb7403db10 con 0x7fbb78106210 2024-09-19T07:53:46.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.089+0000 7fbb7fcc0640 1 -- 172.21.15.98:0/241442504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fbb7802d9b0 con 0x7fbb78106210 2024-09-19T07:53:46.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.090+0000 7fbb6effd640 1 -- 172.21.15.98:0/241442504 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fbb7403d430 con 0x7fbb78106210 2024-09-19T07:53:46.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.091+0000 7fbb7fcc0640 1 -- 172.21.15.98:0/241442504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb78106210 msgr2=0x7fbb7819dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:46.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.091+0000 7fbb7fcc0640 1 --2- 172.21.15.98:0/241442504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb78106210 0x7fbb7819dad0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbb740097f0 tx=0x7fbb74030ec0 comp rx=0 tx=0).stop 2024-09-19T07:53:46.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.091+0000 7fbb7fcc0640 1 -- 172.21.15.98:0/241442504 shutdown_connections 2024-09-19T07:53:46.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.091+0000 7fbb7fcc0640 1 --2- 172.21.15.98:0/241442504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb78106210 0x7fbb7819dad0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:46.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.091+0000 7fbb7fcc0640 1 -- 172.21.15.98:0/241442504 >> 172.21.15.98:0/241442504 conn(0x7fbb78102020 msgr2=0x7fbb78194710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:46.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.092+0000 7fbb7fcc0640 1 -- 172.21.15.98:0/241442504 shutdown_connections 2024-09-19T07:53:46.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:46.092+0000 7fbb7fcc0640 1 -- 172.21.15.98:0/241442504 wait complete. 2024-09-19T07:53:46.724 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-19T07:53:46.746 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-19T07:53:46.747 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mgr... 2024-09-19T07:53:46.747 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-19T07:53:46.747 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-19T07:53:46.993 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1a02525a-765c-11ef-bceb-c7b262605968@mgr.x 2024-09-19T07:53:46.993 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1a02525a-765c-11ef-bceb-c7b262605968@mgr.x.service: Unit ceph-1a02525a-765c-11ef-bceb-c7b262605968@mgr.x.service not loaded. 2024-09-19T07:53:47.142 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1a02525a-765c-11ef-bceb-c7b262605968.target.wants/ceph-1a02525a-765c-11ef-bceb-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-1a02525a-765c-11ef-bceb-c7b262605968@.service. 2024-09-19T07:53:47.374 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:47 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/241442504' entity='client.admin' 2024-09-19T07:53:47.374 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:47 smithi098 systemd[1]: Starting Ceph mgr.x for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T07:53:47.668 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:47 smithi098 podman[31544]: 2024-09-19 07:53:47.276963583 +0000 UTC m=+0.016457991 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:53:47.668 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:47 smithi098 podman[31544]: 2024-09-19 07:53:47.474980679 +0000 UTC m=+0.214475126 container create 9a5fff8af5c4db892866a3a9fce9927bc0ef8e07b39b6dbab212d665658112f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x, RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-19T07:53:47.669 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:47 smithi098 podman[31544]: 2024-09-19 07:53:47.66924345 +0000 UTC m=+0.408737867 container init 9a5fff8af5c4db892866a3a9fce9927bc0ef8e07b39b6dbab212d665658112f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-19T07:53:47.793 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-19T07:53:47.794 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-09-19T07:53:47.805 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-19T07:53:47.805 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-09-19T07:53:47.806 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-09-19T07:53:47.806 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-09-19T07:53:47.812 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-19T07:53:47.812 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-09-19T07:53:47.822 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-19T07:53:47.822 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-09-19T07:53:47.822 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-09-19T07:53:47.822 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-19T07:53:47.822 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr to start... 2024-09-19T07:53:47.822 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr... 2024-09-19T07:53:47.934 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:47 smithi098 podman[31544]: 2024-09-19 07:53:47.671914211 +0000 UTC m=+0.411408613 container start 9a5fff8af5c4db892866a3a9fce9927bc0ef8e07b39b6dbab212d665658112f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=squid-2f599a4) 2024-09-19T07:53:47.935 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:47 smithi098 bash[31544]: 9a5fff8af5c4db892866a3a9fce9927bc0ef8e07b39b6dbab212d665658112f0 2024-09-19T07:53:47.935 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:47 smithi098 systemd[1]: Started Ceph mgr.x for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T07:53:47.935 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:47 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:47.791+0000 7fb3d4d4a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T07:53:47.935 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:47 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:47.859+0000 7fb3d4d4a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T07:53:48.231 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:48.009+0000 7fb3d4d4a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T07:53:48.231 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:48.080+0000 7fb3d4d4a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T07:53:48.232 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:48.229+0000 7fb3d4d4a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T07:53:48.584 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:48 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3543855581' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-19T07:53:48.584 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:48.378+0000 7fb3d4d4a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T07:53:48.867 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:48.703+0000 7fb3d4d4a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T07:53:48.868 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:48.766+0000 7fb3d4d4a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T07:53:48.872 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-19T07:53:48.872 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-09-19T07:53:48.872 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T07:53:48.872 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-09-19T07:53:48.872 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-19T07:53:48.872 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-19T07:53:48.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-19T07:53:48.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:48.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-19T07:53:48.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-19T07:53:48.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-09-19T07:53:48.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-19T07:53:48.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-19T07:53:48.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-09-19T07:53:48.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-19T07:53:48.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-19T07:53:48.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-19T07:53:48.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T07:53:48.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-19T07:53:48.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-19T07:53:48.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:48.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-19T07:53:48.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T07:53:48.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-19T07:53:48.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-19T07:53:48.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-19T07:53:48.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-19T07:53:48.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-19T07:53:48.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-19T07:53:48.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:48.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-19T07:53:48.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-19T07:53:48.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-19T07:53:48.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-19T07:53:48.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-19T07:53:48.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-19T07:53:48.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-19T07:53:48.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-19T07:53:48.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-19T07:53:48.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:48.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-19T07:53:48.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T07:53:48.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "btime": "2024-09-19T07:53:39:127203+0000", 2024-09-19T07:53:48.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-19T07:53:48.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-19T07:53:48.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:48.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-19T07:53:48.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-19T07:53:48.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-19T07:53:48.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-19T07:53:48.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-19T07:53:48.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-19T07:53:48.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-09-19T07:53:48.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-19T07:53:48.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-19T07:53:48.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:48.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-19T07:53:48.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T07:53:48.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-09-19T07:53:39.127858+0000", 2024-09-19T07:53:48.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-19T07:53:48.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:48.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-19T07:53:48.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-09-19T07:53:48.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.266+0000 7fa249ba6640 1 Processor -- start 2024-09-19T07:53:48.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.266+0000 7fa249ba6640 1 -- start start 2024-09-19T07:53:48.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.266+0000 7fa249ba6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2441083e0 0x7fa24410a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:48.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.266+0000 7fa249ba6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa24407bdb0 con 0x7fa2441083e0 2024-09-19T07:53:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.267+0000 7fa2437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2441083e0 0x7fa24410a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.267+0000 7fa2437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2441083e0 0x7fa24410a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49840/0 (socket says 172.21.15.98:49840) 2024-09-19T07:53:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.267+0000 7fa2437fe640 1 -- 172.21.15.98:0/2647350084 learned_addr learned my addr 172.21.15.98:0/2647350084 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.267+0000 7fa2437fe640 1 -- 172.21.15.98:0/2647350084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa24410ad10 con 0x7fa2441083e0 2024-09-19T07:53:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.268+0000 7fa2437fe640 1 --2- 172.21.15.98:0/2647350084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2441083e0 0x7fa24410a7d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa234009820 tx=0x7fa23402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e54b2caf3f9c34eb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.268+0000 7fa242ffd640 1 -- 172.21.15.98:0/2647350084 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa234037070 con 0x7fa2441083e0 2024-09-19T07:53:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.268+0000 7fa242ffd640 1 -- 172.21.15.98:0/2647350084 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa234032900 con 0x7fa2441083e0 2024-09-19T07:53:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.268+0000 7fa242ffd640 1 -- 172.21.15.98:0/2647350084 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa234030850 con 0x7fa2441083e0 2024-09-19T07:53:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.269+0000 7fa249ba6640 1 -- 172.21.15.98:0/2647350084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2441083e0 msgr2=0x7fa24410a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.269+0000 7fa249ba6640 1 --2- 172.21.15.98:0/2647350084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2441083e0 0x7fa24410a7d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa234009820 tx=0x7fa23402a270 comp rx=0 tx=0).stop 2024-09-19T07:53:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.269+0000 7fa249ba6640 1 -- 172.21.15.98:0/2647350084 shutdown_connections 2024-09-19T07:53:48.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.269+0000 7fa249ba6640 1 --2- 172.21.15.98:0/2647350084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2441083e0 0x7fa24410a7d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:48.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.269+0000 7fa249ba6640 1 -- 172.21.15.98:0/2647350084 >> 172.21.15.98:0/2647350084 conn(0x7fa244101ee0 msgr2=0x7fa244104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:48.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.269+0000 7fa249ba6640 1 -- 172.21.15.98:0/2647350084 shutdown_connections 2024-09-19T07:53:48.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.269+0000 7fa249ba6640 1 -- 172.21.15.98:0/2647350084 wait complete. 2024-09-19T07:53:48.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.269+0000 7fa249ba6640 1 Processor -- start 2024-09-19T07:53:48.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.269+0000 7fa249ba6640 1 -- start start 2024-09-19T07:53:48.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.270+0000 7fa249ba6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2441083e0 0x7fa244107c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:48.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.270+0000 7fa249ba6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa24410bc20 con 0x7fa2441083e0 2024-09-19T07:53:48.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.270+0000 7fa2437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2441083e0 0x7fa244107c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:48.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.270+0000 7fa2437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2441083e0 0x7fa244107c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49850/0 (socket says 172.21.15.98:49850) 2024-09-19T07:53:48.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.270+0000 7fa2437fe640 1 -- 172.21.15.98:0/3543855581 learned_addr learned my addr 172.21.15.98:0/3543855581 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:48.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.270+0000 7fa2437fe640 1 -- 172.21.15.98:0/3543855581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2441081a0 con 0x7fa2441083e0 2024-09-19T07:53:48.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.270+0000 7fa2437fe640 1 --2- 172.21.15.98:0/3543855581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2441083e0 0x7fa244107c60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa2340097f0 tx=0x7fa234030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:48.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.270+0000 7fa2417fa640 1 -- 172.21.15.98:0/3543855581 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa234037040 con 0x7fa2441083e0 2024-09-19T07:53:48.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.270+0000 7fa2417fa640 1 -- 172.21.15.98:0/3543855581 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa234040070 con 0x7fa2441083e0 2024-09-19T07:53:48.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.270+0000 7fa249ba6640 1 -- 172.21.15.98:0/3543855581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa244106390 con 0x7fa2441083e0 2024-09-19T07:53:48.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.270+0000 7fa249ba6640 1 -- 172.21.15.98:0/3543855581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa244106870 con 0x7fa2441083e0 2024-09-19T07:53:48.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.270+0000 7fa2417fa640 1 -- 172.21.15.98:0/3543855581 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa23403a390 con 0x7fa2441083e0 2024-09-19T07:53:48.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.271+0000 7fa2417fa640 1 -- 172.21.15.98:0/3543855581 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa23403a5e0 con 0x7fa2441083e0 2024-09-19T07:53:48.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.271+0000 7fa2417fa640 1 -- 172.21.15.98:0/3543855581 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa234045950 con 0x7fa2441083e0 2024-09-19T07:53:48.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.271+0000 7fa249ba6640 1 -- 172.21.15.98:0/3543855581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa244108840 con 0x7fa2441083e0 2024-09-19T07:53:48.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.274+0000 7fa2417fa640 1 -- 172.21.15.98:0/3543855581 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fa234045460 con 0x7fa2441083e0 2024-09-19T07:53:48.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.334+0000 7fa249ba6640 1 -- 172.21.15.98:0/3543855581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa244106da0 con 0x7fa2441083e0 2024-09-19T07:53:48.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.335+0000 7fa2417fa640 1 -- 172.21.15.98:0/3543855581 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7fa23403b430 con 0x7fa2441083e0 2024-09-19T07:53:48.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.336+0000 7fa249ba6640 1 -- 172.21.15.98:0/3543855581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2441083e0 msgr2=0x7fa244107c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:48.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.336+0000 7fa249ba6640 1 --2- 172.21.15.98:0/3543855581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2441083e0 0x7fa244107c60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa2340097f0 tx=0x7fa234030d80 comp rx=0 tx=0).stop 2024-09-19T07:53:48.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.336+0000 7fa249ba6640 1 -- 172.21.15.98:0/3543855581 shutdown_connections 2024-09-19T07:53:48.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.336+0000 7fa249ba6640 1 --2- 172.21.15.98:0/3543855581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2441083e0 0x7fa244107c60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:48.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.336+0000 7fa249ba6640 1 -- 172.21.15.98:0/3543855581 >> 172.21.15.98:0/3543855581 conn(0x7fa244101ee0 msgr2=0x7fa244102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:48.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.336+0000 7fa249ba6640 1 -- 172.21.15.98:0/3543855581 shutdown_connections 2024-09-19T07:53:48.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:48.336+0000 7fa249ba6640 1 -- 172.21.15.98:0/3543855581 wait complete. 2024-09-19T07:53:48.891 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (1/15)... 2024-09-19T07:53:49.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:48.953+0000 7fb3d4d4a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T07:53:50.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:49 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:49.838+0000 7fb3d4d4a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T07:53:50.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:49 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:49.905+0000 7fb3d4d4a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T07:53:50.892 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:50 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:50.619+0000 7fb3d4d4a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-19T07:53:50.892 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:50 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:50.739+0000 7fb3d4d4a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T07:53:50.892 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:50 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:50.890+0000 7fb3d4d4a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T07:53:51.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:50 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:50.955+0000 7fb3d4d4a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T07:53:51.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:51.115+0000 7fb3d4d4a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-19T07:53:51.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:51 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1904467488' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-19T07:53:51.493 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:51.185+0000 7fb3d4d4a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T07:53:51.493 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:51.248+0000 7fb3d4d4a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T07:53:51.493 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:51.420+0000 7fb3d4d4a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T07:53:51.494 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:51.492+0000 7fb3d4d4a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T07:53:51.772 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:51.566+0000 7fb3d4d4a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T07:53:51.772 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:51.653+0000 7fb3d4d4a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T07:53:51.773 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:51.712+0000 7fb3d4d4a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T07:53:51.773 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:51.771+0000 7fb3d4d4a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T07:53:51.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-19T07:53:51.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-09-19T07:53:51.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T07:53:51.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-09-19T07:53:51.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-19T07:53:51.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-19T07:53:51.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-19T07:53:51.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:51.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-19T07:53:51.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-19T07:53:51.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-09-19T07:53:51.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-19T07:53:51.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-19T07:53:51.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-09-19T07:53:51.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-19T07:53:51.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-19T07:53:51.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-19T07:53:51.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T07:53:51.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-19T07:53:51.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-19T07:53:51.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:51.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-19T07:53:51.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T07:53:51.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-19T07:53:51.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-19T07:53:51.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-19T07:53:51.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-19T07:53:51.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-19T07:53:51.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-19T07:53:51.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:51.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-19T07:53:51.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-19T07:53:51.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-19T07:53:51.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-19T07:53:51.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-19T07:53:51.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-19T07:53:51.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-19T07:53:51.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-19T07:53:51.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-19T07:53:51.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:51.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-19T07:53:51.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T07:53:51.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "btime": "2024-09-19T07:53:39:127203+0000", 2024-09-19T07:53:51.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-19T07:53:51.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-19T07:53:51.943 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:51.943 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-19T07:53:51.943 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-19T07:53:51.943 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-19T07:53:51.943 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-19T07:53:51.943 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-19T07:53:51.943 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-19T07:53:51.943 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-09-19T07:53:51.943 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-19T07:53:51.943 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-19T07:53:51.943 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:51.943 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-19T07:53:51.943 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T07:53:51.944 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-09-19T07:53:39.127858+0000", 2024-09-19T07:53:51.944 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-19T07:53:51.944 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:51.944 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-19T07:53:51.944 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-09-19T07:53:51.944 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.298+0000 7ff4101b0640 1 Processor -- start 2024-09-19T07:53:51.944 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.298+0000 7ff4101b0640 1 -- start start 2024-09-19T07:53:51.944 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.298+0000 7ff4101b0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff408108320 0x7ff40810a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:51.944 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.298+0000 7ff4101b0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff40807be90 con 0x7ff408108320 2024-09-19T07:53:51.944 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.299+0000 7ff40df4c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff408108320 0x7ff40810a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:51.944 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.299+0000 7ff40df4c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff408108320 0x7ff40810a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49854/0 (socket says 172.21.15.98:49854) 2024-09-19T07:53:51.944 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.299+0000 7ff40df4c640 1 -- 172.21.15.98:0/3413882309 learned_addr learned my addr 172.21.15.98:0/3413882309 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:51.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.299+0000 7ff40df4c640 1 -- 172.21.15.98:0/3413882309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40810ac50 con 0x7ff408108320 2024-09-19T07:53:51.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.300+0000 7ff40df4c640 1 --2- 172.21.15.98:0/3413882309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff408108320 0x7ff40810a710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff404009820 tx=0x7ff40402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a968474f54beecf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:51.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.300+0000 7ff40cf4a640 1 -- 172.21.15.98:0/3413882309 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff404037070 con 0x7ff408108320 2024-09-19T07:53:51.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.300+0000 7ff40cf4a640 1 -- 172.21.15.98:0/3413882309 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff404032900 con 0x7ff408108320 2024-09-19T07:53:51.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.300+0000 7ff40cf4a640 1 -- 172.21.15.98:0/3413882309 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff404030850 con 0x7ff408108320 2024-09-19T07:53:51.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.301+0000 7ff4101b0640 1 -- 172.21.15.98:0/3413882309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff408108320 msgr2=0x7ff40810a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:51.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.301+0000 7ff4101b0640 1 --2- 172.21.15.98:0/3413882309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff408108320 0x7ff40810a710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff404009820 tx=0x7ff40402a270 comp rx=0 tx=0).stop 2024-09-19T07:53:51.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.301+0000 7ff4101b0640 1 -- 172.21.15.98:0/3413882309 shutdown_connections 2024-09-19T07:53:51.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.301+0000 7ff4101b0640 1 --2- 172.21.15.98:0/3413882309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff408108320 0x7ff40810a710 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:51.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.301+0000 7ff4101b0640 1 -- 172.21.15.98:0/3413882309 >> 172.21.15.98:0/3413882309 conn(0x7ff408101f40 msgr2=0x7ff408104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:51.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.301+0000 7ff4101b0640 1 -- 172.21.15.98:0/3413882309 shutdown_connections 2024-09-19T07:53:51.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.301+0000 7ff4101b0640 1 -- 172.21.15.98:0/3413882309 wait complete. 2024-09-19T07:53:51.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.302+0000 7ff4101b0640 1 Processor -- start 2024-09-19T07:53:51.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.302+0000 7ff4101b0640 1 -- start start 2024-09-19T07:53:51.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.302+0000 7ff4101b0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff408108320 0x7ff40819b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:51.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.302+0000 7ff4101b0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff40810bc80 con 0x7ff408108320 2024-09-19T07:53:51.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.302+0000 7ff40df4c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff408108320 0x7ff40819b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:51.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.303+0000 7ff40df4c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff408108320 0x7ff40819b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49864/0 (socket says 172.21.15.98:49864) 2024-09-19T07:53:51.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.303+0000 7ff40df4c640 1 -- 172.21.15.98:0/1904467488 learned_addr learned my addr 172.21.15.98:0/1904467488 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:51.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.303+0000 7ff40df4c640 1 -- 172.21.15.98:0/1904467488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40819bdb0 con 0x7ff408108320 2024-09-19T07:53:51.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.303+0000 7ff40df4c640 1 --2- 172.21.15.98:0/1904467488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff408108320 0x7ff40819b870 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff4040097f0 tx=0x7ff404030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:51.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.303+0000 7ff3feffd640 1 -- 172.21.15.98:0/1904467488 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff404037040 con 0x7ff408108320 2024-09-19T07:53:51.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.303+0000 7ff4101b0640 1 -- 172.21.15.98:0/1904467488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff40819c040 con 0x7ff408108320 2024-09-19T07:53:51.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.303+0000 7ff3feffd640 1 -- 172.21.15.98:0/1904467488 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff404040070 con 0x7ff408108320 2024-09-19T07:53:51.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.304+0000 7ff3feffd640 1 -- 172.21.15.98:0/1904467488 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff40403a390 con 0x7ff408108320 2024-09-19T07:53:51.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.304+0000 7ff4101b0640 1 -- 172.21.15.98:0/1904467488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff40819ed30 con 0x7ff408108320 2024-09-19T07:53:51.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.304+0000 7ff3feffd640 1 -- 172.21.15.98:0/1904467488 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff40403a660 con 0x7ff408108320 2024-09-19T07:53:51.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.304+0000 7ff3feffd640 1 -- 172.21.15.98:0/1904467488 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff40403b430 con 0x7ff408108320 2024-09-19T07:53:51.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.305+0000 7ff4101b0640 1 -- 172.21.15.98:0/1904467488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff408108780 con 0x7ff408108320 2024-09-19T07:53:51.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.307+0000 7ff3feffd640 1 -- 172.21.15.98:0/1904467488 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7ff40403b630 con 0x7ff408108320 2024-09-19T07:53:51.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.368+0000 7ff4101b0640 1 -- 172.21.15.98:0/1904467488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff40819ef80 con 0x7ff408108320 2024-09-19T07:53:51.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.368+0000 7ff3feffd640 1 -- 172.21.15.98:0/1904467488 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7ff40403b810 con 0x7ff408108320 2024-09-19T07:53:51.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.369+0000 7ff4101b0640 1 -- 172.21.15.98:0/1904467488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff408108320 msgr2=0x7ff40819b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:51.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.369+0000 7ff4101b0640 1 --2- 172.21.15.98:0/1904467488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff408108320 0x7ff40819b870 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff4040097f0 tx=0x7ff404030d80 comp rx=0 tx=0).stop 2024-09-19T07:53:51.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.369+0000 7ff4101b0640 1 -- 172.21.15.98:0/1904467488 shutdown_connections 2024-09-19T07:53:51.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.369+0000 7ff4101b0640 1 --2- 172.21.15.98:0/1904467488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff408108320 0x7ff40819b870 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:51.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.369+0000 7ff4101b0640 1 -- 172.21.15.98:0/1904467488 >> 172.21.15.98:0/1904467488 conn(0x7ff408101f40 msgr2=0x7ff408102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:51.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.370+0000 7ff4101b0640 1 -- 172.21.15.98:0/1904467488 shutdown_connections 2024-09-19T07:53:51.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:51.370+0000 7ff4101b0640 1 -- 172.21.15.98:0/1904467488 wait complete. 2024-09-19T07:53:51.948 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (2/15)... 2024-09-19T07:53:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:52 smithi098 ceph-mon[31333]: Activating manager daemon x 2024-09-19T07:53:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:52 smithi098 ceph-mon[31333]: mgrmap e2: x(active, starting, since 0.00309323s) 2024-09-19T07:53:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:52 smithi098 ceph-mon[31333]: from='mgr.14100 172.21.15.98:0/4140561267' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-19T07:53:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:52 smithi098 ceph-mon[31333]: from='mgr.14100 172.21.15.98:0/4140561267' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-19T07:53:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:52 smithi098 ceph-mon[31333]: from='mgr.14100 172.21.15.98:0/4140561267' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-19T07:53:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:52 smithi098 ceph-mon[31333]: from='mgr.14100 172.21.15.98:0/4140561267' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:53:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:52 smithi098 ceph-mon[31333]: from='mgr.14100 172.21.15.98:0/4140561267' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-19T07:53:52.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:52 smithi098 ceph-mon[31333]: Manager daemon x is now available 2024-09-19T07:53:52.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:52 smithi098 ceph-mon[31333]: from='mgr.14100 172.21.15.98:0/4140561267' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-19T07:53:52.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:52 smithi098 ceph-mon[31333]: from='mgr.14100 172.21.15.98:0/4140561267' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-19T07:53:52.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:52 smithi098 ceph-mon[31333]: from='mgr.14100 172.21.15.98:0/4140561267' entity='mgr.x' 2024-09-19T07:53:52.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:52 smithi098 ceph-mon[31333]: from='mgr.14100 172.21.15.98:0/4140561267' entity='mgr.x' 2024-09-19T07:53:52.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:52 smithi098 ceph-mon[31333]: from='mgr.14100 172.21.15.98:0/4140561267' entity='mgr.x' 2024-09-19T07:53:52.933 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:52 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:52.563+0000 7fb3d4d4a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T07:53:54.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:54 smithi098 ceph-mon[31333]: mgrmap e3: x(active, since 1.00722s) 2024-09-19T07:53:55.164 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-19T07:53:55.165 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-09-19T07:53:55.165 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T07:53:55.165 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-09-19T07:53:55.165 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-19T07:53:55.165 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-19T07:53:55.165 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-19T07:53:55.165 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:55.165 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-19T07:53:55.165 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-19T07:53:55.165 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-09-19T07:53:55.165 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-19T07:53:55.165 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-19T07:53:55.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-09-19T07:53:55.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-19T07:53:55.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-19T07:53:55.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-19T07:53:55.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T07:53:55.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-19T07:53:55.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-19T07:53:55.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:55.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-19T07:53:55.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T07:53:55.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-19T07:53:55.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-19T07:53:55.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-19T07:53:55.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-19T07:53:55.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-19T07:53:55.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-19T07:53:55.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:55.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-19T07:53:55.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-19T07:53:55.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-19T07:53:55.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-19T07:53:55.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-19T07:53:55.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-19T07:53:55.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-19T07:53:55.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-19T07:53:55.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-19T07:53:55.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:55.176 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-19T07:53:55.176 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T07:53:55.176 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "btime": "2024-09-19T07:53:39:127203+0000", 2024-09-19T07:53:55.176 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-19T07:53:55.176 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-19T07:53:55.176 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:55.176 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-19T07:53:55.176 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-19T07:53:55.176 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-19T07:53:55.176 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-19T07:53:55.176 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-19T07:53:55.176 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-19T07:53:55.176 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-09-19T07:53:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-19T07:53:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-19T07:53:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-19T07:53:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T07:53:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-09-19T07:53:39.127858+0000", 2024-09-19T07:53:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-19T07:53:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-19T07:53:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-19T07:53:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-09-19T07:53:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.356+0000 7fedc27a2640 1 Processor -- start 2024-09-19T07:53:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.357+0000 7fedc27a2640 1 -- start start 2024-09-19T07:53:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.357+0000 7fedc27a2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedbc107bf0 0x7fedbc107ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.357+0000 7fedc27a2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1085c0 con 0x7fedbc107bf0 2024-09-19T07:53:55.178 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.357+0000 7fedc17a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedbc107bf0 0x7fedbc107ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:55.178 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.357+0000 7fedc17a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedbc107bf0 0x7fedbc107ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49906/0 (socket says 172.21.15.98:49906) 2024-09-19T07:53:55.178 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.357+0000 7fedc17a0640 1 -- 172.21.15.98:0/2597049566 learned_addr learned my addr 172.21.15.98:0/2597049566 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:55.178 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.357+0000 7fedc17a0640 1 -- 172.21.15.98:0/2597049566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedbc106850 con 0x7fedbc107bf0 2024-09-19T07:53:55.178 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.358+0000 7fedc17a0640 1 --2- 172.21.15.98:0/2597049566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedbc107bf0 0x7fedbc107ff0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fedb8009820 tx=0x7fedb802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41dcb5667aa12d49 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:55.178 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.358+0000 7fedb3fff640 1 -- 172.21.15.98:0/2597049566 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedb8037070 con 0x7fedbc107bf0 2024-09-19T07:53:55.178 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.358+0000 7fedb3fff640 1 -- 172.21.15.98:0/2597049566 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fedb8032900 con 0x7fedbc107bf0 2024-09-19T07:53:55.178 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.358+0000 7fedb3fff640 1 -- 172.21.15.98:0/2597049566 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedb8030850 con 0x7fedbc107bf0 2024-09-19T07:53:55.178 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.359+0000 7fedc27a2640 1 -- 172.21.15.98:0/2597049566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedbc107bf0 msgr2=0x7fedbc107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:55.178 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.359+0000 7fedc27a2640 1 --2- 172.21.15.98:0/2597049566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedbc107bf0 0x7fedbc107ff0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fedb8009820 tx=0x7fedb802a270 comp rx=0 tx=0).stop 2024-09-19T07:53:55.178 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.359+0000 7fedc27a2640 1 -- 172.21.15.98:0/2597049566 shutdown_connections 2024-09-19T07:53:55.178 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.359+0000 7fedc27a2640 1 --2- 172.21.15.98:0/2597049566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedbc107bf0 0x7fedbc107ff0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:55.178 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.359+0000 7fedc27a2640 1 -- 172.21.15.98:0/2597049566 >> 172.21.15.98:0/2597049566 conn(0x7fedbc101530 msgr2=0x7fedbc103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:55.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.359+0000 7fedc27a2640 1 -- 172.21.15.98:0/2597049566 shutdown_connections 2024-09-19T07:53:55.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.359+0000 7fedc27a2640 1 -- 172.21.15.98:0/2597049566 wait complete. 2024-09-19T07:53:55.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.360+0000 7fedc27a2640 1 Processor -- start 2024-09-19T07:53:55.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.360+0000 7fedc27a2640 1 -- start start 2024-09-19T07:53:55.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.360+0000 7fedc27a2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedbc107bf0 0x7fedbc191820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:55.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.360+0000 7fedc27a2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc10ac10 con 0x7fedbc107bf0 2024-09-19T07:53:55.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.360+0000 7fedc17a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedbc107bf0 0x7fedbc191820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:55.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.360+0000 7fedc17a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedbc107bf0 0x7fedbc191820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49916/0 (socket says 172.21.15.98:49916) 2024-09-19T07:53:55.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.360+0000 7fedc17a0640 1 -- 172.21.15.98:0/3307703088 learned_addr learned my addr 172.21.15.98:0/3307703088 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:55.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.360+0000 7fedc17a0640 1 -- 172.21.15.98:0/3307703088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedbc191d60 con 0x7fedbc107bf0 2024-09-19T07:53:55.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.360+0000 7fedc17a0640 1 --2- 172.21.15.98:0/3307703088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedbc107bf0 0x7fedbc191820 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fedb8009940 tx=0x7fedb8032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:55.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.361+0000 7fedb27fc640 1 -- 172.21.15.98:0/3307703088 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedb8042070 con 0x7fedbc107bf0 2024-09-19T07:53:55.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.361+0000 7fedb27fc640 1 -- 172.21.15.98:0/3307703088 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fedb80308c0 con 0x7fedbc107bf0 2024-09-19T07:53:55.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.361+0000 7fedc27a2640 1 -- 172.21.15.98:0/3307703088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedbc18ff50 con 0x7fedbc107bf0 2024-09-19T07:53:55.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.361+0000 7fedb27fc640 1 -- 172.21.15.98:0/3307703088 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedb8037040 con 0x7fedbc107bf0 2024-09-19T07:53:55.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.361+0000 7fedc27a2640 1 -- 172.21.15.98:0/3307703088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedbc190490 con 0x7fedbc107bf0 2024-09-19T07:53:55.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.362+0000 7fedb27fc640 1 -- 172.21.15.98:0/3307703088 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 3) ==== 46344+0+0 (secure 0 0 0) 0x7fedb8045070 con 0x7fedbc107bf0 2024-09-19T07:53:55.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.362+0000 7fedb27fc640 1 --2- 172.21.15.98:0/3307703088 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7feda80390a0 0x7feda803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:55.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.362+0000 7fedc27a2640 1 -- 172.21.15.98:0/3307703088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedbc108070 con 0x7fedbc107bf0 2024-09-19T07:53:55.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.362+0000 7fedb27fc640 1 -- 172.21.15.98:0/3307703088 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fedb806c110 con 0x7fedbc107bf0 2024-09-19T07:53:55.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.362+0000 7fedc0f9f640 1 --2- 172.21.15.98:0/3307703088 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7feda80390a0 0x7feda803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:55.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.362+0000 7fedc0f9f640 1 --2- 172.21.15.98:0/3307703088 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7feda80390a0 0x7feda803b560 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fedac0098c0 tx=0x7fedac006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:55.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.367+0000 7fedb27fc640 1 -- 172.21.15.98:0/3307703088 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fedb8045360 con 0x7fedbc107bf0 2024-09-19T07:53:55.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.570+0000 7fedb27fc640 1 -- 172.21.15.98:0/3307703088 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7fedb803c760 con 0x7fedbc107bf0 2024-09-19T07:53:55.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.598+0000 7fedc27a2640 1 -- 172.21.15.98:0/3307703088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fedbc195300 con 0x7fedbc107bf0 2024-09-19T07:53:55.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.599+0000 7fedb27fc640 1 -- 172.21.15.98:0/3307703088 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1290 (secure 0 0 0) 0x7fedb802ee80 con 0x7fedbc107bf0 2024-09-19T07:53:55.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.600+0000 7fedc27a2640 1 -- 172.21.15.98:0/3307703088 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7feda80390a0 msgr2=0x7feda803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:55.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.600+0000 7fedc27a2640 1 --2- 172.21.15.98:0/3307703088 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7feda80390a0 0x7feda803b560 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fedac0098c0 tx=0x7fedac006db0 comp rx=0 tx=0).stop 2024-09-19T07:53:55.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.600+0000 7fedc27a2640 1 -- 172.21.15.98:0/3307703088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedbc107bf0 msgr2=0x7fedbc191820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:55.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.601+0000 7fedc27a2640 1 --2- 172.21.15.98:0/3307703088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedbc107bf0 0x7fedbc191820 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fedb8009940 tx=0x7fedb8032ce0 comp rx=0 tx=0).stop 2024-09-19T07:53:55.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.601+0000 7fedc27a2640 1 -- 172.21.15.98:0/3307703088 shutdown_connections 2024-09-19T07:53:55.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.601+0000 7fedc27a2640 1 --2- 172.21.15.98:0/3307703088 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7feda80390a0 0x7feda803b560 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:55.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.601+0000 7fedc27a2640 1 --2- 172.21.15.98:0/3307703088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedbc107bf0 0x7fedbc191820 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:55.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.601+0000 7fedc27a2640 1 -- 172.21.15.98:0/3307703088 >> 172.21.15.98:0/3307703088 conn(0x7fedbc101530 msgr2=0x7fedbc102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:55.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.601+0000 7fedc27a2640 1 -- 172.21.15.98:0/3307703088 shutdown_connections 2024-09-19T07:53:55.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:54.601+0000 7fedc27a2640 1 -- 172.21.15.98:0/3307703088 wait complete. 2024-09-19T07:53:55.181 INFO:teuthology.orchestra.run.smithi098.stdout:mgr is available 2024-09-19T07:53:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:55 smithi098 ceph-mon[31333]: mgrmap e4: x(active, since 2s) 2024-09-19T07:53:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:55 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3307703088' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-19T07:53:56.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-19T07:53:56.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-09-19T07:53:56.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:53:56.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-09-19T07:53:56.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-19T07:53:56.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-19T07:53:56.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-19T07:53:56.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-19T07:53:56.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-19T07:53:56.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-19T07:53:56.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-19T07:53:56.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-19T07:53:56.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-09-19T07:53:56.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-19T07:53:56.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-19T07:53:56.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-19T07:53:56.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-19T07:53:56.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-19T07:53:56.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.590+0000 7fd9b3aed640 1 Processor -- start 2024-09-19T07:53:56.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.590+0000 7fd9b3aed640 1 -- start start 2024-09-19T07:53:56.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.590+0000 7fd9b3aed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ac1078b0 0x7fd9ac107cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:56.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.590+0000 7fd9b3aed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac108280 con 0x7fd9ac1078b0 2024-09-19T07:53:56.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.590+0000 7fd9b1889640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ac1078b0 0x7fd9ac107cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:56.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.590+0000 7fd9b1889640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ac1078b0 0x7fd9ac107cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49932/0 (socket says 172.21.15.98:49932) 2024-09-19T07:53:56.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.590+0000 7fd9b1889640 1 -- 172.21.15.98:0/1965856654 learned_addr learned my addr 172.21.15.98:0/1965856654 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:56.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.590+0000 7fd9b1889640 1 -- 172.21.15.98:0/1965856654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9ac108ab0 con 0x7fd9ac1078b0 2024-09-19T07:53:56.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.591+0000 7fd9b1889640 1 --2- 172.21.15.98:0/1965856654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ac1078b0 0x7fd9ac107cb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009820 tx=0x7fd9a802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242e7ad1be495d78 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:56.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.591+0000 7fd9b0887640 1 -- 172.21.15.98:0/1965856654 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9a8037070 con 0x7fd9ac1078b0 2024-09-19T07:53:56.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.591+0000 7fd9b0887640 1 -- 172.21.15.98:0/1965856654 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd9a8032900 con 0x7fd9ac1078b0 2024-09-19T07:53:56.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.591+0000 7fd9b0887640 1 -- 172.21.15.98:0/1965856654 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9a8030850 con 0x7fd9ac1078b0 2024-09-19T07:53:56.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.592+0000 7fd9b3aed640 1 -- 172.21.15.98:0/1965856654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ac1078b0 msgr2=0x7fd9ac107cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:56.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.592+0000 7fd9b3aed640 1 --2- 172.21.15.98:0/1965856654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ac1078b0 0x7fd9ac107cb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009820 tx=0x7fd9a802a270 comp rx=0 tx=0).stop 2024-09-19T07:53:56.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.592+0000 7fd9b3aed640 1 -- 172.21.15.98:0/1965856654 shutdown_connections 2024-09-19T07:53:56.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.592+0000 7fd9b3aed640 1 --2- 172.21.15.98:0/1965856654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ac1078b0 0x7fd9ac107cb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:56.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.592+0000 7fd9b3aed640 1 -- 172.21.15.98:0/1965856654 >> 172.21.15.98:0/1965856654 conn(0x7fd9ac103020 msgr2=0x7fd9ac105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:56.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.592+0000 7fd9b3aed640 1 -- 172.21.15.98:0/1965856654 shutdown_connections 2024-09-19T07:53:56.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.592+0000 7fd9b3aed640 1 -- 172.21.15.98:0/1965856654 wait complete. 2024-09-19T07:53:56.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.593+0000 7fd9b3aed640 1 Processor -- start 2024-09-19T07:53:56.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.593+0000 7fd9b3aed640 1 -- start start 2024-09-19T07:53:56.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.593+0000 7fd9b3aed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ac1078b0 0x7fd9ac19b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:56.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.593+0000 7fd9b3aed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac10be60 con 0x7fd9ac1078b0 2024-09-19T07:53:56.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.593+0000 7fd9b1889640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ac1078b0 0x7fd9ac19b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:56.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.593+0000 7fd9b1889640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ac1078b0 0x7fd9ac19b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49948/0 (socket says 172.21.15.98:49948) 2024-09-19T07:53:56.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.593+0000 7fd9b1889640 1 -- 172.21.15.98:0/264701875 learned_addr learned my addr 172.21.15.98:0/264701875 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:56.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.593+0000 7fd9b1889640 1 -- 172.21.15.98:0/264701875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9ac19be20 con 0x7fd9ac1078b0 2024-09-19T07:53:56.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.593+0000 7fd9b1889640 1 --2- 172.21.15.98:0/264701875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ac1078b0 0x7fd9ac19b8e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009940 tx=0x7fd9a8032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:56.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.593+0000 7fd9a2ffd640 1 -- 172.21.15.98:0/264701875 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9a8042070 con 0x7fd9ac1078b0 2024-09-19T07:53:56.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.593+0000 7fd9a2ffd640 1 -- 172.21.15.98:0/264701875 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd9a80308c0 con 0x7fd9ac1078b0 2024-09-19T07:53:56.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.593+0000 7fd9b3aed640 1 -- 172.21.15.98:0/264701875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9ac19c0b0 con 0x7fd9ac1078b0 2024-09-19T07:53:56.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.593+0000 7fd9a2ffd640 1 -- 172.21.15.98:0/264701875 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9a8037040 con 0x7fd9ac1078b0 2024-09-19T07:53:56.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.594+0000 7fd9b3aed640 1 -- 172.21.15.98:0/264701875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9ac19eda0 con 0x7fd9ac1078b0 2024-09-19T07:53:56.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.594+0000 7fd9a2ffd640 1 -- 172.21.15.98:0/264701875 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7fd9a8045070 con 0x7fd9ac1078b0 2024-09-19T07:53:56.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.595+0000 7fd9a2ffd640 1 --2- 172.21.15.98:0/264701875 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7fd9980391b0 0x7fd99803b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:56.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.595+0000 7fd9a2ffd640 1 -- 172.21.15.98:0/264701875 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd9a806c1e0 con 0x7fd9ac1078b0 2024-09-19T07:53:56.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.595+0000 7fd9b3aed640 1 -- 172.21.15.98:0/264701875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9ac107d30 con 0x7fd9ac1078b0 2024-09-19T07:53:56.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.595+0000 7fd9b1088640 1 --2- 172.21.15.98:0/264701875 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7fd9980391b0 0x7fd99803b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:56.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.595+0000 7fd9b1088640 1 --2- 172.21.15.98:0/264701875 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7fd9980391b0 0x7fd99803b670 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0098c0 tx=0x7fd99c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:56.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.600+0000 7fd9a2ffd640 1 -- 172.21.15.98:0/264701875 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fd9a804b170 con 0x7fd9ac1078b0 2024-09-19T07:53:56.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.765+0000 7fd9b3aed640 1 -- 172.21.15.98:0/264701875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fd9ac1a04d0 con 0x7fd9ac1078b0 2024-09-19T07:53:56.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.766+0000 7fd9a2ffd640 1 -- 172.21.15.98:0/264701875 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+429 (secure 0 0 0) 0x7fd9a802fd90 con 0x7fd9ac1078b0 2024-09-19T07:53:56.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.768+0000 7fd9b3aed640 1 -- 172.21.15.98:0/264701875 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7fd9980391b0 msgr2=0x7fd99803b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:56.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.768+0000 7fd9b3aed640 1 --2- 172.21.15.98:0/264701875 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7fd9980391b0 0x7fd99803b670 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0098c0 tx=0x7fd99c006db0 comp rx=0 tx=0).stop 2024-09-19T07:53:56.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.768+0000 7fd9b3aed640 1 -- 172.21.15.98:0/264701875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ac1078b0 msgr2=0x7fd9ac19b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:56.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.768+0000 7fd9b3aed640 1 --2- 172.21.15.98:0/264701875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ac1078b0 0x7fd9ac19b8e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009940 tx=0x7fd9a8032ce0 comp rx=0 tx=0).stop 2024-09-19T07:53:56.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.768+0000 7fd9b3aed640 1 -- 172.21.15.98:0/264701875 shutdown_connections 2024-09-19T07:53:56.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.768+0000 7fd9b3aed640 1 --2- 172.21.15.98:0/264701875 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7fd9980391b0 0x7fd99803b670 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:56.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.768+0000 7fd9b3aed640 1 --2- 172.21.15.98:0/264701875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ac1078b0 0x7fd9ac19b8e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:56.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.768+0000 7fd9b3aed640 1 -- 172.21.15.98:0/264701875 >> 172.21.15.98:0/264701875 conn(0x7fd9ac103020 msgr2=0x7fd9ac103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:56.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.768+0000 7fd9b3aed640 1 -- 172.21.15.98:0/264701875 shutdown_connections 2024-09-19T07:53:56.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:55.768+0000 7fd9b3aed640 1 -- 172.21.15.98:0/264701875 wait complete. 2024-09-19T07:53:56.322 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling cephadm module... 2024-09-19T07:53:56.899 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:56 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/264701875' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-19T07:53:57.880 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:57 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3843300836' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-19T07:53:57.881 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:57 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: ignoring --setuser ceph since I am not root 2024-09-19T07:53:57.881 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:57 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: ignoring --setgroup ceph since I am not root 2024-09-19T07:53:57.881 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:57 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:57.719+0000 7f8cd76fb040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T07:53:57.881 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:57 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:57.775+0000 7f8cd76fb040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T07:53:58.141 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:57 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:57.926+0000 7f8cd76fb040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T07:53:58.142 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:57 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:57.991+0000 7f8cd76fb040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T07:53:58.142 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:58 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:58.140+0000 7f8cd76fb040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T07:53:58.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.680+0000 7f00d7607640 1 Processor -- start 2024-09-19T07:53:58.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.680+0000 7f00d7607640 1 -- start start 2024-09-19T07:53:58.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.680+0000 7f00d7607640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d007dea0 0x7f00d007c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:58.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.680+0000 7f00d7607640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f00d007c8d0 con 0x7f00d007dea0 2024-09-19T07:53:58.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.680+0000 7f00d53a3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d007dea0 0x7f00d007c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:58.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.680+0000 7f00d53a3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d007dea0 0x7f00d007c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39980/0 (socket says 172.21.15.98:39980) 2024-09-19T07:53:58.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.680+0000 7f00d53a3640 1 -- 172.21.15.98:0/988156985 learned_addr learned my addr 172.21.15.98:0/988156985 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:58.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.680+0000 7f00d53a3640 1 -- 172.21.15.98:0/988156985 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d007ca50 con 0x7f00d007dea0 2024-09-19T07:53:58.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.681+0000 7f00d53a3640 1 --2- 172.21.15.98:0/988156985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d007dea0 0x7f00d007c300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009820 tx=0x7f00cc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef46948b486e1a66 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:58.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.681+0000 7f00c3fff640 1 -- 172.21.15.98:0/988156985 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00cc037070 con 0x7f00d007dea0 2024-09-19T07:53:58.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.681+0000 7f00c3fff640 1 -- 172.21.15.98:0/988156985 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f00cc032900 con 0x7f00d007dea0 2024-09-19T07:53:58.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.681+0000 7f00c3fff640 1 -- 172.21.15.98:0/988156985 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00cc030850 con 0x7f00d007dea0 2024-09-19T07:53:58.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.682+0000 7f00d7607640 1 -- 172.21.15.98:0/988156985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d007dea0 msgr2=0x7f00d007c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:58.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.682+0000 7f00d7607640 1 --2- 172.21.15.98:0/988156985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d007dea0 0x7f00d007c300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009820 tx=0x7f00cc02a270 comp rx=0 tx=0).stop 2024-09-19T07:53:58.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.682+0000 7f00d7607640 1 -- 172.21.15.98:0/988156985 shutdown_connections 2024-09-19T07:53:58.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.682+0000 7f00d7607640 1 --2- 172.21.15.98:0/988156985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d007dea0 0x7f00d007c300 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:58.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.682+0000 7f00d7607640 1 -- 172.21.15.98:0/988156985 >> 172.21.15.98:0/988156985 conn(0x7f00d01030b0 msgr2=0x7f00d01054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:58.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.682+0000 7f00d7607640 1 -- 172.21.15.98:0/988156985 shutdown_connections 2024-09-19T07:53:58.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.682+0000 7f00d7607640 1 -- 172.21.15.98:0/988156985 wait complete. 2024-09-19T07:53:58.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.682+0000 7f00d7607640 1 Processor -- start 2024-09-19T07:53:58.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.682+0000 7f00d7607640 1 -- start start 2024-09-19T07:53:58.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.683+0000 7f00d7607640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d007dea0 0x7f00d019daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:58.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.683+0000 7f00d7607640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f00d010def0 con 0x7f00d007dea0 2024-09-19T07:53:58.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.683+0000 7f00d53a3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d007dea0 0x7f00d019daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:58.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.683+0000 7f00d53a3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d007dea0 0x7f00d019daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39992/0 (socket says 172.21.15.98:39992) 2024-09-19T07:53:58.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.683+0000 7f00d53a3640 1 -- 172.21.15.98:0/3843300836 learned_addr learned my addr 172.21.15.98:0/3843300836 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:58.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.683+0000 7f00d53a3640 1 -- 172.21.15.98:0/3843300836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d019dfe0 con 0x7f00d007dea0 2024-09-19T07:53:58.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.683+0000 7f00d53a3640 1 --2- 172.21.15.98:0/3843300836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d007dea0 0x7f00d019daa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f00cc0097f0 tx=0x7f00cc030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:58.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.683+0000 7f00c27fc640 1 -- 172.21.15.98:0/3843300836 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00cc042070 con 0x7f00d007dea0 2024-09-19T07:53:58.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.683+0000 7f00c27fc640 1 -- 172.21.15.98:0/3843300836 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f00cc032ca0 con 0x7f00d007dea0 2024-09-19T07:53:58.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.683+0000 7f00d7607640 1 -- 172.21.15.98:0/3843300836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d019e270 con 0x7f00d007dea0 2024-09-19T07:53:58.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.683+0000 7f00c27fc640 1 -- 172.21.15.98:0/3843300836 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00cc037040 con 0x7f00d007dea0 2024-09-19T07:53:58.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.683+0000 7f00d7607640 1 -- 172.21.15.98:0/3843300836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d01a0f60 con 0x7f00d007dea0 2024-09-19T07:53:58.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.684+0000 7f00c27fc640 1 -- 172.21.15.98:0/3843300836 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7f00cc045070 con 0x7f00d007dea0 2024-09-19T07:53:58.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.684+0000 7f00c27fc640 1 --2- 172.21.15.98:0/3843300836 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f00bc0391b0 0x7f00bc03b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:58.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.684+0000 7f00d7607640 1 -- 172.21.15.98:0/3843300836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00d007c380 con 0x7f00d007dea0 2024-09-19T07:53:58.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.684+0000 7f00c27fc640 1 -- 172.21.15.98:0/3843300836 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f00cc06c1f0 con 0x7f00d007dea0 2024-09-19T07:53:58.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.684+0000 7f00d4ba2640 1 --2- 172.21.15.98:0/3843300836 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f00bc0391b0 0x7f00bc03b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:58.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.685+0000 7f00d4ba2640 1 --2- 172.21.15.98:0/3843300836 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f00bc0391b0 0x7f00bc03b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f00c40098c0 tx=0x7f00c4006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:58.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.689+0000 7f00c27fc640 1 -- 172.21.15.98:0/3843300836 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f00cc03d770 con 0x7f00d007dea0 2024-09-19T07:53:58.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:56.898+0000 7f00d7607640 1 -- 172.21.15.98:0/3843300836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f00d01a1130 con 0x7f00d007dea0 2024-09-19T07:53:58.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:57.576+0000 7f00c27fc640 1 -- 172.21.15.98:0/3843300836 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f00d01a1130 con 0x7f00d007dea0 2024-09-19T07:53:58.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:57.577+0000 7f00c27fc640 1 -- 172.21.15.98:0/3843300836 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7f00cc03c6a0 con 0x7f00d007dea0 2024-09-19T07:53:58.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:57.579+0000 7f00d7607640 1 -- 172.21.15.98:0/3843300836 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f00bc0391b0 msgr2=0x7f00bc03b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:58.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:57.579+0000 7f00d7607640 1 --2- 172.21.15.98:0/3843300836 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f00bc0391b0 0x7f00bc03b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f00c40098c0 tx=0x7f00c4006db0 comp rx=0 tx=0).stop 2024-09-19T07:53:58.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:57.579+0000 7f00d7607640 1 -- 172.21.15.98:0/3843300836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d007dea0 msgr2=0x7f00d019daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:58.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:57.579+0000 7f00d7607640 1 --2- 172.21.15.98:0/3843300836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d007dea0 0x7f00d019daa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f00cc0097f0 tx=0x7f00cc030ec0 comp rx=0 tx=0).stop 2024-09-19T07:53:58.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:57.579+0000 7f00d7607640 1 -- 172.21.15.98:0/3843300836 shutdown_connections 2024-09-19T07:53:58.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:57.579+0000 7f00d7607640 1 --2- 172.21.15.98:0/3843300836 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f00bc0391b0 0x7f00bc03b670 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:58.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:57.579+0000 7f00d7607640 1 --2- 172.21.15.98:0/3843300836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d007dea0 0x7f00d019daa0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:58.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:57.579+0000 7f00d7607640 1 -- 172.21.15.98:0/3843300836 >> 172.21.15.98:0/3843300836 conn(0x7f00d01030b0 msgr2=0x7f00d0103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:58.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:57.579+0000 7f00d7607640 1 -- 172.21.15.98:0/3843300836 shutdown_connections 2024-09-19T07:53:58.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:57.579+0000 7f00d7607640 1 -- 172.21.15.98:0/3843300836 wait complete. 2024-09-19T07:53:58.427 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:58 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:58.295+0000 7f8cd76fb040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T07:53:58.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:58 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3843300836' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-19T07:53:58.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:58 smithi098 ceph-mon[31333]: mgrmap e5: x(active, since 5s) 2024-09-19T07:53:58.682 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:58 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:58.619+0000 7f8cd76fb040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T07:53:59.128 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:58 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:58.682+0000 7f8cd76fb040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T07:53:59.129 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:58 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:58.875+0000 7f8cd76fb040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T07:53:59.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-09-19T07:53:59.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-19T07:53:59.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-19T07:53:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-19T07:53:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-19T07:53:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-09-19T07:53:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.602+0000 7f52b36ef640 1 Processor -- start 2024-09-19T07:53:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.602+0000 7f52b36ef640 1 -- start start 2024-09-19T07:53:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.602+0000 7f52b36ef640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52ac108320 0x7f52ac10a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.602+0000 7f52b36ef640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac07be90 con 0x7f52ac108320 2024-09-19T07:53:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.603+0000 7f52b148b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52ac108320 0x7f52ac10a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.603+0000 7f52b148b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52ac108320 0x7f52ac10a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40012/0 (socket says 172.21.15.98:40012) 2024-09-19T07:53:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.603+0000 7f52b148b640 1 -- 172.21.15.98:0/475805892 learned_addr learned my addr 172.21.15.98:0/475805892 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.603+0000 7f52b148b640 1 -- 172.21.15.98:0/475805892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52ac10ac50 con 0x7f52ac108320 2024-09-19T07:53:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.604+0000 7f52b148b640 1 --2- 172.21.15.98:0/475805892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52ac108320 0x7f52ac10a710 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f52a8009820 tx=0x7f52a802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac899b264a2e0927 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.604+0000 7f52a3fff640 1 -- 172.21.15.98:0/475805892 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52a8037070 con 0x7f52ac108320 2024-09-19T07:53:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.604+0000 7f52a3fff640 1 -- 172.21.15.98:0/475805892 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f52a8032900 con 0x7f52ac108320 2024-09-19T07:53:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.604+0000 7f52a3fff640 1 -- 172.21.15.98:0/475805892 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52a8030850 con 0x7f52ac108320 2024-09-19T07:53:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.604+0000 7f52b36ef640 1 -- 172.21.15.98:0/475805892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52ac108320 msgr2=0x7f52ac10a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.604+0000 7f52b36ef640 1 --2- 172.21.15.98:0/475805892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52ac108320 0x7f52ac10a710 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f52a8009820 tx=0x7f52a802a270 comp rx=0 tx=0).stop 2024-09-19T07:53:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.605+0000 7f52b36ef640 1 -- 172.21.15.98:0/475805892 shutdown_connections 2024-09-19T07:53:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.605+0000 7f52b36ef640 1 --2- 172.21.15.98:0/475805892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52ac108320 0x7f52ac10a710 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.605+0000 7f52b36ef640 1 -- 172.21.15.98:0/475805892 >> 172.21.15.98:0/475805892 conn(0x7f52ac101f40 msgr2=0x7f52ac104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.605+0000 7f52b36ef640 1 -- 172.21.15.98:0/475805892 shutdown_connections 2024-09-19T07:53:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.605+0000 7f52b36ef640 1 -- 172.21.15.98:0/475805892 wait complete. 2024-09-19T07:53:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.605+0000 7f52b36ef640 1 Processor -- start 2024-09-19T07:53:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.605+0000 7f52b36ef640 1 -- start start 2024-09-19T07:53:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.605+0000 7f52b36ef640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52ac108320 0x7f52ac19b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.605+0000 7f52b36ef640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac10bc80 con 0x7f52ac108320 2024-09-19T07:53:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.605+0000 7f52b148b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52ac108320 0x7f52ac19b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:53:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.605+0000 7f52b148b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52ac108320 0x7f52ac19b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40026/0 (socket says 172.21.15.98:40026) 2024-09-19T07:53:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.605+0000 7f52b148b640 1 -- 172.21.15.98:0/3899281707 learned_addr learned my addr 172.21.15.98:0/3899281707 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:53:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.606+0000 7f52b148b640 1 -- 172.21.15.98:0/3899281707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52ac19bdb0 con 0x7f52ac108320 2024-09-19T07:53:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.606+0000 7f52b148b640 1 --2- 172.21.15.98:0/3899281707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52ac108320 0x7f52ac19b870 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f52a8032ce0 tx=0x7f52a8032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:53:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.606+0000 7f52a27fc640 1 -- 172.21.15.98:0/3899281707 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52a8042070 con 0x7f52ac108320 2024-09-19T07:53:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.606+0000 7f52a27fc640 1 -- 172.21.15.98:0/3899281707 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f52a80308c0 con 0x7f52ac108320 2024-09-19T07:53:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.606+0000 7f52a27fc640 1 -- 172.21.15.98:0/3899281707 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52a8037040 con 0x7f52ac108320 2024-09-19T07:53:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.606+0000 7f52b36ef640 1 -- 172.21.15.98:0/3899281707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52ac19c040 con 0x7f52ac108320 2024-09-19T07:53:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.606+0000 7f52b36ef640 1 -- 172.21.15.98:0/3899281707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52ac19ed30 con 0x7f52ac108320 2024-09-19T07:53:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.607+0000 7f52a27fc640 1 -- 172.21.15.98:0/3899281707 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7f52a8045070 con 0x7f52ac108320 2024-09-19T07:53:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.607+0000 7f52b36ef640 1 -- 172.21.15.98:0/3899281707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52ac108780 con 0x7f52ac108320 2024-09-19T07:53:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.607+0000 7f52a27fc640 1 --2- 172.21.15.98:0/3899281707 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f5298039200 0x7f529803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:53:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.607+0000 7f52b0c8a640 1 -- 172.21.15.98:0/3899281707 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f5298039200 msgr2=0x7f529803b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1560445911 2024-09-19T07:53:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.607+0000 7f52a27fc640 1 -- 172.21.15.98:0/3899281707 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f52a806c2b0 con 0x7f52ac108320 2024-09-19T07:53:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.607+0000 7f52b0c8a640 1 --2- 172.21.15.98:0/3899281707 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f5298039200 0x7f529803b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T07:53:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.612+0000 7f52a27fc640 1 -- 172.21.15.98:0/3899281707 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f52a804b170 con 0x7f52ac108320 2024-09-19T07:53:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.808+0000 7f52b0c8a640 1 -- 172.21.15.98:0/3899281707 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f5298039200 msgr2=0x7f529803b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1560445911 2024-09-19T07:53:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.808+0000 7f52b0c8a640 1 --2- 172.21.15.98:0/3899281707 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f5298039200 0x7f529803b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-19T07:53:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.830+0000 7f52b36ef640 1 -- 172.21.15.98:0/3899281707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f52ac19f3b0 con 0x7f52ac108320 2024-09-19T07:53:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.831+0000 7f52a27fc640 1 -- 172.21.15.98:0/3899281707 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f52a806bcd0 con 0x7f52ac108320 2024-09-19T07:53:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.833+0000 7f52b36ef640 1 -- 172.21.15.98:0/3899281707 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f5298039200 msgr2=0x7f529803b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:53:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.833+0000 7f52b36ef640 1 --2- 172.21.15.98:0/3899281707 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f5298039200 0x7f529803b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.833+0000 7f52b36ef640 1 -- 172.21.15.98:0/3899281707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52ac108320 msgr2=0x7f52ac19b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:53:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.833+0000 7f52b36ef640 1 --2- 172.21.15.98:0/3899281707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52ac108320 0x7f52ac19b870 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f52a8032ce0 tx=0x7f52a8032d10 comp rx=0 tx=0).stop 2024-09-19T07:53:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.833+0000 7f52b36ef640 1 -- 172.21.15.98:0/3899281707 shutdown_connections 2024-09-19T07:53:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.833+0000 7f52b36ef640 1 --2- 172.21.15.98:0/3899281707 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f5298039200 0x7f529803b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:59.392 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.833+0000 7f52b36ef640 1 --2- 172.21.15.98:0/3899281707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52ac108320 0x7f52ac19b870 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:53:59.392 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.833+0000 7f52b36ef640 1 -- 172.21.15.98:0/3899281707 >> 172.21.15.98:0/3899281707 conn(0x7f52ac101f40 msgr2=0x7f52ac102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:53:59.392 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.833+0000 7f52b36ef640 1 -- 172.21.15.98:0/3899281707 shutdown_connections 2024-09-19T07:53:59.392 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:58.833+0000 7f52b36ef640 1 -- 172.21.15.98:0/3899281707 wait complete. 2024-09-19T07:53:59.392 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-09-19T07:53:59.392 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 5... 2024-09-19T07:53:59.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:53:59 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3899281707' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-19T07:54:00.034 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:59 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:59.775+0000 7f8cd76fb040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T07:54:00.034 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:53:59 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:53:59.846+0000 7f8cd76fb040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T07:54:00.804 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:00.548+0000 7f8cd76fb040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-19T07:54:00.804 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:00.663+0000 7f8cd76fb040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T07:54:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:00.802+0000 7f8cd76fb040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T07:54:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:00.862+0000 7f8cd76fb040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T07:54:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:01.011+0000 7f8cd76fb040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-19T07:54:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:01.083+0000 7f8cd76fb040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T07:54:01.365 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:01.142+0000 7f8cd76fb040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T07:54:01.365 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:01.300+0000 7f8cd76fb040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T07:54:01.365 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:01.364+0000 7f8cd76fb040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T07:54:01.631 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:01.429+0000 7f8cd76fb040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T07:54:01.632 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:01.510+0000 7f8cd76fb040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T07:54:01.632 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:01.568+0000 7f8cd76fb040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T07:54:01.632 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:01.630+0000 7f8cd76fb040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T07:54:02.682 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:02.415+0000 7f8cd76fb040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T07:54:02.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: Active manager daemon x restarted 2024-09-19T07:54:02.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: Activating manager daemon x 2024-09-19T07:54:02.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: osdmap e2: 0 total, 0 up, 0 in 2024-09-19T07:54:02.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: mgrmap e6: x(active, starting, since 0.00371808s) 2024-09-19T07:54:02.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:54:02.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-19T07:54:02.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-19T07:54:02.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-19T07:54:02.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-19T07:54:02.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: Manager daemon x is now available 2024-09-19T07:54:02.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' 2024-09-19T07:54:02.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' 2024-09-19T07:54:02.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:54:02.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:54:02.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-19T07:54:02.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:02 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-19T07:54:03.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:03 smithi098 ceph-mon[31333]: Found migration_current of "None". Setting to last migration. 2024-09-19T07:54:03.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:03 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' 2024-09-19T07:54:03.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:03 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' 2024-09-19T07:54:03.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:03 smithi098 ceph-mon[31333]: mgrmap e7: x(active, since 1.00616s) 2024-09-19T07:54:03.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-09-19T07:54:03.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-19T07:54:03.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-19T07:54:03.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-09-19T07:54:03.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.827+0000 7f9a1fafe640 1 Processor -- start 2024-09-19T07:54:03.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.827+0000 7f9a1fafe640 1 -- start start 2024-09-19T07:54:03.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.828+0000 7f9a1fafe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a18106210 0x7f9a18106610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:03.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.828+0000 7f9a1fafe640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1807be10 con 0x7f9a18106210 2024-09-19T07:54:03.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.828+0000 7f9a1d89a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a18106210 0x7f9a18106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:03.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.828+0000 7f9a1d89a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a18106210 0x7f9a18106610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40042/0 (socket says 172.21.15.98:40042) 2024-09-19T07:54:03.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.828+0000 7f9a1d89a640 1 -- 172.21.15.98:0/973767374 learned_addr learned my addr 172.21.15.98:0/973767374 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:03.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.828+0000 7f9a1d89a640 1 -- 172.21.15.98:0/973767374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a18106b50 con 0x7f9a18106210 2024-09-19T07:54:03.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.829+0000 7f9a1d89a640 1 --2- 172.21.15.98:0/973767374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a18106210 0x7f9a18106610 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009820 tx=0x7f9a1402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d620357860f694 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:03.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.829+0000 7f9a1c898640 1 -- 172.21.15.98:0/973767374 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a14037070 con 0x7f9a18106210 2024-09-19T07:54:03.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.829+0000 7f9a1c898640 1 -- 172.21.15.98:0/973767374 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9a14032900 con 0x7f9a18106210 2024-09-19T07:54:03.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.829+0000 7f9a1c898640 1 -- 172.21.15.98:0/973767374 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a14030850 con 0x7f9a18106210 2024-09-19T07:54:03.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.830+0000 7f9a1fafe640 1 -- 172.21.15.98:0/973767374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a18106210 msgr2=0x7f9a18106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:03.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.830+0000 7f9a1fafe640 1 --2- 172.21.15.98:0/973767374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a18106210 0x7f9a18106610 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009820 tx=0x7f9a1402a270 comp rx=0 tx=0).stop 2024-09-19T07:54:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.830+0000 7f9a1fafe640 1 -- 172.21.15.98:0/973767374 shutdown_connections 2024-09-19T07:54:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.830+0000 7f9a1fafe640 1 --2- 172.21.15.98:0/973767374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a18106210 0x7f9a18106610 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.830+0000 7f9a1fafe640 1 -- 172.21.15.98:0/973767374 >> 172.21.15.98:0/973767374 conn(0x7f9a18102020 msgr2=0x7f9a18104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.830+0000 7f9a1fafe640 1 -- 172.21.15.98:0/973767374 shutdown_connections 2024-09-19T07:54:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.830+0000 7f9a1fafe640 1 -- 172.21.15.98:0/973767374 wait complete. 2024-09-19T07:54:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.830+0000 7f9a1fafe640 1 Processor -- start 2024-09-19T07:54:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.830+0000 7f9a1fafe640 1 -- start start 2024-09-19T07:54:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.831+0000 7f9a1fafe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a18106210 0x7f9a1819dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.831+0000 7f9a1fafe640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a18107be0 con 0x7f9a18106210 2024-09-19T07:54:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.831+0000 7f9a1d89a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a18106210 0x7f9a1819dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.831+0000 7f9a1d89a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a18106210 0x7f9a1819dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40054/0 (socket says 172.21.15.98:40054) 2024-09-19T07:54:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.831+0000 7f9a1d89a640 1 -- 172.21.15.98:0/1895528166 learned_addr learned my addr 172.21.15.98:0/1895528166 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.831+0000 7f9a1d89a640 1 -- 172.21.15.98:0/1895528166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1819e010 con 0x7f9a18106210 2024-09-19T07:54:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.831+0000 7f9a1d89a640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a18106210 0x7f9a1819dad0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9a14031000 tx=0x7f9a140326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:03.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.831+0000 7f9a0affd640 1 -- 172.21.15.98:0/1895528166 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a14042070 con 0x7f9a18106210 2024-09-19T07:54:03.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.831+0000 7f9a0affd640 1 -- 172.21.15.98:0/1895528166 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9a14032be0 con 0x7f9a18106210 2024-09-19T07:54:03.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.831+0000 7f9a1fafe640 1 -- 172.21.15.98:0/1895528166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a1819e2a0 con 0x7f9a18106210 2024-09-19T07:54:03.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.831+0000 7f9a0affd640 1 -- 172.21.15.98:0/1895528166 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a14037040 con 0x7f9a18106210 2024-09-19T07:54:03.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.831+0000 7f9a1fafe640 1 -- 172.21.15.98:0/1895528166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a181a0f90 con 0x7f9a18106210 2024-09-19T07:54:03.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.832+0000 7f9a0affd640 1 -- 172.21.15.98:0/1895528166 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7f9a14045070 con 0x7f9a18106210 2024-09-19T07:54:03.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.832+0000 7f9a0affd640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f9a04039200 0x7f9a0403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:03.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.832+0000 7f9a0affd640 1 -- 172.21.15.98:0/1895528166 --> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9a0403bdd0 con 0x7f9a04039200 2024-09-19T07:54:03.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.832+0000 7f9a0affd640 1 -- 172.21.15.98:0/1895528166 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9a1406c1c0 con 0x7f9a18106210 2024-09-19T07:54:03.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.832+0000 7f9a1d099640 1 -- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f9a04039200 msgr2=0x7f9a0403b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1560445911 2024-09-19T07:54:03.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:53:59.832+0000 7f9a1d099640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f9a04039200 0x7f9a0403b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T07:54:03.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:00.033+0000 7f9a1d099640 1 -- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f9a04039200 msgr2=0x7f9a0403b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1560445911 2024-09-19T07:54:03.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:00.033+0000 7f9a1d099640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f9a04039200 0x7f9a0403b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-19T07:54:03.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:00.434+0000 7f9a1d099640 1 -- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f9a04039200 msgr2=0x7f9a0403b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1560445911 2024-09-19T07:54:03.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:00.434+0000 7f9a1d099640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f9a04039200 0x7f9a0403b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-19T07:54:03.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:01.235+0000 7f9a1d099640 1 -- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f9a04039200 msgr2=0x7f9a0403b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1560445911 2024-09-19T07:54:03.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:01.235+0000 7f9a1d099640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f9a04039200 0x7f9a0403b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-19T07:54:03.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:02.419+0000 7f9a0affd640 1 -- 172.21.15.98:0/1895528166 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 6) ==== 46228+0+0 (secure 0 0 0) 0x7f9a1406ad40 con 0x7f9a18106210 2024-09-19T07:54:03.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:02.419+0000 7f9a0affd640 1 -- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f9a04039200 msgr2=0x7f9a0403b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:54:03.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:02.419+0000 7f9a0affd640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f9a04039200 0x7f9a0403b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:03.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.421+0000 7f9a0affd640 1 -- 172.21.15.98:0/1895528166 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 7) ==== 46355+0+0 (secure 0 0 0) 0x7f9a1403d900 con 0x7f9a18106210 2024-09-19T07:54:03.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.421+0000 7f9a0affd640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f9a0403cb30 0x7f9a0403ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:03.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.421+0000 7f9a0affd640 1 -- 172.21.15.98:0/1895528166 --> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9a14044e50 con 0x7f9a0403cb30 2024-09-19T07:54:03.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.421+0000 7f9a1d099640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f9a0403cb30 0x7f9a0403ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:03.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.422+0000 7f9a1d099640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f9a0403cb30 0x7f9a0403ef20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c003980 tx=0x7f9a0c0092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:03.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.422+0000 7f9a0affd640 1 -- 172.21.15.98:0/1895528166 <== mgr.14118 v2:172.21.15.98:6800/16410793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f9a14044e50 con 0x7f9a0403cb30 2024-09-19T07:54:03.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.429+0000 7f9a1fafe640 1 -- 172.21.15.98:0/1895528166 --> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9a18106610 con 0x7f9a0403cb30 2024-09-19T07:54:03.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.430+0000 7f9a0affd640 1 -- 172.21.15.98:0/1895528166 <== mgr.14118 v2:172.21.15.98:6800/16410793 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f9a18106610 con 0x7f9a0403cb30 2024-09-19T07:54:03.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.430+0000 7f9a1fafe640 1 -- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f9a0403cb30 msgr2=0x7f9a0403ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:03.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.430+0000 7f9a1fafe640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f9a0403cb30 0x7f9a0403ef20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c003980 tx=0x7f9a0c0092b0 comp rx=0 tx=0).stop 2024-09-19T07:54:03.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.430+0000 7f9a1fafe640 1 -- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a18106210 msgr2=0x7f9a1819dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:03.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.430+0000 7f9a1fafe640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a18106210 0x7f9a1819dad0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9a14031000 tx=0x7f9a140326e0 comp rx=0 tx=0).stop 2024-09-19T07:54:03.926 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.430+0000 7f9a1fafe640 1 -- 172.21.15.98:0/1895528166 shutdown_connections 2024-09-19T07:54:03.926 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.430+0000 7f9a1fafe640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f9a0403cb30 0x7f9a0403ef20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:03.926 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.430+0000 7f9a1fafe640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:6800/1560445911,v1:172.21.15.98:6801/1560445911] conn(0x7f9a04039200 0x7f9a0403b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:03.926 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.430+0000 7f9a1fafe640 1 --2- 172.21.15.98:0/1895528166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a18106210 0x7f9a1819dad0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:03.926 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.430+0000 7f9a1fafe640 1 -- 172.21.15.98:0/1895528166 >> 172.21.15.98:0/1895528166 conn(0x7f9a18102020 msgr2=0x7f9a18194770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:03.926 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.430+0000 7f9a1fafe640 1 -- 172.21.15.98:0/1895528166 shutdown_connections 2024-09-19T07:54:03.926 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:03.430+0000 7f9a1fafe640 1 -- 172.21.15.98:0/1895528166 wait complete. 2024-09-19T07:54:03.926 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 5 is available 2024-09-19T07:54:03.926 INFO:teuthology.orchestra.run.smithi098.stdout:Setting orchestrator backend to cephadm... 2024-09-19T07:54:04.273 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:04.114+0000 7f8c3ea55640 -1 log_channel(cephadm) log [ERR] : [19/Sep/2024:07:54:04] ENGINE Error in HTTPServer.serve 2024-09-19T07:54:04.273 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: Traceback (most recent call last): 2024-09-19T07:54:04.273 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-19T07:54:04.273 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: self._connections.run(self.expiration_interval) 2024-09-19T07:54:04.273 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-19T07:54:04.274 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: self._run(expiration_interval) 2024-09-19T07:54:04.274 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-19T07:54:04.274 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: new_conn = self._from_server_socket(self.server.socket) 2024-09-19T07:54:04.274 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-19T07:54:04.274 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-19T07:54:04.274 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-19T07:54:04.274 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: s = self.context.wrap_socket( 2024-09-19T07:54:04.274 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-19T07:54:04.275 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: return self.sslsocket_class._create( 2024-09-19T07:54:04.275 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-19T07:54:04.275 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: self.do_handshake() 2024-09-19T07:54:04.275 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-19T07:54:04.275 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: self._sslobj.do_handshake() 2024-09-19T07:54:04.275 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-19T07:54:04.275 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:04.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-mon[31333]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-19T07:54:04.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-mon[31333]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-19T07:54:04.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:54:04.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:04 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' 2024-09-19T07:54:05.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.276+0000 7fb2fa396640 1 Processor -- start 2024-09-19T07:54:05.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.276+0000 7fb2fa396640 1 -- start start 2024-09-19T07:54:05.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.276+0000 7fb2fa396640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f4108420 0x7fb2f410a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:05.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.276+0000 7fb2fa396640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f407bdf0 con 0x7fb2f4108420 2024-09-19T07:54:05.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.276+0000 7fb2f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f4108420 0x7fb2f410a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:05.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.276+0000 7fb2f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f4108420 0x7fb2f410a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40142/0 (socket says 172.21.15.98:40142) 2024-09-19T07:54:05.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.276+0000 7fb2f3fff640 1 -- 172.21.15.98:0/3828308117 learned_addr learned my addr 172.21.15.98:0/3828308117 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:05.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.276+0000 7fb2f3fff640 1 -- 172.21.15.98:0/3828308117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f410ad50 con 0x7fb2f4108420 2024-09-19T07:54:05.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.277+0000 7fb2f3fff640 1 --2- 172.21.15.98:0/3828308117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f4108420 0x7fb2f410a810 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4009820 tx=0x7fb2e402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cca011ab0f78f2d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:05.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.277+0000 7fb2f37fe640 1 -- 172.21.15.98:0/3828308117 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2e4037070 con 0x7fb2f4108420 2024-09-19T07:54:05.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.277+0000 7fb2f37fe640 1 -- 172.21.15.98:0/3828308117 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb2e4032900 con 0x7fb2f4108420 2024-09-19T07:54:05.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.277+0000 7fb2f37fe640 1 -- 172.21.15.98:0/3828308117 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2e4030850 con 0x7fb2f4108420 2024-09-19T07:54:05.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.278+0000 7fb2fa396640 1 -- 172.21.15.98:0/3828308117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f4108420 msgr2=0x7fb2f410a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:05.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.278+0000 7fb2fa396640 1 --2- 172.21.15.98:0/3828308117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f4108420 0x7fb2f410a810 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4009820 tx=0x7fb2e402a270 comp rx=0 tx=0).stop 2024-09-19T07:54:05.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.278+0000 7fb2fa396640 1 -- 172.21.15.98:0/3828308117 shutdown_connections 2024-09-19T07:54:05.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.278+0000 7fb2fa396640 1 --2- 172.21.15.98:0/3828308117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f4108420 0x7fb2f410a810 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:05.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.278+0000 7fb2fa396640 1 -- 172.21.15.98:0/3828308117 >> 172.21.15.98:0/3828308117 conn(0x7fb2f4101fc0 msgr2=0x7fb2f41043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:05.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.278+0000 7fb2fa396640 1 -- 172.21.15.98:0/3828308117 shutdown_connections 2024-09-19T07:54:05.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.278+0000 7fb2fa396640 1 -- 172.21.15.98:0/3828308117 wait complete. 2024-09-19T07:54:05.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.278+0000 7fb2fa396640 1 Processor -- start 2024-09-19T07:54:05.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.278+0000 7fb2fa396640 1 -- start start 2024-09-19T07:54:05.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.279+0000 7fb2fa396640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f4108420 0x7fb2f4107ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:05.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.279+0000 7fb2fa396640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f410bcc0 con 0x7fb2f4108420 2024-09-19T07:54:05.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.279+0000 7fb2f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f4108420 0x7fb2f4107ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:05.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.279+0000 7fb2f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f4108420 0x7fb2f4107ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40156/0 (socket says 172.21.15.98:40156) 2024-09-19T07:54:05.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.279+0000 7fb2f3fff640 1 -- 172.21.15.98:0/4023452636 learned_addr learned my addr 172.21.15.98:0/4023452636 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:05.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.279+0000 7fb2f3fff640 1 -- 172.21.15.98:0/4023452636 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f41081e0 con 0x7fb2f4108420 2024-09-19T07:54:05.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.279+0000 7fb2f3fff640 1 --2- 172.21.15.98:0/4023452636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f4108420 0x7fb2f4107ca0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb2e40097f0 tx=0x7fb2e4030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:05.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.279+0000 7fb2f1ffb640 1 -- 172.21.15.98:0/4023452636 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2e4042070 con 0x7fb2f4108420 2024-09-19T07:54:05.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.279+0000 7fb2f1ffb640 1 -- 172.21.15.98:0/4023452636 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb2e4032ca0 con 0x7fb2f4108420 2024-09-19T07:54:05.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.279+0000 7fb2fa396640 1 -- 172.21.15.98:0/4023452636 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2f41063d0 con 0x7fb2f4108420 2024-09-19T07:54:05.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.279+0000 7fb2f1ffb640 1 -- 172.21.15.98:0/4023452636 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2e4037040 con 0x7fb2f4108420 2024-09-19T07:54:05.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.279+0000 7fb2fa396640 1 -- 172.21.15.98:0/4023452636 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2f4106910 con 0x7fb2f4108420 2024-09-19T07:54:05.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.280+0000 7fb2f1ffb640 1 -- 172.21.15.98:0/4023452636 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) ==== 46355+0+0 (secure 0 0 0) 0x7fb2e4045070 con 0x7fb2f4108420 2024-09-19T07:54:05.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.280+0000 7fb2fa396640 1 -- 172.21.15.98:0/4023452636 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2f4108880 con 0x7fb2f4108420 2024-09-19T07:54:05.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.281+0000 7fb2f1ffb640 1 --2- 172.21.15.98:0/4023452636 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7fb2e00390f0 0x7fb2e003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:05.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.281+0000 7fb2f1ffb640 1 -- 172.21.15.98:0/4023452636 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb2e406c1d0 con 0x7fb2f4108420 2024-09-19T07:54:05.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.281+0000 7fb2ebfff640 1 --2- 172.21.15.98:0/4023452636 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7fb2e00390f0 0x7fb2e003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:05.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.281+0000 7fb2ebfff640 1 --2- 172.21.15.98:0/4023452636 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7fb2e00390f0 0x7fb2e003b5b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0098c0 tx=0x7fb2dc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:05.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.286+0000 7fb2f1ffb640 1 -- 172.21.15.98:0/4023452636 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb2e403b330 con 0x7fb2f4108420 2024-09-19T07:54:05.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.464+0000 7fb2fa396640 1 -- 172.21.15.98:0/4023452636 --> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fb2f41953e0 con 0x7fb2e00390f0 2024-09-19T07:54:05.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.468+0000 7fb2f1ffb640 1 -- 172.21.15.98:0/4023452636 <== mgr.14118 v2:172.21.15.98:6800/16410793 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb2f41953e0 con 0x7fb2e00390f0 2024-09-19T07:54:05.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.471+0000 7fb2fa396640 1 -- 172.21.15.98:0/4023452636 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7fb2e00390f0 msgr2=0x7fb2e003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:05.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.471+0000 7fb2fa396640 1 --2- 172.21.15.98:0/4023452636 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7fb2e00390f0 0x7fb2e003b5b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0098c0 tx=0x7fb2dc006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:05.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.471+0000 7fb2fa396640 1 -- 172.21.15.98:0/4023452636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f4108420 msgr2=0x7fb2f4107ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:05.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.471+0000 7fb2fa396640 1 --2- 172.21.15.98:0/4023452636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f4108420 0x7fb2f4107ca0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb2e40097f0 tx=0x7fb2e4030ec0 comp rx=0 tx=0).stop 2024-09-19T07:54:05.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.472+0000 7fb2fa396640 1 -- 172.21.15.98:0/4023452636 shutdown_connections 2024-09-19T07:54:05.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.472+0000 7fb2fa396640 1 --2- 172.21.15.98:0/4023452636 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7fb2e00390f0 0x7fb2e003b5b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:05.031 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.472+0000 7fb2fa396640 1 --2- 172.21.15.98:0/4023452636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f4108420 0x7fb2f4107ca0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:05.031 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.472+0000 7fb2fa396640 1 -- 172.21.15.98:0/4023452636 >> 172.21.15.98:0/4023452636 conn(0x7fb2f4101fc0 msgr2=0x7fb2f4102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:05.031 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.472+0000 7fb2fa396640 1 -- 172.21.15.98:0/4023452636 shutdown_connections 2024-09-19T07:54:05.031 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:04.472+0000 7fb2fa396640 1 -- 172.21.15.98:0/4023452636 wait complete. 2024-09-19T07:54:05.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:05 smithi098 ceph-mon[31333]: [19/Sep/2024:07:54:03] ENGINE Bus STARTING 2024-09-19T07:54:05.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:05 smithi098 ceph-mon[31333]: [19/Sep/2024:07:54:03] ENGINE Serving on http://172.21.15.98:8765 2024-09-19T07:54:05.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:05 smithi098 ceph-mon[31333]: [19/Sep/2024:07:54:04] ENGINE Serving on https://172.21.15.98:7150 2024-09-19T07:54:05.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:05 smithi098 ceph-mon[31333]: [19/Sep/2024:07:54:04] ENGINE Bus STARTED 2024-09-19T07:54:05.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:05 smithi098 ceph-mon[31333]: [19/Sep/2024:07:54:04] ENGINE Error in HTTPServer.serve 2024-09-19T07:54:05.682 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-09-19T07:54:05.682 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-19T07:54:05.682 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._connections.run(self.expiration_interval) 2024-09-19T07:54:05.682 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-19T07:54:05.682 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._run(expiration_interval) 2024-09-19T07:54:05.682 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-19T07:54:05.683 INFO:journalctl@ceph.mon.a.smithi098.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-19T07:54:05.683 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-19T07:54:05.683 INFO:journalctl@ceph.mon.a.smithi098.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-19T07:54:05.683 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-19T07:54:05.683 INFO:journalctl@ceph.mon.a.smithi098.stdout: s = self.context.wrap_socket( 2024-09-19T07:54:05.683 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-19T07:54:05.683 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.sslsocket_class._create( 2024-09-19T07:54:05.683 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-19T07:54:05.683 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.do_handshake() 2024-09-19T07:54:05.683 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-19T07:54:05.683 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._sslobj.do_handshake() 2024-09-19T07:54:05.683 INFO:journalctl@ceph.mon.a.smithi098.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-19T07:54:05.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:05 smithi098 ceph-mon[31333]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:54:05.684 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:05 smithi098 ceph-mon[31333]: mgrmap e8: x(active, since 2s) 2024-09-19T07:54:05.684 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:05 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:54:06.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-19T07:54:06.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.395+0000 7ff52639b640 1 Processor -- start 2024-09-19T07:54:06.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.395+0000 7ff52639b640 1 -- start start 2024-09-19T07:54:06.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.396+0000 7ff52639b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff520108380 0x7ff52010a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:06.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.396+0000 7ff52639b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff52007be30 con 0x7ff520108380 2024-09-19T07:54:06.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.396+0000 7ff51ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff520108380 0x7ff52010a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:06.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.396+0000 7ff51ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff520108380 0x7ff52010a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40164/0 (socket says 172.21.15.98:40164) 2024-09-19T07:54:06.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.396+0000 7ff51ffff640 1 -- 172.21.15.98:0/634432802 learned_addr learned my addr 172.21.15.98:0/634432802 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:06.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.396+0000 7ff51ffff640 1 -- 172.21.15.98:0/634432802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52010acb0 con 0x7ff520108380 2024-09-19T07:54:06.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.397+0000 7ff51ffff640 1 --2- 172.21.15.98:0/634432802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff520108380 0x7ff52010a770 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff510009820 tx=0x7ff51002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4227ae8d6dfda61 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:06.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.397+0000 7ff51effd640 1 -- 172.21.15.98:0/634432802 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff510037070 con 0x7ff520108380 2024-09-19T07:54:06.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.397+0000 7ff51effd640 1 -- 172.21.15.98:0/634432802 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff510032900 con 0x7ff520108380 2024-09-19T07:54:06.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.397+0000 7ff51effd640 1 -- 172.21.15.98:0/634432802 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff510030850 con 0x7ff520108380 2024-09-19T07:54:06.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.397+0000 7ff52639b640 1 -- 172.21.15.98:0/634432802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff520108380 msgr2=0x7ff52010a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:06.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.397+0000 7ff52639b640 1 --2- 172.21.15.98:0/634432802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff520108380 0x7ff52010a770 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff510009820 tx=0x7ff51002a270 comp rx=0 tx=0).stop 2024-09-19T07:54:06.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.397+0000 7ff52639b640 1 -- 172.21.15.98:0/634432802 shutdown_connections 2024-09-19T07:54:06.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.397+0000 7ff52639b640 1 --2- 172.21.15.98:0/634432802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff520108380 0x7ff52010a770 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:06.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.397+0000 7ff52639b640 1 -- 172.21.15.98:0/634432802 >> 172.21.15.98:0/634432802 conn(0x7ff520101f40 msgr2=0x7ff520104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:06.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.397+0000 7ff52639b640 1 -- 172.21.15.98:0/634432802 shutdown_connections 2024-09-19T07:54:06.186 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.398+0000 7ff52639b640 1 -- 172.21.15.98:0/634432802 wait complete. 2024-09-19T07:54:06.186 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.398+0000 7ff52639b640 1 Processor -- start 2024-09-19T07:54:06.186 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.398+0000 7ff52639b640 1 -- start start 2024-09-19T07:54:06.186 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.398+0000 7ff52639b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff520108380 0x7ff52019b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:06.186 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.398+0000 7ff52639b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff52010bc80 con 0x7ff520108380 2024-09-19T07:54:06.186 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.398+0000 7ff51ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff520108380 0x7ff52019b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:06.186 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.398+0000 7ff51ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff520108380 0x7ff52019b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40174/0 (socket says 172.21.15.98:40174) 2024-09-19T07:54:06.186 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.398+0000 7ff51ffff640 1 -- 172.21.15.98:0/1072650383 learned_addr learned my addr 172.21.15.98:0/1072650383 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:06.187 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.399+0000 7ff51ffff640 1 -- 172.21.15.98:0/1072650383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52019bda0 con 0x7ff520108380 2024-09-19T07:54:06.187 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.399+0000 7ff51ffff640 1 --2- 172.21.15.98:0/1072650383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff520108380 0x7ff52019b860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff510032ce0 tx=0x7ff510032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:06.187 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.399+0000 7ff51d7fa640 1 -- 172.21.15.98:0/1072650383 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff510042070 con 0x7ff520108380 2024-09-19T07:54:06.187 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.399+0000 7ff51d7fa640 1 -- 172.21.15.98:0/1072650383 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff5100308c0 con 0x7ff520108380 2024-09-19T07:54:06.187 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.399+0000 7ff52639b640 1 -- 172.21.15.98:0/1072650383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52019c030 con 0x7ff520108380 2024-09-19T07:54:06.187 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.399+0000 7ff51d7fa640 1 -- 172.21.15.98:0/1072650383 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff510037040 con 0x7ff520108380 2024-09-19T07:54:06.187 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.399+0000 7ff52639b640 1 -- 172.21.15.98:0/1072650383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff52019ed20 con 0x7ff520108380 2024-09-19T07:54:06.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.400+0000 7ff51d7fa640 1 -- 172.21.15.98:0/1072650383 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7ff510045070 con 0x7ff520108380 2024-09-19T07:54:06.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.400+0000 7ff51d7fa640 1 --2- 172.21.15.98:0/1072650383 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7ff5080391b0 0x7ff50803b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:06.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.400+0000 7ff51d7fa640 1 -- 172.21.15.98:0/1072650383 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff51006c410 con 0x7ff520108380 2024-09-19T07:54:06.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.400+0000 7ff51f7fe640 1 --2- 172.21.15.98:0/1072650383 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7ff5080391b0 0x7ff50803b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:06.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.400+0000 7ff52639b640 1 -- 172.21.15.98:0/1072650383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5201087e0 con 0x7ff520108380 2024-09-19T07:54:06.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.401+0000 7ff51f7fe640 1 --2- 172.21.15.98:0/1072650383 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7ff5080391b0 0x7ff50803b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff5140098c0 tx=0x7ff514006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:06.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.405+0000 7ff51d7fa640 1 -- 172.21.15.98:0/1072650383 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ff51006be00 con 0x7ff520108380 2024-09-19T07:54:06.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.576+0000 7ff52639b640 1 -- 172.21.15.98:0/1072650383 --> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7ff520103280 con 0x7ff5080391b0 2024-09-19T07:54:06.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.577+0000 7ff51d7fa640 1 -- 172.21.15.98:0/1072650383 <== mgr.14118 v2:172.21.15.98:6800/16410793 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7ff520103280 con 0x7ff5080391b0 2024-09-19T07:54:06.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.578+0000 7ff52639b640 1 -- 172.21.15.98:0/1072650383 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7ff5080391b0 msgr2=0x7ff50803b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:06.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.578+0000 7ff52639b640 1 --2- 172.21.15.98:0/1072650383 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7ff5080391b0 0x7ff50803b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff5140098c0 tx=0x7ff514006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:06.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.578+0000 7ff52639b640 1 -- 172.21.15.98:0/1072650383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff520108380 msgr2=0x7ff52019b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:06.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.578+0000 7ff52639b640 1 --2- 172.21.15.98:0/1072650383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff520108380 0x7ff52019b860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff510032ce0 tx=0x7ff510032d10 comp rx=0 tx=0).stop 2024-09-19T07:54:06.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.579+0000 7ff52639b640 1 -- 172.21.15.98:0/1072650383 shutdown_connections 2024-09-19T07:54:06.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.579+0000 7ff52639b640 1 --2- 172.21.15.98:0/1072650383 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7ff5080391b0 0x7ff50803b670 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:06.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.579+0000 7ff52639b640 1 --2- 172.21.15.98:0/1072650383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff520108380 0x7ff52019b860 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:06.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.579+0000 7ff52639b640 1 -- 172.21.15.98:0/1072650383 >> 172.21.15.98:0/1072650383 conn(0x7ff520101f40 msgr2=0x7ff520102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:06.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.579+0000 7ff52639b640 1 -- 172.21.15.98:0/1072650383 shutdown_connections 2024-09-19T07:54:06.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:05.579+0000 7ff52639b640 1 -- 172.21.15.98:0/1072650383 wait complete. 2024-09-19T07:54:06.191 INFO:teuthology.orchestra.run.smithi098.stdout:Generating ssh key... 2024-09-19T07:54:06.763 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-mon[31333]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:54:07.181 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: Generating public/private rsa key pair. 2024-09-19T07:54:07.181 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: Your identification has been saved in /tmp/tmpbsquzza8/key 2024-09-19T07:54:07.181 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: Your public key has been saved in /tmp/tmpbsquzza8/key.pub 2024-09-19T07:54:07.181 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: The key fingerprint is: 2024-09-19T07:54:07.181 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: SHA256:Q8VpicB8t9wa7OoganZzP9uczXXjhGMwVDX2hXr6+lY ceph-1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:54:07.181 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: The key's randomart image is: 2024-09-19T07:54:07.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: +---[RSA 3072]----+ 2024-09-19T07:54:07.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: | o.. o.o ..=.| 2024-09-19T07:54:07.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: | o o.* . o +| 2024-09-19T07:54:07.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: | ..= + . .| 2024-09-19T07:54:07.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: | . * o . | 2024-09-19T07:54:07.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: | S. = o | 2024-09-19T07:54:07.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: | .o + . E| 2024-09-19T07:54:07.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: | . . . =.+.| 2024-09-19T07:54:07.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: | o.o...oo +..*..| 2024-09-19T07:54:07.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: | o.. o o+o+ ++.. | 2024-09-19T07:54:07.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: +----[SHA256]-----+ 2024-09-19T07:54:07.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.587+0000 7f18f6f6f640 1 Processor -- start 2024-09-19T07:54:07.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.587+0000 7f18f6f6f640 1 -- start start 2024-09-19T07:54:07.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.587+0000 7f18f6f6f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18f0106830 0x7f18f0106c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:07.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.587+0000 7f18f6f6f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f18f0107200 con 0x7f18f0106830 2024-09-19T07:54:07.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.587+0000 7f18f4d0b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18f0106830 0x7f18f0106c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:07.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.587+0000 7f18f4d0b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18f0106830 0x7f18f0106c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48476/0 (socket says 172.21.15.98:48476) 2024-09-19T07:54:07.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.587+0000 7f18f4d0b640 1 -- 172.21.15.98:0/2221784072 learned_addr learned my addr 172.21.15.98:0/2221784072 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:07.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.587+0000 7f18f4d0b640 1 -- 172.21.15.98:0/2221784072 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f0107a30 con 0x7f18f0106830 2024-09-19T07:54:07.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.588+0000 7f18f4d0b640 1 --2- 172.21.15.98:0/2221784072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18f0106830 0x7f18f0106c30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009820 tx=0x7f18e402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab954048f93010d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:07.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.588+0000 7f18ef7fe640 1 -- 172.21.15.98:0/2221784072 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18e4037070 con 0x7f18f0106830 2024-09-19T07:54:07.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.588+0000 7f18ef7fe640 1 -- 172.21.15.98:0/2221784072 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f18e4032900 con 0x7f18f0106830 2024-09-19T07:54:07.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.588+0000 7f18ef7fe640 1 -- 172.21.15.98:0/2221784072 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18e4030850 con 0x7f18f0106830 2024-09-19T07:54:07.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.589+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2221784072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18f0106830 msgr2=0x7f18f0106c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:07.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.589+0000 7f18f6f6f640 1 --2- 172.21.15.98:0/2221784072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18f0106830 0x7f18f0106c30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009820 tx=0x7f18e402a270 comp rx=0 tx=0).stop 2024-09-19T07:54:07.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.589+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2221784072 shutdown_connections 2024-09-19T07:54:07.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.589+0000 7f18f6f6f640 1 --2- 172.21.15.98:0/2221784072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18f0106830 0x7f18f0106c30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:07.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.589+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2221784072 >> 172.21.15.98:0/2221784072 conn(0x7f18f0102020 msgr2=0x7f18f0104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:07.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.589+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2221784072 shutdown_connections 2024-09-19T07:54:07.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.589+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2221784072 wait complete. 2024-09-19T07:54:07.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.590+0000 7f18f6f6f640 1 Processor -- start 2024-09-19T07:54:07.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.590+0000 7f18f6f6f640 1 -- start start 2024-09-19T07:54:07.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.590+0000 7f18f6f6f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18f0106830 0x7f18f01919f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:07.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.590+0000 7f18f6f6f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f18f010ade0 con 0x7f18f0106830 2024-09-19T07:54:07.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.590+0000 7f18f4d0b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18f0106830 0x7f18f01919f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:07.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.590+0000 7f18f4d0b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18f0106830 0x7f18f01919f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48478/0 (socket says 172.21.15.98:48478) 2024-09-19T07:54:07.491 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.590+0000 7f18f4d0b640 1 -- 172.21.15.98:0/2228488812 learned_addr learned my addr 172.21.15.98:0/2228488812 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:07.491 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.590+0000 7f18f4d0b640 1 -- 172.21.15.98:0/2228488812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f0191f30 con 0x7f18f0106830 2024-09-19T07:54:07.491 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.590+0000 7f18f4d0b640 1 --2- 172.21.15.98:0/2228488812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18f0106830 0x7f18f01919f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009940 tx=0x7f18e40326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:07.492 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.590+0000 7f18edffb640 1 -- 172.21.15.98:0/2228488812 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18e4042070 con 0x7f18f0106830 2024-09-19T07:54:07.492 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.590+0000 7f18edffb640 1 -- 172.21.15.98:0/2228488812 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f18e4032be0 con 0x7f18f0106830 2024-09-19T07:54:07.492 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.590+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2228488812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f0190120 con 0x7f18f0106830 2024-09-19T07:54:07.492 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.590+0000 7f18edffb640 1 -- 172.21.15.98:0/2228488812 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18e4037040 con 0x7f18f0106830 2024-09-19T07:54:07.492 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.591+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2228488812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f0190660 con 0x7f18f0106830 2024-09-19T07:54:07.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.591+0000 7f18edffb640 1 -- 172.21.15.98:0/2228488812 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f18e4045070 con 0x7f18f0106830 2024-09-19T07:54:07.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.591+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2228488812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18f0106cb0 con 0x7f18f0106830 2024-09-19T07:54:07.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.592+0000 7f18edffb640 1 --2- 172.21.15.98:0/2228488812 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f18e00391b0 0x7f18e003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:07.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.592+0000 7f18edffb640 1 -- 172.21.15.98:0/2228488812 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f18e406c1e0 con 0x7f18f0106830 2024-09-19T07:54:07.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.592+0000 7f18effff640 1 --2- 172.21.15.98:0/2228488812 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f18e00391b0 0x7f18e003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:07.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.592+0000 7f18effff640 1 --2- 172.21.15.98:0/2228488812 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f18e00391b0 0x7f18e003b670 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f18dc0098c0 tx=0x7f18dc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:07.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.597+0000 7f18edffb640 1 -- 172.21.15.98:0/2228488812 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f18e4045360 con 0x7f18f0106830 2024-09-19T07:54:07.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.763+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2228488812 --> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f18f0195820 con 0x7f18e00391b0 2024-09-19T07:54:07.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.883+0000 7f18edffb640 1 -- 172.21.15.98:0/2228488812 <== mgr.14118 v2:172.21.15.98:6800/16410793 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f18f0195820 con 0x7f18e00391b0 2024-09-19T07:54:07.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.885+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2228488812 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f18e00391b0 msgr2=0x7f18e003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:07.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.885+0000 7f18f6f6f640 1 --2- 172.21.15.98:0/2228488812 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f18e00391b0 0x7f18e003b670 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f18dc0098c0 tx=0x7f18dc006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:07.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.885+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2228488812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18f0106830 msgr2=0x7f18f01919f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:07.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.885+0000 7f18f6f6f640 1 --2- 172.21.15.98:0/2228488812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18f0106830 0x7f18f01919f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009940 tx=0x7f18e40326e0 comp rx=0 tx=0).stop 2024-09-19T07:54:07.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.886+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2228488812 shutdown_connections 2024-09-19T07:54:07.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.886+0000 7f18f6f6f640 1 --2- 172.21.15.98:0/2228488812 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f18e00391b0 0x7f18e003b670 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:07.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.886+0000 7f18f6f6f640 1 --2- 172.21.15.98:0/2228488812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18f0106830 0x7f18f01919f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:07.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.886+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2228488812 >> 172.21.15.98:0/2228488812 conn(0x7f18f0102020 msgr2=0x7f18f0102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:07.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.886+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2228488812 shutdown_connections 2024-09-19T07:54:07.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:06.886+0000 7f18f6f6f640 1 -- 172.21.15.98:0/2228488812 wait complete. 2024-09-19T07:54:08.071 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:07 smithi098 ceph-mon[31333]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:54:08.072 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:07 smithi098 ceph-mon[31333]: Generating ssh key... 2024-09-19T07:54:08.072 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:07 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' 2024-09-19T07:54:08.072 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:07 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' 2024-09-19T07:54:08.611 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3KEOHcZ00COWq7d9RkUMFiy1T1qOb4K5EkvUBN7iaFERibi35eJLiVvmV9xJ5Yry6pD8cDm3pg0KqvQCNPcf30lBrcOdyXbC2lbPSiECybOAVStT+cvMOLZfHZ7QdwEBXwhlvHf6kdc9PQPwEkimLTPAopBbohe9U8+KhTT5cp/IA/8fkNXGeJMqy+momiw7o4tBuUDZKpuxv3lji0mXZw9bSTyEoY+rip+sR9zcBsjcV0B4aOUFfJoF0sET5LxJYEZq7WmX5/RwBBLn95atmaK3EfWIrdNxyWarHMEL0+bNP9U41nmDgh0LtpYa4xvQw/oN/PmYKt7JnsBCZQ3rjYa28rQdeFZD6qS+aTeO9J3WN3zFyMadFRlUPbQNXixzB2BLfyyeAze7U8ZzxjZKiSUCt2uBQAARiI4MKaZquNNuSqvsbXs+ShC7KS0oV9ky+ccq6YcVA7gKjxuIqT0hVXY79jqPqTBATCQald1c4wjghKc5Olqfr70nbLUfipZ8= ceph-1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:54:08.612 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.895+0000 7f7606072640 1 Processor -- start 2024-09-19T07:54:08.612 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.895+0000 7f7606072640 1 -- start start 2024-09-19T07:54:08.612 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.895+0000 7f7606072640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760007de60 0x7f760007c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:08.612 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.895+0000 7f7606072640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f760007c890 con 0x7f760007de60 2024-09-19T07:54:08.612 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.895+0000 7f75ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760007de60 0x7f760007c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:08.612 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.896+0000 7f75ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760007de60 0x7f760007c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48480/0 (socket says 172.21.15.98:48480) 2024-09-19T07:54:08.612 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.896+0000 7f75ff7fe640 1 -- 172.21.15.98:0/1480635599 learned_addr learned my addr 172.21.15.98:0/1480635599 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:08.612 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.896+0000 7f75ff7fe640 1 -- 172.21.15.98:0/1480635599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f760007ca10 con 0x7f760007de60 2024-09-19T07:54:08.612 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.896+0000 7f75ff7fe640 1 --2- 172.21.15.98:0/1480635599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760007de60 0x7f760007c2c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009820 tx=0x7f75f002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9973ac888ce6304 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:08.612 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.897+0000 7f75fe7fc640 1 -- 172.21.15.98:0/1480635599 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75f0037070 con 0x7f760007de60 2024-09-19T07:54:08.612 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.897+0000 7f75fe7fc640 1 -- 172.21.15.98:0/1480635599 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f75f0032900 con 0x7f760007de60 2024-09-19T07:54:08.613 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.897+0000 7f75fe7fc640 1 -- 172.21.15.98:0/1480635599 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75f0030850 con 0x7f760007de60 2024-09-19T07:54:08.613 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.897+0000 7f7606072640 1 -- 172.21.15.98:0/1480635599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760007de60 msgr2=0x7f760007c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:08.613 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.897+0000 7f7606072640 1 --2- 172.21.15.98:0/1480635599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760007de60 0x7f760007c2c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009820 tx=0x7f75f002a270 comp rx=0 tx=0).stop 2024-09-19T07:54:08.613 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.897+0000 7f7606072640 1 -- 172.21.15.98:0/1480635599 shutdown_connections 2024-09-19T07:54:08.613 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.897+0000 7f7606072640 1 --2- 172.21.15.98:0/1480635599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760007de60 0x7f760007c2c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:08.613 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.897+0000 7f7606072640 1 -- 172.21.15.98:0/1480635599 >> 172.21.15.98:0/1480635599 conn(0x7f76001030b0 msgr2=0x7f76001054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:08.613 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.897+0000 7f7606072640 1 -- 172.21.15.98:0/1480635599 shutdown_connections 2024-09-19T07:54:08.613 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.897+0000 7f7606072640 1 -- 172.21.15.98:0/1480635599 wait complete. 2024-09-19T07:54:08.613 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.898+0000 7f7606072640 1 Processor -- start 2024-09-19T07:54:08.613 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.898+0000 7f7606072640 1 -- start start 2024-09-19T07:54:08.613 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.898+0000 7f7606072640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760007de60 0x7f760019da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:08.613 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.898+0000 7f7606072640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f760010ded0 con 0x7f760007de60 2024-09-19T07:54:08.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.898+0000 7f75ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760007de60 0x7f760019da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:08.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.898+0000 7f75ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760007de60 0x7f760019da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48488/0 (socket says 172.21.15.98:48488) 2024-09-19T07:54:08.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.898+0000 7f75ff7fe640 1 -- 172.21.15.98:0/552668900 learned_addr learned my addr 172.21.15.98:0/552668900 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:08.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.899+0000 7f75ff7fe640 1 -- 172.21.15.98:0/552668900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f760019dfc0 con 0x7f760007de60 2024-09-19T07:54:08.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.899+0000 7f75ff7fe640 1 --2- 172.21.15.98:0/552668900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760007de60 0x7f760019da80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f75f0031000 tx=0x7f75f00326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:08.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.899+0000 7f75fcff9640 1 -- 172.21.15.98:0/552668900 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75f0042070 con 0x7f760007de60 2024-09-19T07:54:08.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.899+0000 7f75fcff9640 1 -- 172.21.15.98:0/552668900 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f75f0032be0 con 0x7f760007de60 2024-09-19T07:54:08.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.899+0000 7f7606072640 1 -- 172.21.15.98:0/552668900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f760019e250 con 0x7f760007de60 2024-09-19T07:54:08.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.899+0000 7f75fcff9640 1 -- 172.21.15.98:0/552668900 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75f0037040 con 0x7f760007de60 2024-09-19T07:54:08.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.899+0000 7f7606072640 1 -- 172.21.15.98:0/552668900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76001a0f40 con 0x7f760007de60 2024-09-19T07:54:08.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.900+0000 7f75fcff9640 1 -- 172.21.15.98:0/552668900 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f75f0045070 con 0x7f760007de60 2024-09-19T07:54:08.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.900+0000 7f7606072640 1 -- 172.21.15.98:0/552668900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f760007c340 con 0x7f760007de60 2024-09-19T07:54:08.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.900+0000 7f75fcff9640 1 --2- 172.21.15.98:0/552668900 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f75e8039200 0x7f75e803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:08.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.900+0000 7f75fcff9640 1 -- 172.21.15.98:0/552668900 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f75f006c3a0 con 0x7f760007de60 2024-09-19T07:54:08.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.900+0000 7f75feffd640 1 --2- 172.21.15.98:0/552668900 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f75e8039200 0x7f75e803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:08.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.900+0000 7f75feffd640 1 --2- 172.21.15.98:0/552668900 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f75e8039200 0x7f75e803b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75f40098c0 tx=0x7f75f4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:08.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:07.905+0000 7f75fcff9640 1 -- 172.21.15.98:0/552668900 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f75f003a780 con 0x7f760007de60 2024-09-19T07:54:08.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.071+0000 7f7606072640 1 -- 172.21.15.98:0/552668900 --> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f7600104350 con 0x7f75e8039200 2024-09-19T07:54:08.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.072+0000 7f75fcff9640 1 -- 172.21.15.98:0/552668900 <== mgr.14118 v2:172.21.15.98:6800/16410793 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f7600104350 con 0x7f75e8039200 2024-09-19T07:54:08.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.074+0000 7f7606072640 1 -- 172.21.15.98:0/552668900 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f75e8039200 msgr2=0x7f75e803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:08.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.074+0000 7f7606072640 1 --2- 172.21.15.98:0/552668900 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f75e8039200 0x7f75e803b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75f40098c0 tx=0x7f75f4006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:08.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.074+0000 7f7606072640 1 -- 172.21.15.98:0/552668900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760007de60 msgr2=0x7f760019da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:08.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.074+0000 7f7606072640 1 --2- 172.21.15.98:0/552668900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760007de60 0x7f760019da80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f75f0031000 tx=0x7f75f00326e0 comp rx=0 tx=0).stop 2024-09-19T07:54:08.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.074+0000 7f7606072640 1 -- 172.21.15.98:0/552668900 shutdown_connections 2024-09-19T07:54:08.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.074+0000 7f7606072640 1 --2- 172.21.15.98:0/552668900 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f75e8039200 0x7f75e803b6c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:08.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.074+0000 7f7606072640 1 --2- 172.21.15.98:0/552668900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760007de60 0x7f760019da80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:08.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.074+0000 7f7606072640 1 -- 172.21.15.98:0/552668900 >> 172.21.15.98:0/552668900 conn(0x7f76001030b0 msgr2=0x7f7600103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:08.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.074+0000 7f7606072640 1 -- 172.21.15.98:0/552668900 shutdown_connections 2024-09-19T07:54:08.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.074+0000 7f7606072640 1 -- 172.21.15.98:0/552668900 wait complete. 2024-09-19T07:54:08.617 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-19T07:54:08.617 INFO:teuthology.orchestra.run.smithi098.stdout:Adding key to root@localhost authorized_keys... 2024-09-19T07:54:08.618 INFO:teuthology.orchestra.run.smithi098.stdout:Adding host smithi098... 2024-09-19T07:54:09.140 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:08 smithi098 ceph-mon[31333]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:54:09.909 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:09 smithi098 ceph-mon[31333]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:54:11.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:10 smithi098 ceph-mon[31333]: Deploying cephadm binary to smithi098 2024-09-19T07:54:12.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Added host 'smithi098' with addr '172.21.15.98' 2024-09-19T07:54:12.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.961+0000 7fb2555bb640 1 Processor -- start 2024-09-19T07:54:12.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.962+0000 7fb2555bb640 1 -- start start 2024-09-19T07:54:12.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.962+0000 7fb2555bb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2501083a0 0x7fb25010a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:12.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.962+0000 7fb2555bb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb25007be10 con 0x7fb2501083a0 2024-09-19T07:54:12.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.962+0000 7fb24effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2501083a0 0x7fb25010a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:12.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.962+0000 7fb24effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2501083a0 0x7fb25010a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48500/0 (socket says 172.21.15.98:48500) 2024-09-19T07:54:12.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.962+0000 7fb24effd640 1 -- 172.21.15.98:0/2540138052 learned_addr learned my addr 172.21.15.98:0/2540138052 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:12.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.962+0000 7fb24effd640 1 -- 172.21.15.98:0/2540138052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb25010acd0 con 0x7fb2501083a0 2024-09-19T07:54:12.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.963+0000 7fb24effd640 1 --2- 172.21.15.98:0/2540138052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2501083a0 0x7fb25010a790 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb240009e80 tx=0x7fb24002a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbcb44c98cf870b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:12.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.963+0000 7fb24dffb640 1 -- 172.21.15.98:0/2540138052 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb240036070 con 0x7fb2501083a0 2024-09-19T07:54:12.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.963+0000 7fb24dffb640 1 -- 172.21.15.98:0/2540138052 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb2400318c0 con 0x7fb2501083a0 2024-09-19T07:54:12.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.963+0000 7fb24dffb640 1 -- 172.21.15.98:0/2540138052 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb240007ab0 con 0x7fb2501083a0 2024-09-19T07:54:12.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.964+0000 7fb2555bb640 1 -- 172.21.15.98:0/2540138052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2501083a0 msgr2=0x7fb25010a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:12.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.964+0000 7fb2555bb640 1 --2- 172.21.15.98:0/2540138052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2501083a0 0x7fb25010a790 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb240009e80 tx=0x7fb24002a7a0 comp rx=0 tx=0).stop 2024-09-19T07:54:12.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.964+0000 7fb2555bb640 1 -- 172.21.15.98:0/2540138052 shutdown_connections 2024-09-19T07:54:12.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.964+0000 7fb2555bb640 1 --2- 172.21.15.98:0/2540138052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2501083a0 0x7fb25010a790 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:12.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.964+0000 7fb2555bb640 1 -- 172.21.15.98:0/2540138052 >> 172.21.15.98:0/2540138052 conn(0x7fb250101f40 msgr2=0x7fb250104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:12.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.964+0000 7fb2555bb640 1 -- 172.21.15.98:0/2540138052 shutdown_connections 2024-09-19T07:54:12.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.964+0000 7fb2555bb640 1 -- 172.21.15.98:0/2540138052 wait complete. 2024-09-19T07:54:12.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.964+0000 7fb2555bb640 1 Processor -- start 2024-09-19T07:54:12.931 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.964+0000 7fb2555bb640 1 -- start start 2024-09-19T07:54:12.931 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.964+0000 7fb2555bb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2501083a0 0x7fb25019b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:12.931 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.965+0000 7fb2555bb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb25010bc40 con 0x7fb2501083a0 2024-09-19T07:54:12.931 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.965+0000 7fb24effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2501083a0 0x7fb25019b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:12.931 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.965+0000 7fb24effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2501083a0 0x7fb25019b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48508/0 (socket says 172.21.15.98:48508) 2024-09-19T07:54:12.931 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.965+0000 7fb24effd640 1 -- 172.21.15.98:0/2006840871 learned_addr learned my addr 172.21.15.98:0/2006840871 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:12.931 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.965+0000 7fb24effd640 1 -- 172.21.15.98:0/2006840871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb25019bdb0 con 0x7fb2501083a0 2024-09-19T07:54:12.931 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.965+0000 7fb24effd640 1 --2- 172.21.15.98:0/2006840871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2501083a0 0x7fb25019b870 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb240031ca0 tx=0x7fb240031cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:12.931 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.965+0000 7fb247fff640 1 -- 172.21.15.98:0/2006840871 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb240036050 con 0x7fb2501083a0 2024-09-19T07:54:12.931 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.965+0000 7fb247fff640 1 -- 172.21.15.98:0/2006840871 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb240042070 con 0x7fb2501083a0 2024-09-19T07:54:12.931 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.965+0000 7fb247fff640 1 -- 172.21.15.98:0/2006840871 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb24003d820 con 0x7fb2501083a0 2024-09-19T07:54:12.931 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.965+0000 7fb2555bb640 1 -- 172.21.15.98:0/2006840871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb25019c040 con 0x7fb2501083a0 2024-09-19T07:54:12.931 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.965+0000 7fb2555bb640 1 -- 172.21.15.98:0/2006840871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb25019c460 con 0x7fb2501083a0 2024-09-19T07:54:12.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.966+0000 7fb247fff640 1 -- 172.21.15.98:0/2006840871 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fb24003b610 con 0x7fb2501083a0 2024-09-19T07:54:12.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.966+0000 7fb247fff640 1 --2- 172.21.15.98:0/2006840871 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7fb23c0391b0 0x7fb23c03b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:12.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.966+0000 7fb247fff640 1 -- 172.21.15.98:0/2006840871 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb240007e00 con 0x7fb2501083a0 2024-09-19T07:54:12.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.966+0000 7fb2555bb640 1 -- 172.21.15.98:0/2006840871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb250108800 con 0x7fb2501083a0 2024-09-19T07:54:12.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.966+0000 7fb24e7fc640 1 --2- 172.21.15.98:0/2006840871 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7fb23c0391b0 0x7fb23c03b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:12.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.967+0000 7fb24e7fc640 1 --2- 172.21.15.98:0/2006840871 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7fb23c0391b0 0x7fb23c03b670 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb2380098c0 tx=0x7fb238006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:12.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:08.971+0000 7fb247fff640 1 -- 172.21.15.98:0/2006840871 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb240042220 con 0x7fb2501083a0 2024-09-19T07:54:12.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:09.140+0000 7fb2555bb640 1 -- 172.21.15.98:0/2006840871 --> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}) -- 0x7fb250103280 con 0x7fb23c0391b0 2024-09-19T07:54:12.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:12.331+0000 7fb247fff640 1 -- 172.21.15.98:0/2006840871 <== mgr.14118 v2:172.21.15.98:6800/16410793 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fb250103280 con 0x7fb23c0391b0 2024-09-19T07:54:12.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:12.334+0000 7fb2555bb640 1 -- 172.21.15.98:0/2006840871 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7fb23c0391b0 msgr2=0x7fb23c03b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:12.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:12.334+0000 7fb2555bb640 1 --2- 172.21.15.98:0/2006840871 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7fb23c0391b0 0x7fb23c03b670 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb2380098c0 tx=0x7fb238006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:12.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:12.334+0000 7fb2555bb640 1 -- 172.21.15.98:0/2006840871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2501083a0 msgr2=0x7fb25019b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:12.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:12.334+0000 7fb2555bb640 1 --2- 172.21.15.98:0/2006840871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2501083a0 0x7fb25019b870 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb240031ca0 tx=0x7fb240031cd0 comp rx=0 tx=0).stop 2024-09-19T07:54:12.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:12.334+0000 7fb2555bb640 1 -- 172.21.15.98:0/2006840871 shutdown_connections 2024-09-19T07:54:12.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:12.334+0000 7fb2555bb640 1 --2- 172.21.15.98:0/2006840871 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7fb23c0391b0 0x7fb23c03b670 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:12.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:12.334+0000 7fb2555bb640 1 --2- 172.21.15.98:0/2006840871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2501083a0 0x7fb25019b870 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:12.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:12.334+0000 7fb2555bb640 1 -- 172.21.15.98:0/2006840871 >> 172.21.15.98:0/2006840871 conn(0x7fb250101f40 msgr2=0x7fb250102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:12.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:12.334+0000 7fb2555bb640 1 -- 172.21.15.98:0/2006840871 shutdown_connections 2024-09-19T07:54:12.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:12.334+0000 7fb2555bb640 1 -- 172.21.15.98:0/2006840871 wait complete. 2024-09-19T07:54:12.933 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mon service... 2024-09-19T07:54:13.569 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:13 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' 2024-09-19T07:54:13.569 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:13 smithi098 ceph-mon[31333]: Added host smithi098 2024-09-19T07:54:13.569 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:13 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:54:14.371 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-19T07:54:14.371 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.364+0000 7f94b7064640 1 Processor -- start 2024-09-19T07:54:14.371 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.364+0000 7f94b7064640 1 -- start start 2024-09-19T07:54:14.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.364+0000 7f94b7064640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b0107900 0x7f94b0107d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:14.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.364+0000 7f94b7064640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94b01082d0 con 0x7f94b0107900 2024-09-19T07:54:14.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.364+0000 7f94b4e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b0107900 0x7f94b0107d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:14.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.364+0000 7f94b4e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b0107900 0x7f94b0107d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48514/0 (socket says 172.21.15.98:48514) 2024-09-19T07:54:14.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.364+0000 7f94b4e00640 1 -- 172.21.15.98:0/1835322691 learned_addr learned my addr 172.21.15.98:0/1835322691 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:14.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.365+0000 7f94b4e00640 1 -- 172.21.15.98:0/1835322691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b0108b00 con 0x7f94b0107900 2024-09-19T07:54:14.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.365+0000 7f94b4e00640 1 --2- 172.21.15.98:0/1835322691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b0107900 0x7f94b0107d00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009820 tx=0x7f94a402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f44dddc877038230 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:14.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.366+0000 7f94af7fe640 1 -- 172.21.15.98:0/1835322691 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94a4037070 con 0x7f94b0107900 2024-09-19T07:54:14.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.366+0000 7f94af7fe640 1 -- 172.21.15.98:0/1835322691 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f94a4032900 con 0x7f94b0107900 2024-09-19T07:54:14.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.366+0000 7f94af7fe640 1 -- 172.21.15.98:0/1835322691 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94a4030850 con 0x7f94b0107900 2024-09-19T07:54:14.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.366+0000 7f94b7064640 1 -- 172.21.15.98:0/1835322691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b0107900 msgr2=0x7f94b0107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:14.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.366+0000 7f94b7064640 1 --2- 172.21.15.98:0/1835322691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b0107900 0x7f94b0107d00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009820 tx=0x7f94a402a270 comp rx=0 tx=0).stop 2024-09-19T07:54:14.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.366+0000 7f94b7064640 1 -- 172.21.15.98:0/1835322691 shutdown_connections 2024-09-19T07:54:14.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.366+0000 7f94b7064640 1 --2- 172.21.15.98:0/1835322691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b0107900 0x7f94b0107d00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:14.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.366+0000 7f94b7064640 1 -- 172.21.15.98:0/1835322691 >> 172.21.15.98:0/1835322691 conn(0x7f94b01030b0 msgr2=0x7f94b01054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:14.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.366+0000 7f94b7064640 1 -- 172.21.15.98:0/1835322691 shutdown_connections 2024-09-19T07:54:14.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.366+0000 7f94b7064640 1 -- 172.21.15.98:0/1835322691 wait complete. 2024-09-19T07:54:14.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.367+0000 7f94b7064640 1 Processor -- start 2024-09-19T07:54:14.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.367+0000 7f94b7064640 1 -- start start 2024-09-19T07:54:14.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.367+0000 7f94b7064640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b0107900 0x7f94b019b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:14.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.367+0000 7f94b7064640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94b010beb0 con 0x7f94b0107900 2024-09-19T07:54:14.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.367+0000 7f94b4e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b0107900 0x7f94b019b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:14.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.367+0000 7f94b4e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b0107900 0x7f94b019b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48518/0 (socket says 172.21.15.98:48518) 2024-09-19T07:54:14.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.367+0000 7f94b4e00640 1 -- 172.21.15.98:0/843123193 learned_addr learned my addr 172.21.15.98:0/843123193 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:14.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.367+0000 7f94b4e00640 1 -- 172.21.15.98:0/843123193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b019be70 con 0x7f94b0107900 2024-09-19T07:54:14.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.368+0000 7f94b4e00640 1 --2- 172.21.15.98:0/843123193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b0107900 0x7f94b019b930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f94a40097f0 tx=0x7f94a4030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:14.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.368+0000 7f94adffb640 1 -- 172.21.15.98:0/843123193 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94a4042070 con 0x7f94b0107900 2024-09-19T07:54:14.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.368+0000 7f94adffb640 1 -- 172.21.15.98:0/843123193 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f94a4032ca0 con 0x7f94b0107900 2024-09-19T07:54:14.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.368+0000 7f94b7064640 1 -- 172.21.15.98:0/843123193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b019c100 con 0x7f94b0107900 2024-09-19T07:54:14.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.368+0000 7f94adffb640 1 -- 172.21.15.98:0/843123193 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94a4037040 con 0x7f94b0107900 2024-09-19T07:54:14.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.368+0000 7f94b7064640 1 -- 172.21.15.98:0/843123193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b019edf0 con 0x7f94b0107900 2024-09-19T07:54:14.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.369+0000 7f94adffb640 1 -- 172.21.15.98:0/843123193 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f94a4045070 con 0x7f94b0107900 2024-09-19T07:54:14.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.369+0000 7f94b7064640 1 -- 172.21.15.98:0/843123193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94b0107d80 con 0x7f94b0107900 2024-09-19T07:54:14.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.369+0000 7f94adffb640 1 --2- 172.21.15.98:0/843123193 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f94a0039200 0x7f94a003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:14.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.369+0000 7f94adffb640 1 -- 172.21.15.98:0/843123193 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f94a406c1b0 con 0x7f94b0107900 2024-09-19T07:54:14.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.369+0000 7f94affff640 1 --2- 172.21.15.98:0/843123193 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f94a0039200 0x7f94a003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:14.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.370+0000 7f94affff640 1 --2- 172.21.15.98:0/843123193 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f94a0039200 0x7f94a003b6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f949c0098c0 tx=0x7f949c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:14.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.374+0000 7f94adffb640 1 -- 172.21.15.98:0/843123193 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f94a4045360 con 0x7f94b0107900 2024-09-19T07:54:14.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.569+0000 7f94b7064640 1 -- 172.21.15.98:0/843123193 --> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f94b0104360 con 0x7f94a0039200 2024-09-19T07:54:14.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.574+0000 7f94adffb640 1 -- 172.21.15.98:0/843123193 <== mgr.14118 v2:172.21.15.98:6800/16410793 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f94b0104360 con 0x7f94a0039200 2024-09-19T07:54:14.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.576+0000 7f94b7064640 1 -- 172.21.15.98:0/843123193 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f94a0039200 msgr2=0x7f94a003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:14.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.576+0000 7f94b7064640 1 --2- 172.21.15.98:0/843123193 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f94a0039200 0x7f94a003b6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f949c0098c0 tx=0x7f949c006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:14.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.576+0000 7f94b7064640 1 -- 172.21.15.98:0/843123193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b0107900 msgr2=0x7f94b019b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:14.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.576+0000 7f94b7064640 1 --2- 172.21.15.98:0/843123193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b0107900 0x7f94b019b930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f94a40097f0 tx=0x7f94a4030ec0 comp rx=0 tx=0).stop 2024-09-19T07:54:14.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.576+0000 7f94b7064640 1 -- 172.21.15.98:0/843123193 shutdown_connections 2024-09-19T07:54:14.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.576+0000 7f94b7064640 1 --2- 172.21.15.98:0/843123193 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f94a0039200 0x7f94a003b6c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:14.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.576+0000 7f94b7064640 1 --2- 172.21.15.98:0/843123193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b0107900 0x7f94b019b930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:14.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.576+0000 7f94b7064640 1 -- 172.21.15.98:0/843123193 >> 172.21.15.98:0/843123193 conn(0x7f94b01030b0 msgr2=0x7f94b0103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:14.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.576+0000 7f94b7064640 1 -- 172.21.15.98:0/843123193 shutdown_connections 2024-09-19T07:54:14.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:13.576+0000 7f94b7064640 1 -- 172.21.15.98:0/843123193 wait complete. 2024-09-19T07:54:14.376 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mgr service... 2024-09-19T07:54:14.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:14 smithi098 ceph-mon[31333]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:54:14.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:14 smithi098 ceph-mon[31333]: Saving service mon spec with placement count:5 2024-09-19T07:54:14.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:14 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' 2024-09-19T07:54:16.377 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:16 smithi098 ceph-mon[31333]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:54:16.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:16 smithi098 ceph-mon[31333]: Saving service mgr spec with placement count:2 2024-09-19T07:54:16.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:16 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' 2024-09-19T07:54:16.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-19T07:54:16.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.814+0000 7f6c1a2c3640 1 Processor -- start 2024-09-19T07:54:16.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.814+0000 7f6c1a2c3640 1 -- start start 2024-09-19T07:54:16.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.814+0000 7f6c1a2c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c14106780 0x7f6c14106b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:16.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.815+0000 7f6c1a2c3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6c14107150 con 0x7f6c14106780 2024-09-19T07:54:16.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.815+0000 7f6c13fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c14106780 0x7f6c14106b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:16.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.815+0000 7f6c13fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c14106780 0x7f6c14106b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48532/0 (socket says 172.21.15.98:48532) 2024-09-19T07:54:16.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.815+0000 7f6c13fff640 1 -- 172.21.15.98:0/137064781 learned_addr learned my addr 172.21.15.98:0/137064781 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:16.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.815+0000 7f6c13fff640 1 -- 172.21.15.98:0/137064781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c14107980 con 0x7f6c14106780 2024-09-19T07:54:16.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.816+0000 7f6c13fff640 1 --2- 172.21.15.98:0/137064781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c14106780 0x7f6c14106b80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009820 tx=0x7f6c0402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1271b74385a6d895 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:16.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.816+0000 7f6c137fe640 1 -- 172.21.15.98:0/137064781 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c04037070 con 0x7f6c14106780 2024-09-19T07:54:16.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.816+0000 7f6c137fe640 1 -- 172.21.15.98:0/137064781 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6c04032900 con 0x7f6c14106780 2024-09-19T07:54:16.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.816+0000 7f6c137fe640 1 -- 172.21.15.98:0/137064781 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c04030850 con 0x7f6c14106780 2024-09-19T07:54:16.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.816+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/137064781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c14106780 msgr2=0x7f6c14106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:16.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.816+0000 7f6c1a2c3640 1 --2- 172.21.15.98:0/137064781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c14106780 0x7f6c14106b80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009820 tx=0x7f6c0402a270 comp rx=0 tx=0).stop 2024-09-19T07:54:16.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.816+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/137064781 shutdown_connections 2024-09-19T07:54:16.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.816+0000 7f6c1a2c3640 1 --2- 172.21.15.98:0/137064781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c14106780 0x7f6c14106b80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:16.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.816+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/137064781 >> 172.21.15.98:0/137064781 conn(0x7f6c14101f10 msgr2=0x7f6c14104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:16.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.816+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/137064781 shutdown_connections 2024-09-19T07:54:16.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.817+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/137064781 wait complete. 2024-09-19T07:54:16.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.817+0000 7f6c1a2c3640 1 Processor -- start 2024-09-19T07:54:16.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.817+0000 7f6c1a2c3640 1 -- start start 2024-09-19T07:54:16.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.817+0000 7f6c1a2c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c14106780 0x7f6c1419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:16.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.817+0000 7f6c1a2c3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1410ad30 con 0x7f6c14106780 2024-09-19T07:54:16.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.817+0000 7f6c13fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c14106780 0x7f6c1419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:16.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.817+0000 7f6c13fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c14106780 0x7f6c1419a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48534/0 (socket says 172.21.15.98:48534) 2024-09-19T07:54:16.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.817+0000 7f6c13fff640 1 -- 172.21.15.98:0/2691852170 learned_addr learned my addr 172.21.15.98:0/2691852170 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:16.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.817+0000 7f6c13fff640 1 -- 172.21.15.98:0/2691852170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1419acd0 con 0x7f6c14106780 2024-09-19T07:54:16.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.818+0000 7f6c13fff640 1 --2- 172.21.15.98:0/2691852170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c14106780 0x7f6c1419a790 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6c040097f0 tx=0x7f6c04030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:16.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.818+0000 7f6c11ffb640 1 -- 172.21.15.98:0/2691852170 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c04037040 con 0x7f6c14106780 2024-09-19T07:54:16.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.818+0000 7f6c11ffb640 1 -- 172.21.15.98:0/2691852170 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6c04040070 con 0x7f6c14106780 2024-09-19T07:54:16.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.818+0000 7f6c11ffb640 1 -- 172.21.15.98:0/2691852170 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c0403a390 con 0x7f6c14106780 2024-09-19T07:54:16.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.818+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/2691852170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1419af60 con 0x7f6c14106780 2024-09-19T07:54:16.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.818+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/2691852170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c1419dc50 con 0x7f6c14106780 2024-09-19T07:54:16.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.819+0000 7f6c11ffb640 1 -- 172.21.15.98:0/2691852170 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f6c04032900 con 0x7f6c14106780 2024-09-19T07:54:16.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.819+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/2691852170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c14106c00 con 0x7f6c14106780 2024-09-19T07:54:16.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.819+0000 7f6c11ffb640 1 --2- 172.21.15.98:0/2691852170 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f6c00039200 0x7f6c0003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:16.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.819+0000 7f6c11ffb640 1 -- 172.21.15.98:0/2691852170 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6c0406c730 con 0x7f6c14106780 2024-09-19T07:54:16.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.819+0000 7f6c0bfff640 1 --2- 172.21.15.98:0/2691852170 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f6c00039200 0x7f6c0003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:16.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.819+0000 7f6c0bfff640 1 --2- 172.21.15.98:0/2691852170 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f6c00039200 0x7f6c0003b6c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0098c0 tx=0x7f6bfc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:16.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:14.824+0000 7f6c11ffb640 1 -- 172.21.15.98:0/2691852170 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f6c0403a590 con 0x7f6c14106780 2024-09-19T07:54:16.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:15.008+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/2691852170 --> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f6c14103220 con 0x7f6c00039200 2024-09-19T07:54:16.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:15.012+0000 7f6c11ffb640 1 -- 172.21.15.98:0/2691852170 <== mgr.14118 v2:172.21.15.98:6800/16410793 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6c14103220 con 0x7f6c00039200 2024-09-19T07:54:16.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:15.014+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/2691852170 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f6c00039200 msgr2=0x7f6c0003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:16.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:15.014+0000 7f6c1a2c3640 1 --2- 172.21.15.98:0/2691852170 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f6c00039200 0x7f6c0003b6c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0098c0 tx=0x7f6bfc006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:16.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:15.014+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/2691852170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c14106780 msgr2=0x7f6c1419a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:16.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:15.014+0000 7f6c1a2c3640 1 --2- 172.21.15.98:0/2691852170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c14106780 0x7f6c1419a790 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6c040097f0 tx=0x7f6c04030d80 comp rx=0 tx=0).stop 2024-09-19T07:54:16.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:15.014+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/2691852170 shutdown_connections 2024-09-19T07:54:16.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:15.014+0000 7f6c1a2c3640 1 --2- 172.21.15.98:0/2691852170 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f6c00039200 0x7f6c0003b6c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:16.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:15.014+0000 7f6c1a2c3640 1 --2- 172.21.15.98:0/2691852170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c14106780 0x7f6c1419a790 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:16.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:15.014+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/2691852170 >> 172.21.15.98:0/2691852170 conn(0x7f6c14101f10 msgr2=0x7f6c14102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:16.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:15.014+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/2691852170 shutdown_connections 2024-09-19T07:54:16.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:15.014+0000 7f6c1a2c3640 1 -- 172.21.15.98:0/2691852170 wait complete. 2024-09-19T07:54:17.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:17 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' 2024-09-19T07:54:17.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:17 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' 2024-09-19T07:54:17.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:17 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2951006487' entity='client.admin' 2024-09-19T07:54:18.054 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.811+0000 7f8b0dcb6640 1 Processor -- start 2024-09-19T07:54:18.054 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.811+0000 7f8b0dcb6640 1 -- start start 2024-09-19T07:54:18.054 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.812+0000 7f8b0dcb6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b08107900 0x7f8b08107d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:18.054 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.812+0000 7f8b0dcb6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b081082d0 con 0x7f8b08107900 2024-09-19T07:54:18.054 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.812+0000 7f8b077fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b08107900 0x7f8b08107d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:18.054 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.812+0000 7f8b077fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b08107900 0x7f8b08107d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40874/0 (socket says 172.21.15.98:40874) 2024-09-19T07:54:18.054 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.812+0000 7f8b077fe640 1 -- 172.21.15.98:0/3793394282 learned_addr learned my addr 172.21.15.98:0/3793394282 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:18.054 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.812+0000 7f8b077fe640 1 -- 172.21.15.98:0/3793394282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b08108ab0 con 0x7f8b08107900 2024-09-19T07:54:18.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.813+0000 7f8b077fe640 1 --2- 172.21.15.98:0/3793394282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b08107900 0x7f8b08107d00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8af8009820 tx=0x7f8af802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c97c6bfce9508aa server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:18.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.813+0000 7f8b067fc640 1 -- 172.21.15.98:0/3793394282 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8af8037070 con 0x7f8b08107900 2024-09-19T07:54:18.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.813+0000 7f8b067fc640 1 -- 172.21.15.98:0/3793394282 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8af8032920 con 0x7f8b08107900 2024-09-19T07:54:18.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.813+0000 7f8b067fc640 1 -- 172.21.15.98:0/3793394282 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8af8030810 con 0x7f8b08107900 2024-09-19T07:54:18.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.814+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/3793394282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b08107900 msgr2=0x7f8b08107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:18.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.814+0000 7f8b0dcb6640 1 --2- 172.21.15.98:0/3793394282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b08107900 0x7f8b08107d00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8af8009820 tx=0x7f8af802a270 comp rx=0 tx=0).stop 2024-09-19T07:54:18.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.814+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/3793394282 shutdown_connections 2024-09-19T07:54:18.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.814+0000 7f8b0dcb6640 1 --2- 172.21.15.98:0/3793394282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b08107900 0x7f8b08107d00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:18.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.814+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/3793394282 >> 172.21.15.98:0/3793394282 conn(0x7f8b08103090 msgr2=0x7f8b081054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:18.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.814+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/3793394282 shutdown_connections 2024-09-19T07:54:18.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.814+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/3793394282 wait complete. 2024-09-19T07:54:18.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.814+0000 7f8b0dcb6640 1 Processor -- start 2024-09-19T07:54:18.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.815+0000 7f8b0dcb6640 1 -- start start 2024-09-19T07:54:18.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.815+0000 7f8b0dcb6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b08107900 0x7f8b0819b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:18.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.815+0000 7f8b0dcb6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0810be60 con 0x7f8b08107900 2024-09-19T07:54:18.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.815+0000 7f8b077fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b08107900 0x7f8b0819b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:18.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.815+0000 7f8b077fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b08107900 0x7f8b0819b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40878/0 (socket says 172.21.15.98:40878) 2024-09-19T07:54:18.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.815+0000 7f8b077fe640 1 -- 172.21.15.98:0/2951006487 learned_addr learned my addr 172.21.15.98:0/2951006487 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:18.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.815+0000 7f8b077fe640 1 -- 172.21.15.98:0/2951006487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b0819bdd0 con 0x7f8b08107900 2024-09-19T07:54:18.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.815+0000 7f8b077fe640 1 --2- 172.21.15.98:0/2951006487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b08107900 0x7f8b0819b890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8af8030560 tx=0x7f8af8032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:18.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.815+0000 7f8b04ff9640 1 -- 172.21.15.98:0/2951006487 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8af8037040 con 0x7f8b08107900 2024-09-19T07:54:18.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.816+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/2951006487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b0819c060 con 0x7f8b08107900 2024-09-19T07:54:18.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.816+0000 7f8b04ff9640 1 -- 172.21.15.98:0/2951006487 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8af8040070 con 0x7f8b08107900 2024-09-19T07:54:18.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.816+0000 7f8b04ff9640 1 -- 172.21.15.98:0/2951006487 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8af803a740 con 0x7f8b08107900 2024-09-19T07:54:18.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.816+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/2951006487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b0819ed50 con 0x7f8b08107900 2024-09-19T07:54:18.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.817+0000 7f8b04ff9640 1 -- 172.21.15.98:0/2951006487 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f8af803a8e0 con 0x7f8b08107900 2024-09-19T07:54:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.817+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/2951006487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b08107d80 con 0x7f8b08107900 2024-09-19T07:54:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.817+0000 7f8b04ff9640 1 --2- 172.21.15.98:0/2951006487 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f8af0039200 0x7f8af003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.817+0000 7f8b04ff9640 1 -- 172.21.15.98:0/2951006487 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8af806c930 con 0x7f8b08107900 2024-09-19T07:54:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.817+0000 7f8b06ffd640 1 --2- 172.21.15.98:0/2951006487 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f8af0039200 0x7f8af003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.818+0000 7f8b06ffd640 1 --2- 172.21.15.98:0/2951006487 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f8af0039200 0x7f8af003b6c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0098c0 tx=0x7f8afc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:16.822+0000 7f8b04ff9640 1 -- 172.21.15.98:0/2951006487 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f8af8044c70 con 0x7f8b08107900 2024-09-19T07:54:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:17.016+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/2951006487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f8b0819f350 con 0x7f8b08107900 2024-09-19T07:54:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:17.018+0000 7f8b04ff9640 1 -- 172.21.15.98:0/2951006487 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f8af8041270 con 0x7f8b08107900 2024-09-19T07:54:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:17.020+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/2951006487 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f8af0039200 msgr2=0x7f8af003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:17.020+0000 7f8b0dcb6640 1 --2- 172.21.15.98:0/2951006487 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f8af0039200 0x7f8af003b6c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0098c0 tx=0x7f8afc006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:17.020+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/2951006487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b08107900 msgr2=0x7f8b0819b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:17.020+0000 7f8b0dcb6640 1 --2- 172.21.15.98:0/2951006487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b08107900 0x7f8b0819b890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8af8030560 tx=0x7f8af8032670 comp rx=0 tx=0).stop 2024-09-19T07:54:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:17.020+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/2951006487 shutdown_connections 2024-09-19T07:54:18.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:17.020+0000 7f8b0dcb6640 1 --2- 172.21.15.98:0/2951006487 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f8af0039200 0x7f8af003b6c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:18.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:17.020+0000 7f8b0dcb6640 1 --2- 172.21.15.98:0/2951006487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b08107900 0x7f8b0819b890 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:18.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:17.021+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/2951006487 >> 172.21.15.98:0/2951006487 conn(0x7f8b08103090 msgr2=0x7f8b08103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:18.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:17.021+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/2951006487 shutdown_connections 2024-09-19T07:54:18.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:17.021+0000 7f8b0dcb6640 1 -- 172.21.15.98:0/2951006487 wait complete. 2024-09-19T07:54:19.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.471+0000 7f0bb2a0f640 1 Processor -- start 2024-09-19T07:54:19.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.471+0000 7f0bb2a0f640 1 -- start start 2024-09-19T07:54:19.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.472+0000 7f0bb2a0f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bac1067b0 0x7f0bac106bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:19.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.472+0000 7f0bb2a0f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac107180 con 0x7f0bac1067b0 2024-09-19T07:54:19.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.472+0000 7f0babfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bac1067b0 0x7f0bac106bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:19.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.472+0000 7f0babfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bac1067b0 0x7f0bac106bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40880/0 (socket says 172.21.15.98:40880) 2024-09-19T07:54:19.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.472+0000 7f0babfff640 1 -- 172.21.15.98:0/2200993733 learned_addr learned my addr 172.21.15.98:0/2200993733 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:19.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.472+0000 7f0babfff640 1 -- 172.21.15.98:0/2200993733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bac1079b0 con 0x7f0bac1067b0 2024-09-19T07:54:19.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.473+0000 7f0babfff640 1 --2- 172.21.15.98:0/2200993733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bac1067b0 0x7f0bac106bb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c009820 tx=0x7f0b9c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cef061bed39e07d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:19.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.473+0000 7f0baaffd640 1 -- 172.21.15.98:0/2200993733 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b9c037070 con 0x7f0bac1067b0 2024-09-19T07:54:19.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.473+0000 7f0baaffd640 1 -- 172.21.15.98:0/2200993733 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b9c032920 con 0x7f0bac1067b0 2024-09-19T07:54:19.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.473+0000 7f0baaffd640 1 -- 172.21.15.98:0/2200993733 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b9c030810 con 0x7f0bac1067b0 2024-09-19T07:54:19.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.473+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/2200993733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bac1067b0 msgr2=0x7f0bac106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:19.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.473+0000 7f0bb2a0f640 1 --2- 172.21.15.98:0/2200993733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bac1067b0 0x7f0bac106bb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c009820 tx=0x7f0b9c02a270 comp rx=0 tx=0).stop 2024-09-19T07:54:19.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.474+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/2200993733 shutdown_connections 2024-09-19T07:54:19.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.474+0000 7f0bb2a0f640 1 --2- 172.21.15.98:0/2200993733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bac1067b0 0x7f0bac106bb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:19.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.474+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/2200993733 >> 172.21.15.98:0/2200993733 conn(0x7f0bac101f40 msgr2=0x7f0bac104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:19.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.474+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/2200993733 shutdown_connections 2024-09-19T07:54:19.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.474+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/2200993733 wait complete. 2024-09-19T07:54:19.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.474+0000 7f0bb2a0f640 1 Processor -- start 2024-09-19T07:54:19.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.474+0000 7f0bb2a0f640 1 -- start start 2024-09-19T07:54:19.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.475+0000 7f0bb2a0f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bac1067b0 0x7f0bac07ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:19.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.475+0000 7f0bb2a0f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac10ad60 con 0x7f0bac1067b0 2024-09-19T07:54:19.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.475+0000 7f0babfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bac1067b0 0x7f0bac07ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:19.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.475+0000 7f0babfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bac1067b0 0x7f0bac07ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40892/0 (socket says 172.21.15.98:40892) 2024-09-19T07:54:19.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.475+0000 7f0babfff640 1 -- 172.21.15.98:0/3137592575 learned_addr learned my addr 172.21.15.98:0/3137592575 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:19.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.475+0000 7f0babfff640 1 -- 172.21.15.98:0/3137592575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bac19e630 con 0x7f0bac1067b0 2024-09-19T07:54:19.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.475+0000 7f0babfff640 1 --2- 172.21.15.98:0/3137592575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bac1067b0 0x7f0bac07ba30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c0097f0 tx=0x7f0b9c0327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:19.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.475+0000 7f0ba97fa640 1 -- 172.21.15.98:0/3137592575 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b9c037040 con 0x7f0bac1067b0 2024-09-19T07:54:19.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.475+0000 7f0ba97fa640 1 -- 172.21.15.98:0/3137592575 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b9c030cc0 con 0x7f0bac1067b0 2024-09-19T07:54:19.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.475+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/3137592575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bac19e800 con 0x7f0bac1067b0 2024-09-19T07:54:19.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.475+0000 7f0ba97fa640 1 -- 172.21.15.98:0/3137592575 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b9c03a640 con 0x7f0bac1067b0 2024-09-19T07:54:19.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.476+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/3137592575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bac19eba0 con 0x7f0bac1067b0 2024-09-19T07:54:19.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.476+0000 7f0ba97fa640 1 -- 172.21.15.98:0/3137592575 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f0b9c03a8f0 con 0x7f0bac1067b0 2024-09-19T07:54:19.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.476+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/3137592575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bac106c30 con 0x7f0bac1067b0 2024-09-19T07:54:19.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.477+0000 7f0ba97fa640 1 --2- 172.21.15.98:0/3137592575 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f0b94039200 0x7f0b9403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:19.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.477+0000 7f0ba97fa640 1 -- 172.21.15.98:0/3137592575 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0b9c06c8c0 con 0x7f0bac1067b0 2024-09-19T07:54:19.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.477+0000 7f0bab7fe640 1 --2- 172.21.15.98:0/3137592575 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f0b94039200 0x7f0b9403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:19.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.477+0000 7f0bab7fe640 1 --2- 172.21.15.98:0/3137592575 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f0b94039200 0x7f0b9403b6c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0ba00098c0 tx=0x7f0ba0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:19.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.482+0000 7f0ba97fa640 1 -- 172.21.15.98:0/3137592575 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f0b9c0328c0 con 0x7f0bac1067b0 2024-09-19T07:54:19.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.656+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/3137592575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f0bac19f4a0 con 0x7f0bac1067b0 2024-09-19T07:54:19.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.658+0000 7f0ba97fa640 1 -- 172.21.15.98:0/3137592575 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f0b9c031990 con 0x7f0bac1067b0 2024-09-19T07:54:19.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.661+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/3137592575 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f0b94039200 msgr2=0x7f0b9403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:19.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.661+0000 7f0bb2a0f640 1 --2- 172.21.15.98:0/3137592575 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f0b94039200 0x7f0b9403b6c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0ba00098c0 tx=0x7f0ba0006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:19.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.661+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/3137592575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bac1067b0 msgr2=0x7f0bac07ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:19.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.661+0000 7f0bb2a0f640 1 --2- 172.21.15.98:0/3137592575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bac1067b0 0x7f0bac07ba30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c0097f0 tx=0x7f0b9c0327b0 comp rx=0 tx=0).stop 2024-09-19T07:54:19.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.661+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/3137592575 shutdown_connections 2024-09-19T07:54:19.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.661+0000 7f0bb2a0f640 1 --2- 172.21.15.98:0/3137592575 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f0b94039200 0x7f0b9403b6c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:19.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.661+0000 7f0bb2a0f640 1 --2- 172.21.15.98:0/3137592575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bac1067b0 0x7f0bac07ba30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:19.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.661+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/3137592575 >> 172.21.15.98:0/3137592575 conn(0x7f0bac101f40 msgr2=0x7f0bac102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:19.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.661+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/3137592575 shutdown_connections 2024-09-19T07:54:19.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:18.661+0000 7f0bb2a0f640 1 -- 172.21.15.98:0/3137592575 wait complete. 2024-09-19T07:54:19.233 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling the dashboard module... 2024-09-19T07:54:19.883 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:19 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' 2024-09-19T07:54:19.883 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:19 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3137592575' entity='client.admin' 2024-09-19T07:54:19.884 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:19 smithi098 ceph-mon[31333]: from='mgr.14118 172.21.15.98:0/3397050790' entity='mgr.x' 2024-09-19T07:54:20.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:20 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3399188735' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-19T07:54:20.742 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:20 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: ignoring --setuser ceph since I am not root 2024-09-19T07:54:20.742 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:20 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: ignoring --setgroup ceph since I am not root 2024-09-19T07:54:20.742 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:20 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:20.741+0000 7f69aef91040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T07:54:21.022 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:20 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:20.801+0000 7f69aef91040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T07:54:21.022 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:20 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:20.953+0000 7f69aef91040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T07:54:21.022 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:21 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:21.022+0000 7f69aef91040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T07:54:21.281 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:21 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:21.176+0000 7f69aef91040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T07:54:21.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.677+0000 7f5a026e1640 1 Processor -- start 2024-09-19T07:54:21.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.677+0000 7f5a026e1640 1 -- start start 2024-09-19T07:54:21.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.677+0000 7f5a026e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59fc105c00 0x7f59fc106000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:21.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.677+0000 7f5a026e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc1065d0 con 0x7f59fc105c00 2024-09-19T07:54:21.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.678+0000 7f59fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59fc105c00 0x7f59fc106000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:21.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.678+0000 7f59fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59fc105c00 0x7f59fc106000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40908/0 (socket says 172.21.15.98:40908) 2024-09-19T07:54:21.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.678+0000 7f59fbfff640 1 -- 172.21.15.98:0/191400975 learned_addr learned my addr 172.21.15.98:0/191400975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:21.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.678+0000 7f59fbfff640 1 -- 172.21.15.98:0/191400975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59fc106750 con 0x7f59fc105c00 2024-09-19T07:54:21.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.679+0000 7f59fbfff640 1 --2- 172.21.15.98:0/191400975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59fc105c00 0x7f59fc106000 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009820 tx=0x7f59ec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bcff77915c22acf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:21.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.679+0000 7f59fb7fe640 1 -- 172.21.15.98:0/191400975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59ec037070 con 0x7f59fc105c00 2024-09-19T07:54:21.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.679+0000 7f59fb7fe640 1 -- 172.21.15.98:0/191400975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59ec032920 con 0x7f59fc105c00 2024-09-19T07:54:21.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.679+0000 7f59fb7fe640 1 -- 172.21.15.98:0/191400975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59ec030810 con 0x7f59fc105c00 2024-09-19T07:54:21.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.679+0000 7f5a026e1640 1 -- 172.21.15.98:0/191400975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59fc105c00 msgr2=0x7f59fc106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:21.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.679+0000 7f5a026e1640 1 --2- 172.21.15.98:0/191400975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59fc105c00 0x7f59fc106000 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009820 tx=0x7f59ec02a270 comp rx=0 tx=0).stop 2024-09-19T07:54:21.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.679+0000 7f5a026e1640 1 -- 172.21.15.98:0/191400975 shutdown_connections 2024-09-19T07:54:21.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.679+0000 7f5a026e1640 1 --2- 172.21.15.98:0/191400975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59fc105c00 0x7f59fc106000 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:21.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.680+0000 7f5a026e1640 1 -- 172.21.15.98:0/191400975 >> 172.21.15.98:0/191400975 conn(0x7f59fc101790 msgr2=0x7f59fc103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:21.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.680+0000 7f5a026e1640 1 -- 172.21.15.98:0/191400975 shutdown_connections 2024-09-19T07:54:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.680+0000 7f5a026e1640 1 -- 172.21.15.98:0/191400975 wait complete. 2024-09-19T07:54:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.680+0000 7f5a026e1640 1 Processor -- start 2024-09-19T07:54:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.680+0000 7f5a026e1640 1 -- start start 2024-09-19T07:54:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.680+0000 7f5a026e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59fc105c00 0x7f59fc19b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.680+0000 7f5a026e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc10beb0 con 0x7f59fc105c00 2024-09-19T07:54:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.680+0000 7f59fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59fc105c00 0x7f59fc19b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.680+0000 7f59fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59fc105c00 0x7f59fc19b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40920/0 (socket says 172.21.15.98:40920) 2024-09-19T07:54:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.680+0000 7f59fbfff640 1 -- 172.21.15.98:0/3399188735 learned_addr learned my addr 172.21.15.98:0/3399188735 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.681+0000 7f59fbfff640 1 -- 172.21.15.98:0/3399188735 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59fc19bdc0 con 0x7f59fc105c00 2024-09-19T07:54:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.681+0000 7f59fbfff640 1 --2- 172.21.15.98:0/3399188735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59fc105c00 0x7f59fc19b880 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f59ec030560 tx=0x7f59ec032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.681+0000 7f59f9ffb640 1 -- 172.21.15.98:0/3399188735 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59ec037040 con 0x7f59fc105c00 2024-09-19T07:54:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.681+0000 7f5a026e1640 1 -- 172.21.15.98:0/3399188735 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59fc19c050 con 0x7f59fc105c00 2024-09-19T07:54:21.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.681+0000 7f59f9ffb640 1 -- 172.21.15.98:0/3399188735 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59ec040070 con 0x7f59fc105c00 2024-09-19T07:54:21.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.681+0000 7f59f9ffb640 1 -- 172.21.15.98:0/3399188735 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59ec03a740 con 0x7f59fc105c00 2024-09-19T07:54:21.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.681+0000 7f5a026e1640 1 -- 172.21.15.98:0/3399188735 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59fc19ed40 con 0x7f59fc105c00 2024-09-19T07:54:21.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.682+0000 7f59f9ffb640 1 -- 172.21.15.98:0/3399188735 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f59ec03b420 con 0x7f59fc105c00 2024-09-19T07:54:21.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.682+0000 7f5a026e1640 1 -- 172.21.15.98:0/3399188735 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59fc106080 con 0x7f59fc105c00 2024-09-19T07:54:21.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.682+0000 7f59f9ffb640 1 --2- 172.21.15.98:0/3399188735 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f59e8039200 0x7f59e803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:21.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.682+0000 7f59f9ffb640 1 -- 172.21.15.98:0/3399188735 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f59ec06c870 con 0x7f59fc105c00 2024-09-19T07:54:21.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.682+0000 7f59f3fff640 1 --2- 172.21.15.98:0/3399188735 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f59e8039200 0x7f59e803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:21.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.683+0000 7f59f3fff640 1 --2- 172.21.15.98:0/3399188735 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f59e8039200 0x7f59e803b6c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f59e40098c0 tx=0x7f59e4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:21.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.687+0000 7f59f9ffb640 1 -- 172.21.15.98:0/3399188735 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f59ec04a200 con 0x7f59fc105c00 2024-09-19T07:54:21.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:19.911+0000 7f5a026e1640 1 -- 172.21.15.98:0/3399188735 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f59fc19ef10 con 0x7f59fc105c00 2024-09-19T07:54:21.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:20.583+0000 7f59f9ffb640 1 -- 172.21.15.98:0/3399188735 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f59fc19ef10 con 0x7f59fc105c00 2024-09-19T07:54:21.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:20.585+0000 7f59f9ffb640 1 -- 172.21.15.98:0/3399188735 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7f59ec03a8e0 con 0x7f59fc105c00 2024-09-19T07:54:21.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:20.586+0000 7f5a026e1640 1 -- 172.21.15.98:0/3399188735 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f59e8039200 msgr2=0x7f59e803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:21.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:20.586+0000 7f5a026e1640 1 --2- 172.21.15.98:0/3399188735 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f59e8039200 0x7f59e803b6c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f59e40098c0 tx=0x7f59e4006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:21.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:20.586+0000 7f5a026e1640 1 -- 172.21.15.98:0/3399188735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59fc105c00 msgr2=0x7f59fc19b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:21.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:20.586+0000 7f5a026e1640 1 --2- 172.21.15.98:0/3399188735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59fc105c00 0x7f59fc19b880 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f59ec030560 tx=0x7f59ec032670 comp rx=0 tx=0).stop 2024-09-19T07:54:21.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:20.586+0000 7f5a026e1640 1 -- 172.21.15.98:0/3399188735 shutdown_connections 2024-09-19T07:54:21.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:20.586+0000 7f5a026e1640 1 --2- 172.21.15.98:0/3399188735 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f59e8039200 0x7f59e803b6c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:21.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:20.586+0000 7f5a026e1640 1 --2- 172.21.15.98:0/3399188735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59fc105c00 0x7f59fc19b880 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:21.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:20.586+0000 7f5a026e1640 1 -- 172.21.15.98:0/3399188735 >> 172.21.15.98:0/3399188735 conn(0x7f59fc101790 msgr2=0x7f59fc102370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:21.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:20.586+0000 7f5a026e1640 1 -- 172.21.15.98:0/3399188735 shutdown_connections 2024-09-19T07:54:21.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:20.586+0000 7f5a026e1640 1 -- 172.21.15.98:0/3399188735 wait complete. 2024-09-19T07:54:21.544 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:21 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:21.324+0000 7f69aef91040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T07:54:21.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:21 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3399188735' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-19T07:54:21.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:21 smithi098 ceph-mon[31333]: mgrmap e9: x(active, since 18s) 2024-09-19T07:54:21.850 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:21 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:21.648+0000 7f69aef91040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T07:54:21.850 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:21 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:21.711+0000 7f69aef91040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T07:54:22.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:21 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:21.906+0000 7f69aef91040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T07:54:22.699 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:22 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1855378505' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-19T07:54:22.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-09-19T07:54:22.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-19T07:54:22.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-19T07:54:22.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-19T07:54:22.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-19T07:54:22.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-09-19T07:54:22.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.849+0000 7f37e544e640 1 Processor -- start 2024-09-19T07:54:22.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.849+0000 7f37e544e640 1 -- start start 2024-09-19T07:54:22.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.849+0000 7f37e544e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e0105be0 0x7f37e0105fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:22.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.849+0000 7f37e544e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37e01065b0 con 0x7f37e0105be0 2024-09-19T07:54:22.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.849+0000 7f37deffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e0105be0 0x7f37e0105fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:22.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.849+0000 7f37deffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e0105be0 0x7f37e0105fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40954/0 (socket says 172.21.15.98:40954) 2024-09-19T07:54:22.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.849+0000 7f37deffd640 1 -- 172.21.15.98:0/663057963 learned_addr learned my addr 172.21.15.98:0/663057963 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:22.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.850+0000 7f37deffd640 1 -- 172.21.15.98:0/663057963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e0106730 con 0x7f37e0105be0 2024-09-19T07:54:22.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.850+0000 7f37deffd640 1 --2- 172.21.15.98:0/663057963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e0105be0 0x7f37e0105fe0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f37d0009820 tx=0x7f37d002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463458f694a9295d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:22.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.851+0000 7f37de7fc640 1 -- 172.21.15.98:0/663057963 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37d0037070 con 0x7f37e0105be0 2024-09-19T07:54:22.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.851+0000 7f37de7fc640 1 -- 172.21.15.98:0/663057963 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37d0032920 con 0x7f37e0105be0 2024-09-19T07:54:22.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.851+0000 7f37de7fc640 1 -- 172.21.15.98:0/663057963 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37d0030810 con 0x7f37e0105be0 2024-09-19T07:54:22.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.851+0000 7f37e544e640 1 -- 172.21.15.98:0/663057963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e0105be0 msgr2=0x7f37e0105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:22.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.851+0000 7f37e544e640 1 --2- 172.21.15.98:0/663057963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e0105be0 0x7f37e0105fe0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f37d0009820 tx=0x7f37d002a270 comp rx=0 tx=0).stop 2024-09-19T07:54:22.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.851+0000 7f37e544e640 1 -- 172.21.15.98:0/663057963 shutdown_connections 2024-09-19T07:54:22.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.851+0000 7f37e544e640 1 --2- 172.21.15.98:0/663057963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e0105be0 0x7f37e0105fe0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:22.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.851+0000 7f37e544e640 1 -- 172.21.15.98:0/663057963 >> 172.21.15.98:0/663057963 conn(0x7f37e0101790 msgr2=0x7f37e0103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:22.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.851+0000 7f37e544e640 1 -- 172.21.15.98:0/663057963 shutdown_connections 2024-09-19T07:54:22.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.851+0000 7f37e544e640 1 -- 172.21.15.98:0/663057963 wait complete. 2024-09-19T07:54:22.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.852+0000 7f37e544e640 1 Processor -- start 2024-09-19T07:54:22.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.852+0000 7f37e544e640 1 -- start start 2024-09-19T07:54:22.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.852+0000 7f37e544e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e0105be0 0x7f37e019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:22.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.852+0000 7f37e544e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37e010be90 con 0x7f37e0105be0 2024-09-19T07:54:22.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.852+0000 7f37deffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e0105be0 0x7f37e019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:22.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.852+0000 7f37deffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e0105be0 0x7f37e019b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40968/0 (socket says 172.21.15.98:40968) 2024-09-19T07:54:22.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.852+0000 7f37deffd640 1 -- 172.21.15.98:0/1855378505 learned_addr learned my addr 172.21.15.98:0/1855378505 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:22.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.853+0000 7f37deffd640 1 -- 172.21.15.98:0/1855378505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e019be10 con 0x7f37e0105be0 2024-09-19T07:54:22.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.853+0000 7f37deffd640 1 --2- 172.21.15.98:0/1855378505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e0105be0 0x7f37e019b8d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f37d0030560 tx=0x7f37d0032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:22.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.853+0000 7f37dcff9640 1 -- 172.21.15.98:0/1855378505 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37d0037040 con 0x7f37e0105be0 2024-09-19T07:54:22.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.853+0000 7f37dcff9640 1 -- 172.21.15.98:0/1855378505 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37d0040070 con 0x7f37e0105be0 2024-09-19T07:54:22.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.853+0000 7f37e544e640 1 -- 172.21.15.98:0/1855378505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e019c0a0 con 0x7f37e0105be0 2024-09-19T07:54:22.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.853+0000 7f37dcff9640 1 -- 172.21.15.98:0/1855378505 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37d003a740 con 0x7f37e0105be0 2024-09-19T07:54:22.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.853+0000 7f37e544e640 1 -- 172.21.15.98:0/1855378505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e019ed90 con 0x7f37e0105be0 2024-09-19T07:54:22.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.854+0000 7f37dcff9640 1 -- 172.21.15.98:0/1855378505 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7f37d003b420 con 0x7f37e0105be0 2024-09-19T07:54:22.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.854+0000 7f37e544e640 1 -- 172.21.15.98:0/1855378505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37e0106060 con 0x7f37e0105be0 2024-09-19T07:54:22.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.855+0000 7f37dcff9640 1 --2- 172.21.15.98:0/1855378505 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f37cc039260 0x7f37cc03b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:22.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.855+0000 7f37dcff9640 1 -- 172.21.15.98:0/1855378505 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f37d006c870 con 0x7f37e0105be0 2024-09-19T07:54:22.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.855+0000 7f37d7fff640 1 -- 172.21.15.98:0/1855378505 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f37cc039260 msgr2=0x7f37cc03b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/16410793 2024-09-19T07:54:22.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.855+0000 7f37d7fff640 1 --2- 172.21.15.98:0/1855378505 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f37cc039260 0x7f37cc03b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T07:54:22.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:21.860+0000 7f37dcff9640 1 -- 172.21.15.98:0/1855378505 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f37d004a200 con 0x7f37e0105be0 2024-09-19T07:54:22.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:22.056+0000 7f37d7fff640 1 -- 172.21.15.98:0/1855378505 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f37cc039260 msgr2=0x7f37cc03b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/16410793 2024-09-19T07:54:22.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:22.056+0000 7f37d7fff640 1 --2- 172.21.15.98:0/1855378505 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f37cc039260 0x7f37cc03b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-19T07:54:22.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:22.079+0000 7f37e544e640 1 -- 172.21.15.98:0/1855378505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f37e019f380 con 0x7f37e0105be0 2024-09-19T07:54:22.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:22.080+0000 7f37dcff9640 1 -- 172.21.15.98:0/1855378505 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f37d0031e80 con 0x7f37e0105be0 2024-09-19T07:54:22.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:22.083+0000 7f37e544e640 1 -- 172.21.15.98:0/1855378505 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f37cc039260 msgr2=0x7f37cc03b720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:54:22.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:22.083+0000 7f37e544e640 1 --2- 172.21.15.98:0/1855378505 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f37cc039260 0x7f37cc03b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:22.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:22.083+0000 7f37e544e640 1 -- 172.21.15.98:0/1855378505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e0105be0 msgr2=0x7f37e019b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:22.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:22.083+0000 7f37e544e640 1 --2- 172.21.15.98:0/1855378505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e0105be0 0x7f37e019b8d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f37d0030560 tx=0x7f37d0032670 comp rx=0 tx=0).stop 2024-09-19T07:54:22.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:22.083+0000 7f37e544e640 1 -- 172.21.15.98:0/1855378505 shutdown_connections 2024-09-19T07:54:22.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:22.083+0000 7f37e544e640 1 --2- 172.21.15.98:0/1855378505 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f37cc039260 0x7f37cc03b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:22.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:22.083+0000 7f37e544e640 1 --2- 172.21.15.98:0/1855378505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e0105be0 0x7f37e019b8d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:22.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:22.083+0000 7f37e544e640 1 -- 172.21.15.98:0/1855378505 >> 172.21.15.98:0/1855378505 conn(0x7f37e0101790 msgr2=0x7f37e0102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:22.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:22.083+0000 7f37e544e640 1 -- 172.21.15.98:0/1855378505 shutdown_connections 2024-09-19T07:54:22.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:22.083+0000 7f37e544e640 1 -- 172.21.15.98:0/1855378505 wait complete. 2024-09-19T07:54:22.712 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-09-19T07:54:22.712 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 9... 2024-09-19T07:54:23.009 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:22 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:22.811+0000 7f69aef91040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T07:54:23.010 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:22 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:22.879+0000 7f69aef91040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T07:54:23.860 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:23.601+0000 7f69aef91040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-19T07:54:23.860 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:23.717+0000 7f69aef91040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T07:54:24.138 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:23.860+0000 7f69aef91040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T07:54:24.138 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:23.921+0000 7f69aef91040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T07:54:24.138 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:24.069+0000 7f69aef91040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-19T07:54:24.138 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:24.138+0000 7f69aef91040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T07:54:24.418 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:24.197+0000 7f69aef91040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T07:54:24.418 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:24.353+0000 7f69aef91040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T07:54:24.418 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:24.417+0000 7f69aef91040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T07:54:24.682 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:24.484+0000 7f69aef91040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T07:54:24.682 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:24.566+0000 7f69aef91040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T07:54:24.682 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:24.626+0000 7f69aef91040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T07:54:25.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:24.685+0000 7f69aef91040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T07:54:25.932 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:25 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:25.473+0000 7f69aef91040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T07:54:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:25 smithi098 ceph-mon[31333]: Active manager daemon x restarted 2024-09-19T07:54:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:25 smithi098 ceph-mon[31333]: Activating manager daemon x 2024-09-19T07:54:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:25 smithi098 ceph-mon[31333]: osdmap e3: 0 total, 0 up, 0 in 2024-09-19T07:54:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:25 smithi098 ceph-mon[31333]: mgrmap e10: x(active, starting, since 0.00304146s) 2024-09-19T07:54:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:54:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-19T07:54:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-19T07:54:25.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-19T07:54:25.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-19T07:54:25.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:25 smithi098 ceph-mon[31333]: Manager daemon x is now available 2024-09-19T07:54:25.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:54:25.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-19T07:54:25.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-19T07:54:26.801 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:26.728+0000 7f691a2a5640 -1 log_channel(cephadm) log [ERR] : [19/Sep/2024:07:54:26] ENGINE Error in HTTPServer.serve 2024-09-19T07:54:26.801 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: Traceback (most recent call last): 2024-09-19T07:54:26.801 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-19T07:54:26.801 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: self._connections.run(self.expiration_interval) 2024-09-19T07:54:26.801 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-19T07:54:26.801 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: self._run(expiration_interval) 2024-09-19T07:54:26.801 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-19T07:54:26.801 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: new_conn = self._from_server_socket(self.server.socket) 2024-09-19T07:54:26.801 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-19T07:54:26.801 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-19T07:54:26.802 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-19T07:54:26.802 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: s = self.context.wrap_socket( 2024-09-19T07:54:26.802 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-19T07:54:26.802 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: return self.sslsocket_class._create( 2024-09-19T07:54:26.802 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-19T07:54:26.802 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: self.do_handshake() 2024-09-19T07:54:26.802 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-19T07:54:26.802 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: self._sslobj.do_handshake() 2024-09-19T07:54:26.802 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-19T07:54:26.802 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 07:54:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T07:54:27.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-09-19T07:54:27.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-19T07:54:27.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-19T07:54:27.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-09-19T07:54:27.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.089+0000 7f7875c48640 1 Processor -- start 2024-09-19T07:54:27.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.089+0000 7f7875c48640 1 -- start start 2024-09-19T07:54:27.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.089+0000 7f7875c48640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7870108320 0x7f787010a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:27.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.089+0000 7f7875c48640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f787007be90 con 0x7f7870108320 2024-09-19T07:54:27.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.089+0000 7f786f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7870108320 0x7f787010a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:27.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.090+0000 7f786f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7870108320 0x7f787010a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40970/0 (socket says 172.21.15.98:40970) 2024-09-19T07:54:27.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.090+0000 7f786f7fe640 1 -- 172.21.15.98:0/1648692283 learned_addr learned my addr 172.21.15.98:0/1648692283 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:27.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.090+0000 7f786f7fe640 1 -- 172.21.15.98:0/1648692283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f787010ac50 con 0x7f7870108320 2024-09-19T07:54:27.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.090+0000 7f786f7fe640 1 --2- 172.21.15.98:0/1648692283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7870108320 0x7f787010a710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7860009820 tx=0x7f786002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a137bf62210f2ae3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:27.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.091+0000 7f786effd640 1 -- 172.21.15.98:0/1648692283 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7860037070 con 0x7f7870108320 2024-09-19T07:54:27.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.091+0000 7f786effd640 1 -- 172.21.15.98:0/1648692283 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7860032920 con 0x7f7870108320 2024-09-19T07:54:27.070 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.091+0000 7f786effd640 1 -- 172.21.15.98:0/1648692283 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7860030810 con 0x7f7870108320 2024-09-19T07:54:27.070 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.091+0000 7f7875c48640 1 -- 172.21.15.98:0/1648692283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7870108320 msgr2=0x7f787010a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:27.070 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.091+0000 7f7875c48640 1 --2- 172.21.15.98:0/1648692283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7870108320 0x7f787010a710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7860009820 tx=0x7f786002a270 comp rx=0 tx=0).stop 2024-09-19T07:54:27.070 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.092+0000 7f7875c48640 1 -- 172.21.15.98:0/1648692283 shutdown_connections 2024-09-19T07:54:27.070 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.092+0000 7f7875c48640 1 --2- 172.21.15.98:0/1648692283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7870108320 0x7f787010a710 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:27.070 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.092+0000 7f7875c48640 1 -- 172.21.15.98:0/1648692283 >> 172.21.15.98:0/1648692283 conn(0x7f7870101f40 msgr2=0x7f7870104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:27.070 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.092+0000 7f7875c48640 1 -- 172.21.15.98:0/1648692283 shutdown_connections 2024-09-19T07:54:27.070 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.092+0000 7f7875c48640 1 -- 172.21.15.98:0/1648692283 wait complete. 2024-09-19T07:54:27.070 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.092+0000 7f7875c48640 1 Processor -- start 2024-09-19T07:54:27.070 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.092+0000 7f7875c48640 1 -- start start 2024-09-19T07:54:27.070 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.092+0000 7f7875c48640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7870108320 0x7f787019da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:27.070 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.092+0000 7f7875c48640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7870106f50 con 0x7f7870108320 2024-09-19T07:54:27.071 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.092+0000 7f786f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7870108320 0x7f787019da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:27.071 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.092+0000 7f786f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7870108320 0x7f787019da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40982/0 (socket says 172.21.15.98:40982) 2024-09-19T07:54:27.071 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.092+0000 7f786f7fe640 1 -- 172.21.15.98:0/1972792924 learned_addr learned my addr 172.21.15.98:0/1972792924 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:27.071 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.093+0000 7f786f7fe640 1 -- 172.21.15.98:0/1972792924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f787019dfd0 con 0x7f7870108320 2024-09-19T07:54:27.071 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.093+0000 7f786f7fe640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7870108320 0x7f787019da90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7860030380 tx=0x7f7860032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:27.071 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.093+0000 7f786d7fa640 1 -- 172.21.15.98:0/1972792924 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7860037040 con 0x7f7870108320 2024-09-19T07:54:27.071 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.093+0000 7f786d7fa640 1 -- 172.21.15.98:0/1972792924 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7860040070 con 0x7f7870108320 2024-09-19T07:54:27.071 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.093+0000 7f7875c48640 1 -- 172.21.15.98:0/1972792924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f787019e260 con 0x7f7870108320 2024-09-19T07:54:27.071 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.093+0000 7f786d7fa640 1 -- 172.21.15.98:0/1972792924 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f786003a7b0 con 0x7f7870108320 2024-09-19T07:54:27.071 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.093+0000 7f7875c48640 1 -- 172.21.15.98:0/1972792924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78701a0f50 con 0x7f7870108320 2024-09-19T07:54:27.071 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.094+0000 7f786d7fa640 1 -- 172.21.15.98:0/1972792924 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7f786003aa30 con 0x7f7870108320 2024-09-19T07:54:27.071 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.094+0000 7f786d7fa640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f785c039200 0x7f785c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:27.071 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.094+0000 7f786d7fa640 1 -- 172.21.15.98:0/1972792924 --> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f785c03bdd0 con 0x7f785c039200 2024-09-19T07:54:27.072 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.094+0000 7f786d7fa640 1 -- 172.21.15.98:0/1972792924 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f786006c970 con 0x7f7870108320 2024-09-19T07:54:27.072 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.094+0000 7f7867fff640 1 -- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f785c039200 msgr2=0x7f785c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/16410793 2024-09-19T07:54:27.072 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.094+0000 7f7867fff640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f785c039200 0x7f785c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T07:54:27.072 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.295+0000 7f7867fff640 1 -- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f785c039200 msgr2=0x7f785c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/16410793 2024-09-19T07:54:27.072 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.295+0000 7f7867fff640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f785c039200 0x7f785c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-19T07:54:27.072 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.696+0000 7f7867fff640 1 -- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f785c039200 msgr2=0x7f785c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/16410793 2024-09-19T07:54:27.072 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:23.696+0000 7f7867fff640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f785c039200 0x7f785c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-19T07:54:27.072 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:24.497+0000 7f7867fff640 1 -- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f785c039200 msgr2=0x7f785c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/16410793 2024-09-19T07:54:27.072 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:24.497+0000 7f7867fff640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f785c039200 0x7f785c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-19T07:54:27.072 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:25.476+0000 7f786d7fa640 1 -- 172.21.15.98:0/1972792924 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 10) ==== 46241+0+0 (secure 0 0 0) 0x7f786006b420 con 0x7f7870108320 2024-09-19T07:54:27.072 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:25.477+0000 7f786d7fa640 1 -- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f785c039200 msgr2=0x7f785c03b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:54:27.072 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:25.477+0000 7f786d7fa640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f785c039200 0x7f785c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:27.073 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.478+0000 7f786d7fa640 1 -- 172.21.15.98:0/1972792924 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 11) ==== 46368+0+0 (secure 0 0 0) 0x7f7860045460 con 0x7f7870108320 2024-09-19T07:54:27.073 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.478+0000 7f786d7fa640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f785c03ccc0 0x7f785c03f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:27.073 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.478+0000 7f786d7fa640 1 -- 172.21.15.98:0/1972792924 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f786002b9f0 con 0x7f785c03ccc0 2024-09-19T07:54:27.073 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.479+0000 7f7867fff640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f785c03ccc0 0x7f785c03f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:27.073 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.480+0000 7f7867fff640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f785c03ccc0 0x7f785c03f0b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7858003980 tx=0x7f7858006e10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:27.073 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.480+0000 7f786d7fa640 1 -- 172.21.15.98:0/1972792924 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f786002b9f0 con 0x7f785c03ccc0 2024-09-19T07:54:27.073 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.487+0000 7f7875c48640 1 -- 172.21.15.98:0/1972792924 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f78701075d0 con 0x7f785c03ccc0 2024-09-19T07:54:27.073 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.487+0000 7f786d7fa640 1 -- 172.21.15.98:0/1972792924 <== mgr.14150 v2:172.21.15.98:6800/470997812 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f78701075d0 con 0x7f785c03ccc0 2024-09-19T07:54:27.073 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.487+0000 7f7875c48640 1 -- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f785c03ccc0 msgr2=0x7f785c03f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:27.073 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.487+0000 7f7875c48640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f785c03ccc0 0x7f785c03f0b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7858003980 tx=0x7f7858006e10 comp rx=0 tx=0).stop 2024-09-19T07:54:27.073 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.488+0000 7f7875c48640 1 -- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7870108320 msgr2=0x7f787019da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:27.073 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.488+0000 7f7875c48640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7870108320 0x7f787019da90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7860030380 tx=0x7f7860032670 comp rx=0 tx=0).stop 2024-09-19T07:54:27.073 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.488+0000 7f7875c48640 1 -- 172.21.15.98:0/1972792924 shutdown_connections 2024-09-19T07:54:27.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.488+0000 7f7875c48640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f785c03ccc0 0x7f785c03f0b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:27.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.488+0000 7f7875c48640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:6800/16410793,v1:172.21.15.98:6801/16410793] conn(0x7f785c039200 0x7f785c03b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:27.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.488+0000 7f7875c48640 1 --2- 172.21.15.98:0/1972792924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7870108320 0x7f787019da90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:27.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.488+0000 7f7875c48640 1 -- 172.21.15.98:0/1972792924 >> 172.21.15.98:0/1972792924 conn(0x7f7870101f40 msgr2=0x7f7870102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:27.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.488+0000 7f7875c48640 1 -- 172.21.15.98:0/1972792924 shutdown_connections 2024-09-19T07:54:27.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:26.488+0000 7f7875c48640 1 -- 172.21.15.98:0/1972792924 wait complete. 2024-09-19T07:54:27.074 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 9 is available 2024-09-19T07:54:27.074 INFO:teuthology.orchestra.run.smithi098.stdout:Generating a dashboard self-signed certificate... 2024-09-19T07:54:27.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:27 smithi098 ceph-mon[31333]: mgrmap e11: x(active, since 1.00568s) 2024-09-19T07:54:27.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:27 smithi098 ceph-mon[31333]: [19/Sep/2024:07:54:26] ENGINE Bus STARTING 2024-09-19T07:54:27.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:27 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:28.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-19T07:54:28.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.497+0000 7fb9d0f06640 1 Processor -- start 2024-09-19T07:54:28.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.497+0000 7fb9d0f06640 1 -- start start 2024-09-19T07:54:28.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.498+0000 7fb9d0f06640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9cc07de60 0x7fb9cc07c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:28.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.498+0000 7fb9d0f06640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc07c890 con 0x7fb9cc07de60 2024-09-19T07:54:28.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.498+0000 7fb9ca59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9cc07de60 0x7fb9cc07c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:28.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.498+0000 7fb9ca59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9cc07de60 0x7fb9cc07c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49114/0 (socket says 172.21.15.98:49114) 2024-09-19T07:54:28.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.498+0000 7fb9ca59c640 1 -- 172.21.15.98:0/1581760195 learned_addr learned my addr 172.21.15.98:0/1581760195 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:28.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.498+0000 7fb9ca59c640 1 -- 172.21.15.98:0/1581760195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9cc07ca10 con 0x7fb9cc07de60 2024-09-19T07:54:28.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.499+0000 7fb9ca59c640 1 --2- 172.21.15.98:0/1581760195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9cc07de60 0x7fb9cc07c2c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc009820 tx=0x7fb9bc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1491df6897a7f62 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:28.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.499+0000 7fb9c959a640 1 -- 172.21.15.98:0/1581760195 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9bc037070 con 0x7fb9cc07de60 2024-09-19T07:54:28.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.499+0000 7fb9c959a640 1 -- 172.21.15.98:0/1581760195 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9bc032920 con 0x7fb9cc07de60 2024-09-19T07:54:28.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.499+0000 7fb9c959a640 1 -- 172.21.15.98:0/1581760195 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9bc030810 con 0x7fb9cc07de60 2024-09-19T07:54:28.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.499+0000 7fb9d0f06640 1 -- 172.21.15.98:0/1581760195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9cc07de60 msgr2=0x7fb9cc07c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:28.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.499+0000 7fb9d0f06640 1 --2- 172.21.15.98:0/1581760195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9cc07de60 0x7fb9cc07c2c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc009820 tx=0x7fb9bc02a270 comp rx=0 tx=0).stop 2024-09-19T07:54:28.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.500+0000 7fb9d0f06640 1 -- 172.21.15.98:0/1581760195 shutdown_connections 2024-09-19T07:54:28.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.500+0000 7fb9d0f06640 1 --2- 172.21.15.98:0/1581760195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9cc07de60 0x7fb9cc07c2c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:28.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.500+0000 7fb9d0f06640 1 -- 172.21.15.98:0/1581760195 >> 172.21.15.98:0/1581760195 conn(0x7fb9cc1030b0 msgr2=0x7fb9cc1054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:28.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.500+0000 7fb9d0f06640 1 -- 172.21.15.98:0/1581760195 shutdown_connections 2024-09-19T07:54:28.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.500+0000 7fb9d0f06640 1 -- 172.21.15.98:0/1581760195 wait complete. 2024-09-19T07:54:28.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.500+0000 7fb9d0f06640 1 Processor -- start 2024-09-19T07:54:28.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.500+0000 7fb9d0f06640 1 -- start start 2024-09-19T07:54:28.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.501+0000 7fb9d0f06640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9cc07de60 0x7fb9cc19dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:28.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.501+0000 7fb9d0f06640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc10df20 con 0x7fb9cc07de60 2024-09-19T07:54:28.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.501+0000 7fb9ca59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9cc07de60 0x7fb9cc19dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:28.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.501+0000 7fb9ca59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9cc07de60 0x7fb9cc19dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49116/0 (socket says 172.21.15.98:49116) 2024-09-19T07:54:28.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.501+0000 7fb9ca59c640 1 -- 172.21.15.98:0/4247305375 learned_addr learned my addr 172.21.15.98:0/4247305375 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:28.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.501+0000 7fb9ca59c640 1 -- 172.21.15.98:0/4247305375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9cc19dff0 con 0x7fb9cc07de60 2024-09-19T07:54:28.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.501+0000 7fb9ca59c640 1 --2- 172.21.15.98:0/4247305375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9cc07de60 0x7fb9cc19dab0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0097f0 tx=0x7fb9bc0327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:28.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.501+0000 7fb9bb7fe640 1 -- 172.21.15.98:0/4247305375 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9bc037040 con 0x7fb9cc07de60 2024-09-19T07:54:28.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.501+0000 7fb9bb7fe640 1 -- 172.21.15.98:0/4247305375 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9bc030c40 con 0x7fb9cc07de60 2024-09-19T07:54:28.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.501+0000 7fb9bb7fe640 1 -- 172.21.15.98:0/4247305375 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9bc03a5d0 con 0x7fb9cc07de60 2024-09-19T07:54:28.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.501+0000 7fb9d0f06640 1 -- 172.21.15.98:0/4247305375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9cc19e280 con 0x7fb9cc07de60 2024-09-19T07:54:28.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.501+0000 7fb9d0f06640 1 -- 172.21.15.98:0/4247305375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9cc1a0f70 con 0x7fb9cc07de60 2024-09-19T07:54:28.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.502+0000 7fb9bb7fe640 1 -- 172.21.15.98:0/4247305375 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) ==== 46368+0+0 (secure 0 0 0) 0x7fb9bc03a7c0 con 0x7fb9cc07de60 2024-09-19T07:54:28.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.502+0000 7fb9bb7fe640 1 --2- 172.21.15.98:0/4247305375 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb9b4039140 0x7fb9b403b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:28.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.502+0000 7fb9bb7fe640 1 -- 172.21.15.98:0/4247305375 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb9bc06c6a0 con 0x7fb9cc07de60 2024-09-19T07:54:28.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.502+0000 7fb9c9d9b640 1 --2- 172.21.15.98:0/4247305375 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb9b4039140 0x7fb9b403b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:28.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.502+0000 7fb9d0f06640 1 -- 172.21.15.98:0/4247305375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9cc07c340 con 0x7fb9cc07de60 2024-09-19T07:54:28.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.503+0000 7fb9c9d9b640 1 --2- 172.21.15.98:0/4247305375 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb9b4039140 0x7fb9b403b600 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00098c0 tx=0x7fb9c0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:28.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.507+0000 7fb9bb7fe640 1 -- 172.21.15.98:0/4247305375 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb9bc0371e0 con 0x7fb9cc07de60 2024-09-19T07:54:28.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.700+0000 7fb9d0f06640 1 -- 172.21.15.98:0/4247305375 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fb9cc104350 con 0x7fb9b4039140 2024-09-19T07:54:28.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.754+0000 7fb9bb7fe640 1 -- 172.21.15.98:0/4247305375 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fb9cc104350 con 0x7fb9b4039140 2024-09-19T07:54:28.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.756+0000 7fb9d0f06640 1 -- 172.21.15.98:0/4247305375 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb9b4039140 msgr2=0x7fb9b403b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:28.369 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.756+0000 7fb9d0f06640 1 --2- 172.21.15.98:0/4247305375 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb9b4039140 0x7fb9b403b600 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00098c0 tx=0x7fb9c0006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:28.369 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.757+0000 7fb9d0f06640 1 -- 172.21.15.98:0/4247305375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9cc07de60 msgr2=0x7fb9cc19dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:28.369 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.757+0000 7fb9d0f06640 1 --2- 172.21.15.98:0/4247305375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9cc07de60 0x7fb9cc19dab0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0097f0 tx=0x7fb9bc0327b0 comp rx=0 tx=0).stop 2024-09-19T07:54:28.369 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.757+0000 7fb9d0f06640 1 -- 172.21.15.98:0/4247305375 shutdown_connections 2024-09-19T07:54:28.369 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.757+0000 7fb9d0f06640 1 --2- 172.21.15.98:0/4247305375 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb9b4039140 0x7fb9b403b600 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:28.369 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.757+0000 7fb9d0f06640 1 --2- 172.21.15.98:0/4247305375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9cc07de60 0x7fb9cc19dab0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:28.369 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.757+0000 7fb9d0f06640 1 -- 172.21.15.98:0/4247305375 >> 172.21.15.98:0/4247305375 conn(0x7fb9cc1030b0 msgr2=0x7fb9cc103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:28.369 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.757+0000 7fb9d0f06640 1 -- 172.21.15.98:0/4247305375 shutdown_connections 2024-09-19T07:54:28.369 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:27.757+0000 7fb9d0f06640 1 -- 172.21.15.98:0/4247305375 wait complete. 2024-09-19T07:54:28.369 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial admin user... 2024-09-19T07:54:28.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:28 smithi098 ceph-mon[31333]: [19/Sep/2024:07:54:26] ENGINE Serving on https://172.21.15.98:7150 2024-09-19T07:54:28.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:28 smithi098 ceph-mon[31333]: [19/Sep/2024:07:54:26] ENGINE Error in HTTPServer.serve 2024-09-19T07:54:28.612 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-09-19T07:54:28.613 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-19T07:54:28.613 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._connections.run(self.expiration_interval) 2024-09-19T07:54:28.613 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-19T07:54:28.613 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._run(expiration_interval) 2024-09-19T07:54:28.613 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-19T07:54:28.613 INFO:journalctl@ceph.mon.a.smithi098.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-19T07:54:28.613 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-19T07:54:28.613 INFO:journalctl@ceph.mon.a.smithi098.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-19T07:54:28.613 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-19T07:54:28.613 INFO:journalctl@ceph.mon.a.smithi098.stdout: s = self.context.wrap_socket( 2024-09-19T07:54:28.614 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-19T07:54:28.614 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.sslsocket_class._create( 2024-09-19T07:54:28.614 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-19T07:54:28.614 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.do_handshake() 2024-09-19T07:54:28.614 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-19T07:54:28.614 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._sslobj.do_handshake() 2024-09-19T07:54:28.614 INFO:journalctl@ceph.mon.a.smithi098.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-19T07:54:28.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:28 smithi098 ceph-mon[31333]: [19/Sep/2024:07:54:26] ENGINE Serving on http://172.21.15.98:8765 2024-09-19T07:54:28.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:28 smithi098 ceph-mon[31333]: [19/Sep/2024:07:54:26] ENGINE Bus STARTED 2024-09-19T07:54:28.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:28.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:28.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:28.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:28 smithi098 ceph-mon[31333]: mgrmap e12: x(active, since 2s) 2024-09-19T07:54:29.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:29 smithi098 ceph-mon[31333]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:54:29.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:29 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$wu4lHlNVUL0YU1alhWPDPefhX..kFl/6/GIO18HJEQSmSE5zGx3Bi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726732469, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-19T07:54:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.819+0000 7f0403abc640 1 Processor -- start 2024-09-19T07:54:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.819+0000 7f0403abc640 1 -- start start 2024-09-19T07:54:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.819+0000 7f0403abc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc106190 0x7f03fc106590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.819+0000 7f0403abc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03fc07be10 con 0x7f03fc106190 2024-09-19T07:54:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.819+0000 7f0401858640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc106190 0x7f03fc106590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.819+0000 7f0401858640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc106190 0x7f03fc106590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49126/0 (socket says 172.21.15.98:49126) 2024-09-19T07:54:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.819+0000 7f0401858640 1 -- 172.21.15.98:0/1514152855 learned_addr learned my addr 172.21.15.98:0/1514152855 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.819+0000 7f0401858640 1 -- 172.21.15.98:0/1514152855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03fc106ad0 con 0x7f03fc106190 2024-09-19T07:54:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.820+0000 7f0401858640 1 --2- 172.21.15.98:0/1514152855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc106190 0x7f03fc106590 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f03f8009820 tx=0x7f03f802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e068b09ff0a864ed server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.820+0000 7f0400856640 1 -- 172.21.15.98:0/1514152855 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03f8037070 con 0x7f03fc106190 2024-09-19T07:54:30.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.820+0000 7f0400856640 1 -- 172.21.15.98:0/1514152855 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03f8032920 con 0x7f03fc106190 2024-09-19T07:54:30.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.820+0000 7f0400856640 1 -- 172.21.15.98:0/1514152855 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03f8030810 con 0x7f03fc106190 2024-09-19T07:54:30.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.821+0000 7f0403abc640 1 -- 172.21.15.98:0/1514152855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc106190 msgr2=0x7f03fc106590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:30.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.821+0000 7f0403abc640 1 --2- 172.21.15.98:0/1514152855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc106190 0x7f03fc106590 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f03f8009820 tx=0x7f03f802a270 comp rx=0 tx=0).stop 2024-09-19T07:54:30.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.821+0000 7f0403abc640 1 -- 172.21.15.98:0/1514152855 shutdown_connections 2024-09-19T07:54:30.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.821+0000 7f0403abc640 1 --2- 172.21.15.98:0/1514152855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc106190 0x7f03fc106590 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:30.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.821+0000 7f0403abc640 1 -- 172.21.15.98:0/1514152855 >> 172.21.15.98:0/1514152855 conn(0x7f03fc101f40 msgr2=0x7f03fc104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:30.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.821+0000 7f0403abc640 1 -- 172.21.15.98:0/1514152855 shutdown_connections 2024-09-19T07:54:30.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.821+0000 7f0403abc640 1 -- 172.21.15.98:0/1514152855 wait complete. 2024-09-19T07:54:30.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.821+0000 7f0403abc640 1 Processor -- start 2024-09-19T07:54:30.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.822+0000 7f0403abc640 1 -- start start 2024-09-19T07:54:30.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.822+0000 7f0403abc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc106190 0x7f03fc19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:30.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.822+0000 7f0403abc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03fc107ba0 con 0x7f03fc106190 2024-09-19T07:54:30.472 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.822+0000 7f0401858640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc106190 0x7f03fc19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:30.472 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.822+0000 7f0401858640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc106190 0x7f03fc19da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49138/0 (socket says 172.21.15.98:49138) 2024-09-19T07:54:30.472 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.822+0000 7f0401858640 1 -- 172.21.15.98:0/1118247562 learned_addr learned my addr 172.21.15.98:0/1118247562 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:30.472 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.822+0000 7f0401858640 1 -- 172.21.15.98:0/1118247562 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03fc19dfa0 con 0x7f03fc106190 2024-09-19T07:54:30.472 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.822+0000 7f0401858640 1 --2- 172.21.15.98:0/1118247562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc106190 0x7f03fc19da60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f03f8030380 tx=0x7f03f8032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:30.472 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.822+0000 7f03f2ffd640 1 -- 172.21.15.98:0/1118247562 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03f8037040 con 0x7f03fc106190 2024-09-19T07:54:30.472 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.822+0000 7f03f2ffd640 1 -- 172.21.15.98:0/1118247562 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03f8040070 con 0x7f03fc106190 2024-09-19T07:54:30.472 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.822+0000 7f0403abc640 1 -- 172.21.15.98:0/1118247562 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03fc19e230 con 0x7f03fc106190 2024-09-19T07:54:30.472 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.822+0000 7f03f2ffd640 1 -- 172.21.15.98:0/1118247562 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03f803a7b0 con 0x7f03fc106190 2024-09-19T07:54:30.472 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.822+0000 7f0403abc640 1 -- 172.21.15.98:0/1118247562 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03fc1a0f20 con 0x7f03fc106190 2024-09-19T07:54:30.472 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.823+0000 7f03f2ffd640 1 -- 172.21.15.98:0/1118247562 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7f03f803bc00 con 0x7f03fc106190 2024-09-19T07:54:30.472 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.823+0000 7f0403abc640 1 -- 172.21.15.98:0/1118247562 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03fc106590 con 0x7f03fc106190 2024-09-19T07:54:30.472 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.824+0000 7f03f2ffd640 1 --2- 172.21.15.98:0/1118247562 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f03e8039260 0x7f03e803b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:30.473 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.824+0000 7f03f2ffd640 1 -- 172.21.15.98:0/1118247562 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f03f806ca40 con 0x7f03fc106190 2024-09-19T07:54:30.473 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.824+0000 7f0401057640 1 --2- 172.21.15.98:0/1118247562 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f03e8039260 0x7f03e803b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:30.473 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.824+0000 7f0401057640 1 --2- 172.21.15.98:0/1118247562 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f03e8039260 0x7f03e803b720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f03ec0098c0 tx=0x7f03ec006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:30.473 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:28.829+0000 7f03f2ffd640 1 -- 172.21.15.98:0/1118247562 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f03f806ee20 con 0x7f03fc106190 2024-09-19T07:54:30.473 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:29.031+0000 7f0403abc640 1 -- 172.21.15.98:0/1118247562 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f03fc19e710 con 0x7f03e8039260 2024-09-19T07:54:30.473 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:29.257+0000 7f03f2ffd640 1 -- 172.21.15.98:0/1118247562 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f03fc19e710 con 0x7f03e8039260 2024-09-19T07:54:30.473 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:29.260+0000 7f0403abc640 1 -- 172.21.15.98:0/1118247562 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f03e8039260 msgr2=0x7f03e803b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:30.473 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:29.260+0000 7f0403abc640 1 --2- 172.21.15.98:0/1118247562 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f03e8039260 0x7f03e803b720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f03ec0098c0 tx=0x7f03ec006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:30.473 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:29.260+0000 7f0403abc640 1 -- 172.21.15.98:0/1118247562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc106190 msgr2=0x7f03fc19da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:30.473 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:29.260+0000 7f0403abc640 1 --2- 172.21.15.98:0/1118247562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc106190 0x7f03fc19da60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f03f8030380 tx=0x7f03f8032670 comp rx=0 tx=0).stop 2024-09-19T07:54:30.473 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:29.260+0000 7f0403abc640 1 -- 172.21.15.98:0/1118247562 shutdown_connections 2024-09-19T07:54:30.473 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:29.260+0000 7f0403abc640 1 --2- 172.21.15.98:0/1118247562 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f03e8039260 0x7f03e803b720 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:30.473 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:29.260+0000 7f0403abc640 1 --2- 172.21.15.98:0/1118247562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc106190 0x7f03fc19da60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:30.474 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:29.260+0000 7f0403abc640 1 -- 172.21.15.98:0/1118247562 >> 172.21.15.98:0/1118247562 conn(0x7f03fc101f40 msgr2=0x7f03fc191d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:30.474 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:29.260+0000 7f0403abc640 1 -- 172.21.15.98:0/1118247562 shutdown_connections 2024-09-19T07:54:30.474 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:29.260+0000 7f0403abc640 1 -- 172.21.15.98:0/1118247562 wait complete. 2024-09-19T07:54:30.474 INFO:teuthology.orchestra.run.smithi098.stdout:Fetching dashboard port number... 2024-09-19T07:54:30.713 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:30 smithi098 ceph-mon[31333]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:54:31.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:31 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3714008214' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-19T07:54:31.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 8443 2024-09-19T07:54:31.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.902+0000 7f0698c10640 1 Processor -- start 2024-09-19T07:54:31.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.902+0000 7f0698c10640 1 -- start start 2024-09-19T07:54:31.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.903+0000 7f0698c10640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0694107900 0x7f0694107d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:31.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.903+0000 7f0698c10640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f06941082d0 con 0x7f0694107900 2024-09-19T07:54:31.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.903+0000 7f069259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0694107900 0x7f0694107d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:31.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.903+0000 7f069259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0694107900 0x7f0694107d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49144/0 (socket says 172.21.15.98:49144) 2024-09-19T07:54:31.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.903+0000 7f069259c640 1 -- 172.21.15.98:0/3951203246 learned_addr learned my addr 172.21.15.98:0/3951203246 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:31.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.903+0000 7f069259c640 1 -- 172.21.15.98:0/3951203246 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0694108b00 con 0x7f0694107900 2024-09-19T07:54:31.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.904+0000 7f069259c640 1 --2- 172.21.15.98:0/3951203246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0694107900 0x7f0694107d00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0684009820 tx=0x7f068402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ea4609918f2ea5e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:31.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.904+0000 7f069159a640 1 -- 172.21.15.98:0/3951203246 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0684037070 con 0x7f0694107900 2024-09-19T07:54:31.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.904+0000 7f069159a640 1 -- 172.21.15.98:0/3951203246 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0684032920 con 0x7f0694107900 2024-09-19T07:54:31.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.904+0000 7f069159a640 1 -- 172.21.15.98:0/3951203246 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0684030810 con 0x7f0694107900 2024-09-19T07:54:31.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.904+0000 7f0698c10640 1 -- 172.21.15.98:0/3951203246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0694107900 msgr2=0x7f0694107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:31.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.904+0000 7f0698c10640 1 --2- 172.21.15.98:0/3951203246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0694107900 0x7f0694107d00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0684009820 tx=0x7f068402a270 comp rx=0 tx=0).stop 2024-09-19T07:54:31.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.905+0000 7f0698c10640 1 -- 172.21.15.98:0/3951203246 shutdown_connections 2024-09-19T07:54:31.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.905+0000 7f0698c10640 1 --2- 172.21.15.98:0/3951203246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0694107900 0x7f0694107d00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:31.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.905+0000 7f0698c10640 1 -- 172.21.15.98:0/3951203246 >> 172.21.15.98:0/3951203246 conn(0x7f06941030b0 msgr2=0x7f06941054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:31.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.905+0000 7f0698c10640 1 -- 172.21.15.98:0/3951203246 shutdown_connections 2024-09-19T07:54:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.905+0000 7f0698c10640 1 -- 172.21.15.98:0/3951203246 wait complete. 2024-09-19T07:54:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.905+0000 7f0698c10640 1 Processor -- start 2024-09-19T07:54:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.905+0000 7f0698c10640 1 -- start start 2024-09-19T07:54:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.905+0000 7f0698c10640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0694107900 0x7f069419b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.906+0000 7f0698c10640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f069410beb0 con 0x7f0694107900 2024-09-19T07:54:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.906+0000 7f069259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0694107900 0x7f069419b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.906+0000 7f069259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0694107900 0x7f069419b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49148/0 (socket says 172.21.15.98:49148) 2024-09-19T07:54:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.906+0000 7f069259c640 1 -- 172.21.15.98:0/3714008214 learned_addr learned my addr 172.21.15.98:0/3714008214 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.906+0000 7f069259c640 1 -- 172.21.15.98:0/3714008214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f069419be20 con 0x7f0694107900 2024-09-19T07:54:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.906+0000 7f069259c640 1 --2- 172.21.15.98:0/3714008214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0694107900 0x7f069419b8e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0684030560 tx=0x7f0684032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.906+0000 7f06837fe640 1 -- 172.21.15.98:0/3714008214 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0684037040 con 0x7f0694107900 2024-09-19T07:54:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.906+0000 7f0698c10640 1 -- 172.21.15.98:0/3714008214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f069419c0b0 con 0x7f0694107900 2024-09-19T07:54:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.906+0000 7f06837fe640 1 -- 172.21.15.98:0/3714008214 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0684040070 con 0x7f0694107900 2024-09-19T07:54:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.906+0000 7f06837fe640 1 -- 172.21.15.98:0/3714008214 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f068403a6d0 con 0x7f0694107900 2024-09-19T07:54:31.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.906+0000 7f0698c10640 1 -- 172.21.15.98:0/3714008214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f069419eda0 con 0x7f0694107900 2024-09-19T07:54:31.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.907+0000 7f06837fe640 1 -- 172.21.15.98:0/3714008214 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7f068403a940 con 0x7f0694107900 2024-09-19T07:54:31.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.907+0000 7f06837fe640 1 --2- 172.21.15.98:0/3714008214 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f067c039200 0x7f067c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:31.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.907+0000 7f0698c10640 1 -- 172.21.15.98:0/3714008214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0694107d80 con 0x7f0694107900 2024-09-19T07:54:31.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.907+0000 7f06837fe640 1 -- 172.21.15.98:0/3714008214 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f068406c930 con 0x7f0694107900 2024-09-19T07:54:31.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.907+0000 7f0691d9b640 1 --2- 172.21.15.98:0/3714008214 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f067c039200 0x7f067c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:31.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.908+0000 7f0691d9b640 1 --2- 172.21.15.98:0/3714008214 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f067c039200 0x7f067c03b6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f06880098c0 tx=0x7f0688006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:31.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:30.912+0000 7f06837fe640 1 -- 172.21.15.98:0/3714008214 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f0684044c70 con 0x7f0694107900 2024-09-19T07:54:31.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:31.077+0000 7f0698c10640 1 -- 172.21.15.98:0/3714008214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f069419f390 con 0x7f0694107900 2024-09-19T07:54:31.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:31.077+0000 7f06837fe640 1 -- 172.21.15.98:0/3714008214 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f0684008080 con 0x7f0694107900 2024-09-19T07:54:31.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:31.080+0000 7f0698c10640 1 -- 172.21.15.98:0/3714008214 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f067c039200 msgr2=0x7f067c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:31.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:31.080+0000 7f0698c10640 1 --2- 172.21.15.98:0/3714008214 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f067c039200 0x7f067c03b6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f06880098c0 tx=0x7f0688006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:31.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:31.080+0000 7f0698c10640 1 -- 172.21.15.98:0/3714008214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0694107900 msgr2=0x7f069419b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:31.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:31.080+0000 7f0698c10640 1 --2- 172.21.15.98:0/3714008214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0694107900 0x7f069419b8e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0684030560 tx=0x7f0684032670 comp rx=0 tx=0).stop 2024-09-19T07:54:31.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:31.080+0000 7f0698c10640 1 -- 172.21.15.98:0/3714008214 shutdown_connections 2024-09-19T07:54:31.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:31.080+0000 7f0698c10640 1 --2- 172.21.15.98:0/3714008214 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f067c039200 0x7f067c03b6c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:31.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:31.080+0000 7f0698c10640 1 --2- 172.21.15.98:0/3714008214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0694107900 0x7f069419b8e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:31.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:31.080+0000 7f0698c10640 1 -- 172.21.15.98:0/3714008214 >> 172.21.15.98:0/3714008214 conn(0x7f06941030b0 msgr2=0x7f0694103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:31.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:31.080+0000 7f0698c10640 1 -- 172.21.15.98:0/3714008214 shutdown_connections 2024-09-19T07:54:31.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:31.080+0000 7f0698c10640 1 -- 172.21.15.98:0/3714008214 wait complete. 2024-09-19T07:54:31.675 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-19T07:54:31.675 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-09-19T07:54:31.683 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-19T07:54:31.683 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-09-19T07:54:31.683 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-09-19T07:54:31.683 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-19T07:54:31.685 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph Dashboard is now available at: 2024-09-19T07:54:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:54:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: URL: https://smithi098.front.sepia.ceph.com:8443/ 2024-09-19T07:54:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: User: admin 2024-09-19T07:54:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: Password: pjdjdv72sv 2024-09-19T07:54:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:54:31.686 INFO:teuthology.orchestra.run.smithi098.stdout:Saving cluster configuration to /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config directory 2024-09-19T07:54:32.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:32 smithi098 ceph-mon[31333]: mgrmap e13: x(active, since 6s) 2024-09-19T07:54:32.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:32 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1300749327' entity='client.admin' 2024-09-19T07:54:32.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.051+0000 7f0a0606d640 1 Processor -- start 2024-09-19T07:54:32.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.051+0000 7f0a0606d640 1 -- start start 2024-09-19T07:54:32.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.051+0000 7f0a0606d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a00106780 0x7f0a00106b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:32.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.051+0000 7f0a0606d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a00107150 con 0x7f0a00106780 2024-09-19T07:54:32.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.052+0000 7f09ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a00106780 0x7f0a00106b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:32.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.052+0000 7f09ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a00106780 0x7f0a00106b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49150/0 (socket says 172.21.15.98:49150) 2024-09-19T07:54:32.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.052+0000 7f09ff7fe640 1 -- 172.21.15.98:0/1650272962 learned_addr learned my addr 172.21.15.98:0/1650272962 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:32.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.052+0000 7f09ff7fe640 1 -- 172.21.15.98:0/1650272962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a00107980 con 0x7f0a00106780 2024-09-19T07:54:32.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.053+0000 7f09ff7fe640 1 --2- 172.21.15.98:0/1650272962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a00106780 0x7f0a00106b80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009e80 tx=0x7f09f002a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804135f8f17a126e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:32.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.053+0000 7f09fe7fc640 1 -- 172.21.15.98:0/1650272962 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09f0036070 con 0x7f0a00106780 2024-09-19T07:54:32.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.053+0000 7f09fe7fc640 1 -- 172.21.15.98:0/1650272962 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09f00318e0 con 0x7f0a00106780 2024-09-19T07:54:32.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.053+0000 7f09fe7fc640 1 -- 172.21.15.98:0/1650272962 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09f0007a00 con 0x7f0a00106780 2024-09-19T07:54:32.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.053+0000 7f0a0606d640 1 -- 172.21.15.98:0/1650272962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a00106780 msgr2=0x7f0a00106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:32.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.053+0000 7f0a0606d640 1 --2- 172.21.15.98:0/1650272962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a00106780 0x7f0a00106b80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009e80 tx=0x7f09f002a7a0 comp rx=0 tx=0).stop 2024-09-19T07:54:32.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.053+0000 7f0a0606d640 1 -- 172.21.15.98:0/1650272962 shutdown_connections 2024-09-19T07:54:32.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.053+0000 7f0a0606d640 1 --2- 172.21.15.98:0/1650272962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a00106780 0x7f0a00106b80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:32.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.053+0000 7f0a0606d640 1 -- 172.21.15.98:0/1650272962 >> 172.21.15.98:0/1650272962 conn(0x7f0a00101f10 msgr2=0x7f0a00104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:32.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.054+0000 7f0a0606d640 1 -- 172.21.15.98:0/1650272962 shutdown_connections 2024-09-19T07:54:32.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.054+0000 7f0a0606d640 1 -- 172.21.15.98:0/1650272962 wait complete. 2024-09-19T07:54:32.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.054+0000 7f0a0606d640 1 Processor -- start 2024-09-19T07:54:32.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.054+0000 7f0a0606d640 1 -- start start 2024-09-19T07:54:32.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.054+0000 7f0a0606d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a00106780 0x7f0a0019b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:32.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.054+0000 7f0a0606d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0010ad30 con 0x7f0a00106780 2024-09-19T07:54:32.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.055+0000 7f09ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a00106780 0x7f0a0019b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:32.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.055+0000 7f09ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a00106780 0x7f0a0019b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49152/0 (socket says 172.21.15.98:49152) 2024-09-19T07:54:32.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.055+0000 7f09ff7fe640 1 -- 172.21.15.98:0/1300749327 learned_addr learned my addr 172.21.15.98:0/1300749327 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:32.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.055+0000 7f09ff7fe640 1 -- 172.21.15.98:0/1300749327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0019be20 con 0x7f0a00106780 2024-09-19T07:54:32.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.055+0000 7f09ff7fe640 1 --2- 172.21.15.98:0/1300749327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a00106780 0x7f0a0019b8e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f09f0031cc0 tx=0x7f09f0031cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:32.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.055+0000 7f09fcff9640 1 -- 172.21.15.98:0/1300749327 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09f0036050 con 0x7f0a00106780 2024-09-19T07:54:32.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.055+0000 7f0a0606d640 1 -- 172.21.15.98:0/1300749327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a0019c0b0 con 0x7f0a00106780 2024-09-19T07:54:32.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.055+0000 7f09fcff9640 1 -- 172.21.15.98:0/1300749327 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09f0042070 con 0x7f0a00106780 2024-09-19T07:54:32.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.055+0000 7f09fcff9640 1 -- 172.21.15.98:0/1300749327 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09f003d8e0 con 0x7f0a00106780 2024-09-19T07:54:32.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.055+0000 7f0a0606d640 1 -- 172.21.15.98:0/1300749327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a0019eda0 con 0x7f0a00106780 2024-09-19T07:54:32.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.056+0000 7f09fcff9640 1 -- 172.21.15.98:0/1300749327 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f09f003db90 con 0x7f0a00106780 2024-09-19T07:54:32.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.056+0000 7f09fcff9640 1 --2- 172.21.15.98:0/1300749327 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f09e8039300 0x7f09e803b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:32.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.056+0000 7f0a0606d640 1 -- 172.21.15.98:0/1300749327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a00106c00 con 0x7f0a00106780 2024-09-19T07:54:32.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.056+0000 7f09fcff9640 1 -- 172.21.15.98:0/1300749327 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f09f006d880 con 0x7f0a00106780 2024-09-19T07:54:32.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.056+0000 7f09feffd640 1 --2- 172.21.15.98:0/1300749327 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f09e8039300 0x7f09e803b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:32.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.057+0000 7f09feffd640 1 --2- 172.21.15.98:0/1300749327 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f09e8039300 0x7f09e803b7c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f09f40098c0 tx=0x7f09f4006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:32.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.061+0000 7f09fcff9640 1 -- 172.21.15.98:0/1300749327 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f09f0044310 con 0x7f0a00106780 2024-09-19T07:54:32.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.315+0000 7f0a0606d640 1 -- 172.21.15.98:0/1300749327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f0a0002d9f0 con 0x7f0a00106780 2024-09-19T07:54:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.317+0000 7f09fcff9640 1 -- 172.21.15.98:0/1300749327 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f09f0045520 con 0x7f0a00106780 2024-09-19T07:54:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-19T07:54:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.319+0000 7f0a0606d640 1 -- 172.21.15.98:0/1300749327 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f09e8039300 msgr2=0x7f09e803b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.319+0000 7f0a0606d640 1 --2- 172.21.15.98:0/1300749327 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f09e8039300 0x7f09e803b7c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f09f40098c0 tx=0x7f09f4006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.319+0000 7f0a0606d640 1 -- 172.21.15.98:0/1300749327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a00106780 msgr2=0x7f0a0019b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.319+0000 7f0a0606d640 1 --2- 172.21.15.98:0/1300749327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a00106780 0x7f0a0019b8e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f09f0031cc0 tx=0x7f09f0031cf0 comp rx=0 tx=0).stop 2024-09-19T07:54:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.319+0000 7f0a0606d640 1 -- 172.21.15.98:0/1300749327 shutdown_connections 2024-09-19T07:54:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.319+0000 7f0a0606d640 1 --2- 172.21.15.98:0/1300749327 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f09e8039300 0x7f09e803b7c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.319+0000 7f0a0606d640 1 --2- 172.21.15.98:0/1300749327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a00106780 0x7f0a0019b8e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.319+0000 7f0a0606d640 1 -- 172.21.15.98:0/1300749327 >> 172.21.15.98:0/1300749327 conn(0x7f0a00101f10 msgr2=0x7f0a00102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.320+0000 7f0a0606d640 1 -- 172.21.15.98:0/1300749327 shutdown_connections 2024-09-19T07:54:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-09-19T07:54:32.320+0000 7f0a0606d640 1 -- 172.21.15.98:0/1300749327 wait complete. 2024-09-19T07:54:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout:Or, if you are only running a single cluster on this host: 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: ceph telemetry on 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout:For more information see: 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-19T07:54:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:54:32.890 INFO:teuthology.orchestra.run.smithi098.stdout:Bootstrap complete. 2024-09-19T07:54:32.924 INFO:tasks.cephadm:Fetching config... 2024-09-19T07:54:32.924 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T07:54:32.924 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-19T07:54:32.941 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-19T07:54:32.941 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T07:54:32.942 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-19T07:54:32.998 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-19T07:54:32.998 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T07:54:32.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-19T07:54:33.063 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-19T07:54:33.063 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T07:54:33.063 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-19T07:54:33.118 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-19T07:54:33.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3KEOHcZ00COWq7d9RkUMFiy1T1qOb4K5EkvUBN7iaFERibi35eJLiVvmV9xJ5Yry6pD8cDm3pg0KqvQCNPcf30lBrcOdyXbC2lbPSiECybOAVStT+cvMOLZfHZ7QdwEBXwhlvHf6kdc9PQPwEkimLTPAopBbohe9U8+KhTT5cp/IA/8fkNXGeJMqy+momiw7o4tBuUDZKpuxv3lji0mXZw9bSTyEoY+rip+sR9zcBsjcV0B4aOUFfJoF0sET5LxJYEZq7WmX5/RwBBLn95atmaK3EfWIrdNxyWarHMEL0+bNP9U41nmDgh0LtpYa4xvQw/oN/PmYKt7JnsBCZQ3rjYa28rQdeFZD6qS+aTeO9J3WN3zFyMadFRlUPbQNXixzB2BLfyyeAze7U8ZzxjZKiSUCt2uBQAARiI4MKaZquNNuSqvsbXs+ShC7KS0oV9ky+ccq6YcVA7gKjxuIqT0hVXY79jqPqTBATCQald1c4wjghKc5Olqfr70nbLUfipZ8= ceph-1a02525a-765c-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-19T07:54:33.198 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3KEOHcZ00COWq7d9RkUMFiy1T1qOb4K5EkvUBN7iaFERibi35eJLiVvmV9xJ5Yry6pD8cDm3pg0KqvQCNPcf30lBrcOdyXbC2lbPSiECybOAVStT+cvMOLZfHZ7QdwEBXwhlvHf6kdc9PQPwEkimLTPAopBbohe9U8+KhTT5cp/IA/8fkNXGeJMqy+momiw7o4tBuUDZKpuxv3lji0mXZw9bSTyEoY+rip+sR9zcBsjcV0B4aOUFfJoF0sET5LxJYEZq7WmX5/RwBBLn95atmaK3EfWIrdNxyWarHMEL0+bNP9U41nmDgh0LtpYa4xvQw/oN/PmYKt7JnsBCZQ3rjYa28rQdeFZD6qS+aTeO9J3WN3zFyMadFRlUPbQNXixzB2BLfyyeAze7U8ZzxjZKiSUCt2uBQAARiI4MKaZquNNuSqvsbXs+ShC7KS0oV9ky+ccq6YcVA7gKjxuIqT0hVXY79jqPqTBATCQald1c4wjghKc5Olqfr70nbLUfipZ8= ceph-1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:54:33.208 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3KEOHcZ00COWq7d9RkUMFiy1T1qOb4K5EkvUBN7iaFERibi35eJLiVvmV9xJ5Yry6pD8cDm3pg0KqvQCNPcf30lBrcOdyXbC2lbPSiECybOAVStT+cvMOLZfHZ7QdwEBXwhlvHf6kdc9PQPwEkimLTPAopBbohe9U8+KhTT5cp/IA/8fkNXGeJMqy+momiw7o4tBuUDZKpuxv3lji0mXZw9bSTyEoY+rip+sR9zcBsjcV0B4aOUFfJoF0sET5LxJYEZq7WmX5/RwBBLn95atmaK3EfWIrdNxyWarHMEL0+bNP9U41nmDgh0LtpYa4xvQw/oN/PmYKt7JnsBCZQ3rjYa28rQdeFZD6qS+aTeO9J3WN3zFyMadFRlUPbQNXixzB2BLfyyeAze7U8ZzxjZKiSUCt2uBQAARiI4MKaZquNNuSqvsbXs+ShC7KS0oV9ky+ccq6YcVA7gKjxuIqT0hVXY79jqPqTBATCQald1c4wjghKc5Olqfr70nbLUfipZ8= ceph-1a02525a-765c-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-19T07:54:33.250 INFO:teuthology.orchestra.run.smithi139.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3KEOHcZ00COWq7d9RkUMFiy1T1qOb4K5EkvUBN7iaFERibi35eJLiVvmV9xJ5Yry6pD8cDm3pg0KqvQCNPcf30lBrcOdyXbC2lbPSiECybOAVStT+cvMOLZfHZ7QdwEBXwhlvHf6kdc9PQPwEkimLTPAopBbohe9U8+KhTT5cp/IA/8fkNXGeJMqy+momiw7o4tBuUDZKpuxv3lji0mXZw9bSTyEoY+rip+sR9zcBsjcV0B4aOUFfJoF0sET5LxJYEZq7WmX5/RwBBLn95atmaK3EfWIrdNxyWarHMEL0+bNP9U41nmDgh0LtpYa4xvQw/oN/PmYKt7JnsBCZQ3rjYa28rQdeFZD6qS+aTeO9J3WN3zFyMadFRlUPbQNXixzB2BLfyyeAze7U8ZzxjZKiSUCt2uBQAARiI4MKaZquNNuSqvsbXs+ShC7KS0oV9ky+ccq6YcVA7gKjxuIqT0hVXY79jqPqTBATCQald1c4wjghKc5Olqfr70nbLUfipZ8= ceph-1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:54:33.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3KEOHcZ00COWq7d9RkUMFiy1T1qOb4K5EkvUBN7iaFERibi35eJLiVvmV9xJ5Yry6pD8cDm3pg0KqvQCNPcf30lBrcOdyXbC2lbPSiECybOAVStT+cvMOLZfHZ7QdwEBXwhlvHf6kdc9PQPwEkimLTPAopBbohe9U8+KhTT5cp/IA/8fkNXGeJMqy+momiw7o4tBuUDZKpuxv3lji0mXZw9bSTyEoY+rip+sR9zcBsjcV0B4aOUFfJoF0sET5LxJYEZq7WmX5/RwBBLn95atmaK3EfWIrdNxyWarHMEL0+bNP9U41nmDgh0LtpYa4xvQw/oN/PmYKt7JnsBCZQ3rjYa28rQdeFZD6qS+aTeO9J3WN3zFyMadFRlUPbQNXixzB2BLfyyeAze7U8ZzxjZKiSUCt2uBQAARiI4MKaZquNNuSqvsbXs+ShC7KS0oV9ky+ccq6YcVA7gKjxuIqT0hVXY79jqPqTBATCQald1c4wjghKc5Olqfr70nbLUfipZ8= ceph-1a02525a-765c-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-19T07:54:33.305 INFO:teuthology.orchestra.run.smithi162.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3KEOHcZ00COWq7d9RkUMFiy1T1qOb4K5EkvUBN7iaFERibi35eJLiVvmV9xJ5Yry6pD8cDm3pg0KqvQCNPcf30lBrcOdyXbC2lbPSiECybOAVStT+cvMOLZfHZ7QdwEBXwhlvHf6kdc9PQPwEkimLTPAopBbohe9U8+KhTT5cp/IA/8fkNXGeJMqy+momiw7o4tBuUDZKpuxv3lji0mXZw9bSTyEoY+rip+sR9zcBsjcV0B4aOUFfJoF0sET5LxJYEZq7WmX5/RwBBLn95atmaK3EfWIrdNxyWarHMEL0+bNP9U41nmDgh0LtpYa4xvQw/oN/PmYKt7JnsBCZQ3rjYa28rQdeFZD6qS+aTeO9J3WN3zFyMadFRlUPbQNXixzB2BLfyyeAze7U8ZzxjZKiSUCt2uBQAARiI4MKaZquNNuSqvsbXs+ShC7KS0oV9ky+ccq6YcVA7gKjxuIqT0hVXY79jqPqTBATCQald1c4wjghKc5Olqfr70nbLUfipZ8= ceph-1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:54:33.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-19T07:54:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:54:34.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.048+0000 7fb06cca3640 1 -- 172.21.15.98:0/402936212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb068100510 msgr2=0x7fb068100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:34.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.048+0000 7fb06cca3640 1 --2- 172.21.15.98:0/402936212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb068100510 0x7fb068100910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb058009880 tx=0x7fb05802a5d0 comp rx=0 tx=0).stop 2024-09-19T07:54:34.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.048+0000 7fb06cca3640 1 -- 172.21.15.98:0/402936212 shutdown_connections 2024-09-19T07:54:34.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.048+0000 7fb06cca3640 1 --2- 172.21.15.98:0/402936212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb068100510 0x7fb068100910 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:34.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.048+0000 7fb06cca3640 1 -- 172.21.15.98:0/402936212 >> 172.21.15.98:0/402936212 conn(0x7fb0680fbc80 msgr2=0x7fb0680fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:34.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.048+0000 7fb06cca3640 1 -- 172.21.15.98:0/402936212 shutdown_connections 2024-09-19T07:54:34.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.048+0000 7fb06cca3640 1 -- 172.21.15.98:0/402936212 wait complete. 2024-09-19T07:54:34.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.049+0000 7fb06cca3640 1 Processor -- start 2024-09-19T07:54:34.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.049+0000 7fb06cca3640 1 -- start start 2024-09-19T07:54:34.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.049+0000 7fb06cca3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb068100510 0x7fb068194bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:34.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.049+0000 7fb06cca3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb068105520 con 0x7fb068100510 2024-09-19T07:54:34.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.049+0000 7fb06659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb068100510 0x7fb068194bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:34.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.049+0000 7fb06659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb068100510 0x7fb068194bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49162/0 (socket says 172.21.15.98:49162) 2024-09-19T07:54:34.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.049+0000 7fb06659c640 1 -- 172.21.15.98:0/3680425567 learned_addr learned my addr 172.21.15.98:0/3680425567 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:34.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.049+0000 7fb06659c640 1 -- 172.21.15.98:0/3680425567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb068195100 con 0x7fb068100510 2024-09-19T07:54:34.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.050+0000 7fb06659c640 1 --2- 172.21.15.98:0/3680425567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb068100510 0x7fb068194bc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb058004f60 tx=0x7fb0580048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:34.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.050+0000 7fb0577fe640 1 -- 172.21.15.98:0/3680425567 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb05803f070 con 0x7fb068100510 2024-09-19T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.050+0000 7fb0577fe640 1 -- 172.21.15.98:0/3680425567 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb058004d40 con 0x7fb068100510 2024-09-19T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.050+0000 7fb0577fe640 1 -- 172.21.15.98:0/3680425567 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb058035030 con 0x7fb068100510 2024-09-19T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.050+0000 7fb06cca3640 1 -- 172.21.15.98:0/3680425567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb068195390 con 0x7fb068100510 2024-09-19T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.050+0000 7fb06cca3640 1 -- 172.21.15.98:0/3680425567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb068198080 con 0x7fb068100510 2024-09-19T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.051+0000 7fb0577fe640 1 -- 172.21.15.98:0/3680425567 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fb058042070 con 0x7fb068100510 2024-09-19T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.051+0000 7fb0577fe640 1 --2- 172.21.15.98:0/3680425567 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb05003b550 0x7fb05003da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.051+0000 7fb0577fe640 1 -- 172.21.15.98:0/3680425567 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb05806cff0 con 0x7fb068100510 2024-09-19T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.051+0000 7fb065d9b640 1 --2- 172.21.15.98:0/3680425567 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb05003b550 0x7fb05003da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.051+0000 7fb06cca3640 1 -- 172.21.15.98:0/3680425567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb068105ca0 con 0x7fb068100510 2024-09-19T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.051+0000 7fb065d9b640 1 --2- 172.21.15.98:0/3680425567 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb05003b550 0x7fb05003da10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0098c0 tx=0x7fb05c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.056+0000 7fb0577fe640 1 -- 172.21.15.98:0/3680425567 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb058042360 con 0x7fb068100510 2024-09-19T07:54:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.225+0000 7fb06cca3640 1 -- 172.21.15.98:0/3680425567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fb06802d7c0 con 0x7fb068100510 2024-09-19T07:54:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.228+0000 7fb0577fe640 1 -- 172.21.15.98:0/3680425567 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fb05802f300 con 0x7fb068100510 2024-09-19T07:54:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.230+0000 7fb06cca3640 1 -- 172.21.15.98:0/3680425567 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb05003b550 msgr2=0x7fb05003da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.230+0000 7fb06cca3640 1 --2- 172.21.15.98:0/3680425567 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb05003b550 0x7fb05003da10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0098c0 tx=0x7fb05c006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.230+0000 7fb06cca3640 1 -- 172.21.15.98:0/3680425567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb068100510 msgr2=0x7fb068194bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.230+0000 7fb06cca3640 1 --2- 172.21.15.98:0/3680425567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb068100510 0x7fb068194bc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb058004f60 tx=0x7fb0580048f0 comp rx=0 tx=0).stop 2024-09-19T07:54:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.230+0000 7fb06cca3640 1 -- 172.21.15.98:0/3680425567 shutdown_connections 2024-09-19T07:54:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.230+0000 7fb06cca3640 1 --2- 172.21.15.98:0/3680425567 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb05003b550 0x7fb05003da10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.230+0000 7fb06cca3640 1 --2- 172.21.15.98:0/3680425567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb068100510 0x7fb068194bc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.230+0000 7fb06cca3640 1 -- 172.21.15.98:0/3680425567 >> 172.21.15.98:0/3680425567 conn(0x7fb0680fbc80 msgr2=0x7fb0680fc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.230+0000 7fb06cca3640 1 -- 172.21.15.98:0/3680425567 shutdown_connections 2024-09-19T07:54:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:34.230+0000 7fb06cca3640 1 -- 172.21.15.98:0/3680425567 wait complete. 2024-09-19T07:54:34.850 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-19T07:54:34.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-19T07:54:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:54:35.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:35 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3680425567' entity='client.admin' 2024-09-19T07:54:36.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.066+0000 7fc3eacef640 1 -- 172.21.15.98:0/3097405319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3e4101600 msgr2=0x7fc3e4101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:36.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.066+0000 7fc3eacef640 1 --2- 172.21.15.98:0/3097405319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3e4101600 0x7fc3e4101a00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009880 tx=0x7fc3d402a5d0 comp rx=0 tx=0).stop 2024-09-19T07:54:36.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.066+0000 7fc3eacef640 1 -- 172.21.15.98:0/3097405319 shutdown_connections 2024-09-19T07:54:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.066+0000 7fc3eacef640 1 --2- 172.21.15.98:0/3097405319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3e4101600 0x7fc3e4101a00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.066+0000 7fc3eacef640 1 -- 172.21.15.98:0/3097405319 >> 172.21.15.98:0/3097405319 conn(0x7fc3e40fcdb0 msgr2=0x7fc3e40ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.066+0000 7fc3eacef640 1 -- 172.21.15.98:0/3097405319 shutdown_connections 2024-09-19T07:54:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.066+0000 7fc3eacef640 1 -- 172.21.15.98:0/3097405319 wait complete. 2024-09-19T07:54:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.067+0000 7fc3eacef640 1 Processor -- start 2024-09-19T07:54:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.067+0000 7fc3eacef640 1 -- start start 2024-09-19T07:54:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.067+0000 7fc3eacef640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3e4101600 0x7fc3e4196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.067+0000 7fc3eacef640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e4107710 con 0x7fc3e4101600 2024-09-19T07:54:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.067+0000 7fc3e8a8b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3e4101600 0x7fc3e4196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.067+0000 7fc3e8a8b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3e4101600 0x7fc3e4196da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49170/0 (socket says 172.21.15.98:49170) 2024-09-19T07:54:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.067+0000 7fc3e8a8b640 1 -- 172.21.15.98:0/2070906115 learned_addr learned my addr 172.21.15.98:0/2070906115 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.068+0000 7fc3e8a8b640 1 -- 172.21.15.98:0/2070906115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e41972e0 con 0x7fc3e4101600 2024-09-19T07:54:36.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.068+0000 7fc3e8a8b640 1 --2- 172.21.15.98:0/2070906115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3e4101600 0x7fc3e4196da0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4004f60 tx=0x7fc3d40048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:36.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.068+0000 7fc3e1ffb640 1 -- 172.21.15.98:0/2070906115 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3d403f070 con 0x7fc3e4101600 2024-09-19T07:54:36.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.068+0000 7fc3e1ffb640 1 -- 172.21.15.98:0/2070906115 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3d4004d40 con 0x7fc3e4101600 2024-09-19T07:54:36.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.068+0000 7fc3eacef640 1 -- 172.21.15.98:0/2070906115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e4197570 con 0x7fc3e4101600 2024-09-19T07:54:36.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.068+0000 7fc3e1ffb640 1 -- 172.21.15.98:0/2070906115 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3d4035030 con 0x7fc3e4101600 2024-09-19T07:54:36.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.068+0000 7fc3eacef640 1 -- 172.21.15.98:0/2070906115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e419a220 con 0x7fc3e4101600 2024-09-19T07:54:36.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.069+0000 7fc3e1ffb640 1 -- 172.21.15.98:0/2070906115 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fc3d4042070 con 0x7fc3e4101600 2024-09-19T07:54:36.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.069+0000 7fc3e1ffb640 1 --2- 172.21.15.98:0/2070906115 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc3cc0392a0 0x7fc3cc03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:36.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.069+0000 7fc3eacef640 1 -- 172.21.15.98:0/2070906115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3e4107e90 con 0x7fc3e4101600 2024-09-19T07:54:36.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.069+0000 7fc3e1ffb640 1 -- 172.21.15.98:0/2070906115 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc3d406d1a0 con 0x7fc3e4101600 2024-09-19T07:54:36.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.069+0000 7fc3e3fff640 1 --2- 172.21.15.98:0/2070906115 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc3cc0392a0 0x7fc3cc03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:36.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.070+0000 7fc3e3fff640 1 --2- 172.21.15.98:0/2070906115 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc3cc0392a0 0x7fc3cc03b760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80098c0 tx=0x7fc3d8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.074+0000 7fc3e1ffb640 1 -- 172.21.15.98:0/2070906115 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fc3d403b550 con 0x7fc3e4101600 2024-09-19T07:54:36.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.252+0000 7fc3eacef640 1 -- 172.21.15.98:0/2070906115 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fc3e402d940 con 0x7fc3cc0392a0 2024-09-19T07:54:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.256+0000 7fc3e1ffb640 1 -- 172.21.15.98:0/2070906115 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc3e402d940 con 0x7fc3cc0392a0 2024-09-19T07:54:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.258+0000 7fc3eacef640 1 -- 172.21.15.98:0/2070906115 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc3cc0392a0 msgr2=0x7fc3cc03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.258+0000 7fc3eacef640 1 --2- 172.21.15.98:0/2070906115 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc3cc0392a0 0x7fc3cc03b760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80098c0 tx=0x7fc3d8006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.258+0000 7fc3eacef640 1 -- 172.21.15.98:0/2070906115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3e4101600 msgr2=0x7fc3e4196da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.258+0000 7fc3eacef640 1 --2- 172.21.15.98:0/2070906115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3e4101600 0x7fc3e4196da0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4004f60 tx=0x7fc3d40048f0 comp rx=0 tx=0).stop 2024-09-19T07:54:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.259+0000 7fc3eacef640 1 -- 172.21.15.98:0/2070906115 shutdown_connections 2024-09-19T07:54:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.259+0000 7fc3eacef640 1 --2- 172.21.15.98:0/2070906115 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc3cc0392a0 0x7fc3cc03b760 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.259+0000 7fc3eacef640 1 --2- 172.21.15.98:0/2070906115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3e4101600 0x7fc3e4196da0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.259+0000 7fc3eacef640 1 -- 172.21.15.98:0/2070906115 >> 172.21.15.98:0/2070906115 conn(0x7fc3e40fcdb0 msgr2=0x7fc3e40fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.259+0000 7fc3eacef640 1 -- 172.21.15.98:0/2070906115 shutdown_connections 2024-09-19T07:54:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:36.259+0000 7fc3eacef640 1 -- 172.21.15.98:0/2070906115 wait complete. 2024-09-19T07:54:36.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:36.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:36.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:54:36.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:36.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:54:36.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:37.260 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi139 2024-09-19T07:54:37.260 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-19T07:54:37.260 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/etc/ceph/ceph.conf 2024-09-19T07:54:37.280 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-19T07:54:37.280 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-19T07:54:37.338 INFO:tasks.cephadm:Adding host smithi139 to orchestrator... 2024-09-19T07:54:37.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch host add smithi139 2024-09-19T07:54:37.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:37 smithi098 ceph-mon[31333]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:54:37.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:37 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:37.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:37 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:54:37.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:37 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:54:37.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:37 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:54:37.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:37 smithi098 ceph-mon[31333]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-19T07:54:37.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:37 smithi098 ceph-mon[31333]: Updating smithi098:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:54:37.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:37 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:37.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:37 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:37.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:37 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:37.759 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:54:38.298 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:38 smithi098 ceph-mon[31333]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-19T07:54:38.298 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:38 smithi098 ceph-mon[31333]: Updating smithi098:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T07:54:38.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.297+0000 7f4a2280f640 1 -- 172.21.15.98:0/2866412363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a1c106560 msgr2=0x7f4a1c106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.297+0000 7f4a2280f640 1 --2- 172.21.15.98:0/2866412363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a1c106560 0x7f4a1c106940 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c009880 tx=0x7f4a0c02a5d0 comp rx=0 tx=0).stop 2024-09-19T07:54:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.297+0000 7f4a2280f640 1 -- 172.21.15.98:0/2866412363 shutdown_connections 2024-09-19T07:54:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.297+0000 7f4a2280f640 1 --2- 172.21.15.98:0/2866412363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a1c106560 0x7f4a1c106940 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.297+0000 7f4a2280f640 1 -- 172.21.15.98:0/2866412363 >> 172.21.15.98:0/2866412363 conn(0x7f4a1c0fc1b0 msgr2=0x7f4a1c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.297+0000 7f4a2280f640 1 -- 172.21.15.98:0/2866412363 shutdown_connections 2024-09-19T07:54:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.297+0000 7f4a2280f640 1 -- 172.21.15.98:0/2866412363 wait complete. 2024-09-19T07:54:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.298+0000 7f4a2280f640 1 Processor -- start 2024-09-19T07:54:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.298+0000 7f4a2280f640 1 -- start start 2024-09-19T07:54:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.298+0000 7f4a2280f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a1c106560 0x7f4a1c194ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.298+0000 7f4a2280f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c109aa0 con 0x7f4a1c106560 2024-09-19T07:54:38.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.299+0000 7f4a1bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a1c106560 0x7f4a1c194ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:38.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.299+0000 7f4a1bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a1c106560 0x7f4a1c194ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33492/0 (socket says 172.21.15.98:33492) 2024-09-19T07:54:38.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.299+0000 7f4a1bfff640 1 -- 172.21.15.98:0/1191444562 learned_addr learned my addr 172.21.15.98:0/1191444562 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:38.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.299+0000 7f4a1bfff640 1 -- 172.21.15.98:0/1191444562 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1c195430 con 0x7f4a1c106560 2024-09-19T07:54:38.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.299+0000 7f4a1bfff640 1 --2- 172.21.15.98:0/1191444562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a1c106560 0x7f4a1c194ef0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c009230 tx=0x7f4a0c003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:38.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.299+0000 7f4a197fa640 1 -- 172.21.15.98:0/1191444562 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a0c03f070 con 0x7f4a1c106560 2024-09-19T07:54:38.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.299+0000 7f4a197fa640 1 -- 172.21.15.98:0/1191444562 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a0c03bc80 con 0x7f4a1c106560 2024-09-19T07:54:38.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.299+0000 7f4a197fa640 1 -- 172.21.15.98:0/1191444562 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a0c035030 con 0x7f4a1c106560 2024-09-19T07:54:38.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.299+0000 7f4a2280f640 1 -- 172.21.15.98:0/1191444562 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a1c1956c0 con 0x7f4a1c106560 2024-09-19T07:54:38.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.299+0000 7f4a2280f640 1 -- 172.21.15.98:0/1191444562 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a1c1997c0 con 0x7f4a1c106560 2024-09-19T07:54:38.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.300+0000 7f4a197fa640 1 -- 172.21.15.98:0/1191444562 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f4a0c042070 con 0x7f4a1c106560 2024-09-19T07:54:38.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.300+0000 7f4a197fa640 1 --2- 172.21.15.98:0/1191444562 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4a0403b550 0x7f4a0403da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:38.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.300+0000 7f4a2280f640 1 -- 172.21.15.98:0/1191444562 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a1c101bd0 con 0x7f4a1c106560 2024-09-19T07:54:38.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.301+0000 7f4a197fa640 1 -- 172.21.15.98:0/1191444562 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4a0c06cf30 con 0x7f4a1c106560 2024-09-19T07:54:38.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.301+0000 7f4a1b7fe640 1 --2- 172.21.15.98:0/1191444562 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4a0403b550 0x7f4a0403da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:38.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.301+0000 7f4a1b7fe640 1 --2- 172.21.15.98:0/1191444562 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4a0403b550 0x7f4a0403da10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4a100098c0 tx=0x7f4a10006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:38.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.305+0000 7f4a197fa640 1 -- 172.21.15.98:0/1191444562 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f4a0c0314c0 con 0x7f4a1c106560 2024-09-19T07:54:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:38.476+0000 7f4a2280f640 1 -- 172.21.15.98:0/1191444562 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi139", "target": ["mon-mgr", ""]}) -- 0x7f4a1c196250 con 0x7f4a0403b550 2024-09-19T07:54:39.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:39 smithi098 ceph-mon[31333]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi139", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:54:40.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:40 smithi098 ceph-mon[31333]: Deploying cephadm binary to smithi139 2024-09-19T07:54:41.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:41.620+0000 7f4a197fa640 1 -- 172.21.15.98:0/1191444562 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f4a1c196250 con 0x7f4a0403b550 2024-09-19T07:54:41.620 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi139' with addr '172.21.15.139' 2024-09-19T07:54:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:41.622+0000 7f4a2280f640 1 -- 172.21.15.98:0/1191444562 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4a0403b550 msgr2=0x7f4a0403da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:41.622+0000 7f4a2280f640 1 --2- 172.21.15.98:0/1191444562 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4a0403b550 0x7f4a0403da10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4a100098c0 tx=0x7f4a10006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:41.622+0000 7f4a2280f640 1 -- 172.21.15.98:0/1191444562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a1c106560 msgr2=0x7f4a1c194ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:41.622+0000 7f4a2280f640 1 --2- 172.21.15.98:0/1191444562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a1c106560 0x7f4a1c194ef0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c009230 tx=0x7f4a0c003d40 comp rx=0 tx=0).stop 2024-09-19T07:54:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:41.622+0000 7f4a2280f640 1 -- 172.21.15.98:0/1191444562 shutdown_connections 2024-09-19T07:54:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:41.622+0000 7f4a2280f640 1 --2- 172.21.15.98:0/1191444562 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4a0403b550 0x7f4a0403da10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:41.622+0000 7f4a2280f640 1 --2- 172.21.15.98:0/1191444562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a1c106560 0x7f4a1c194ef0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:41.622+0000 7f4a2280f640 1 -- 172.21.15.98:0/1191444562 >> 172.21.15.98:0/1191444562 conn(0x7f4a1c0fc1b0 msgr2=0x7f4a1c0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:41.622+0000 7f4a2280f640 1 -- 172.21.15.98:0/1191444562 shutdown_connections 2024-09-19T07:54:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:41.622+0000 7f4a2280f640 1 -- 172.21.15.98:0/1191444562 wait complete. 2024-09-19T07:54:42.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-19T07:54:42.461 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:54:42.872 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:42.872 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:42 smithi098 ceph-mon[31333]: Added host smithi139 2024-09-19T07:54:42.872 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:54:42.872 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:42.872 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.877+0000 7f3638540640 1 -- 172.21.15.98:0/355744964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3630107660 msgr2=0x7f3630107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.877+0000 7f3638540640 1 --2- 172.21.15.98:0/355744964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3630107660 0x7f3630107a40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f362c009880 tx=0x7f362c02a5d0 comp rx=0 tx=0).stop 2024-09-19T07:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.877+0000 7f3638540640 1 -- 172.21.15.98:0/355744964 shutdown_connections 2024-09-19T07:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.877+0000 7f3638540640 1 --2- 172.21.15.98:0/355744964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3630107660 0x7f3630107a40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.877+0000 7f3638540640 1 -- 172.21.15.98:0/355744964 >> 172.21.15.98:0/355744964 conn(0x7f36300fd270 msgr2=0x7f36300ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.877+0000 7f3638540640 1 -- 172.21.15.98:0/355744964 shutdown_connections 2024-09-19T07:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.877+0000 7f3638540640 1 -- 172.21.15.98:0/355744964 wait complete. 2024-09-19T07:54:42.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.878+0000 7f3638540640 1 Processor -- start 2024-09-19T07:54:42.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.878+0000 7f3638540640 1 -- start start 2024-09-19T07:54:42.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.878+0000 7f3638540640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3630107660 0x7f36300744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:42.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.878+0000 7f3638540640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f363010bc90 con 0x7f3630107660 2024-09-19T07:54:42.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.878+0000 7f36362dc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3630107660 0x7f36300744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:42.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.878+0000 7f36362dc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3630107660 0x7f36300744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33512/0 (socket says 172.21.15.98:33512) 2024-09-19T07:54:42.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.878+0000 7f36362dc640 1 -- 172.21.15.98:0/2978291768 learned_addr learned my addr 172.21.15.98:0/2978291768 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:42.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.879+0000 7f36362dc640 1 -- 172.21.15.98:0/2978291768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36300763e0 con 0x7f3630107660 2024-09-19T07:54:42.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.879+0000 7f36362dc640 1 --2- 172.21.15.98:0/2978291768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3630107660 0x7f36300744f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f362c009230 tx=0x7f362c003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:42.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.879+0000 7f36277fe640 1 -- 172.21.15.98:0/2978291768 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f362c03f070 con 0x7f3630107660 2024-09-19T07:54:42.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.879+0000 7f36277fe640 1 -- 172.21.15.98:0/2978291768 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f362c03bd40 con 0x7f3630107660 2024-09-19T07:54:42.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.879+0000 7f3638540640 1 -- 172.21.15.98:0/2978291768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3630074a30 con 0x7f3630107660 2024-09-19T07:54:42.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.879+0000 7f36277fe640 1 -- 172.21.15.98:0/2978291768 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f362c035030 con 0x7f3630107660 2024-09-19T07:54:42.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.879+0000 7f3638540640 1 -- 172.21.15.98:0/2978291768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3630074ef0 con 0x7f3630107660 2024-09-19T07:54:42.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.880+0000 7f36277fe640 1 -- 172.21.15.98:0/2978291768 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f362c042070 con 0x7f3630107660 2024-09-19T07:54:42.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.880+0000 7f36277fe640 1 --2- 172.21.15.98:0/2978291768 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f361c0392a0 0x7f361c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:42.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.880+0000 7f36277fe640 1 -- 172.21.15.98:0/2978291768 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f362c06cfb0 con 0x7f3630107660 2024-09-19T07:54:42.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.880+0000 7f3638540640 1 -- 172.21.15.98:0/2978291768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3630102cd0 con 0x7f3630107660 2024-09-19T07:54:42.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.880+0000 7f3635adb640 1 --2- 172.21.15.98:0/2978291768 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f361c0392a0 0x7f361c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:42.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.880+0000 7f3635adb640 1 --2- 172.21.15.98:0/2978291768 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f361c0392a0 0x7f361c03b760 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f36200098c0 tx=0x7f3620006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:42.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:42.885+0000 7f36277fe640 1 -- 172.21.15.98:0/2978291768 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f362c0314c0 con 0x7f3630107660 2024-09-19T07:54:43.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:43.057+0000 7f3638540640 1 -- 172.21.15.98:0/2978291768 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3630075a20 con 0x7f361c0392a0 2024-09-19T07:54:43.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:43.058+0000 7f36277fe640 1 -- 172.21.15.98:0/2978291768 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f3630075a20 con 0x7f361c0392a0 2024-09-19T07:54:43.058 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:54:43.059 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.139", "hostname": "smithi139", "labels": [], "status": ""}] 2024-09-19T07:54:43.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:43.060+0000 7f3638540640 1 -- 172.21.15.98:0/2978291768 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f361c0392a0 msgr2=0x7f361c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:43.060+0000 7f3638540640 1 --2- 172.21.15.98:0/2978291768 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f361c0392a0 0x7f361c03b760 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f36200098c0 tx=0x7f3620006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:43.060+0000 7f3638540640 1 -- 172.21.15.98:0/2978291768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3630107660 msgr2=0x7f36300744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:43.060+0000 7f3638540640 1 --2- 172.21.15.98:0/2978291768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3630107660 0x7f36300744f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f362c009230 tx=0x7f362c003d80 comp rx=0 tx=0).stop 2024-09-19T07:54:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:43.060+0000 7f3638540640 1 -- 172.21.15.98:0/2978291768 shutdown_connections 2024-09-19T07:54:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:43.060+0000 7f3638540640 1 --2- 172.21.15.98:0/2978291768 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f361c0392a0 0x7f361c03b760 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:43.060+0000 7f3638540640 1 --2- 172.21.15.98:0/2978291768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3630107660 0x7f36300744f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:43.060+0000 7f3638540640 1 -- 172.21.15.98:0/2978291768 >> 172.21.15.98:0/2978291768 conn(0x7f36300fd270 msgr2=0x7f36300fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:43.061+0000 7f3638540640 1 -- 172.21.15.98:0/2978291768 shutdown_connections 2024-09-19T07:54:43.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:43.061+0000 7f3638540640 1 -- 172.21.15.98:0/2978291768 wait complete. 2024-09-19T07:54:43.719 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi162 2024-09-19T07:54:43.720 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-19T07:54:43.720 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.conf 2024-09-19T07:54:43.741 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-19T07:54:43.741 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-19T07:54:43.803 INFO:tasks.cephadm:Adding host smithi162 to orchestrator... 2024-09-19T07:54:43.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch host add smithi162 2024-09-19T07:54:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:54:44.278 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:43 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:44.278 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:43 smithi098 ceph-mon[31333]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T07:54:44.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.484+0000 7fac9ecff640 1 -- 172.21.15.98:0/853079687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98106560 msgr2=0x7fac98106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:44.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.484+0000 7fac9ecff640 1 --2- 172.21.15.98:0/853079687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98106560 0x7fac98106940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fac94009880 tx=0x7fac9402a5d0 comp rx=0 tx=0).stop 2024-09-19T07:54:44.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.484+0000 7fac9ecff640 1 -- 172.21.15.98:0/853079687 shutdown_connections 2024-09-19T07:54:44.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.484+0000 7fac9ecff640 1 --2- 172.21.15.98:0/853079687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98106560 0x7fac98106940 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:44.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.484+0000 7fac9ecff640 1 -- 172.21.15.98:0/853079687 >> 172.21.15.98:0/853079687 conn(0x7fac980fc1b0 msgr2=0x7fac980fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:44.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.484+0000 7fac9ecff640 1 -- 172.21.15.98:0/853079687 shutdown_connections 2024-09-19T07:54:44.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.484+0000 7fac9ecff640 1 -- 172.21.15.98:0/853079687 wait complete. 2024-09-19T07:54:44.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.485+0000 7fac9ecff640 1 Processor -- start 2024-09-19T07:54:44.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.485+0000 7fac9ecff640 1 -- start start 2024-09-19T07:54:44.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.485+0000 7fac9ecff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98106560 0x7fac98197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:44.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.485+0000 7fac9ecff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac9810ab90 con 0x7fac98106560 2024-09-19T07:54:44.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.485+0000 7fac9ca9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98106560 0x7fac98197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:44.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.485+0000 7fac9ca9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98106560 0x7fac98197190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33524/0 (socket says 172.21.15.98:33524) 2024-09-19T07:54:44.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.485+0000 7fac9ca9b640 1 -- 172.21.15.98:0/3600657251 learned_addr learned my addr 172.21.15.98:0/3600657251 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:44.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.486+0000 7fac9ca9b640 1 -- 172.21.15.98:0/3600657251 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac981976d0 con 0x7fac98106560 2024-09-19T07:54:44.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.486+0000 7fac9ca9b640 1 --2- 172.21.15.98:0/3600657251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98106560 0x7fac98197190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fac94009230 tx=0x7fac94003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:44.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.486+0000 7fac8dffb640 1 -- 172.21.15.98:0/3600657251 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac9403f070 con 0x7fac98106560 2024-09-19T07:54:44.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.486+0000 7fac8dffb640 1 -- 172.21.15.98:0/3600657251 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac9403bc80 con 0x7fac98106560 2024-09-19T07:54:44.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.486+0000 7fac9ecff640 1 -- 172.21.15.98:0/3600657251 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac98197960 con 0x7fac98106560 2024-09-19T07:54:44.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.486+0000 7fac8dffb640 1 -- 172.21.15.98:0/3600657251 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac94035030 con 0x7fac98106560 2024-09-19T07:54:44.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.486+0000 7fac9ecff640 1 -- 172.21.15.98:0/3600657251 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac98197dc0 con 0x7fac98106560 2024-09-19T07:54:44.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.487+0000 7fac8dffb640 1 -- 172.21.15.98:0/3600657251 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fac94042070 con 0x7fac98106560 2024-09-19T07:54:44.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.487+0000 7fac8dffb640 1 --2- 172.21.15.98:0/3600657251 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fac80039300 0x7fac8003b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:44.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.487+0000 7fac9ecff640 1 -- 172.21.15.98:0/3600657251 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac98101bd0 con 0x7fac98106560 2024-09-19T07:54:44.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.487+0000 7fac8dffb640 1 -- 172.21.15.98:0/3600657251 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fac9406d240 con 0x7fac98106560 2024-09-19T07:54:44.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.487+0000 7fac8ffff640 1 --2- 172.21.15.98:0/3600657251 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fac80039300 0x7fac8003b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:44.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.488+0000 7fac8ffff640 1 --2- 172.21.15.98:0/3600657251 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fac80039300 0x7fac8003b7c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fac880098c0 tx=0x7fac88006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:44.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.492+0000 7fac8dffb640 1 -- 172.21.15.98:0/3600657251 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fac940351d0 con 0x7fac98106560 2024-09-19T07:54:44.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:44.665+0000 7fac9ecff640 1 -- 172.21.15.98:0/3600657251 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}) -- 0x7fac98198550 con 0x7fac80039300 2024-09-19T07:54:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:46 smithi098 ceph-mon[31333]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:54:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:46 smithi098 ceph-mon[31333]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:54:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:46 smithi098 ceph-mon[31333]: Deploying cephadm binary to smithi162 2024-09-19T07:54:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:46.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:46.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:46.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:46.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi139", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:54:46.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:54:46.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:54:47.770 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:47 smithi098 ceph-mon[31333]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-19T07:54:47.770 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:47 smithi098 ceph-mon[31333]: Updating smithi139:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:54:47.770 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:47 smithi098 ceph-mon[31333]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-09-19T07:54:47.770 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:47 smithi098 ceph-mon[31333]: Updating smithi139:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T07:54:47.770 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:47 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:47.770 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:47 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:47.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:47 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:47.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:47.769+0000 7fac8dffb640 1 -- 172.21.15.98:0/3600657251 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fac98198550 con 0x7fac80039300 2024-09-19T07:54:47.771 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi162' with addr '172.21.15.162' 2024-09-19T07:54:47.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:47.772+0000 7fac9ecff640 1 -- 172.21.15.98:0/3600657251 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fac80039300 msgr2=0x7fac8003b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:47.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:47.772+0000 7fac9ecff640 1 --2- 172.21.15.98:0/3600657251 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fac80039300 0x7fac8003b7c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fac880098c0 tx=0x7fac88006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:47.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:47.772+0000 7fac9ecff640 1 -- 172.21.15.98:0/3600657251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98106560 msgr2=0x7fac98197190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:47.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:47.772+0000 7fac9ecff640 1 --2- 172.21.15.98:0/3600657251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98106560 0x7fac98197190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fac94009230 tx=0x7fac94003d40 comp rx=0 tx=0).stop 2024-09-19T07:54:47.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:47.772+0000 7fac9ecff640 1 -- 172.21.15.98:0/3600657251 shutdown_connections 2024-09-19T07:54:47.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:47.772+0000 7fac9ecff640 1 --2- 172.21.15.98:0/3600657251 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fac80039300 0x7fac8003b7c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:47.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:47.772+0000 7fac9ecff640 1 --2- 172.21.15.98:0/3600657251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98106560 0x7fac98197190 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:47.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:47.772+0000 7fac9ecff640 1 -- 172.21.15.98:0/3600657251 >> 172.21.15.98:0/3600657251 conn(0x7fac980fc1b0 msgr2=0x7fac98069ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:47.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:47.772+0000 7fac9ecff640 1 -- 172.21.15.98:0/3600657251 shutdown_connections 2024-09-19T07:54:47.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:47.772+0000 7fac9ecff640 1 -- 172.21.15.98:0/3600657251 wait complete. 2024-09-19T07:54:48.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-19T07:54:48.635 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:54:48.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:48 smithi098 ceph-mon[31333]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:54:48.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:48.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:54:48.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:48.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:49.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.094+0000 7f86ecc62640 1 -- 172.21.15.98:0/699996447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e8107660 msgr2=0x7f86e8107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:49.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.094+0000 7f86ecc62640 1 --2- 172.21.15.98:0/699996447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e8107660 0x7f86e8107a40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009880 tx=0x7f86dc02a5d0 comp rx=0 tx=0).stop 2024-09-19T07:54:49.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.094+0000 7f86ecc62640 1 -- 172.21.15.98:0/699996447 shutdown_connections 2024-09-19T07:54:49.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.094+0000 7f86ecc62640 1 --2- 172.21.15.98:0/699996447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e8107660 0x7f86e8107a40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:49.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.094+0000 7f86ecc62640 1 -- 172.21.15.98:0/699996447 >> 172.21.15.98:0/699996447 conn(0x7f86e80fd270 msgr2=0x7f86e80ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:49.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.094+0000 7f86ecc62640 1 -- 172.21.15.98:0/699996447 shutdown_connections 2024-09-19T07:54:49.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.094+0000 7f86ecc62640 1 -- 172.21.15.98:0/699996447 wait complete. 2024-09-19T07:54:49.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.095+0000 7f86ecc62640 1 Processor -- start 2024-09-19T07:54:49.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.095+0000 7f86ecc62640 1 -- start start 2024-09-19T07:54:49.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.095+0000 7f86ecc62640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e8107660 0x7f86e8197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:49.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.095+0000 7f86ecc62640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86e810bc90 con 0x7f86e8107660 2024-09-19T07:54:49.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.095+0000 7f86e659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e8107660 0x7f86e8197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:49.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.095+0000 7f86e659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e8107660 0x7f86e8197130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44320/0 (socket says 172.21.15.98:44320) 2024-09-19T07:54:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.095+0000 7f86e659c640 1 -- 172.21.15.98:0/1568033969 learned_addr learned my addr 172.21.15.98:0/1568033969 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.095+0000 7f86e659c640 1 -- 172.21.15.98:0/1568033969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e8197670 con 0x7f86e8107660 2024-09-19T07:54:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.096+0000 7f86e659c640 1 --2- 172.21.15.98:0/1568033969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e8107660 0x7f86e8197130 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009230 tx=0x7f86dc003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.096+0000 7f86d77fe640 1 -- 172.21.15.98:0/1568033969 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86dc03f070 con 0x7f86e8107660 2024-09-19T07:54:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.096+0000 7f86ecc62640 1 -- 172.21.15.98:0/1568033969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e8197900 con 0x7f86e8107660 2024-09-19T07:54:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.096+0000 7f86d77fe640 1 -- 172.21.15.98:0/1568033969 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86dc03bc80 con 0x7f86e8107660 2024-09-19T07:54:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.096+0000 7f86d77fe640 1 -- 172.21.15.98:0/1568033969 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86dc035030 con 0x7f86e8107660 2024-09-19T07:54:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.096+0000 7f86ecc62640 1 -- 172.21.15.98:0/1568033969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e819ba00 con 0x7f86e8107660 2024-09-19T07:54:49.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.097+0000 7f86d77fe640 1 -- 172.21.15.98:0/1568033969 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f86dc042070 con 0x7f86e8107660 2024-09-19T07:54:49.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.097+0000 7f86ecc62640 1 -- 172.21.15.98:0/1568033969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86e8102cd0 con 0x7f86e8107660 2024-09-19T07:54:49.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.097+0000 7f86d77fe640 1 --2- 172.21.15.98:0/1568033969 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f86d0039300 0x7f86d003b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:49.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.097+0000 7f86d77fe640 1 -- 172.21.15.98:0/1568033969 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f86dc06cf60 con 0x7f86e8107660 2024-09-19T07:54:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.097+0000 7f86e5d9b640 1 --2- 172.21.15.98:0/1568033969 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f86d0039300 0x7f86d003b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.098+0000 7f86e5d9b640 1 --2- 172.21.15.98:0/1568033969 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f86d0039300 0x7f86d003b7c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f86d80098c0 tx=0x7f86d8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:49.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.102+0000 7f86d77fe640 1 -- 172.21.15.98:0/1568033969 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f86dc0314c0 con 0x7f86e8107660 2024-09-19T07:54:49.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.277+0000 7f86ecc62640 1 -- 172.21.15.98:0/1568033969 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f86e8198490 con 0x7f86d0039300 2024-09-19T07:54:49.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.278+0000 7f86d77fe640 1 -- 172.21.15.98:0/1568033969 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f86e8198490 con 0x7f86d0039300 2024-09-19T07:54:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:54:49.278 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.139", "hostname": "smithi139", "labels": [], "status": ""}, {"addr": "172.21.15.162", "hostname": "smithi162", "labels": [], "status": ""}] 2024-09-19T07:54:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.280+0000 7f86ecc62640 1 -- 172.21.15.98:0/1568033969 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f86d0039300 msgr2=0x7f86d003b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.280+0000 7f86ecc62640 1 --2- 172.21.15.98:0/1568033969 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f86d0039300 0x7f86d003b7c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f86d80098c0 tx=0x7f86d8006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.280+0000 7f86ecc62640 1 -- 172.21.15.98:0/1568033969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e8107660 msgr2=0x7f86e8197130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.280+0000 7f86ecc62640 1 --2- 172.21.15.98:0/1568033969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e8107660 0x7f86e8197130 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009230 tx=0x7f86dc003d40 comp rx=0 tx=0).stop 2024-09-19T07:54:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.281+0000 7f86ecc62640 1 -- 172.21.15.98:0/1568033969 shutdown_connections 2024-09-19T07:54:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.281+0000 7f86ecc62640 1 --2- 172.21.15.98:0/1568033969 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f86d0039300 0x7f86d003b7c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.281+0000 7f86ecc62640 1 --2- 172.21.15.98:0/1568033969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e8107660 0x7f86e8197130 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.281+0000 7f86ecc62640 1 -- 172.21.15.98:0/1568033969 >> 172.21.15.98:0/1568033969 conn(0x7f86e80fd270 msgr2=0x7f86e80fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.281+0000 7f86ecc62640 1 -- 172.21.15.98:0/1568033969 shutdown_connections 2024-09-19T07:54:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:49.281+0000 7f86ecc62640 1 -- 172.21.15.98:0/1568033969 wait complete. 2024-09-19T07:54:49.631 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:49 smithi098 ceph-mon[31333]: Added host smithi162 2024-09-19T07:54:49.632 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:49.934 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-19T07:54:49.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-19T07:54:50.179 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:54:50.535 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:50 smithi098 ceph-mon[31333]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T07:54:50.536 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:50 smithi098 ceph-mon[31333]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:54:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.617+0000 7f8ffe954640 1 -- 172.21.15.98:0/1176717795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8102790 msgr2=0x7f8ff8102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.617+0000 7f8ffe954640 1 --2- 172.21.15.98:0/1176717795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8102790 0x7f8ff8102b70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009810 tx=0x7f8fe802a540 comp rx=0 tx=0).stop 2024-09-19T07:54:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.617+0000 7f8ffe954640 1 -- 172.21.15.98:0/1176717795 shutdown_connections 2024-09-19T07:54:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.617+0000 7f8ffe954640 1 --2- 172.21.15.98:0/1176717795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8102790 0x7f8ff8102b70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.617+0000 7f8ffe954640 1 -- 172.21.15.98:0/1176717795 >> 172.21.15.98:0/1176717795 conn(0x7f8ff8077310 msgr2=0x7f8ff80fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.617+0000 7f8ffe954640 1 -- 172.21.15.98:0/1176717795 shutdown_connections 2024-09-19T07:54:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.617+0000 7f8ffe954640 1 -- 172.21.15.98:0/1176717795 wait complete. 2024-09-19T07:54:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.617+0000 7f8ffe954640 1 Processor -- start 2024-09-19T07:54:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.618+0000 7f8ffe954640 1 -- start start 2024-09-19T07:54:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.618+0000 7f8ffe954640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8102790 0x7f8ff81971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.618+0000 7f8ffe954640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff810bc80 con 0x7f8ff8102790 2024-09-19T07:54:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.618+0000 7f8ff7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8102790 0x7f8ff81971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.618+0000 7f8ff7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8102790 0x7f8ff81971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44328/0 (socket says 172.21.15.98:44328) 2024-09-19T07:54:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.618+0000 7f8ff7fff640 1 -- 172.21.15.98:0/1864013099 learned_addr learned my addr 172.21.15.98:0/1864013099 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:54:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.618+0000 7f8ff7fff640 1 -- 172.21.15.98:0/1864013099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff8197720 con 0x7f8ff8102790 2024-09-19T07:54:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.618+0000 7f8ff7fff640 1 --2- 172.21.15.98:0/1864013099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8102790 0x7f8ff81971e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8fe80045e0 tx=0x7f8fe8004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.618+0000 7f8ff5ffb640 1 -- 172.21.15.98:0/1864013099 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fe8030e70 con 0x7f8ff8102790 2024-09-19T07:54:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.618+0000 7f8ff5ffb640 1 -- 172.21.15.98:0/1864013099 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fe80318e0 con 0x7f8ff8102790 2024-09-19T07:54:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.618+0000 7f8ffe954640 1 -- 172.21.15.98:0/1864013099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff81979b0 con 0x7f8ff8102790 2024-09-19T07:54:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.619+0000 7f8ff5ffb640 1 -- 172.21.15.98:0/1864013099 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fe80049c0 con 0x7f8ff8102790 2024-09-19T07:54:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.619+0000 7f8ffe954640 1 -- 172.21.15.98:0/1864013099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff8197e10 con 0x7f8ff8102790 2024-09-19T07:54:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.620+0000 7f8ffe954640 1 -- 172.21.15.98:0/1864013099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff8069f30 con 0x7f8ff8102790 2024-09-19T07:54:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.620+0000 7f8ff5ffb640 1 -- 172.21.15.98:0/1864013099 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f8fe8039040 con 0x7f8ff8102790 2024-09-19T07:54:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.620+0000 7f8ff5ffb640 1 --2- 172.21.15.98:0/1864013099 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f8fe4039300 0x7f8fe403b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.620+0000 7f8ff5ffb640 1 -- 172.21.15.98:0/1864013099 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8fe806d560 con 0x7f8ff8102790 2024-09-19T07:54:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.620+0000 7f8feffff640 1 --2- 172.21.15.98:0/1864013099 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f8fe4039300 0x7f8fe403b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.621+0000 7f8feffff640 1 --2- 172.21.15.98:0/1864013099 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f8fe4039300 0x7f8fe403b7c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00098c0 tx=0x7f8fe0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.625+0000 7f8ff5ffb640 1 -- 172.21.15.98:0/1864013099 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f8fe8040030 con 0x7f8ff8102790 2024-09-19T07:54:50.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:50.793+0000 7f8ffe954640 1 -- 172.21.15.98:0/1864013099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f8ff8198540 con 0x7f8ff8102790 2024-09-19T07:54:51.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:51.487+0000 7f8ff5ffb640 1 -- 172.21.15.98:0/1864013099 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f8fe803b3b0 con 0x7f8ff8102790 2024-09-19T07:54:51.487 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-09-19T07:54:51.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:51.489+0000 7f8ffe954640 1 -- 172.21.15.98:0/1864013099 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f8fe4039300 msgr2=0x7f8fe403b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:51.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:51.489+0000 7f8ffe954640 1 --2- 172.21.15.98:0/1864013099 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f8fe4039300 0x7f8fe403b7c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00098c0 tx=0x7f8fe0006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:51.489+0000 7f8ffe954640 1 -- 172.21.15.98:0/1864013099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8102790 msgr2=0x7f8ff81971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:51.489+0000 7f8ffe954640 1 --2- 172.21.15.98:0/1864013099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8102790 0x7f8ff81971e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8fe80045e0 tx=0x7f8fe8004610 comp rx=0 tx=0).stop 2024-09-19T07:54:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:51.490+0000 7f8ffe954640 1 -- 172.21.15.98:0/1864013099 shutdown_connections 2024-09-19T07:54:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:51.490+0000 7f8ffe954640 1 --2- 172.21.15.98:0/1864013099 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f8fe4039300 0x7f8fe403b7c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:51.490+0000 7f8ffe954640 1 --2- 172.21.15.98:0/1864013099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8102790 0x7f8ff81971e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:51.490+0000 7f8ffe954640 1 -- 172.21.15.98:0/1864013099 >> 172.21.15.98:0/1864013099 conn(0x7f8ff8077310 msgr2=0x7f8ff8103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:51.490+0000 7f8ffe954640 1 -- 172.21.15.98:0/1864013099 shutdown_connections 2024-09-19T07:54:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:54:51.490+0000 7f8ffe954640 1 -- 172.21.15.98:0/1864013099 wait complete. 2024-09-19T07:54:51.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:51 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1864013099' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-19T07:54:52.088 INFO:tasks.cephadm:Adding mon.a on smithi098 2024-09-19T07:54:52.089 INFO:tasks.cephadm:Adding mon.b on smithi139 2024-09-19T07:54:52.089 INFO:tasks.cephadm:Adding mon.c on smithi162 2024-09-19T07:54:52.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi098:172.21.15.98=a;smithi139:172.21.15.139=b;smithi162:172.21.15.162=c' 2024-09-19T07:54:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:54:52.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.834+0000 7f471f05e640 1 -- 172.21.15.162:0/2387622676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47181097c0 msgr2=0x7f4718109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:52.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.834+0000 7f471f05e640 1 --2- 172.21.15.162:0/2387622676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47181097c0 0x7f4718109ba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f470c009880 tx=0x7f470c02a5d0 comp rx=0 tx=0).stop 2024-09-19T07:54:52.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.834+0000 7f471f05e640 1 -- 172.21.15.162:0/2387622676 shutdown_connections 2024-09-19T07:54:52.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.834+0000 7f471f05e640 1 --2- 172.21.15.162:0/2387622676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47181097c0 0x7f4718109ba0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:52.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.834+0000 7f471f05e640 1 -- 172.21.15.162:0/2387622676 >> 172.21.15.162:0/2387622676 conn(0x7f47180ff480 msgr2=0x7f47181018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:52.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.834+0000 7f471f05e640 1 -- 172.21.15.162:0/2387622676 shutdown_connections 2024-09-19T07:54:52.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.835+0000 7f471f05e640 1 -- 172.21.15.162:0/2387622676 wait complete. 2024-09-19T07:54:52.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.835+0000 7f471f05e640 1 Processor -- start 2024-09-19T07:54:52.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.835+0000 7f471f05e640 1 -- start start 2024-09-19T07:54:52.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.835+0000 7f471f05e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47181097c0 0x7f4718199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:52.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.835+0000 7f471f05e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f471810de80 con 0x7f47181097c0 2024-09-19T07:54:52.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.836+0000 7f471cdfa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47181097c0 0x7f4718199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:52.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.836+0000 7f471cdfa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47181097c0 0x7f4718199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.162:49716/0 (socket says 172.21.15.162:49716) 2024-09-19T07:54:52.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.836+0000 7f471cdfa640 1 -- 172.21.15.162:0/461131695 learned_addr learned my addr 172.21.15.162:0/461131695 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-19T07:54:52.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.836+0000 7f471cdfa640 1 -- 172.21.15.162:0/461131695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4718199860 con 0x7f47181097c0 2024-09-19T07:54:52.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.836+0000 7f471cdfa640 1 --2- 172.21.15.162:0/461131695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47181097c0 0x7f4718199320 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f470c009230 tx=0x7f470c003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:52.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.836+0000 7f4715ffb640 1 -- 172.21.15.162:0/461131695 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f470c03f070 con 0x7f47181097c0 2024-09-19T07:54:52.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.836+0000 7f4715ffb640 1 -- 172.21.15.162:0/461131695 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f470c03bc80 con 0x7f47181097c0 2024-09-19T07:54:52.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.836+0000 7f471f05e640 1 -- 172.21.15.162:0/461131695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4718199b50 con 0x7f47181097c0 2024-09-19T07:54:52.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.836+0000 7f4715ffb640 1 -- 172.21.15.162:0/461131695 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f470c035030 con 0x7f47181097c0 2024-09-19T07:54:52.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.837+0000 7f471f05e640 1 -- 172.21.15.162:0/461131695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f471819dbf0 con 0x7f47181097c0 2024-09-19T07:54:52.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.838+0000 7f4715ffb640 1 -- 172.21.15.162:0/461131695 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f470c042070 con 0x7f47181097c0 2024-09-19T07:54:52.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.838+0000 7f4715ffb640 1 --2- 172.21.15.162:0/461131695 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f47080392a0 0x7f470803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:52.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.838+0000 7f471f05e640 1 -- 172.21.15.162:0/461131695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4718104f00 con 0x7f47181097c0 2024-09-19T07:54:52.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.838+0000 7f4715ffb640 1 -- 172.21.15.162:0/461131695 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f470c06cfb0 con 0x7f47181097c0 2024-09-19T07:54:52.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.838+0000 7f4717fff640 1 --2- 172.21.15.162:0/461131695 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f47080392a0 0x7f470803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:52.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.839+0000 7f4717fff640 1 --2- 172.21.15.162:0/461131695 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f47080392a0 0x7f470803b760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f47040098c0 tx=0x7f4704006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:52.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:52.843+0000 7f4715ffb640 1 -- 172.21.15.162:0/461131695 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f470c04c350 con 0x7f47181097c0 2024-09-19T07:54:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:52 smithi098 ceph-mon[31333]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:54:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:52 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1864013099' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-19T07:54:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:52 smithi098 ceph-mon[31333]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T07:54:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:52.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:54:52.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:54:52.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:54:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:53.026+0000 7f471f05e640 1 -- 172.21.15.162:0/461131695 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi139:172.21.15.139=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}) -- 0x7f471819a680 con 0x7f47080392a0 2024-09-19T07:54:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:53.031+0000 7f4715ffb640 1 -- 172.21.15.162:0/461131695 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f471819a680 con 0x7f47080392a0 2024-09-19T07:54:53.031 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mon update... 2024-09-19T07:54:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:53.033+0000 7f471f05e640 1 -- 172.21.15.162:0/461131695 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f47080392a0 msgr2=0x7f470803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:53.033+0000 7f471f05e640 1 --2- 172.21.15.162:0/461131695 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f47080392a0 0x7f470803b760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f47040098c0 tx=0x7f4704006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:53.033+0000 7f471f05e640 1 -- 172.21.15.162:0/461131695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47181097c0 msgr2=0x7f4718199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:53.033+0000 7f471f05e640 1 --2- 172.21.15.162:0/461131695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47181097c0 0x7f4718199320 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f470c009230 tx=0x7f470c003d40 comp rx=0 tx=0).stop 2024-09-19T07:54:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:53.033+0000 7f471f05e640 1 -- 172.21.15.162:0/461131695 shutdown_connections 2024-09-19T07:54:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:53.033+0000 7f471f05e640 1 --2- 172.21.15.162:0/461131695 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f47080392a0 0x7f470803b760 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:53.033+0000 7f471f05e640 1 --2- 172.21.15.162:0/461131695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47181097c0 0x7f4718199320 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:53.033+0000 7f471f05e640 1 -- 172.21.15.162:0/461131695 >> 172.21.15.162:0/461131695 conn(0x7f47180ff480 msgr2=0x7f4718100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:53.033+0000 7f471f05e640 1 -- 172.21.15.162:0/461131695 shutdown_connections 2024-09-19T07:54:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:53.033+0000 7f471f05e640 1 -- 172.21.15.162:0/461131695 wait complete. 2024-09-19T07:54:53.735 DEBUG:teuthology.orchestra.run.smithi139:mon.b> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mon.b.service 2024-09-19T07:54:53.738 DEBUG:teuthology.orchestra.run.smithi162:mon.c> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mon.c.service 2024-09-19T07:54:53.741 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-19T07:54:53.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-19T07:54:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:53 smithi098 ceph-mon[31333]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-19T07:54:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:53 smithi098 ceph-mon[31333]: Updating smithi162:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:54:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:53 smithi098 ceph-mon[31333]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-19T07:54:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:53 smithi098 ceph-mon[31333]: Updating smithi162:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T07:54:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:53.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:53.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:53.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:54:53.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:54:53.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:54:53.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:53.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T07:54:53.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:54:54.428 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:54:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.817+0000 7f2d60f4d640 1 -- 172.21.15.162:0/3486652644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5c106560 msgr2=0x7f2d5c106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.817+0000 7f2d60f4d640 1 --2- 172.21.15.162:0/3486652644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5c106560 0x7f2d5c106940 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c009c20 tx=0x7f2d4c02aa90 comp rx=0 tx=0).stop 2024-09-19T07:54:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.817+0000 7f2d60f4d640 1 -- 172.21.15.162:0/3486652644 shutdown_connections 2024-09-19T07:54:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.817+0000 7f2d60f4d640 1 --2- 172.21.15.162:0/3486652644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5c106560 0x7f2d5c106940 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.817+0000 7f2d60f4d640 1 -- 172.21.15.162:0/3486652644 >> 172.21.15.162:0/3486652644 conn(0x7f2d5c0fc1b0 msgr2=0x7f2d5c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.817+0000 7f2d60f4d640 1 -- 172.21.15.162:0/3486652644 shutdown_connections 2024-09-19T07:54:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.817+0000 7f2d60f4d640 1 -- 172.21.15.162:0/3486652644 wait complete. 2024-09-19T07:54:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.818+0000 7f2d60f4d640 1 Processor -- start 2024-09-19T07:54:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.818+0000 7f2d60f4d640 1 -- start start 2024-09-19T07:54:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.818+0000 7f2d60f4d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5c106560 0x7f2d5c0fc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.818+0000 7f2d60f4d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5c10ab90 con 0x7f2d5c106560 2024-09-19T07:54:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.819+0000 7f2d5a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5c106560 0x7f2d5c0fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.819+0000 7f2d5a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5c106560 0x7f2d5c0fc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.162:49742/0 (socket says 172.21.15.162:49742) 2024-09-19T07:54:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.819+0000 7f2d5a59c640 1 -- 172.21.15.162:0/2360677075 learned_addr learned my addr 172.21.15.162:0/2360677075 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-19T07:54:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.819+0000 7f2d5a59c640 1 -- 172.21.15.162:0/2360677075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d5c0fe550 con 0x7f2d5c106560 2024-09-19T07:54:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.819+0000 7f2d5a59c640 1 --2- 172.21.15.162:0/2360677075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5c106560 0x7f2d5c0fc660 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c005e30 tx=0x7f2d4c02fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.819+0000 7f2d4b7fe640 1 -- 172.21.15.162:0/2360677075 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d4c047070 con 0x7f2d5c106560 2024-09-19T07:54:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.819+0000 7f2d60f4d640 1 -- 172.21.15.162:0/2360677075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d5c0fcba0 con 0x7f2d5c106560 2024-09-19T07:54:54.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.820+0000 7f2d4b7fe640 1 -- 172.21.15.162:0/2360677075 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d4c003df0 con 0x7f2d5c106560 2024-09-19T07:54:54.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.820+0000 7f2d4b7fe640 1 -- 172.21.15.162:0/2360677075 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d4c033050 con 0x7f2d5c106560 2024-09-19T07:54:54.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.820+0000 7f2d60f4d640 1 -- 172.21.15.162:0/2360677075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d5c0fd060 con 0x7f2d5c106560 2024-09-19T07:54:54.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.820+0000 7f2d4b7fe640 1 -- 172.21.15.162:0/2360677075 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f2d4c04a070 con 0x7f2d5c106560 2024-09-19T07:54:54.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.820+0000 7f2d60f4d640 1 -- 172.21.15.162:0/2360677075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d5c101bd0 con 0x7f2d5c106560 2024-09-19T07:54:54.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.821+0000 7f2d4b7fe640 1 --2- 172.21.15.162:0/2360677075 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2d440392a0 0x7f2d4403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:54.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.821+0000 7f2d4b7fe640 1 -- 172.21.15.162:0/2360677075 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2d4c03e030 con 0x7f2d5c106560 2024-09-19T07:54:54.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.821+0000 7f2d59d9b640 1 --2- 172.21.15.162:0/2360677075 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2d440392a0 0x7f2d4403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:54.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.821+0000 7f2d59d9b640 1 --2- 172.21.15.162:0/2360677075 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2d440392a0 0x7f2d4403b760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2d500098c0 tx=0x7f2d50006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:54.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:54.826+0000 7f2d4b7fe640 1 -- 172.21.15.162:0/2360677075 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2d4c04a360 con 0x7f2d5c106560 2024-09-19T07:54:54.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:54 smithi098 ceph-mon[31333]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi139:172.21.15.139=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:54:54.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:54 smithi098 ceph-mon[31333]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi139:172.21.15.139=b;smithi162:172.21.15.162=c;count:3 2024-09-19T07:54:54.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:54 smithi098 ceph-mon[31333]: Deploying daemon mon.c on smithi162 2024-09-19T07:54:54.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:54 smithi098 ceph-mon[31333]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:54:55.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:55.069+0000 7f2d60f4d640 1 -- 172.21.15.162:0/2360677075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f2d5c0fdb90 con 0x7f2d5c106560 2024-09-19T07:54:55.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:55.070+0000 7f2d4b7fe640 1 -- 172.21.15.162:0/2360677075 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f2d4c051300 con 0x7f2d5c106560 2024-09-19T07:54:55.070 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:54:55.070 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":1,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","modified":"2024-09-19T07:53:35.879566Z","created":"2024-09-19T07:53:35.879566Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-19T07:54:55.070 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 1 2024-09-19T07:54:55.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:55.072+0000 7f2d60f4d640 1 -- 172.21.15.162:0/2360677075 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2d440392a0 msgr2=0x7f2d4403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:55.072+0000 7f2d60f4d640 1 --2- 172.21.15.162:0/2360677075 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2d440392a0 0x7f2d4403b760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2d500098c0 tx=0x7f2d50006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:55.072+0000 7f2d60f4d640 1 -- 172.21.15.162:0/2360677075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5c106560 msgr2=0x7f2d5c0fc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:55.072+0000 7f2d60f4d640 1 --2- 172.21.15.162:0/2360677075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5c106560 0x7f2d5c0fc660 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c005e30 tx=0x7f2d4c02fc10 comp rx=0 tx=0).stop 2024-09-19T07:54:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:55.072+0000 7f2d60f4d640 1 -- 172.21.15.162:0/2360677075 shutdown_connections 2024-09-19T07:54:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:55.072+0000 7f2d60f4d640 1 --2- 172.21.15.162:0/2360677075 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2d440392a0 0x7f2d4403b760 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:55.072+0000 7f2d60f4d640 1 --2- 172.21.15.162:0/2360677075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5c106560 0x7f2d5c0fc660 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:55.072+0000 7f2d60f4d640 1 -- 172.21.15.162:0/2360677075 >> 172.21.15.162:0/2360677075 conn(0x7f2d5c0fc1b0 msgr2=0x7f2d5c069ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:55.072+0000 7f2d60f4d640 1 -- 172.21.15.162:0/2360677075 shutdown_connections 2024-09-19T07:54:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:55.072+0000 7f2d60f4d640 1 -- 172.21.15.162:0/2360677075 wait complete. 2024-09-19T07:54:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:55 smithi098 ceph-mon[31333]: from='client.? 172.21.15.162:0/2360677075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T07:54:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:56 smithi098 ceph-mon[31333]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:54:57.032 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-19T07:54:57.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-19T07:54:57.417 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T07:54:58.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.001+0000 7fc74b160640 1 -- 172.21.15.162:0/2111549152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc744106650 msgr2=0x7fc744106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:58.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.001+0000 7fc74b160640 1 --2- 172.21.15.162:0/2111549152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc744106650 0x7fc744106a30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc738009880 tx=0x7fc73802a5d0 comp rx=0 tx=0).stop 2024-09-19T07:54:58.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.002+0000 7fc74b160640 1 -- 172.21.15.162:0/2111549152 shutdown_connections 2024-09-19T07:54:58.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.002+0000 7fc74b160640 1 --2- 172.21.15.162:0/2111549152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc744106650 0x7fc744106a30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:58.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.002+0000 7fc74b160640 1 -- 172.21.15.162:0/2111549152 >> 172.21.15.162:0/2111549152 conn(0x7fc7440fc280 msgr2=0x7fc7440fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:58.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.002+0000 7fc74b160640 1 -- 172.21.15.162:0/2111549152 shutdown_connections 2024-09-19T07:54:58.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.002+0000 7fc74b160640 1 -- 172.21.15.162:0/2111549152 wait complete. 2024-09-19T07:54:58.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.002+0000 7fc74b160640 1 Processor -- start 2024-09-19T07:54:58.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.002+0000 7fc74b160640 1 -- start start 2024-09-19T07:54:58.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.003+0000 7fc74b160640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc744106650 0x7fc744196120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:58.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.003+0000 7fc74b160640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc74410ac80 con 0x7fc744106650 2024-09-19T07:54:58.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.003+0000 7fc748efc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc744106650 0x7fc744196120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:58.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.003+0000 7fc748efc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc744106650 0x7fc744196120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.162:49760/0 (socket says 172.21.15.162:49760) 2024-09-19T07:54:58.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.003+0000 7fc748efc640 1 -- 172.21.15.162:0/34253752 learned_addr learned my addr 172.21.15.162:0/34253752 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-19T07:54:58.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.003+0000 7fc748efc640 1 -- 172.21.15.162:0/34253752 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc744196660 con 0x7fc744106650 2024-09-19T07:54:58.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.004+0000 7fc748efc640 1 --2- 172.21.15.162:0/34253752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc744106650 0x7fc744196120 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc738009230 tx=0x7fc738003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:58.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.004+0000 7fc741ffb640 1 -- 172.21.15.162:0/34253752 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc73803f070 con 0x7fc744106650 2024-09-19T07:54:58.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.004+0000 7fc74b160640 1 -- 172.21.15.162:0/34253752 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7441968f0 con 0x7fc744106650 2024-09-19T07:54:58.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.004+0000 7fc741ffb640 1 -- 172.21.15.162:0/34253752 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc73803bc80 con 0x7fc744106650 2024-09-19T07:54:58.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.004+0000 7fc741ffb640 1 -- 172.21.15.162:0/34253752 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc738035030 con 0x7fc744106650 2024-09-19T07:54:58.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.004+0000 7fc74b160640 1 -- 172.21.15.162:0/34253752 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc74419a9f0 con 0x7fc744106650 2024-09-19T07:54:58.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.005+0000 7fc741ffb640 1 -- 172.21.15.162:0/34253752 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fc738042070 con 0x7fc744106650 2024-09-19T07:54:58.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.005+0000 7fc74b160640 1 -- 172.21.15.162:0/34253752 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc744101cc0 con 0x7fc744106650 2024-09-19T07:54:58.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.005+0000 7fc741ffb640 1 --2- 172.21.15.162:0/34253752 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc7340392a0 0x7fc73403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:54:58.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.005+0000 7fc741ffb640 1 -- 172.21.15.162:0/34253752 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc73806d0f0 con 0x7fc744106650 2024-09-19T07:54:58.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.005+0000 7fc743fff640 1 --2- 172.21.15.162:0/34253752 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc7340392a0 0x7fc73403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:54:58.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.006+0000 7fc743fff640 1 --2- 172.21.15.162:0/34253752 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc7340392a0 0x7fc73403b760 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc7300098c0 tx=0x7fc730006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:54:58.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.010+0000 7fc741ffb640 1 -- 172.21.15.162:0/34253752 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fc7380372d0 con 0x7fc744106650 2024-09-19T07:54:58.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.278+0000 7fc74b160640 1 -- 172.21.15.162:0/34253752 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc744197480 con 0x7fc744106650 2024-09-19T07:54:58.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.278+0000 7fc741ffb640 1 -- 172.21.15.162:0/34253752 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fc738031570 con 0x7fc744106650 2024-09-19T07:54:58.280 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:54:58.280 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":1,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","modified":"2024-09-19T07:53:35.879566Z","created":"2024-09-19T07:53:35.879566Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-19T07:54:58.281 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 1 2024-09-19T07:54:58.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.280+0000 7fc74b160640 1 -- 172.21.15.162:0/34253752 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc7340392a0 msgr2=0x7fc73403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:58.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.280+0000 7fc74b160640 1 --2- 172.21.15.162:0/34253752 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc7340392a0 0x7fc73403b760 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc7300098c0 tx=0x7fc730006db0 comp rx=0 tx=0).stop 2024-09-19T07:54:58.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.280+0000 7fc74b160640 1 -- 172.21.15.162:0/34253752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc744106650 msgr2=0x7fc744196120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:54:58.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.280+0000 7fc74b160640 1 --2- 172.21.15.162:0/34253752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc744106650 0x7fc744196120 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc738009230 tx=0x7fc738003d40 comp rx=0 tx=0).stop 2024-09-19T07:54:58.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.280+0000 7fc74b160640 1 -- 172.21.15.162:0/34253752 shutdown_connections 2024-09-19T07:54:58.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.280+0000 7fc74b160640 1 --2- 172.21.15.162:0/34253752 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc7340392a0 0x7fc73403b760 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:58.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.280+0000 7fc74b160640 1 --2- 172.21.15.162:0/34253752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc744106650 0x7fc744196120 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:54:58.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.280+0000 7fc74b160640 1 -- 172.21.15.162:0/34253752 >> 172.21.15.162:0/34253752 conn(0x7fc7440fc280 msgr2=0x7fc7440fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:54:58.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.280+0000 7fc74b160640 1 -- 172.21.15.162:0/34253752 shutdown_connections 2024-09-19T07:54:58.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:54:58.281+0000 7fc74b160640 1 -- 172.21.15.162:0/34253752 wait complete. 2024-09-19T07:54:58.552 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 podman[35229]: 2024-09-19 07:54:58.551812172 +0000 UTC m=+0.438182929 container init 500b1c467f929a37e7f3fa083bd2c476bd8dd7588bbeaea2c706ce9f6b545123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS) 2024-09-19T07:54:58.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 podman[35229]: 2024-09-19 07:54:58.555770082 +0000 UTC m=+0.442140837 container start 500b1c467f929a37e7f3fa083bd2c476bd8dd7588bbeaea2c706ce9f6b545123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-19T07:54:58.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: set uid:gid to 167:167 (ceph:ceph) 2024-09-19T07:54:58.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable), process ceph-mon, pid 2 2024-09-19T07:54:58.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: pidfile_write: ignore empty --pid-file 2024-09-19T07:54:58.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: load: jerasure load: lrc 2024-09-19T07:54:58.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: RocksDB version: 7.9.2 2024-09-19T07:54:58.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Git sha 0 2024-09-19T07:54:58.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Compile date 2024-09-18 16:29:48 2024-09-19T07:54:58.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: DB SUMMARY 2024-09-19T07:54:58.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: DB Session ID: DZI87HCZKUO298PVR9OP 2024-09-19T07:54:58.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: CURRENT file: CURRENT 2024-09-19T07:54:58.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: IDENTITY file: IDENTITY 2024-09-19T07:54:58.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-19T07:54:58.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-19T07:54:58.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-09-19T07:54:58.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.error_if_exists: 0 2024-09-19T07:54:58.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.create_if_missing: 0 2024-09-19T07:54:58.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.paranoid_checks: 1 2024-09-19T07:54:58.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-19T07:54:58.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-19T07:54:58.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-19T07:54:58.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.env: 0x55aba577f940 2024-09-19T07:54:58.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.fs: PosixFileSystem 2024-09-19T07:54:58.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.info_log: 0x55aba658b6e0 2024-09-19T07:54:58.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_file_opening_threads: 16 2024-09-19T07:54:58.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.statistics: (nil) 2024-09-19T07:54:58.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.use_fsync: 0 2024-09-19T07:54:58.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_log_file_size: 0 2024-09-19T07:54:58.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-19T07:54:58.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-19T07:54:58.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.keep_log_file_num: 1000 2024-09-19T07:54:58.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.recycle_log_file_num: 0 2024-09-19T07:54:58.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.allow_fallocate: 1 2024-09-19T07:54:58.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.allow_mmap_reads: 0 2024-09-19T07:54:58.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.allow_mmap_writes: 0 2024-09-19T07:54:58.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.use_direct_reads: 0 2024-09-19T07:54:58.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-19T07:54:58.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.create_missing_column_families: 0 2024-09-19T07:54:58.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.db_log_dir: 2024-09-19T07:54:58.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.wal_dir: 2024-09-19T07:54:58.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-19T07:54:58.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-19T07:54:58.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-19T07:54:58.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-19T07:54:58.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-19T07:54:58.876 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-19T07:54:58.876 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.advise_random_on_open: 1 2024-09-19T07:54:58.876 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.db_write_buffer_size: 0 2024-09-19T07:54:58.876 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.write_buffer_manager: 0x55aba659adc0 2024-09-19T07:54:58.876 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-19T07:54:58.876 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-19T07:54:58.876 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-19T07:54:58.876 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.rate_limiter: (nil) 2024-09-19T07:54:58.876 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-19T07:54:58.876 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.wal_recovery_mode: 2 2024-09-19T07:54:58.876 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.enable_thread_tracking: 0 2024-09-19T07:54:58.876 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.enable_pipelined_write: 0 2024-09-19T07:54:58.876 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.unordered_write: 0 2024-09-19T07:54:58.877 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-19T07:54:58.877 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-19T07:54:58.877 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-19T07:54:58.877 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-19T07:54:58.877 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.row_cache: None 2024-09-19T07:54:58.877 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.wal_filter: None 2024-09-19T07:54:58.877 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-19T07:54:58.877 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.allow_ingest_behind: 0 2024-09-19T07:54:58.877 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.two_write_queues: 0 2024-09-19T07:54:58.877 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.manual_wal_flush: 0 2024-09-19T07:54:58.878 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.wal_compression: 0 2024-09-19T07:54:58.878 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.atomic_flush: 0 2024-09-19T07:54:58.878 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-19T07:54:58.878 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-19T07:54:58.878 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-19T07:54:58.878 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.log_readahead_size: 0 2024-09-19T07:54:58.878 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-19T07:54:58.878 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.best_efforts_recovery: 0 2024-09-19T07:54:58.878 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-19T07:54:58.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-19T07:54:58.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.allow_data_in_errors: 0 2024-09-19T07:54:58.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.db_host_id: __hostname__ 2024-09-19T07:54:58.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-19T07:54:58.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_background_jobs: 2 2024-09-19T07:54:58.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_background_compactions: -1 2024-09-19T07:54:58.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_subcompactions: 1 2024-09-19T07:54:58.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-19T07:54:58.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-19T07:54:58.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-19T07:54:58.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_total_wal_size: 0 2024-09-19T07:54:58.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-19T07:54:58.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-19T07:54:58.881 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-19T07:54:58.881 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-19T07:54:58.881 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_open_files: -1 2024-09-19T07:54:58.881 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.bytes_per_sync: 0 2024-09-19T07:54:58.881 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-19T07:54:58.881 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-19T07:54:58.882 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compaction_readahead_size: 0 2024-09-19T07:54:58.882 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_background_flushes: -1 2024-09-19T07:54:58.882 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Compression algorithms supported: 2024-09-19T07:54:58.882 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: kZSTD supported: 0 2024-09-19T07:54:58.882 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: kXpressCompression supported: 0 2024-09-19T07:54:58.882 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: kBZip2Compression supported: 0 2024-09-19T07:54:58.882 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-19T07:54:58.883 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: kLZ4Compression supported: 1 2024-09-19T07:54:58.883 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: kZlibCompression supported: 1 2024-09-19T07:54:58.883 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: kLZ4HCCompression supported: 1 2024-09-19T07:54:58.883 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: kSnappyCompression supported: 1 2024-09-19T07:54:58.883 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-19T07:54:58.883 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-19T07:54:58.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-19T07:54:58.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-19T07:54:58.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-19T07:54:58.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.merge_operator: 2024-09-19T07:54:58.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compaction_filter: None 2024-09-19T07:54:58.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compaction_filter_factory: None 2024-09-19T07:54:58.885 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.sst_partitioner_factory: None 2024-09-19T07:54:58.885 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-19T07:54:58.885 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-19T07:54:58.885 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55aba658b320) 2024-09-19T07:54:58.885 INFO:journalctl@ceph.mon.c.smithi162.stdout: cache_index_and_filter_blocks: 1 2024-09-19T07:54:58.885 INFO:journalctl@ceph.mon.c.smithi162.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-19T07:54:58.886 INFO:journalctl@ceph.mon.c.smithi162.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-19T07:54:58.886 INFO:journalctl@ceph.mon.c.smithi162.stdout: pin_top_level_index_and_filter: 1 2024-09-19T07:54:58.886 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_type: 0 2024-09-19T07:54:58.886 INFO:journalctl@ceph.mon.c.smithi162.stdout: data_block_index_type: 0 2024-09-19T07:54:58.886 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_shortening: 1 2024-09-19T07:54:58.886 INFO:journalctl@ceph.mon.c.smithi162.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-19T07:54:58.887 INFO:journalctl@ceph.mon.c.smithi162.stdout: checksum: 4 2024-09-19T07:54:58.887 INFO:journalctl@ceph.mon.c.smithi162.stdout: no_block_cache: 0 2024-09-19T07:54:58.887 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache: 0x55aba657f1f0 2024-09-19T07:54:58.887 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_name: BinnedLRUCache 2024-09-19T07:54:58.887 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_options: 2024-09-19T07:54:58.887 INFO:journalctl@ceph.mon.c.smithi162.stdout: capacity : 536870912 2024-09-19T07:54:58.888 INFO:journalctl@ceph.mon.c.smithi162.stdout: num_shard_bits : 4 2024-09-19T07:54:58.888 INFO:journalctl@ceph.mon.c.smithi162.stdout: strict_capacity_limit : 0 2024-09-19T07:54:58.888 INFO:journalctl@ceph.mon.c.smithi162.stdout: high_pri_pool_ratio: 0.000 2024-09-19T07:54:58.888 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_compressed: (nil) 2024-09-19T07:54:58.888 INFO:journalctl@ceph.mon.c.smithi162.stdout: persistent_cache: (nil) 2024-09-19T07:54:58.889 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_size: 4096 2024-09-19T07:54:58.889 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_size_deviation: 10 2024-09-19T07:54:58.889 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_restart_interval: 16 2024-09-19T07:54:58.889 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_block_restart_interval: 1 2024-09-19T07:54:58.889 INFO:journalctl@ceph.mon.c.smithi162.stdout: metadata_block_size: 4096 2024-09-19T07:54:58.889 INFO:journalctl@ceph.mon.c.smithi162.stdout: partition_filters: 0 2024-09-19T07:54:58.890 INFO:journalctl@ceph.mon.c.smithi162.stdout: use_delta_encoding: 1 2024-09-19T07:54:58.890 INFO:journalctl@ceph.mon.c.smithi162.stdout: filter_policy: bloomfilter 2024-09-19T07:54:58.890 INFO:journalctl@ceph.mon.c.smithi162.stdout: whole_key_filtering: 1 2024-09-19T07:54:58.890 INFO:journalctl@ceph.mon.c.smithi162.stdout: verify_compression: 0 2024-09-19T07:54:58.890 INFO:journalctl@ceph.mon.c.smithi162.stdout: read_amp_bytes_per_bit: 0 2024-09-19T07:54:58.890 INFO:journalctl@ceph.mon.c.smithi162.stdout: format_version: 5 2024-09-19T07:54:58.891 INFO:journalctl@ceph.mon.c.smithi162.stdout: enable_index_compression: 1 2024-09-19T07:54:58.891 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_align: 0 2024-09-19T07:54:58.891 INFO:journalctl@ceph.mon.c.smithi162.stdout: max_auto_readahead_size: 262144 2024-09-19T07:54:58.891 INFO:journalctl@ceph.mon.c.smithi162.stdout: prepopulate_block_cache: 0 2024-09-19T07:54:58.891 INFO:journalctl@ceph.mon.c.smithi162.stdout: initial_auto_readahead_size: 8192 2024-09-19T07:54:58.892 INFO:journalctl@ceph.mon.c.smithi162.stdout: num_file_reads_for_auto_readahead: 2 2024-09-19T07:54:58.892 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.write_buffer_size: 33554432 2024-09-19T07:54:58.892 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_write_buffer_number: 2 2024-09-19T07:54:58.892 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compression: NoCompression 2024-09-19T07:54:58.892 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.bottommost_compression: Disabled 2024-09-19T07:54:58.892 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.prefix_extractor: nullptr 2024-09-19T07:54:58.893 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-19T07:54:58.893 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.num_levels: 7 2024-09-19T07:54:58.893 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-19T07:54:58.893 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-19T07:54:58.893 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-19T07:54:58.893 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-19T07:54:58.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-19T07:54:58.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-19T07:54:58.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-19T07:54:58.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-19T07:54:58.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-19T07:54:58.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-19T07:54:58.895 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-19T07:54:58.895 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-19T07:54:58.895 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-19T07:54:58.895 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compression_opts.level: 32767 2024-09-19T07:54:58.895 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compression_opts.strategy: 0 2024-09-19T07:54:58.895 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-19T07:54:58.896 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-19T07:54:58.896 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-19T07:54:58.896 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-19T07:54:58.896 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compression_opts.enabled: false 2024-09-19T07:54:58.896 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-19T07:54:58.896 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-19T07:54:58.897 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-19T07:54:58.897 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-19T07:54:58.897 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.target_file_size_base: 67108864 2024-09-19T07:54:58.897 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-19T07:54:58.897 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-19T07:54:58.897 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-19T07:54:58.898 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-19T07:54:58.898 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-19T07:54:58.898 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-19T07:54:58.898 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-19T07:54:58.898 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-19T07:54:58.898 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-19T07:54:58.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-19T07:54:58.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-19T07:54:58.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-19T07:54:58.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-19T07:54:58.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-19T07:54:58.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.arena_block_size: 1048576 2024-09-19T07:54:58.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-19T07:54:58.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-19T07:54:58.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.disable_auto_compactions: 0 2024-09-19T07:54:58.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-19T07:54:58.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-19T07:54:58.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-19T07:54:58.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-19T07:54:58.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-19T07:54:58.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-19T07:54:58.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-19T07:54:58.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-19T07:54:58.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-19T07:54:58.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-19T07:54:58.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-19T07:54:58.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.inplace_update_support: 0 2024-09-19T07:54:58.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-19T07:54:58.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-19T07:54:58.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-19T07:54:58.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-19T07:54:58.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.bloom_locality: 0 2024-09-19T07:54:58.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.max_successive_merges: 0 2024-09-19T07:54:58.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-19T07:54:58.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.paranoid_file_checks: 0 2024-09-19T07:54:58.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.force_consistency_checks: 1 2024-09-19T07:54:58.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.report_bg_io_stats: 0 2024-09-19T07:54:58.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.ttl: 2592000 2024-09-19T07:54:58.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-19T07:54:58.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-19T07:54:58.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-19T07:54:58.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.enable_blob_files: false 2024-09-19T07:54:58.905 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.min_blob_size: 0 2024-09-19T07:54:58.905 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.blob_file_size: 268435456 2024-09-19T07:54:58.905 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-19T07:54:58.905 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-19T07:54:58.905 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-19T07:54:58.905 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-19T07:54:58.905 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-19T07:54:58.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.blob_file_starting_level: 0 2024-09-19T07:54:58.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-19T07:54:58.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-19T07:54:58.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-19T07:54:58.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: caff10de-b238-4e73-b683-9eb57e3df22f 2024-09-19T07:54:58.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726732498584116, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-19T07:54:58.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-19T07:54:58.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726732498584747, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726732498, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "caff10de-b238-4e73-b683-9eb57e3df22f", "db_session_id": "DZI87HCZKUO298PVR9OP", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-19T07:54:58.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726732498584875, "job": 1, "event": "recovery_finished"} 2024-09-19T07:54:58.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-19T07:54:58.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-19T07:54:58.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55aba65ace00 2024-09-19T07:54:58.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: DB pointer 0x55aba66a8000 2024-09-19T07:54:58.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-19T07:54:58.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-19T07:54:58.907 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** DB Stats ** 2024-09-19T07:54:58.907 INFO:journalctl@ceph.mon.c.smithi162.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-19T07:54:58.907 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-19T07:54:58.907 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-19T07:54:58.907 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-19T07:54:58.907 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-19T07:54:58.908 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-19T07:54:58.908 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-19T07:54:58.908 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-09-19T07:54:58.908 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** Compaction Stats [default] ** 2024-09-19T07:54:58.908 INFO:journalctl@ceph.mon.c.smithi162.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-19T07:54:58.908 INFO:journalctl@ceph.mon.c.smithi162.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-19T07:54:58.908 INFO:journalctl@ceph.mon.c.smithi162.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-19T07:54:58.908 INFO:journalctl@ceph.mon.c.smithi162.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-19T07:54:58.908 INFO:journalctl@ceph.mon.c.smithi162.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-19T07:54:58.908 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-09-19T07:54:58.908 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** Compaction Stats [default] ** 2024-09-19T07:54:58.908 INFO:journalctl@ceph.mon.c.smithi162.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-19T07:54:58.908 INFO:journalctl@ceph.mon.c.smithi162.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-19T07:54:58.909 INFO:journalctl@ceph.mon.c.smithi162.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-19T07:54:58.909 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-09-19T07:54:58.909 INFO:journalctl@ceph.mon.c.smithi162.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-19T07:54:58.909 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-09-19T07:54:58.909 INFO:journalctl@ceph.mon.c.smithi162.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-19T07:54:58.909 INFO:journalctl@ceph.mon.c.smithi162.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-19T07:54:58.909 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-19T07:54:58.909 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-19T07:54:58.909 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-19T07:54:58.909 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-19T07:54:58.909 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative compaction: 0.00 GB write, 0.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-19T07:54:58.909 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval compaction: 0.00 GB write, 0.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-19T07:54:58.909 INFO:journalctl@ceph.mon.c.smithi162.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-19T07:54:58.910 INFO:journalctl@ceph.mon.c.smithi162.stdout: Block cache BinnedLRUCache@0x55aba657f1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-19T07:54:58.910 INFO:journalctl@ceph.mon.c.smithi162.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-19T07:54:58.910 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-09-19T07:54:58.910 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-19T07:54:58.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-19T07:54:58.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: using public_addr v2:172.21.15.162:0/0 -> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-09-19T07:54:58.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: starting mon.c rank -1 at public addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] at bind addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:54:58.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c@-1(???) e0 preinit fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:54:58.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c@-1(synchronizing).mds e1 new map 2024-09-19T07:54:58.911 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-19T07:54:58.911 INFO:journalctl@ceph.mon.c.smithi162.stdout: e1 2024-09-19T07:54:58.911 INFO:journalctl@ceph.mon.c.smithi162.stdout: btime 2024-09-19T07:53:39:127203+0000 2024-09-19T07:54:58.911 INFO:journalctl@ceph.mon.c.smithi162.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-19T07:54:58.911 INFO:journalctl@ceph.mon.c.smithi162.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-19T07:54:58.911 INFO:journalctl@ceph.mon.c.smithi162.stdout: legacy client fscid: -1 2024-09-19T07:54:58.911 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-09-19T07:54:58.911 INFO:journalctl@ceph.mon.c.smithi162.stdout: No filesystems configured 2024-09-19T07:54:58.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-19T07:54:58.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-19T07:54:58.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-19T07:54:58.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-19T07:54:58.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-19T07:54:58.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-19T07:54:58.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-19T07:54:58.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-19T07:54:58.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-19T07:54:58.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-19T07:54:58.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: Added host smithi162 2024-09-19T07:54:58.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:58.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T07:54:58.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:54:58.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1864013099' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-19T07:54:58.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:54:58.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1864013099' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-19T07:54:58.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T07:54:58.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:58.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:58.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:58.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:58.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:54:58.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:54:58.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:54:58.916 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-19T07:54:58.916 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: Updating smithi162:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:54:58.916 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-19T07:54:58.916 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: Updating smithi162:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T07:54:58.916 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:58.916 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:58.916 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:58.916 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:58.916 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:54:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:54:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:54:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:54:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T07:54:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:54:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi139:172.21.15.139=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:54:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi139:172.21.15.139=b;smithi162:172.21.15.162=c;count:3 2024-09-19T07:54:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: Deploying daemon mon.c on smithi162 2024-09-19T07:54:58.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:54:58.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='client.? 172.21.15.162:0/2360677075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T07:54:58.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:54:58.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:54:58.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: from='client.? 172.21.15.162:0/34253752' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T07:54:58.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 ceph-mon[35260]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-19T07:54:58.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 bash[35229]: 500b1c467f929a37e7f3fa083bd2c476bd8dd7588bbeaea2c706ce9f6b545123 2024-09-19T07:54:58.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:54:58 smithi162 systemd[1]: Started Ceph mon.c for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T07:54:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:58 smithi098 ceph-mon[31333]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:54:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:54:58 smithi098 ceph-mon[31333]: from='client.? 172.21.15.162:0/34253752' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T07:55:00.076 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-19T07:55:00.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-19T07:55:00.340 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T07:55:03.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.738+0000 7f58079c3640 1 -- 172.21.15.162:0/3622632365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57ec003530 msgr2=0x7f57ec0059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:03.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.738+0000 7f58079c3640 1 --2- 172.21.15.162:0/3622632365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57ec003530 0x7f57ec0059c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f58001016c0 tx=0x7f57f402bce0 comp rx=0 tx=0).stop 2024-09-19T07:55:03.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.738+0000 7f58079c3640 1 -- 172.21.15.162:0/3622632365 shutdown_connections 2024-09-19T07:55:03.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.738+0000 7f58079c3640 1 --2- 172.21.15.162:0/3622632365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57ec003530 0x7f57ec0059c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:03.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.738+0000 7f58079c3640 1 -- 172.21.15.162:0/3622632365 >> 172.21.15.162:0/3622632365 conn(0x7f58000fc280 msgr2=0x7f58000fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:55:03.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.738+0000 7f58079c3640 1 -- 172.21.15.162:0/3622632365 shutdown_connections 2024-09-19T07:55:03.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.738+0000 7f58079c3640 1 -- 172.21.15.162:0/3622632365 wait complete. 2024-09-19T07:55:03.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.739+0000 7f58079c3640 1 Processor -- start 2024-09-19T07:55:03.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.739+0000 7f58079c3640 1 -- start start 2024-09-19T07:55:03.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.739+0000 7f58079c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57ec003530 0x7f58001994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:03.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.739+0000 7f58079c3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5800199a20 0x7f580019ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:03.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.739+0000 7f58079c3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5800108690 con 0x7f57ec003530 2024-09-19T07:55:03.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.739+0000 7f58079c3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5800108510 con 0x7f5800199a20 2024-09-19T07:55:03.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.740+0000 7f5804f5e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5800199a20 0x7f580019ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:03.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.740+0000 7f5804f5e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5800199a20 0x7f580019ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44948/0 (socket says 172.21.15.162:44948) 2024-09-19T07:55:03.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.740+0000 7f5804f5e640 1 -- 172.21.15.162:0/607128733 learned_addr learned my addr 172.21.15.162:0/607128733 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-19T07:55:03.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.740+0000 7f580575f640 1 --2- 172.21.15.162:0/607128733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57ec003530 0x7f58001994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:03.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.740+0000 7f5804f5e640 1 -- 172.21.15.162:0/607128733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5800199a20 msgr2=0x7f580019ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-19T07:55:03.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.740+0000 7f5804f5e640 1 -- 172.21.15.162:0/607128733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5800199a20 msgr2=0x7f580019ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-19T07:55:03.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.740+0000 7f5804f5e640 1 --2- 172.21.15.162:0/607128733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5800199a20 0x7f580019ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-19T07:55:03.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.740+0000 7f5804f5e640 1 --2- 172.21.15.162:0/607128733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5800199a20 0x7f580019ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T07:55:03.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.740+0000 7f580575f640 1 -- 172.21.15.162:0/607128733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5800199a20 msgr2=0x7f580019ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:55:03.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.740+0000 7f580575f640 1 --2- 172.21.15.162:0/607128733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5800199a20 0x7f580019ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:03.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.740+0000 7f580575f640 1 -- 172.21.15.162:0/607128733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f580019a040 con 0x7f57ec003530 2024-09-19T07:55:03.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.740+0000 7f580575f640 1 --2- 172.21.15.162:0/607128733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57ec003530 0x7f58001994e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f57f4011c00 tx=0x7f57f402ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:55:03.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.741+0000 7f57f27fc640 1 -- 172.21.15.162:0/607128733 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f57f40051c0 con 0x7f57ec003530 2024-09-19T07:55:03.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.741+0000 7f57f27fc640 1 -- 172.21.15.162:0/607128733 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57f4005360 con 0x7f57ec003530 2024-09-19T07:55:03.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.741+0000 7f58079c3640 1 -- 172.21.15.162:0/607128733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f580019e2f0 con 0x7f57ec003530 2024-09-19T07:55:03.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.741+0000 7f57f27fc640 1 -- 172.21.15.162:0/607128733 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f57f4004bb0 con 0x7f57ec003530 2024-09-19T07:55:03.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.741+0000 7f58079c3640 1 -- 172.21.15.162:0/607128733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f580019e850 con 0x7f57ec003530 2024-09-19T07:55:03.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.742+0000 7f57f27fc640 1 -- 172.21.15.162:0/607128733 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f57f4047060 con 0x7f57ec003530 2024-09-19T07:55:03.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.742+0000 7f57f27fc640 1 --2- 172.21.15.162:0/607128733 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f57e4039040 0x7f57e403b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:03.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.742+0000 7f58079c3640 1 -- 172.21.15.162:0/607128733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5800101cc0 con 0x7f57ec003530 2024-09-19T07:55:03.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.742+0000 7f57f27fc640 1 -- 172.21.15.162:0/607128733 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f57f403d070 con 0x7f57ec003530 2024-09-19T07:55:03.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.742+0000 7f5804f5e640 1 --2- 172.21.15.162:0/607128733 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f57e4039040 0x7f57e403b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:03.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.743+0000 7f5804f5e640 1 --2- 172.21.15.162:0/607128733 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f57e4039040 0x7f57e403b500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009100 tx=0x7f57fc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:55:03.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.748+0000 7f57f27fc640 1 -- 172.21.15.162:0/607128733 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f57f4037d20 con 0x7f57ec003530 2024-09-19T07:55:03.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:55:03.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:03.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: mon.a calling monitor election 2024-09-19T07:55:03.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:03.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:03.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:03.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: mon.c calling monitor election 2024-09-19T07:55:03.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:03.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:03.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:03.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:03.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:03.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-19T07:55:03.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: monmap epoch 2 2024-09-19T07:55:03.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:55:03.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: last_changed 2024-09-19T07:54:58.612818+0000 2024-09-19T07:55:03.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: created 2024-09-19T07:53:35.879566+0000 2024-09-19T07:55:03.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: min_mon_release 19 (squid) 2024-09-19T07:55:03.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: election_strategy: 1 2024-09-19T07:55:03.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-19T07:55:03.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-09-19T07:55:03.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: fsmap 2024-09-19T07:55:03.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T07:55:03.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: mgrmap e13: x(active, since 38s) 2024-09-19T07:55:03.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: overall HEALTH_OK 2024-09-19T07:55:03.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:03.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:03.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:03.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T07:55:03.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:03.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.992+0000 7f58079c3640 1 -- 172.21.15.162:0/607128733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f580019eb60 con 0x7f57ec003530 2024-09-19T07:55:03.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:55:03.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:03.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: mon.a calling monitor election 2024-09-19T07:55:03.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:03.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:03.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:03.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: mon.c calling monitor election 2024-09-19T07:55:03.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:03.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:03.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:03.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:03.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:03.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-19T07:55:03.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: monmap epoch 2 2024-09-19T07:55:03.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:55:03.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: last_changed 2024-09-19T07:54:58.612818+0000 2024-09-19T07:55:03.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: created 2024-09-19T07:53:35.879566+0000 2024-09-19T07:55:03.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: min_mon_release 19 (squid) 2024-09-19T07:55:03.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: election_strategy: 1 2024-09-19T07:55:03.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-19T07:55:03.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-09-19T07:55:03.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: fsmap 2024-09-19T07:55:03.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T07:55:03.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: mgrmap e13: x(active, since 38s) 2024-09-19T07:55:03.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: overall HEALTH_OK 2024-09-19T07:55:03.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:03.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:03.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:03.995 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T07:55:03.995 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:03.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.992+0000 7f57f27fc640 1 -- 172.21.15.162:0/607128733 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f57f4004e40 con 0x7f57ec003530 2024-09-19T07:55:03.995 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:55:03.996 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":2,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","modified":"2024-09-19T07:54:58.612818Z","created":"2024-09-19T07:53:35.879566Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-19T07:55:03.996 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 2 2024-09-19T07:55:03.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.995+0000 7f58079c3640 1 -- 172.21.15.162:0/607128733 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f57e4039040 msgr2=0x7f57e403b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:03.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.995+0000 7f58079c3640 1 --2- 172.21.15.162:0/607128733 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f57e4039040 0x7f57e403b500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009100 tx=0x7f57fc008040 comp rx=0 tx=0).stop 2024-09-19T07:55:03.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.995+0000 7f58079c3640 1 -- 172.21.15.162:0/607128733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57ec003530 msgr2=0x7f58001994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:03.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.995+0000 7f58079c3640 1 --2- 172.21.15.162:0/607128733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57ec003530 0x7f58001994e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f57f4011c00 tx=0x7f57f402ca70 comp rx=0 tx=0).stop 2024-09-19T07:55:03.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.995+0000 7f58079c3640 1 -- 172.21.15.162:0/607128733 shutdown_connections 2024-09-19T07:55:03.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.995+0000 7f58079c3640 1 --2- 172.21.15.162:0/607128733 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f57e4039040 0x7f57e403b500 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:03.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.995+0000 7f58079c3640 1 --2- 172.21.15.162:0/607128733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5800199a20 0x7f580019ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:03.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.995+0000 7f58079c3640 1 --2- 172.21.15.162:0/607128733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57ec003530 0x7f58001994e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:03.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.995+0000 7f58079c3640 1 -- 172.21.15.162:0/607128733 >> 172.21.15.162:0/607128733 conn(0x7f58000fc280 msgr2=0x7f580006a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:55:03.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.995+0000 7f58079c3640 1 -- 172.21.15.162:0/607128733 shutdown_connections 2024-09-19T07:55:03.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:03.995+0000 7f58079c3640 1 -- 172.21.15.162:0/607128733 wait complete. 2024-09-19T07:55:04.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:04 smithi162 ceph-mon[35260]: Deploying daemon mon.b on smithi139 2024-09-19T07:55:04.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:04 smithi162 ceph-mon[35260]: from='client.? 172.21.15.162:0/607128733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T07:55:04.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:05.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:04 smithi098 ceph-mon[31333]: Deploying daemon mon.b on smithi139 2024-09-19T07:55:05.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:04 smithi098 ceph-mon[31333]: from='client.? 172.21.15.162:0/607128733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T07:55:05.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:05.632 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-19T07:55:05.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-19T07:55:05.879 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T07:55:06.000 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:05 smithi162 ceph-mon[35260]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:06.185 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:05 smithi098 ceph-mon[31333]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:06.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.276+0000 7f2df7dec640 1 -- 172.21.15.162:0/2714521447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0102840 msgr2=0x7f2df0102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:06.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.276+0000 7f2df7dec640 1 --2- 172.21.15.162:0/2714521447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0102840 0x7f2df0102c20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009c20 tx=0x7f2de002ab20 comp rx=0 tx=0).stop 2024-09-19T07:55:06.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.276+0000 7f2df7dec640 1 -- 172.21.15.162:0/2714521447 shutdown_connections 2024-09-19T07:55:06.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.276+0000 7f2df7dec640 1 --2- 172.21.15.162:0/2714521447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0102840 0x7f2df0102c20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:06.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.276+0000 7f2df7dec640 1 -- 172.21.15.162:0/2714521447 >> 172.21.15.162:0/2714521447 conn(0x7f2df00fc280 msgr2=0x7f2df00fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:55:06.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.277+0000 7f2df7dec640 1 -- 172.21.15.162:0/2714521447 shutdown_connections 2024-09-19T07:55:06.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.277+0000 7f2df7dec640 1 -- 172.21.15.162:0/2714521447 wait complete. 2024-09-19T07:55:06.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.277+0000 7f2df7dec640 1 Processor -- start 2024-09-19T07:55:06.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.277+0000 7f2df7dec640 1 -- start start 2024-09-19T07:55:06.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df7dec640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2df0102840 0x7f2df01973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:06.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df7dec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0197900 0x7f2df019bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:06.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df7dec640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2df010bdb0 con 0x7f2df0197900 2024-09-19T07:55:06.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df7dec640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2df010bc30 con 0x7f2df0102840 2024-09-19T07:55:06.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df5b88640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2df0102840 0x7f2df01973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:06.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df5387640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0197900 0x7f2df019bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:06.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df5387640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0197900 0x7f2df019bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.162:49228/0 (socket says 172.21.15.162:49228) 2024-09-19T07:55:06.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df5b88640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2df0102840 0x7f2df01973c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44958/0 (socket says 172.21.15.162:44958) 2024-09-19T07:55:06.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df5387640 1 -- 172.21.15.162:0/1903962775 learned_addr learned my addr 172.21.15.162:0/1903962775 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-19T07:55:06.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df5b88640 1 -- 172.21.15.162:0/1903962775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2df0102840 msgr2=0x7f2df01973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-19T07:55:06.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df5b88640 1 -- 172.21.15.162:0/1903962775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2df0102840 msgr2=0x7f2df01973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-19T07:55:06.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df5b88640 1 --2- 172.21.15.162:0/1903962775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2df0102840 0x7f2df01973c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-19T07:55:06.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df5b88640 1 --2- 172.21.15.162:0/1903962775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2df0102840 0x7f2df01973c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T07:55:06.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df5387640 1 -- 172.21.15.162:0/1903962775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2df0102840 msgr2=0x7f2df01973c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:55:06.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df5387640 1 --2- 172.21.15.162:0/1903962775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2df0102840 0x7f2df01973c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:06.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df5387640 1 -- 172.21.15.162:0/1903962775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df0197f20 con 0x7f2df0197900 2024-09-19T07:55:06.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.278+0000 7f2df5387640 1 --2- 172.21.15.162:0/1903962775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0197900 0x7f2df019bc90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2de400b4a0 tx=0x7f2de400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:55:06.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.279+0000 7f2deeffd640 1 -- 172.21.15.162:0/1903962775 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f2de4010610 con 0x7f2df0197900 2024-09-19T07:55:06.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.279+0000 7f2df7dec640 1 -- 172.21.15.162:0/1903962775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2df019c1d0 con 0x7f2df0197900 2024-09-19T07:55:06.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.279+0000 7f2deeffd640 1 -- 172.21.15.162:0/1903962775 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2de4010c90 con 0x7f2df0197900 2024-09-19T07:55:06.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.279+0000 7f2deeffd640 1 -- 172.21.15.162:0/1903962775 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f2de4017e60 con 0x7f2df0197900 2024-09-19T07:55:06.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.279+0000 7f2df7dec640 1 -- 172.21.15.162:0/1903962775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2df019c710 con 0x7f2df0197900 2024-09-19T07:55:06.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.280+0000 7f2deeffd640 1 -- 172.21.15.162:0/1903962775 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f2de4017070 con 0x7f2df0197900 2024-09-19T07:55:06.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.280+0000 7f2df7dec640 1 -- 172.21.15.162:0/1903962775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2df006a4f0 con 0x7f2df0197900 2024-09-19T07:55:06.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.280+0000 7f2deeffd640 1 --2- 172.21.15.162:0/1903962775 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2dd8038ff0 0x7f2dd803b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:06.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.280+0000 7f2deeffd640 1 -- 172.21.15.162:0/1903962775 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2de400d040 con 0x7f2df0197900 2024-09-19T07:55:06.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.280+0000 7f2df5b88640 1 --2- 172.21.15.162:0/1903962775 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2dd8038ff0 0x7f2dd803b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:06.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.281+0000 7f2df5b88640 1 --2- 172.21.15.162:0/1903962775 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2dd8038ff0 0x7f2dd803b4b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009a60 tx=0x7f2de0005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:55:06.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.285+0000 7f2deeffd640 1 -- 172.21.15.162:0/1903962775 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2de401e020 con 0x7f2df0197900 2024-09-19T07:55:06.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.513+0000 7f2df7dec640 1 -- 172.21.15.162:0/1903962775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f2df019c9d0 con 0x7f2df0197900 2024-09-19T07:55:06.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.514+0000 7f2deeffd640 1 -- 172.21.15.162:0/1903962775 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f2de400eb40 con 0x7f2df0197900 2024-09-19T07:55:06.514 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:55:06.514 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":2,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","modified":"2024-09-19T07:54:58.612818Z","created":"2024-09-19T07:53:35.879566Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-19T07:55:06.515 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 2 2024-09-19T07:55:06.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.516+0000 7f2df7dec640 1 -- 172.21.15.162:0/1903962775 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2dd8038ff0 msgr2=0x7f2dd803b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:06.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.516+0000 7f2df7dec640 1 --2- 172.21.15.162:0/1903962775 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2dd8038ff0 0x7f2dd803b4b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009a60 tx=0x7f2de0005da0 comp rx=0 tx=0).stop 2024-09-19T07:55:06.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.516+0000 7f2df7dec640 1 -- 172.21.15.162:0/1903962775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0197900 msgr2=0x7f2df019bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:06.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.516+0000 7f2df7dec640 1 --2- 172.21.15.162:0/1903962775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0197900 0x7f2df019bc90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2de400b4a0 tx=0x7f2de400b960 comp rx=0 tx=0).stop 2024-09-19T07:55:06.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.516+0000 7f2df7dec640 1 -- 172.21.15.162:0/1903962775 shutdown_connections 2024-09-19T07:55:06.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.516+0000 7f2df7dec640 1 --2- 172.21.15.162:0/1903962775 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2dd8038ff0 0x7f2dd803b4b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:06.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.516+0000 7f2df7dec640 1 --2- 172.21.15.162:0/1903962775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0197900 0x7f2df019bc90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:06.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.516+0000 7f2df7dec640 1 --2- 172.21.15.162:0/1903962775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2df0102840 0x7f2df01973c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:06.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.516+0000 7f2df7dec640 1 -- 172.21.15.162:0/1903962775 >> 172.21.15.162:0/1903962775 conn(0x7f2df00fc280 msgr2=0x7f2df0105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:55:06.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.516+0000 7f2df7dec640 1 -- 172.21.15.162:0/1903962775 shutdown_connections 2024-09-19T07:55:06.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:06.516+0000 7f2df7dec640 1 -- 172.21.15.162:0/1903962775 wait complete. 2024-09-19T07:55:06.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:06 smithi162 ceph-mon[35260]: from='client.? 172.21.15.162:0/1903962775' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T07:55:07.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:06 smithi098 ceph-mon[31333]: from='client.? 172.21.15.162:0/1903962775' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T07:55:07.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:07 smithi139 systemd[1]: Started Ceph mon.b for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T07:55:08.078 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-19T07:55:08.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-19T07:55:08.332 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T07:55:12.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:55:12.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:12.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: mon.a calling monitor election 2024-09-19T07:55:12.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: mon.c calling monitor election 2024-09-19T07:55:12.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:12.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:12.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:12.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:12.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:12.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:12.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:12.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:12.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:12.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-19T07:55:12.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: monmap epoch 3 2024-09-19T07:55:12.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:55:12.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: last_changed 2024-09-19T07:55:07.444500+0000 2024-09-19T07:55:12.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: created 2024-09-19T07:53:35.879566+0000 2024-09-19T07:55:12.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: min_mon_release 19 (squid) 2024-09-19T07:55:12.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: election_strategy: 1 2024-09-19T07:55:12.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-19T07:55:12.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-09-19T07:55:12.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-09-19T07:55:12.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: fsmap 2024-09-19T07:55:12.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T07:55:12.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: mgrmap e13: x(active, since 46s) 2024-09-19T07:55:12.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: overall HEALTH_OK 2024-09-19T07:55:12.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:12.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:12.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:12.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:12.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:55:12.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:12.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: mon.a calling monitor election 2024-09-19T07:55:12.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: mon.c calling monitor election 2024-09-19T07:55:12.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-19T07:55:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: monmap epoch 3 2024-09-19T07:55:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:55:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: last_changed 2024-09-19T07:55:07.444500+0000 2024-09-19T07:55:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: created 2024-09-19T07:53:35.879566+0000 2024-09-19T07:55:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: min_mon_release 19 (squid) 2024-09-19T07:55:12.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: election_strategy: 1 2024-09-19T07:55:12.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-19T07:55:12.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-09-19T07:55:12.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-09-19T07:55:12.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: fsmap 2024-09-19T07:55:12.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T07:55:12.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: mgrmap e13: x(active, since 46s) 2024-09-19T07:55:12.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: overall HEALTH_OK 2024-09-19T07:55:12.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:12.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:12.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:12.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:13.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.111+0000 7f7670b8c640 1 -- 172.21.15.162:0/2769717331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f766c106650 msgr2=0x7f7658005580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:13.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.111+0000 7f7670b8c640 1 --2- 172.21.15.162:0/2769717331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f766c106650 0x7f7658005580 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f765c012c60 tx=0x7f765c02c620 comp rx=0 tx=0).stop 2024-09-19T07:55:13.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.111+0000 7f7670b8c640 1 -- 172.21.15.162:0/2769717331 shutdown_connections 2024-09-19T07:55:13.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.111+0000 7f7670b8c640 1 --2- 172.21.15.162:0/2769717331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f766c106650 0x7f7658005580 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:13.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.111+0000 7f7670b8c640 1 -- 172.21.15.162:0/2769717331 >> 172.21.15.162:0/2769717331 conn(0x7f766c0fc280 msgr2=0x7f766c0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:55:13.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.111+0000 7f7670b8c640 1 -- 172.21.15.162:0/2769717331 shutdown_connections 2024-09-19T07:55:13.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.111+0000 7f7670b8c640 1 -- 172.21.15.162:0/2769717331 wait complete. 2024-09-19T07:55:13.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7670b8c640 1 Processor -- start 2024-09-19T07:55:13.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7670b8c640 1 -- start start 2024-09-19T07:55:13.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7670b8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f766c106650 0x7f766c1972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:13.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7670b8c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f766c197830 0x7f766c19bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:13.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7670b8c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766c19c100 0x7f766c19c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:13.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7670b8c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f766c108680 con 0x7f766c106650 2024-09-19T07:55:13.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7670b8c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f766c108500 con 0x7f766c19c100 2024-09-19T07:55:13.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7670b8c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f766c108800 con 0x7f766c197830 2024-09-19T07:55:13.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7669d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f766c197830 0x7f766c19bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:13.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7669d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f766c197830 0x7f766c19bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:55376/0 (socket says 172.21.15.162:55376) 2024-09-19T07:55:13.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7669d9b640 1 -- 172.21.15.162:0/1601573034 learned_addr learned my addr 172.21.15.162:0/1601573034 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-19T07:55:13.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7669d9b640 1 -- 172.21.15.162:0/1601573034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f766c197830 msgr2=0x7f766c19bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-19T07:55:13.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7669d9b640 1 -- 172.21.15.162:0/1601573034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f766c197830 msgr2=0x7f766c19bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-19T07:55:13.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7669d9b640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f766c197830 0x7f766c19bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-19T07:55:13.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f766a59c640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f766c106650 0x7f766c1972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:13.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f7669d9b640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f766c197830 0x7f766c19bbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T07:55:13.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.112+0000 7f766ad9d640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766c19c100 0x7f766c19c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:13.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f766ad9d640 1 -- 172.21.15.162:0/1601573034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766c19c100 msgr2=0x7f766c19c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-19T07:55:13.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f766ad9d640 1 -- 172.21.15.162:0/1601573034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766c19c100 msgr2=0x7f766c19c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-19T07:55:13.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f766ad9d640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766c19c100 0x7f766c19c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-19T07:55:13.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f766ad9d640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766c19c100 0x7f766c19c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T07:55:13.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f766a59c640 1 -- 172.21.15.162:0/1601573034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f766c197830 msgr2=0x7f766c19bbc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:55:13.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f766a59c640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f766c197830 0x7f766c19bbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:13.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f766a59c640 1 -- 172.21.15.162:0/1601573034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766c19c100 msgr2=0x7f766c19c5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:55:13.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f766a59c640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766c19c100 0x7f766c19c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:13.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f766a59c640 1 -- 172.21.15.162:0/1601573034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f766c1a3e70 con 0x7f766c106650 2024-09-19T07:55:13.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f766a59c640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f766c106650 0x7f766c1972f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f765c034000 tx=0x7f765c005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:55:13.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f76637fe640 1 -- 172.21.15.162:0/1601573034 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765c041020 con 0x7f766c106650 2024-09-19T07:55:13.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f7670b8c640 1 -- 172.21.15.162:0/1601573034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f766c1a4040 con 0x7f766c106650 2024-09-19T07:55:13.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f76637fe640 1 -- 172.21.15.162:0/1601573034 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f765c004ce0 con 0x7f766c106650 2024-09-19T07:55:13.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f76637fe640 1 -- 172.21.15.162:0/1601573034 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765c03dd30 con 0x7f766c106650 2024-09-19T07:55:13.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.113+0000 7f7670b8c640 1 -- 172.21.15.162:0/1601573034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f766c1a45d0 con 0x7f766c106650 2024-09-19T07:55:13.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.114+0000 7f76637fe640 1 -- 172.21.15.162:0/1601573034 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f765c044020 con 0x7f766c106650 2024-09-19T07:55:13.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.114+0000 7f7670b8c640 1 -- 172.21.15.162:0/1601573034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f766c109770 con 0x7f766c106650 2024-09-19T07:55:13.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.114+0000 7f76637fe640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f764c039730 0x7f764c03bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:13.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.114+0000 7f76637fe640 1 -- 172.21.15.162:0/1601573034 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f765c06ef50 con 0x7f766c106650 2024-09-19T07:55:13.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.114+0000 7f7669d9b640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f764c039730 0x7f764c03bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:13.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.115+0000 7f7669d9b640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f764c039730 0x7f764c03bbf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7654009100 tx=0x7f7654008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:55:13.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.120+0000 7f76637fe640 1 -- 172.21.15.162:0/1601573034 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f765c04b020 con 0x7f766c106650 2024-09-19T07:55:13.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.366+0000 7f7670b8c640 1 -- 172.21.15.162:0/1601573034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f766c1a4860 con 0x7f766c106650 2024-09-19T07:55:13.587 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.367+0000 7f76637fe640 1 -- 172.21.15.162:0/1601573034 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f765c04d300 con 0x7f766c106650 2024-09-19T07:55:13.588 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:55:13.588 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":3,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","modified":"2024-09-19T07:55:07.444500Z","created":"2024-09-19T07:53:35.879566Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-19T07:55:13.589 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 3 2024-09-19T07:55:13.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.369+0000 7f7670b8c640 1 -- 172.21.15.162:0/1601573034 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f764c039730 msgr2=0x7f764c03bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:13.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.369+0000 7f7670b8c640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f764c039730 0x7f764c03bbf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7654009100 tx=0x7f7654008040 comp rx=0 tx=0).stop 2024-09-19T07:55:13.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.369+0000 7f7670b8c640 1 -- 172.21.15.162:0/1601573034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f766c106650 msgr2=0x7f766c1972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:13.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.369+0000 7f7670b8c640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f766c106650 0x7f766c1972f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f765c034000 tx=0x7f765c005270 comp rx=0 tx=0).stop 2024-09-19T07:55:13.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.369+0000 7f7670b8c640 1 -- 172.21.15.162:0/1601573034 shutdown_connections 2024-09-19T07:55:13.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.369+0000 7f7670b8c640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f764c039730 0x7f764c03bbf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:13.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.369+0000 7f7670b8c640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766c19c100 0x7f766c19c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:13.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.369+0000 7f7670b8c640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f766c197830 0x7f766c19bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:13.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.369+0000 7f7670b8c640 1 --2- 172.21.15.162:0/1601573034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f766c106650 0x7f766c1972f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:13.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.369+0000 7f7670b8c640 1 -- 172.21.15.162:0/1601573034 >> 172.21.15.162:0/1601573034 conn(0x7f766c0fc280 msgr2=0x7f766c06a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:55:13.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.369+0000 7f7670b8c640 1 -- 172.21.15.162:0/1601573034 shutdown_connections 2024-09-19T07:55:13.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:13.369+0000 7f7670b8c640 1 -- 172.21.15.162:0/1601573034 wait complete. 2024-09-19T07:55:13.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:13 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:13.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:13 smithi162 ceph-mon[35260]: from='client.? 172.21.15.162:0/1601573034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T07:55:13.624 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:13 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:13.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:13 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:13.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:13 smithi098 ceph-mon[31333]: from='client.? 172.21.15.162:0/1601573034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T07:55:13.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:13 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:14.366 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-19T07:55:14.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-19T07:55:14.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:55:14.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:14.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: mon.a calling monitor election 2024-09-19T07:55:14.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: mon.c calling monitor election 2024-09-19T07:55:14.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:55:14.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:14.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:14.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:14.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:14.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:14.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:14.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:14.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:14.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-19T07:55:14.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: monmap epoch 3 2024-09-19T07:55:14.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:55:14.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: last_changed 2024-09-19T07:55:07.444500+0000 2024-09-19T07:55:14.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: created 2024-09-19T07:53:35.879566+0000 2024-09-19T07:55:14.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: min_mon_release 19 (squid) 2024-09-19T07:55:14.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: election_strategy: 1 2024-09-19T07:55:14.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-19T07:55:14.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-09-19T07:55:14.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-09-19T07:55:14.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: fsmap 2024-09-19T07:55:14.930 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T07:55:14.930 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: mgrmap e13: x(active, since 46s) 2024-09-19T07:55:14.930 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: overall HEALTH_OK 2024-09-19T07:55:14.930 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:14.930 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:14.930 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:14.930 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:14.930 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:14.930 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='client.? 172.21.15.162:0/1601573034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T07:55:14.930 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:15.480 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: mon.b calling monitor election 2024-09-19T07:55:15.480 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-19T07:55:15.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-19T07:55:15.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-19T07:55:15.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: Updating smithi139:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:55:15.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: Updating smithi162:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:55:15.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: Updating smithi098:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:55:15.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: Reconfiguring mon.a (unknown last config time)... 2024-09-19T07:55:15.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: Reconfiguring daemon mon.a on smithi098 2024-09-19T07:55:15.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: mon.b calling monitor election 2024-09-19T07:55:15.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: mon.c calling monitor election 2024-09-19T07:55:15.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: mon.a calling monitor election 2024-09-19T07:55:15.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-19T07:55:15.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: monmap epoch 3 2024-09-19T07:55:15.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:55:15.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: last_changed 2024-09-19T07:55:07.444500+0000 2024-09-19T07:55:15.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: created 2024-09-19T07:53:35.879566+0000 2024-09-19T07:55:15.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: min_mon_release 19 (squid) 2024-09-19T07:55:15.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: election_strategy: 1 2024-09-19T07:55:15.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-19T07:55:15.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-09-19T07:55:15.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-09-19T07:55:15.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: fsmap 2024-09-19T07:55:15.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T07:55:15.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: mgrmap e13: x(active, since 48s) 2024-09-19T07:55:15.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:15 smithi098 ceph-mon[31333]: overall HEALTH_OK 2024-09-19T07:55:15.553 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:55:15.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: mon.b calling monitor election 2024-09-19T07:55:15.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-19T07:55:15.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-19T07:55:15.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-19T07:55:15.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: Updating smithi139:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:55:15.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: Updating smithi162:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:55:15.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: Updating smithi098:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:55:15.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: Reconfiguring mon.a (unknown last config time)... 2024-09-19T07:55:15.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: Reconfiguring daemon mon.a on smithi098 2024-09-19T07:55:15.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: mon.b calling monitor election 2024-09-19T07:55:15.622 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: mon.c calling monitor election 2024-09-19T07:55:15.622 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: mon.a calling monitor election 2024-09-19T07:55:15.622 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-19T07:55:15.622 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: monmap epoch 3 2024-09-19T07:55:15.622 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:55:15.622 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: last_changed 2024-09-19T07:55:07.444500+0000 2024-09-19T07:55:15.622 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: created 2024-09-19T07:53:35.879566+0000 2024-09-19T07:55:15.622 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: min_mon_release 19 (squid) 2024-09-19T07:55:15.622 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: election_strategy: 1 2024-09-19T07:55:15.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-19T07:55:15.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-09-19T07:55:15.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-09-19T07:55:15.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: fsmap 2024-09-19T07:55:15.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T07:55:15.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: mgrmap e13: x(active, since 48s) 2024-09-19T07:55:15.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:15 smithi162 ceph-mon[35260]: overall HEALTH_OK 2024-09-19T07:55:15.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: mon.b calling monitor election 2024-09-19T07:55:15.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-19T07:55:15.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-19T07:55:15.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-19T07:55:15.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: Updating smithi139:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:55:15.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: Updating smithi162:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:55:15.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: Updating smithi098:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T07:55:15.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: Reconfiguring mon.a (unknown last config time)... 2024-09-19T07:55:15.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: Reconfiguring daemon mon.a on smithi098 2024-09-19T07:55:15.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: mon.b calling monitor election 2024-09-19T07:55:15.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: mon.c calling monitor election 2024-09-19T07:55:15.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: mon.a calling monitor election 2024-09-19T07:55:15.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-19T07:55:15.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: monmap epoch 3 2024-09-19T07:55:15.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:55:15.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: last_changed 2024-09-19T07:55:07.444500+0000 2024-09-19T07:55:15.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: created 2024-09-19T07:53:35.879566+0000 2024-09-19T07:55:15.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: min_mon_release 19 (squid) 2024-09-19T07:55:15.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: election_strategy: 1 2024-09-19T07:55:15.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-19T07:55:15.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-09-19T07:55:15.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-09-19T07:55:15.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: fsmap 2024-09-19T07:55:15.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T07:55:15.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: mgrmap e13: x(active, since 48s) 2024-09-19T07:55:15.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:15 smithi139 ceph-mon[34874]: overall HEALTH_OK 2024-09-19T07:55:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.026+0000 7f26fe607640 1 -- 172.21.15.98:0/2659440839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f8100e20 msgr2=0x7f26f8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.026+0000 7f26fe607640 1 --2- 172.21.15.98:0/2659440839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f8100e20 0x7f26f8101200 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f26e8009900 tx=0x7f26e802a480 comp rx=0 tx=0).stop 2024-09-19T07:55:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.026+0000 7f26fe607640 1 -- 172.21.15.98:0/2659440839 shutdown_connections 2024-09-19T07:55:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.026+0000 7f26fe607640 1 --2- 172.21.15.98:0/2659440839 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f26f8102ef0 0x7f26f810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.026+0000 7f26fe607640 1 --2- 172.21.15.98:0/2659440839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26f8102570 0x7f26f81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.026+0000 7f26fe607640 1 --2- 172.21.15.98:0/2659440839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f8100e20 0x7f26f8101200 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.026+0000 7f26fe607640 1 -- 172.21.15.98:0/2659440839 >> 172.21.15.98:0/2659440839 conn(0x7f26f80fc7d0 msgr2=0x7f26f80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:55:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.026+0000 7f26fe607640 1 -- 172.21.15.98:0/2659440839 shutdown_connections 2024-09-19T07:55:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.026+0000 7f26fe607640 1 -- 172.21.15.98:0/2659440839 wait complete. 2024-09-19T07:55:16.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.027+0000 7f26fe607640 1 Processor -- start 2024-09-19T07:55:16.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.027+0000 7f26fe607640 1 -- start start 2024-09-19T07:55:16.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.027+0000 7f26fe607640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f8100e20 0x7f26f8198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:16.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.027+0000 7f26fe607640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f26f8102570 0x7f26f8198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:16.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.027+0000 7f26fe607640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26f8102ef0 0x7f26f819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:16.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.027+0000 7f26fe607640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f26f8110f80 con 0x7f26f8100e20 2024-09-19T07:55:16.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.027+0000 7f26fe607640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f26f8110e00 con 0x7f26f8102570 2024-09-19T07:55:16.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.027+0000 7f26fe607640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f26f8111100 con 0x7f26f8102ef0 2024-09-19T07:55:16.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26f7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f8100e20 0x7f26f8198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:16.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26f7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f8100e20 0x7f26f8198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57964/0 (socket says 172.21.15.98:57964) 2024-09-19T07:55:16.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26f7fff640 1 -- 172.21.15.98:0/1502306425 learned_addr learned my addr 172.21.15.98:0/1502306425 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:55:16.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26f77fe640 1 --2- 172.21.15.98:0/1502306425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f26f8102570 0x7f26f8198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:16.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26fcba4640 1 --2- 172.21.15.98:0/1502306425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26f8102ef0 0x7f26f819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:16.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26f7fff640 1 -- 172.21.15.98:0/1502306425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26f8102ef0 msgr2=0x7f26f819cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:16.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26f7fff640 1 --2- 172.21.15.98:0/1502306425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26f8102ef0 0x7f26f819cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:16.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26f7fff640 1 -- 172.21.15.98:0/1502306425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f26f8102570 msgr2=0x7f26f8198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:16.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26f7fff640 1 --2- 172.21.15.98:0/1502306425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f26f8102570 0x7f26f8198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:16.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26f7fff640 1 -- 172.21.15.98:0/1502306425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f819d4d0 con 0x7f26f8100e20 2024-09-19T07:55:16.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26f7fff640 1 --2- 172.21.15.98:0/1502306425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f8100e20 0x7f26f8198520 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f26e8006010 tx=0x7f26e8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:55:16.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26f57fa640 1 -- 172.21.15.98:0/1502306425 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e803f070 con 0x7f26f8100e20 2024-09-19T07:55:16.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26f57fa640 1 -- 172.21.15.98:0/1502306425 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26e8031c30 con 0x7f26f8100e20 2024-09-19T07:55:16.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26fe607640 1 -- 172.21.15.98:0/1502306425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f819d760 con 0x7f26f8100e20 2024-09-19T07:55:16.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26f57fa640 1 -- 172.21.15.98:0/1502306425 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e8035030 con 0x7f26f8100e20 2024-09-19T07:55:16.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.028+0000 7f26fe607640 1 -- 172.21.15.98:0/1502306425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f81a50a0 con 0x7f26f8100e20 2024-09-19T07:55:16.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.029+0000 7f26f57fa640 1 -- 172.21.15.98:0/1502306425 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f26e8042070 con 0x7f26f8100e20 2024-09-19T07:55:16.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.029+0000 7f26fe607640 1 -- 172.21.15.98:0/1502306425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26f8101200 con 0x7f26f8100e20 2024-09-19T07:55:16.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.029+0000 7f26f57fa640 1 --2- 172.21.15.98:0/1502306425 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f26e40394d0 0x7f26e403b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:16.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.030+0000 7f26f57fa640 1 -- 172.21.15.98:0/1502306425 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f26e806cd90 con 0x7f26f8100e20 2024-09-19T07:55:16.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.030+0000 7f26f77fe640 1 --2- 172.21.15.98:0/1502306425 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f26e40394d0 0x7f26e403b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:16.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.030+0000 7f26f77fe640 1 --2- 172.21.15.98:0/1502306425 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f26e40394d0 0x7f26e403b990 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f26f80fff40 tx=0x7f26e000a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:55:16.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.035+0000 7f26f57fa640 1 -- 172.21.15.98:0/1502306425 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f26e80433b0 con 0x7f26f8100e20 2024-09-19T07:55:16.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.204+0000 7f26fe607640 1 -- 172.21.15.98:0/1502306425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f26f81a5270 con 0x7f26f8100e20 2024-09-19T07:55:16.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.204+0000 7f26f57fa640 1 -- 172.21.15.98:0/1502306425 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f26e8040260 con 0x7f26f8100e20 2024-09-19T07:55:16.205 INFO:teuthology.orchestra.run.smithi098.stdout:# minimal ceph.conf for 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:55:16.206 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-09-19T07:55:16.206 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T07:55:16.206 INFO:teuthology.orchestra.run.smithi098.stdout: mon_host = [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-09-19T07:55:16.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.207+0000 7f26fe607640 1 -- 172.21.15.98:0/1502306425 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f26e40394d0 msgr2=0x7f26e403b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:16.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.207+0000 7f26fe607640 1 --2- 172.21.15.98:0/1502306425 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f26e40394d0 0x7f26e403b990 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f26f80fff40 tx=0x7f26e000a250 comp rx=0 tx=0).stop 2024-09-19T07:55:16.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.207+0000 7f26fe607640 1 -- 172.21.15.98:0/1502306425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f8100e20 msgr2=0x7f26f8198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:16.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.207+0000 7f26fe607640 1 --2- 172.21.15.98:0/1502306425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f8100e20 0x7f26f8198520 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f26e8006010 tx=0x7f26e8003c10 comp rx=0 tx=0).stop 2024-09-19T07:55:16.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.207+0000 7f26fe607640 1 -- 172.21.15.98:0/1502306425 shutdown_connections 2024-09-19T07:55:16.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.208+0000 7f26fe607640 1 --2- 172.21.15.98:0/1502306425 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f26e40394d0 0x7f26e403b990 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:16.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.208+0000 7f26fe607640 1 --2- 172.21.15.98:0/1502306425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26f8102ef0 0x7f26f819cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:16.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.208+0000 7f26fe607640 1 --2- 172.21.15.98:0/1502306425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f26f8102570 0x7f26f8198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:16.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.208+0000 7f26fe607640 1 --2- 172.21.15.98:0/1502306425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f8100e20 0x7f26f8198520 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:16.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.208+0000 7f26fe607640 1 -- 172.21.15.98:0/1502306425 >> 172.21.15.98:0/1502306425 conn(0x7f26f80fc7d0 msgr2=0x7f26f806a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:55:16.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.208+0000 7f26fe607640 1 -- 172.21.15.98:0/1502306425 shutdown_connections 2024-09-19T07:55:16.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:16.208+0000 7f26fe607640 1 -- 172.21.15.98:0/1502306425 wait complete. 2024-09-19T07:55:16.462 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:16.462 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:16 smithi098 ceph-mon[31333]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:16.462 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:16.462 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:16.462 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:16.462 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:16 smithi098 ceph-mon[31333]: Reconfiguring mon.b (monmap changed)... 2024-09-19T07:55:16.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T07:55:16.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T07:55:16.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:16.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:16 smithi098 ceph-mon[31333]: Reconfiguring daemon mon.b on smithi139 2024-09-19T07:55:16.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:16 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1502306425' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:16.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:16.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:16 smithi162 ceph-mon[35260]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:16.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:16.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:16.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:16.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:16 smithi162 ceph-mon[35260]: Reconfiguring mon.b (monmap changed)... 2024-09-19T07:55:16.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T07:55:16.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T07:55:16.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:16.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:16 smithi162 ceph-mon[35260]: Reconfiguring daemon mon.b on smithi139 2024-09-19T07:55:16.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:16 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1502306425' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:16.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:55:16.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:16 smithi139 ceph-mon[34874]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:16.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:16.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:16.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:16.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:16 smithi139 ceph-mon[34874]: Reconfiguring mon.b (monmap changed)... 2024-09-19T07:55:16.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T07:55:16.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T07:55:16.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:16.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:16 smithi139 ceph-mon[34874]: Reconfiguring daemon mon.b on smithi139 2024-09-19T07:55:16.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:16 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1502306425' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:16.807 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-19T07:55:16.807 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T07:55:16.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-09-19T07:55:16.835 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T07:55:16.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-19T07:55:16.901 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-19T07:55:16.901 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.conf 2024-09-19T07:55:16.930 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-19T07:55:16.930 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-19T07:55:16.995 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-19T07:55:16.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.conf 2024-09-19T07:55:17.022 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-19T07:55:17.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-19T07:55:17.089 INFO:tasks.cephadm:Adding mgr.x on smithi098 2024-09-19T07:55:17.090 INFO:tasks.cephadm:Adding mgr.y on smithi139 2024-09-19T07:55:17.090 INFO:tasks.cephadm:Adding mgr.z on smithi162 2024-09-19T07:55:17.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch apply mgr '3;smithi098=x;smithi139=y;smithi162=z' 2024-09-19T07:55:18.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:17 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:18.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:17 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:18.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:17 smithi162 ceph-mon[35260]: Reconfiguring mon.c (monmap changed)... 2024-09-19T07:55:18.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:17 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T07:55:18.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:17 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T07:55:18.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:17 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:18.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:17 smithi162 ceph-mon[35260]: Reconfiguring daemon mon.c on smithi162 2024-09-19T07:55:18.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:17 smithi162 ceph-mon[35260]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:18.136 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T07:55:18.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:17 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:18.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:17 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:18.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:17 smithi139 ceph-mon[34874]: Reconfiguring mon.c (monmap changed)... 2024-09-19T07:55:18.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:17 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T07:55:18.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:17 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T07:55:18.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:17 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:18.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:17 smithi139 ceph-mon[34874]: Reconfiguring daemon mon.c on smithi162 2024-09-19T07:55:18.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:17 smithi139 ceph-mon[34874]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:18.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:17 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:18.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:17 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:18.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:17 smithi098 ceph-mon[31333]: Reconfiguring mon.c (monmap changed)... 2024-09-19T07:55:18.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:17 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T07:55:18.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:17 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T07:55:18.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:17 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:18.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:17 smithi098 ceph-mon[31333]: Reconfiguring daemon mon.c on smithi162 2024-09-19T07:55:18.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:17 smithi098 ceph-mon[31333]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:18.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.566+0000 7fdbdc303640 1 -- 172.21.15.162:0/3300965911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbd4101c00 msgr2=0x7fdbd4102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:18.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.566+0000 7fdbdc303640 1 --2- 172.21.15.162:0/3300965911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbd4101c00 0x7fdbd4102000 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009900 tx=0x7fdbd002a410 comp rx=0 tx=0).stop 2024-09-19T07:55:18.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.566+0000 7fdbdc303640 1 -- 172.21.15.162:0/3300965911 shutdown_connections 2024-09-19T07:55:18.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.566+0000 7fdbdc303640 1 --2- 172.21.15.162:0/3300965911 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbd41037c0 0x7fdbd410a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:18.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.566+0000 7fdbdc303640 1 --2- 172.21.15.162:0/3300965911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd4102e00 0x7fdbd4103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:18.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.566+0000 7fdbdc303640 1 --2- 172.21.15.162:0/3300965911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbd4101c00 0x7fdbd4102000 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:18.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.566+0000 7fdbdc303640 1 -- 172.21.15.162:0/3300965911 >> 172.21.15.162:0/3300965911 conn(0x7fdbd40fd390 msgr2=0x7fdbd40ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:55:18.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.566+0000 7fdbdc303640 1 -- 172.21.15.162:0/3300965911 shutdown_connections 2024-09-19T07:55:18.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.566+0000 7fdbdc303640 1 -- 172.21.15.162:0/3300965911 wait complete. 2024-09-19T07:55:18.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.567+0000 7fdbdc303640 1 Processor -- start 2024-09-19T07:55:18.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.567+0000 7fdbdc303640 1 -- start start 2024-09-19T07:55:18.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.567+0000 7fdbdc303640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbd4101c00 0x7fdbd419a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:18.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.567+0000 7fdbdc303640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbd4102e00 0x7fdbd419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:18.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.567+0000 7fdbdc303640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd41037c0 0x7fdbd41a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:18.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.567+0000 7fdbdc303640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd410d890 con 0x7fdbd41037c0 2024-09-19T07:55:18.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.567+0000 7fdbdc303640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd410d710 con 0x7fdbd4102e00 2024-09-19T07:55:18.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.567+0000 7fdbdc303640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd410da10 con 0x7fdbd4101c00 2024-09-19T07:55:18.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.567+0000 7fdbda09f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbd4101c00 0x7fdbd419a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:18.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.567+0000 7fdbda09f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbd4101c00 0x7fdbd419a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:55402/0 (socket says 172.21.15.162:55402) 2024-09-19T07:55:18.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.567+0000 7fdbda09f640 1 -- 172.21.15.162:0/2582015044 learned_addr learned my addr 172.21.15.162:0/2582015044 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-19T07:55:18.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.567+0000 7fdbda8a0640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd41037c0 0x7fdbd41a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:18.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.568+0000 7fdbd989e640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbd4102e00 0x7fdbd419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:18.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.568+0000 7fdbda09f640 1 -- 172.21.15.162:0/2582015044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbd4102e00 msgr2=0x7fdbd419a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.568+0000 7fdbda09f640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbd4102e00 0x7fdbd419a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.568+0000 7fdbda09f640 1 -- 172.21.15.162:0/2582015044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd41037c0 msgr2=0x7fdbd41a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.568+0000 7fdbda09f640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd41037c0 0x7fdbd41a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.568+0000 7fdbda09f640 1 -- 172.21.15.162:0/2582015044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd41a1d70 con 0x7fdbd4101c00 2024-09-19T07:55:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.568+0000 7fdbd989e640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbd4102e00 0x7fdbd419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:55:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.568+0000 7fdbda09f640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbd4101c00 0x7fdbd419a140 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0031610 tx=0x7fdbd00043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:55:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.568+0000 7fdbcb7fe640 1 -- 172.21.15.162:0/2582015044 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd003f030 con 0x7fdbd4101c00 2024-09-19T07:55:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.568+0000 7fdbcb7fe640 1 -- 172.21.15.162:0/2582015044 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbd0003ce0 con 0x7fdbd4101c00 2024-09-19T07:55:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.568+0000 7fdbdc303640 1 -- 172.21.15.162:0/2582015044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd41a2000 con 0x7fdbd4101c00 2024-09-19T07:55:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.568+0000 7fdbcb7fe640 1 -- 172.21.15.162:0/2582015044 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0035070 con 0x7fdbd4101c00 2024-09-19T07:55:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.568+0000 7fdbdc303640 1 -- 172.21.15.162:0/2582015044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbd41a24e0 con 0x7fdbd4101c00 2024-09-19T07:55:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.569+0000 7fdbcb7fe640 1 -- 172.21.15.162:0/2582015044 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fdbd0042030 con 0x7fdbd4101c00 2024-09-19T07:55:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.569+0000 7fdbdc303640 1 -- 172.21.15.162:0/2582015044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbd41a2800 con 0x7fdbd4101c00 2024-09-19T07:55:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.569+0000 7fdbcb7fe640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdbc0039500 0x7fdbc003b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:18.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.570+0000 7fdbcb7fe640 1 -- 172.21.15.162:0/2582015044 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdbd006c9d0 con 0x7fdbd4101c00 2024-09-19T07:55:18.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.570+0000 7fdbd989e640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdbc0039500 0x7fdbc003b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:18.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.570+0000 7fdbd989e640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdbc0039500 0x7fdbc003b9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4005d00 tx=0x7fdbc4005ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:55:18.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.574+0000 7fdbcb7fe640 1 -- 172.21.15.162:0/2582015044 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fdbd003bd70 con 0x7fdbd4101c00 2024-09-19T07:55:18.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.753+0000 7fdbdc303640 1 -- 172.21.15.162:0/2582015044 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi139=y;smithi162=z", "target": ["mon-mgr", ""]}) -- 0x7fdbd4109cf0 con 0x7fdbc0039500 2024-09-19T07:55:18.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.759+0000 7fdbcb7fe640 1 -- 172.21.15.162:0/2582015044 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fdbd4109cf0 con 0x7fdbc0039500 2024-09-19T07:55:18.759 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mgr update... 2024-09-19T07:55:18.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.761+0000 7fdbdc303640 1 -- 172.21.15.162:0/2582015044 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdbc0039500 msgr2=0x7fdbc003b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:18.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.761+0000 7fdbdc303640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdbc0039500 0x7fdbc003b9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4005d00 tx=0x7fdbc4005ed0 comp rx=0 tx=0).stop 2024-09-19T07:55:18.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.761+0000 7fdbdc303640 1 -- 172.21.15.162:0/2582015044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbd4101c00 msgr2=0x7fdbd419a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:18.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.761+0000 7fdbdc303640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbd4101c00 0x7fdbd419a140 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0031610 tx=0x7fdbd00043b0 comp rx=0 tx=0).stop 2024-09-19T07:55:18.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.761+0000 7fdbdc303640 1 -- 172.21.15.162:0/2582015044 shutdown_connections 2024-09-19T07:55:18.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.761+0000 7fdbdc303640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdbc0039500 0x7fdbc003b9c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:18.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.761+0000 7fdbdc303640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd41037c0 0x7fdbd41a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:18.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.761+0000 7fdbdc303640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbd4102e00 0x7fdbd419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:18.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.761+0000 7fdbdc303640 1 --2- 172.21.15.162:0/2582015044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbd4101c00 0x7fdbd419a140 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:18.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.761+0000 7fdbdc303640 1 -- 172.21.15.162:0/2582015044 >> 172.21.15.162:0/2582015044 conn(0x7fdbd40fd390 msgr2=0x7fdbd41085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:55:18.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.761+0000 7fdbdc303640 1 -- 172.21.15.162:0/2582015044 shutdown_connections 2024-09-19T07:55:18.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:55:18.761+0000 7fdbdc303640 1 -- 172.21.15.162:0/2582015044 wait complete. 2024-09-19T07:55:19.343 DEBUG:teuthology.orchestra.run.smithi139:mgr.y> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mgr.y.service 2024-09-19T07:55:19.346 DEBUG:teuthology.orchestra.run.smithi162:mgr.z> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mgr.z.service 2024-09-19T07:55:19.348 INFO:tasks.cephadm:Deploying OSDs... 2024-09-19T07:55:19.348 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T07:55:19.348 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-09-19T07:55:19.368 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-19T07:55:19.368 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-09-19T07:55:19.428 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-19T07:55:19.428 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:55:19.428 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-19T07:55:19.428 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:55:19.428 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T07:55:19.428 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-19 07:54:36.322354240 +0000 2024-09-19T07:55:19.428 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-19 07:49:51.524088293 +0000 2024-09-19T07:55:19.429 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-19 07:49:51.524088293 +0000 2024-09-19T07:55:19.429 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-09-19 07:49:51.524088293 +0000 2024-09-19T07:55:19.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-19T07:55:19.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.430 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.430 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:19.430 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:19.430 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:19.430 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.430 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.430 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:19.430 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:19.430 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:19.430 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.431 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T07:55:19.431 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-19T07:55:19.431 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T07:55:19.431 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:19.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:19.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:19.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:19.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:19.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:19.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:19.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T07:55:19.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-19T07:55:19.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T07:55:19.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:19.460 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-19T07:55:19.460 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-19T07:55:19.460 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000228747 s, 2.2 MB/s 2024-09-19T07:55:19.461 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-19T07:55:19.515 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.515 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.515 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:19.515 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:19.515 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:19.515 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.515 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.515 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:19.515 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:19.515 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:19.515 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:19.516 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T07:55:19.516 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-19T07:55:19.516 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T07:55:19.516 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:19.520 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-09-19T07:55:19.577 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-19T07:55:19.577 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:55:19.577 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-19T07:55:19.577 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:55:19.577 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T07:55:19.577 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-19 07:54:36.359353773 +0000 2024-09-19T07:55:19.577 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-19 07:49:51.850084121 +0000 2024-09-19T07:55:19.577 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-19 07:49:51.850084121 +0000 2024-09-19T07:55:19.578 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-09-19 07:49:51.850084121 +0000 2024-09-19T07:55:19.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-19T07:55:19.641 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-19T07:55:19.641 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-19T07:55:19.641 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000226367 s, 2.3 MB/s 2024-09-19T07:55:19.642 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-19T07:55:19.700 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-09-19T07:55:19.756 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-19T07:55:19.756 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:55:19.756 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-19T07:55:19.756 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:55:19.756 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T07:55:19.756 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-19 07:54:36.396353305 +0000 2024-09-19T07:55:19.756 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-19 07:49:52.196079694 +0000 2024-09-19T07:55:19.756 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-19 07:49:52.196079694 +0000 2024-09-19T07:55:19.756 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-09-19 07:49:52.196079694 +0000 2024-09-19T07:55:19.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-19T07:55:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-19T07:55:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-19T07:55:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000199152 s, 2.6 MB/s 2024-09-19T07:55:19.821 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-19T07:55:19.878 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-09-19T07:55:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-19T07:55:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:55:19.934 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-19T07:55:19.934 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:55:19.934 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T07:55:19.934 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-19 07:54:36.433352838 +0000 2024-09-19T07:55:19.935 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-19 07:49:52.525075483 +0000 2024-09-19T07:55:19.935 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-19 07:49:52.525075483 +0000 2024-09-19T07:55:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-09-19 07:49:52.525075483 +0000 2024-09-19T07:55:19.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-19T07:55:19.998 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-19T07:55:19.998 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-19T07:55:19.998 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000236245 s, 2.2 MB/s 2024-09-19T07:55:19.999 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-19T07:55:20.056 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-19T07:55:20.056 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/scratch_devs of=/dev/stdout 2024-09-19T07:55:20.076 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-19T07:55:20.076 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_1 2024-09-19T07:55:20.135 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-19T07:55:20.136 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:55:20.136 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-19T07:55:20.136 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:55:20.136 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T07:55:20.136 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-09-19 07:55:06.893387007 +0000 2024-09-19T07:55:20.137 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-09-19 07:50:06.570170688 +0000 2024-09-19T07:55:20.137 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-09-19 07:50:06.570170688 +0000 2024-09-19T07:55:20.137 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-09-19 07:50:06.570170688 +0000 2024-09-19T07:55:20.137 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-19T07:55:20.203 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-09-19T07:55:20.204 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-09-19T07:55:20.204 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000284842 s, 1.8 MB/s 2024-09-19T07:55:20.205 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-19T07:55:20.264 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_2 2024-09-19T07:55:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-19T07:55:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:55:20.321 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-19T07:55:20.321 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:55:20.321 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T07:55:20.321 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-09-19 07:55:06.893387007 +0000 2024-09-19T07:55:20.321 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-09-19 07:50:06.892163705 +0000 2024-09-19T07:55:20.321 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-09-19 07:50:06.892163705 +0000 2024-09-19T07:55:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-09-19 07:50:06.892163705 +0000 2024-09-19T07:55:20.322 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-19T07:55:20.386 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-09-19T07:55:20.386 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-09-19T07:55:20.386 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000240959 s, 2.1 MB/s 2024-09-19T07:55:20.387 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-19T07:55:20.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:20 smithi139 ceph-mon[34874]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi139=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:55:20.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:20 smithi139 ceph-mon[34874]: Saving service mgr spec with placement smithi098=x;smithi139=y;smithi162=z;count:3 2024-09-19T07:55:20.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:20 smithi139 ceph-mon[34874]: Deploying daemon mgr.z on smithi162 2024-09-19T07:55:20.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:20 smithi139 ceph-mon[34874]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:20.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:20 smithi098 ceph-mon[31333]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi139=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:55:20.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:20 smithi098 ceph-mon[31333]: Saving service mgr spec with placement smithi098=x;smithi139=y;smithi162=z;count:3 2024-09-19T07:55:20.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:20 smithi098 ceph-mon[31333]: Deploying daemon mgr.z on smithi162 2024-09-19T07:55:20.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:20 smithi098 ceph-mon[31333]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:20.437 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:20 smithi162 ceph-mon[35260]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi139=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:55:20.437 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:20 smithi162 ceph-mon[35260]: Saving service mgr spec with placement smithi098=x;smithi139=y;smithi162=z;count:3 2024-09-19T07:55:20.437 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:20 smithi162 ceph-mon[35260]: Deploying daemon mgr.z on smithi162 2024-09-19T07:55:20.438 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:20 smithi162 ceph-mon[35260]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:20.444 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_3 2024-09-19T07:55:20.501 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-19T07:55:20.501 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:55:20.501 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-19T07:55:20.501 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:55:20.501 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T07:55:20.501 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-09-19 07:55:06.893387007 +0000 2024-09-19T07:55:20.501 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-09-19 07:50:07.227156439 +0000 2024-09-19T07:55:20.501 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-09-19 07:50:07.227156439 +0000 2024-09-19T07:55:20.502 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-09-19 07:50:07.227156439 +0000 2024-09-19T07:55:20.502 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-19T07:55:20.565 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-09-19T07:55:20.565 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-09-19T07:55:20.565 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000178483 s, 2.9 MB/s 2024-09-19T07:55:20.566 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-19T07:55:20.623 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_4 2024-09-19T07:55:20.678 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-19T07:55:20.678 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:55:20.678 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-09-19T07:55:20.678 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:55:20.678 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T07:55:20.678 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-09-19 07:55:06.894386986 +0000 2024-09-19T07:55:20.678 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-09-19 07:50:07.556149304 +0000 2024-09-19T07:55:20.679 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-09-19 07:50:07.556149304 +0000 2024-09-19T07:55:20.679 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-09-19 07:50:07.556149304 +0000 2024-09-19T07:55:20.679 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-19T07:55:20.744 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-09-19T07:55:20.744 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-09-19T07:55:20.744 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000201242 s, 2.5 MB/s 2024-09-19T07:55:20.745 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-19T07:55:20.803 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-19T07:55:20.803 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-09-19T07:55:20.820 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-19T07:55:20.820 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-09-19T07:55:20.876 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-19T07:55:20.877 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:55:20.877 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-19T07:55:20.877 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:55:20.877 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T07:55:20.877 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-19 07:55:20.812509139 +0000 2024-09-19T07:55:20.877 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-19 07:50:05.411343853 +0000 2024-09-19T07:55:20.877 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-19 07:50:05.411343853 +0000 2024-09-19T07:55:20.877 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-09-19 07:50:05.411343853 +0000 2024-09-19T07:55:20.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-19T07:55:20.945 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-19T07:55:20.945 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-19T07:55:20.946 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000212323 s, 2.4 MB/s 2024-09-19T07:55:20.947 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-19T07:55:21.005 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-09-19T07:55:21.060 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-19T07:55:21.061 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:55:21.061 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-09-19T07:55:21.061 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:55:21.061 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T07:55:21.061 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-19 07:55:20.813509119 +0000 2024-09-19T07:55:21.061 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-19 07:50:05.755336584 +0000 2024-09-19T07:55:21.061 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-19 07:50:05.755336584 +0000 2024-09-19T07:55:21.061 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-09-19 07:50:05.755336584 +0000 2024-09-19T07:55:21.061 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-19T07:55:21.125 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-19T07:55:21.129 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-19T07:55:21.129 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000196606 s, 2.6 MB/s 2024-09-19T07:55:21.130 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-19T07:55:21.186 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-09-19T07:55:21.241 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-19T07:55:21.242 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:55:21.242 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-19T07:55:21.242 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:55:21.242 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T07:55:21.242 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-19 07:55:20.813509119 +0000 2024-09-19T07:55:21.242 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-19 07:50:06.085329611 +0000 2024-09-19T07:55:21.242 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-19 07:50:06.085329611 +0000 2024-09-19T07:55:21.242 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-09-19 07:50:06.085329611 +0000 2024-09-19T07:55:21.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-19T07:55:21.305 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-19T07:55:21.305 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-19T07:55:21.305 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000191572 s, 2.7 MB/s 2024-09-19T07:55:21.306 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-19T07:55:21.363 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-09-19T07:55:21.370 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:21 smithi162 podman[36915]: 2024-09-19 07:55:21.345352615 +0000 UTC m=+0.362791458 container start cf1f1c2f4f8eb6792292902fe23e0a57a48761e094d4aed064e727fb8082f199 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-2f599a4, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-19T07:55:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-19T07:55:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:55:21.385 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-19T07:55:21.385 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:55:21.385 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T07:55:21.385 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-19 07:55:20.814509098 +0000 2024-09-19T07:55:21.385 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-19 07:50:06.400322955 +0000 2024-09-19T07:55:21.385 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-19 07:50:06.400322955 +0000 2024-09-19T07:55:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-09-19 07:50:06.400322955 +0000 2024-09-19T07:55:21.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-19T07:55:21.450 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-19T07:55:21.451 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-19T07:55:21.451 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000181695 s, 2.8 MB/s 2024-09-19T07:55:21.452 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-19T07:55:21.508 INFO:tasks.cephadm:Deploying osd.0 on smithi098 with /dev/vg_nvme/lv_4... 2024-09-19T07:55:21.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-19T07:55:21.680 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:21 smithi162 ceph-mgr[37040]: -- 172.21.15.162:0/2182518152 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a7e3c210e0 con 0x55a7e3bfe800 2024-09-19T07:55:21.680 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:21 smithi162 bash[36915]: cf1f1c2f4f8eb6792292902fe23e0a57a48761e094d4aed064e727fb8082f199 2024-09-19T07:55:21.680 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:21 smithi162 systemd[1]: Started Ceph mgr.z for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T07:55:21.680 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:21 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:21.471+0000 7ff8264b2040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T07:55:21.680 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:21 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:21.532+0000 7ff8264b2040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T07:55:21.681 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:21 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:21.679+0000 7ff8264b2040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T07:55:21.758 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:55:22.028 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:21 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:21.745+0000 7ff8264b2040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T07:55:22.028 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:21 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:21.890+0000 7ff8264b2040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T07:55:22.028 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:22 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:22.027+0000 7ff8264b2040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T07:55:22.620 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:22 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:22.338+0000 7ff8264b2040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T07:55:22.620 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:22 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:22.398+0000 7ff8264b2040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T07:55:22.620 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:22 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:22.582+0000 7ff8264b2040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T07:55:22.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:22 smithi162 ceph-mon[35260]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:22.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:22 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:22.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:22 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:22.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:22 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:22.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:22 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T07:55:22.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:22 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-19T07:55:22.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:22 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T07:55:22.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:22 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:22.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:22 smithi162 ceph-mon[35260]: Deploying daemon mgr.y on smithi139 2024-09-19T07:55:22.819 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:22 smithi098 ceph-mon[31333]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:22.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:22 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:22.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:22 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:22.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:22 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:22.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:22 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T07:55:22.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:22 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-19T07:55:22.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:22 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T07:55:22.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:22 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:22.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:22 smithi098 ceph-mon[31333]: Deploying daemon mgr.y on smithi139 2024-09-19T07:55:22.852 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:22 smithi139 ceph-mon[34874]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:22.853 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:22 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:22.853 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:22 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:22.853 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:22 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:22.853 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:22 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T07:55:22.853 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:22 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-19T07:55:22.853 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:22 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T07:55:22.853 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:22 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:22.853 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:22 smithi139 ceph-mon[34874]: Deploying daemon mgr.y on smithi139 2024-09-19T07:55:23.427 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:23 smithi139 systemd[1]: Starting Ceph mgr.y for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T07:55:23.765 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:23 smithi139 podman[36259]: 2024-09-19 07:55:23.40870186 +0000 UTC m=+0.015448512 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T07:55:23.765 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:23 smithi139 podman[36259]: 2024-09-19 07:55:23.547007826 +0000 UTC m=+0.153754472 container create dcdc610cf1f4703ebd0ebb3faa815a9c0b88ff5eb98dc8317aeebdad612973f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731) 2024-09-19T07:55:23.765 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:23 smithi139 podman[36259]: 2024-09-19 07:55:23.76584183 +0000 UTC m=+0.372588475 container init dcdc610cf1f4703ebd0ebb3faa815a9c0b88ff5eb98dc8317aeebdad612973f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-19T07:55:23.870 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:23.454+0000 7ff8264b2040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T07:55:23.870 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:23.521+0000 7ff8264b2040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T07:55:24.065 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:55:24.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-09-19T07:55:24.107 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:23 smithi139 podman[36259]: 2024-09-19 07:55:23.769327329 +0000 UTC m=+0.376073977 container start dcdc610cf1f4703ebd0ebb3faa815a9c0b88ff5eb98dc8317aeebdad612973f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-2f599a4, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-19T07:55:24.107 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:23 smithi139 ceph-mgr[36278]: -- 172.21.15.139:0/819506202 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bc318bf0e0 con 0x55bc3189d000 2024-09-19T07:55:24.107 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:23 smithi139 bash[36259]: dcdc610cf1f4703ebd0ebb3faa815a9c0b88ff5eb98dc8317aeebdad612973f1 2024-09-19T07:55:24.107 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:23 smithi139 systemd[1]: Started Ceph mgr.y for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T07:55:24.107 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:23.892+0000 7f840fa51040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T07:55:24.107 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:23.949+0000 7f840fa51040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T07:55:24.107 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:24 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:24.106+0000 7f840fa51040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T07:55:24.333 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:55:24.427 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:24 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:24.175+0000 7f840fa51040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T07:55:24.427 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:24 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:24.338+0000 7f840fa51040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T07:55:24.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:24 smithi162 ceph-mon[35260]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:24.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:24 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:24.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:24 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:24.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:24 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:24.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:24 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:24.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:24 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:24.530 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:24 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:24.252+0000 7ff8264b2040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-19T07:55:24.530 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:24 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:24.384+0000 7ff8264b2040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T07:55:24.530 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:24 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:24.529+0000 7ff8264b2040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T07:55:24.715 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:24 smithi098 ceph-mon[31333]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:24.715 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:24 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:24.716 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:24 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:24.716 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:24 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:24.716 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:24 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:24.716 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:24 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:24.817 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:24 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:24.487+0000 7f840fa51040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T07:55:24.817 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:24 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:24.813+0000 7f840fa51040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T07:55:24.817 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:24 smithi139 ceph-mon[34874]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:24.817 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:24 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:24.818 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:24 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:24.818 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:24 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:24.818 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:24 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:24.818 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:24 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:24.820 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:24 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:24.591+0000 7ff8264b2040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T07:55:24.820 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:24 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:24.751+0000 7ff8264b2040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-19T07:55:24.820 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:24 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:24.819+0000 7ff8264b2040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T07:55:24.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.905+0000 7f7d9ed54640 1 -- 172.21.15.98:0/2958693725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d98101c00 msgr2=0x7f7d98102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:24.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.905+0000 7f7d9ed54640 1 --2- 172.21.15.98:0/2958693725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d98101c00 0x7f7d98102000 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c009900 tx=0x7f7d8c02a410 comp rx=0 tx=0).stop 2024-09-19T07:55:24.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 -- 172.21.15.98:0/2958693725 shutdown_connections 2024-09-19T07:55:24.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 --2- 172.21.15.98:0/2958693725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7d981037c0 0x7f7d9810a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:24.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 --2- 172.21.15.98:0/2958693725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d98102e00 0x7f7d98103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:24.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 --2- 172.21.15.98:0/2958693725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d98101c00 0x7f7d98102000 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:24.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 -- 172.21.15.98:0/2958693725 >> 172.21.15.98:0/2958693725 conn(0x7f7d980fd390 msgr2=0x7f7d980ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:55:24.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 -- 172.21.15.98:0/2958693725 shutdown_connections 2024-09-19T07:55:24.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 -- 172.21.15.98:0/2958693725 wait complete. 2024-09-19T07:55:24.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 Processor -- start 2024-09-19T07:55:24.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 -- start start 2024-09-19T07:55:24.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d98101c00 0x7f7d98197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:24.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d98102e00 0x7f7d98198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:24.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7d981037c0 0x7f7d9819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:24.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9810d880 con 0x7f7d98102e00 2024-09-19T07:55:24.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9810d700 con 0x7f7d981037c0 2024-09-19T07:55:24.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.906+0000 7f7d9ed54640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9810da00 con 0x7f7d98101c00 2024-09-19T07:55:24.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d97fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d98102e00 0x7f7d98198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:24.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d97fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d98102e00 0x7f7d98198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38766/0 (socket says 172.21.15.98:38766) 2024-09-19T07:55:24.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d97fff640 1 -- 172.21.15.98:0/3959610648 learned_addr learned my addr 172.21.15.98:0/3959610648 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:55:24.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d9caf0640 1 --2- 172.21.15.98:0/3959610648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d98101c00 0x7f7d98197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:24.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d9d2f1640 1 --2- 172.21.15.98:0/3959610648 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7d981037c0 0x7f7d9819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:24.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d97fff640 1 -- 172.21.15.98:0/3959610648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d98101c00 msgr2=0x7f7d98197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:24.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d97fff640 1 --2- 172.21.15.98:0/3959610648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d98101c00 0x7f7d98197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:24.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d97fff640 1 -- 172.21.15.98:0/3959610648 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7d981037c0 msgr2=0x7f7d9819f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:24.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d97fff640 1 --2- 172.21.15.98:0/3959610648 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7d981037c0 0x7f7d9819f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:24.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d97fff640 1 -- 172.21.15.98:0/3959610648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d9819fb00 con 0x7f7d98102e00 2024-09-19T07:55:24.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d97fff640 1 --2- 172.21.15.98:0/3959610648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d98102e00 0x7f7d98198410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7d8400e9f0 tx=0x7f7d8400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:55:24.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d95ffb640 1 -- 172.21.15.98:0/3959610648 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8400cd80 con 0x7f7d98102e00 2024-09-19T07:55:24.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d95ffb640 1 -- 172.21.15.98:0/3959610648 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d84005500 con 0x7f7d98102e00 2024-09-19T07:55:24.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d9ed54640 1 -- 172.21.15.98:0/3959610648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d9819fdf0 con 0x7f7d98102e00 2024-09-19T07:55:24.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d95ffb640 1 -- 172.21.15.98:0/3959610648 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d84010690 con 0x7f7d98102e00 2024-09-19T07:55:24.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.907+0000 7f7d9ed54640 1 -- 172.21.15.98:0/3959610648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d981a0250 con 0x7f7d98102e00 2024-09-19T07:55:24.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.908+0000 7f7d95ffb640 1 -- 172.21.15.98:0/3959610648 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f7d84010830 con 0x7f7d98102e00 2024-09-19T07:55:24.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.908+0000 7f7d95ffb640 1 --2- 172.21.15.98:0/3959610648 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7d88039500 0x7f7d8803b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:24.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.909+0000 7f7d95ffb640 1 -- 172.21.15.98:0/3959610648 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7d8404cb30 con 0x7f7d98102e00 2024-09-19T07:55:24.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.909+0000 7f7d9caf0640 1 --2- 172.21.15.98:0/3959610648 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7d88039500 0x7f7d8803b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:24.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.909+0000 7f7d9ed54640 1 -- 172.21.15.98:0/3959610648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d98108ff0 con 0x7f7d98102e00 2024-09-19T07:55:24.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.909+0000 7f7d9caf0640 1 --2- 172.21.15.98:0/3959610648 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7d88039500 0x7f7d8803b9c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c005ab0 tx=0x7f7d8c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:55:24.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:24.914+0000 7f7d95ffb640 1 -- 172.21.15.98:0/3959610648 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f7d840203a0 con 0x7f7d98102e00 2024-09-19T07:55:25.074 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:24 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:24.874+0000 7f840fa51040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T07:55:25.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:25.091+0000 7f7d9ed54640 1 -- 172.21.15.98:0/3959610648 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7d98062630 con 0x7f7d88039500 2024-09-19T07:55:25.112 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:24 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:24.880+0000 7ff8264b2040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T07:55:25.112 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:25.039+0000 7ff8264b2040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T07:55:25.112 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:25.111+0000 7ff8264b2040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T07:55:25.370 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:25.180+0000 7ff8264b2040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T07:55:25.370 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:25.266+0000 7ff8264b2040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T07:55:25.370 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:25.324+0000 7ff8264b2040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T07:55:25.427 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:25.073+0000 7f840fa51040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T07:55:25.870 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:25.383+0000 7ff8264b2040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T07:55:26.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:55:26.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:55:26.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:55:26.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:26.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:26.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:26.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: Reconfiguring mgr.x (unknown last config time)... 2024-09-19T07:55:26.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T07:55:26.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T07:55:26.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:26.184 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: Reconfiguring daemon mgr.x on smithi098 2024-09-19T07:55:26.184 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:26.184 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.370 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 07:55:26 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T07:55:26.166+0000 7ff8264b2040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T07:55:26.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:55:26.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:55:26.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:55:26.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:26.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:26.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:26.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: Reconfiguring mgr.x (unknown last config time)... 2024-09-19T07:55:26.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T07:55:26.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T07:55:26.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:26.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: Reconfiguring daemon mgr.x on smithi098 2024-09-19T07:55:26.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:26.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.427 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:25.950+0000 7f840fa51040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T07:55:26.427 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:26 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:26.017+0000 7f840fa51040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T07:55:26.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:55:26.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:55:26.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:55:26.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:26.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:26.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:26.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: Reconfiguring mgr.x (unknown last config time)... 2024-09-19T07:55:26.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T07:55:26.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T07:55:26.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:26.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: Reconfiguring daemon mgr.x on smithi098 2024-09-19T07:55:26.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:26.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:26.982 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:26 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:26.726+0000 7f840fa51040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-19T07:55:26.983 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:26 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:26.840+0000 7f840fa51040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T07:55:26.983 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:26 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:26.981+0000 7f840fa51040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T07:55:27.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:27.143+0000 7f7d95ffb640 1 -- 172.21.15.98:0/3959610648 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 14) ==== 92408+0+0 (secure 0 0 0) 0x7f7d84007d10 con 0x7f7d98102e00 2024-09-19T07:55:27.260 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-mon[34874]: Standby manager daemon z started 2024-09-19T07:55:27.260 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.162:0/272322325' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-19T07:55:27.260 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.162:0/272322325' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T07:55:27.260 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.162:0/272322325' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-19T07:55:27.261 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.162:0/272322325' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T07:55:27.261 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:27.042+0000 7f840fa51040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T07:55:27.261 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:27.191+0000 7f840fa51040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-19T07:55:27.261 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:27.259+0000 7f840fa51040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T07:55:27.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:27 smithi098 ceph-mon[31333]: Standby manager daemon z started 2024-09-19T07:55:27.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:27 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.162:0/272322325' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-19T07:55:27.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:27 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.162:0/272322325' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T07:55:27.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:27 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.162:0/272322325' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-19T07:55:27.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:27 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.162:0/272322325' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T07:55:27.542 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:27.318+0000 7f840fa51040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T07:55:27.542 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:27.476+0000 7f840fa51040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T07:55:27.542 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:27.540+0000 7f840fa51040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T07:55:27.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:27 smithi162 ceph-mon[35260]: Standby manager daemon z started 2024-09-19T07:55:27.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:27 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.162:0/272322325' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-19T07:55:27.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:27 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.162:0/272322325' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T07:55:27.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:27 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.162:0/272322325' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-19T07:55:27.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:27 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.162:0/272322325' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T07:55:27.805 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:27.606+0000 7f840fa51040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T07:55:27.805 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:27.687+0000 7f840fa51040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T07:55:27.806 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:27.745+0000 7f840fa51040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T07:55:27.806 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:27.804+0000 7f840fa51040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T07:55:28.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:28 smithi098 ceph-mon[31333]: mgrmap e14: x(active, since 61s), standbys: z 2024-09-19T07:55:28.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-19T07:55:28.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:28 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/4184533086' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7d01ce6-165d-493d-ae98-d8ad9e5d4134"}]: dispatch 2024-09-19T07:55:28.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:28 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/4184533086' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7d01ce6-165d-493d-ae98-d8ad9e5d4134"}]': finished 2024-09-19T07:55:28.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:28 smithi098 ceph-mon[31333]: osdmap e5: 1 total, 0 up, 1 in 2024-09-19T07:55:28.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:28.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:28 smithi098 ceph-mon[31333]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:28.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:28 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/4106217330' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:55:28.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:28 smithi139 ceph-mon[34874]: mgrmap e14: x(active, since 61s), standbys: z 2024-09-19T07:55:28.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-19T07:55:28.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:28 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/4184533086' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7d01ce6-165d-493d-ae98-d8ad9e5d4134"}]: dispatch 2024-09-19T07:55:28.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:28 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/4184533086' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7d01ce6-165d-493d-ae98-d8ad9e5d4134"}]': finished 2024-09-19T07:55:28.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:28 smithi139 ceph-mon[34874]: osdmap e5: 1 total, 0 up, 1 in 2024-09-19T07:55:28.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:28.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:28 smithi139 ceph-mon[34874]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:28.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:28 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/4106217330' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:55:28.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:28 smithi162 ceph-mon[35260]: mgrmap e14: x(active, since 61s), standbys: z 2024-09-19T07:55:28.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-19T07:55:28.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:28 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/4184533086' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7d01ce6-165d-493d-ae98-d8ad9e5d4134"}]: dispatch 2024-09-19T07:55:28.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:28 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/4184533086' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7d01ce6-165d-493d-ae98-d8ad9e5d4134"}]': finished 2024-09-19T07:55:28.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:28 smithi162 ceph-mon[35260]: osdmap e5: 1 total, 0 up, 1 in 2024-09-19T07:55:28.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:28.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:28 smithi162 ceph-mon[35260]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:28.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:28 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/4106217330' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:55:28.927 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 07:55:28 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T07:55:28.585+0000 7f840fa51040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T07:55:29.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:29.425+0000 7f7d95ffb640 1 -- 172.21.15.98:0/3959610648 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f7d84021440 con 0x7f7d98102e00 2024-09-19T07:55:29.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:29 smithi139 ceph-mon[34874]: Standby manager daemon y started 2024-09-19T07:55:29.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:29 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.139:0/3192822793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-19T07:55:29.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:29 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.139:0/3192822793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T07:55:29.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:29 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.139:0/3192822793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-19T07:55:29.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:29 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.139:0/3192822793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T07:55:29.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:29 smithi098 ceph-mon[31333]: Standby manager daemon y started 2024-09-19T07:55:29.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:29 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.139:0/3192822793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-19T07:55:29.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:29 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.139:0/3192822793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T07:55:29.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:29 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.139:0/3192822793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-19T07:55:29.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:29 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.139:0/3192822793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T07:55:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:29 smithi162 ceph-mon[35260]: Standby manager daemon y started 2024-09-19T07:55:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:29 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.139:0/3192822793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-19T07:55:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:29 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.139:0/3192822793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T07:55:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:29 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.139:0/3192822793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-19T07:55:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:29 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.139:0/3192822793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T07:55:30.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:30 smithi139 ceph-mon[34874]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-09-19T07:55:30.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:30 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-19T07:55:30.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:30 smithi139 ceph-mon[34874]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:30.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:30 smithi098 ceph-mon[31333]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-09-19T07:55:30.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:30 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-19T07:55:30.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:30 smithi098 ceph-mon[31333]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:30.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:30 smithi162 ceph-mon[35260]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-09-19T07:55:30.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:30 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-19T07:55:30.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:30 smithi162 ceph-mon[35260]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:32.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:32 smithi098 ceph-mon[31333]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:32 smithi162 ceph-mon[35260]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:32.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:32 smithi139 ceph-mon[34874]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:34 smithi098 ceph-mon[31333]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:34 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:34 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:34 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:34.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:34 smithi162 ceph-mon[35260]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:34.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:34 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:34.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:34 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:34.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:34 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:34.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:34 smithi139 ceph-mon[34874]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:34.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:34 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:34.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:34 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:34.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:34 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:35.832 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:35 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:35.832 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:35 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:35.832 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:35 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:35.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:35 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:35.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:35 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:35.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:35 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:35.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:35 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:35.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:35 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:35.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:35 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:36.710 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:36 smithi098 ceph-mon[31333]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:36.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:36 smithi162 ceph-mon[35260]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:36.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:36 smithi139 ceph-mon[34874]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:37.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:37 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-19T07:55:37.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:37 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:37.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:37 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-19T07:55:37.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:37 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:37.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:37 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-19T07:55:37.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:37 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:38.566 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:38 smithi098 ceph-mon[31333]: Deploying daemon osd.0 on smithi098 2024-09-19T07:55:38.566 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:38 smithi098 ceph-mon[31333]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:38.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:38 smithi162 ceph-mon[35260]: Deploying daemon osd.0 on smithi098 2024-09-19T07:55:38.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:38 smithi162 ceph-mon[35260]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:38.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:38 smithi139 ceph-mon[34874]: Deploying daemon osd.0 on smithi098 2024-09-19T07:55:38.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:38 smithi139 ceph-mon[34874]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:40.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:40 smithi162 ceph-mon[35260]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:40.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:40 smithi139 ceph-mon[34874]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:40.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:40 smithi098 ceph-mon[31333]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:42.833 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:42 smithi098 ceph-mon[31333]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:42 smithi162 ceph-mon[35260]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:42.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:42 smithi139 ceph-mon[34874]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:43.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:43 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:43.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:43 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:43.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:43 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:43.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:43 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:43.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:43 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:43.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:43 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:43.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:43 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:43.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:43 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:43.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:43 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:44.680 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:44 smithi098 ceph-mon[31333]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:44.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:44 smithi162 ceph-mon[35260]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:44.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:44 smithi139 ceph-mon[34874]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:46.835 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:46 smithi098 ceph-mon[31333]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:46.835 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:46.835 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:46.835 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:46.835 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:46.835 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:46.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:46 smithi162 ceph-mon[35260]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:46.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:46.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:46.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:46.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:46.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:46.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:46 smithi139 ceph-mon[34874]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:46.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:46.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:46.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:46.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:46.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:47.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:47.149+0000 7f7d95ffb640 1 -- 172.21.15.98:0/3959610648 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7d98062630 con 0x7f7d88039500 2024-09-19T07:55:47.150 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 0 on host 'smithi098' 2024-09-19T07:55:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:47.152+0000 7f7d9ed54640 1 -- 172.21.15.98:0/3959610648 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7d88039500 msgr2=0x7f7d8803b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:47.152+0000 7f7d9ed54640 1 --2- 172.21.15.98:0/3959610648 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7d88039500 0x7f7d8803b9c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c005ab0 tx=0x7f7d8c000bc0 comp rx=0 tx=0).stop 2024-09-19T07:55:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:47.152+0000 7f7d9ed54640 1 -- 172.21.15.98:0/3959610648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d98102e00 msgr2=0x7f7d98198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:47.152+0000 7f7d9ed54640 1 --2- 172.21.15.98:0/3959610648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d98102e00 0x7f7d98198410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7d8400e9f0 tx=0x7f7d8400edf0 comp rx=0 tx=0).stop 2024-09-19T07:55:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:47.152+0000 7f7d9ed54640 1 -- 172.21.15.98:0/3959610648 shutdown_connections 2024-09-19T07:55:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:47.152+0000 7f7d9ed54640 1 --2- 172.21.15.98:0/3959610648 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7d88039500 0x7f7d8803b9c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:47.152+0000 7f7d9ed54640 1 --2- 172.21.15.98:0/3959610648 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7d981037c0 0x7f7d9819f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:47.152+0000 7f7d9ed54640 1 --2- 172.21.15.98:0/3959610648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d98102e00 0x7f7d98198410 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:47.152+0000 7f7d9ed54640 1 --2- 172.21.15.98:0/3959610648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d98101c00 0x7f7d98197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:47.152+0000 7f7d9ed54640 1 -- 172.21.15.98:0/3959610648 >> 172.21.15.98:0/3959610648 conn(0x7f7d980fd390 msgr2=0x7f7d981061c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:55:47.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:47.152+0000 7f7d9ed54640 1 -- 172.21.15.98:0/3959610648 shutdown_connections 2024-09-19T07:55:47.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:55:47.152+0000 7f7d9ed54640 1 -- 172.21.15.98:0/3959610648 wait complete. 2024-09-19T07:55:47.732 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:47 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:47.732 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:47 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:47.732 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:47 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:47.734 DEBUG:teuthology.orchestra.run.smithi098:osd.0> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.0.service 2024-09-19T07:55:47.737 INFO:tasks.cephadm:Deploying osd.1 on smithi139 with /dev/vg_nvme/lv_4... 2024-09-19T07:55:47.738 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-19T07:55:47.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:47 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:47.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:47 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:47.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:47 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:47.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:47 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:47.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:47 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:47.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:47 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:47.988 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T07:55:48.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:48 smithi139 ceph-mon[34874]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:48.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:48 smithi162 ceph-mon[35260]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:48.904 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:48 smithi098 ceph-mon[31333]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:49.182 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 07:55:49 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0[45294]: 2024-09-19T07:55:49.028+0000 7f7b41b0d640 -1 osd.0 0 log_to_monitors true 2024-09-19T07:55:49.662 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:49 smithi139 ceph-mon[34874]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-19T07:55:49.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:49 smithi162 ceph-mon[35260]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-19T07:55:49.875 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:49 smithi098 ceph-mon[31333]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-19T07:55:50.258 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:55:50.286 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_4 2024-09-19T07:55:50.533 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T07:55:50.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:50 smithi098 ceph-mon[31333]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:50.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:50 smithi098 ceph-mon[31333]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-19T07:55:50.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:50 smithi098 ceph-mon[31333]: osdmap e6: 1 total, 0 up, 1 in 2024-09-19T07:55:50.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:50 smithi098 ceph-mon[31333]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-19T07:55:50.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:50 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:50.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:50 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:50.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:50 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:50.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:50 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:55:50.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:50 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:50.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:50 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:50.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:50 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:50.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:50 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:50.891 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:50 smithi139 ceph-mon[34874]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:50.891 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:50 smithi139 ceph-mon[34874]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-19T07:55:50.891 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:50 smithi139 ceph-mon[34874]: osdmap e6: 1 total, 0 up, 1 in 2024-09-19T07:55:50.891 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:50 smithi139 ceph-mon[34874]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-19T07:55:50.892 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:50 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:50.892 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:50 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:50.892 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:50 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:50.892 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:50 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:55:50.892 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:50 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:50.892 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:50 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:50.892 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:50 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:50.892 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:50 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:50.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.976+0000 7f1994d78640 1 -- 172.21.15.139:0/1675270117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1990100da0 msgr2=0x7f1990101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:50.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.976+0000 7f1994d78640 1 --2- 172.21.15.139:0/1675270117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1990100da0 0x7f1990101220 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1978009900 tx=0x7f197802a410 comp rx=0 tx=0).stop 2024-09-19T07:55:50.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.976+0000 7f1994d78640 1 -- 172.21.15.139:0/1675270117 shutdown_connections 2024-09-19T07:55:50.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.976+0000 7f1994d78640 1 --2- 172.21.15.139:0/1675270117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1990101760 0x7f1990109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:50.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.976+0000 7f1994d78640 1 --2- 172.21.15.139:0/1675270117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1990100da0 0x7f1990101220 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:50.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.976+0000 7f1994d78640 1 --2- 172.21.15.139:0/1675270117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19901020f0 0x7f19901024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:50.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.976+0000 7f1994d78640 1 -- 172.21.15.139:0/1675270117 >> 172.21.15.139:0/1675270117 conn(0x7f19900fba40 msgr2=0x7f19900fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:55:50.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.976+0000 7f1994d78640 1 -- 172.21.15.139:0/1675270117 shutdown_connections 2024-09-19T07:55:50.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.976+0000 7f1994d78640 1 -- 172.21.15.139:0/1675270117 wait complete. 2024-09-19T07:55:50.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1994d78640 1 Processor -- start 2024-09-19T07:55:50.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1994d78640 1 -- start start 2024-09-19T07:55:50.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1994d78640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1990100da0 0x7f1990197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:50.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1994d78640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1990101760 0x7f19901984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:50.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1994d78640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19901020f0 0x7f199019f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:50.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1994d78640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f199010c790 con 0x7f1990100da0 2024-09-19T07:55:50.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1994d78640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f199010c610 con 0x7f1990101760 2024-09-19T07:55:50.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1994d78640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f199010c910 con 0x7f19901020f0 2024-09-19T07:55:50.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1987fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1990101760 0x7f19901984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:50.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1987fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1990101760 0x7f19901984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:49048/0 (socket says 172.21.15.139:49048) 2024-09-19T07:55:50.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1987fff640 1 -- 172.21.15.139:0/2776292408 learned_addr learned my addr 172.21.15.139:0/2776292408 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-09-19T07:55:50.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f198e59c640 1 --2- 172.21.15.139:0/2776292408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1990100da0 0x7f1990197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:50.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f198ed9d640 1 --2- 172.21.15.139:0/2776292408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19901020f0 0x7f199019f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:50.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1987fff640 1 -- 172.21.15.139:0/2776292408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19901020f0 msgr2=0x7f199019f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:50.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1987fff640 1 --2- 172.21.15.139:0/2776292408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19901020f0 0x7f199019f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:50.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1987fff640 1 -- 172.21.15.139:0/2776292408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1990100da0 msgr2=0x7f1990197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:55:50.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1987fff640 1 --2- 172.21.15.139:0/2776292408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1990100da0 0x7f1990197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:55:50.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1987fff640 1 -- 172.21.15.139:0/2776292408 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199019fba0 con 0x7f1990101760 2024-09-19T07:55:50.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.977+0000 7f1987fff640 1 --2- 172.21.15.139:0/2776292408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1990101760 0x7f19901984b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1978031470 tx=0x7f1978032c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:55:50.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.978+0000 7f19877fe640 1 -- 172.21.15.139:0/2776292408 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197803f030 con 0x7f1990101760 2024-09-19T07:55:50.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.978+0000 7f1994d78640 1 -- 172.21.15.139:0/2776292408 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199019fe30 con 0x7f1990101760 2024-09-19T07:55:50.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.978+0000 7f19877fe640 1 -- 172.21.15.139:0/2776292408 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1978003ce0 con 0x7f1990101760 2024-09-19T07:55:50.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.978+0000 7f19877fe640 1 -- 172.21.15.139:0/2776292408 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1978035070 con 0x7f1990101760 2024-09-19T07:55:50.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.978+0000 7f1994d78640 1 -- 172.21.15.139:0/2776292408 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19901a0290 con 0x7f1990101760 2024-09-19T07:55:50.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.979+0000 7f1994d78640 1 -- 172.21.15.139:0/2776292408 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19901a0580 con 0x7f1990101760 2024-09-19T07:55:50.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.980+0000 7f19877fe640 1 -- 172.21.15.139:0/2776292408 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f1978042030 con 0x7f1990101760 2024-09-19T07:55:50.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.981+0000 7f19877fe640 1 --2- 172.21.15.139:0/2776292408 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f197c0a31e0 0x7f197c0a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:55:50.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.981+0000 7f19877fe640 1 -- 172.21.15.139:0/2776292408 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1487+0+0 (secure 0 0 0) 0x7f19780ee1a0 con 0x7f1990101760 2024-09-19T07:55:50.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.981+0000 7f198e59c640 1 --2- 172.21.15.139:0/2776292408 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f197c0a31e0 0x7f197c0a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:55:50.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.981+0000 7f198e59c640 1 --2- 172.21.15.139:0/2776292408 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f197c0a31e0 0x7f197c0a56a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1980005fd0 tx=0x7f198000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:55:50.986 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:50.985+0000 7f19877fe640 1 -- 172.21.15.139:0/2776292408 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f19780bc400 con 0x7f1990101760 2024-09-19T07:55:51.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:50 smithi162 ceph-mon[35260]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:51.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:50 smithi162 ceph-mon[35260]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-19T07:55:51.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:50 smithi162 ceph-mon[35260]: osdmap e6: 1 total, 0 up, 1 in 2024-09-19T07:55:51.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:50 smithi162 ceph-mon[35260]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-19T07:55:51.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:50 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:51.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:50 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:51.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:50 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:51.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:50 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:55:51.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:50 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:51.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:50 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:51.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:50 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:51.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:50 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:55:51.164 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:55:51.163+0000 7f1994d78640 1 -- 172.21.15.139:0/2776292408 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1990100410 con 0x7f197c0a31e0 2024-09-19T07:55:51.682 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 07:55:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0[45294]: 2024-09-19T07:55:51.418+0000 7f7b3d26d640 -1 osd.0 0 waiting for initial osdmap 2024-09-19T07:55:51.682 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 07:55:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0[45294]: 2024-09-19T07:55:51.422+0000 7f7b3589f640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:55:51.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:51 smithi098 ceph-mon[31333]: Detected new or changed devices on smithi098 2024-09-19T07:55:51.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:51 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi098 to 17088M 2024-09-19T07:55:51.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:51 smithi098 ceph-mon[31333]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-19T07:55:51.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:51 smithi098 ceph-mon[31333]: osdmap e7: 1 total, 0 up, 1 in 2024-09-19T07:55:51.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:51 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:51.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:51 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:51.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:51 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:55:51.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:51 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:55:51.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:51 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:51.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:51 smithi098 ceph-mon[31333]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' 2024-09-19T07:55:51.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:51 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:51.843 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:51 smithi139 ceph-mon[34874]: Detected new or changed devices on smithi098 2024-09-19T07:55:51.843 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:51 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi098 to 17088M 2024-09-19T07:55:51.843 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:51 smithi139 ceph-mon[34874]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-19T07:55:51.843 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:51 smithi139 ceph-mon[34874]: osdmap e7: 1 total, 0 up, 1 in 2024-09-19T07:55:51.844 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:51 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:51.844 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:51 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:51.844 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:51 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:55:51.844 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:51 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:55:51.844 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:51 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:51.844 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:51 smithi139 ceph-mon[34874]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' 2024-09-19T07:55:51.844 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:51 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:52.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:51 smithi162 ceph-mon[35260]: Detected new or changed devices on smithi098 2024-09-19T07:55:52.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:51 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi098 to 17088M 2024-09-19T07:55:52.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:51 smithi162 ceph-mon[35260]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-19T07:55:52.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:51 smithi162 ceph-mon[35260]: osdmap e7: 1 total, 0 up, 1 in 2024-09-19T07:55:52.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:51 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:52.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:51 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:52.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:51 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:55:52.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:51 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:55:52.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:51 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:52.122 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:51 smithi162 ceph-mon[35260]: from='osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533]' entity='osd.0' 2024-09-19T07:55:52.122 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:51 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:52.811 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:52 smithi139 ceph-mon[34874]: purged_snaps scrub starts 2024-09-19T07:55:52.811 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:52 smithi139 ceph-mon[34874]: purged_snaps scrub ok 2024-09-19T07:55:52.811 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:52 smithi139 ceph-mon[34874]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:55:52.811 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:52 smithi139 ceph-mon[34874]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:52.811 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:52 smithi139 ceph-mon[34874]: osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533] boot 2024-09-19T07:55:52.812 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:52 smithi139 ceph-mon[34874]: osdmap e8: 1 total, 1 up, 1 in 2024-09-19T07:55:52.812 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:52 smithi098 ceph-mon[31333]: purged_snaps scrub starts 2024-09-19T07:55:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:52 smithi098 ceph-mon[31333]: purged_snaps scrub ok 2024-09-19T07:55:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:52 smithi098 ceph-mon[31333]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:55:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:52 smithi098 ceph-mon[31333]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:52 smithi098 ceph-mon[31333]: osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533] boot 2024-09-19T07:55:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:52 smithi098 ceph-mon[31333]: osdmap e8: 1 total, 1 up, 1 in 2024-09-19T07:55:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:53.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:52 smithi162 ceph-mon[35260]: purged_snaps scrub starts 2024-09-19T07:55:53.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:52 smithi162 ceph-mon[35260]: purged_snaps scrub ok 2024-09-19T07:55:53.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:52 smithi162 ceph-mon[35260]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:55:53.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:52 smithi162 ceph-mon[35260]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T07:55:53.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:52 smithi162 ceph-mon[35260]: osd.0 [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533] boot 2024-09-19T07:55:53.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:52 smithi162 ceph-mon[35260]: osdmap e8: 1 total, 1 up, 1 in 2024-09-19T07:55:53.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T07:55:53.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:53 smithi139 ceph-mon[34874]: from='client.? 172.21.15.139:0/1000228701' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34724bd5-0738-4473-b087-c09031c1ee2e"}]: dispatch 2024-09-19T07:55:53.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:53 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34724bd5-0738-4473-b087-c09031c1ee2e"}]: dispatch 2024-09-19T07:55:53.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:53 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34724bd5-0738-4473-b087-c09031c1ee2e"}]': finished 2024-09-19T07:55:53.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:53 smithi139 ceph-mon[34874]: osdmap e9: 2 total, 1 up, 2 in 2024-09-19T07:55:53.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:53 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:55:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:53 smithi098 ceph-mon[31333]: from='client.? 172.21.15.139:0/1000228701' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34724bd5-0738-4473-b087-c09031c1ee2e"}]: dispatch 2024-09-19T07:55:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:53 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34724bd5-0738-4473-b087-c09031c1ee2e"}]: dispatch 2024-09-19T07:55:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:53 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34724bd5-0738-4473-b087-c09031c1ee2e"}]': finished 2024-09-19T07:55:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:53 smithi098 ceph-mon[31333]: osdmap e9: 2 total, 1 up, 2 in 2024-09-19T07:55:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:55:54.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:53 smithi162 ceph-mon[35260]: from='client.? 172.21.15.139:0/1000228701' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34724bd5-0738-4473-b087-c09031c1ee2e"}]: dispatch 2024-09-19T07:55:54.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:53 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34724bd5-0738-4473-b087-c09031c1ee2e"}]: dispatch 2024-09-19T07:55:54.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:53 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34724bd5-0738-4473-b087-c09031c1ee2e"}]': finished 2024-09-19T07:55:54.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:53 smithi162 ceph-mon[35260]: osdmap e9: 2 total, 1 up, 2 in 2024-09-19T07:55:54.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:53 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:55:54.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:54 smithi139 ceph-mon[34874]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:55:54.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:54 smithi139 ceph-mon[34874]: from='client.? 172.21.15.139:0/2713168151' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:55:54.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:54 smithi098 ceph-mon[31333]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:55:54.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:54 smithi098 ceph-mon[31333]: from='client.? 172.21.15.139:0/2713168151' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:55:55.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:54 smithi162 ceph-mon[35260]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:55:55.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:54 smithi162 ceph-mon[35260]: from='client.? 172.21.15.139:0/2713168151' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:55:55.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:55 smithi139 ceph-mon[34874]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:55:55.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:55 smithi098 ceph-mon[31333]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:55:56.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:55 smithi162 ceph-mon[35260]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:55:58.844 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:55:58 smithi139 ceph-mon[34874]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:55:58.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:55:58 smithi162 ceph-mon[35260]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:55:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:55:58 smithi098 ceph-mon[31333]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:00.760 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:00 smithi139 ceph-mon[34874]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:00.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:00 smithi162 ceph-mon[35260]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:00.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:00 smithi098 ceph-mon[31333]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:02.811 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:02 smithi139 ceph-mon[34874]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:02.811 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-19T07:56:02.811 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:02.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:02 smithi162 ceph-mon[35260]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:02.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-19T07:56:02.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:02 smithi098 ceph-mon[31333]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-19T07:56:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:03.842 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:03 smithi139 ceph-mon[34874]: Deploying daemon osd.1 on smithi139 2024-09-19T07:56:03.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:03 smithi162 ceph-mon[35260]: Deploying daemon osd.1 on smithi139 2024-09-19T07:56:03.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:03 smithi098 ceph-mon[31333]: Deploying daemon osd.1 on smithi139 2024-09-19T07:56:04.797 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:04 smithi139 ceph-mon[34874]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:04.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:04 smithi162 ceph-mon[35260]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:04.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:04 smithi098 ceph-mon[31333]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:06.841 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:06 smithi139 ceph-mon[34874]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:06.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:06 smithi162 ceph-mon[35260]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:06.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:06 smithi098 ceph-mon[31333]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:08.850 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:08 smithi139 ceph-mon[34874]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:08.850 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:08 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:08.850 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:08 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:08.851 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:08 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:08.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:08 smithi162 ceph-mon[35260]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:08.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:08 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:08.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:08 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:08.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:08 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:08.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:08 smithi098 ceph-mon[31333]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:08.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:08 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:08.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:08 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:08.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:08 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:10.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:10 smithi139 ceph-mon[34874]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:10.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:10 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:10.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:10 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:10.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:10 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:10.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:10 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:56:10.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:10 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:10.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:10 smithi162 ceph-mon[35260]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:10.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:10 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:10.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:10 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:10.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:10 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:10.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:10 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:56:10.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:10 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:10.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:10 smithi098 ceph-mon[31333]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:10.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:10 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:10.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:10 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:10.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:10 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:10.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:10 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:56:10.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:10 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:12.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:56:12.542+0000 7f19877fe640 1 -- 172.21.15.139:0/2776292408 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1990100410 con 0x7f197c0a31e0 2024-09-19T07:56:12.544 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 1 on host 'smithi139' 2024-09-19T07:56:12.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:56:12.544+0000 7f1994d78640 1 -- 172.21.15.139:0/2776292408 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f197c0a31e0 msgr2=0x7f197c0a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:56:12.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:56:12.544+0000 7f1994d78640 1 --2- 172.21.15.139:0/2776292408 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f197c0a31e0 0x7f197c0a56a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1980005fd0 tx=0x7f198000b410 comp rx=0 tx=0).stop 2024-09-19T07:56:12.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:56:12.544+0000 7f1994d78640 1 -- 172.21.15.139:0/2776292408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1990101760 msgr2=0x7f19901984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:56:12.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:56:12.544+0000 7f1994d78640 1 --2- 172.21.15.139:0/2776292408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1990101760 0x7f19901984b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1978031470 tx=0x7f1978032c70 comp rx=0 tx=0).stop 2024-09-19T07:56:12.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:56:12.545+0000 7f1994d78640 1 -- 172.21.15.139:0/2776292408 shutdown_connections 2024-09-19T07:56:12.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:56:12.545+0000 7f1994d78640 1 --2- 172.21.15.139:0/2776292408 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f197c0a31e0 0x7f197c0a56a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:12.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:56:12.545+0000 7f1994d78640 1 --2- 172.21.15.139:0/2776292408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19901020f0 0x7f199019f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:12.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:56:12.545+0000 7f1994d78640 1 --2- 172.21.15.139:0/2776292408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1990101760 0x7f19901984b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:12.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:56:12.545+0000 7f1994d78640 1 --2- 172.21.15.139:0/2776292408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1990100da0 0x7f1990197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:12.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:56:12.545+0000 7f1994d78640 1 -- 172.21.15.139:0/2776292408 >> 172.21.15.139:0/2776292408 conn(0x7f19900fba40 msgr2=0x7f19900ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:56:12.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:56:12.545+0000 7f1994d78640 1 -- 172.21.15.139:0/2776292408 shutdown_connections 2024-09-19T07:56:12.547 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:56:12.545+0000 7f1994d78640 1 -- 172.21.15.139:0/2776292408 wait complete. 2024-09-19T07:56:12.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:12 smithi162 ceph-mon[35260]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:12.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:12.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:12.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:12 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:12.874 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:12 smithi139 ceph-mon[34874]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:12.874 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:12 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:12.874 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:12 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:12.874 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:12 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:12.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:12 smithi098 ceph-mon[31333]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:12.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:12 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:13.176 DEBUG:teuthology.orchestra.run.smithi139:osd.1> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.1.service 2024-09-19T07:56:13.179 INFO:tasks.cephadm:Deploying osd.2 on smithi162 with /dev/vg_nvme/lv_4... 2024-09-19T07:56:13.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-19T07:56:13.431 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T07:56:14.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:14 smithi139 ceph-mon[34874]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:14.678 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 07:56:14 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1[39414]: 2024-09-19T07:56:14.633+0000 7f6cd208a640 -1 osd.1 0 log_to_monitors true 2024-09-19T07:56:14.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:14 smithi162 ceph-mon[35260]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:14.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:14 smithi098 ceph-mon[31333]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:15.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:15 smithi162 ceph-mon[35260]: from='osd.1 [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-19T07:56:15.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:15 smithi162 ceph-mon[35260]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-19T07:56:15.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:15 smithi139 ceph-mon[34874]: from='osd.1 [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-19T07:56:15.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:15 smithi139 ceph-mon[34874]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-19T07:56:15.745 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:56:15.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_4 2024-09-19T07:56:15.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:15 smithi098 ceph-mon[31333]: from='osd.1 [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-19T07:56:15.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:15 smithi098 ceph-mon[31333]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-19T07:56:16.025 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T07:56:16.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.437+0000 7f7edcc11640 1 -- 172.21.15.162:0/1703149071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed8101d10 msgr2=0x7f7ed8102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:56:16.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.437+0000 7f7edcc11640 1 --2- 172.21.15.162:0/1703149071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed8101d10 0x7f7ed8102190 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc009900 tx=0x7f7ecc02a410 comp rx=0 tx=0).stop 2024-09-19T07:56:16.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.437+0000 7f7edcc11640 1 -- 172.21.15.162:0/1703149071 shutdown_connections 2024-09-19T07:56:16.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.437+0000 7f7edcc11640 1 --2- 172.21.15.162:0/1703149071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed81026d0 0x7f7ed8108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:16.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.437+0000 7f7edcc11640 1 --2- 172.21.15.162:0/1703149071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed8101d10 0x7f7ed8102190 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:16.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.437+0000 7f7edcc11640 1 --2- 172.21.15.162:0/1703149071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ed8100b10 0x7f7ed8100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:16.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.437+0000 7f7edcc11640 1 -- 172.21.15.162:0/1703149071 >> 172.21.15.162:0/1703149071 conn(0x7f7ed80fc2a0 msgr2=0x7f7ed80fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:56:16.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.437+0000 7f7edcc11640 1 -- 172.21.15.162:0/1703149071 shutdown_connections 2024-09-19T07:56:16.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.437+0000 7f7edcc11640 1 -- 172.21.15.162:0/1703149071 wait complete. 2024-09-19T07:56:16.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.438+0000 7f7edcc11640 1 Processor -- start 2024-09-19T07:56:16.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.438+0000 7f7edcc11640 1 -- start start 2024-09-19T07:56:16.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.438+0000 7f7edcc11640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed8100b10 0x7f7ed806b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:56:16.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.438+0000 7f7edcc11640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ed8101d10 0x7f7ed819ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:56:16.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.438+0000 7f7edcc11640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed81026d0 0x7f7ed819b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:56:16.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.438+0000 7f7edcc11640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed810c890 con 0x7f7ed81026d0 2024-09-19T07:56:16.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.438+0000 7f7edcc11640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed810c710 con 0x7f7ed8101d10 2024-09-19T07:56:16.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.438+0000 7f7edcc11640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed810ca10 con 0x7f7ed8100b10 2024-09-19T07:56:16.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.438+0000 7f7ed659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed8100b10 0x7f7ed806b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:56:16.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.438+0000 7f7ed659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed8100b10 0x7f7ed806b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:33118/0 (socket says 172.21.15.162:33118) 2024-09-19T07:56:16.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.438+0000 7f7ed659c640 1 -- 172.21.15.162:0/2732266114 learned_addr learned my addr 172.21.15.162:0/2732266114 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-19T07:56:16.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.439+0000 7f7ed5d9b640 1 --2- 172.21.15.162:0/2732266114 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ed8101d10 0x7f7ed819ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:56:16.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.439+0000 7f7ed6d9d640 1 --2- 172.21.15.162:0/2732266114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed81026d0 0x7f7ed819b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:56:16.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.439+0000 7f7ed659c640 1 -- 172.21.15.162:0/2732266114 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ed8101d10 msgr2=0x7f7ed819ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:56:16.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.439+0000 7f7ed659c640 1 --2- 172.21.15.162:0/2732266114 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ed8101d10 0x7f7ed819ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:16.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.439+0000 7f7ed659c640 1 -- 172.21.15.162:0/2732266114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed81026d0 msgr2=0x7f7ed819b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:56:16.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.439+0000 7f7ed659c640 1 --2- 172.21.15.162:0/2732266114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed81026d0 0x7f7ed819b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:16.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.439+0000 7f7ed659c640 1 -- 172.21.15.162:0/2732266114 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed819ba90 con 0x7f7ed8100b10 2024-09-19T07:56:16.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.439+0000 7f7ed659c640 1 --2- 172.21.15.162:0/2732266114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed8100b10 0x7f7ed806b330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7ec800b450 tx=0x7f7ec800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:56:16.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.439+0000 7f7ec77fe640 1 -- 172.21.15.162:0/2732266114 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8015020 con 0x7f7ed8100b10 2024-09-19T07:56:16.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.439+0000 7f7ec77fe640 1 -- 172.21.15.162:0/2732266114 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ec8011d70 con 0x7f7ed8100b10 2024-09-19T07:56:16.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.439+0000 7f7ec77fe640 1 -- 172.21.15.162:0/2732266114 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec800f340 con 0x7f7ed8100b10 2024-09-19T07:56:16.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.439+0000 7f7edcc11640 1 -- 172.21.15.162:0/2732266114 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed819bd80 con 0x7f7ed8100b10 2024-09-19T07:56:16.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.439+0000 7f7edcc11640 1 -- 172.21.15.162:0/2732266114 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed819c2c0 con 0x7f7ed8100b10 2024-09-19T07:56:16.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.440+0000 7f7edcc11640 1 -- 172.21.15.162:0/2732266114 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed819cd40 con 0x7f7ed8100b10 2024-09-19T07:56:16.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.441+0000 7f7ec77fe640 1 -- 172.21.15.162:0/2732266114 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f7ec8018020 con 0x7f7ed8100b10 2024-09-19T07:56:16.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.442+0000 7f7ec77fe640 1 --2- 172.21.15.162:0/2732266114 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7ec00a3230 0x7f7ec00a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:56:16.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.442+0000 7f7ec77fe640 1 -- 172.21.15.162:0/2732266114 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1903+0+0 (secure 0 0 0) 0x7f7ec80cd9d0 con 0x7f7ed8100b10 2024-09-19T07:56:16.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.442+0000 7f7ed5d9b640 1 --2- 172.21.15.162:0/2732266114 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7ec00a3230 0x7f7ec00a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:56:16.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.443+0000 7f7ed5d9b640 1 --2- 172.21.15.162:0/2732266114 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7ec00a3230 0x7f7ec00a56f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc000c00 tx=0x7f7ecc031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:56:16.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.446+0000 7f7ec77fe640 1 -- 172.21.15.162:0/2732266114 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f7ec801f020 con 0x7f7ed8100b10 2024-09-19T07:56:16.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:16.623+0000 7f7edcc11640 1 -- 172.21.15.162:0/2732266114 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7ed8107ec0 con 0x7f7ec00a3230 2024-09-19T07:56:16.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:16 smithi162 ceph-mon[35260]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:16.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:16 smithi162 ceph-mon[35260]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-19T07:56:16.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:16 smithi162 ceph-mon[35260]: osdmap e10: 2 total, 1 up, 2 in 2024-09-19T07:56:16.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:16.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:16 smithi162 ceph-mon[35260]: from='osd.1 [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:56:16.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:16 smithi162 ceph-mon[35260]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:56:16.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:16.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:16.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:56:16.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:16.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:16.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:56:16.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:16 smithi139 ceph-mon[34874]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:16 smithi139 ceph-mon[34874]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-19T07:56:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:16 smithi139 ceph-mon[34874]: osdmap e10: 2 total, 1 up, 2 in 2024-09-19T07:56:16.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:16.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:16 smithi139 ceph-mon[34874]: from='osd.1 [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:56:16.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:16 smithi139 ceph-mon[34874]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:56:16.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:16.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:16.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:56:16.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:16.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:16.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:56:16.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:16.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:16 smithi098 ceph-mon[31333]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:16.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:16 smithi098 ceph-mon[31333]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-19T07:56:16.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:16 smithi098 ceph-mon[31333]: osdmap e10: 2 total, 1 up, 2 in 2024-09-19T07:56:16.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:16.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:16 smithi098 ceph-mon[31333]: from='osd.1 [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:56:16.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:16 smithi098 ceph-mon[31333]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:56:16.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:16.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:16.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:56:16.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:16.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:16.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:56:16.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:17.677 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 07:56:17 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1[39414]: 2024-09-19T07:56:17.339+0000 7f6cce821640 -1 osd.1 0 waiting for initial osdmap 2024-09-19T07:56:17.677 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 07:56:17 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1[39414]: 2024-09-19T07:56:17.343+0000 7f6cc5e1c640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:56:17.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:17 smithi139 ceph-mon[34874]: Detected new or changed devices on smithi139 2024-09-19T07:56:17.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:17 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi139 to 17088M 2024-09-19T07:56:17.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:17 smithi139 ceph-mon[34874]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-19T07:56:17.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:17 smithi139 ceph-mon[34874]: osdmap e11: 2 total, 1 up, 2 in 2024-09-19T07:56:17.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:17 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:17.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:17 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:17.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:17 smithi139 ceph-mon[34874]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:56:17.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:17 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:56:17.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:17 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:56:17.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:17 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:17.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:17 smithi139 ceph-mon[34874]: from='osd.1 ' entity='osd.1' 2024-09-19T07:56:17.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:17 smithi162 ceph-mon[35260]: Detected new or changed devices on smithi139 2024-09-19T07:56:17.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:17 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi139 to 17088M 2024-09-19T07:56:17.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:17 smithi162 ceph-mon[35260]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-19T07:56:17.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:17 smithi162 ceph-mon[35260]: osdmap e11: 2 total, 1 up, 2 in 2024-09-19T07:56:17.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:17 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:17.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:17 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:17.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:17 smithi162 ceph-mon[35260]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:56:17.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:17 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:56:17.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:17 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:56:17.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:17 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:17.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:17 smithi162 ceph-mon[35260]: from='osd.1 ' entity='osd.1' 2024-09-19T07:56:17.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:17 smithi098 ceph-mon[31333]: Detected new or changed devices on smithi139 2024-09-19T07:56:17.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:17 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi139 to 17088M 2024-09-19T07:56:17.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:17 smithi098 ceph-mon[31333]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-19T07:56:17.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:17 smithi098 ceph-mon[31333]: osdmap e11: 2 total, 1 up, 2 in 2024-09-19T07:56:17.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:17 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:17.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:17 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:17.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:17 smithi098 ceph-mon[31333]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:56:17.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:17 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:56:17.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:17 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:56:17.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:17 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:17.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:17 smithi098 ceph-mon[31333]: from='osd.1 ' entity='osd.1' 2024-09-19T07:56:18.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:18 smithi162 ceph-mon[35260]: purged_snaps scrub starts 2024-09-19T07:56:18.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:18 smithi162 ceph-mon[35260]: purged_snaps scrub ok 2024-09-19T07:56:18.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:18 smithi162 ceph-mon[35260]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:18.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:18 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:18.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:18 smithi162 ceph-mon[35260]: osd.1 [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626] boot 2024-09-19T07:56:18.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:18 smithi162 ceph-mon[35260]: osdmap e12: 2 total, 2 up, 2 in 2024-09-19T07:56:18.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:18 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:18.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:18 smithi139 ceph-mon[34874]: purged_snaps scrub starts 2024-09-19T07:56:18.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:18 smithi139 ceph-mon[34874]: purged_snaps scrub ok 2024-09-19T07:56:18.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:18 smithi139 ceph-mon[34874]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:18.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:18 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:18.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:18 smithi139 ceph-mon[34874]: osd.1 [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626] boot 2024-09-19T07:56:18.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:18 smithi139 ceph-mon[34874]: osdmap e12: 2 total, 2 up, 2 in 2024-09-19T07:56:18.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:18 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:18.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:18 smithi098 ceph-mon[31333]: purged_snaps scrub starts 2024-09-19T07:56:18.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:18 smithi098 ceph-mon[31333]: purged_snaps scrub ok 2024-09-19T07:56:18.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:18 smithi098 ceph-mon[31333]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-19T07:56:18.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:18 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:18.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:18 smithi098 ceph-mon[31333]: osd.1 [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626] boot 2024-09-19T07:56:18.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:18 smithi098 ceph-mon[31333]: osdmap e12: 2 total, 2 up, 2 in 2024-09-19T07:56:18.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:18 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T07:56:19.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:19 smithi162 ceph-mon[35260]: from='client.? 172.21.15.162:0/1202251395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16985ebe-0e32-42ee-bded-b7838e290afc"}]: dispatch 2024-09-19T07:56:19.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:19 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16985ebe-0e32-42ee-bded-b7838e290afc"}]: dispatch 2024-09-19T07:56:19.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:19 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16985ebe-0e32-42ee-bded-b7838e290afc"}]': finished 2024-09-19T07:56:19.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:19 smithi162 ceph-mon[35260]: osdmap e13: 3 total, 2 up, 3 in 2024-09-19T07:56:19.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:19.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:19 smithi162 ceph-mon[35260]: from='client.? 172.21.15.162:0/3530349458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:56:19.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:19 smithi139 ceph-mon[34874]: from='client.? 172.21.15.162:0/1202251395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16985ebe-0e32-42ee-bded-b7838e290afc"}]: dispatch 2024-09-19T07:56:19.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:19 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16985ebe-0e32-42ee-bded-b7838e290afc"}]: dispatch 2024-09-19T07:56:19.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:19 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16985ebe-0e32-42ee-bded-b7838e290afc"}]': finished 2024-09-19T07:56:19.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:19 smithi139 ceph-mon[34874]: osdmap e13: 3 total, 2 up, 3 in 2024-09-19T07:56:19.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:19.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:19 smithi139 ceph-mon[34874]: from='client.? 172.21.15.162:0/3530349458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:56:19.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:19 smithi098 ceph-mon[31333]: from='client.? 172.21.15.162:0/1202251395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16985ebe-0e32-42ee-bded-b7838e290afc"}]: dispatch 2024-09-19T07:56:19.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:19 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16985ebe-0e32-42ee-bded-b7838e290afc"}]: dispatch 2024-09-19T07:56:19.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:19 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16985ebe-0e32-42ee-bded-b7838e290afc"}]': finished 2024-09-19T07:56:19.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:19 smithi098 ceph-mon[31333]: osdmap e13: 3 total, 2 up, 3 in 2024-09-19T07:56:19.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:19.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:19 smithi098 ceph-mon[31333]: from='client.? 172.21.15.162:0/3530349458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:56:20.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:20 smithi098 ceph-mon[31333]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:20.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:20 smithi162 ceph-mon[35260]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:20.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:20 smithi139 ceph-mon[34874]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:22.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:22 smithi162 ceph-mon[35260]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:22.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:22 smithi139 ceph-mon[34874]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:22.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:22 smithi098 ceph-mon[31333]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:24.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:24 smithi162 ceph-mon[35260]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:24.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:24 smithi139 ceph-mon[34874]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:24.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:24 smithi098 ceph-mon[31333]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:26.712 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:26 smithi162 ceph-mon[35260]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:26.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:26 smithi139 ceph-mon[34874]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:26 smithi098 ceph-mon[31333]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:27.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:27 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-19T07:56:27.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:27 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:27.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:27 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-19T07:56:27.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:27 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:27.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:27 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-19T07:56:27.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:27 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:28.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:28 smithi162 ceph-mon[35260]: Deploying daemon osd.2 on smithi162 2024-09-19T07:56:28.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:28 smithi162 ceph-mon[35260]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:28.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:28 smithi139 ceph-mon[34874]: Deploying daemon osd.2 on smithi162 2024-09-19T07:56:28.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:28 smithi139 ceph-mon[34874]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:28.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:28 smithi098 ceph-mon[31333]: Deploying daemon osd.2 on smithi162 2024-09-19T07:56:28.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:28 smithi098 ceph-mon[31333]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:30.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:30 smithi162 ceph-mon[35260]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:30.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:30 smithi139 ceph-mon[34874]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:30.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:30 smithi098 ceph-mon[31333]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:32.742 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:32 smithi162 ceph-mon[35260]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:32.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:32 smithi139 ceph-mon[34874]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:32.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:32 smithi098 ceph-mon[31333]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:33.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:33 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:33.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:33 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:33.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:33 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:33.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:33 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:33.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:33 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:33.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:33 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:33.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:33 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:33.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:33 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:33.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:33 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:34.672 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:34 smithi162 ceph-mon[35260]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:34.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:34 smithi139 ceph-mon[34874]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:34.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:34 smithi098 ceph-mon[31333]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:36.437 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 2 on host 'smithi162' 2024-09-19T07:56:36.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:36.435+0000 7f7ec77fe640 1 -- 172.21.15.162:0/2732266114 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7ed8107ec0 con 0x7f7ec00a3230 2024-09-19T07:56:36.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:36.437+0000 7f7edcc11640 1 -- 172.21.15.162:0/2732266114 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7ec00a3230 msgr2=0x7f7ec00a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:56:36.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:36.437+0000 7f7edcc11640 1 --2- 172.21.15.162:0/2732266114 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7ec00a3230 0x7f7ec00a56f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc000c00 tx=0x7f7ecc031400 comp rx=0 tx=0).stop 2024-09-19T07:56:36.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:36.437+0000 7f7edcc11640 1 -- 172.21.15.162:0/2732266114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed8100b10 msgr2=0x7f7ed806b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:56:36.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:36.437+0000 7f7edcc11640 1 --2- 172.21.15.162:0/2732266114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed8100b10 0x7f7ed806b330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7ec800b450 tx=0x7f7ec800b910 comp rx=0 tx=0).stop 2024-09-19T07:56:36.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:36.437+0000 7f7edcc11640 1 -- 172.21.15.162:0/2732266114 shutdown_connections 2024-09-19T07:56:36.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:36.437+0000 7f7edcc11640 1 --2- 172.21.15.162:0/2732266114 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7ec00a3230 0x7f7ec00a56f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:36.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:36.437+0000 7f7edcc11640 1 --2- 172.21.15.162:0/2732266114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed81026d0 0x7f7ed819b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:36.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:36.437+0000 7f7edcc11640 1 --2- 172.21.15.162:0/2732266114 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ed8101d10 0x7f7ed819ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:36.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:36.437+0000 7f7edcc11640 1 --2- 172.21.15.162:0/2732266114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed8100b10 0x7f7ed806b330 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:36.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:36.437+0000 7f7edcc11640 1 -- 172.21.15.162:0/2732266114 >> 172.21.15.162:0/2732266114 conn(0x7f7ed80fc2a0 msgr2=0x7f7ed8105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:56:36.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:36.437+0000 7f7edcc11640 1 -- 172.21.15.162:0/2732266114 shutdown_connections 2024-09-19T07:56:36.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:56:36.437+0000 7f7edcc11640 1 -- 172.21.15.162:0/2732266114 wait complete. 2024-09-19T07:56:36.739 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:36 smithi162 ceph-mon[35260]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:36.739 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:36 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:36.739 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:36 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:36.740 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:36 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:36.740 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:36 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:56:36.740 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:36 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:36.740 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:36 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:36.740 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:36 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:36.740 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:36 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:36.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:36 smithi139 ceph-mon[34874]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:36.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:36 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:36.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:36 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:36.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:36 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:36.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:36 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:56:36.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:36 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:36.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:36 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:36.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:36 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:36.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:36 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:36.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:36 smithi098 ceph-mon[31333]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:36.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:36.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:36.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:36.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:56:36.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:36.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:36.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:36.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:37.040 DEBUG:teuthology.orchestra.run.smithi162:osd.2> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.2.service 2024-09-19T07:56:37.042 INFO:tasks.cephadm:Deploying osd.3 on smithi098 with /dev/vg_nvme/lv_3... 2024-09-19T07:56:37.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-19T07:56:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:56:38.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:38 smithi162 ceph-mon[35260]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:38.637 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:38 smithi098 ceph-mon[31333]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:38.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:38 smithi139 ceph-mon[34874]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:39.602 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:39 smithi162 ceph-mon[35260]: from='osd.2 [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-19T07:56:39.602 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:39 smithi162 ceph-mon[35260]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-19T07:56:39.602 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 07:56:39 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2[40270]: 2024-09-19T07:56:39.319+0000 7f8a93e7c640 -1 osd.2 0 log_to_monitors true 2024-09-19T07:56:39.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:39 smithi098 ceph-mon[31333]: from='osd.2 [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-19T07:56:39.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:39 smithi098 ceph-mon[31333]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-19T07:56:39.726 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:56:39.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-09-19T07:56:39.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:39 smithi139 ceph-mon[34874]: from='osd.2 [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-19T07:56:39.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:39 smithi139 ceph-mon[34874]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-19T07:56:40.004 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:56:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.447+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2757268535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c880ffed0 msgr2=0x7f5c881002d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:56:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.447+0000 7f5c8ed3a640 1 --2- 172.21.15.98:0/2757268535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c880ffed0 0x7f5c881002d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009930 tx=0x7f5c8402a420 comp rx=0 tx=0).stop 2024-09-19T07:56:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.447+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2757268535 shutdown_connections 2024-09-19T07:56:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.447+0000 7f5c8ed3a640 1 --2- 172.21.15.98:0/2757268535 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5c88103870 0x7f5c8810a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.447+0000 7f5c8ed3a640 1 --2- 172.21.15.98:0/2757268535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c88102eb0 0x7f5c88103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.447+0000 7f5c8ed3a640 1 --2- 172.21.15.98:0/2757268535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c880ffed0 0x7f5c881002d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.447+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2757268535 >> 172.21.15.98:0/2757268535 conn(0x7f5c880fba40 msgr2=0x7f5c880fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:56:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.447+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2757268535 shutdown_connections 2024-09-19T07:56:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.447+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2757268535 wait complete. 2024-09-19T07:56:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8ed3a640 1 Processor -- start 2024-09-19T07:56:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8ed3a640 1 -- start start 2024-09-19T07:56:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8ed3a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5c880ffed0 0x7f5c88198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:56:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8ed3a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c88102eb0 0x7f5c88198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:56:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8ed3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c88103870 0x7f5c8819f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:56:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8ed3a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8810d910 con 0x7f5c88103870 2024-09-19T07:56:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8ed3a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8810d790 con 0x7f5c880ffed0 2024-09-19T07:56:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8ed3a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8810da90 con 0x7f5c88102eb0 2024-09-19T07:56:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8d2d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c88103870 0x7f5c8819f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:56:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8d2d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c88103870 0x7f5c8819f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54868/0 (socket says 172.21.15.98:54868) 2024-09-19T07:56:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8d2d7640 1 -- 172.21.15.98:0/2696577197 learned_addr learned my addr 172.21.15.98:0/2696577197 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:56:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8cad6640 1 --2- 172.21.15.98:0/2696577197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5c880ffed0 0x7f5c88198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:56:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c7ffff640 1 --2- 172.21.15.98:0/2696577197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c88102eb0 0x7f5c88198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:56:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8d2d7640 1 -- 172.21.15.98:0/2696577197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c88102eb0 msgr2=0x7f5c88198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:56:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8d2d7640 1 --2- 172.21.15.98:0/2696577197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c88102eb0 0x7f5c88198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8d2d7640 1 -- 172.21.15.98:0/2696577197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5c880ffed0 msgr2=0x7f5c88198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:56:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8d2d7640 1 --2- 172.21.15.98:0/2696577197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5c880ffed0 0x7f5c88198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:56:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.448+0000 7f5c8d2d7640 1 -- 172.21.15.98:0/2696577197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c8819fc50 con 0x7f5c88103870 2024-09-19T07:56:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.449+0000 7f5c8d2d7640 1 --2- 172.21.15.98:0/2696577197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c88103870 0x7f5c8819f5e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5c8000b970 tx=0x7f5c8000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:56:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.449+0000 7f5c7dffb640 1 -- 172.21.15.98:0/2696577197 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c80019070 con 0x7f5c88103870 2024-09-19T07:56:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.449+0000 7f5c7dffb640 1 -- 172.21.15.98:0/2696577197 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c8000cb40 con 0x7f5c88103870 2024-09-19T07:56:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.449+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2696577197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c8819ff40 con 0x7f5c88103870 2024-09-19T07:56:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.449+0000 7f5c7dffb640 1 -- 172.21.15.98:0/2696577197 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c80007650 con 0x7f5c88103870 2024-09-19T07:56:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.449+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2696577197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c881a0470 con 0x7f5c88103870 2024-09-19T07:56:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.450+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2696577197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c88109170 con 0x7f5c88103870 2024-09-19T07:56:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.451+0000 7f5c7dffb640 1 -- 172.21.15.98:0/2696577197 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f5c8001c070 con 0x7f5c88103870 2024-09-19T07:56:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.451+0000 7f5c7dffb640 1 --2- 172.21.15.98:0/2696577197 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5c700a3260 0x7f5c700a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:56:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.452+0000 7f5c8cad6640 1 --2- 172.21.15.98:0/2696577197 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5c700a3260 0x7f5c700a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:56:40.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.452+0000 7f5c7dffb640 1 -- 172.21.15.98:0/2696577197 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2445+0+0 (secure 0 0 0) 0x7f5c800cd600 con 0x7f5c88103870 2024-09-19T07:56:40.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.452+0000 7f5c8cad6640 1 --2- 172.21.15.98:0/2696577197 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5c700a3260 0x7f5c700a5720 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009900 tx=0x7f5c840315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:56:40.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.456+0000 7f5c7dffb640 1 -- 172.21.15.98:0/2696577197 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f5c8009b4a0 con 0x7f5c88103870 2024-09-19T07:56:40.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:40.634+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2696577197 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f5c88062630 con 0x7f5c700a3260 2024-09-19T07:56:40.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:40.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-19T07:56:40.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: osdmap e14: 3 total, 2 up, 3 in 2024-09-19T07:56:40.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:40.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: from='osd.2 [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:56:40.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:56:40.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: Detected new or changed devices on smithi162 2024-09-19T07:56:40.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:40.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:40.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:56:40.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi162 to 17088M 2024-09-19T07:56:40.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:40.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:40.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:56:40.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:40.870 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2[40270]: 2024-09-19T07:56:40.543+0000 7f8a8f5dc640 -1 osd.2 0 waiting for initial osdmap 2024-09-19T07:56:40.870 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2[40270]: 2024-09-19T07:56:40.546+0000 7f8a8bc16640 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:56:40.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:40.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-19T07:56:40.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: osdmap e14: 3 total, 2 up, 3 in 2024-09-19T07:56:40.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:40.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: from='osd.2 [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:56:40.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:56:40.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: Detected new or changed devices on smithi162 2024-09-19T07:56:40.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:40.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:40.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:56:40.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi162 to 17088M 2024-09-19T07:56:40.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:40.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:40.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:56:40.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:40.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:40.963 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-19T07:56:40.963 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: osdmap e14: 3 total, 2 up, 3 in 2024-09-19T07:56:40.963 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:40.963 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: from='osd.2 [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:56:40.963 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:56:40.963 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: Detected new or changed devices on smithi162 2024-09-19T07:56:40.964 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:40.964 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:40.964 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:56:40.964 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi162 to 17088M 2024-09-19T07:56:40.964 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:40.964 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:40.964 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:56:40.964 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:41.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:41 smithi098 ceph-mon[31333]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-19T07:56:41.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:41 smithi098 ceph-mon[31333]: osdmap e15: 3 total, 2 up, 3 in 2024-09-19T07:56:41.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:41 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:41.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:41 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:41.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:41 smithi098 ceph-mon[31333]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:56:41.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:41 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:56:41.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:41 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:56:41.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:41 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:41.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:41 smithi162 ceph-mon[35260]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-19T07:56:41.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:41 smithi162 ceph-mon[35260]: osdmap e15: 3 total, 2 up, 3 in 2024-09-19T07:56:41.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:41 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:41.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:41 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:41.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:41 smithi162 ceph-mon[35260]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:56:41.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:41 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:56:41.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:41 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:56:41.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:41 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:41.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:41 smithi139 ceph-mon[34874]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-19T07:56:41.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:41 smithi139 ceph-mon[34874]: osdmap e15: 3 total, 2 up, 3 in 2024-09-19T07:56:41.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:41 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:41.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:41 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:41.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:41 smithi139 ceph-mon[34874]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:56:41.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:41 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:56:41.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:41 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:56:41.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:41 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:42 smithi162 ceph-mon[35260]: purged_snaps scrub starts 2024-09-19T07:56:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:42 smithi162 ceph-mon[35260]: purged_snaps scrub ok 2024-09-19T07:56:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:42 smithi162 ceph-mon[35260]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:42 smithi162 ceph-mon[35260]: osd.2 [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891] boot 2024-09-19T07:56:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:42 smithi162 ceph-mon[35260]: osdmap e16: 3 total, 3 up, 3 in 2024-09-19T07:56:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:42.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:42 smithi139 ceph-mon[34874]: purged_snaps scrub starts 2024-09-19T07:56:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:42 smithi139 ceph-mon[34874]: purged_snaps scrub ok 2024-09-19T07:56:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:42 smithi139 ceph-mon[34874]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:42 smithi139 ceph-mon[34874]: osd.2 [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891] boot 2024-09-19T07:56:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:42 smithi139 ceph-mon[34874]: osdmap e16: 3 total, 3 up, 3 in 2024-09-19T07:56:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:42 smithi098 ceph-mon[31333]: purged_snaps scrub starts 2024-09-19T07:56:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:42 smithi098 ceph-mon[31333]: purged_snaps scrub ok 2024-09-19T07:56:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:42 smithi098 ceph-mon[31333]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-19T07:56:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:42 smithi098 ceph-mon[31333]: osd.2 [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891] boot 2024-09-19T07:56:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:42 smithi098 ceph-mon[31333]: osdmap e16: 3 total, 3 up, 3 in 2024-09-19T07:56:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T07:56:44.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:43 smithi162 ceph-mon[35260]: osdmap e17: 3 total, 3 up, 3 in 2024-09-19T07:56:44.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:43 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2672845076' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b140e466-3119-49cb-aec2-ba6ca4d10d11"}]: dispatch 2024-09-19T07:56:44.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:43 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2672845076' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b140e466-3119-49cb-aec2-ba6ca4d10d11"}]': finished 2024-09-19T07:56:44.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:43 smithi162 ceph-mon[35260]: osdmap e18: 4 total, 3 up, 4 in 2024-09-19T07:56:44.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:43 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:56:44.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:43 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3682644369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:56:44.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:43 smithi139 ceph-mon[34874]: osdmap e17: 3 total, 3 up, 3 in 2024-09-19T07:56:44.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:43 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2672845076' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b140e466-3119-49cb-aec2-ba6ca4d10d11"}]: dispatch 2024-09-19T07:56:44.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:43 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2672845076' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b140e466-3119-49cb-aec2-ba6ca4d10d11"}]': finished 2024-09-19T07:56:44.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:43 smithi139 ceph-mon[34874]: osdmap e18: 4 total, 3 up, 4 in 2024-09-19T07:56:44.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:43 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:56:44.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:43 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3682644369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:56:44.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:43 smithi098 ceph-mon[31333]: osdmap e17: 3 total, 3 up, 3 in 2024-09-19T07:56:44.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:43 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2672845076' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b140e466-3119-49cb-aec2-ba6ca4d10d11"}]: dispatch 2024-09-19T07:56:44.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:43 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2672845076' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b140e466-3119-49cb-aec2-ba6ca4d10d11"}]': finished 2024-09-19T07:56:44.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:43 smithi098 ceph-mon[31333]: osdmap e18: 4 total, 3 up, 4 in 2024-09-19T07:56:44.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:43 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:56:44.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:43 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3682644369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:56:45.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:44 smithi162 ceph-mon[35260]: pgmap v77: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:45.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:44 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-19T07:56:45.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:44 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-19T07:56:45.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:44 smithi162 ceph-mon[35260]: osdmap e19: 4 total, 3 up, 4 in 2024-09-19T07:56:45.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:44 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:56:45.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:44 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-19T07:56:45.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:44 smithi139 ceph-mon[34874]: pgmap v77: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:45.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:44 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-19T07:56:45.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:44 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-19T07:56:45.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:44 smithi139 ceph-mon[34874]: osdmap e19: 4 total, 3 up, 4 in 2024-09-19T07:56:45.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:44 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:56:45.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:44 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-19T07:56:45.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:44 smithi098 ceph-mon[31333]: pgmap v77: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:45.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:44 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-19T07:56:45.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:44 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-19T07:56:45.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:44 smithi098 ceph-mon[31333]: osdmap e19: 4 total, 3 up, 4 in 2024-09-19T07:56:45.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:44 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:56:45.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:44 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-19T07:56:45.617 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 07:56:45 smithi098 sudo[50335]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-19T07:56:45.617 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 07:56:45 smithi098 sudo[50335]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T07:56:45.617 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 07:56:45 smithi098 sudo[50335]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T07:56:45.617 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 07:56:45 smithi098 sudo[50335]: pam_unix(sudo:session): session closed for user root 2024-09-19T07:56:45.932 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 07:56:45 smithi098 sudo[50339]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-19T07:56:45.932 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 07:56:45 smithi098 sudo[50339]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T07:56:45.932 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 07:56:45 smithi098 sudo[50339]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T07:56:45.932 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 07:56:45 smithi098 sudo[50339]: pam_unix(sudo:session): session closed for user root 2024-09-19T07:56:46.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:56:46.027+0000 7f5c7dffb640 1 -- 172.21.15.98:0/2696577197 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f5c8000ccf0 con 0x7f5c88103870 2024-09-19T07:56:46.148 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 07:56:45 smithi139 sudo[43150]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-19T07:56:46.148 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 07:56:45 smithi139 sudo[43150]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T07:56:46.148 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 07:56:45 smithi139 sudo[43150]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T07:56:46.148 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 07:56:45 smithi139 sudo[43150]: pam_unix(sudo:session): session closed for user root 2024-09-19T07:56:46.149 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-19T07:56:46.149 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:46 smithi139 ceph-mon[34874]: osdmap e20: 4 total, 3 up, 4 in 2024-09-19T07:56:46.149 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:56:46.149 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:46 smithi139 ceph-mon[34874]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:46.149 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:46.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-19T07:56:46.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:46 smithi162 ceph-mon[35260]: osdmap e20: 4 total, 3 up, 4 in 2024-09-19T07:56:46.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:56:46.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:46 smithi162 ceph-mon[35260]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:46.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:46.427 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 07:56:46 smithi139 sudo[43153]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-19T07:56:46.427 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 07:56:46 smithi139 sudo[43153]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T07:56:46.427 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 07:56:46 smithi139 sudo[43153]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T07:56:46.428 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 07:56:46 smithi139 sudo[43153]: pam_unix(sudo:session): session closed for user root 2024-09-19T07:56:46.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-19T07:56:46.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:46 smithi098 ceph-mon[31333]: osdmap e20: 4 total, 3 up, 4 in 2024-09-19T07:56:46.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:56:46.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:46 smithi098 ceph-mon[31333]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:46.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:46.677 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 07:56:46 smithi162 sudo[44060]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-19T07:56:46.677 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 07:56:46 smithi162 sudo[44060]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T07:56:46.677 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 07:56:46 smithi162 sudo[44060]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T07:56:46.677 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 07:56:46 smithi162 sudo[44060]: pam_unix(sudo:session): session closed for user root 2024-09-19T07:56:47.120 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 07:56:46 smithi162 sudo[44063]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-19T07:56:47.120 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 07:56:46 smithi162 sudo[44063]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T07:56:47.120 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 07:56:46 smithi162 sudo[44063]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T07:56:47.120 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 07:56:46 smithi162 sudo[44063]: pam_unix(sudo:session): session closed for user root 2024-09-19T07:56:47.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:46 smithi098 sudo[50721]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-19T07:56:47.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:46 smithi098 sudo[50721]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T07:56:47.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:46 smithi098 sudo[50721]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T07:56:47.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:46 smithi098 sudo[50721]: pam_unix(sudo:session): session closed for user root 2024-09-19T07:56:47.460 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:47 smithi162 ceph-mon[35260]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-19T07:56:47.460 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:47 smithi162 ceph-mon[35260]: osdmap e21: 4 total, 3 up, 4 in 2024-09-19T07:56:47.460 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:47 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:56:47.460 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:47 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:47.460 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:47 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:47.460 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:47 smithi162 ceph-mon[35260]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T07:56:47.460 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:47 smithi162 sudo[44066]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-19T07:56:47.460 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:47 smithi162 sudo[44066]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T07:56:47.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:47 smithi162 sudo[44066]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T07:56:47.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:47 smithi162 sudo[44066]: pam_unix(sudo:session): session closed for user root 2024-09-19T07:56:47.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:47 smithi139 ceph-mon[34874]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-19T07:56:47.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:47 smithi139 ceph-mon[34874]: osdmap e21: 4 total, 3 up, 4 in 2024-09-19T07:56:47.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:47 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:56:47.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:47 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:47.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:47 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:47.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:47 smithi139 ceph-mon[34874]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T07:56:47.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:47 smithi098 sudo[50909]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-19T07:56:47.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:47 smithi098 sudo[50909]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T07:56:47.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:47 smithi098 sudo[50909]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T07:56:47.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:47 smithi098 sudo[50909]: pam_unix(sudo:session): session closed for user root 2024-09-19T07:56:47.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:47 smithi098 ceph-mon[31333]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-19T07:56:47.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:47 smithi098 ceph-mon[31333]: osdmap e21: 4 total, 3 up, 4 in 2024-09-19T07:56:47.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:47 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:56:47.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:47 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:47.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:47 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:47.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:47 smithi098 ceph-mon[31333]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T07:56:47.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:47 smithi162 sudo[44069]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-19T07:56:47.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:47 smithi162 sudo[44069]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T07:56:47.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:47 smithi162 sudo[44069]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T07:56:47.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:47 smithi162 sudo[44069]: pam_unix(sudo:session): session closed for user root 2024-09-19T07:56:48.193 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:47 smithi139 sudo[43156]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-19T07:56:48.193 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:47 smithi139 sudo[43156]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T07:56:48.193 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:47 smithi139 sudo[43156]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T07:56:48.193 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:47 smithi139 sudo[43156]: pam_unix(sudo:session): session closed for user root 2024-09-19T07:56:48.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T07:56:48.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:56:48.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:56:48.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:56:48.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T07:56:48.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:56:48.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:56:48.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:56:48.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:48.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T07:56:48.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T07:56:48.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:56:48.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:56:48.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:56:48.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:48 smithi098 ceph-mon[31333]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T07:56:48.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T07:56:48.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:56:48.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:56:48.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:56:48.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T07:56:48.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:56:48.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:56:48.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:56:48.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:48.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T07:56:48.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T07:56:48.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:56:48.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:56:48.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:56:48.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:48 smithi162 ceph-mon[35260]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T07:56:48.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 sudo[43159]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-19T07:56:48.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 sudo[43159]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T07:56:48.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 sudo[43159]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T07:56:48.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 sudo[43159]: pam_unix(sudo:session): session closed for user root 2024-09-19T07:56:48.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T07:56:48.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:56:48.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:56:48.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:56:48.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T07:56:48.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:56:48.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:56:48.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:56:48.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:48.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T07:56:48.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T07:56:48.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T07:56:48.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T07:56:48.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T07:56:48.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:48 smithi139 ceph-mon[34874]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T07:56:50.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:50 smithi098 ceph-mon[31333]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:50.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:50 smithi162 ceph-mon[35260]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:50.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:50 smithi139 ceph-mon[34874]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:51.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:51 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-19T07:56:51.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:51 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:51.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:51 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-19T07:56:51.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:51 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:51.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:51 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-19T07:56:51.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:51 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:52.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:52 smithi162 ceph-mon[35260]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:52.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:52 smithi162 ceph-mon[35260]: Deploying daemon osd.3 on smithi098 2024-09-19T07:56:52.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:52 smithi139 ceph-mon[34874]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:52.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:52 smithi139 ceph-mon[34874]: Deploying daemon osd.3 on smithi098 2024-09-19T07:56:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:52 smithi098 ceph-mon[31333]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:52 smithi098 ceph-mon[31333]: Deploying daemon osd.3 on smithi098 2024-09-19T07:56:54.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:54 smithi162 ceph-mon[35260]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:54.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:54 smithi098 ceph-mon[31333]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:54.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:54 smithi139 ceph-mon[34874]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:56.831 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:56 smithi098 ceph-mon[31333]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:56.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:56 smithi162 ceph-mon[35260]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:56.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:56 smithi139 ceph-mon[34874]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:57.869 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:57 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:57.869 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:57 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:57.869 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:57 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:57.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:57 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:57.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:57 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:57.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:57 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:57.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:57 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:57.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:57 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:57.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:57 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:56:58.808 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:56:58 smithi098 ceph-mon[31333]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:58.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:56:58 smithi162 ceph-mon[35260]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:56:58.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:56:58 smithi139 ceph-mon[34874]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:57:00.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:00 smithi098 ceph-mon[31333]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:57:00.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:00 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:00.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:00 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:00.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:00 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:00.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:00 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:00.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:00 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:00.656+0000 7f5c7dffb640 1 -- 172.21.15.98:0/2696577197 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5c88062630 con 0x7f5c700a3260 2024-09-19T07:57:00.657 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 3 on host 'smithi098' 2024-09-19T07:57:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:00.659+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2696577197 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5c700a3260 msgr2=0x7f5c700a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:00.659+0000 7f5c8ed3a640 1 --2- 172.21.15.98:0/2696577197 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5c700a3260 0x7f5c700a5720 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009900 tx=0x7f5c840315b0 comp rx=0 tx=0).stop 2024-09-19T07:57:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:00.659+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2696577197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c88103870 msgr2=0x7f5c8819f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:00.659+0000 7f5c8ed3a640 1 --2- 172.21.15.98:0/2696577197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c88103870 0x7f5c8819f5e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5c8000b970 tx=0x7f5c8000be30 comp rx=0 tx=0).stop 2024-09-19T07:57:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:00.659+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2696577197 shutdown_connections 2024-09-19T07:57:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:00.659+0000 7f5c8ed3a640 1 --2- 172.21.15.98:0/2696577197 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5c700a3260 0x7f5c700a5720 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:00.659+0000 7f5c8ed3a640 1 --2- 172.21.15.98:0/2696577197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c88103870 0x7f5c8819f5e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:00.659+0000 7f5c8ed3a640 1 --2- 172.21.15.98:0/2696577197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c88102eb0 0x7f5c88198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:00.659+0000 7f5c8ed3a640 1 --2- 172.21.15.98:0/2696577197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5c880ffed0 0x7f5c88198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:00.659+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2696577197 >> 172.21.15.98:0/2696577197 conn(0x7f5c880fba40 msgr2=0x7f5c88106340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:57:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:00.659+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2696577197 shutdown_connections 2024-09-19T07:57:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:00.659+0000 7f5c8ed3a640 1 -- 172.21.15.98:0/2696577197 wait complete. 2024-09-19T07:57:00.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:00 smithi162 ceph-mon[35260]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:57:00.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:00 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:00.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:00 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:00.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:00 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:00.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:00 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:00.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:00 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:00.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:00 smithi139 ceph-mon[34874]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:57:00.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:00 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:00.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:00 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:00.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:00 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:00.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:00 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:00.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:00 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:01.240 DEBUG:teuthology.orchestra.run.smithi098:osd.3> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.3.service 2024-09-19T07:57:01.243 INFO:tasks.cephadm:Deploying osd.4 on smithi139 with /dev/vg_nvme/lv_3... 2024-09-19T07:57:01.243 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-19T07:57:01.496 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T07:57:01.769 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:01 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:01.769 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:01 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:01.769 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:01 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:01.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:01 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:01.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:01 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:01.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:01 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:01.911 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:01 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:01.911 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:01 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:01.911 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:01 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:02.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:02 smithi098 ceph-mon[31333]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:57:02.844 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:02 smithi139 ceph-mon[34874]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:57:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:02 smithi162 ceph-mon[35260]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:57:03.932 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:57:03.959 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_3 2024-09-19T07:57:03.980 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 07:57:03 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3[51957]: 2024-09-19T07:57:03.707+0000 7ffa3870b640 -1 osd.3 0 log_to_monitors true 2024-09-19T07:57:04.214 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T07:57:04.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.635+0000 7f2a40bea640 1 -- 172.21.15.139:0/2955541341 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3c1026a0 msgr2=0x7f2a3c104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:04.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.635+0000 7f2a40bea640 1 --2- 172.21.15.139:0/2955541341 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3c1026a0 0x7f2a3c104a90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009900 tx=0x7f2a2c02a410 comp rx=0 tx=0).stop 2024-09-19T07:57:04.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.636+0000 7f2a40bea640 1 -- 172.21.15.139:0/2955541341 shutdown_connections 2024-09-19T07:57:04.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.636+0000 7f2a40bea640 1 --2- 172.21.15.139:0/2955541341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a3c107900 0x7f2a3c109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:04.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.636+0000 7f2a40bea640 1 --2- 172.21.15.139:0/2955541341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c104fd0 0x7f2a3c1073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:04.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.636+0000 7f2a40bea640 1 --2- 172.21.15.139:0/2955541341 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3c1026a0 0x7f2a3c104a90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:04.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.636+0000 7f2a40bea640 1 -- 172.21.15.139:0/2955541341 >> 172.21.15.139:0/2955541341 conn(0x7f2a3c0fc2a0 msgr2=0x7f2a3c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:57:04.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.636+0000 7f2a40bea640 1 -- 172.21.15.139:0/2955541341 shutdown_connections 2024-09-19T07:57:04.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.636+0000 7f2a40bea640 1 -- 172.21.15.139:0/2955541341 wait complete. 2024-09-19T07:57:04.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a40bea640 1 Processor -- start 2024-09-19T07:57:04.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a40bea640 1 -- start start 2024-09-19T07:57:04.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a40bea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c1026a0 0x7f2a3c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:57:04.676 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a40bea640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3c104fd0 0x7f2a3c1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:57:04.676 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a40bea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a3c107900 0x7f2a3c19f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:57:04.676 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a40bea640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c10d8b0 con 0x7f2a3c1026a0 2024-09-19T07:57:04.676 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a40bea640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c10d730 con 0x7f2a3c104fd0 2024-09-19T07:57:04.676 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a40bea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c10da30 con 0x7f2a3c107900 2024-09-19T07:57:04.676 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a39d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3c104fd0 0x7f2a3c1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:57:04.676 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a39d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3c104fd0 0x7f2a3c1984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:53848/0 (socket says 172.21.15.139:53848) 2024-09-19T07:57:04.676 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a39d9b640 1 -- 172.21.15.139:0/4161411670 learned_addr learned my addr 172.21.15.139:0/4161411670 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-09-19T07:57:04.676 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a3a59c640 1 --2- 172.21.15.139:0/4161411670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c1026a0 0x7f2a3c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:57:04.676 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a39d9b640 1 -- 172.21.15.139:0/4161411670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a3c107900 msgr2=0x7f2a3c19f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:04.676 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a39d9b640 1 --2- 172.21.15.139:0/4161411670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a3c107900 0x7f2a3c19f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:04.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a39d9b640 1 -- 172.21.15.139:0/4161411670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c1026a0 msgr2=0x7f2a3c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:04.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a39d9b640 1 --2- 172.21.15.139:0/4161411670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c1026a0 0x7f2a3c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:04.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a39d9b640 1 -- 172.21.15.139:0/4161411670 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3c19fba0 con 0x7f2a3c104fd0 2024-09-19T07:57:04.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a3a59c640 1 --2- 172.21.15.139:0/4161411670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c1026a0 0x7f2a3c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:57:04.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.637+0000 7f2a39d9b640 1 --2- 172.21.15.139:0/4161411670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3c104fd0 0x7f2a3c1984b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2a30005fd0 tx=0x7f2a3000dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:57:04.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.638+0000 7f2a2b7fe640 1 -- 172.21.15.139:0/4161411670 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a30017070 con 0x7f2a3c104fd0 2024-09-19T07:57:04.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.638+0000 7f2a2b7fe640 1 -- 172.21.15.139:0/4161411670 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a30009c30 con 0x7f2a3c104fd0 2024-09-19T07:57:04.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.638+0000 7f2a40bea640 1 -- 172.21.15.139:0/4161411670 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a3c19fe90 con 0x7f2a3c104fd0 2024-09-19T07:57:04.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.638+0000 7f2a2b7fe640 1 -- 172.21.15.139:0/4161411670 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a30010460 con 0x7f2a3c104fd0 2024-09-19T07:57:04.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.638+0000 7f2a40bea640 1 -- 172.21.15.139:0/4161411670 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a3c1a0340 con 0x7f2a3c104fd0 2024-09-19T07:57:04.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.639+0000 7f2a40bea640 1 -- 172.21.15.139:0/4161411670 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a3c0fdd90 con 0x7f2a3c104fd0 2024-09-19T07:57:04.678 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.640+0000 7f2a2b7fe640 1 -- 172.21.15.139:0/4161411670 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2a30003e70 con 0x7f2a3c104fd0 2024-09-19T07:57:04.678 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.640+0000 7f2a2b7fe640 1 --2- 172.21.15.139:0/4161411670 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2a240a3270 0x7f2a240a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:57:04.678 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.641+0000 7f2a2b7fe640 1 -- 172.21.15.139:0/4161411670 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3352+0+0 (secure 0 0 0) 0x7f2a300ceb10 con 0x7f2a3c104fd0 2024-09-19T07:57:04.678 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.641+0000 7f2a3a59c640 1 --2- 172.21.15.139:0/4161411670 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2a240a3270 0x7f2a240a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:57:04.678 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.641+0000 7f2a3a59c640 1 --2- 172.21.15.139:0/4161411670 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2a240a3270 0x7f2a240a5730 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c000c00 tx=0x7f2a2c0095f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:57:04.678 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.644+0000 7f2a2b7fe640 1 -- 172.21.15.139:0/4161411670 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2a30020030 con 0x7f2a3c104fd0 2024-09-19T07:57:04.823 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:04.822+0000 7f2a40bea640 1 -- 172.21.15.139:0/4161411670 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f2a3c062630 con 0x7f2a240a3270 2024-09-19T07:57:04.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:04 smithi162 ceph-mon[35260]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:57:04.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:04 smithi162 ceph-mon[35260]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-19T07:57:04.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:04.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:04.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:04.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:04.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:04.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:04.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:04.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:04.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:04 smithi139 ceph-mon[34874]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:57:04.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:04 smithi139 ceph-mon[34874]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-19T07:57:04.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:04 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:04.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:04 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:04.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:04 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:04.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:04 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:04.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:04 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:04.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:04 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:04.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:04 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:04.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:04 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:04 smithi098 ceph-mon[31333]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:57:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:04 smithi098 ceph-mon[31333]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-19T07:57:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:05.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:05 smithi139 ceph-mon[34874]: Detected new or changed devices on smithi098 2024-09-19T07:57:05.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:05 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi098 to 8544M 2024-09-19T07:57:05.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:05 smithi139 ceph-mon[34874]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-19T07:57:05.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:05 smithi139 ceph-mon[34874]: osdmap e22: 4 total, 3 up, 4 in 2024-09-19T07:57:05.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:05 smithi139 ceph-mon[34874]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-19T07:57:05.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:05 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:57:05.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:05 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:57:05.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:05 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:57:05.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:05 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:06.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:05 smithi162 ceph-mon[35260]: Detected new or changed devices on smithi098 2024-09-19T07:57:06.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:05 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi098 to 8544M 2024-09-19T07:57:06.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:05 smithi162 ceph-mon[35260]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-19T07:57:06.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:05 smithi162 ceph-mon[35260]: osdmap e22: 4 total, 3 up, 4 in 2024-09-19T07:57:06.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:05 smithi162 ceph-mon[35260]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-19T07:57:06.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:05 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:57:06.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:05 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:57:06.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:05 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:57:06.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:05 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:06.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:05 smithi098 ceph-mon[31333]: Detected new or changed devices on smithi098 2024-09-19T07:57:06.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:05 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi098 to 8544M 2024-09-19T07:57:06.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:05 smithi098 ceph-mon[31333]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-19T07:57:06.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:05 smithi098 ceph-mon[31333]: osdmap e22: 4 total, 3 up, 4 in 2024-09-19T07:57:06.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:05 smithi098 ceph-mon[31333]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-19T07:57:06.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:05 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:57:06.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:05 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:57:06.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:05 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:57:06.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:05 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:06.698 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 07:57:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3[51957]: 2024-09-19T07:57:06.434+0000 7ffa34ea2640 -1 osd.3 0 waiting for initial osdmap 2024-09-19T07:57:06.698 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 07:57:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3[51957]: 2024-09-19T07:57:06.440+0000 7ffa2c49d640 -1 osd.3 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:57:07.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:06 smithi162 ceph-mon[35260]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:57:07.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:06 smithi162 ceph-mon[35260]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:57:07.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:06 smithi162 ceph-mon[35260]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-19T07:57:07.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:06 smithi162 ceph-mon[35260]: osdmap e23: 4 total, 3 up, 4 in 2024-09-19T07:57:07.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:57:07.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:57:07.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:06 smithi162 ceph-mon[35260]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' 2024-09-19T07:57:07.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:06 smithi162 ceph-mon[35260]: osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014] boot 2024-09-19T07:57:07.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:06 smithi162 ceph-mon[35260]: osdmap e24: 4 total, 4 up, 4 in 2024-09-19T07:57:07.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:57:07.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:06 smithi139 ceph-mon[34874]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:57:07.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:06 smithi139 ceph-mon[34874]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:57:07.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:06 smithi139 ceph-mon[34874]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-19T07:57:07.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:06 smithi139 ceph-mon[34874]: osdmap e23: 4 total, 3 up, 4 in 2024-09-19T07:57:07.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:57:07.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:57:07.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:06 smithi139 ceph-mon[34874]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' 2024-09-19T07:57:07.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:06 smithi139 ceph-mon[34874]: osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014] boot 2024-09-19T07:57:07.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:06 smithi139 ceph-mon[34874]: osdmap e24: 4 total, 4 up, 4 in 2024-09-19T07:57:07.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:57:07.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:06 smithi098 ceph-mon[31333]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:57:07.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:06 smithi098 ceph-mon[31333]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-19T07:57:07.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:06 smithi098 ceph-mon[31333]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-19T07:57:07.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:06 smithi098 ceph-mon[31333]: osdmap e23: 4 total, 3 up, 4 in 2024-09-19T07:57:07.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:57:07.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:57:07.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:06 smithi098 ceph-mon[31333]: from='osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014]' entity='osd.3' 2024-09-19T07:57:07.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:06 smithi098 ceph-mon[31333]: osd.3 [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014] boot 2024-09-19T07:57:07.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:06 smithi098 ceph-mon[31333]: osdmap e24: 4 total, 4 up, 4 in 2024-09-19T07:57:07.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T07:57:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:07 smithi162 ceph-mon[35260]: purged_snaps scrub starts 2024-09-19T07:57:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:07 smithi162 ceph-mon[35260]: purged_snaps scrub ok 2024-09-19T07:57:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:07 smithi162 ceph-mon[35260]: from='client.? 172.21.15.139:0/4155403082' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4412813-df57-4ac6-8fb7-f57b69c74768"}]: dispatch 2024-09-19T07:57:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:07 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4412813-df57-4ac6-8fb7-f57b69c74768"}]: dispatch 2024-09-19T07:57:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:07 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4412813-df57-4ac6-8fb7-f57b69c74768"}]': finished 2024-09-19T07:57:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:07 smithi162 ceph-mon[35260]: osdmap e25: 5 total, 4 up, 5 in 2024-09-19T07:57:08.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:07 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:08.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:07 smithi162 ceph-mon[35260]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 521 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:08.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:07 smithi162 ceph-mon[35260]: from='client.? 172.21.15.139:0/4056040071' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:57:08.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:07 smithi139 ceph-mon[34874]: purged_snaps scrub starts 2024-09-19T07:57:08.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:07 smithi139 ceph-mon[34874]: purged_snaps scrub ok 2024-09-19T07:57:08.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:07 smithi139 ceph-mon[34874]: from='client.? 172.21.15.139:0/4155403082' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4412813-df57-4ac6-8fb7-f57b69c74768"}]: dispatch 2024-09-19T07:57:08.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:07 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4412813-df57-4ac6-8fb7-f57b69c74768"}]: dispatch 2024-09-19T07:57:08.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:07 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4412813-df57-4ac6-8fb7-f57b69c74768"}]': finished 2024-09-19T07:57:08.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:07 smithi139 ceph-mon[34874]: osdmap e25: 5 total, 4 up, 5 in 2024-09-19T07:57:08.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:07 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:08.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:07 smithi139 ceph-mon[34874]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 521 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:08.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:07 smithi139 ceph-mon[34874]: from='client.? 172.21.15.139:0/4056040071' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:57:08.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:07 smithi098 ceph-mon[31333]: purged_snaps scrub starts 2024-09-19T07:57:08.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:07 smithi098 ceph-mon[31333]: purged_snaps scrub ok 2024-09-19T07:57:08.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:07 smithi098 ceph-mon[31333]: from='client.? 172.21.15.139:0/4155403082' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4412813-df57-4ac6-8fb7-f57b69c74768"}]: dispatch 2024-09-19T07:57:08.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:07 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4412813-df57-4ac6-8fb7-f57b69c74768"}]: dispatch 2024-09-19T07:57:08.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:07 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4412813-df57-4ac6-8fb7-f57b69c74768"}]': finished 2024-09-19T07:57:08.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:07 smithi098 ceph-mon[31333]: osdmap e25: 5 total, 4 up, 5 in 2024-09-19T07:57:08.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:07 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:08.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:07 smithi098 ceph-mon[31333]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 521 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:08.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:07 smithi098 ceph-mon[31333]: from='client.? 172.21.15.139:0/4056040071' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:57:09.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:09 smithi162 ceph-mon[35260]: osdmap e26: 5 total, 4 up, 5 in 2024-09-19T07:57:09.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:09 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:09.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:09 smithi139 ceph-mon[34874]: osdmap e26: 5 total, 4 up, 5 in 2024-09-19T07:57:09.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:09 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:09.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:09 smithi098 ceph-mon[31333]: osdmap e26: 5 total, 4 up, 5 in 2024-09-19T07:57:09.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:09 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:10.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:10 smithi162 ceph-mon[35260]: osdmap e27: 5 total, 4 up, 5 in 2024-09-19T07:57:10.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:10 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:10.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:10 smithi162 ceph-mon[35260]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 521 MiB used, 357 GiB / 358 GiB avail; 150 KiB/s, 0 objects/s recovering 2024-09-19T07:57:10.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:10 smithi139 ceph-mon[34874]: osdmap e27: 5 total, 4 up, 5 in 2024-09-19T07:57:10.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:10 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:10.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:10 smithi139 ceph-mon[34874]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 521 MiB used, 357 GiB / 358 GiB avail; 150 KiB/s, 0 objects/s recovering 2024-09-19T07:57:10.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:10 smithi098 ceph-mon[31333]: osdmap e27: 5 total, 4 up, 5 in 2024-09-19T07:57:10.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:10 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:10.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:10 smithi098 ceph-mon[31333]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 521 MiB used, 357 GiB / 358 GiB avail; 150 KiB/s, 0 objects/s recovering 2024-09-19T07:57:11.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:11 smithi162 ceph-mon[35260]: osdmap e28: 5 total, 4 up, 5 in 2024-09-19T07:57:11.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:11 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:11.760 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:11 smithi098 ceph-mon[31333]: osdmap e28: 5 total, 4 up, 5 in 2024-09-19T07:57:11.760 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:11 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:11.760 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:11 smithi139 ceph-mon[34874]: osdmap e28: 5 total, 4 up, 5 in 2024-09-19T07:57:11.760 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:11 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:12.547 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:12 smithi139 ceph-mon[34874]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 521 MiB used, 357 GiB / 358 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-09-19T07:57:12.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:12 smithi162 ceph-mon[35260]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 521 MiB used, 357 GiB / 358 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-09-19T07:57:12.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:12 smithi098 ceph-mon[31333]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 521 MiB used, 357 GiB / 358 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-09-19T07:57:14.851 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:14 smithi139 ceph-mon[34874]: pgmap v102: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-19T07:57:14.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:14 smithi162 ceph-mon[35260]: pgmap v102: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-19T07:57:14.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:14 smithi098 ceph-mon[31333]: pgmap v102: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-19T07:57:16.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:16 smithi162 ceph-mon[35260]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-19T07:57:16.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-19T07:57:16.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:16 smithi139 ceph-mon[34874]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-19T07:57:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-19T07:57:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:16.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:16 smithi098 ceph-mon[31333]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-19T07:57:16.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-19T07:57:16.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:17.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:17 smithi162 ceph-mon[35260]: Deploying daemon osd.4 on smithi139 2024-09-19T07:57:17.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:17 smithi139 ceph-mon[34874]: Deploying daemon osd.4 on smithi139 2024-09-19T07:57:17.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:17 smithi098 ceph-mon[31333]: Deploying daemon osd.4 on smithi139 2024-09-19T07:57:18.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:18 smithi162 ceph-mon[35260]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:18.899 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:18 smithi139 ceph-mon[34874]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:18.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:18 smithi098 ceph-mon[31333]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:20.762 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:20 smithi139 ceph-mon[34874]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:20.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:20 smithi162 ceph-mon[35260]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:20 smithi098 ceph-mon[31333]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:22.660 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:22 smithi139 ceph-mon[34874]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:22.660 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:22 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:22.660 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:22 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:22.661 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:22 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:22.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:22 smithi162 ceph-mon[35260]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:22.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:22 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:22.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:22 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:22.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:22 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:22.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:22 smithi098 ceph-mon[31333]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:22.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:22 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:22.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:22 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:22.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:22 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:24.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:24 smithi162 ceph-mon[35260]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:24.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:24 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:24.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:24 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:24.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:24 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:24.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:24 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:24.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:24 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:24.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:24 smithi139 ceph-mon[34874]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:24.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:24 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:24.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:24 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:24.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:24 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:24.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:24 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:24.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:24 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:24.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:24 smithi098 ceph-mon[31333]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:24.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:24 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:24.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:24 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:24.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:24 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:24.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:24 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:24.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:24 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:25.739 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 4 on host 'smithi139' 2024-09-19T07:57:25.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:25.738+0000 7f2a2b7fe640 1 -- 172.21.15.139:0/4161411670 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2a3c062630 con 0x7f2a240a3270 2024-09-19T07:57:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:25.741+0000 7f2a40bea640 1 -- 172.21.15.139:0/4161411670 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2a240a3270 msgr2=0x7f2a240a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:25.741+0000 7f2a40bea640 1 --2- 172.21.15.139:0/4161411670 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2a240a3270 0x7f2a240a5730 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c000c00 tx=0x7f2a2c0095f0 comp rx=0 tx=0).stop 2024-09-19T07:57:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:25.741+0000 7f2a40bea640 1 -- 172.21.15.139:0/4161411670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3c104fd0 msgr2=0x7f2a3c1984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:25.741+0000 7f2a40bea640 1 --2- 172.21.15.139:0/4161411670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3c104fd0 0x7f2a3c1984b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2a30005fd0 tx=0x7f2a3000dde0 comp rx=0 tx=0).stop 2024-09-19T07:57:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:25.741+0000 7f2a40bea640 1 -- 172.21.15.139:0/4161411670 shutdown_connections 2024-09-19T07:57:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:25.741+0000 7f2a40bea640 1 --2- 172.21.15.139:0/4161411670 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2a240a3270 0x7f2a240a5730 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:25.741+0000 7f2a40bea640 1 --2- 172.21.15.139:0/4161411670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a3c107900 0x7f2a3c19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:25.741+0000 7f2a40bea640 1 --2- 172.21.15.139:0/4161411670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3c104fd0 0x7f2a3c1984b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:25.741+0000 7f2a40bea640 1 --2- 172.21.15.139:0/4161411670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c1026a0 0x7f2a3c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:25.741+0000 7f2a40bea640 1 -- 172.21.15.139:0/4161411670 >> 172.21.15.139:0/4161411670 conn(0x7f2a3c0fc2a0 msgr2=0x7f2a3c103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:57:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:25.741+0000 7f2a40bea640 1 -- 172.21.15.139:0/4161411670 shutdown_connections 2024-09-19T07:57:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:57:25.741+0000 7f2a40bea640 1 -- 172.21.15.139:0/4161411670 wait complete. 2024-09-19T07:57:26.346 DEBUG:teuthology.orchestra.run.smithi139:osd.4> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.4.service 2024-09-19T07:57:26.348 INFO:tasks.cephadm:Deploying osd.5 on smithi162 with /dev/vg_nvme/lv_3... 2024-09-19T07:57:26.348 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-19T07:57:26.603 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T07:57:26.670 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:26 smithi139 ceph-mon[34874]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:26.670 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:26 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:26.671 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:26 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:26.671 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:26 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:26.721 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:26 smithi162 ceph-mon[35260]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:26.721 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:26 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:26.721 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:26 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:26.721 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:26 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:26.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:26 smithi098 ceph-mon[31333]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:26.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:26 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:26.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:26 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:26.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:26 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:28.582 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 07:57:28 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4[45921]: 2024-09-19T07:57:28.180+0000 7f767092a640 -1 osd.4 0 log_to_monitors true 2024-09-19T07:57:28.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:28 smithi162 ceph-mon[35260]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:28.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:28 smithi162 ceph-mon[35260]: from='osd.4 [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-19T07:57:28.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:28 smithi162 ceph-mon[35260]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-19T07:57:28.901 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:57:28.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_3 2024-09-19T07:57:28.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:28 smithi139 ceph-mon[34874]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:28.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:28 smithi139 ceph-mon[34874]: from='osd.4 [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-19T07:57:28.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:28 smithi139 ceph-mon[34874]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-19T07:57:28.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:28 smithi098 ceph-mon[31333]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:28.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:28 smithi098 ceph-mon[31333]: from='osd.4 [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-19T07:57:28.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:28 smithi098 ceph-mon[31333]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-19T07:57:29.179 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T07:57:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.594+0000 7f31062e9640 1 -- 172.21.15.162:0/3237945447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31001079a0 msgr2=0x7f3100109d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.594+0000 7f31062e9640 1 --2- 172.21.15.162:0/3237945447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31001079a0 0x7f3100109d90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f30f8009ca0 tx=0x7f30f802a8c0 comp rx=0 tx=0).stop 2024-09-19T07:57:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.594+0000 7f31062e9640 1 -- 172.21.15.162:0/3237945447 shutdown_connections 2024-09-19T07:57:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.594+0000 7f31062e9640 1 --2- 172.21.15.162:0/3237945447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31001079a0 0x7f3100109d90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.594+0000 7f31062e9640 1 --2- 172.21.15.162:0/3237945447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3100105070 0x7f3100107460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.594+0000 7f31062e9640 1 --2- 172.21.15.162:0/3237945447 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3100102740 0x7f3100104b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.594+0000 7f31062e9640 1 -- 172.21.15.162:0/3237945447 >> 172.21.15.162:0/3237945447 conn(0x7f31000fc2a0 msgr2=0x7f31000fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:57:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.594+0000 7f31062e9640 1 -- 172.21.15.162:0/3237945447 shutdown_connections 2024-09-19T07:57:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.594+0000 7f31062e9640 1 -- 172.21.15.162:0/3237945447 wait complete. 2024-09-19T07:57:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.595+0000 7f31062e9640 1 Processor -- start 2024-09-19T07:57:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.595+0000 7f31062e9640 1 -- start start 2024-09-19T07:57:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.595+0000 7f31062e9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3100102740 0x7f3100197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:57:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.595+0000 7f31062e9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3100105070 0x7f31001983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:57:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.595+0000 7f31062e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31001079a0 0x7f310019f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:57:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.595+0000 7f31062e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f310010d8f0 con 0x7f31001079a0 2024-09-19T07:57:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.595+0000 7f31062e9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f310010d770 con 0x7f3100102740 2024-09-19T07:57:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.595+0000 7f31062e9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f310010da70 con 0x7f3100105070 2024-09-19T07:57:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.595+0000 7f30ff7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3100105070 0x7f31001983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:57:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.595+0000 7f30ff7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3100105070 0x7f31001983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:50042/0 (socket says 172.21.15.162:50042) 2024-09-19T07:57:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.595+0000 7f30ff7fe640 1 -- 172.21.15.162:0/2415583821 learned_addr learned my addr 172.21.15.162:0/2415583821 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-19T07:57:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.595+0000 7f30fffff640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3100102740 0x7f3100197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:57:29.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.596+0000 7f3104886640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31001079a0 0x7f310019f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:57:29.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.596+0000 7f30ff7fe640 1 -- 172.21.15.162:0/2415583821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3100102740 msgr2=0x7f3100197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:29.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.596+0000 7f30ff7fe640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3100102740 0x7f3100197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:29.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.596+0000 7f30ff7fe640 1 -- 172.21.15.162:0/2415583821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31001079a0 msgr2=0x7f310019f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:29.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.596+0000 7f30ff7fe640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31001079a0 0x7f310019f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:29.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.596+0000 7f30ff7fe640 1 -- 172.21.15.162:0/2415583821 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f310019fad0 con 0x7f3100105070 2024-09-19T07:57:29.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.596+0000 7f30fffff640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3100102740 0x7f3100197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:57:29.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.596+0000 7f30ff7fe640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3100105070 0x7f31001983e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f30f4005fd0 tx=0x7f30f400dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:57:29.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.596+0000 7f30fd7fa640 1 -- 172.21.15.162:0/2415583821 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f4017070 con 0x7f3100105070 2024-09-19T07:57:29.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.596+0000 7f30fd7fa640 1 -- 172.21.15.162:0/2415583821 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30f4009c30 con 0x7f3100105070 2024-09-19T07:57:29.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.596+0000 7f31062e9640 1 -- 172.21.15.162:0/2415583821 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f310019fdc0 con 0x7f3100105070 2024-09-19T07:57:29.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.596+0000 7f30fd7fa640 1 -- 172.21.15.162:0/2415583821 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f4010460 con 0x7f3100105070 2024-09-19T07:57:29.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.596+0000 7f31062e9640 1 -- 172.21.15.162:0/2415583821 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31001a0270 con 0x7f3100105070 2024-09-19T07:57:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.598+0000 7f31062e9640 1 -- 172.21.15.162:0/2415583821 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31000fdd90 con 0x7f3100105070 2024-09-19T07:57:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.599+0000 7f30fd7fa640 1 -- 172.21.15.162:0/2415583821 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f30f4003e70 con 0x7f3100105070 2024-09-19T07:57:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.599+0000 7f30fd7fa640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30e80a3270 0x7f30e80a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:57:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.599+0000 7f30fd7fa640 1 -- 172.21.15.162:0/2415583821 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 3821+0+0 (secure 0 0 0) 0x7f30f40ce320 con 0x7f3100105070 2024-09-19T07:57:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.599+0000 7f30fffff640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30e80a3270 0x7f30e80a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:57:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.600+0000 7f30fffff640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30e80a3270 0x7f30e80a5730 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f30f00058c0 tx=0x7f30f000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:57:29.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.604+0000 7f30fd7fa640 1 -- 172.21.15.162:0/2415583821 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f30f4020030 con 0x7f3100105070 2024-09-19T07:57:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:29.779+0000 7f31062e9640 1 -- 172.21.15.162:0/2415583821 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f3100062630 con 0x7f30e80a3270 2024-09-19T07:57:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:29 smithi162 ceph-mon[35260]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-19T07:57:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:29 smithi162 ceph-mon[35260]: from='osd.4 [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:57:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:29 smithi162 ceph-mon[35260]: osdmap e29: 5 total, 4 up, 5 in 2024-09-19T07:57:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:29 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:29 smithi162 ceph-mon[35260]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:57:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:29 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:29 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:29 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:29.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:29 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:29.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:29 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:29.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:29 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:29.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:29 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:29.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:29 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:29.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:29 smithi139 ceph-mon[34874]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-19T07:57:29.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:29 smithi139 ceph-mon[34874]: from='osd.4 [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:57:29.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:29 smithi139 ceph-mon[34874]: osdmap e29: 5 total, 4 up, 5 in 2024-09-19T07:57:29.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:29 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:29.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:29 smithi139 ceph-mon[34874]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:57:29.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:29 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:29.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:29 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:29.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:29 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:29.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:29 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:29.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:29 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:29.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:29 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:29.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:29 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:29.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:29 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:29.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:29 smithi098 ceph-mon[31333]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-19T07:57:29.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:29 smithi098 ceph-mon[31333]: from='osd.4 [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:57:29.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:29 smithi098 ceph-mon[31333]: osdmap e29: 5 total, 4 up, 5 in 2024-09-19T07:57:29.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:29 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:29.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:29 smithi098 ceph-mon[31333]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:57:29.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:29 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:29.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:29 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:29.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:29 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:29.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:29 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:29.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:29 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:29.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:29 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:29.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:29 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:29.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:29 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:30.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:30 smithi162 ceph-mon[35260]: Detected new or changed devices on smithi139 2024-09-19T07:57:30.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:30 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi139 to 8544M 2024-09-19T07:57:30.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:30 smithi162 ceph-mon[35260]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:30.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:30 smithi162 ceph-mon[35260]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-19T07:57:30.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:30 smithi162 ceph-mon[35260]: osdmap e30: 5 total, 4 up, 5 in 2024-09-19T07:57:30.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:30 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:30.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:30 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:30.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:30 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:57:30.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:30 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:57:30.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:30 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:30.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:30 smithi162 ceph-mon[35260]: from='osd.4 ' entity='osd.4' 2024-09-19T07:57:30.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:30 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:30.927 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 07:57:30 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4[45921]: 2024-09-19T07:57:30.433+0000 7f766c08a640 -1 osd.4 0 waiting for initial osdmap 2024-09-19T07:57:30.927 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 07:57:30 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4[45921]: 2024-09-19T07:57:30.440+0000 7f76646bc640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:57:30.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:30 smithi139 ceph-mon[34874]: Detected new or changed devices on smithi139 2024-09-19T07:57:30.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:30 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi139 to 8544M 2024-09-19T07:57:30.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:30 smithi139 ceph-mon[34874]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:30.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:30 smithi139 ceph-mon[34874]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-19T07:57:30.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:30 smithi139 ceph-mon[34874]: osdmap e30: 5 total, 4 up, 5 in 2024-09-19T07:57:30.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:30 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:30.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:30 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:30.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:30 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:57:30.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:30 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:57:30.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:30 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:30.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:30 smithi139 ceph-mon[34874]: from='osd.4 ' entity='osd.4' 2024-09-19T07:57:30.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:30 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:30.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:30 smithi098 ceph-mon[31333]: Detected new or changed devices on smithi139 2024-09-19T07:57:30.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:30 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi139 to 8544M 2024-09-19T07:57:30.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:30 smithi098 ceph-mon[31333]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:30.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:30 smithi098 ceph-mon[31333]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-19T07:57:30.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:30 smithi098 ceph-mon[31333]: osdmap e30: 5 total, 4 up, 5 in 2024-09-19T07:57:30.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:30 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:30.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:30 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:30.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:30 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:57:30.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:30 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:57:30.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:30 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:30.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:30 smithi098 ceph-mon[31333]: from='osd.4 ' entity='osd.4' 2024-09-19T07:57:30.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:30 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:31.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:31 smithi162 ceph-mon[35260]: purged_snaps scrub starts 2024-09-19T07:57:31.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:31 smithi162 ceph-mon[35260]: purged_snaps scrub ok 2024-09-19T07:57:31.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:31 smithi162 ceph-mon[35260]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:57:31.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:31 smithi162 ceph-mon[35260]: osd.4 [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520] boot 2024-09-19T07:57:31.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:31 smithi162 ceph-mon[35260]: osdmap e31: 5 total, 5 up, 5 in 2024-09-19T07:57:31.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:31 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:31.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:31 smithi139 ceph-mon[34874]: purged_snaps scrub starts 2024-09-19T07:57:31.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:31 smithi139 ceph-mon[34874]: purged_snaps scrub ok 2024-09-19T07:57:31.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:31 smithi139 ceph-mon[34874]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:57:31.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:31 smithi139 ceph-mon[34874]: osd.4 [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520] boot 2024-09-19T07:57:31.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:31 smithi139 ceph-mon[34874]: osdmap e31: 5 total, 5 up, 5 in 2024-09-19T07:57:31.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:31 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:31.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:31 smithi098 ceph-mon[31333]: purged_snaps scrub starts 2024-09-19T07:57:31.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:31 smithi098 ceph-mon[31333]: purged_snaps scrub ok 2024-09-19T07:57:31.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:31 smithi098 ceph-mon[31333]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:57:31.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:31 smithi098 ceph-mon[31333]: osd.4 [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520] boot 2024-09-19T07:57:31.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:31 smithi098 ceph-mon[31333]: osdmap e31: 5 total, 5 up, 5 in 2024-09-19T07:57:31.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:31 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T07:57:32.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:32 smithi162 ceph-mon[35260]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:32.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:32 smithi162 ceph-mon[35260]: from='client.? 172.21.15.162:0/1208759940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c"}]: dispatch 2024-09-19T07:57:32.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:32 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c"}]: dispatch 2024-09-19T07:57:32.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:32 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c"}]': finished 2024-09-19T07:57:32.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:32 smithi162 ceph-mon[35260]: osdmap e32: 6 total, 5 up, 6 in 2024-09-19T07:57:32.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:32 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:32.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:32 smithi162 ceph-mon[35260]: from='client.? 172.21.15.162:0/2942686945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:57:32.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:32 smithi139 ceph-mon[34874]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:32.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:32 smithi139 ceph-mon[34874]: from='client.? 172.21.15.162:0/1208759940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c"}]: dispatch 2024-09-19T07:57:32.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:32 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c"}]: dispatch 2024-09-19T07:57:32.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:32 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c"}]': finished 2024-09-19T07:57:32.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:32 smithi139 ceph-mon[34874]: osdmap e32: 6 total, 5 up, 6 in 2024-09-19T07:57:32.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:32 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:32.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:32 smithi139 ceph-mon[34874]: from='client.? 172.21.15.162:0/2942686945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:57:32.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:32 smithi098 ceph-mon[31333]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-19T07:57:32.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:32 smithi098 ceph-mon[31333]: from='client.? 172.21.15.162:0/1208759940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c"}]: dispatch 2024-09-19T07:57:32.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:32 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c"}]: dispatch 2024-09-19T07:57:32.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:32 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c"}]': finished 2024-09-19T07:57:32.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:32 smithi098 ceph-mon[31333]: osdmap e32: 6 total, 5 up, 6 in 2024-09-19T07:57:32.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:32 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:32.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:32 smithi098 ceph-mon[31333]: from='client.? 172.21.15.162:0/2942686945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:57:34.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:34 smithi162 ceph-mon[35260]: osdmap e33: 6 total, 5 up, 6 in 2024-09-19T07:57:34.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:34 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:34.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:34 smithi162 ceph-mon[35260]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:34.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:34 smithi139 ceph-mon[34874]: osdmap e33: 6 total, 5 up, 6 in 2024-09-19T07:57:34.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:34 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:34.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:34 smithi139 ceph-mon[34874]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:34.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:34 smithi098 ceph-mon[31333]: osdmap e33: 6 total, 5 up, 6 in 2024-09-19T07:57:34.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:34 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:34.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:34 smithi098 ceph-mon[31333]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:36.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:36 smithi162 ceph-mon[35260]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:36.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:36 smithi139 ceph-mon[34874]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:36.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:36 smithi098 ceph-mon[31333]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:38.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:38 smithi162 ceph-mon[35260]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:38.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:38 smithi139 ceph-mon[34874]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:38.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:38 smithi098 ceph-mon[31333]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:40.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:40 smithi162 ceph-mon[35260]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:40.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-19T07:57:40.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:40.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:40 smithi139 ceph-mon[34874]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:40.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-19T07:57:40.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:40.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:40 smithi098 ceph-mon[31333]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:40.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-19T07:57:40.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:41.824 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:41 smithi162 ceph-mon[35260]: Deploying daemon osd.5 on smithi162 2024-09-19T07:57:41.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:41 smithi139 ceph-mon[34874]: Deploying daemon osd.5 on smithi162 2024-09-19T07:57:41.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:41 smithi098 ceph-mon[31333]: Deploying daemon osd.5 on smithi162 2024-09-19T07:57:42.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:42 smithi162 ceph-mon[35260]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:42.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:42 smithi139 ceph-mon[34874]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:42.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:42 smithi098 ceph-mon[31333]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:44.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:44 smithi162 ceph-mon[35260]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:44.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:44 smithi139 ceph-mon[34874]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:44.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:44 smithi098 ceph-mon[31333]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:46.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:46 smithi162 ceph-mon[35260]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:46.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:46.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:46.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:46.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:46 smithi139 ceph-mon[34874]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:46.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:46.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:46.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:46 smithi098 ceph-mon[31333]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:48.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:48 smithi162 ceph-mon[35260]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:48.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:48 smithi139 ceph-mon[34874]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:48.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:48 smithi098 ceph-mon[31333]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:49.587 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 5 on host 'smithi162' 2024-09-19T07:57:49.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:49.587+0000 7f30fd7fa640 1 -- 172.21.15.162:0/2415583821 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3100062630 con 0x7f30e80a3270 2024-09-19T07:57:49.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:49.589+0000 7f31062e9640 1 -- 172.21.15.162:0/2415583821 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30e80a3270 msgr2=0x7f30e80a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:49.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:49.589+0000 7f31062e9640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30e80a3270 0x7f30e80a5730 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f30f00058c0 tx=0x7f30f000a2d0 comp rx=0 tx=0).stop 2024-09-19T07:57:49.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:49.589+0000 7f31062e9640 1 -- 172.21.15.162:0/2415583821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3100105070 msgr2=0x7f31001983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:49.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:49.589+0000 7f31062e9640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3100105070 0x7f31001983e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f30f4005fd0 tx=0x7f30f400dde0 comp rx=0 tx=0).stop 2024-09-19T07:57:49.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:49.590+0000 7f31062e9640 1 -- 172.21.15.162:0/2415583821 shutdown_connections 2024-09-19T07:57:49.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:49.590+0000 7f31062e9640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30e80a3270 0x7f30e80a5730 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:49.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:49.590+0000 7f31062e9640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31001079a0 0x7f310019f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:49.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:49.590+0000 7f31062e9640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3100105070 0x7f31001983e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:49.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:49.590+0000 7f31062e9640 1 --2- 172.21.15.162:0/2415583821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3100102740 0x7f3100197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:49.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:49.590+0000 7f31062e9640 1 -- 172.21.15.162:0/2415583821 >> 172.21.15.162:0/2415583821 conn(0x7f31000fc2a0 msgr2=0x7f3100102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:57:49.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:49.590+0000 7f31062e9640 1 -- 172.21.15.162:0/2415583821 shutdown_connections 2024-09-19T07:57:49.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:57:49.590+0000 7f31062e9640 1 -- 172.21.15.162:0/2415583821 wait complete. 2024-09-19T07:57:50.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:50.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:50.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:49 smithi162 ceph-mon[35260]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:50.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:50.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:50.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:50.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:49 smithi139 ceph-mon[34874]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:50.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:50.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:50.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:49 smithi098 ceph-mon[31333]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:50.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:50.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:50.195 DEBUG:teuthology.orchestra.run.smithi162:osd.5> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.5.service 2024-09-19T07:57:50.198 INFO:tasks.cephadm:Deploying osd.6 on smithi098 with /dev/vg_nvme/lv_2... 2024-09-19T07:57:50.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-19T07:57:50.463 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:57:52.812 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:57:52.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:52 smithi098 ceph-mon[31333]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:52.839 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-09-19T07:57:52.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:52 smithi162 ceph-mon[35260]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:52.841 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 07:57:52 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T07:57:52.654+0000 7f6c087c6640 -1 osd.5 0 log_to_monitors true 2024-09-19T07:57:52.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:52 smithi139 ceph-mon[34874]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:53.091 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:57:53.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.512+0000 7f352883b640 1 -- 172.21.15.98:0/1290166601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3524102e00 msgr2=0x7f3524103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:53.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.512+0000 7f352883b640 1 --2- 172.21.15.98:0/1290166601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3524102e00 0x7f3524103280 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3518009900 tx=0x7f351802a410 comp rx=0 tx=0).stop 2024-09-19T07:57:53.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.512+0000 7f352883b640 1 -- 172.21.15.98:0/1290166601 shutdown_connections 2024-09-19T07:57:53.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.512+0000 7f352883b640 1 --2- 172.21.15.98:0/1290166601 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35241037c0 0x7f352410a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:53.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.512+0000 7f352883b640 1 --2- 172.21.15.98:0/1290166601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3524102e00 0x7f3524103280 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:53.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.512+0000 7f352883b640 1 --2- 172.21.15.98:0/1290166601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3524101c00 0x7f3524102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:53.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.512+0000 7f352883b640 1 -- 172.21.15.98:0/1290166601 >> 172.21.15.98:0/1290166601 conn(0x7f35240fd390 msgr2=0x7f35240ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:57:53.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.512+0000 7f352883b640 1 -- 172.21.15.98:0/1290166601 shutdown_connections 2024-09-19T07:57:53.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.512+0000 7f352883b640 1 -- 172.21.15.98:0/1290166601 wait complete. 2024-09-19T07:57:53.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.513+0000 7f352883b640 1 Processor -- start 2024-09-19T07:57:53.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.513+0000 7f352883b640 1 -- start start 2024-09-19T07:57:53.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.513+0000 7f352883b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3524101c00 0x7f3524075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:57:53.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.513+0000 7f352883b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3524102e00 0x7f35240745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:57:53.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.513+0000 7f352883b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35241037c0 0x7f3524074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:57:53.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.513+0000 7f352883b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f352410d900 con 0x7f3524101c00 2024-09-19T07:57:53.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.513+0000 7f352883b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f352410d780 con 0x7f35241037c0 2024-09-19T07:57:53.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.513+0000 7f352883b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f352410da80 con 0x7f3524102e00 2024-09-19T07:57:53.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.513+0000 7f352259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3524101c00 0x7f3524075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:57:53.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.513+0000 7f352259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3524101c00 0x7f3524075f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59606/0 (socket says 172.21.15.98:59606) 2024-09-19T07:57:53.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.513+0000 7f352259c640 1 -- 172.21.15.98:0/4176079649 learned_addr learned my addr 172.21.15.98:0/4176079649 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:57:53.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.513+0000 7f3521d9b640 1 --2- 172.21.15.98:0/4176079649 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3524102e00 0x7f35240745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:57:53.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.514+0000 7f3522d9d640 1 --2- 172.21.15.98:0/4176079649 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35241037c0 0x7f3524074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:57:53.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.514+0000 7f352259c640 1 -- 172.21.15.98:0/4176079649 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3524102e00 msgr2=0x7f35240745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:53.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.514+0000 7f352259c640 1 --2- 172.21.15.98:0/4176079649 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3524102e00 0x7f35240745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:53.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.514+0000 7f352259c640 1 -- 172.21.15.98:0/4176079649 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35241037c0 msgr2=0x7f3524074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:57:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.514+0000 7f352259c640 1 --2- 172.21.15.98:0/4176079649 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35241037c0 0x7f3524074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:57:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.514+0000 7f352259c640 1 -- 172.21.15.98:0/4176079649 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35240753e0 con 0x7f3524101c00 2024-09-19T07:57:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.514+0000 7f352259c640 1 --2- 172.21.15.98:0/4176079649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3524101c00 0x7f3524075f90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f351400d950 tx=0x7f351400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:57:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.514+0000 7f35137fe640 1 -- 172.21.15.98:0/4176079649 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514017070 con 0x7f3524101c00 2024-09-19T07:57:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.514+0000 7f352883b640 1 -- 172.21.15.98:0/4176079649 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35241a2110 con 0x7f3524101c00 2024-09-19T07:57:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.514+0000 7f35137fe640 1 -- 172.21.15.98:0/4176079649 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3514009c30 con 0x7f3524101c00 2024-09-19T07:57:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.514+0000 7f35137fe640 1 -- 172.21.15.98:0/4176079649 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351400f600 con 0x7f3524101c00 2024-09-19T07:57:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.514+0000 7f352883b640 1 -- 172.21.15.98:0/4176079649 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35241a2650 con 0x7f3524101c00 2024-09-19T07:57:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.515+0000 7f352883b640 1 -- 172.21.15.98:0/4176079649 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3524108fa0 con 0x7f3524101c00 2024-09-19T07:57:53.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.516+0000 7f35137fe640 1 -- 172.21.15.98:0/4176079649 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f351401a070 con 0x7f3524101c00 2024-09-19T07:57:53.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.517+0000 7f35137fe640 1 --2- 172.21.15.98:0/4176079649 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f350c0a3240 0x7f350c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:57:53.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.517+0000 7f3521d9b640 1 --2- 172.21.15.98:0/4176079649 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f350c0a3240 0x7f350c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:57:53.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.517+0000 7f35137fe640 1 -- 172.21.15.98:0/4176079649 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4216+0+0 (secure 0 0 0) 0x7f35140ce3b0 con 0x7f3524101c00 2024-09-19T07:57:53.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.517+0000 7f3521d9b640 1 --2- 172.21.15.98:0/4176079649 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f350c0a3240 0x7f350c0a5700 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f35180314e0 tx=0x7f3518000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:57:53.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.522+0000 7f35137fe640 1 -- 172.21.15.98:0/4176079649 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f3514009de0 con 0x7f3524101c00 2024-09-19T07:57:53.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:57:53.702+0000 7f352883b640 1 -- 172.21.15.98:0/4176079649 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3524062630 con 0x7f350c0a3240 2024-09-19T07:57:53.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:53 smithi162 ceph-mon[35260]: from='osd.5 [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-19T07:57:53.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:53 smithi162 ceph-mon[35260]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-19T07:57:53.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:53 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:53.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:53 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:53.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:53 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:53.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:53 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:53.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:53 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:53.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:53 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:53.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:53 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:53.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:53 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:53.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:53 smithi139 ceph-mon[34874]: from='osd.5 [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-19T07:57:53.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:53 smithi139 ceph-mon[34874]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-19T07:57:53.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:53 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:53.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:53 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:53.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:53 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:53.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:53 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:53.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:53 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:53.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:53 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:53.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:53 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:53.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:53 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:53 smithi098 ceph-mon[31333]: from='osd.5 [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-19T07:57:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:53 smithi098 ceph-mon[31333]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-19T07:57:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:57:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:53.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:57:54.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:54 smithi098 ceph-mon[31333]: Detected new or changed devices on smithi162 2024-09-19T07:57:54.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:54 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi162 to 8544M 2024-09-19T07:57:54.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:54 smithi098 ceph-mon[31333]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:54.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:54 smithi098 ceph-mon[31333]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-19T07:57:54.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:54 smithi098 ceph-mon[31333]: from='osd.5 [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:57:54.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:54 smithi098 ceph-mon[31333]: osdmap e34: 6 total, 5 up, 6 in 2024-09-19T07:57:54.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:54 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:54.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:54 smithi098 ceph-mon[31333]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:57:54.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:54 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:57:54.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:54 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:57:54.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:54 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:54.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:54 smithi162 ceph-mon[35260]: Detected new or changed devices on smithi162 2024-09-19T07:57:54.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:54 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi162 to 8544M 2024-09-19T07:57:54.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:54 smithi162 ceph-mon[35260]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:54.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:54 smithi162 ceph-mon[35260]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-19T07:57:54.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:54 smithi162 ceph-mon[35260]: from='osd.5 [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:57:54.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:54 smithi162 ceph-mon[35260]: osdmap e34: 6 total, 5 up, 6 in 2024-09-19T07:57:54.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:54 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:54.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:54 smithi162 ceph-mon[35260]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:57:54.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:54 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:57:54.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:54 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:57:54.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:54 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:54.872 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 07:57:54 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T07:57:54.585+0000 7f6c03f26640 -1 osd.5 0 waiting for initial osdmap 2024-09-19T07:57:54.872 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 07:57:54 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T07:57:54.593+0000 7f6c00560640 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:57:54.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:54 smithi139 ceph-mon[34874]: Detected new or changed devices on smithi162 2024-09-19T07:57:54.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:54 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi162 to 8544M 2024-09-19T07:57:54.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:54 smithi139 ceph-mon[34874]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:54.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:54 smithi139 ceph-mon[34874]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-19T07:57:54.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:54 smithi139 ceph-mon[34874]: from='osd.5 [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:57:54.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:54 smithi139 ceph-mon[34874]: osdmap e34: 6 total, 5 up, 6 in 2024-09-19T07:57:54.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:54 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:54.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:54 smithi139 ceph-mon[34874]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:57:54.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:54 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:57:54.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:54 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:57:54.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:54 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:55.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:55 smithi098 ceph-mon[31333]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:57:55.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:55 smithi098 ceph-mon[31333]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-19T07:57:55.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:55 smithi098 ceph-mon[31333]: osdmap e35: 6 total, 5 up, 6 in 2024-09-19T07:57:55.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:55.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:55.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:55 smithi162 ceph-mon[35260]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:57:55.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:55 smithi162 ceph-mon[35260]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-19T07:57:55.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:55 smithi162 ceph-mon[35260]: osdmap e35: 6 total, 5 up, 6 in 2024-09-19T07:57:55.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:55.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:55.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:55 smithi139 ceph-mon[34874]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:57:55.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:55 smithi139 ceph-mon[34874]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-19T07:57:55.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:55 smithi139 ceph-mon[34874]: osdmap e35: 6 total, 5 up, 6 in 2024-09-19T07:57:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:56.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:56 smithi162 ceph-mon[35260]: purged_snaps scrub starts 2024-09-19T07:57:56.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:56 smithi162 ceph-mon[35260]: purged_snaps scrub ok 2024-09-19T07:57:56.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:56 smithi162 ceph-mon[35260]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:56.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:56 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:56.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:56 smithi162 ceph-mon[35260]: osd.5 [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297] boot 2024-09-19T07:57:56.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:56 smithi162 ceph-mon[35260]: osdmap e36: 6 total, 6 up, 6 in 2024-09-19T07:57:56.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:56 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:56.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:56 smithi162 ceph-mon[35260]: osdmap e37: 6 total, 6 up, 6 in 2024-09-19T07:57:56.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:56 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2395600892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1c1590-6aea-40dc-90c8-e21b88896909"}]: dispatch 2024-09-19T07:57:56.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:56 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2395600892' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba1c1590-6aea-40dc-90c8-e21b88896909"}]': finished 2024-09-19T07:57:56.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:56 smithi162 ceph-mon[35260]: osdmap e38: 7 total, 6 up, 7 in 2024-09-19T07:57:56.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:56 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:57:56.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:56 smithi139 ceph-mon[34874]: purged_snaps scrub starts 2024-09-19T07:57:56.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:56 smithi139 ceph-mon[34874]: purged_snaps scrub ok 2024-09-19T07:57:56.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:56 smithi139 ceph-mon[34874]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:56.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:56 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:56.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:56 smithi139 ceph-mon[34874]: osd.5 [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297] boot 2024-09-19T07:57:56.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:56 smithi139 ceph-mon[34874]: osdmap e36: 6 total, 6 up, 6 in 2024-09-19T07:57:56.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:56 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:56.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:56 smithi139 ceph-mon[34874]: osdmap e37: 6 total, 6 up, 6 in 2024-09-19T07:57:56.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:56 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2395600892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1c1590-6aea-40dc-90c8-e21b88896909"}]: dispatch 2024-09-19T07:57:56.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:56 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2395600892' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba1c1590-6aea-40dc-90c8-e21b88896909"}]': finished 2024-09-19T07:57:56.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:56 smithi139 ceph-mon[34874]: osdmap e38: 7 total, 6 up, 7 in 2024-09-19T07:57:56.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:56 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:57:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:56 smithi098 ceph-mon[31333]: purged_snaps scrub starts 2024-09-19T07:57:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:56 smithi098 ceph-mon[31333]: purged_snaps scrub ok 2024-09-19T07:57:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:56 smithi098 ceph-mon[31333]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-19T07:57:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:56 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:56.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:56 smithi098 ceph-mon[31333]: osd.5 [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297] boot 2024-09-19T07:57:56.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:56 smithi098 ceph-mon[31333]: osdmap e36: 6 total, 6 up, 6 in 2024-09-19T07:57:56.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:56 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T07:57:56.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:56 smithi098 ceph-mon[31333]: osdmap e37: 6 total, 6 up, 6 in 2024-09-19T07:57:56.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:56 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2395600892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1c1590-6aea-40dc-90c8-e21b88896909"}]: dispatch 2024-09-19T07:57:56.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:56 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2395600892' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba1c1590-6aea-40dc-90c8-e21b88896909"}]': finished 2024-09-19T07:57:56.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:56 smithi098 ceph-mon[31333]: osdmap e38: 7 total, 6 up, 7 in 2024-09-19T07:57:56.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:56 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:57:57.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:57 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/4263970227' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:57:57.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:57 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/4263970227' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:57:57.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:57 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/4263970227' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:57:58.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:58 smithi162 ceph-mon[35260]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:57:58.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:58 smithi162 ceph-mon[35260]: osdmap e39: 7 total, 6 up, 7 in 2024-09-19T07:57:58.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:57:58.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:58 smithi139 ceph-mon[34874]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:57:58.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:58 smithi139 ceph-mon[34874]: osdmap e39: 7 total, 6 up, 7 in 2024-09-19T07:57:58.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:58 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:57:58.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:58 smithi098 ceph-mon[31333]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:57:58.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:58 smithi098 ceph-mon[31333]: osdmap e39: 7 total, 6 up, 7 in 2024-09-19T07:57:58.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:58 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:57:59.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:59 smithi162 ceph-mon[35260]: osdmap e40: 7 total, 6 up, 7 in 2024-09-19T07:57:59.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:57:59 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:57:59.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:59 smithi139 ceph-mon[34874]: osdmap e40: 7 total, 6 up, 7 in 2024-09-19T07:57:59.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:57:59 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:57:59.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:59 smithi098 ceph-mon[31333]: osdmap e40: 7 total, 6 up, 7 in 2024-09-19T07:57:59.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:57:59 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:00.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:00 smithi162 ceph-mon[35260]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-09-19T07:58:00.871 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:00 smithi098 ceph-mon[31333]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-09-19T07:58:00.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:00 smithi139 ceph-mon[34874]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-09-19T07:58:02.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:02 smithi162 ceph-mon[35260]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-19T07:58:02.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:02 smithi139 ceph-mon[34874]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-19T07:58:02.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:02 smithi098 ceph-mon[31333]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-19T07:58:04.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:04 smithi162 ceph-mon[35260]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-19T07:58:04.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-19T07:58:04.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:04.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:04 smithi139 ceph-mon[34874]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-19T07:58:04.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:04 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-19T07:58:04.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:04 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:04 smithi098 ceph-mon[31333]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-19T07:58:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-19T07:58:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:05.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:05 smithi162 ceph-mon[35260]: Deploying daemon osd.6 on smithi098 2024-09-19T07:58:05.920 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:05 smithi098 ceph-mon[31333]: Deploying daemon osd.6 on smithi098 2024-09-19T07:58:05.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:05 smithi139 ceph-mon[34874]: Deploying daemon osd.6 on smithi098 2024-09-19T07:58:06.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:06 smithi098 ceph-mon[31333]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T07:58:06.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:06 smithi139 ceph-mon[34874]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T07:58:07.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:06 smithi162 ceph-mon[35260]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T07:58:08.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:08 smithi139 ceph-mon[34874]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-19T07:58:08.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:08 smithi098 ceph-mon[31333]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-19T07:58:09.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:08 smithi162 ceph-mon[35260]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-19T07:58:10.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:10 smithi098 ceph-mon[31333]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-19T07:58:10.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:10 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:10.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:10 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:10.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:10 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:10.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:10 smithi139 ceph-mon[34874]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-19T07:58:10.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:10 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:10.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:10 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:10.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:10 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:10 smithi162 ceph-mon[35260]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-19T07:58:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:10 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:10 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:10 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:12.821 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:12 smithi098 ceph-mon[31333]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:12.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:12 smithi139 ceph-mon[34874]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:13.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:12 smithi162 ceph-mon[35260]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:14.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:58:14.091+0000 7f35137fe640 1 -- 172.21.15.98:0/4176079649 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3524062630 con 0x7f350c0a3240 2024-09-19T07:58:14.091 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 6 on host 'smithi098' 2024-09-19T07:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:58:14.093+0000 7f352883b640 1 -- 172.21.15.98:0/4176079649 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f350c0a3240 msgr2=0x7f350c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:58:14.093+0000 7f352883b640 1 --2- 172.21.15.98:0/4176079649 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f350c0a3240 0x7f350c0a5700 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f35180314e0 tx=0x7f3518000bc0 comp rx=0 tx=0).stop 2024-09-19T07:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:58:14.093+0000 7f352883b640 1 -- 172.21.15.98:0/4176079649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3524101c00 msgr2=0x7f3524075f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:58:14.093+0000 7f352883b640 1 --2- 172.21.15.98:0/4176079649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3524101c00 0x7f3524075f90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f351400d950 tx=0x7f351400de10 comp rx=0 tx=0).stop 2024-09-19T07:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:58:14.093+0000 7f352883b640 1 -- 172.21.15.98:0/4176079649 shutdown_connections 2024-09-19T07:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:58:14.093+0000 7f352883b640 1 --2- 172.21.15.98:0/4176079649 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f350c0a3240 0x7f350c0a5700 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:58:14.093+0000 7f352883b640 1 --2- 172.21.15.98:0/4176079649 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35241037c0 0x7f3524074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:58:14.093+0000 7f352883b640 1 --2- 172.21.15.98:0/4176079649 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3524102e00 0x7f35240745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:58:14.093+0000 7f352883b640 1 --2- 172.21.15.98:0/4176079649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3524101c00 0x7f3524075f90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:58:14.093+0000 7f352883b640 1 -- 172.21.15.98:0/4176079649 >> 172.21.15.98:0/4176079649 conn(0x7f35240fd390 msgr2=0x7f3524106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:58:14.093+0000 7f352883b640 1 -- 172.21.15.98:0/4176079649 shutdown_connections 2024-09-19T07:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:58:14.093+0000 7f352883b640 1 -- 172.21.15.98:0/4176079649 wait complete. 2024-09-19T07:58:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:14 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:14 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:14 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:14 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:58:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:14 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:14 smithi162 ceph-mon[35260]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:58:14.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:14 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:14.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:14 smithi139 ceph-mon[34874]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:14.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:14 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:14.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:14 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:14.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:14 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:14.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:14 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:58:14.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:14 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:14.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:14 smithi098 ceph-mon[31333]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:14.765 DEBUG:teuthology.orchestra.run.smithi098:osd.6> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.6.service 2024-09-19T07:58:14.767 INFO:tasks.cephadm:Deploying osd.7 on smithi139 with /dev/vg_nvme/lv_2... 2024-09-19T07:58:14.768 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-19T07:58:15.023 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T07:58:15.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:15 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:15.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:15 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:15.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:15 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:15.370 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:15 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:15.370 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:15 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:15.370 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:15 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:15.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:15 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:15.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:15 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:15.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:15 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:16.165 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:16 smithi098 ceph-mon[31333]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:16 smithi162 ceph-mon[35260]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:16.415 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:16 smithi139 ceph-mon[34874]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:16.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 07:58:16 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T07:58:16.408+0000 7f827f456640 -1 osd.6 0 log_to_monitors true 2024-09-19T07:58:17.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:17 smithi098 ceph-mon[31333]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-19T07:58:17.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:17 smithi139 ceph-mon[34874]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-19T07:58:17.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:17 smithi162 ceph-mon[35260]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-19T07:58:17.659 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:58:17.686 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_2 2024-09-19T07:58:17.943 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T07:58:18.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-19T07:58:18.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: osdmap e41: 7 total, 6 up, 7 in 2024-09-19T07:58:18.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-19T07:58:18.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:18.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:18.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: Detected new or changed devices on smithi098 2024-09-19T07:58:18.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:18.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:18.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:18.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:18.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:18.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi098 to 5696M 2024-09-19T07:58:18.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:18.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:18.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:58:18.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:18 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-19T07:58:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: osdmap e41: 7 total, 6 up, 7 in 2024-09-19T07:58:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-19T07:58:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: Detected new or changed devices on smithi098 2024-09-19T07:58:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi098 to 5696M 2024-09-19T07:58:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:58:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:18 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:18.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.409+0000 7fccb244e640 1 -- 172.21.15.139:0/1426395597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccac100b10 msgr2=0x7fccac100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:58:18.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.409+0000 7fccb244e640 1 --2- 172.21.15.139:0/1426395597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccac100b10 0x7fccac100f10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009900 tx=0x7fcc9c02a410 comp rx=0 tx=0).stop 2024-09-19T07:58:18.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.409+0000 7fccb244e640 1 -- 172.21.15.139:0/1426395597 shutdown_connections 2024-09-19T07:58:18.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.409+0000 7fccb244e640 1 --2- 172.21.15.139:0/1426395597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccac1026d0 0x7fccac108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:18.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.409+0000 7fccb244e640 1 --2- 172.21.15.139:0/1426395597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccac101d10 0x7fccac102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:18.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.409+0000 7fccb244e640 1 --2- 172.21.15.139:0/1426395597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccac100b10 0x7fccac100f10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:18.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.409+0000 7fccb244e640 1 -- 172.21.15.139:0/1426395597 >> 172.21.15.139:0/1426395597 conn(0x7fccac0fc2a0 msgr2=0x7fccac0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:58:18.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.409+0000 7fccb244e640 1 -- 172.21.15.139:0/1426395597 shutdown_connections 2024-09-19T07:58:18.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.409+0000 7fccb244e640 1 -- 172.21.15.139:0/1426395597 wait complete. 2024-09-19T07:58:18.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.410+0000 7fccb244e640 1 Processor -- start 2024-09-19T07:58:18.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.410+0000 7fccb244e640 1 -- start start 2024-09-19T07:58:18.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.410+0000 7fccb244e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccac100b10 0x7fccac196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:58:18.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.410+0000 7fccb244e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccac101d10 0x7fccac1973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:58:18.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.410+0000 7fccb244e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccac1026d0 0x7fccac19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:58:18.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.410+0000 7fccb244e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fccac10c7f0 con 0x7fccac100b10 2024-09-19T07:58:18.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.410+0000 7fccb244e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fccac10c670 con 0x7fccac1026d0 2024-09-19T07:58:18.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.410+0000 7fccb244e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fccac10c970 con 0x7fccac101d10 2024-09-19T07:58:18.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.410+0000 7fccb09eb640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccac1026d0 0x7fccac19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:58:18.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.410+0000 7fccb09eb640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccac1026d0 0x7fccac19e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:39284/0 (socket says 172.21.15.139:39284) 2024-09-19T07:58:18.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.410+0000 7fccabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccac100b10 0x7fccac196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:58:18.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.410+0000 7fccb09eb640 1 -- 172.21.15.139:0/723923668 learned_addr learned my addr 172.21.15.139:0/723923668 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-09-19T07:58:18.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.411+0000 7fccb09eb640 1 -- 172.21.15.139:0/723923668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccac101d10 msgr2=0x7fccac1973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:58:18.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.411+0000 7fccab7fe640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccac101d10 0x7fccac1973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:58:18.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.411+0000 7fccb09eb640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccac101d10 0x7fccac1973c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:18.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.411+0000 7fccb09eb640 1 -- 172.21.15.139:0/723923668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccac100b10 msgr2=0x7fccac196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:58:18.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.411+0000 7fccb09eb640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccac100b10 0x7fccac196e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:18.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.411+0000 7fccb09eb640 1 -- 172.21.15.139:0/723923668 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccac19eab0 con 0x7fccac1026d0 2024-09-19T07:58:18.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.411+0000 7fccab7fe640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccac101d10 0x7fccac1973c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:58:18.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.411+0000 7fccabfff640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccac100b10 0x7fccac196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:58:18.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.411+0000 7fccb09eb640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccac1026d0 0x7fccac19e440 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcca400eaa0 tx=0x7fcca400ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:58:18.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.411+0000 7fcca97fa640 1 -- 172.21.15.139:0/723923668 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca400cdb0 con 0x7fccac1026d0 2024-09-19T07:58:18.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.411+0000 7fccb244e640 1 -- 172.21.15.139:0/723923668 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccac19eda0 con 0x7fccac1026d0 2024-09-19T07:58:18.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.411+0000 7fcca97fa640 1 -- 172.21.15.139:0/723923668 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcca4005020 con 0x7fccac1026d0 2024-09-19T07:58:18.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.411+0000 7fcca97fa640 1 -- 172.21.15.139:0/723923668 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca40116b0 con 0x7fccac1026d0 2024-09-19T07:58:18.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.411+0000 7fccb244e640 1 -- 172.21.15.139:0/723923668 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccac19f2d0 con 0x7fccac1026d0 2024-09-19T07:58:18.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.412+0000 7fccb244e640 1 -- 172.21.15.139:0/723923668 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccac107f60 con 0x7fccac1026d0 2024-09-19T07:58:18.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.413+0000 7fcca97fa640 1 -- 172.21.15.139:0/723923668 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fcca4005560 con 0x7fccac1026d0 2024-09-19T07:58:18.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.414+0000 7fcca97fa640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fcc980a3270 0x7fcc980a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:58:18.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.414+0000 7fcca97fa640 1 -- 172.21.15.139:0/723923668 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4685+0+0 (secure 0 0 0) 0x7fcca40cf090 con 0x7fccac1026d0 2024-09-19T07:58:18.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.414+0000 7fccabfff640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fcc980a3270 0x7fcc980a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:58:18.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.415+0000 7fccabfff640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fcc980a3270 0x7fcc980a5730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c005ab0 tx=0x7fcc9c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:58:18.421 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.417+0000 7fcca97fa640 1 -- 172.21.15.139:0/723923668 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fcca4016030 con 0x7fccac1026d0 2024-09-19T07:58:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-19T07:58:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: osdmap e41: 7 total, 6 up, 7 in 2024-09-19T07:58:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-19T07:58:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: Detected new or changed devices on smithi098 2024-09-19T07:58:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi098 to 5696M 2024-09-19T07:58:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:58:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:18.593 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:18.592+0000 7fccb244e640 1 -- 172.21.15.139:0/723923668 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fccac062630 con 0x7fcc980a3270 2024-09-19T07:58:19.182 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T07:58:18.894+0000 7f827abb6640 -1 osd.6 0 waiting for initial osdmap 2024-09-19T07:58:19.182 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 07:58:18 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T07:58:18.906+0000 7f82731e8640 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:58:19.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:19 smithi098 ceph-mon[31333]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-19T07:58:19.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:19 smithi098 ceph-mon[31333]: osdmap e42: 7 total, 6 up, 7 in 2024-09-19T07:58:19.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:19.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:19.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:19 smithi098 ceph-mon[31333]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:58:19.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:58:19.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:58:19.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:19.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:19 smithi098 ceph-mon[31333]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' 2024-09-19T07:58:19.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:19 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:19.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:19 smithi139 ceph-mon[34874]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-19T07:58:19.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:19 smithi139 ceph-mon[34874]: osdmap e42: 7 total, 6 up, 7 in 2024-09-19T07:58:19.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:19.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:19.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:19 smithi139 ceph-mon[34874]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:58:19.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:58:19.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:58:19.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:19.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:19 smithi139 ceph-mon[34874]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' 2024-09-19T07:58:19.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:19 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:19.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:19 smithi162 ceph-mon[35260]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-19T07:58:19.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:19 smithi162 ceph-mon[35260]: osdmap e42: 7 total, 6 up, 7 in 2024-09-19T07:58:19.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:19.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:19.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:19 smithi162 ceph-mon[35260]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:58:19.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:58:19.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:58:19.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:19.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:19 smithi162 ceph-mon[35260]: from='osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267]' entity='osd.6' 2024-09-19T07:58:19.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:19 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:20.176 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:20 smithi139 ceph-mon[34874]: purged_snaps scrub starts 2024-09-19T07:58:20.176 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:20 smithi139 ceph-mon[34874]: purged_snaps scrub ok 2024-09-19T07:58:20.176 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:20 smithi139 ceph-mon[34874]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:20.176 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:20 smithi139 ceph-mon[34874]: osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267] boot 2024-09-19T07:58:20.176 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:20 smithi139 ceph-mon[34874]: osdmap e43: 7 total, 7 up, 7 in 2024-09-19T07:58:20.176 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:20.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:20 smithi098 ceph-mon[31333]: purged_snaps scrub starts 2024-09-19T07:58:20.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:20 smithi098 ceph-mon[31333]: purged_snaps scrub ok 2024-09-19T07:58:20.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:20 smithi098 ceph-mon[31333]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:20.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:20 smithi098 ceph-mon[31333]: osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267] boot 2024-09-19T07:58:20.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:20 smithi098 ceph-mon[31333]: osdmap e43: 7 total, 7 up, 7 in 2024-09-19T07:58:20.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:20.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:20 smithi162 ceph-mon[35260]: purged_snaps scrub starts 2024-09-19T07:58:20.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:20 smithi162 ceph-mon[35260]: purged_snaps scrub ok 2024-09-19T07:58:20.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:20 smithi162 ceph-mon[35260]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-19T07:58:20.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:20 smithi162 ceph-mon[35260]: osd.6 [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267] boot 2024-09-19T07:58:20.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:20 smithi162 ceph-mon[35260]: osdmap e43: 7 total, 7 up, 7 in 2024-09-19T07:58:20.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T07:58:21.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:21 smithi162 ceph-mon[35260]: from='client.? 172.21.15.139:0/3468008620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb9093ad-a068-4a63-ab87-0b50522cbc29"}]: dispatch 2024-09-19T07:58:21.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:21 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb9093ad-a068-4a63-ab87-0b50522cbc29"}]: dispatch 2024-09-19T07:58:21.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:21 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb9093ad-a068-4a63-ab87-0b50522cbc29"}]': finished 2024-09-19T07:58:21.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:21 smithi162 ceph-mon[35260]: osdmap e44: 8 total, 7 up, 8 in 2024-09-19T07:58:21.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:21 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:21.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:21 smithi139 ceph-mon[34874]: from='client.? 172.21.15.139:0/3468008620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb9093ad-a068-4a63-ab87-0b50522cbc29"}]: dispatch 2024-09-19T07:58:21.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:21 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb9093ad-a068-4a63-ab87-0b50522cbc29"}]: dispatch 2024-09-19T07:58:21.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:21 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb9093ad-a068-4a63-ab87-0b50522cbc29"}]': finished 2024-09-19T07:58:21.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:21 smithi139 ceph-mon[34874]: osdmap e44: 8 total, 7 up, 8 in 2024-09-19T07:58:21.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:21 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:21.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:21 smithi098 ceph-mon[31333]: from='client.? 172.21.15.139:0/3468008620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb9093ad-a068-4a63-ab87-0b50522cbc29"}]: dispatch 2024-09-19T07:58:21.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:21 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb9093ad-a068-4a63-ab87-0b50522cbc29"}]: dispatch 2024-09-19T07:58:21.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:21 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb9093ad-a068-4a63-ab87-0b50522cbc29"}]': finished 2024-09-19T07:58:21.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:21 smithi098 ceph-mon[31333]: osdmap e44: 8 total, 7 up, 8 in 2024-09-19T07:58:21.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:21 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:22.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:22 smithi162 ceph-mon[35260]: from='client.? 172.21.15.139:0/651952983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:58:22.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:22 smithi162 ceph-mon[35260]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:22.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:22 smithi139 ceph-mon[34874]: from='client.? 172.21.15.139:0/651952983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:58:22.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:22 smithi139 ceph-mon[34874]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:22.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:22 smithi098 ceph-mon[31333]: from='client.? 172.21.15.139:0/651952983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:58:22.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:22 smithi098 ceph-mon[31333]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:24.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:24 smithi162 ceph-mon[35260]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:24.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:24 smithi139 ceph-mon[34874]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:24.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:24 smithi098 ceph-mon[31333]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:26.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:26 smithi162 ceph-mon[35260]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:26.897 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:26 smithi139 ceph-mon[34874]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:26.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:26 smithi098 ceph-mon[31333]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:28.828 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:28 smithi139 ceph-mon[34874]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:28.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:28 smithi162 ceph-mon[35260]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:28.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:28 smithi098 ceph-mon[31333]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:29 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-19T07:58:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:29 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:29.877 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:29 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-19T07:58:29.877 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:29 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:29.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:29 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-19T07:58:29.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:29 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:30.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:30 smithi162 ceph-mon[35260]: Deploying daemon osd.7 on smithi139 2024-09-19T07:58:30.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:30 smithi162 ceph-mon[35260]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:30.887 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:30 smithi139 ceph-mon[34874]: Deploying daemon osd.7 on smithi139 2024-09-19T07:58:30.887 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:30 smithi139 ceph-mon[34874]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:30.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:30 smithi098 ceph-mon[31333]: Deploying daemon osd.7 on smithi139 2024-09-19T07:58:30.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:30 smithi098 ceph-mon[31333]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:32.855 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:32 smithi139 ceph-mon[34874]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:32 smithi162 ceph-mon[35260]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:32.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:32 smithi098 ceph-mon[31333]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:34.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:34 smithi139 ceph-mon[34874]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:34.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:34 smithi162 ceph-mon[35260]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:34.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:34 smithi098 ceph-mon[31333]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:35.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:35 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:35.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:35 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:35.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:35 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:35.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:35 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:35.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:35 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:35.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:35 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:35.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:35 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:35.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:35 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:35.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:35 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:36.672 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:36 smithi139 ceph-mon[34874]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:36.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:36 smithi162 ceph-mon[35260]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:36.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:36 smithi098 ceph-mon[31333]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:38.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:38 smithi162 ceph-mon[35260]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:38.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:38 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:38.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:38 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:38.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:38 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:38.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:38 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:58:38.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:38 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:38.906 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:38 smithi139 ceph-mon[34874]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:38.906 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:38 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:38.906 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:38 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:38.907 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:38 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:38.907 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:38 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:58:38.907 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:38 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:38.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:38 smithi098 ceph-mon[31333]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:38.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:38 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:38.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:38 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:38.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:38 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:38.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:38 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:58:38.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:38 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:38.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:38.949+0000 7fcca97fa640 1 -- 172.21.15.139:0/723923668 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fccac062630 con 0x7fcc980a3270 2024-09-19T07:58:38.951 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 7 on host 'smithi139' 2024-09-19T07:58:38.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:38.951+0000 7fccb244e640 1 -- 172.21.15.139:0/723923668 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fcc980a3270 msgr2=0x7fcc980a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:58:38.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:38.951+0000 7fccb244e640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fcc980a3270 0x7fcc980a5730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c005ab0 tx=0x7fcc9c000bc0 comp rx=0 tx=0).stop 2024-09-19T07:58:38.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:38.951+0000 7fccb244e640 1 -- 172.21.15.139:0/723923668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccac1026d0 msgr2=0x7fccac19e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:58:38.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:38.951+0000 7fccb244e640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccac1026d0 0x7fccac19e440 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcca400eaa0 tx=0x7fcca400ef60 comp rx=0 tx=0).stop 2024-09-19T07:58:38.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:38.951+0000 7fccb244e640 1 -- 172.21.15.139:0/723923668 shutdown_connections 2024-09-19T07:58:38.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:38.951+0000 7fccb244e640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fcc980a3270 0x7fcc980a5730 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:38.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:38.951+0000 7fccb244e640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccac1026d0 0x7fccac19e440 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:38.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:38.951+0000 7fccb244e640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccac101d10 0x7fccac1973c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:38.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:38.951+0000 7fccb244e640 1 --2- 172.21.15.139:0/723923668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccac100b10 0x7fccac196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:38.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:38.951+0000 7fccb244e640 1 -- 172.21.15.139:0/723923668 >> 172.21.15.139:0/723923668 conn(0x7fccac0fc2a0 msgr2=0x7fccac105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:58:38.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:38.951+0000 7fccb244e640 1 -- 172.21.15.139:0/723923668 shutdown_connections 2024-09-19T07:58:38.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:58:38.951+0000 7fccb244e640 1 -- 172.21.15.139:0/723923668 wait complete. 2024-09-19T07:58:39.595 DEBUG:teuthology.orchestra.run.smithi139:osd.7> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.7.service 2024-09-19T07:58:39.597 INFO:tasks.cephadm:Deploying osd.8 on smithi162 with /dev/vg_nvme/lv_2... 2024-09-19T07:58:39.597 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-19T07:58:39.854 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T07:58:39.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:39 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:39.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:39 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:39.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:39 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:39.910 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:39 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:39.910 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:39 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:39.910 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:39 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:39.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:39 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:39.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:39 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:39.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:39 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:40.775 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:40 smithi139 ceph-mon[34874]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:40.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:40 smithi162 ceph-mon[35260]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:40.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:40 smithi098 ceph-mon[31333]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:41.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:41 smithi162 ceph-mon[35260]: from='osd.7 [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-19T07:58:41.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:41 smithi162 ceph-mon[35260]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-19T07:58:41.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:41 smithi139 ceph-mon[34874]: from='osd.7 [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-19T07:58:41.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:41 smithi139 ceph-mon[34874]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-19T07:58:41.928 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 07:58:41 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T07:58:41.592+0000 7f3306a69640 -1 osd.7 0 log_to_monitors true 2024-09-19T07:58:41.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:41 smithi098 ceph-mon[31333]: from='osd.7 [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-19T07:58:41.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:41 smithi098 ceph-mon[31333]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-19T07:58:42.165 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:58:42.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_2 2024-09-19T07:58:42.447 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T07:58:42.816 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:42.816 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-19T07:58:42.816 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: osdmap e45: 8 total, 7 up, 8 in 2024-09-19T07:58:42.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:42.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: from='osd.7 [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:58:42.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:58:42.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:42.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:42.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:42.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:42.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:42.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:42.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:42.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:58:42.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:42.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.902+0000 7f34c12d4640 1 -- 172.21.15.162:0/3458971327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34bc1026c0 msgr2=0x7f34bc108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:58:42.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.902+0000 7f34c12d4640 1 --2- 172.21.15.162:0/3458971327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34bc1026c0 0x7f34bc108f50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f34b4009ca0 tx=0x7f34b402a8c0 comp rx=0 tx=0).stop 2024-09-19T07:58:42.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.902+0000 7f34c12d4640 1 -- 172.21.15.162:0/3458971327 shutdown_connections 2024-09-19T07:58:42.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.902+0000 7f34c12d4640 1 --2- 172.21.15.162:0/3458971327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34bc1026c0 0x7f34bc108f50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:42.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.902+0000 7f34c12d4640 1 --2- 172.21.15.162:0/3458971327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc101d00 0x7f34bc102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:42.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.902+0000 7f34c12d4640 1 --2- 172.21.15.162:0/3458971327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34bc100b00 0x7f34bc100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:42.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.902+0000 7f34c12d4640 1 -- 172.21.15.162:0/3458971327 >> 172.21.15.162:0/3458971327 conn(0x7f34bc0fc290 msgr2=0x7f34bc0fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:58:42.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.902+0000 7f34c12d4640 1 -- 172.21.15.162:0/3458971327 shutdown_connections 2024-09-19T07:58:42.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.902+0000 7f34c12d4640 1 -- 172.21.15.162:0/3458971327 wait complete. 2024-09-19T07:58:42.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.902+0000 7f34c12d4640 1 Processor -- start 2024-09-19T07:58:42.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34c12d4640 1 -- start start 2024-09-19T07:58:42.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34c12d4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34bc100b00 0x7f34bc104a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:58:42.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34c12d4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34bc101d00 0x7f34bc1030e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:58:42.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34c12d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc1026c0 0x7f34bc103620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:58:42.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34c12d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34bc10c780 con 0x7f34bc1026c0 2024-09-19T07:58:42.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34c12d4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f34bc10c600 con 0x7f34bc101d00 2024-09-19T07:58:42.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34c12d4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f34bc10c900 con 0x7f34bc100b00 2024-09-19T07:58:42.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34baffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34bc100b00 0x7f34bc104a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:58:42.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34baffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34bc100b00 0x7f34bc104a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38198/0 (socket says 172.21.15.162:38198) 2024-09-19T07:58:42.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34baffd640 1 -- 172.21.15.162:0/3676447725 learned_addr learned my addr 172.21.15.162:0/3676447725 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-19T07:58:42.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34ba7fc640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34bc101d00 0x7f34bc1030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:58:42.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34bb7fe640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc1026c0 0x7f34bc103620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:58:42.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34baffd640 1 -- 172.21.15.162:0/3676447725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34bc101d00 msgr2=0x7f34bc1030e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:58:42.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34baffd640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34bc101d00 0x7f34bc1030e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:42.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34baffd640 1 -- 172.21.15.162:0/3676447725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc1026c0 msgr2=0x7f34bc103620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:58:42.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34baffd640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc1026c0 0x7f34bc103620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:58:42.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34baffd640 1 -- 172.21.15.162:0/3676447725 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34bc103ee0 con 0x7f34bc100b00 2024-09-19T07:58:42.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34bb7fe640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc1026c0 0x7f34bc103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:58:42.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.903+0000 7f34ba7fc640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34bc101d00 0x7f34bc1030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:58:42.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.904+0000 7f34baffd640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34bc100b00 0x7f34bc104a90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00e9c0 tx=0x7f34ac00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:58:42.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.904+0000 7f34a3fff640 1 -- 172.21.15.162:0/3676447725 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac00cd80 con 0x7f34bc100b00 2024-09-19T07:58:42.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.904+0000 7f34a3fff640 1 -- 172.21.15.162:0/3676447725 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34ac0035f0 con 0x7f34bc100b00 2024-09-19T07:58:42.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.904+0000 7f34c12d4640 1 -- 172.21.15.162:0/3676447725 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34bc1a1ff0 con 0x7f34bc100b00 2024-09-19T07:58:42.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.904+0000 7f34a3fff640 1 -- 172.21.15.162:0/3676447725 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac010690 con 0x7f34bc100b00 2024-09-19T07:58:42.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.904+0000 7f34c12d4640 1 -- 172.21.15.162:0/3676447725 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34bc1a23e0 con 0x7f34bc100b00 2024-09-19T07:58:42.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.905+0000 7f34c12d4640 1 -- 172.21.15.162:0/3676447725 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34bc1a2700 con 0x7f34bc100b00 2024-09-19T07:58:42.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.906+0000 7f34a3fff640 1 -- 172.21.15.162:0/3676447725 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f34ac0037a0 con 0x7f34bc100b00 2024-09-19T07:58:42.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.907+0000 7f34a3fff640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f34a40a3240 0x7f34a40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:58:42.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.907+0000 7f34a3fff640 1 -- 172.21.15.162:0/3676447725 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5117+0+0 (secure 0 0 0) 0x7f34ac0cf050 con 0x7f34bc100b00 2024-09-19T07:58:42.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.907+0000 7f34ba7fc640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f34a40a3240 0x7f34a40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:58:42.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.908+0000 7f34ba7fc640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f34a40a3240 0x7f34a40a5700 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f34bc104810 tx=0x7f34b0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:58:42.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:42.911+0000 7f34a3fff640 1 -- 172.21.15.162:0/3676447725 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f34ac09c9f0 con 0x7f34bc100b00 2024-09-19T07:58:42.927 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T07:58:42.612+0000 7f33021c9640 -1 osd.7 0 waiting for initial osdmap 2024-09-19T07:58:42.927 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T07:58:42.621+0000 7f32fe803640 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:58:42.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-19T07:58:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: osdmap e45: 8 total, 7 up, 8 in 2024-09-19T07:58:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: from='osd.7 [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:58:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:58:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:58:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-19T07:58:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: osdmap e45: 8 total, 7 up, 8 in 2024-09-19T07:58:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: from='osd.7 [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:58:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:58:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:42.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:42.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:42.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:42.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:58:42.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:42.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:42.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:58:42.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:43.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:58:43.089+0000 7f34c12d4640 1 -- 172.21.15.162:0/3676447725 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f34bc06aea0 con 0x7f34a40a3240 2024-09-19T07:58:43.785 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:43 smithi162 ceph-mon[35260]: Detected new or changed devices on smithi139 2024-09-19T07:58:43.785 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:43 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi139 to 5696M 2024-09-19T07:58:43.785 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:43 smithi162 ceph-mon[35260]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-19T07:58:43.785 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:43 smithi162 ceph-mon[35260]: osdmap e46: 8 total, 7 up, 8 in 2024-09-19T07:58:43.785 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:43 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:43.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:43 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:43.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:43 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:58:43.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:43 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:58:43.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:43 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:43.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:43 smithi139 ceph-mon[34874]: Detected new or changed devices on smithi139 2024-09-19T07:58:43.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:43 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi139 to 5696M 2024-09-19T07:58:43.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:43 smithi139 ceph-mon[34874]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-19T07:58:43.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:43 smithi139 ceph-mon[34874]: osdmap e46: 8 total, 7 up, 8 in 2024-09-19T07:58:43.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:43 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:43.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:43 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:43.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:43 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:58:43.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:43 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:58:43.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:43 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:43.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:43 smithi098 ceph-mon[31333]: Detected new or changed devices on smithi139 2024-09-19T07:58:43.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:43 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi139 to 5696M 2024-09-19T07:58:43.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:43 smithi098 ceph-mon[31333]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-19T07:58:43.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:43 smithi098 ceph-mon[31333]: osdmap e46: 8 total, 7 up, 8 in 2024-09-19T07:58:43.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:43 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:43.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:43 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:43.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:43 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:58:43.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:43 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:58:43.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:43 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:44.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:44 smithi162 ceph-mon[35260]: purged_snaps scrub starts 2024-09-19T07:58:44.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:44 smithi162 ceph-mon[35260]: purged_snaps scrub ok 2024-09-19T07:58:44.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:44 smithi162 ceph-mon[35260]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:58:44.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:44 smithi162 ceph-mon[35260]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:44.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:44 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:44.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:44 smithi162 ceph-mon[35260]: osd.7 [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650] boot 2024-09-19T07:58:44.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:44 smithi162 ceph-mon[35260]: osdmap e47: 8 total, 8 up, 8 in 2024-09-19T07:58:44.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:44 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:44.908 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:44 smithi139 ceph-mon[34874]: purged_snaps scrub starts 2024-09-19T07:58:44.908 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:44 smithi139 ceph-mon[34874]: purged_snaps scrub ok 2024-09-19T07:58:44.908 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:44 smithi139 ceph-mon[34874]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:58:44.908 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:44 smithi139 ceph-mon[34874]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:44.908 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:44 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:44.908 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:44 smithi139 ceph-mon[34874]: osd.7 [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650] boot 2024-09-19T07:58:44.909 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:44 smithi139 ceph-mon[34874]: osdmap e47: 8 total, 8 up, 8 in 2024-09-19T07:58:44.909 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:44 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:44 smithi098 ceph-mon[31333]: purged_snaps scrub starts 2024-09-19T07:58:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:44 smithi098 ceph-mon[31333]: purged_snaps scrub ok 2024-09-19T07:58:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:44 smithi098 ceph-mon[31333]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:58:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:44 smithi098 ceph-mon[31333]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-19T07:58:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:44 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:44 smithi098 ceph-mon[31333]: osd.7 [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650] boot 2024-09-19T07:58:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:44 smithi098 ceph-mon[31333]: osdmap e47: 8 total, 8 up, 8 in 2024-09-19T07:58:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:44 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T07:58:45.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:45 smithi139 ceph-mon[34874]: osdmap e48: 8 total, 8 up, 8 in 2024-09-19T07:58:45.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:45 smithi139 ceph-mon[34874]: from='client.? 172.21.15.162:0/395418135' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31ca330a-fc45-466d-81c2-ef4386b3cc6e"}]: dispatch 2024-09-19T07:58:45.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:45 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31ca330a-fc45-466d-81c2-ef4386b3cc6e"}]: dispatch 2024-09-19T07:58:45.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:45 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31ca330a-fc45-466d-81c2-ef4386b3cc6e"}]': finished 2024-09-19T07:58:45.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:45 smithi139 ceph-mon[34874]: osdmap e49: 9 total, 8 up, 9 in 2024-09-19T07:58:45.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:45 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:58:45.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:45 smithi098 ceph-mon[31333]: osdmap e48: 8 total, 8 up, 8 in 2024-09-19T07:58:45.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:45 smithi098 ceph-mon[31333]: from='client.? 172.21.15.162:0/395418135' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31ca330a-fc45-466d-81c2-ef4386b3cc6e"}]: dispatch 2024-09-19T07:58:45.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:45 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31ca330a-fc45-466d-81c2-ef4386b3cc6e"}]: dispatch 2024-09-19T07:58:45.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:45 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31ca330a-fc45-466d-81c2-ef4386b3cc6e"}]': finished 2024-09-19T07:58:45.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:45 smithi098 ceph-mon[31333]: osdmap e49: 9 total, 8 up, 9 in 2024-09-19T07:58:45.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:45 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:58:45.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:45 smithi162 ceph-mon[35260]: osdmap e48: 8 total, 8 up, 8 in 2024-09-19T07:58:45.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:45 smithi162 ceph-mon[35260]: from='client.? 172.21.15.162:0/395418135' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31ca330a-fc45-466d-81c2-ef4386b3cc6e"}]: dispatch 2024-09-19T07:58:45.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:45 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31ca330a-fc45-466d-81c2-ef4386b3cc6e"}]: dispatch 2024-09-19T07:58:45.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:45 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31ca330a-fc45-466d-81c2-ef4386b3cc6e"}]': finished 2024-09-19T07:58:45.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:45 smithi162 ceph-mon[35260]: osdmap e49: 9 total, 8 up, 9 in 2024-09-19T07:58:45.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:45 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:58:46.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:46 smithi139 ceph-mon[34874]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:46.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:46 smithi139 ceph-mon[34874]: osdmap e50: 9 total, 8 up, 9 in 2024-09-19T07:58:46.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:58:46.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:46 smithi139 ceph-mon[34874]: from='client.? 172.21.15.162:0/2670276697' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:58:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:46 smithi098 ceph-mon[31333]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:46 smithi098 ceph-mon[31333]: osdmap e50: 9 total, 8 up, 9 in 2024-09-19T07:58:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:58:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:46 smithi098 ceph-mon[31333]: from='client.? 172.21.15.162:0/2670276697' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:58:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:46 smithi162 ceph-mon[35260]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:46 smithi162 ceph-mon[35260]: osdmap e50: 9 total, 8 up, 9 in 2024-09-19T07:58:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:58:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:46 smithi162 ceph-mon[35260]: from='client.? 172.21.15.162:0/2670276697' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:58:47.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:47 smithi139 ceph-mon[34874]: osdmap e51: 9 total, 8 up, 9 in 2024-09-19T07:58:47.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:47 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:58:47.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:47 smithi098 ceph-mon[31333]: osdmap e51: 9 total, 8 up, 9 in 2024-09-19T07:58:47.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:47 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:58:48.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:47 smithi162 ceph-mon[35260]: osdmap e51: 9 total, 8 up, 9 in 2024-09-19T07:58:48.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:47 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:58:48.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:48 smithi139 ceph-mon[34874]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:48.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:48 smithi098 ceph-mon[31333]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:49.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:48 smithi162 ceph-mon[35260]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:50.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:50 smithi162 ceph-mon[35260]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:50.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:50 smithi098 ceph-mon[31333]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:51.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:50 smithi139 ceph-mon[34874]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:52.813 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:52 smithi162 ceph-mon[35260]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:53.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:52 smithi139 ceph-mon[34874]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:53.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:52 smithi098 ceph-mon[31333]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:54.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:53 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-19T07:58:54.076 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:53 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:54.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:53 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-19T07:58:54.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:53 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:54.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-19T07:58:54.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:53 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:54.955 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:54 smithi162 ceph-mon[35260]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:54.955 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:54 smithi162 ceph-mon[35260]: Deploying daemon osd.8 on smithi162 2024-09-19T07:58:55.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:54 smithi139 ceph-mon[34874]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:55.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:54 smithi139 ceph-mon[34874]: Deploying daemon osd.8 on smithi162 2024-09-19T07:58:55.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:54 smithi098 ceph-mon[31333]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:55.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:54 smithi098 ceph-mon[31333]: Deploying daemon osd.8 on smithi162 2024-09-19T07:58:55.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:55 smithi162 ceph-mon[35260]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:56.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:55 smithi139 ceph-mon[34874]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:56.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:55 smithi098 ceph-mon[31333]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:58.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:58 smithi162 ceph-mon[35260]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:58.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:58 smithi139 ceph-mon[34874]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:58.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:58 smithi098 ceph-mon[31333]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:58:59.858 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:59 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:59.858 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:59 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:59.858 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:58:59 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:59.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:59 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:59.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:59 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:59.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:58:59 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:59.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:59 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:59.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:59 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:58:59.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:58:59 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:00.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:00 smithi162 ceph-mon[35260]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:59:00.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:00 smithi139 ceph-mon[34874]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:59:00.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:00 smithi098 ceph-mon[31333]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:59:02.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:02.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:02.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:02.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:02.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:02.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:02 smithi162 ceph-mon[35260]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:59:02.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:02.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:02.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:02.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:02.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:02.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:02 smithi139 ceph-mon[34874]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:59:02.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:02.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:02.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:02.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:02.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:02.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:02 smithi098 ceph-mon[31333]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:59:02.718 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 8 on host 'smithi162' 2024-09-19T07:59:02.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:02.717+0000 7f34a3fff640 1 -- 172.21.15.162:0/3676447725 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f34bc06aea0 con 0x7f34a40a3240 2024-09-19T07:59:02.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:02.719+0000 7f34c12d4640 1 -- 172.21.15.162:0/3676447725 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f34a40a3240 msgr2=0x7f34a40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:02.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:02.719+0000 7f34c12d4640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f34a40a3240 0x7f34a40a5700 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f34bc104810 tx=0x7f34b0005f90 comp rx=0 tx=0).stop 2024-09-19T07:59:02.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:02.719+0000 7f34c12d4640 1 -- 172.21.15.162:0/3676447725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34bc100b00 msgr2=0x7f34bc104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:02.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:02.719+0000 7f34c12d4640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34bc100b00 0x7f34bc104a90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00e9c0 tx=0x7f34ac00ee80 comp rx=0 tx=0).stop 2024-09-19T07:59:02.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:02.719+0000 7f34c12d4640 1 -- 172.21.15.162:0/3676447725 shutdown_connections 2024-09-19T07:59:02.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:02.719+0000 7f34c12d4640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f34a40a3240 0x7f34a40a5700 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:02.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:02.719+0000 7f34c12d4640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc1026c0 0x7f34bc103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:02.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:02.719+0000 7f34c12d4640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34bc101d00 0x7f34bc1030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:02.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:02.719+0000 7f34c12d4640 1 --2- 172.21.15.162:0/3676447725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34bc100b00 0x7f34bc104a90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:02.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:02.719+0000 7f34c12d4640 1 -- 172.21.15.162:0/3676447725 >> 172.21.15.162:0/3676447725 conn(0x7f34bc0fc290 msgr2=0x7f34bc06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:59:02.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:02.719+0000 7f34c12d4640 1 -- 172.21.15.162:0/3676447725 shutdown_connections 2024-09-19T07:59:02.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:02.719+0000 7f34c12d4640 1 -- 172.21.15.162:0/3676447725 wait complete. 2024-09-19T07:59:03.289 DEBUG:teuthology.orchestra.run.smithi162:osd.8> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.8.service 2024-09-19T07:59:03.291 INFO:tasks.cephadm:Deploying osd.9 on smithi098 with /dev/vg_nvme/lv_1... 2024-09-19T07:59:03.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-19T07:59:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:59:03.570 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:03.570 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:03.570 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:03 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:03.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:03.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:03.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:03 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:03.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:03 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:03.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:03 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:03.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:03 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:04.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:04 smithi162 ceph-mon[35260]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:59:04.583 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:04 smithi139 ceph-mon[34874]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:59:04.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:04 smithi098 ceph-mon[31333]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:59:05.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:05 smithi098 ceph-mon[31333]: from='osd.8 [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-19T07:59:05.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:05 smithi098 ceph-mon[31333]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-19T07:59:05.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:05 smithi162 ceph-mon[35260]: from='osd.8 [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-19T07:59:05.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:05 smithi162 ceph-mon[35260]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-19T07:59:05.606 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 07:59:05 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T07:59:05.326+0000 7f6dfac00640 -1 osd.8 0 log_to_monitors true 2024-09-19T07:59:05.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:05 smithi139 ceph-mon[34874]: from='osd.8 [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-19T07:59:05.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:05 smithi139 ceph-mon[34874]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-19T07:59:06.130 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T07:59:06.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2024-09-19T07:59:06.409 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T07:59:06.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-19T07:59:06.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: from='osd.8 [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:59:06.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: osdmap e52: 9 total, 8 up, 9 in 2024-09-19T07:59:06.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:59:06.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:59:06.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:59:06.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:06.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:06.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:06.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:06.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:06.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:06.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:06.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:06.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:06.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:06 smithi098 ceph-mon[31333]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-19T07:59:06.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-19T07:59:06.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: from='osd.8 [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:59:06.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: osdmap e52: 9 total, 8 up, 9 in 2024-09-19T07:59:06.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:59:06.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:59:06.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:59:06.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:06.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:06.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:06.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:06.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:06.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:06.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:06.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:06.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:06.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:06 smithi139 ceph-mon[34874]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-19T07:59:06.870 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T07:59:06.390+0000 7f6df6360640 -1 osd.8 0 waiting for initial osdmap 2024-09-19T07:59:06.870 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T07:59:06.403+0000 7f6df299a640 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:59:06.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-19T07:59:06.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: from='osd.8 [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:59:06.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: osdmap e52: 9 total, 8 up, 9 in 2024-09-19T07:59:06.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:59:06.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T07:59:06.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-19T07:59:06.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:06.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:06.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:06.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:06.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:06.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:06.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:06.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:06.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:06.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:06 smithi162 ceph-mon[35260]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-19T07:59:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.872+0000 7f2e02854640 1 -- 172.21.15.98:0/3349717717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc104fd0 msgr2=0x7f2dfc1073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.872+0000 7f2e02854640 1 --2- 172.21.15.98:0/3349717717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc104fd0 0x7f2dfc1073c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009900 tx=0x7f2df002a410 comp rx=0 tx=0).stop 2024-09-19T07:59:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.872+0000 7f2e02854640 1 -- 172.21.15.98:0/3349717717 shutdown_connections 2024-09-19T07:59:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.872+0000 7f2e02854640 1 --2- 172.21.15.98:0/3349717717 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2dfc107900 0x7f2dfc109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.872+0000 7f2e02854640 1 --2- 172.21.15.98:0/3349717717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc104fd0 0x7f2dfc1073c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.872+0000 7f2e02854640 1 --2- 172.21.15.98:0/3349717717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2dfc1026a0 0x7f2dfc104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:06.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.872+0000 7f2e02854640 1 -- 172.21.15.98:0/3349717717 >> 172.21.15.98:0/3349717717 conn(0x7f2dfc0fc280 msgr2=0x7f2dfc0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:59:06.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.872+0000 7f2e02854640 1 -- 172.21.15.98:0/3349717717 shutdown_connections 2024-09-19T07:59:06.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.872+0000 7f2e02854640 1 -- 172.21.15.98:0/3349717717 wait complete. 2024-09-19T07:59:06.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2e02854640 1 Processor -- start 2024-09-19T07:59:06.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2e02854640 1 -- start start 2024-09-19T07:59:06.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2e02854640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc1026a0 0x7f2dfc197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:59:06.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2e02854640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2dfc104fd0 0x7f2dfc1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:59:06.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2e02854640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2dfc107900 0x7f2dfc19f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:59:06.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2e02854640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc10d850 con 0x7f2dfc1026a0 2024-09-19T07:59:06.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2e02854640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc10d6d0 con 0x7f2dfc107900 2024-09-19T07:59:06.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2e02854640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc10d9d0 con 0x7f2dfc104fd0 2024-09-19T07:59:06.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2dfbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc1026a0 0x7f2dfc197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:59:06.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2dfbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc1026a0 0x7f2dfc197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43720/0 (socket says 172.21.15.98:43720) 2024-09-19T07:59:06.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2dfbfff640 1 -- 172.21.15.98:0/606802003 learned_addr learned my addr 172.21.15.98:0/606802003 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T07:59:06.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2dfb7fe640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2dfc104fd0 0x7f2dfc1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:59:06.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2e00df1640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2dfc107900 0x7f2dfc19f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:59:06.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2dfbfff640 1 -- 172.21.15.98:0/606802003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2dfc104fd0 msgr2=0x7f2dfc1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:06.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2dfbfff640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2dfc104fd0 0x7f2dfc1984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:06.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2dfbfff640 1 -- 172.21.15.98:0/606802003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2dfc107900 msgr2=0x7f2dfc19f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:06.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2dfbfff640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2dfc107900 0x7f2dfc19f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:06.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2dfbfff640 1 -- 172.21.15.98:0/606802003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dfc19fba0 con 0x7f2dfc1026a0 2024-09-19T07:59:06.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.873+0000 7f2e00df1640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2dfc107900 0x7f2dfc19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:59:06.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.874+0000 7f2dfbfff640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc1026a0 0x7f2dfc197f70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00e9f0 tx=0x7f2dec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:59:06.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.874+0000 7f2df97fa640 1 -- 172.21.15.98:0/606802003 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec00cd80 con 0x7f2dfc1026a0 2024-09-19T07:59:06.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.874+0000 7f2df97fa640 1 -- 172.21.15.98:0/606802003 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dec005500 con 0x7f2dfc1026a0 2024-09-19T07:59:06.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.874+0000 7f2e02854640 1 -- 172.21.15.98:0/606802003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dfc19fe90 con 0x7f2dfc1026a0 2024-09-19T07:59:06.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.874+0000 7f2df97fa640 1 -- 172.21.15.98:0/606802003 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec010690 con 0x7f2dfc1026a0 2024-09-19T07:59:06.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.874+0000 7f2e02854640 1 -- 172.21.15.98:0/606802003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dfc1a02e0 con 0x7f2dfc1026a0 2024-09-19T07:59:06.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.875+0000 7f2e02854640 1 -- 172.21.15.98:0/606802003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dfc0fdd70 con 0x7f2dfc1026a0 2024-09-19T07:59:06.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.876+0000 7f2df97fa640 1 -- 172.21.15.98:0/606802003 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2dec0046c0 con 0x7f2dfc1026a0 2024-09-19T07:59:06.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.876+0000 7f2df97fa640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2de40a32f0 0x7f2de40a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:59:06.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.877+0000 7f2dfb7fe640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2de40a32f0 0x7f2de40a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:59:06.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.877+0000 7f2df97fa640 1 -- 172.21.15.98:0/606802003 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5549+0+0 (secure 0 0 0) 0x7f2dec0cf230 con 0x7f2dfc1026a0 2024-09-19T07:59:06.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.877+0000 7f2dfb7fe640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2de40a32f0 0x7f2de40a57b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2df0005ab0 tx=0x7f2df0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:59:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:06.881+0000 7f2df97fa640 1 -- 172.21.15.98:0/606802003 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2dec09ca20 con 0x7f2dfc1026a0 2024-09-19T07:59:07.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:07.055+0000 7f2e02854640 1 -- 172.21.15.98:0/606802003 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2dfc062630 con 0x7f2de40a32f0 2024-09-19T07:59:07.677 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:07 smithi098 ceph-mon[31333]: Detected new or changed devices on smithi162 2024-09-19T07:59:07.677 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:07 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi162 to 5696M 2024-09-19T07:59:07.677 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:07 smithi098 ceph-mon[31333]: osdmap e53: 9 total, 8 up, 9 in 2024-09-19T07:59:07.677 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:07 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:59:07.677 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:07 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:59:07.677 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:07 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:59:07.677 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:07 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:59:07.677 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:07 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:07.677 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:07 smithi098 ceph-mon[31333]: osd.8 [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458] boot 2024-09-19T07:59:07.678 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:07 smithi098 ceph-mon[31333]: osdmap e54: 9 total, 9 up, 9 in 2024-09-19T07:59:07.678 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:07 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:59:07.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:07 smithi139 ceph-mon[34874]: Detected new or changed devices on smithi162 2024-09-19T07:59:07.679 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:07 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi162 to 5696M 2024-09-19T07:59:07.680 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:07 smithi139 ceph-mon[34874]: osdmap e53: 9 total, 8 up, 9 in 2024-09-19T07:59:07.680 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:07 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:59:07.680 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:07 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:59:07.680 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:07 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:59:07.680 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:07 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:59:07.680 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:07 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:07.680 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:07 smithi139 ceph-mon[34874]: osd.8 [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458] boot 2024-09-19T07:59:07.680 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:07 smithi139 ceph-mon[34874]: osdmap e54: 9 total, 9 up, 9 in 2024-09-19T07:59:07.680 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:07 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:59:07.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:07 smithi162 ceph-mon[35260]: Detected new or changed devices on smithi162 2024-09-19T07:59:07.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:07 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi162 to 5696M 2024-09-19T07:59:07.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:07 smithi162 ceph-mon[35260]: osdmap e53: 9 total, 8 up, 9 in 2024-09-19T07:59:07.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:07 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:59:07.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:07 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:59:07.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:07 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:59:07.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:07 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:59:07.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:07 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:07.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:07 smithi162 ceph-mon[35260]: osd.8 [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458] boot 2024-09-19T07:59:07.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:07 smithi162 ceph-mon[35260]: osdmap e54: 9 total, 9 up, 9 in 2024-09-19T07:59:07.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:07 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T07:59:08.642 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:08 smithi098 ceph-mon[31333]: purged_snaps scrub starts 2024-09-19T07:59:08.642 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:08 smithi098 ceph-mon[31333]: purged_snaps scrub ok 2024-09-19T07:59:08.642 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:08 smithi098 ceph-mon[31333]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:59:08.642 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:08 smithi098 ceph-mon[31333]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:08.642 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:08 smithi098 ceph-mon[31333]: osdmap e55: 9 total, 9 up, 9 in 2024-09-19T07:59:08.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:08 smithi139 ceph-mon[34874]: purged_snaps scrub starts 2024-09-19T07:59:08.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:08 smithi139 ceph-mon[34874]: purged_snaps scrub ok 2024-09-19T07:59:08.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:08 smithi139 ceph-mon[34874]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:59:08.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:08 smithi139 ceph-mon[34874]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:08.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:08 smithi139 ceph-mon[34874]: osdmap e55: 9 total, 9 up, 9 in 2024-09-19T07:59:08.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:08 smithi162 ceph-mon[35260]: purged_snaps scrub starts 2024-09-19T07:59:08.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:08 smithi162 ceph-mon[35260]: purged_snaps scrub ok 2024-09-19T07:59:08.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:08 smithi162 ceph-mon[35260]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:59:08.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:08 smithi162 ceph-mon[35260]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:08.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:08 smithi162 ceph-mon[35260]: osdmap e55: 9 total, 9 up, 9 in 2024-09-19T07:59:09.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:09 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3917868630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c4f216f-aef1-4367-9c58-b913b0c4f7db"}]: dispatch 2024-09-19T07:59:09.711 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:09 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3917868630' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c4f216f-aef1-4367-9c58-b913b0c4f7db"}]': finished 2024-09-19T07:59:09.712 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:09 smithi098 ceph-mon[31333]: osdmap e56: 10 total, 9 up, 10 in 2024-09-19T07:59:09.712 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:09 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:09.712 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:09 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3917868630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c4f216f-aef1-4367-9c58-b913b0c4f7db"}]: dispatch 2024-09-19T07:59:09.712 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:09 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3917868630' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c4f216f-aef1-4367-9c58-b913b0c4f7db"}]': finished 2024-09-19T07:59:09.713 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:09 smithi139 ceph-mon[34874]: osdmap e56: 10 total, 9 up, 10 in 2024-09-19T07:59:09.713 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:09 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:09.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:09 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3917868630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c4f216f-aef1-4367-9c58-b913b0c4f7db"}]: dispatch 2024-09-19T07:59:09.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:09 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3917868630' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c4f216f-aef1-4367-9c58-b913b0c4f7db"}]': finished 2024-09-19T07:59:09.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:09 smithi162 ceph-mon[35260]: osdmap e56: 10 total, 9 up, 10 in 2024-09-19T07:59:09.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:09 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:10.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:10 smithi162 ceph-mon[35260]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:10.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:10 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2534066947' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:59:10.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:10 smithi139 ceph-mon[34874]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:10.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:10 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2534066947' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:59:10.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:10 smithi098 ceph-mon[31333]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:10.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:10 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2534066947' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:59:11.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:11 smithi162 ceph-mon[35260]: osdmap e57: 10 total, 9 up, 10 in 2024-09-19T07:59:11.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:11 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:11.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:11 smithi162 ceph-mon[35260]: osdmap e58: 10 total, 9 up, 10 in 2024-09-19T07:59:11.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:11 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:11.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:11 smithi139 ceph-mon[34874]: osdmap e57: 10 total, 9 up, 10 in 2024-09-19T07:59:11.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:11 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:11.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:11 smithi139 ceph-mon[34874]: osdmap e58: 10 total, 9 up, 10 in 2024-09-19T07:59:11.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:11 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:11.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:11 smithi098 ceph-mon[31333]: osdmap e57: 10 total, 9 up, 10 in 2024-09-19T07:59:11.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:11 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:11.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:11 smithi098 ceph-mon[31333]: osdmap e58: 10 total, 9 up, 10 in 2024-09-19T07:59:11.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:11 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:12.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:12 smithi162 ceph-mon[35260]: pgmap v191: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-19T07:59:12.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:12 smithi139 ceph-mon[34874]: pgmap v191: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-19T07:59:12.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:12 smithi098 ceph-mon[31333]: pgmap v191: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-19T07:59:14.840 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:14 smithi098 ceph-mon[31333]: pgmap v192: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-09-19T07:59:14.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:14 smithi162 ceph-mon[35260]: pgmap v192: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-09-19T07:59:14.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:14 smithi139 ceph-mon[34874]: pgmap v192: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-09-19T07:59:16.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:16 smithi162 ceph-mon[35260]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-19T07:59:16.902 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:16 smithi098 ceph-mon[31333]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-19T07:59:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:16 smithi139 ceph-mon[34874]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-19T07:59:18.728 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:18 smithi098 ceph-mon[31333]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T07:59:18.728 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:18 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-19T07:59:18.729 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:18 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:18.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:18 smithi162 ceph-mon[35260]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T07:59:18.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:18 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-19T07:59:18.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:18 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:18.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:18 smithi139 ceph-mon[34874]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T07:59:18.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:18 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-19T07:59:18.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:18 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:19.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:19 smithi162 ceph-mon[35260]: Deploying daemon osd.9 on smithi098 2024-09-19T07:59:19.885 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:19 smithi098 ceph-mon[31333]: Deploying daemon osd.9 on smithi098 2024-09-19T07:59:19.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:19 smithi139 ceph-mon[34874]: Deploying daemon osd.9 on smithi098 2024-09-19T07:59:20.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:20 smithi098 ceph-mon[31333]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:20.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:20 smithi162 ceph-mon[35260]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:20.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:20 smithi139 ceph-mon[34874]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:22.787 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:22 smithi098 ceph-mon[31333]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:22.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:22 smithi162 ceph-mon[35260]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:22.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:22 smithi139 ceph-mon[34874]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:24.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:24 smithi162 ceph-mon[35260]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:24.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:24 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:24.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:24 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:24.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:24 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:24.883 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:24 smithi098 ceph-mon[31333]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:24.883 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:24 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:24.883 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:24 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:24.883 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:24 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:24.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:24 smithi139 ceph-mon[34874]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:24.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:24 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:24.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:24 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:24.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:24 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:26.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:26 smithi162 ceph-mon[35260]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:26.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:26 smithi139 ceph-mon[34874]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:26.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:26 smithi098 ceph-mon[31333]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:27.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:27.763+0000 7f2df97fa640 1 -- 172.21.15.98:0/606802003 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2dfc062630 con 0x7f2de40a32f0 2024-09-19T07:59:27.764 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 9 on host 'smithi098' 2024-09-19T07:59:27.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:27.765+0000 7f2e02854640 1 -- 172.21.15.98:0/606802003 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2de40a32f0 msgr2=0x7f2de40a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:27.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:27.765+0000 7f2e02854640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2de40a32f0 0x7f2de40a57b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2df0005ab0 tx=0x7f2df0000bc0 comp rx=0 tx=0).stop 2024-09-19T07:59:27.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:27.765+0000 7f2e02854640 1 -- 172.21.15.98:0/606802003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc1026a0 msgr2=0x7f2dfc197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:27.765+0000 7f2e02854640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc1026a0 0x7f2dfc197f70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00e9f0 tx=0x7f2dec00edf0 comp rx=0 tx=0).stop 2024-09-19T07:59:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:27.766+0000 7f2e02854640 1 -- 172.21.15.98:0/606802003 shutdown_connections 2024-09-19T07:59:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:27.766+0000 7f2e02854640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2de40a32f0 0x7f2de40a57b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:27.766+0000 7f2e02854640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2dfc107900 0x7f2dfc19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:27.766+0000 7f2e02854640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2dfc104fd0 0x7f2dfc1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:27.766+0000 7f2e02854640 1 --2- 172.21.15.98:0/606802003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc1026a0 0x7f2dfc197f70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:27.766+0000 7f2e02854640 1 -- 172.21.15.98:0/606802003 >> 172.21.15.98:0/606802003 conn(0x7f2dfc0fc280 msgr2=0x7f2dfc103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:59:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:27.766+0000 7f2e02854640 1 -- 172.21.15.98:0/606802003 shutdown_connections 2024-09-19T07:59:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T07:59:27.766+0000 7f2e02854640 1 -- 172.21.15.98:0/606802003 wait complete. 2024-09-19T07:59:28.354 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.354 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.354 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:28.355 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:28.355 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.355 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:28 smithi098 ceph-mon[31333]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:28.355 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:28.355 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.355 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:28 smithi162 ceph-mon[35260]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:28.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.381 DEBUG:teuthology.orchestra.run.smithi098:osd.9> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.9.service 2024-09-19T07:59:28.384 INFO:tasks.cephadm:Deploying osd.10 on smithi139 with /dev/vg_nvme/lv_1... 2024-09-19T07:59:28.384 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-19T07:59:28.413 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.413 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.413 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:28.413 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:28.413 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.413 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:28 smithi139 ceph-mon[34874]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:28.414 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:28.414 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.414 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:28.640 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T07:59:30.182 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 07:59:30 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T07:59:30.112+0000 7f8bdee16640 -1 osd.9 0 log_to_monitors true 2024-09-19T07:59:30.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:30 smithi162 ceph-mon[35260]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:30.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:30 smithi162 ceph-mon[35260]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-19T07:59:30.880 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:30 smithi098 ceph-mon[31333]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:30.880 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:30 smithi098 ceph-mon[31333]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-19T07:59:30.918 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:30 smithi139 ceph-mon[34874]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:30.918 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:30 smithi139 ceph-mon[34874]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-19T07:59:31.035 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-19T07:59:31.061 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_1 2024-09-19T07:59:31.350 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T07:59:31.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-19T07:59:31.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: osdmap e59: 10 total, 9 up, 10 in 2024-09-19T07:59:31.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-19T07:59:31.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:31.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:31.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:31.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:31.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:31.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:31.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:31.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:31.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:31.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:31.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:31.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:31 smithi098 ceph-mon[31333]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-19T07:59:31.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-19T07:59:31.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: osdmap e59: 10 total, 9 up, 10 in 2024-09-19T07:59:31.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-19T07:59:31.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:31.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:31.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:31.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:31.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:31.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:31.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:31.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:31.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:31.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:31.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:31.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:31 smithi139 ceph-mon[34874]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-19T07:59:31.844 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.836+0000 7fb192e50640 1 -- 172.21.15.139:0/1802687170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb18c104f90 msgr2=0x7fb18c107380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:31.844 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.836+0000 7fb192e50640 1 --2- 172.21.15.139:0/1802687170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb18c104f90 0x7fb18c107380 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb178009900 tx=0x7fb17802a410 comp rx=0 tx=0).stop 2024-09-19T07:59:31.844 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.836+0000 7fb192e50640 1 -- 172.21.15.139:0/1802687170 shutdown_connections 2024-09-19T07:59:31.844 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.836+0000 7fb192e50640 1 --2- 172.21.15.139:0/1802687170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb18c1078c0 0x7fb18c109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:31.844 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.836+0000 7fb192e50640 1 --2- 172.21.15.139:0/1802687170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb18c104f90 0x7fb18c107380 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:31.844 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.836+0000 7fb192e50640 1 --2- 172.21.15.139:0/1802687170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb18c102660 0x7fb18c104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:31.844 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.836+0000 7fb192e50640 1 -- 172.21.15.139:0/1802687170 >> 172.21.15.139:0/1802687170 conn(0x7fb18c0fc280 msgr2=0x7fb18c0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:59:31.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.836+0000 7fb192e50640 1 -- 172.21.15.139:0/1802687170 shutdown_connections 2024-09-19T07:59:31.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.836+0000 7fb192e50640 1 -- 172.21.15.139:0/1802687170 wait complete. 2024-09-19T07:59:31.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb192e50640 1 Processor -- start 2024-09-19T07:59:31.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb192e50640 1 -- start start 2024-09-19T07:59:31.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb192e50640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb18c102660 0x7fb18c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:59:31.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb192e50640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb18c104f90 0x7fb18c1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:59:31.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb192e50640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb18c1078c0 0x7fb18c19f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:59:31.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb192e50640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c10d8c0 con 0x7fb18c102660 2024-09-19T07:59:31.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb192e50640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c10d740 con 0x7fb18c104f90 2024-09-19T07:59:31.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb192e50640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c10da40 con 0x7fb18c1078c0 2024-09-19T07:59:31.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb18bfff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb18c104f90 0x7fb18c1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:59:31.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb18bfff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb18c104f90 0x7fb18c1984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:45828/0 (socket says 172.21.15.139:45828) 2024-09-19T07:59:31.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb18bfff640 1 -- 172.21.15.139:0/2992501279 learned_addr learned my addr 172.21.15.139:0/2992501279 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-09-19T07:59:31.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb190bec640 1 --2- 172.21.15.139:0/2992501279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb18c102660 0x7fb18c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:59:31.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb1913ed640 1 --2- 172.21.15.139:0/2992501279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb18c1078c0 0x7fb18c19f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:59:31.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb18bfff640 1 -- 172.21.15.139:0/2992501279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb18c1078c0 msgr2=0x7fb18c19f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:31.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb18bfff640 1 --2- 172.21.15.139:0/2992501279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb18c1078c0 0x7fb18c19f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:31.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb18bfff640 1 -- 172.21.15.139:0/2992501279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb18c102660 msgr2=0x7fb18c197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:31.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb18bfff640 1 --2- 172.21.15.139:0/2992501279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb18c102660 0x7fb18c197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:31.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb18bfff640 1 -- 172.21.15.139:0/2992501279 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18c19fbe0 con 0x7fb18c104f90 2024-09-19T07:59:31.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.837+0000 7fb18bfff640 1 --2- 172.21.15.139:0/2992501279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb18c104f90 0x7fb18c1984f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb178031610 tx=0x7fb1780043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:59:31.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.838+0000 7fb189ffb640 1 -- 172.21.15.139:0/2992501279 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17803f030 con 0x7fb18c104f90 2024-09-19T07:59:31.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.838+0000 7fb189ffb640 1 -- 172.21.15.139:0/2992501279 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb178003ce0 con 0x7fb18c104f90 2024-09-19T07:59:31.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.838+0000 7fb192e50640 1 -- 172.21.15.139:0/2992501279 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18c19fe70 con 0x7fb18c104f90 2024-09-19T07:59:31.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.838+0000 7fb189ffb640 1 -- 172.21.15.139:0/2992501279 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb178035070 con 0x7fb18c104f90 2024-09-19T07:59:31.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.838+0000 7fb192e50640 1 -- 172.21.15.139:0/2992501279 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb18c1a0350 con 0x7fb18c104f90 2024-09-19T07:59:31.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.839+0000 7fb192e50640 1 -- 172.21.15.139:0/2992501279 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb18c0fdd70 con 0x7fb18c104f90 2024-09-19T07:59:31.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.840+0000 7fb189ffb640 1 -- 172.21.15.139:0/2992501279 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fb178042030 con 0x7fb18c104f90 2024-09-19T07:59:31.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.841+0000 7fb189ffb640 1 --2- 172.21.15.139:0/2992501279 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb17c0a32f0 0x7fb17c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:59:31.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.841+0000 7fb189ffb640 1 -- 172.21.15.139:0/2992501279 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5981+0+0 (secure 0 0 0) 0x7fb1780ef760 con 0x7fb18c104f90 2024-09-19T07:59:31.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.841+0000 7fb190bec640 1 --2- 172.21.15.139:0/2992501279 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb17c0a32f0 0x7fb17c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:59:31.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.841+0000 7fb190bec640 1 --2- 172.21.15.139:0/2992501279 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb17c0a32f0 0x7fb17c0a57b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb180009990 tx=0x7fb180009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:59:31.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:31.844+0000 7fb189ffb640 1 -- 172.21.15.139:0/2992501279 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb1780bcda0 con 0x7fb18c104f90 2024-09-19T07:59:31.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-19T07:59:31.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: osdmap e59: 10 total, 9 up, 10 in 2024-09-19T07:59:31.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-19T07:59:31.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:31.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:31.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:31.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:31.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:31.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:31.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:31.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:31.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:31.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:31.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:31.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:31 smithi162 ceph-mon[35260]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-19T07:59:32.020 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:32.020+0000 7fb192e50640 1 -- 172.21.15.139:0/2992501279 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb18c062630 con 0x7fb17c0a32f0 2024-09-19T07:59:32.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:32 smithi139 ceph-mon[34874]: Detected new or changed devices on smithi098 2024-09-19T07:59:32.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:32 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi098 to 4272M 2024-09-19T07:59:32.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:32 smithi139 ceph-mon[34874]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:32.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:32 smithi139 ceph-mon[34874]: osdmap e60: 10 total, 9 up, 10 in 2024-09-19T07:59:32.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:32 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:32.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:32 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:32.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:32 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:59:32.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:32 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:59:32.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:32 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:32.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:32 smithi139 ceph-mon[34874]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' 2024-09-19T07:59:32.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:32 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:32.682 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 07:59:32 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T07:59:32.368+0000 7f8bda576640 -1 osd.9 0 waiting for initial osdmap 2024-09-19T07:59:32.682 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 07:59:32 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T07:59:32.382+0000 7f8bd2ba8640 -1 osd.9 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:59:32.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:32 smithi098 ceph-mon[31333]: Detected new or changed devices on smithi098 2024-09-19T07:59:32.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:32 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi098 to 4272M 2024-09-19T07:59:32.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:32 smithi098 ceph-mon[31333]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:32.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:32 smithi098 ceph-mon[31333]: osdmap e60: 10 total, 9 up, 10 in 2024-09-19T07:59:32.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:32 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:32.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:32 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:32.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:32 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:59:32.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:32 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:59:32.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:32 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:32.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:32 smithi098 ceph-mon[31333]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' 2024-09-19T07:59:32.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:32 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:32 smithi162 ceph-mon[35260]: Detected new or changed devices on smithi098 2024-09-19T07:59:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:32 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi098 to 4272M 2024-09-19T07:59:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:32 smithi162 ceph-mon[35260]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:32 smithi162 ceph-mon[35260]: osdmap e60: 10 total, 9 up, 10 in 2024-09-19T07:59:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:32 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:32 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:32 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:59:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:32 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:59:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:32 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:32 smithi162 ceph-mon[35260]: from='osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244]' entity='osd.9' 2024-09-19T07:59:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:32 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:33 smithi162 ceph-mon[35260]: purged_snaps scrub starts 2024-09-19T07:59:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:33 smithi162 ceph-mon[35260]: purged_snaps scrub ok 2024-09-19T07:59:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:33 smithi162 ceph-mon[35260]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:59:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:33 smithi162 ceph-mon[35260]: osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244] boot 2024-09-19T07:59:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:33 smithi162 ceph-mon[35260]: osdmap e61: 10 total, 10 up, 10 in 2024-09-19T07:59:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:33 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:33.895 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:33 smithi139 ceph-mon[34874]: purged_snaps scrub starts 2024-09-19T07:59:33.896 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:33 smithi139 ceph-mon[34874]: purged_snaps scrub ok 2024-09-19T07:59:33.896 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:33 smithi139 ceph-mon[34874]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:59:33.896 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:33 smithi139 ceph-mon[34874]: osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244] boot 2024-09-19T07:59:33.896 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:33 smithi139 ceph-mon[34874]: osdmap e61: 10 total, 10 up, 10 in 2024-09-19T07:59:33.896 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:33 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:33.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:33 smithi098 ceph-mon[31333]: purged_snaps scrub starts 2024-09-19T07:59:33.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:33 smithi098 ceph-mon[31333]: purged_snaps scrub ok 2024-09-19T07:59:33.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:33 smithi098 ceph-mon[31333]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:59:33.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:33 smithi098 ceph-mon[31333]: osd.9 [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244] boot 2024-09-19T07:59:33.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:33 smithi098 ceph-mon[31333]: osdmap e61: 10 total, 10 up, 10 in 2024-09-19T07:59:33.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:33 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T07:59:34.766 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:34 smithi139 ceph-mon[34874]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:34.767 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:34 smithi139 ceph-mon[34874]: from='client.? 172.21.15.139:0/1130084618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03cc7df6-84ae-4d9a-a3d7-74c91e1744e9"}]: dispatch 2024-09-19T07:59:34.767 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:34 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03cc7df6-84ae-4d9a-a3d7-74c91e1744e9"}]: dispatch 2024-09-19T07:59:34.767 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:34 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03cc7df6-84ae-4d9a-a3d7-74c91e1744e9"}]': finished 2024-09-19T07:59:34.767 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:34 smithi139 ceph-mon[34874]: osdmap e62: 11 total, 10 up, 11 in 2024-09-19T07:59:34.767 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:34 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:34.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:34 smithi162 ceph-mon[35260]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:34.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:34 smithi162 ceph-mon[35260]: from='client.? 172.21.15.139:0/1130084618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03cc7df6-84ae-4d9a-a3d7-74c91e1744e9"}]: dispatch 2024-09-19T07:59:34.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:34 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03cc7df6-84ae-4d9a-a3d7-74c91e1744e9"}]: dispatch 2024-09-19T07:59:34.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:34 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03cc7df6-84ae-4d9a-a3d7-74c91e1744e9"}]': finished 2024-09-19T07:59:34.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:34 smithi162 ceph-mon[35260]: osdmap e62: 11 total, 10 up, 11 in 2024-09-19T07:59:34.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:34 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:34.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:34 smithi098 ceph-mon[31333]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-09-19T07:59:34.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:34 smithi098 ceph-mon[31333]: from='client.? 172.21.15.139:0/1130084618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03cc7df6-84ae-4d9a-a3d7-74c91e1744e9"}]: dispatch 2024-09-19T07:59:34.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:34 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03cc7df6-84ae-4d9a-a3d7-74c91e1744e9"}]: dispatch 2024-09-19T07:59:34.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:34 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03cc7df6-84ae-4d9a-a3d7-74c91e1744e9"}]': finished 2024-09-19T07:59:34.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:34 smithi098 ceph-mon[31333]: osdmap e62: 11 total, 10 up, 11 in 2024-09-19T07:59:34.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:34 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:35.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:35 smithi162 ceph-mon[35260]: from='client.? 172.21.15.139:0/3958876669' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:59:35.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:35 smithi139 ceph-mon[34874]: from='client.? 172.21.15.139:0/3958876669' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:59:35.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:35 smithi098 ceph-mon[31333]: from='client.? 172.21.15.139:0/3958876669' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:59:36.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:36 smithi162 ceph-mon[35260]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:36.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:36 smithi139 ceph-mon[34874]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:36.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:36 smithi098 ceph-mon[31333]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:38.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:38 smithi162 ceph-mon[35260]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:38.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:38 smithi139 ceph-mon[34874]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:38.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:38 smithi098 ceph-mon[31333]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:40.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:40 smithi162 ceph-mon[35260]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:40.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:40 smithi139 ceph-mon[34874]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:40.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:40 smithi098 ceph-mon[31333]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:42 smithi162 ceph-mon[35260]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:42.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:42 smithi139 ceph-mon[34874]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:42.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:42 smithi098 ceph-mon[31333]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:43.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:43 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-19T07:59:43.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:43 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:43.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:43 smithi139 ceph-mon[34874]: Deploying daemon osd.10 on smithi139 2024-09-19T07:59:43.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:43 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-19T07:59:43.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:43 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:43.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:43 smithi098 ceph-mon[31333]: Deploying daemon osd.10 on smithi139 2024-09-19T07:59:44.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:43 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-19T07:59:44.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:43 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:44.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:43 smithi162 ceph-mon[35260]: Deploying daemon osd.10 on smithi139 2024-09-19T07:59:44.647 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:44 smithi139 ceph-mon[34874]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:44 smithi098 ceph-mon[31333]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:45.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:44 smithi162 ceph-mon[35260]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:46.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:46 smithi139 ceph-mon[34874]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:46.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:46 smithi162 ceph-mon[35260]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:46 smithi098 ceph-mon[31333]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:48.902 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:48 smithi139 ceph-mon[34874]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:48.902 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:48 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:48.902 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:48 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:48.903 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:48 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:48.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:48 smithi098 ceph-mon[31333]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:48.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:48.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:48.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:48 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:49.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:48 smithi162 ceph-mon[35260]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:49.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:48 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:49.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:48 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:49.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:48 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:50.815 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:50 smithi139 ceph-mon[34874]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:50.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:50 smithi098 ceph-mon[31333]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:51.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:50 smithi162 ceph-mon[35260]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:51.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:51.783+0000 7fb189ffb640 1 -- 172.21.15.139:0/2992501279 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fb18c062630 con 0x7fb17c0a32f0 2024-09-19T07:59:51.784 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 10 on host 'smithi139' 2024-09-19T07:59:51.785 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:51.785+0000 7fb192e50640 1 -- 172.21.15.139:0/2992501279 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb17c0a32f0 msgr2=0x7fb17c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:51.785 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:51.785+0000 7fb192e50640 1 --2- 172.21.15.139:0/2992501279 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb17c0a32f0 0x7fb17c0a57b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb180009990 tx=0x7fb180009110 comp rx=0 tx=0).stop 2024-09-19T07:59:51.785 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:51.785+0000 7fb192e50640 1 -- 172.21.15.139:0/2992501279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb18c104f90 msgr2=0x7fb18c1984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:51.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:51.785+0000 7fb192e50640 1 --2- 172.21.15.139:0/2992501279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb18c104f90 0x7fb18c1984f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb178031610 tx=0x7fb1780043b0 comp rx=0 tx=0).stop 2024-09-19T07:59:51.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:51.785+0000 7fb192e50640 1 -- 172.21.15.139:0/2992501279 shutdown_connections 2024-09-19T07:59:51.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:51.785+0000 7fb192e50640 1 --2- 172.21.15.139:0/2992501279 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb17c0a32f0 0x7fb17c0a57b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:51.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:51.785+0000 7fb192e50640 1 --2- 172.21.15.139:0/2992501279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb18c1078c0 0x7fb18c19f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:51.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:51.785+0000 7fb192e50640 1 --2- 172.21.15.139:0/2992501279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb18c104f90 0x7fb18c1984f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:51.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:51.785+0000 7fb192e50640 1 --2- 172.21.15.139:0/2992501279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb18c102660 0x7fb18c197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:51.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:51.785+0000 7fb192e50640 1 -- 172.21.15.139:0/2992501279 >> 172.21.15.139:0/2992501279 conn(0x7fb18c0fc280 msgr2=0x7fb18c103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:59:51.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:51.785+0000 7fb192e50640 1 -- 172.21.15.139:0/2992501279 shutdown_connections 2024-09-19T07:59:51.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-19T07:59:51.785+0000 7fb192e50640 1 -- 172.21.15.139:0/2992501279 wait complete. 2024-09-19T07:59:52.349 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.349 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.349 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:52.350 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:52.350 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.350 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:52 smithi139 ceph-mon[34874]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:52.350 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:52.350 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.350 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.377 DEBUG:teuthology.orchestra.run.smithi139:osd.10> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.10.service 2024-09-19T07:59:52.379 INFO:tasks.cephadm:Deploying osd.11 on smithi162 with /dev/vg_nvme/lv_1... 2024-09-19T07:59:52.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-19T07:59:52.408 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.408 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.408 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:52.408 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:52.408 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.409 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:52 smithi162 ceph-mon[35260]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:52.409 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:52.409 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.409 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:52.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:52.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:52 smithi098 ceph-mon[31333]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:52.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:52.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:52.636 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T07:59:54.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:54 smithi162 ceph-mon[35260]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:54.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:54 smithi139 ceph-mon[34874]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:54.928 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 07:59:54 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T07:59:54.651+0000 7fabd7233640 -1 osd.10 0 log_to_monitors true 2024-09-19T07:59:54.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:54 smithi098 ceph-mon[31333]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:55.172 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-19T07:59:55.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_1 2024-09-19T07:59:55.493 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T07:59:55.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:55 smithi162 ceph-mon[35260]: from='osd.10 [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-19T07:59:55.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:55 smithi162 ceph-mon[35260]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-19T07:59:55.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:55.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:55.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:55.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:55.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:55.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:55.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:55.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:55.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:55.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:55.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:55 smithi139 ceph-mon[34874]: from='osd.10 [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-19T07:59:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:55 smithi139 ceph-mon[34874]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-19T07:59:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:55 smithi098 ceph-mon[31333]: from='osd.10 [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-19T07:59:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:55 smithi098 ceph-mon[31333]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-19T07:59:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:59:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:55.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:55.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T07:59:55.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.933+0000 7f467cadc640 1 -- 172.21.15.162:0/1836184243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4678100ad0 msgr2=0x7f4678100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:55.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.933+0000 7f467cadc640 1 --2- 172.21.15.162:0/1836184243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4678100ad0 0x7f4678100ed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4668009900 tx=0x7f466802a410 comp rx=0 tx=0).stop 2024-09-19T07:59:55.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.933+0000 7f467cadc640 1 -- 172.21.15.162:0/1836184243 shutdown_connections 2024-09-19T07:59:55.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.933+0000 7f467cadc640 1 --2- 172.21.15.162:0/1836184243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678102690 0x7f4678108f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:55.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.933+0000 7f467cadc640 1 --2- 172.21.15.162:0/1836184243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4678101cd0 0x7f4678102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:55.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.933+0000 7f467cadc640 1 --2- 172.21.15.162:0/1836184243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4678100ad0 0x7f4678100ed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:55.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.933+0000 7f467cadc640 1 -- 172.21.15.162:0/1836184243 >> 172.21.15.162:0/1836184243 conn(0x7f46780fc280 msgr2=0x7f46780fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:59:55.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.933+0000 7f467cadc640 1 -- 172.21.15.162:0/1836184243 shutdown_connections 2024-09-19T07:59:55.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.933+0000 7f467cadc640 1 -- 172.21.15.162:0/1836184243 wait complete. 2024-09-19T07:59:55.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.934+0000 7f467cadc640 1 Processor -- start 2024-09-19T07:59:55.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.934+0000 7f467cadc640 1 -- start start 2024-09-19T07:59:55.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.934+0000 7f467cadc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4678100ad0 0x7f467806aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:59:55.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.934+0000 7f467cadc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678101cd0 0x7f467819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:59:55.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.934+0000 7f467cadc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4678102690 0x7f467819f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:59:55.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f467cadc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f467810c730 con 0x7f4678101cd0 2024-09-19T07:59:55.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f467cadc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f467810c5b0 con 0x7f4678102690 2024-09-19T07:59:55.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f467cadc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f467810c8b0 con 0x7f4678100ad0 2024-09-19T07:59:55.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f467659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4678100ad0 0x7f467806aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:59:55.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f467659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4678100ad0 0x7f467806aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:45070/0 (socket says 172.21.15.162:45070) 2024-09-19T07:59:55.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f467659c640 1 -- 172.21.15.162:0/1969097553 learned_addr learned my addr 172.21.15.162:0/1969097553 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-19T07:59:55.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f4675d9b640 1 --2- 172.21.15.162:0/1969097553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678101cd0 0x7f467819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:59:55.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f4676d9d640 1 --2- 172.21.15.162:0/1969097553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4678102690 0x7f467819f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:59:55.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f467659c640 1 -- 172.21.15.162:0/1969097553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4678102690 msgr2=0x7f467819f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:55.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f467659c640 1 --2- 172.21.15.162:0/1969097553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4678102690 0x7f467819f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:55.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f467659c640 1 -- 172.21.15.162:0/1969097553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678101cd0 msgr2=0x7f467819cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:59:55.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f467659c640 1 --2- 172.21.15.162:0/1969097553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678101cd0 0x7f467819cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:59:55.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f467659c640 1 -- 172.21.15.162:0/1969097553 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467819fb60 con 0x7f4678100ad0 2024-09-19T07:59:55.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f467659c640 1 --2- 172.21.15.162:0/1969097553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4678100ad0 0x7f467806aad0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4668031610 tx=0x7f46680043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:59:55.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f46677fe640 1 -- 172.21.15.162:0/1969097553 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466803f030 con 0x7f4678100ad0 2024-09-19T07:59:55.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f467cadc640 1 -- 172.21.15.162:0/1969097553 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f467819fd90 con 0x7f4678100ad0 2024-09-19T07:59:55.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f46677fe640 1 -- 172.21.15.162:0/1969097553 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4668003ce0 con 0x7f4678100ad0 2024-09-19T07:59:55.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f46677fe640 1 -- 172.21.15.162:0/1969097553 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668035070 con 0x7f4678100ad0 2024-09-19T07:59:55.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.935+0000 7f467cadc640 1 -- 172.21.15.162:0/1969097553 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46781a0260 con 0x7f4678100ad0 2024-09-19T07:59:55.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.936+0000 7f467cadc640 1 -- 172.21.15.162:0/1969097553 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4678107ed0 con 0x7f4678100ad0 2024-09-19T07:59:55.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.938+0000 7f46677fe640 1 -- 172.21.15.162:0/1969097553 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f4668042030 con 0x7f4678100ad0 2024-09-19T07:59:55.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.939+0000 7f46677fe640 1 --2- 172.21.15.162:0/1969097553 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f46600a3240 0x7f46600a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:59:55.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.939+0000 7f46677fe640 1 -- 172.21.15.162:0/1969097553 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6398+0+0 (secure 0 0 0) 0x7f46680ef940 con 0x7f4678100ad0 2024-09-19T07:59:55.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.939+0000 7f4675d9b640 1 --2- 172.21.15.162:0/1969097553 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f46600a3240 0x7f46600a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:59:55.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.940+0000 7f4675d9b640 1 --2- 172.21.15.162:0/1969097553 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f46600a3240 0x7f46600a5700 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f467806b050 tx=0x7f466c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:59:55.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:55.942+0000 7f46677fe640 1 -- 172.21.15.162:0/1969097553 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f46680bc790 con 0x7f4678100ad0 2024-09-19T07:59:56.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T07:59:56.123+0000 7f467cadc640 1 -- 172.21.15.162:0/1969097553 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f4678062630 con 0x7f46600a3240 2024-09-19T07:59:56.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:56 smithi162 ceph-mon[35260]: Detected new or changed devices on smithi139 2024-09-19T07:59:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:56 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi139 to 4272M 2024-09-19T07:59:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:56 smithi162 ceph-mon[35260]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:56 smithi162 ceph-mon[35260]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-19T07:59:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:56 smithi162 ceph-mon[35260]: osdmap e63: 11 total, 10 up, 11 in 2024-09-19T07:59:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:56 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:56 smithi162 ceph-mon[35260]: from='osd.10 [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:59:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:56 smithi162 ceph-mon[35260]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:59:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:56 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:59:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:56 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:59:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:56 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:56.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:56 smithi139 ceph-mon[34874]: Detected new or changed devices on smithi139 2024-09-19T07:59:56.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:56 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi139 to 4272M 2024-09-19T07:59:56.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:56 smithi139 ceph-mon[34874]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:56.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:56 smithi139 ceph-mon[34874]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-19T07:59:56.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:56 smithi139 ceph-mon[34874]: osdmap e63: 11 total, 10 up, 11 in 2024-09-19T07:59:56.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:56 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:56.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:56 smithi139 ceph-mon[34874]: from='osd.10 [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:59:56.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:56 smithi139 ceph-mon[34874]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:59:56.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:56 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:59:56.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:56 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:59:56.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:56 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:56 smithi098 ceph-mon[31333]: Detected new or changed devices on smithi139 2024-09-19T07:59:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:56 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi139 to 4272M 2024-09-19T07:59:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:56 smithi098 ceph-mon[31333]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:56 smithi098 ceph-mon[31333]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-19T07:59:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:56 smithi098 ceph-mon[31333]: osdmap e63: 11 total, 10 up, 11 in 2024-09-19T07:59:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:56 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:56 smithi098 ceph-mon[31333]: from='osd.10 [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:59:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:56 smithi098 ceph-mon[31333]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-19T07:59:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:56 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T07:59:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:56 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T07:59:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:56 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:57.677 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 07:59:57 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T07:59:57.368+0000 7fabd39ca640 -1 osd.10 0 waiting for initial osdmap 2024-09-19T07:59:57.677 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 07:59:57 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T07:59:57.388+0000 7fabcafc5640 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:59:57.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:57 smithi139 ceph-mon[34874]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:59:57.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:57 smithi139 ceph-mon[34874]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-19T07:59:57.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:57 smithi139 ceph-mon[34874]: osdmap e64: 11 total, 10 up, 11 in 2024-09-19T07:59:57.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:57 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:57.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:57 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:57.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:57 smithi139 ceph-mon[34874]: from='osd.10 ' entity='osd.10' 2024-09-19T07:59:57.853 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:57 smithi162 ceph-mon[35260]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:59:57.853 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:57 smithi162 ceph-mon[35260]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-19T07:59:57.854 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:57 smithi162 ceph-mon[35260]: osdmap e64: 11 total, 10 up, 11 in 2024-09-19T07:59:57.854 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:57 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:57.854 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:57 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:57.854 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:57 smithi162 ceph-mon[35260]: from='osd.10 ' entity='osd.10' 2024-09-19T07:59:57.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:57 smithi098 ceph-mon[31333]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T07:59:57.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:57 smithi098 ceph-mon[31333]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-19T07:59:57.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:57 smithi098 ceph-mon[31333]: osdmap e64: 11 total, 10 up, 11 in 2024-09-19T07:59:57.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:57 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:57.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:57 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:57.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:57 smithi098 ceph-mon[31333]: from='osd.10 ' entity='osd.10' 2024-09-19T07:59:58.865 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:58 smithi162 ceph-mon[35260]: purged_snaps scrub starts 2024-09-19T07:59:58.865 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:58 smithi162 ceph-mon[35260]: purged_snaps scrub ok 2024-09-19T07:59:58.865 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:58 smithi162 ceph-mon[35260]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:58.866 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:58.866 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:58 smithi162 ceph-mon[35260]: osd.10 [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639] boot 2024-09-19T07:59:58.866 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:58 smithi162 ceph-mon[35260]: osdmap e65: 11 total, 11 up, 11 in 2024-09-19T07:59:58.866 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:58.866 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:58 smithi162 ceph-mon[35260]: from='client.? 172.21.15.162:0/3240051781' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29e95498-4952-46d4-8fa6-6d8e2b862111"}]: dispatch 2024-09-19T07:59:58.866 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:58 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29e95498-4952-46d4-8fa6-6d8e2b862111"}]: dispatch 2024-09-19T07:59:58.866 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:58 smithi162 ceph-mon[35260]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "29e95498-4952-46d4-8fa6-6d8e2b862111"}]': finished 2024-09-19T07:59:58.866 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:58 smithi162 ceph-mon[35260]: osdmap e66: 12 total, 11 up, 12 in 2024-09-19T07:59:58.866 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T07:59:58.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:58 smithi139 ceph-mon[34874]: purged_snaps scrub starts 2024-09-19T07:59:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:58 smithi139 ceph-mon[34874]: purged_snaps scrub ok 2024-09-19T07:59:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:58 smithi139 ceph-mon[34874]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:58 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:58 smithi139 ceph-mon[34874]: osd.10 [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639] boot 2024-09-19T07:59:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:58 smithi139 ceph-mon[34874]: osdmap e65: 11 total, 11 up, 11 in 2024-09-19T07:59:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:58 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:58 smithi139 ceph-mon[34874]: from='client.? 172.21.15.162:0/3240051781' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29e95498-4952-46d4-8fa6-6d8e2b862111"}]: dispatch 2024-09-19T07:59:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:58 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29e95498-4952-46d4-8fa6-6d8e2b862111"}]: dispatch 2024-09-19T07:59:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:58 smithi139 ceph-mon[34874]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "29e95498-4952-46d4-8fa6-6d8e2b862111"}]': finished 2024-09-19T07:59:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:58 smithi139 ceph-mon[34874]: osdmap e66: 12 total, 11 up, 12 in 2024-09-19T07:59:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:58 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T07:59:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:58 smithi098 ceph-mon[31333]: purged_snaps scrub starts 2024-09-19T07:59:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:58 smithi098 ceph-mon[31333]: purged_snaps scrub ok 2024-09-19T07:59:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:58 smithi098 ceph-mon[31333]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-19T07:59:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:58 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:58 smithi098 ceph-mon[31333]: osd.10 [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639] boot 2024-09-19T07:59:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:58 smithi098 ceph-mon[31333]: osdmap e65: 11 total, 11 up, 11 in 2024-09-19T07:59:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:58 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T07:59:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:58 smithi098 ceph-mon[31333]: from='client.? 172.21.15.162:0/3240051781' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29e95498-4952-46d4-8fa6-6d8e2b862111"}]: dispatch 2024-09-19T07:59:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:58 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29e95498-4952-46d4-8fa6-6d8e2b862111"}]: dispatch 2024-09-19T07:59:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:58 smithi098 ceph-mon[31333]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "29e95498-4952-46d4-8fa6-6d8e2b862111"}]': finished 2024-09-19T07:59:58.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:58 smithi098 ceph-mon[31333]: osdmap e66: 12 total, 11 up, 12 in 2024-09-19T07:59:58.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:58 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T07:59:59.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:59 smithi139 ceph-mon[34874]: from='client.? 172.21.15.162:0/3757400657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:59:59.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:59 smithi139 ceph-mon[34874]: osdmap e67: 12 total, 11 up, 12 in 2024-09-19T07:59:59.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 07:59:59 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T07:59:59.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:59 smithi098 ceph-mon[31333]: from='client.? 172.21.15.162:0/3757400657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T07:59:59.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:59 smithi098 ceph-mon[31333]: osdmap e67: 12 total, 11 up, 12 in 2024-09-19T07:59:59.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 07:59:59 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:00.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:59 smithi162 ceph-mon[35260]: from='client.? 172.21.15.162:0/3757400657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T08:00:00.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:59 smithi162 ceph-mon[35260]: osdmap e67: 12 total, 11 up, 12 in 2024-09-19T08:00:00.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 07:59:59 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:00.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:00 smithi139 ceph-mon[34874]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail 2024-09-19T08:00:00.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:00 smithi139 ceph-mon[34874]: overall HEALTH_OK 2024-09-19T08:00:00.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:00 smithi098 ceph-mon[31333]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail 2024-09-19T08:00:00.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:00 smithi098 ceph-mon[31333]: overall HEALTH_OK 2024-09-19T08:00:01.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:00 smithi162 ceph-mon[35260]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail 2024-09-19T08:00:01.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:00 smithi162 ceph-mon[35260]: overall HEALTH_OK 2024-09-19T08:00:01.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:01 smithi139 ceph-mon[34874]: osdmap e68: 12 total, 11 up, 12 in 2024-09-19T08:00:01.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:01 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:01.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:01 smithi098 ceph-mon[31333]: osdmap e68: 12 total, 11 up, 12 in 2024-09-19T08:00:01.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:01 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:02.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:01 smithi162 ceph-mon[35260]: osdmap e68: 12 total, 11 up, 12 in 2024-09-19T08:00:02.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:01 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:02.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:02 smithi139 ceph-mon[34874]: pgmap v226: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 728 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-09-19T08:00:02.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:02 smithi139 ceph-mon[34874]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-09-19T08:00:02.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:02 smithi139 ceph-mon[34874]: osdmap e69: 12 total, 11 up, 12 in 2024-09-19T08:00:02.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:02 smithi098 ceph-mon[31333]: pgmap v226: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 728 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-09-19T08:00:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:02 smithi098 ceph-mon[31333]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-09-19T08:00:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:02 smithi098 ceph-mon[31333]: osdmap e69: 12 total, 11 up, 12 in 2024-09-19T08:00:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:02 smithi162 ceph-mon[35260]: pgmap v226: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 728 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-09-19T08:00:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:02 smithi162 ceph-mon[35260]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-09-19T08:00:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:02 smithi162 ceph-mon[35260]: osdmap e69: 12 total, 11 up, 12 in 2024-09-19T08:00:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:04.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:04 smithi139 ceph-mon[34874]: pgmap v228: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-09-19T08:00:04.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:04 smithi098 ceph-mon[31333]: pgmap v228: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-09-19T08:00:04.964 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:04 smithi162 ceph-mon[35260]: pgmap v228: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-09-19T08:00:05.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:05 smithi162 ceph-mon[35260]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-19T08:00:05.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:05 smithi162 ceph-mon[35260]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-09-19T08:00:05.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:05 smithi162 ceph-mon[35260]: Cluster is now healthy 2024-09-19T08:00:06.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:05 smithi139 ceph-mon[34874]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-19T08:00:06.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:05 smithi139 ceph-mon[34874]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-09-19T08:00:06.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:05 smithi139 ceph-mon[34874]: Cluster is now healthy 2024-09-19T08:00:06.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:05 smithi098 ceph-mon[31333]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-19T08:00:06.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:05 smithi098 ceph-mon[31333]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-09-19T08:00:06.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:05 smithi098 ceph-mon[31333]: Cluster is now healthy 2024-09-19T08:00:07.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:07 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-19T08:00:07.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:07 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:07.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:07 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-19T08:00:07.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:07 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:07.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:07 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-19T08:00:07.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:07 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:08.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:08 smithi139 ceph-mon[34874]: Deploying daemon osd.11 on smithi162 2024-09-19T08:00:08.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:08 smithi139 ceph-mon[34874]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T08:00:08.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:08 smithi098 ceph-mon[31333]: Deploying daemon osd.11 on smithi162 2024-09-19T08:00:08.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:08 smithi098 ceph-mon[31333]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T08:00:08.442 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:08 smithi162 ceph-mon[35260]: Deploying daemon osd.11 on smithi162 2024-09-19T08:00:08.442 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:08 smithi162 ceph-mon[35260]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T08:00:10.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:10 smithi162 ceph-mon[35260]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-19T08:00:10.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:10 smithi139 ceph-mon[34874]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-19T08:00:10.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:10 smithi098 ceph-mon[31333]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-19T08:00:12.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:12 smithi162 ceph-mon[35260]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-19T08:00:12.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:12 smithi139 ceph-mon[34874]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-19T08:00:12.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:12 smithi098 ceph-mon[31333]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-19T08:00:13.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:13 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:00:13.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:13 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:13.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:13 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:13.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:13 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:00:13.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:13 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:13.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:13 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:13.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:13 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:00:13.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:13 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:13.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:13 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:14.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:14 smithi162 ceph-mon[35260]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-19T08:00:14.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:14 smithi139 ceph-mon[34874]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-19T08:00:14.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:14 smithi098 ceph-mon[31333]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-19T08:00:15.780 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 11 on host 'smithi162' 2024-09-19T08:00:15.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:15.779+0000 7f46677fe640 1 -- 172.21.15.162:0/1969097553 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f4678062630 con 0x7f46600a3240 2024-09-19T08:00:15.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:15.782+0000 7f467cadc640 1 -- 172.21.15.162:0/1969097553 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f46600a3240 msgr2=0x7f46600a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:15.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:15.782+0000 7f467cadc640 1 --2- 172.21.15.162:0/1969097553 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f46600a3240 0x7f46600a5700 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f467806b050 tx=0x7f466c00b410 comp rx=0 tx=0).stop 2024-09-19T08:00:15.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:15.782+0000 7f467cadc640 1 -- 172.21.15.162:0/1969097553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4678100ad0 msgr2=0x7f467806aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:15.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:15.782+0000 7f467cadc640 1 --2- 172.21.15.162:0/1969097553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4678100ad0 0x7f467806aad0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4668031610 tx=0x7f46680043b0 comp rx=0 tx=0).stop 2024-09-19T08:00:15.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:15.782+0000 7f467cadc640 1 -- 172.21.15.162:0/1969097553 shutdown_connections 2024-09-19T08:00:15.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:15.782+0000 7f467cadc640 1 --2- 172.21.15.162:0/1969097553 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f46600a3240 0x7f46600a5700 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:15.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:15.782+0000 7f467cadc640 1 --2- 172.21.15.162:0/1969097553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4678102690 0x7f467819f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:15.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:15.782+0000 7f467cadc640 1 --2- 172.21.15.162:0/1969097553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678101cd0 0x7f467819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:15.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:15.782+0000 7f467cadc640 1 --2- 172.21.15.162:0/1969097553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4678100ad0 0x7f467806aad0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:15.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:15.782+0000 7f467cadc640 1 -- 172.21.15.162:0/1969097553 >> 172.21.15.162:0/1969097553 conn(0x7f46780fc280 msgr2=0x7f46781050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:15.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:15.782+0000 7f467cadc640 1 -- 172.21.15.162:0/1969097553 shutdown_connections 2024-09-19T08:00:15.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:15.782+0000 7f467cadc640 1 -- 172.21.15.162:0/1969097553 wait complete. 2024-09-19T08:00:16.382 DEBUG:teuthology.orchestra.run.smithi162:osd.11> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.11.service 2024-09-19T08:00:16.385 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-19T08:00:16.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-19T08:00:16.645 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:00:16.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:16 smithi098 ceph-mon[31333]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-19T08:00:16.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:16.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:16.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:16.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:16 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:16 smithi139 ceph-mon[34874]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-19T08:00:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:16 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:16.946 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:16 smithi162 ceph-mon[35260]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-19T08:00:16.946 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:16.946 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:16.947 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:16.947 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:16 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.237+0000 7fce29f59640 1 -- 172.21.15.98:0/1732030197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce24104fd0 msgr2=0x7fce241073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.237+0000 7fce29f59640 1 --2- 172.21.15.98:0/1732030197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce24104fd0 0x7fce241073c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fce0c009900 tx=0x7fce0c02a410 comp rx=0 tx=0).stop 2024-09-19T08:00:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.237+0000 7fce29f59640 1 -- 172.21.15.98:0/1732030197 shutdown_connections 2024-09-19T08:00:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.237+0000 7fce29f59640 1 --2- 172.21.15.98:0/1732030197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce24107900 0x7fce24109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.237+0000 7fce29f59640 1 --2- 172.21.15.98:0/1732030197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce24104fd0 0x7fce241073c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.237+0000 7fce29f59640 1 --2- 172.21.15.98:0/1732030197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce241026a0 0x7fce24104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.237+0000 7fce29f59640 1 -- 172.21.15.98:0/1732030197 >> 172.21.15.98:0/1732030197 conn(0x7fce240fc2a0 msgr2=0x7fce240fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.237+0000 7fce29f59640 1 -- 172.21.15.98:0/1732030197 shutdown_connections 2024-09-19T08:00:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.237+0000 7fce29f59640 1 -- 172.21.15.98:0/1732030197 wait complete. 2024-09-19T08:00:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce29f59640 1 Processor -- start 2024-09-19T08:00:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce29f59640 1 -- start start 2024-09-19T08:00:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce29f59640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce241026a0 0x7fce24197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce29f59640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce24104fd0 0x7fce241984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce29f59640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce24107900 0x7fce2419f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce29f59640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fce2410d8a0 con 0x7fce24104fd0 2024-09-19T08:00:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce29f59640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fce2410d720 con 0x7fce24107900 2024-09-19T08:00:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce29f59640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce2410da20 con 0x7fce241026a0 2024-09-19T08:00:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce22ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce24104fd0 0x7fce241984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce22ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce24104fd0 0x7fce241984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50644/0 (socket says 172.21.15.98:50644) 2024-09-19T08:00:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce22ffd640 1 -- 172.21.15.98:0/3316791133 learned_addr learned my addr 172.21.15.98:0/3316791133 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:00:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce237fe640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce241026a0 0x7fce24197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce23fff640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce24107900 0x7fce2419f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce22ffd640 1 -- 172.21.15.98:0/3316791133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce241026a0 msgr2=0x7fce24197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce22ffd640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce241026a0 0x7fce24197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce22ffd640 1 -- 172.21.15.98:0/3316791133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce24107900 msgr2=0x7fce2419f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce22ffd640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce24107900 0x7fce2419f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce22ffd640 1 -- 172.21.15.98:0/3316791133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce2419fbe0 con 0x7fce24104fd0 2024-09-19T08:00:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.238+0000 7fce23fff640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce24107900 0x7fce2419f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:00:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.239+0000 7fce22ffd640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce24104fd0 0x7fce241984f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fce0c031610 tx=0x7fce0c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.239+0000 7fce20ff9640 1 -- 172.21.15.98:0/3316791133 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c03f030 con 0x7fce24104fd0 2024-09-19T08:00:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.239+0000 7fce20ff9640 1 -- 172.21.15.98:0/3316791133 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce0c003ce0 con 0x7fce24104fd0 2024-09-19T08:00:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.239+0000 7fce29f59640 1 -- 172.21.15.98:0/3316791133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce2419fe70 con 0x7fce24104fd0 2024-09-19T08:00:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.239+0000 7fce20ff9640 1 -- 172.21.15.98:0/3316791133 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c035070 con 0x7fce24104fd0 2024-09-19T08:00:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.239+0000 7fce29f59640 1 -- 172.21.15.98:0/3316791133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce241a02c0 con 0x7fce24104fd0 2024-09-19T08:00:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.240+0000 7fce29f59640 1 -- 172.21.15.98:0/3316791133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce240fdd90 con 0x7fce24104fd0 2024-09-19T08:00:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.241+0000 7fce20ff9640 1 -- 172.21.15.98:0/3316791133 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fce0c042030 con 0x7fce24104fd0 2024-09-19T08:00:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.242+0000 7fce20ff9640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fce100a3270 0x7fce100a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.242+0000 7fce237fe640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fce100a3270 0x7fce100a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.242+0000 7fce20ff9640 1 -- 172.21.15.98:0/3316791133 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6809+0+0 (secure 0 0 0) 0x7fce0c0ef880 con 0x7fce24104fd0 2024-09-19T08:00:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.242+0000 7fce237fe640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fce100a3270 0x7fce100a5730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fce14005fd0 tx=0x7fce1400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.247+0000 7fce20ff9640 1 -- 172.21.15.98:0/3316791133 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fce0c0bcb80 con 0x7fce24104fd0 2024-09-19T08:00:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.420+0000 7fce29f59640 1 -- 172.21.15.98:0/3316791133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fce2402da10 con 0x7fce24104fd0 2024-09-19T08:00:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.421+0000 7fce20ff9640 1 -- 172.21.15.98:0/3316791133 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v69) ==== 74+0+133 (secure 0 0 0) 0x7fce0c0bc480 con 0x7fce24104fd0 2024-09-19T08:00:17.421 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:00:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.423+0000 7fce29f59640 1 -- 172.21.15.98:0/3316791133 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fce100a3270 msgr2=0x7fce100a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.423+0000 7fce29f59640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fce100a3270 0x7fce100a5730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fce14005fd0 tx=0x7fce1400b390 comp rx=0 tx=0).stop 2024-09-19T08:00:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.423+0000 7fce29f59640 1 -- 172.21.15.98:0/3316791133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce24104fd0 msgr2=0x7fce241984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.423+0000 7fce29f59640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce24104fd0 0x7fce241984f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fce0c031610 tx=0x7fce0c0043b0 comp rx=0 tx=0).stop 2024-09-19T08:00:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.423+0000 7fce29f59640 1 -- 172.21.15.98:0/3316791133 shutdown_connections 2024-09-19T08:00:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.423+0000 7fce29f59640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fce100a3270 0x7fce100a5730 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.423+0000 7fce29f59640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce24107900 0x7fce2419f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.423+0000 7fce29f59640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce24104fd0 0x7fce241984f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.423+0000 7fce29f59640 1 --2- 172.21.15.98:0/3316791133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce241026a0 0x7fce24197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.423+0000 7fce29f59640 1 -- 172.21.15.98:0/3316791133 >> 172.21.15.98:0/3316791133 conn(0x7fce240fc2a0 msgr2=0x7fce24103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:17.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.423+0000 7fce29f59640 1 -- 172.21.15.98:0/3316791133 shutdown_connections 2024-09-19T08:00:17.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:17.423+0000 7fce29f59640 1 -- 172.21.15.98:0/3316791133 wait complete. 2024-09-19T08:00:17.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:17 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3316791133' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T08:00:17.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:17 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3316791133' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T08:00:17.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:17 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3316791133' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T08:00:18.174 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":69,"num_osds":12,"num_up_osds":11,"osd_up_since":1726732797,"num_in_osds":12,"osd_in_since":1726732798,"num_remapped_pgs":0} 2024-09-19T08:00:18.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:18 smithi139 ceph-mon[34874]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-09-19T08:00:18.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:18 smithi098 ceph-mon[31333]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-09-19T08:00:18.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:18 smithi162 ceph-mon[35260]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-09-19T08:00:18.994 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:00:18 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:00:18.649+0000 7f447b000640 -1 osd.11 0 log_to_monitors true 2024-09-19T08:00:19.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-19T08:00:19.431 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:00:19.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:19 smithi098 ceph-mon[31333]: from='osd.11 [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-19T08:00:19.867 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:19 smithi098 ceph-mon[31333]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-19T08:00:19.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:19 smithi139 ceph-mon[34874]: from='osd.11 [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-19T08:00:19.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:19 smithi139 ceph-mon[34874]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-19T08:00:19.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.950+0000 7fda257e6640 1 -- 172.21.15.98:0/3659857870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20105010 msgr2=0x7fda20107400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.950+0000 7fda257e6640 1 --2- 172.21.15.98:0/3659857870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20105010 0x7fda20107400 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fda08009900 tx=0x7fda0802a410 comp rx=0 tx=0).stop 2024-09-19T08:00:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.950+0000 7fda257e6640 1 -- 172.21.15.98:0/3659857870 shutdown_connections 2024-09-19T08:00:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.950+0000 7fda257e6640 1 --2- 172.21.15.98:0/3659857870 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda20107940 0x7fda20109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.950+0000 7fda257e6640 1 --2- 172.21.15.98:0/3659857870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20105010 0x7fda20107400 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.950+0000 7fda257e6640 1 --2- 172.21.15.98:0/3659857870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda201026e0 0x7fda20104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.950+0000 7fda257e6640 1 -- 172.21.15.98:0/3659857870 >> 172.21.15.98:0/3659857870 conn(0x7fda200fc280 msgr2=0x7fda200fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.950+0000 7fda257e6640 1 -- 172.21.15.98:0/3659857870 shutdown_connections 2024-09-19T08:00:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.950+0000 7fda257e6640 1 -- 172.21.15.98:0/3659857870 wait complete. 2024-09-19T08:00:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.950+0000 7fda257e6640 1 Processor -- start 2024-09-19T08:00:19.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda257e6640 1 -- start start 2024-09-19T08:00:19.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda257e6640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda201026e0 0x7fda20102040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:19.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda257e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20105010 0x7fda20100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:19.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda257e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda20107940 0x7fda20100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:19.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda257e6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fda2010d830 con 0x7fda20105010 2024-09-19T08:00:19.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda257e6640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fda2010d6b0 con 0x7fda201026e0 2024-09-19T08:00:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda257e6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fda2010d9b0 con 0x7fda20107940 2024-09-19T08:00:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda167fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20105010 0x7fda20100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda1effd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda201026e0 0x7fda20102040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda167fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20105010 0x7fda20100690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50658/0 (socket says 172.21.15.98:50658) 2024-09-19T08:00:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda1effd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda201026e0 0x7fda20102040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:55280/0 (socket says 172.21.15.98:55280) 2024-09-19T08:00:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda167fc640 1 -- 172.21.15.98:0/4208408412 learned_addr learned my addr 172.21.15.98:0/4208408412 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:00:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda1f7fe640 1 --2- 172.21.15.98:0/4208408412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda20107940 0x7fda20100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda167fc640 1 -- 172.21.15.98:0/4208408412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda20107940 msgr2=0x7fda20100bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda167fc640 1 --2- 172.21.15.98:0/4208408412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda20107940 0x7fda20100bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda167fc640 1 -- 172.21.15.98:0/4208408412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda201026e0 msgr2=0x7fda20102040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda167fc640 1 --2- 172.21.15.98:0/4208408412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda201026e0 0x7fda20102040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.951+0000 7fda167fc640 1 -- 172.21.15.98:0/4208408412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda20101490 con 0x7fda20105010 2024-09-19T08:00:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.952+0000 7fda167fc640 1 --2- 172.21.15.98:0/4208408412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20105010 0x7fda20100690 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fda08031610 tx=0x7fda080043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.952+0000 7fda1cff9640 1 -- 172.21.15.98:0/4208408412 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0803f030 con 0x7fda20105010 2024-09-19T08:00:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.952+0000 7fda1cff9640 1 -- 172.21.15.98:0/4208408412 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda08003ce0 con 0x7fda20105010 2024-09-19T08:00:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.952+0000 7fda257e6640 1 -- 172.21.15.98:0/4208408412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda201a2070 con 0x7fda20105010 2024-09-19T08:00:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.952+0000 7fda1cff9640 1 -- 172.21.15.98:0/4208408412 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda08035070 con 0x7fda20105010 2024-09-19T08:00:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.952+0000 7fda257e6640 1 -- 172.21.15.98:0/4208408412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda201a2520 con 0x7fda20105010 2024-09-19T08:00:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.953+0000 7fda257e6640 1 -- 172.21.15.98:0/4208408412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda200fdd70 con 0x7fda20105010 2024-09-19T08:00:19.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.954+0000 7fda1cff9640 1 -- 172.21.15.98:0/4208408412 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fda08042030 con 0x7fda20105010 2024-09-19T08:00:19.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.955+0000 7fda1cff9640 1 --2- 172.21.15.98:0/4208408412 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fda0c0a3240 0x7fda0c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:19.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.955+0000 7fda1effd640 1 --2- 172.21.15.98:0/4208408412 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fda0c0a3240 0x7fda0c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:19.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.955+0000 7fda1cff9640 1 -- 172.21.15.98:0/4208408412 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6831+0+0 (secure 0 0 0) 0x7fda080efea0 con 0x7fda20105010 2024-09-19T08:00:19.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.955+0000 7fda1effd640 1 --2- 172.21.15.98:0/4208408412 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fda0c0a3240 0x7fda0c0a5700 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fda10005d00 tx=0x7fda1000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:19.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:19.958+0000 7fda1cff9640 1 -- 172.21.15.98:0/4208408412 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fda08049030 con 0x7fda20105010 2024-09-19T08:00:20.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:19 smithi162 ceph-mon[35260]: from='osd.11 [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-19T08:00:20.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:19 smithi162 ceph-mon[35260]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-19T08:00:20.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:20.126+0000 7fda257e6640 1 -- 172.21.15.98:0/4208408412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fda2002da10 con 0x7fda20105010 2024-09-19T08:00:20.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:20.127+0000 7fda1cff9640 1 -- 172.21.15.98:0/4208408412 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v70) ==== 74+0+133 (secure 0 0 0) 0x7fda080bd0f0 con 0x7fda20105010 2024-09-19T08:00:20.127 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:00:20.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:20.130+0000 7fda257e6640 1 -- 172.21.15.98:0/4208408412 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fda0c0a3240 msgr2=0x7fda0c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:20.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:20.130+0000 7fda257e6640 1 --2- 172.21.15.98:0/4208408412 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fda0c0a3240 0x7fda0c0a5700 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fda10005d00 tx=0x7fda1000b410 comp rx=0 tx=0).stop 2024-09-19T08:00:20.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:20.130+0000 7fda257e6640 1 -- 172.21.15.98:0/4208408412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20105010 msgr2=0x7fda20100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:20.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:20.130+0000 7fda257e6640 1 --2- 172.21.15.98:0/4208408412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20105010 0x7fda20100690 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fda08031610 tx=0x7fda080043b0 comp rx=0 tx=0).stop 2024-09-19T08:00:20.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:20.130+0000 7fda257e6640 1 -- 172.21.15.98:0/4208408412 shutdown_connections 2024-09-19T08:00:20.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:20.130+0000 7fda257e6640 1 --2- 172.21.15.98:0/4208408412 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fda0c0a3240 0x7fda0c0a5700 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:20.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:20.130+0000 7fda257e6640 1 --2- 172.21.15.98:0/4208408412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda20107940 0x7fda20100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:20.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:20.130+0000 7fda257e6640 1 --2- 172.21.15.98:0/4208408412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20105010 0x7fda20100690 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:20.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:20.130+0000 7fda257e6640 1 --2- 172.21.15.98:0/4208408412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda201026e0 0x7fda20102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:20.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:20.130+0000 7fda257e6640 1 -- 172.21.15.98:0/4208408412 >> 172.21.15.98:0/4208408412 conn(0x7fda200fc280 msgr2=0x7fda20103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:20.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:20.130+0000 7fda257e6640 1 -- 172.21.15.98:0/4208408412 shutdown_connections 2024-09-19T08:00:20.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:20.130+0000 7fda257e6640 1 -- 172.21.15.98:0/4208408412 wait complete. 2024-09-19T08:00:20.756 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":70,"num_osds":12,"num_up_osds":11,"osd_up_since":1726732797,"num_in_osds":12,"osd_in_since":1726732798,"num_remapped_pgs":0} 2024-09-19T08:00:20.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-09-19T08:00:20.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-19T08:00:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: osdmap e70: 12 total, 11 up, 12 in 2024-09-19T08:00:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='osd.11 [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T08:00:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T08:00:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:00:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:00:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:00:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:00:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:20.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:00:20.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:20.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:00:20.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:20.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:00:20.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:20.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:20 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/4208408412' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T08:00:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-09-19T08:00:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-19T08:00:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: osdmap e70: 12 total, 11 up, 12 in 2024-09-19T08:00:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='osd.11 [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T08:00:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T08:00:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:00:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:00:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:00:20.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:00:20.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:20.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:20.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:00:20.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:20.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:00:20.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:20.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:00:20.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:20.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:20 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/4208408412' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T08:00:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-09-19T08:00:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-19T08:00:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: osdmap e70: 12 total, 11 up, 12 in 2024-09-19T08:00:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='osd.11 [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T08:00:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-09-19T08:00:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:00:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:00:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:00:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:00:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:00:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:00:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:00:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:20 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/4208408412' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T08:00:21.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-19T08:00:21.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:00:21 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:00:21.464+0000 7f4476760640 -1 osd.11 0 waiting for initial osdmap 2024-09-19T08:00:21.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:00:21 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:00:21.481+0000 7f446ed92640 -1 osd.11 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T08:00:21.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:21 smithi162 ceph-mon[35260]: Detected new or changed devices on smithi162 2024-09-19T08:00:21.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:21 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi162 to 4272M 2024-09-19T08:00:21.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:21 smithi162 ceph-mon[35260]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-19T08:00:21.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:21 smithi162 ceph-mon[35260]: osdmap e71: 12 total, 11 up, 12 in 2024-09-19T08:00:21.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:21 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:21.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:21 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:21.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:21 smithi162 ceph-mon[35260]: from='osd.11 ' entity='osd.11' 2024-09-19T08:00:21.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:21 smithi139 ceph-mon[34874]: Detected new or changed devices on smithi162 2024-09-19T08:00:21.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:21 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi162 to 4272M 2024-09-19T08:00:21.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:21 smithi139 ceph-mon[34874]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-19T08:00:21.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:21 smithi139 ceph-mon[34874]: osdmap e71: 12 total, 11 up, 12 in 2024-09-19T08:00:21.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:21 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:21.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:21 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:21.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:21 smithi139 ceph-mon[34874]: from='osd.11 ' entity='osd.11' 2024-09-19T08:00:21.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:21 smithi098 ceph-mon[31333]: Detected new or changed devices on smithi162 2024-09-19T08:00:21.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:21 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi162 to 4272M 2024-09-19T08:00:21.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:21 smithi098 ceph-mon[31333]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-19T08:00:21.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:21 smithi098 ceph-mon[31333]: osdmap e71: 12 total, 11 up, 12 in 2024-09-19T08:00:21.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:21 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:21.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:21 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:21.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:21 smithi098 ceph-mon[31333]: from='osd.11 ' entity='osd.11' 2024-09-19T08:00:22.016 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:00:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.443+0000 7f1c271d9640 1 -- 172.21.15.98:0/2728151138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20101cd0 msgr2=0x7f1c20102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.443+0000 7f1c271d9640 1 --2- 172.21.15.98:0/2728151138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20101cd0 0x7f1c20102150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009900 tx=0x7f1c0802a410 comp rx=0 tx=0).stop 2024-09-19T08:00:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.443+0000 7f1c271d9640 1 -- 172.21.15.98:0/2728151138 shutdown_connections 2024-09-19T08:00:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.443+0000 7f1c271d9640 1 --2- 172.21.15.98:0/2728151138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c20102690 0x7f1c20108f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.443+0000 7f1c271d9640 1 --2- 172.21.15.98:0/2728151138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20101cd0 0x7f1c20102150 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.443+0000 7f1c271d9640 1 --2- 172.21.15.98:0/2728151138 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c20100ad0 0x7f1c20100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.443+0000 7f1c271d9640 1 -- 172.21.15.98:0/2728151138 >> 172.21.15.98:0/2728151138 conn(0x7f1c200fc280 msgr2=0x7f1c200fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.443+0000 7f1c271d9640 1 -- 172.21.15.98:0/2728151138 shutdown_connections 2024-09-19T08:00:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.443+0000 7f1c271d9640 1 -- 172.21.15.98:0/2728151138 wait complete. 2024-09-19T08:00:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c271d9640 1 Processor -- start 2024-09-19T08:00:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c271d9640 1 -- start start 2024-09-19T08:00:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c271d9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c20100ad0 0x7f1c201980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c271d9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c20101cd0 0x7f1c201985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c271d9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20102690 0x7f1c2019f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c271d9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2010d960 con 0x7f1c20102690 2024-09-19T08:00:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c271d9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2010d7e0 con 0x7f1c20100ad0 2024-09-19T08:00:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c271d9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2010dae0 con 0x7f1c20101cd0 2024-09-19T08:00:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c25776640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20102690 0x7f1c2019f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c25776640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20102690 0x7f1c2019f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50670/0 (socket says 172.21.15.98:50670) 2024-09-19T08:00:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c25776640 1 -- 172.21.15.98:0/3345720879 learned_addr learned my addr 172.21.15.98:0/3345720879 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:00:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c1ffff640 1 --2- 172.21.15.98:0/3345720879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c20101cd0 0x7f1c201985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c24f75640 1 --2- 172.21.15.98:0/3345720879 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c20100ad0 0x7f1c201980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c25776640 1 -- 172.21.15.98:0/3345720879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c20101cd0 msgr2=0x7f1c201985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c25776640 1 --2- 172.21.15.98:0/3345720879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c20101cd0 0x7f1c201985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c25776640 1 -- 172.21.15.98:0/3345720879 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c20100ad0 msgr2=0x7f1c201980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c25776640 1 --2- 172.21.15.98:0/3345720879 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c20100ad0 0x7f1c201980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c25776640 1 -- 172.21.15.98:0/3345720879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c2019fcd0 con 0x7f1c20102690 2024-09-19T08:00:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c25776640 1 --2- 172.21.15.98:0/3345720879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20102690 0x7f1c2019f660 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800b970 tx=0x7f1c1800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.444+0000 7f1c1dffb640 1 -- 172.21.15.98:0/3345720879 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c18019070 con 0x7f1c20102690 2024-09-19T08:00:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.445+0000 7f1c1dffb640 1 -- 172.21.15.98:0/3345720879 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c18014440 con 0x7f1c20102690 2024-09-19T08:00:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.445+0000 7f1c271d9640 1 -- 172.21.15.98:0/3345720879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c2019ffc0 con 0x7f1c20102690 2024-09-19T08:00:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.445+0000 7f1c1dffb640 1 -- 172.21.15.98:0/3345720879 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c180075d0 con 0x7f1c20102690 2024-09-19T08:00:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.445+0000 7f1c271d9640 1 -- 172.21.15.98:0/3345720879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c201a04f0 con 0x7f1c20102690 2024-09-19T08:00:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.446+0000 7f1c271d9640 1 -- 172.21.15.98:0/3345720879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c20107f60 con 0x7f1c20102690 2024-09-19T08:00:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.447+0000 7f1c1dffb640 1 -- 172.21.15.98:0/3345720879 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f1c1801c070 con 0x7f1c20102690 2024-09-19T08:00:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.448+0000 7f1c1dffb640 1 --2- 172.21.15.98:0/3345720879 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1c0c0a3270 0x7f1c0c0a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.448+0000 7f1c24f75640 1 --2- 172.21.15.98:0/3345720879 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1c0c0a3270 0x7f1c0c0a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.448+0000 7f1c1dffb640 1 -- 172.21.15.98:0/3345720879 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6847+0+0 (secure 0 0 0) 0x7f1c180ced10 con 0x7f1c20102690 2024-09-19T08:00:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.448+0000 7f1c24f75640 1 --2- 172.21.15.98:0/3345720879 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1c0c0a3270 0x7f1c0c0a5730 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1c20101b30 tx=0x7f1c10009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.451+0000 7f1c1dffb640 1 -- 172.21.15.98:0/3345720879 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f1c1809b9a0 con 0x7f1c20102690 2024-09-19T08:00:22.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.619+0000 7f1c271d9640 1 -- 172.21.15.98:0/3345720879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f1c2002d990 con 0x7f1c20102690 2024-09-19T08:00:22.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.619+0000 7f1c1dffb640 1 -- 172.21.15.98:0/3345720879 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7f1c1809fca0 con 0x7f1c20102690 2024-09-19T08:00:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:00:22.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.621+0000 7f1c271d9640 1 -- 172.21.15.98:0/3345720879 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1c0c0a3270 msgr2=0x7f1c0c0a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:22.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.621+0000 7f1c271d9640 1 --2- 172.21.15.98:0/3345720879 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1c0c0a3270 0x7f1c0c0a5730 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1c20101b30 tx=0x7f1c10009110 comp rx=0 tx=0).stop 2024-09-19T08:00:22.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.621+0000 7f1c271d9640 1 -- 172.21.15.98:0/3345720879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20102690 msgr2=0x7f1c2019f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:22.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.621+0000 7f1c271d9640 1 --2- 172.21.15.98:0/3345720879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20102690 0x7f1c2019f660 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800b970 tx=0x7f1c1800bd70 comp rx=0 tx=0).stop 2024-09-19T08:00:22.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.621+0000 7f1c271d9640 1 -- 172.21.15.98:0/3345720879 shutdown_connections 2024-09-19T08:00:22.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.621+0000 7f1c271d9640 1 --2- 172.21.15.98:0/3345720879 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1c0c0a3270 0x7f1c0c0a5730 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:22.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.621+0000 7f1c271d9640 1 --2- 172.21.15.98:0/3345720879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20102690 0x7f1c2019f660 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:22.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.621+0000 7f1c271d9640 1 --2- 172.21.15.98:0/3345720879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c20101cd0 0x7f1c201985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:22.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.621+0000 7f1c271d9640 1 --2- 172.21.15.98:0/3345720879 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c20100ad0 0x7f1c201980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:22.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.621+0000 7f1c271d9640 1 -- 172.21.15.98:0/3345720879 >> 172.21.15.98:0/3345720879 conn(0x7f1c200fc280 msgr2=0x7f1c20105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:22.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.621+0000 7f1c271d9640 1 -- 172.21.15.98:0/3345720879 shutdown_connections 2024-09-19T08:00:22.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:22.621+0000 7f1c271d9640 1 -- 172.21.15.98:0/3345720879 wait complete. 2024-09-19T08:00:22.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:22 smithi139 ceph-mon[34874]: purged_snaps scrub starts 2024-09-19T08:00:22.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:22 smithi139 ceph-mon[34874]: purged_snaps scrub ok 2024-09-19T08:00:22.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:22 smithi139 ceph-mon[34874]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-09-19T08:00:22.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:22 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:22.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:22 smithi139 ceph-mon[34874]: osd.11 [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554] boot 2024-09-19T08:00:22.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:22 smithi139 ceph-mon[34874]: osdmap e72: 12 total, 12 up, 12 in 2024-09-19T08:00:22.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:22 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:22.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:22 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3345720879' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T08:00:22.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:22 smithi098 ceph-mon[31333]: purged_snaps scrub starts 2024-09-19T08:00:22.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:22 smithi098 ceph-mon[31333]: purged_snaps scrub ok 2024-09-19T08:00:22.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:22 smithi098 ceph-mon[31333]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-09-19T08:00:22.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:22 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:22.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:22 smithi098 ceph-mon[31333]: osd.11 [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554] boot 2024-09-19T08:00:22.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:22 smithi098 ceph-mon[31333]: osdmap e72: 12 total, 12 up, 12 in 2024-09-19T08:00:22.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:22 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:22.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:22 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3345720879' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T08:00:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:22 smithi162 ceph-mon[35260]: purged_snaps scrub starts 2024-09-19T08:00:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:22 smithi162 ceph-mon[35260]: purged_snaps scrub ok 2024-09-19T08:00:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:22 smithi162 ceph-mon[35260]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-09-19T08:00:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:22 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:22 smithi162 ceph-mon[35260]: osd.11 [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554] boot 2024-09-19T08:00:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:22 smithi162 ceph-mon[35260]: osdmap e72: 12 total, 12 up, 12 in 2024-09-19T08:00:23.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:22 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:00:23.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:22 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3345720879' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T08:00:23.367 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1726732822,"num_in_osds":12,"osd_in_since":1726732798,"num_remapped_pgs":0} 2024-09-19T08:00:23.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T08:00:23.611 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:00:24.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.079+0000 7f303b95d640 1 -- 172.21.15.98:0/815500394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3034102d90 msgr2=0x7f3034103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:24.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.079+0000 7f303b95d640 1 --2- 172.21.15.98:0/815500394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3034102d90 0x7f3034103170 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3030009930 tx=0x7f303002a420 comp rx=0 tx=0).stop 2024-09-19T08:00:24.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.079+0000 7f303b95d640 1 -- 172.21.15.98:0/815500394 shutdown_connections 2024-09-19T08:00:24.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.079+0000 7f303b95d640 1 --2- 172.21.15.98:0/815500394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3034108290 0x7f303410ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:24.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.079+0000 7f303b95d640 1 --2- 172.21.15.98:0/815500394 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3034103740 0x7f3034107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:24.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.079+0000 7f303b95d640 1 --2- 172.21.15.98:0/815500394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3034102d90 0x7f3034103170 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:24.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.079+0000 7f303b95d640 1 -- 172.21.15.98:0/815500394 >> 172.21.15.98:0/815500394 conn(0x7f30340fc7d0 msgr2=0x7f30340febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:24.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.079+0000 7f303b95d640 1 -- 172.21.15.98:0/815500394 shutdown_connections 2024-09-19T08:00:24.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.079+0000 7f303b95d640 1 -- 172.21.15.98:0/815500394 wait complete. 2024-09-19T08:00:24.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.079+0000 7f303b95d640 1 Processor -- start 2024-09-19T08:00:24.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f303b95d640 1 -- start start 2024-09-19T08:00:24.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f303b95d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3034102d90 0x7f30341985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:24.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f303b95d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3034103740 0x7f3034198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:24.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f303b95d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3034108290 0x7f303419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:24.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f303b95d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3034112090 con 0x7f3034102d90 2024-09-19T08:00:24.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f303b95d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3034111f10 con 0x7f3034103740 2024-09-19T08:00:24.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f303b95d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3034112210 con 0x7f3034108290 2024-09-19T08:00:24.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f30396f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3034102d90 0x7f30341985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:24.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f30396f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3034102d90 0x7f30341985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50688/0 (socket says 172.21.15.98:50688) 2024-09-19T08:00:24.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f30396f9640 1 -- 172.21.15.98:0/3536665059 learned_addr learned my addr 172.21.15.98:0/3536665059 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:00:24.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f3038ef8640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3034103740 0x7f3034198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:24.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f3039efa640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3034108290 0x7f303419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:24.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f30396f9640 1 -- 172.21.15.98:0/3536665059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3034108290 msgr2=0x7f303419cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:24.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f30396f9640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3034108290 0x7f303419cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:24.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f30396f9640 1 -- 172.21.15.98:0/3536665059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3034103740 msgr2=0x7f3034198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:24.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f30396f9640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3034103740 0x7f3034198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:24.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f30396f9640 1 -- 172.21.15.98:0/3536665059 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f303419d5a0 con 0x7f3034102d90 2024-09-19T08:00:24.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f3038ef8640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3034103740 0x7f3034198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:00:24.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f30396f9640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3034102d90 0x7f30341985f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3030009230 tx=0x7f3030004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:24.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f302a7fc640 1 -- 172.21.15.98:0/3536665059 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303003f030 con 0x7f3034102d90 2024-09-19T08:00:24.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f302a7fc640 1 -- 172.21.15.98:0/3536665059 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3030003ad0 con 0x7f3034102d90 2024-09-19T08:00:24.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.080+0000 7f303b95d640 1 -- 172.21.15.98:0/3536665059 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f303419d830 con 0x7f3034102d90 2024-09-19T08:00:24.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.081+0000 7f302a7fc640 1 -- 172.21.15.98:0/3536665059 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3030035070 con 0x7f3034102d90 2024-09-19T08:00:24.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.081+0000 7f303b95d640 1 -- 172.21.15.98:0/3536665059 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30341a5170 con 0x7f3034102d90 2024-09-19T08:00:24.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.081+0000 7f303b95d640 1 -- 172.21.15.98:0/3536665059 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f303406a7f0 con 0x7f3034102d90 2024-09-19T08:00:24.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.083+0000 7f302a7fc640 1 -- 172.21.15.98:0/3536665059 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f3030042030 con 0x7f3034102d90 2024-09-19T08:00:24.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.083+0000 7f302a7fc640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30200a32c0 0x7f30200a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:24.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.083+0000 7f3038ef8640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30200a32c0 0x7f30200a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:24.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.083+0000 7f302a7fc640 1 -- 172.21.15.98:0/3536665059 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f30300efd90 con 0x7f3034102d90 2024-09-19T08:00:24.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.084+0000 7f3038ef8640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30200a32c0 0x7f30200a5780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f30340fede0 tx=0x7f3024005ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:24.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.087+0000 7f302a7fc640 1 -- 172.21.15.98:0/3536665059 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f30300bcf50 con 0x7f3034102d90 2024-09-19T08:00:24.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.256+0000 7f303b95d640 1 -- 172.21.15.98:0/3536665059 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f303410a2a0 con 0x7f3034102d90 2024-09-19T08:00:24.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.257+0000 7f302a7fc640 1 -- 172.21.15.98:0/3536665059 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+18490 (secure 0 0 0) 0x7f30300bc850 con 0x7f3034102d90 2024-09-19T08:00:24.258 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:00:24.258 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":73,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","created":"2024-09-19T07:53:39.127398+0000","modified":"2024-09-19T08:00:23.646440+0000","last_up_change":"2024-09-19T08:00:22.467777+0000","last_in_change":"2024-09-19T07:59:58.538492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:56:43.773892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d01ce6-165d-493d-ae98-d8ad9e5d4134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6803","nonce":1719360533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6805","nonce":1719360533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6809","nonce":1719360533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6807","nonce":1719360533}]},"public_addr":"172.21.15.98:6803/1719360533","cluster_addr":"172.21.15.98:6805/1719360533","heartbeat_back_addr":"172.21.15.98:6809/1719360533","heartbeat_front_addr":"172.21.15.98:6807/1719360533","state":["exists","up"]},{"osd":1,"uuid":"34724bd5-0738-4473-b087-c09031c1ee2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036962626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036962626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036962626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036962626}]},"public_addr":"172.21.15.139:6801/4036962626","cluster_addr":"172.21.15.139:6803/4036962626","heartbeat_back_addr":"172.21.15.139:6807/4036962626","heartbeat_front_addr":"172.21.15.139:6805/4036962626","state":["exists","up"]},{"osd":2,"uuid":"16985ebe-0e32-42ee-bded-b7838e290afc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6801","nonce":2811412891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6803","nonce":2811412891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6807","nonce":2811412891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6805","nonce":2811412891}]},"public_addr":"172.21.15.162:6801/2811412891","cluster_addr":"172.21.15.162:6803/2811412891","heartbeat_back_addr":"172.21.15.162:6807/2811412891","heartbeat_front_addr":"172.21.15.162:6805/2811412891","state":["exists","up"]},{"osd":3,"uuid":"b140e466-3119-49cb-aec2-ba6ca4d10d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6811","nonce":1383392014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6813","nonce":1383392014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6817","nonce":1383392014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6815","nonce":1383392014}]},"public_addr":"172.21.15.98:6811/1383392014","cluster_addr":"172.21.15.98:6813/1383392014","heartbeat_back_addr":"172.21.15.98:6817/1383392014","heartbeat_front_addr":"172.21.15.98:6815/1383392014","state":["exists","up"]},{"osd":4,"uuid":"b4412813-df57-4ac6-8fb7-f57b69c74768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6809","nonce":116453520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6811","nonce":116453520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6815","nonce":116453520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6813","nonce":116453520}]},"public_addr":"172.21.15.139:6809/116453520","cluster_addr":"172.21.15.139:6811/116453520","heartbeat_back_addr":"172.21.15.139:6815/116453520","heartbeat_front_addr":"172.21.15.139:6813/116453520","state":["exists","up"]},{"osd":5,"uuid":"d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6809","nonce":3769530297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6811","nonce":3769530297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6815","nonce":3769530297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6813","nonce":3769530297}]},"public_addr":"172.21.15.162:6809/3769530297","cluster_addr":"172.21.15.162:6811/3769530297","heartbeat_back_addr":"172.21.15.162:6815/3769530297","heartbeat_front_addr":"172.21.15.162:6813/3769530297","state":["exists","up"]},{"osd":6,"uuid":"ba1c1590-6aea-40dc-90c8-e21b88896909","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6819","nonce":692373267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6821","nonce":692373267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6825","nonce":692373267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6823","nonce":692373267}]},"public_addr":"172.21.15.98:6819/692373267","cluster_addr":"172.21.15.98:6821/692373267","heartbeat_back_addr":"172.21.15.98:6825/692373267","heartbeat_front_addr":"172.21.15.98:6823/692373267","state":["exists","up"]},{"osd":7,"uuid":"eb9093ad-a068-4a63-ab87-0b50522cbc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6817","nonce":1963811650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6819","nonce":1963811650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6823","nonce":1963811650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6821","nonce":1963811650}]},"public_addr":"172.21.15.139:6817/1963811650","cluster_addr":"172.21.15.139:6819/1963811650","heartbeat_back_addr":"172.21.15.139:6823/1963811650","heartbeat_front_addr":"172.21.15.139:6821/1963811650","state":["exists","up"]},{"osd":8,"uuid":"31ca330a-fc45-466d-81c2-ef4386b3cc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6817","nonce":3607966458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6819","nonce":3607966458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6823","nonce":3607966458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6821","nonce":3607966458}]},"public_addr":"172.21.15.162:6817/3607966458","cluster_addr":"172.21.15.162:6819/3607966458","heartbeat_back_addr":"172.21.15.162:6823/3607966458","heartbeat_front_addr":"172.21.15.162:6821/3607966458","state":["exists","up"]},{"osd":9,"uuid":"9c4f216f-aef1-4367-9c58-b913b0c4f7db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6827","nonce":2050391244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6829","nonce":2050391244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6833","nonce":2050391244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6831","nonce":2050391244}]},"public_addr":"172.21.15.98:6827/2050391244","cluster_addr":"172.21.15.98:6829/2050391244","heartbeat_back_addr":"172.21.15.98:6833/2050391244","heartbeat_front_addr":"172.21.15.98:6831/2050391244","state":["exists","up"]},{"osd":10,"uuid":"03cc7df6-84ae-4d9a-a3d7-74c91e1744e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6825","nonce":2336761639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6827","nonce":2336761639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6831","nonce":2336761639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6829","nonce":2336761639}]},"public_addr":"172.21.15.139:6825/2336761639","cluster_addr":"172.21.15.139:6827/2336761639","heartbeat_back_addr":"172.21.15.139:6831/2336761639","heartbeat_front_addr":"172.21.15.139:6829/2336761639","state":["exists","up"]},{"osd":11,"uuid":"29e95498-4952-46d4-8fa6-6d8e2b862111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6825","nonce":1231011554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6827","nonce":1231011554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6831","nonce":1231011554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6829","nonce":1231011554}]},"public_addr":"172.21.15.162:6825/1231011554","cluster_addr":"172.21.15.162:6827/1231011554","heartbeat_back_addr":"172.21.15.162:6831/1231011554","heartbeat_front_addr":"172.21.15.162:6829/1231011554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:55:49.995244+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:15.655543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:40.331165+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:04.688881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:29.174966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:53.663161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:17.407551+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:42.584296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:06.317219+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:31.078625+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:55.689269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T08:00:19.659748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/3842627255":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3838590070":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/1238193420":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6800/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6800/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3194864054":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3866031018":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/1497156658":"2024-09-20T07:54:02.418785+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T08:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.259+0000 7f303b95d640 1 -- 172.21.15.98:0/3536665059 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30200a32c0 msgr2=0x7f30200a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.259+0000 7f303b95d640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30200a32c0 0x7f30200a5780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f30340fede0 tx=0x7f3024005ec0 comp rx=0 tx=0).stop 2024-09-19T08:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.259+0000 7f303b95d640 1 -- 172.21.15.98:0/3536665059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3034102d90 msgr2=0x7f30341985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.259+0000 7f303b95d640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3034102d90 0x7f30341985f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3030009230 tx=0x7f3030004270 comp rx=0 tx=0).stop 2024-09-19T08:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.259+0000 7f303b95d640 1 -- 172.21.15.98:0/3536665059 shutdown_connections 2024-09-19T08:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.259+0000 7f303b95d640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30200a32c0 0x7f30200a5780 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.259+0000 7f303b95d640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3034108290 0x7f303419cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.259+0000 7f303b95d640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3034103740 0x7f3034198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.259+0000 7f303b95d640 1 --2- 172.21.15.98:0/3536665059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3034102d90 0x7f30341985f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.259+0000 7f303b95d640 1 -- 172.21.15.98:0/3536665059 >> 172.21.15.98:0/3536665059 conn(0x7f30340fc7d0 msgr2=0x7f3034106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.259+0000 7f303b95d640 1 -- 172.21.15.98:0/3536665059 shutdown_connections 2024-09-19T08:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:24.259+0000 7f303b95d640 1 -- 172.21.15.98:0/3536665059 wait complete. 2024-09-19T08:00:24.892 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-19T07:56:43.773892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-19T08:00:24.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-19T08:00:24.919 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:24 smithi098 ceph-mon[31333]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:24.920 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:24 smithi098 ceph-mon[31333]: osdmap e73: 12 total, 12 up, 12 in 2024-09-19T08:00:24.920 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:24 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3536665059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:00:24.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:24 smithi139 ceph-mon[34874]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:24.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:24 smithi139 ceph-mon[34874]: osdmap e73: 12 total, 12 up, 12 in 2024-09-19T08:00:24.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:24 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3536665059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:00:25.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:24 smithi162 ceph-mon[35260]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:25.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:24 smithi162 ceph-mon[35260]: osdmap e73: 12 total, 12 up, 12 in 2024-09-19T08:00:25.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:24 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3536665059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:00:25.137 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:00:25.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.560+0000 7ff881c08640 1 -- 172.21.15.98:0/3738991129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff87c106b80 msgr2=0x7ff87c106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:25.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.560+0000 7ff881c08640 1 --2- 172.21.15.98:0/3738991129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff87c106b80 0x7ff87c106f60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff86c009900 tx=0x7ff86c02a410 comp rx=0 tx=0).stop 2024-09-19T08:00:25.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.560+0000 7ff881c08640 1 -- 172.21.15.98:0/3738991129 shutdown_connections 2024-09-19T08:00:25.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.560+0000 7ff881c08640 1 --2- 172.21.15.98:0/3738991129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff87c100aa0 0x7ff87c10ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:25.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.560+0000 7ff881c08640 1 --2- 172.21.15.98:0/3738991129 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff87c100100 0x7ff87c100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:25.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.560+0000 7ff881c08640 1 --2- 172.21.15.98:0/3738991129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff87c106b80 0x7ff87c106f60 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:25.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.560+0000 7ff881c08640 1 -- 172.21.15.98:0/3738991129 >> 172.21.15.98:0/3738991129 conn(0x7ff87c077230 msgr2=0x7ff87c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:25.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.560+0000 7ff881c08640 1 -- 172.21.15.98:0/3738991129 shutdown_connections 2024-09-19T08:00:25.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.560+0000 7ff881c08640 1 -- 172.21.15.98:0/3738991129 wait complete. 2024-09-19T08:00:25.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.561+0000 7ff881c08640 1 Processor -- start 2024-09-19T08:00:25.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.561+0000 7ff881c08640 1 -- start start 2024-09-19T08:00:25.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.561+0000 7ff881c08640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff87c100100 0x7ff87c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:25.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.561+0000 7ff881c08640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff87c100aa0 0x7ff87c198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:25.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.561+0000 7ff881c08640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff87c106b80 0x7ff87c19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.561+0000 7ff881c08640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff87c112090 con 0x7ff87c100aa0 2024-09-19T08:00:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.561+0000 7ff881c08640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff87c111f10 con 0x7ff87c100100 2024-09-19T08:00:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.561+0000 7ff881c08640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff87c112210 con 0x7ff87c106b80 2024-09-19T08:00:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.561+0000 7ff87affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff87c100aa0 0x7ff87c198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.561+0000 7ff87affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff87c100aa0 0x7ff87c198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50716/0 (socket says 172.21.15.98:50716) 2024-09-19T08:00:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.561+0000 7ff87affd640 1 -- 172.21.15.98:0/1481752457 learned_addr learned my addr 172.21.15.98:0/1481752457 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:00:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.561+0000 7ff87b7fe640 1 --2- 172.21.15.98:0/1481752457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff87c100100 0x7ff87c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.562+0000 7ff87affd640 1 -- 172.21.15.98:0/1481752457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff87c106b80 msgr2=0x7ff87c19ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.562+0000 7ff87affd640 1 --2- 172.21.15.98:0/1481752457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff87c106b80 0x7ff87c19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.562+0000 7ff87affd640 1 -- 172.21.15.98:0/1481752457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff87c100100 msgr2=0x7ff87c198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.562+0000 7ff87affd640 1 --2- 172.21.15.98:0/1481752457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff87c100100 0x7ff87c198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.562+0000 7ff87affd640 1 -- 172.21.15.98:0/1481752457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87c19d540 con 0x7ff87c100aa0 2024-09-19T08:00:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.562+0000 7ff87affd640 1 --2- 172.21.15.98:0/1481752457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff87c100aa0 0x7ff87c198ad0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff87000d950 tx=0x7ff87000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.562+0000 7ff878ff9640 1 -- 172.21.15.98:0/1481752457 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870017070 con 0x7ff87c100aa0 2024-09-19T08:00:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.562+0000 7ff878ff9640 1 -- 172.21.15.98:0/1481752457 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff870009c30 con 0x7ff87c100aa0 2024-09-19T08:00:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.562+0000 7ff881c08640 1 -- 172.21.15.98:0/1481752457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff87c19d830 con 0x7ff87c100aa0 2024-09-19T08:00:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.562+0000 7ff878ff9640 1 -- 172.21.15.98:0/1481752457 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87000f6b0 con 0x7ff87c100aa0 2024-09-19T08:00:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.562+0000 7ff881c08640 1 -- 172.21.15.98:0/1481752457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff87c1a5110 con 0x7ff87c100aa0 2024-09-19T08:00:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.563+0000 7ff881c08640 1 -- 172.21.15.98:0/1481752457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff87c100560 con 0x7ff87c100aa0 2024-09-19T08:00:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.564+0000 7ff878ff9640 1 -- 172.21.15.98:0/1481752457 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ff8700109d0 con 0x7ff87c100aa0 2024-09-19T08:00:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.565+0000 7ff878ff9640 1 --2- 172.21.15.98:0/1481752457 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff8640a32c0 0x7ff8640a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.565+0000 7ff87b7fe640 1 --2- 172.21.15.98:0/1481752457 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff8640a32c0 0x7ff8640a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.565+0000 7ff878ff9640 1 -- 172.21.15.98:0/1481752457 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7ff8700cf030 con 0x7ff87c100aa0 2024-09-19T08:00:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.565+0000 7ff87b7fe640 1 --2- 172.21.15.98:0/1481752457 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff8640a32c0 0x7ff8640a5780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff86c009230 tx=0x7ff86c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.568+0000 7ff878ff9640 1 -- 172.21.15.98:0/1481752457 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ff870009de0 con 0x7ff87c100aa0 2024-09-19T08:00:25.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.734+0000 7ff881c08640 1 -- 172.21.15.98:0/1481752457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff87c02d7c0 con 0x7ff87c100aa0 2024-09-19T08:00:25.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.735+0000 7ff878ff9640 1 -- 172.21.15.98:0/1481752457 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7ff87009c270 con 0x7ff87c100aa0 2024-09-19T08:00:25.736 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-09-19T08:00:25.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.737+0000 7ff881c08640 1 -- 172.21.15.98:0/1481752457 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff8640a32c0 msgr2=0x7ff8640a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:25.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.737+0000 7ff881c08640 1 --2- 172.21.15.98:0/1481752457 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff8640a32c0 0x7ff8640a5780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff86c009230 tx=0x7ff86c0058d0 comp rx=0 tx=0).stop 2024-09-19T08:00:25.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.737+0000 7ff881c08640 1 -- 172.21.15.98:0/1481752457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff87c100aa0 msgr2=0x7ff87c198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:25.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.737+0000 7ff881c08640 1 --2- 172.21.15.98:0/1481752457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff87c100aa0 0x7ff87c198ad0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff87000d950 tx=0x7ff87000de10 comp rx=0 tx=0).stop 2024-09-19T08:00:25.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.737+0000 7ff881c08640 1 -- 172.21.15.98:0/1481752457 shutdown_connections 2024-09-19T08:00:25.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.737+0000 7ff881c08640 1 --2- 172.21.15.98:0/1481752457 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff8640a32c0 0x7ff8640a5780 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:25.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.737+0000 7ff881c08640 1 --2- 172.21.15.98:0/1481752457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff87c106b80 0x7ff87c19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:25.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.737+0000 7ff881c08640 1 --2- 172.21.15.98:0/1481752457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff87c100aa0 0x7ff87c198ad0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:25.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.737+0000 7ff881c08640 1 --2- 172.21.15.98:0/1481752457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff87c100100 0x7ff87c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:25.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.737+0000 7ff881c08640 1 -- 172.21.15.98:0/1481752457 >> 172.21.15.98:0/1481752457 conn(0x7ff87c077230 msgr2=0x7ff87c104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:25.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.737+0000 7ff881c08640 1 -- 172.21.15.98:0/1481752457 shutdown_connections 2024-09-19T08:00:25.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:25.737+0000 7ff881c08640 1 -- 172.21.15.98:0/1481752457 wait complete. 2024-09-19T08:00:26.415 INFO:tasks.cephadm:Adding mds.a on smithi098 2024-09-19T08:00:26.415 INFO:tasks.cephadm:Adding mds.d on smithi098 2024-09-19T08:00:26.415 INFO:tasks.cephadm:Adding mds.g on smithi098 2024-09-19T08:00:26.415 INFO:tasks.cephadm:Adding mds.j on smithi098 2024-09-19T08:00:26.415 INFO:tasks.cephadm:Adding mds.b on smithi139 2024-09-19T08:00:26.415 INFO:tasks.cephadm:Adding mds.e on smithi139 2024-09-19T08:00:26.415 INFO:tasks.cephadm:Adding mds.h on smithi139 2024-09-19T08:00:26.415 INFO:tasks.cephadm:Adding mds.k on smithi139 2024-09-19T08:00:26.416 INFO:tasks.cephadm:Adding mds.c on smithi162 2024-09-19T08:00:26.416 INFO:tasks.cephadm:Adding mds.f on smithi162 2024-09-19T08:00:26.416 INFO:tasks.cephadm:Adding mds.i on smithi162 2024-09-19T08:00:26.416 INFO:tasks.cephadm:Adding mds.l on smithi162 2024-09-19T08:00:26.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph orch apply mds all '12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l' 2024-09-19T08:00:26.679 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T08:00:26.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:26 smithi162 ceph-mon[35260]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:26.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:26 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1481752457' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-19T08:00:26.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:26 smithi139 ceph-mon[34874]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:26.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:26 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1481752457' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-19T08:00:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:26 smithi098 ceph-mon[31333]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:26 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1481752457' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-19T08:00:27.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.105+0000 7fd742331640 1 -- 172.21.15.162:0/3444741655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd73c102e00 msgr2=0x7fd73c103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:27.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.105+0000 7fd742331640 1 --2- 172.21.15.162:0/3444741655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd73c102e00 0x7fd73c103280 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd724009900 tx=0x7fd72402a410 comp rx=0 tx=0).stop 2024-09-19T08:00:27.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.105+0000 7fd742331640 1 -- 172.21.15.162:0/3444741655 shutdown_connections 2024-09-19T08:00:27.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.105+0000 7fd742331640 1 --2- 172.21.15.162:0/3444741655 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd73c1037c0 0x7fd73c10a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:27.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.105+0000 7fd742331640 1 --2- 172.21.15.162:0/3444741655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd73c102e00 0x7fd73c103280 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:27.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.105+0000 7fd742331640 1 --2- 172.21.15.162:0/3444741655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd73c101c00 0x7fd73c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:27.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.105+0000 7fd742331640 1 -- 172.21.15.162:0/3444741655 >> 172.21.15.162:0/3444741655 conn(0x7fd73c0fd3b0 msgr2=0x7fd73c0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:27.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.105+0000 7fd742331640 1 -- 172.21.15.162:0/3444741655 shutdown_connections 2024-09-19T08:00:27.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.105+0000 7fd742331640 1 -- 172.21.15.162:0/3444741655 wait complete. 2024-09-19T08:00:27.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.106+0000 7fd742331640 1 Processor -- start 2024-09-19T08:00:27.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.106+0000 7fd742331640 1 -- start start 2024-09-19T08:00:27.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd742331640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd73c101c00 0x7fd73c197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:27.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd742331640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd73c102e00 0x7fd73c198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:27.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd742331640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd73c1037c0 0x7fd73c19f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:27.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd742331640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c10d860 con 0x7fd73c101c00 2024-09-19T08:00:27.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd742331640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c10d6e0 con 0x7fd73c102e00 2024-09-19T08:00:27.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd742331640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c10d9e0 con 0x7fd73c1037c0 2024-09-19T08:00:27.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd73b7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd73c102e00 0x7fd73c198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:27.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd73bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd73c101c00 0x7fd73c197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:27.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd73b7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd73c102e00 0x7fd73c198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.162:41632/0 (socket says 172.21.15.162:41632) 2024-09-19T08:00:27.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd73bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd73c101c00 0x7fd73c197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.162:55770/0 (socket says 172.21.15.162:55770) 2024-09-19T08:00:27.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd73b7fe640 1 -- 172.21.15.162:0/80880960 learned_addr learned my addr 172.21.15.162:0/80880960 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-19T08:00:27.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd7408ce640 1 --2- 172.21.15.162:0/80880960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd73c1037c0 0x7fd73c19f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:27.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd73bfff640 1 -- 172.21.15.162:0/80880960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd73c1037c0 msgr2=0x7fd73c19f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:27.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd73bfff640 1 --2- 172.21.15.162:0/80880960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd73c1037c0 0x7fd73c19f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:27.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd73bfff640 1 -- 172.21.15.162:0/80880960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd73c102e00 msgr2=0x7fd73c198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:27.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd73bfff640 1 --2- 172.21.15.162:0/80880960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd73c102e00 0x7fd73c198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:27.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd73bfff640 1 -- 172.21.15.162:0/80880960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73c19fb10 con 0x7fd73c101c00 2024-09-19T08:00:27.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd73b7fe640 1 --2- 172.21.15.162:0/80880960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd73c102e00 0x7fd73c198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:00:27.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.107+0000 7fd73bfff640 1 --2- 172.21.15.162:0/80880960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd73c101c00 0x7fd73c197ee0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd72c00d6a0 tx=0x7fd72c00db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:27.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.108+0000 7fd7397fa640 1 -- 172.21.15.162:0/80880960 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72c017070 con 0x7fd73c101c00 2024-09-19T08:00:27.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.108+0000 7fd742331640 1 -- 172.21.15.162:0/80880960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd73c074480 con 0x7fd73c101c00 2024-09-19T08:00:27.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.108+0000 7fd7397fa640 1 -- 172.21.15.162:0/80880960 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd72c009d00 con 0x7fd73c101c00 2024-09-19T08:00:27.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.108+0000 7fd7397fa640 1 -- 172.21.15.162:0/80880960 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72c003c60 con 0x7fd73c101c00 2024-09-19T08:00:27.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.108+0000 7fd742331640 1 -- 172.21.15.162:0/80880960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd73c0749e0 con 0x7fd73c101c00 2024-09-19T08:00:27.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.109+0000 7fd742331640 1 -- 172.21.15.162:0/80880960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd73c109050 con 0x7fd73c101c00 2024-09-19T08:00:27.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.110+0000 7fd7397fa640 1 -- 172.21.15.162:0/80880960 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fd72c009820 con 0x7fd73c101c00 2024-09-19T08:00:27.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.110+0000 7fd7397fa640 1 --2- 172.21.15.162:0/80880960 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd7280a3270 0x7fd7280a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:27.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.111+0000 7fd7397fa640 1 -- 172.21.15.162:0/80880960 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fd72c0cf1b0 con 0x7fd73c101c00 2024-09-19T08:00:27.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.111+0000 7fd73b7fe640 1 --2- 172.21.15.162:0/80880960 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd7280a3270 0x7fd7280a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:27.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.111+0000 7fd73b7fe640 1 --2- 172.21.15.162:0/80880960 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd7280a3270 0x7fd7280a5730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd724031440 tx=0x7fd7240315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:27.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.114+0000 7fd7397fa640 1 -- 172.21.15.162:0/80880960 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fd72c09bc80 con 0x7fd73c101c00 2024-09-19T08:00:27.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.290+0000 7fd742331640 1 -- 172.21.15.162:0/80880960 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}) -- 0x7fd73c0627f0 con 0x7fd7280a3270 2024-09-19T08:00:27.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.297+0000 7fd7397fa640 1 -- 172.21.15.162:0/80880960 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fd73c0627f0 con 0x7fd7280a3270 2024-09-19T08:00:27.298 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mds.all update... 2024-09-19T08:00:27.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.299+0000 7fd742331640 1 -- 172.21.15.162:0/80880960 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd7280a3270 msgr2=0x7fd7280a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:27.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.299+0000 7fd742331640 1 --2- 172.21.15.162:0/80880960 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd7280a3270 0x7fd7280a5730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd724031440 tx=0x7fd7240315b0 comp rx=0 tx=0).stop 2024-09-19T08:00:27.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.299+0000 7fd742331640 1 -- 172.21.15.162:0/80880960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd73c101c00 msgr2=0x7fd73c197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:27.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.299+0000 7fd742331640 1 --2- 172.21.15.162:0/80880960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd73c101c00 0x7fd73c197ee0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd72c00d6a0 tx=0x7fd72c00db60 comp rx=0 tx=0).stop 2024-09-19T08:00:27.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.300+0000 7fd742331640 1 -- 172.21.15.162:0/80880960 shutdown_connections 2024-09-19T08:00:27.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.300+0000 7fd742331640 1 --2- 172.21.15.162:0/80880960 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd7280a3270 0x7fd7280a5730 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:27.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.300+0000 7fd742331640 1 --2- 172.21.15.162:0/80880960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd73c1037c0 0x7fd73c19f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:27.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.300+0000 7fd742331640 1 --2- 172.21.15.162:0/80880960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd73c102e00 0x7fd73c198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:27.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.300+0000 7fd742331640 1 --2- 172.21.15.162:0/80880960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd73c101c00 0x7fd73c197ee0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:27.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.300+0000 7fd742331640 1 -- 172.21.15.162:0/80880960 >> 172.21.15.162:0/80880960 conn(0x7fd73c0fd3b0 msgr2=0x7fd73c106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:27.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.300+0000 7fd742331640 1 -- 172.21.15.162:0/80880960 shutdown_connections 2024-09-19T08:00:27.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-19T08:00:27.300+0000 7fd742331640 1 -- 172.21.15.162:0/80880960 wait complete. 2024-09-19T08:00:27.880 DEBUG:teuthology.orchestra.run.smithi098:mds.a> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.a.service 2024-09-19T08:00:27.884 DEBUG:teuthology.orchestra.run.smithi098:mds.d> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.d.service 2024-09-19T08:00:27.927 DEBUG:teuthology.orchestra.run.smithi098:mds.g> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.g.service 2024-09-19T08:00:27.970 DEBUG:teuthology.orchestra.run.smithi098:mds.j> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.j.service 2024-09-19T08:00:28.015 DEBUG:teuthology.orchestra.run.smithi139:mds.b> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.b.service 2024-09-19T08:00:28.018 DEBUG:teuthology.orchestra.run.smithi139:mds.e> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.e.service 2024-09-19T08:00:28.062 DEBUG:teuthology.orchestra.run.smithi139:mds.h> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.h.service 2024-09-19T08:00:28.105 DEBUG:teuthology.orchestra.run.smithi139:mds.k> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.k.service 2024-09-19T08:00:28.148 DEBUG:teuthology.orchestra.run.smithi162:mds.c> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.c.service 2024-09-19T08:00:28.151 DEBUG:teuthology.orchestra.run.smithi162:mds.f> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.f.service 2024-09-19T08:00:28.194 DEBUG:teuthology.orchestra.run.smithi162:mds.i> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.i.service 2024-09-19T08:00:28.237 DEBUG:teuthology.orchestra.run.smithi162:mds.l> sudo journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.l.service 2024-09-19T08:00:28.280 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-09-19T08:00:28.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:00:28.411 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:28 smithi162 ceph-mon[35260]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:00:28.411 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:28 smithi162 ceph-mon[35260]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-09-19T08:00:28.411 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:28.411 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:00:28.411 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:28 smithi162 ceph-mon[35260]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:28.411 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:28.411 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:00:28.411 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:28.411 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:28.412 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:28.412 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:28 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:28.412 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:28 smithi162 ceph-mon[35260]: Deploying daemon mds.l on smithi162 2024-09-19T08:00:28.523 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:00:28.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:28 smithi098 ceph-mon[31333]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:00:28.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:28 smithi098 ceph-mon[31333]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-09-19T08:00:28.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:28.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:00:28.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:28 smithi098 ceph-mon[31333]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:28.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:28.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:00:28.646 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:28.646 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:28.646 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:28.646 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:28 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:28.646 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:28 smithi098 ceph-mon[31333]: Deploying daemon mds.l on smithi162 2024-09-19T08:00:28.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:28 smithi139 ceph-mon[34874]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:00:28.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:28 smithi139 ceph-mon[34874]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-09-19T08:00:28.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:28.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:00:28.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:28 smithi139 ceph-mon[34874]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:28.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:28.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:00:28.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:28.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:28.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:28.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:28 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:28.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:28 smithi139 ceph-mon[34874]: Deploying daemon mds.l on smithi162 2024-09-19T08:00:28.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.970+0000 7f51c097f640 1 -- 172.21.15.98:0/268548083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51bc101420 msgr2=0x7f51bc10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:28.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.970+0000 7f51c097f640 1 --2- 172.21.15.98:0/268548083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51bc101420 0x7f51bc10d950 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009cd0 tx=0x7f51b402a8e0 comp rx=0 tx=0).stop 2024-09-19T08:00:28.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.970+0000 7f51c097f640 1 -- 172.21.15.98:0/268548083 shutdown_connections 2024-09-19T08:00:28.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.970+0000 7f51c097f640 1 --2- 172.21.15.98:0/268548083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51bc101420 0x7f51bc10d950 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:28.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.970+0000 7f51c097f640 1 --2- 172.21.15.98:0/268548083 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51bc100a80 0x7f51bc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:28.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.970+0000 7f51c097f640 1 --2- 172.21.15.98:0/268548083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51bc106a80 0x7f51bc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:28.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.970+0000 7f51c097f640 1 -- 172.21.15.98:0/268548083 >> 172.21.15.98:0/268548083 conn(0x7f51bc0fc7a0 msgr2=0x7f51bc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.970+0000 7f51c097f640 1 -- 172.21.15.98:0/268548083 shutdown_connections 2024-09-19T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.970+0000 7f51c097f640 1 -- 172.21.15.98:0/268548083 wait complete. 2024-09-19T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51c097f640 1 Processor -- start 2024-09-19T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51c097f640 1 -- start start 2024-09-19T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51c097f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51bc100a80 0x7f51bc1993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51c097f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51bc101420 0x7f51bc199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51c097f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51bc106a80 0x7f51bc1a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51c097f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc110f20 con 0x7f51bc100a80 2024-09-19T08:00:28.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51c097f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc110da0 con 0x7f51bc101420 2024-09-19T08:00:28.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51c097f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc1110a0 con 0x7f51bc106a80 2024-09-19T08:00:28.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51ba59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51bc100a80 0x7f51bc1993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:28.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51ba59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51bc100a80 0x7f51bc1993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34420/0 (socket says 172.21.15.98:34420) 2024-09-19T08:00:28.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51ba59c640 1 -- 172.21.15.98:0/786558502 learned_addr learned my addr 172.21.15.98:0/786558502 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:00:28.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51b9d9b640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51bc101420 0x7f51bc199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:28.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51bad9d640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51bc106a80 0x7f51bc1a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:28.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51ba59c640 1 -- 172.21.15.98:0/786558502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51bc106a80 msgr2=0x7f51bc1a4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:28.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51ba59c640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51bc106a80 0x7f51bc1a4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:28.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51ba59c640 1 -- 172.21.15.98:0/786558502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51bc101420 msgr2=0x7f51bc199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:28.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51ba59c640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51bc101420 0x7f51bc199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:28.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51ba59c640 1 -- 172.21.15.98:0/786558502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51bc1a49b0 con 0x7f51bc100a80 2024-09-19T08:00:28.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51bad9d640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51bc106a80 0x7f51bc1a4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:00:28.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.971+0000 7f51ba59c640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51bc100a80 0x7f51bc1993f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f51ac00b4a0 tx=0x7f51ac00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:28.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.972+0000 7f51b37fe640 1 -- 172.21.15.98:0/786558502 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac015020 con 0x7f51bc100a80 2024-09-19T08:00:28.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.972+0000 7f51b37fe640 1 -- 172.21.15.98:0/786558502 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51ac011d10 con 0x7f51bc100a80 2024-09-19T08:00:28.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.972+0000 7f51c097f640 1 -- 172.21.15.98:0/786558502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51bc1a4b80 con 0x7f51bc100a80 2024-09-19T08:00:28.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.972+0000 7f51b37fe640 1 -- 172.21.15.98:0/786558502 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac00f2e0 con 0x7f51bc100a80 2024-09-19T08:00:28.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.972+0000 7f51c097f640 1 -- 172.21.15.98:0/786558502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51bc1a5040 con 0x7f51bc100a80 2024-09-19T08:00:28.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.973+0000 7f51c097f640 1 -- 172.21.15.98:0/786558502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51bc100f60 con 0x7f51bc100a80 2024-09-19T08:00:28.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.974+0000 7f51b37fe640 1 -- 172.21.15.98:0/786558502 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f51ac018020 con 0x7f51bc100a80 2024-09-19T08:00:28.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.975+0000 7f51b37fe640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f51a80a3240 0x7f51a80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:28.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.975+0000 7f51b9d9b640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f51a80a3240 0x7f51a80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:28.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.975+0000 7f51b37fe640 1 -- 172.21.15.98:0/786558502 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f51ac0cedb0 con 0x7f51bc100a80 2024-09-19T08:00:28.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.975+0000 7f51b9d9b640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f51a80a3240 0x7f51a80a5700 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f51bc06b2e0 tx=0x7f51a4009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:28.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:28.979+0000 7f51b37fe640 1 -- 172.21.15.98:0/786558502 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f51ac01f020 con 0x7f51bc100a80 2024-09-19T08:00:29.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:29.180+0000 7f51c097f640 1 -- 172.21.15.98:0/786558502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51bc109010 con 0x7f51bc100a80 2024-09-19T08:00:29.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:29.181+0000 7f51b37fe640 1 -- 172.21.15.98:0/786558502 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7f51ac09bf70 con 0x7f51bc100a80 2024-09-19T08:00:29.182 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:00:29.182 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"btime":"2024-09-19T07:53:39:127203+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-09-19T08:00:29.182 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 1 2024-09-19T08:00:29.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:29.183+0000 7f51c097f640 1 -- 172.21.15.98:0/786558502 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f51a80a3240 msgr2=0x7f51a80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:29.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:29.183+0000 7f51c097f640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f51a80a3240 0x7f51a80a5700 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f51bc06b2e0 tx=0x7f51a4009110 comp rx=0 tx=0).stop 2024-09-19T08:00:29.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:29.183+0000 7f51c097f640 1 -- 172.21.15.98:0/786558502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51bc100a80 msgr2=0x7f51bc1993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:29.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:29.183+0000 7f51c097f640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51bc100a80 0x7f51bc1993f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f51ac00b4a0 tx=0x7f51ac00b8a0 comp rx=0 tx=0).stop 2024-09-19T08:00:29.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:29.183+0000 7f51c097f640 1 -- 172.21.15.98:0/786558502 shutdown_connections 2024-09-19T08:00:29.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:29.183+0000 7f51c097f640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f51a80a3240 0x7f51a80a5700 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:29.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:29.183+0000 7f51c097f640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51bc106a80 0x7f51bc1a4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:29.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:29.183+0000 7f51c097f640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51bc101420 0x7f51bc199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:29.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:29.183+0000 7f51c097f640 1 --2- 172.21.15.98:0/786558502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51bc100a80 0x7f51bc1993f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:29.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:29.183+0000 7f51c097f640 1 -- 172.21.15.98:0/786558502 >> 172.21.15.98:0/786558502 conn(0x7f51bc0fc7a0 msgr2=0x7f51bc0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:29.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:29.183+0000 7f51c097f640 1 -- 172.21.15.98:0/786558502 shutdown_connections 2024-09-19T08:00:29.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:29.183+0000 7f51c097f640 1 -- 172.21.15.98:0/786558502 wait complete. 2024-09-19T08:00:29.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:29 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/786558502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:29.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:29 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/786558502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:29.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:29 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/786558502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:30.306 INFO:journalctl@ceph.mds.l.smithi162.stdout:Sep 19 08:00:30 smithi162 systemd[1]: Started Ceph mds.l for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:00:30.604 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:30 smithi139 ceph-mon[34874]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:30.605 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:30 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] up:boot 2024-09-19T08:00:30.605 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:30 smithi139 ceph-mon[34874]: fsmap 1 up:standby 2024-09-19T08:00:30.605 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:30 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-19T08:00:30.605 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:30 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:30.605 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:30 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:30.605 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:30 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:30.605 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:30 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:30.605 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:30 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:30.605 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:30 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:30.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:30 smithi162 ceph-mon[35260]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:30.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:30 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] up:boot 2024-09-19T08:00:30.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:30 smithi162 ceph-mon[35260]: fsmap 1 up:standby 2024-09-19T08:00:30.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:30 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-19T08:00:30.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:30 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:30.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:30 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:30.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:30 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:30.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:30 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:30.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:30 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:30.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:30 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:30.685 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:30 smithi098 ceph-mon[31333]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:30.685 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:30 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] up:boot 2024-09-19T08:00:30.685 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:30 smithi098 ceph-mon[31333]: fsmap 1 up:standby 2024-09-19T08:00:30.685 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:30 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-19T08:00:30.685 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:30 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:30.686 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:30 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:30.686 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:30 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:30.686 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:30 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:30.686 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:30 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:30.686 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:30 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:31.608 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:31 smithi139 ceph-mon[34874]: Deploying daemon mds.k on smithi139 2024-09-19T08:00:31.619 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:31 smithi162 ceph-mon[35260]: Deploying daemon mds.k on smithi139 2024-09-19T08:00:31.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:31 smithi098 ceph-mon[31333]: Deploying daemon mds.k on smithi139 2024-09-19T08:00:31.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:00:32.071 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:00:32.485 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:32 smithi098 ceph-mon[31333]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:32.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.568+0000 7f431b25c640 1 -- 172.21.15.98:0/1392735291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4314102790 msgr2=0x7f43141067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:32.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.568+0000 7f431b25c640 1 --2- 172.21.15.98:0/1392735291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4314102790 0x7f43141067e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4300009900 tx=0x7f430002a410 comp rx=0 tx=0).stop 2024-09-19T08:00:32.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.568+0000 7f431b25c640 1 -- 172.21.15.98:0/1392735291 shutdown_connections 2024-09-19T08:00:32.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.568+0000 7f431b25c640 1 --2- 172.21.15.98:0/1392735291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4314106f10 0x7f431410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:32.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.568+0000 7f431b25c640 1 --2- 172.21.15.98:0/1392735291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4314102790 0x7f43141067e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:32.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.568+0000 7f431b25c640 1 --2- 172.21.15.98:0/1392735291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4314101de0 0x7f43141021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:32.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.568+0000 7f431b25c640 1 -- 172.21.15.98:0/1392735291 >> 172.21.15.98:0/1392735291 conn(0x7f43140fd880 msgr2=0x7f43140ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:32.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.568+0000 7f431b25c640 1 -- 172.21.15.98:0/1392735291 shutdown_connections 2024-09-19T08:00:32.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.568+0000 7f431b25c640 1 -- 172.21.15.98:0/1392735291 wait complete. 2024-09-19T08:00:32.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f431b25c640 1 Processor -- start 2024-09-19T08:00:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f431b25c640 1 -- start start 2024-09-19T08:00:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f431b25c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4314101de0 0x7f43140744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f431b25c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4314102790 0x7f4314074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f431b25c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4314106f10 0x7f4314074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f431b25c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4314111f70 con 0x7f4314106f10 2024-09-19T08:00:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f431b25c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4314111df0 con 0x7f4314101de0 2024-09-19T08:00:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f431b25c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f43141120f0 con 0x7f4314102790 2024-09-19T08:00:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f43197f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4314106f10 0x7f4314074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f43197f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4314106f10 0x7f4314074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34430/0 (socket says 172.21.15.98:34430) 2024-09-19T08:00:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f4318ff8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4314101de0 0x7f43140744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f43197f9640 1 -- 172.21.15.98:0/1524657243 learned_addr learned my addr 172.21.15.98:0/1524657243 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:00:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f4313fff640 1 --2- 172.21.15.98:0/1524657243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4314102790 0x7f4314074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f43197f9640 1 -- 172.21.15.98:0/1524657243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4314102790 msgr2=0x7f4314074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f43197f9640 1 --2- 172.21.15.98:0/1524657243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4314102790 0x7f4314074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f43197f9640 1 -- 172.21.15.98:0/1524657243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4314101de0 msgr2=0x7f43140744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f43197f9640 1 --2- 172.21.15.98:0/1524657243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4314101de0 0x7f43140744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f43197f9640 1 -- 172.21.15.98:0/1524657243 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43141a6920 con 0x7f4314106f10 2024-09-19T08:00:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.569+0000 7f4318ff8640 1 --2- 172.21.15.98:0/1524657243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4314101de0 0x7f43140744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:00:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.570+0000 7f43197f9640 1 --2- 172.21.15.98:0/1524657243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4314106f10 0x7f4314074f70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f430c009fd0 tx=0x7f430c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.571+0000 7f4311ffb640 1 -- 172.21.15.98:0/1524657243 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c017070 con 0x7f4314106f10 2024-09-19T08:00:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.571+0000 7f431b25c640 1 -- 172.21.15.98:0/1524657243 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43141a6c10 con 0x7f4314106f10 2024-09-19T08:00:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.571+0000 7f4311ffb640 1 -- 172.21.15.98:0/1524657243 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f430c003e50 con 0x7f4314106f10 2024-09-19T08:00:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.571+0000 7f4311ffb640 1 -- 172.21.15.98:0/1524657243 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c020060 con 0x7f4314106f10 2024-09-19T08:00:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.571+0000 7f431b25c640 1 -- 172.21.15.98:0/1524657243 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43141a7120 con 0x7f4314106f10 2024-09-19T08:00:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.572+0000 7f431b25c640 1 -- 172.21.15.98:0/1524657243 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43141021c0 con 0x7f4314106f10 2024-09-19T08:00:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.573+0000 7f4311ffb640 1 -- 172.21.15.98:0/1524657243 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f430c0040b0 con 0x7f4314106f10 2024-09-19T08:00:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.574+0000 7f4311ffb640 1 --2- 172.21.15.98:0/1524657243 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43040a3240 0x7f43040a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.574+0000 7f4318ff8640 1 --2- 172.21.15.98:0/1524657243 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43040a3240 0x7f43040a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.574+0000 7f4311ffb640 1 -- 172.21.15.98:0/1524657243 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f430c0cee70 con 0x7f4314106f10 2024-09-19T08:00:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.575+0000 7f4318ff8640 1 --2- 172.21.15.98:0/1524657243 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43040a3240 0x7f43040a5700 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4308009fc0 tx=0x7f4308005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.581+0000 7f4311ffb640 1 -- 172.21.15.98:0/1524657243 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f430c0047d0 con 0x7f4314106f10 2024-09-19T08:00:32.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:32 smithi162 ceph-mon[35260]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:32.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:32 smithi139 ceph-mon[34874]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:32.678 INFO:journalctl@ceph.mds.k.smithi139.stdout:Sep 19 08:00:32 smithi139 systemd[1]: Started Ceph mds.k for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:00:32.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.790+0000 7f431b25c640 1 -- 172.21.15.98:0/1524657243 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f431402d7c0 con 0x7f4314106f10 2024-09-19T08:00:32.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.791+0000 7f4311ffb640 1 -- 172.21.15.98:0/1524657243 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1438 (secure 0 0 0) 0x7f430c09b940 con 0x7f4314106f10 2024-09-19T08:00:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:00:32.792 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":2,"btime":"2024-09-19T08:00:29:942102+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-19T08:00:32.793 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 2 2024-09-19T08:00:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.794+0000 7f431b25c640 1 -- 172.21.15.98:0/1524657243 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43040a3240 msgr2=0x7f43040a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.794+0000 7f431b25c640 1 --2- 172.21.15.98:0/1524657243 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43040a3240 0x7f43040a5700 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4308009fc0 tx=0x7f4308005f90 comp rx=0 tx=0).stop 2024-09-19T08:00:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.794+0000 7f431b25c640 1 -- 172.21.15.98:0/1524657243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4314106f10 msgr2=0x7f4314074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.794+0000 7f431b25c640 1 --2- 172.21.15.98:0/1524657243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4314106f10 0x7f4314074f70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f430c009fd0 tx=0x7f430c00b260 comp rx=0 tx=0).stop 2024-09-19T08:00:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.795+0000 7f431b25c640 1 -- 172.21.15.98:0/1524657243 shutdown_connections 2024-09-19T08:00:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.795+0000 7f431b25c640 1 --2- 172.21.15.98:0/1524657243 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43040a3240 0x7f43040a5700 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.795+0000 7f431b25c640 1 --2- 172.21.15.98:0/1524657243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4314106f10 0x7f4314074f70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.795+0000 7f431b25c640 1 --2- 172.21.15.98:0/1524657243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4314102790 0x7f4314074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:32.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.795+0000 7f431b25c640 1 --2- 172.21.15.98:0/1524657243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4314101de0 0x7f43140744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:32.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.795+0000 7f431b25c640 1 -- 172.21.15.98:0/1524657243 >> 172.21.15.98:0/1524657243 conn(0x7f43140fd880 msgr2=0x7f43140fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:32.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.795+0000 7f431b25c640 1 -- 172.21.15.98:0/1524657243 shutdown_connections 2024-09-19T08:00:32.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:32.795+0000 7f431b25c640 1 -- 172.21.15.98:0/1524657243 wait complete. 2024-09-19T08:00:33.819 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:33 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:33.819 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:33 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:33.819 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:33 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:33.819 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:33 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:33.819 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:33 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:33.819 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:33 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:33.819 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:33 smithi139 ceph-mon[34874]: Deploying daemon mds.j on smithi098 2024-09-19T08:00:33.820 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:33 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1524657243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:33.820 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:33 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.139:6832/2267154969,v1:172.21.15.139:6833/2267154969] up:boot 2024-09-19T08:00:33.820 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:33 smithi139 ceph-mon[34874]: fsmap 2 up:standby 2024-09-19T08:00:33.820 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:33 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-19T08:00:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:33 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:33 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:33 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:33 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:33 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:33 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:33 smithi162 ceph-mon[35260]: Deploying daemon mds.j on smithi098 2024-09-19T08:00:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:33 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1524657243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:33 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.139:6832/2267154969,v1:172.21.15.139:6833/2267154969] up:boot 2024-09-19T08:00:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:33 smithi162 ceph-mon[35260]: fsmap 2 up:standby 2024-09-19T08:00:33.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:33 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-19T08:00:33.885 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:33 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:33.885 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:33 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:33.885 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:33 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:33.885 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:33 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:33.885 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:33 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:33.885 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:33 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:33.886 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:33 smithi098 ceph-mon[31333]: Deploying daemon mds.j on smithi098 2024-09-19T08:00:33.886 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:33 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1524657243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:33.886 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:33 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.139:6832/2267154969,v1:172.21.15.139:6833/2267154969] up:boot 2024-09-19T08:00:33.886 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:33 smithi098 ceph-mon[31333]: fsmap 2 up:standby 2024-09-19T08:00:33.886 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:33 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-19T08:00:34.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:34 smithi098 ceph-mon[31333]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:34.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:34 smithi162 ceph-mon[35260]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:34.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:34 smithi139 ceph-mon[34874]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:35.385 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:00:35 smithi098 systemd[1]: Starting Ceph mds.j for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:00:35.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:00:35.656 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:00:35 smithi098 podman[70740]: 2024-09-19 08:00:35.287550008 +0000 UTC m=+0.015813907 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:00:35.656 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:00:35 smithi098 podman[70740]: 2024-09-19 08:00:35.417627399 +0000 UTC m=+0.145891284 container create d4b63495349adcabf78435929b2fa0ac66ec855ab7b3d1cb60cfc1e84ea8e359 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-19T08:00:35.656 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:00:35 smithi098 podman[70740]: 2024-09-19 08:00:35.628876605 +0000 UTC m=+0.357140491 container init d4b63495349adcabf78435929b2fa0ac66ec855ab7b3d1cb60cfc1e84ea8e359 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-19T08:00:35.656 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:00:35 smithi098 podman[70740]: 2024-09-19 08:00:35.632550572 +0000 UTC m=+0.360814462 container start d4b63495349adcabf78435929b2fa0ac66ec855ab7b3d1cb60cfc1e84ea8e359 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-j, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:00:35.827 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:00:35.931 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:00:35 smithi098 ceph-mds[70777]: -- 172.21.15.98:0/1151851490 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55883ff43860 con 0x55883ff20800 2024-09-19T08:00:35.932 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:00:35 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-j[70773]: starting mds.j at 2024-09-19T08:00:35.932 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:00:35 smithi098 bash[70740]: d4b63495349adcabf78435929b2fa0ac66ec855ab7b3d1cb60cfc1e84ea8e359 2024-09-19T08:00:35.932 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:00:35 smithi098 systemd[1]: Started Ceph mds.j for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:00:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.278+0000 7faef8ec8640 1 -- 172.21.15.98:0/560463884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef4106f50 msgr2=0x7faef410eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.278+0000 7faef8ec8640 1 --2- 172.21.15.98:0/560463884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef4106f50 0x7faef410eaf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7faeec009cd0 tx=0x7faeec02a8e0 comp rx=0 tx=0).stop 2024-09-19T08:00:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.278+0000 7faef8ec8640 1 -- 172.21.15.98:0/560463884 shutdown_connections 2024-09-19T08:00:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.278+0000 7faef8ec8640 1 --2- 172.21.15.98:0/560463884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef4106f50 0x7faef410eaf0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.278+0000 7faef8ec8640 1 --2- 172.21.15.98:0/560463884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faef4100ca0 0x7faef4106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.278+0000 7faef8ec8640 1 --2- 172.21.15.98:0/560463884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faef41002f0 0x7faef41006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.278+0000 7faef8ec8640 1 -- 172.21.15.98:0/560463884 >> 172.21.15.98:0/560463884 conn(0x7faef4077210 msgr2=0x7faef40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.278+0000 7faef8ec8640 1 -- 172.21.15.98:0/560463884 shutdown_connections 2024-09-19T08:00:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.278+0000 7faef8ec8640 1 -- 172.21.15.98:0/560463884 wait complete. 2024-09-19T08:00:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.279+0000 7faef8ec8640 1 Processor -- start 2024-09-19T08:00:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.279+0000 7faef8ec8640 1 -- start start 2024-09-19T08:00:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.279+0000 7faef8ec8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faef41002f0 0x7faef41a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef8ec8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef4100ca0 0x7faef41a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef8ec8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faef4106f50 0x7faef419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef8ec8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faef4112080 con 0x7faef4100ca0 2024-09-19T08:00:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef8ec8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7faef4111f00 con 0x7faef41002f0 2024-09-19T08:00:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef8ec8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faef4112200 con 0x7faef4106f50 2024-09-19T08:00:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef4100ca0 0x7faef41a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef4100ca0 0x7faef41a0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34850/0 (socket says 172.21.15.98:34850) 2024-09-19T08:00:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef1d9b640 1 -- 172.21.15.98:0/819659645 learned_addr learned my addr 172.21.15.98:0/819659645 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:00:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef259c640 1 --2- 172.21.15.98:0/819659645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faef41002f0 0x7faef41a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef1d9b640 1 -- 172.21.15.98:0/819659645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faef4106f50 msgr2=0x7faef419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef1d9b640 1 --2- 172.21.15.98:0/819659645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faef4106f50 0x7faef419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef1d9b640 1 -- 172.21.15.98:0/819659645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faef41002f0 msgr2=0x7faef41a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef1d9b640 1 --2- 172.21.15.98:0/819659645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faef41002f0 0x7faef41a0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef1d9b640 1 -- 172.21.15.98:0/819659645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef419aea0 con 0x7faef4100ca0 2024-09-19T08:00:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef1d9b640 1 --2- 172.21.15.98:0/819659645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef4100ca0 0x7faef41a0a50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7faee800b450 tx=0x7faee800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faee37fe640 1 -- 172.21.15.98:0/819659645 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee8015020 con 0x7faef4100ca0 2024-09-19T08:00:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef8ec8640 1 -- 172.21.15.98:0/819659645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef419b190 con 0x7faef4100ca0 2024-09-19T08:00:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faee37fe640 1 -- 172.21.15.98:0/819659645 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faee8011d70 con 0x7faef4100ca0 2024-09-19T08:00:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faee37fe640 1 -- 172.21.15.98:0/819659645 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee800f340 con 0x7faef4100ca0 2024-09-19T08:00:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.280+0000 7faef8ec8640 1 -- 172.21.15.98:0/819659645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef41a7500 con 0x7faef4100ca0 2024-09-19T08:00:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.281+0000 7faef8ec8640 1 -- 172.21.15.98:0/819659645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faef4100750 con 0x7faef4100ca0 2024-09-19T08:00:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.282+0000 7faee37fe640 1 -- 172.21.15.98:0/819659645 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7faee8018020 con 0x7faef4100ca0 2024-09-19T08:00:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.283+0000 7faee37fe640 1 --2- 172.21.15.98:0/819659645 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7faedc0a3240 0x7faedc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.283+0000 7faef259c640 1 --2- 172.21.15.98:0/819659645 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7faedc0a3240 0x7faedc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.283+0000 7faee37fe640 1 -- 172.21.15.98:0/819659645 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7faee80cf4e0 con 0x7faef4100ca0 2024-09-19T08:00:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.283+0000 7faef259c640 1 --2- 172.21.15.98:0/819659645 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7faedc0a3240 0x7faedc0a5700 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faee4009890 tx=0x7faee4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.286+0000 7faee37fe640 1 -- 172.21.15.98:0/819659645 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7faee801f020 con 0x7faef4100ca0 2024-09-19T08:00:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.502+0000 7faef8ec8640 1 -- 172.21.15.98:0/819659645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faef419bb80 con 0x7faef4100ca0 2024-09-19T08:00:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.503+0000 7faee37fe640 1 -- 172.21.15.98:0/819659645 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2266 (secure 0 0 0) 0x7faee800d040 con 0x7faef4100ca0 2024-09-19T08:00:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:00:36.504 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"btime":"2024-09-19T08:00:33:312950+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-19T08:00:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 3 2024-09-19T08:00:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.505+0000 7faef8ec8640 1 -- 172.21.15.98:0/819659645 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7faedc0a3240 msgr2=0x7faedc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.505+0000 7faef8ec8640 1 --2- 172.21.15.98:0/819659645 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7faedc0a3240 0x7faedc0a5700 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faee4009890 tx=0x7faee4008040 comp rx=0 tx=0).stop 2024-09-19T08:00:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.505+0000 7faef8ec8640 1 -- 172.21.15.98:0/819659645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef4100ca0 msgr2=0x7faef41a0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.505+0000 7faef8ec8640 1 --2- 172.21.15.98:0/819659645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef4100ca0 0x7faef41a0a50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7faee800b450 tx=0x7faee800b910 comp rx=0 tx=0).stop 2024-09-19T08:00:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.505+0000 7faef8ec8640 1 -- 172.21.15.98:0/819659645 shutdown_connections 2024-09-19T08:00:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.505+0000 7faef8ec8640 1 --2- 172.21.15.98:0/819659645 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7faedc0a3240 0x7faedc0a5700 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.505+0000 7faef8ec8640 1 --2- 172.21.15.98:0/819659645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faef4106f50 0x7faef419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.505+0000 7faef8ec8640 1 --2- 172.21.15.98:0/819659645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef4100ca0 0x7faef41a0a50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.505+0000 7faef8ec8640 1 --2- 172.21.15.98:0/819659645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faef41002f0 0x7faef41a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.505+0000 7faef8ec8640 1 -- 172.21.15.98:0/819659645 >> 172.21.15.98:0/819659645 conn(0x7faef4077210 msgr2=0x7faef4105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.505+0000 7faef8ec8640 1 -- 172.21.15.98:0/819659645 shutdown_connections 2024-09-19T08:00:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:36.505+0000 7faef8ec8640 1 -- 172.21.15.98:0/819659645 wait complete. 2024-09-19T08:00:36.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:36 smithi162 ceph-mon[35260]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:36.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:36 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:36.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:36 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:36.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:36 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:36.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:36 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:36.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:36 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:36.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:36 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:36.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:36 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/819659645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:36.887 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:36 smithi098 ceph-mon[31333]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:36.887 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:36.887 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:36.887 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:36.887 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:36.887 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:36.887 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:36 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:36.887 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:36 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/819659645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:36.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:36 smithi139 ceph-mon[34874]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:36.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:36 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:36.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:36 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:36.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:36 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:36.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:36 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:36.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:36 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:36.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:36 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:36.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:36 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/819659645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:37.674 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:37 smithi098 ceph-mon[31333]: Deploying daemon mds.d on smithi098 2024-09-19T08:00:37.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:37 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.98:6834/85442870,v1:172.21.15.98:6835/85442870] up:boot 2024-09-19T08:00:37.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:37 smithi098 ceph-mon[31333]: fsmap 3 up:standby 2024-09-19T08:00:37.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:37 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-19T08:00:37.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:37 smithi162 ceph-mon[35260]: Deploying daemon mds.d on smithi098 2024-09-19T08:00:37.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:37 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.98:6834/85442870,v1:172.21.15.98:6835/85442870] up:boot 2024-09-19T08:00:37.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:37 smithi162 ceph-mon[35260]: fsmap 3 up:standby 2024-09-19T08:00:37.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:37 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-19T08:00:37.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:37 smithi139 ceph-mon[34874]: Deploying daemon mds.d on smithi098 2024-09-19T08:00:37.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:37 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.98:6834/85442870,v1:172.21.15.98:6835/85442870] up:boot 2024-09-19T08:00:37.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:37 smithi139 ceph-mon[34874]: fsmap 3 up:standby 2024-09-19T08:00:37.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:37 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-19T08:00:38.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:38 smithi098 ceph-mon[31333]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:38.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:38 smithi162 ceph-mon[35260]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:38.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:38 smithi139 ceph-mon[34874]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:39.042 INFO:journalctl@ceph.mds.d.smithi098.stdout:Sep 19 08:00:39 smithi098 podman[71153]: 2024-09-19 08:00:39.019113517 +0000 UTC m=+0.353698240 container start 0f7fe9582b6a761dbf7f0165c6ce9f01b7c49e2e96dd692aeceac7067cc566a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-2f599a4, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-19T08:00:39.432 INFO:journalctl@ceph.mds.d.smithi098.stdout:Sep 19 08:00:39 smithi098 ceph-mds[71173]: -- 172.21.15.98:0/429651152 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55706529f860 con 0x55706527c800 2024-09-19T08:00:39.508 INFO:journalctl@ceph.mds.d.smithi098.stdout:Sep 19 08:00:39 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-d[71169]: starting mds.d at 2024-09-19T08:00:39.508 INFO:journalctl@ceph.mds.d.smithi098.stdout:Sep 19 08:00:39 smithi098 bash[71153]: 0f7fe9582b6a761dbf7f0165c6ce9f01b7c49e2e96dd692aeceac7067cc566a0 2024-09-19T08:00:39.508 INFO:journalctl@ceph.mds.d.smithi098.stdout:Sep 19 08:00:39 smithi098 systemd[1]: Started Ceph mds.d for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:00:39.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:00:39.956 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:00:40.370 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:40.370 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:40.370 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:40.370 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:40.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:40.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:40.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:40 smithi098 ceph-mon[31333]: Deploying daemon mds.e on smithi139 2024-09-19T08:00:40.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:40 smithi098 ceph-mon[31333]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:40.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:40 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.98:6836/3958359300,v1:172.21.15.98:6837/3958359300] up:boot 2024-09-19T08:00:40.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:40 smithi098 ceph-mon[31333]: fsmap 4 up:standby 2024-09-19T08:00:40.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:40 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-19T08:00:40.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:40.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:40.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:40.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:40.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:40.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:40.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:40 smithi139 ceph-mon[34874]: Deploying daemon mds.e on smithi139 2024-09-19T08:00:40.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:40 smithi139 ceph-mon[34874]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:40.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:40 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.98:6836/3958359300,v1:172.21.15.98:6837/3958359300] up:boot 2024-09-19T08:00:40.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:40 smithi139 ceph-mon[34874]: fsmap 4 up:standby 2024-09-19T08:00:40.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:40 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-19T08:00:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.450+0000 7fcaa2518640 1 -- 172.21.15.98:0/898499703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca9c102d40 msgr2=0x7fca9c103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.450+0000 7fcaa2518640 1 --2- 172.21.15.98:0/898499703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca9c102d40 0x7fca9c103120 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fca8c009900 tx=0x7fca8c02a480 comp rx=0 tx=0).stop 2024-09-19T08:00:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.450+0000 7fcaa2518640 1 -- 172.21.15.98:0/898499703 shutdown_connections 2024-09-19T08:00:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.450+0000 7fcaa2518640 1 --2- 172.21.15.98:0/898499703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca9c10c170 0x7fca9c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.450+0000 7fcaa2518640 1 --2- 172.21.15.98:0/898499703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca9c103660 0x7fca9c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.450+0000 7fcaa2518640 1 --2- 172.21.15.98:0/898499703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca9c102d40 0x7fca9c103120 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.450+0000 7fcaa2518640 1 -- 172.21.15.98:0/898499703 >> 172.21.15.98:0/898499703 conn(0x7fca9c0fc7a0 msgr2=0x7fca9c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.450+0000 7fcaa2518640 1 -- 172.21.15.98:0/898499703 shutdown_connections 2024-09-19T08:00:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.450+0000 7fcaa2518640 1 -- 172.21.15.98:0/898499703 wait complete. 2024-09-19T08:00:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.451+0000 7fcaa2518640 1 Processor -- start 2024-09-19T08:00:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.451+0000 7fcaa2518640 1 -- start start 2024-09-19T08:00:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.451+0000 7fcaa2518640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca9c102d40 0x7fca9c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fcaa2518640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca9c103660 0x7fca9c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fcaa2518640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca9c10c170 0x7fca9c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fcaa2518640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c111fb0 con 0x7fca9c103660 2024-09-19T08:00:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fcaa2518640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c111e30 con 0x7fca9c10c170 2024-09-19T08:00:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fcaa2518640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c112130 con 0x7fca9c102d40 2024-09-19T08:00:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fca9b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca9c103660 0x7fca9c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fca9b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca9c103660 0x7fca9c1989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34900/0 (socket says 172.21.15.98:34900) 2024-09-19T08:00:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fca9b7fe640 1 -- 172.21.15.98:0/1410876248 learned_addr learned my addr 172.21.15.98:0/1410876248 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:00:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fca9bfff640 1 --2- 172.21.15.98:0/1410876248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca9c102d40 0x7fca9c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fcaa0ab5640 1 --2- 172.21.15.98:0/1410876248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca9c10c170 0x7fca9c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fca9b7fe640 1 -- 172.21.15.98:0/1410876248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca9c102d40 msgr2=0x7fca9c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fca9b7fe640 1 --2- 172.21.15.98:0/1410876248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca9c102d40 0x7fca9c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fca9b7fe640 1 -- 172.21.15.98:0/1410876248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca9c10c170 msgr2=0x7fca9c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fca9b7fe640 1 --2- 172.21.15.98:0/1410876248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca9c10c170 0x7fca9c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fca9b7fe640 1 -- 172.21.15.98:0/1410876248 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca9c19d440 con 0x7fca9c103660 2024-09-19T08:00:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fca9b7fe640 1 --2- 172.21.15.98:0/1410876248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca9c103660 0x7fca9c1989d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fca9000d950 tx=0x7fca9000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fca997fa640 1 -- 172.21.15.98:0/1410876248 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca90017030 con 0x7fca9c103660 2024-09-19T08:00:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fca997fa640 1 -- 172.21.15.98:0/1410876248 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca90004020 con 0x7fca9c103660 2024-09-19T08:00:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fcaa2518640 1 -- 172.21.15.98:0/1410876248 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca9c19d730 con 0x7fca9c103660 2024-09-19T08:00:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fca997fa640 1 -- 172.21.15.98:0/1410876248 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca900047a0 con 0x7fca9c103660 2024-09-19T08:00:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.452+0000 7fcaa2518640 1 -- 172.21.15.98:0/1410876248 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca9c1a5010 con 0x7fca9c103660 2024-09-19T08:00:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.453+0000 7fcaa2518640 1 -- 172.21.15.98:0/1410876248 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca9c103120 con 0x7fca9c103660 2024-09-19T08:00:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.454+0000 7fca997fa640 1 -- 172.21.15.98:0/1410876248 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fca900041d0 con 0x7fca9c103660 2024-09-19T08:00:40.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.455+0000 7fca997fa640 1 --2- 172.21.15.98:0/1410876248 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fca840a3240 0x7fca840a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:40.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.455+0000 7fca9bfff640 1 --2- 172.21.15.98:0/1410876248 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fca840a3240 0x7fca840a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:40.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.455+0000 7fca997fa640 1 -- 172.21.15.98:0/1410876248 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fca900cf380 con 0x7fca9c103660 2024-09-19T08:00:40.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.456+0000 7fca9bfff640 1 --2- 172.21.15.98:0/1410876248 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fca840a3240 0x7fca840a5700 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fca8c009230 tx=0x7fca8c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:40.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.459+0000 7fca997fa640 1 -- 172.21.15.98:0/1410876248 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fca9009c540 con 0x7fca9c103660 2024-09-19T08:00:40.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:40.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:40.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:40.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:40.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:40.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:40.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:40 smithi162 ceph-mon[35260]: Deploying daemon mds.e on smithi139 2024-09-19T08:00:40.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:40 smithi162 ceph-mon[35260]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:40.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:40 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.98:6836/3958359300,v1:172.21.15.98:6837/3958359300] up:boot 2024-09-19T08:00:40.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:40 smithi162 ceph-mon[35260]: fsmap 4 up:standby 2024-09-19T08:00:40.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:40 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-19T08:00:40.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.658+0000 7fcaa2518640 1 -- 172.21.15.98:0/1410876248 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca9c1997f0 con 0x7fca9c103660 2024-09-19T08:00:40.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.659+0000 7fca997fa640 1 -- 172.21.15.98:0/1410876248 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3910 (secure 0 0 0) 0x7fca9c1997f0 con 0x7fca9c103660 2024-09-19T08:00:40.660 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:00:40.660 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"btime":"2024-09-19T08:00:39:576915+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-19T08:00:40.661 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 5 2024-09-19T08:00:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.661+0000 7fcaa2518640 1 -- 172.21.15.98:0/1410876248 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fca840a3240 msgr2=0x7fca840a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.661+0000 7fcaa2518640 1 --2- 172.21.15.98:0/1410876248 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fca840a3240 0x7fca840a5700 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fca8c009230 tx=0x7fca8c0058d0 comp rx=0 tx=0).stop 2024-09-19T08:00:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.661+0000 7fcaa2518640 1 -- 172.21.15.98:0/1410876248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca9c103660 msgr2=0x7fca9c1989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.661+0000 7fcaa2518640 1 --2- 172.21.15.98:0/1410876248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca9c103660 0x7fca9c1989d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fca9000d950 tx=0x7fca9000de10 comp rx=0 tx=0).stop 2024-09-19T08:00:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.661+0000 7fcaa2518640 1 -- 172.21.15.98:0/1410876248 shutdown_connections 2024-09-19T08:00:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.661+0000 7fcaa2518640 1 --2- 172.21.15.98:0/1410876248 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fca840a3240 0x7fca840a5700 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.661+0000 7fcaa2518640 1 --2- 172.21.15.98:0/1410876248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca9c10c170 0x7fca9c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.661+0000 7fcaa2518640 1 --2- 172.21.15.98:0/1410876248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca9c103660 0x7fca9c1989d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.661+0000 7fcaa2518640 1 --2- 172.21.15.98:0/1410876248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca9c102d40 0x7fca9c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.661+0000 7fcaa2518640 1 -- 172.21.15.98:0/1410876248 >> 172.21.15.98:0/1410876248 conn(0x7fca9c0fc7a0 msgr2=0x7fca9c0fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.661+0000 7fcaa2518640 1 -- 172.21.15.98:0/1410876248 shutdown_connections 2024-09-19T08:00:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:40.661+0000 7fcaa2518640 1 -- 172.21.15.98:0/1410876248 wait complete. 2024-09-19T08:00:41.178 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:00:41 smithi139 systemd[1]: Starting Ceph mds.e for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:00:41.372 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:41 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1410876248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:41.515 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:41 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1410876248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:41.515 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:00:41 smithi139 podman[64559]: 2024-09-19 08:00:41.134482467 +0000 UTC m=+0.015895850 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:00:41.516 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:00:41 smithi139 podman[64559]: 2024-09-19 08:00:41.2628736 +0000 UTC m=+0.144287032 container create e6725063e11070cf28ffa827f927418fe602e251b6bb2d5158b424ddead3ec0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-e, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:00:41.516 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:00:41 smithi139 podman[64559]: 2024-09-19 08:00:41.514890282 +0000 UTC m=+0.396303666 container init e6725063e11070cf28ffa827f927418fe602e251b6bb2d5158b424ddead3ec0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux ) 2024-09-19T08:00:41.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:41 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1410876248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:41.927 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:00:41 smithi139 podman[64559]: 2024-09-19 08:00:41.517853163 +0000 UTC m=+0.399266547 container start e6725063e11070cf28ffa827f927418fe602e251b6bb2d5158b424ddead3ec0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-19T08:00:41.928 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:00:41 smithi139 ceph-mds[64578]: -- 172.21.15.139:0/3231549325 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5604f0351860 con 0x5604f032f000 2024-09-19T08:00:41.928 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:00:41 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-e[64574]: starting mds.e at 2024-09-19T08:00:41.928 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:00:41 smithi139 bash[64559]: e6725063e11070cf28ffa827f927418fe602e251b6bb2d5158b424ddead3ec0b 2024-09-19T08:00:41.928 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:00:41 smithi139 systemd[1]: Started Ceph mds.e for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:00:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:42 smithi162 ceph-mon[35260]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:42 smithi162 ceph-mon[35260]: Deploying daemon mds.f on smithi162 2024-09-19T08:00:42.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:42 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] up:boot 2024-09-19T08:00:42.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:42 smithi162 ceph-mon[35260]: fsmap 5 up:standby 2024-09-19T08:00:42.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:42 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-19T08:00:42.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:42 smithi139 ceph-mon[34874]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:42 smithi139 ceph-mon[34874]: Deploying daemon mds.f on smithi162 2024-09-19T08:00:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:42 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] up:boot 2024-09-19T08:00:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:42 smithi139 ceph-mon[34874]: fsmap 5 up:standby 2024-09-19T08:00:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:42 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-19T08:00:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:42 smithi098 ceph-mon[31333]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:42.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:42.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:42.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:42.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:42 smithi098 ceph-mon[31333]: Deploying daemon mds.f on smithi162 2024-09-19T08:00:42.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:42 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] up:boot 2024-09-19T08:00:42.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:42 smithi098 ceph-mon[31333]: fsmap 5 up:standby 2024-09-19T08:00:42.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:42 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-19T08:00:43.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:00:43.633 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:00:43.773 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:00:43 smithi162 systemd[1]: Starting Ceph mds.f for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:00:44.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:00:43 smithi162 podman[65279]: 2024-09-19 08:00:43.676009721 +0000 UTC m=+0.016228208 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:00:44.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:00:43 smithi162 podman[65279]: 2024-09-19 08:00:43.784753471 +0000 UTC m=+0.124971947 container create 8096b91bd9820b2ac1925d31a072992d2aadf4b2740a5f17e174531765473bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-f, RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-19T08:00:44.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:00:43 smithi162 podman[65279]: 2024-09-19 08:00:43.995560719 +0000 UTC m=+0.335779206 container init 8096b91bd9820b2ac1925d31a072992d2aadf4b2740a5f17e174531765473bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-19T08:00:44.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:00:43 smithi162 podman[65279]: 2024-09-19 08:00:43.998236627 +0000 UTC m=+0.338455096 container start 8096b91bd9820b2ac1925d31a072992d2aadf4b2740a5f17e174531765473bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-f, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-2f599a4, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-09-19T08:00:44.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:00:44 smithi162 ceph-mds[65298]: -- 172.21.15.162:0/4032858252 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559cc6561860 con 0x559cc653f000 2024-09-19T08:00:44.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:00:44 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-f[65294]: starting mds.f at 2024-09-19T08:00:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.065+0000 7f16a468a640 1 -- 172.21.15.98:0/1429640540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f169c102570 msgr2=0x7f169c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.065+0000 7f16a468a640 1 --2- 172.21.15.98:0/1429640540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f169c102570 0x7f169c1029b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f168c009900 tx=0x7f168c02a480 comp rx=0 tx=0).stop 2024-09-19T08:00:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.065+0000 7f16a468a640 1 -- 172.21.15.98:0/1429640540 shutdown_connections 2024-09-19T08:00:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.065+0000 7f16a468a640 1 --2- 172.21.15.98:0/1429640540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f169c102ef0 0x7f169c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.065+0000 7f16a468a640 1 --2- 172.21.15.98:0/1429640540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f169c102570 0x7f169c1029b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.065+0000 7f16a468a640 1 --2- 172.21.15.98:0/1429640540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f169c100e20 0x7f169c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.065+0000 7f16a468a640 1 -- 172.21.15.98:0/1429640540 >> 172.21.15.98:0/1429640540 conn(0x7f169c0fc7d0 msgr2=0x7f169c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.066+0000 7f16a468a640 1 -- 172.21.15.98:0/1429640540 shutdown_connections 2024-09-19T08:00:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.066+0000 7f16a468a640 1 -- 172.21.15.98:0/1429640540 wait complete. 2024-09-19T08:00:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.066+0000 7f16a468a640 1 Processor -- start 2024-09-19T08:00:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a468a640 1 -- start start 2024-09-19T08:00:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a468a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f169c100e20 0x7f169c0fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a468a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f169c102570 0x7f169c0fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a468a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f169c102ef0 0x7f169c0fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a468a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f169c110f80 con 0x7f169c100e20 2024-09-19T08:00:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a468a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f169c110e00 con 0x7f169c102ef0 2024-09-19T08:00:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a468a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f169c111100 con 0x7f169c102570 2024-09-19T08:00:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a2426640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f169c100e20 0x7f169c0fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a2426640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f169c100e20 0x7f169c0fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34920/0 (socket says 172.21.15.98:34920) 2024-09-19T08:00:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a2426640 1 -- 172.21.15.98:0/3254944095 learned_addr learned my addr 172.21.15.98:0/3254944095 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:00:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a1c25640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f169c102570 0x7f169c0fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a2c27640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f169c102ef0 0x7f169c0fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a2426640 1 -- 172.21.15.98:0/3254944095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f169c102570 msgr2=0x7f169c0fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a2426640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f169c102570 0x7f169c0fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a2426640 1 -- 172.21.15.98:0/3254944095 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f169c102ef0 msgr2=0x7f169c0fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a2426640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f169c102ef0 0x7f169c0fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a2426640 1 -- 172.21.15.98:0/3254944095 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c1a69a0 con 0x7f169c100e20 2024-09-19T08:00:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a1c25640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f169c102570 0x7f169c0fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:00:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16a2426640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f169c100e20 0x7f169c0fcc80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f169800d950 tx=0x7f169800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.067+0000 7f16937fe640 1 -- 172.21.15.98:0/3254944095 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1698017070 con 0x7f169c100e20 2024-09-19T08:00:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.068+0000 7f16937fe640 1 -- 172.21.15.98:0/3254944095 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1698009c30 con 0x7f169c100e20 2024-09-19T08:00:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.068+0000 7f16a468a640 1 -- 172.21.15.98:0/3254944095 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f169c1a6c90 con 0x7f169c100e20 2024-09-19T08:00:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.068+0000 7f16937fe640 1 -- 172.21.15.98:0/3254944095 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169800f6b0 con 0x7f169c100e20 2024-09-19T08:00:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.068+0000 7f16a468a640 1 -- 172.21.15.98:0/3254944095 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f169c1a71a0 con 0x7f169c100e20 2024-09-19T08:00:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.069+0000 7f16a468a640 1 -- 172.21.15.98:0/3254944095 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f169c101200 con 0x7f169c100e20 2024-09-19T08:00:44.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.070+0000 7f16937fe640 1 -- 172.21.15.98:0/3254944095 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f16980109d0 con 0x7f169c100e20 2024-09-19T08:00:44.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.071+0000 7f16937fe640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f16880a3240 0x7f16880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:44.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.071+0000 7f16a1c25640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f16880a3240 0x7f16880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:44.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.071+0000 7f16937fe640 1 -- 172.21.15.98:0/3254944095 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f16980cf010 con 0x7f169c100e20 2024-09-19T08:00:44.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.071+0000 7f16a1c25640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f16880a3240 0x7f16880a5700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f168c009230 tx=0x7f168c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:44.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.075+0000 7f16937fe640 1 -- 172.21.15.98:0/3254944095 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f1698009de0 con 0x7f169c100e20 2024-09-19T08:00:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.279+0000 7f16a468a640 1 -- 172.21.15.98:0/3254944095 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f169c02d7c0 con 0x7f169c100e20 2024-09-19T08:00:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.280+0000 7f16937fe640 1 -- 172.21.15.98:0/3254944095 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4738 (secure 0 0 0) 0x7f169809bb80 con 0x7f169c100e20 2024-09-19T08:00:44.280 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:00:44.280 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":6,"btime":"2024-09-19T08:00:42:179664+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-09-19T08:00:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 6 2024-09-19T08:00:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.282+0000 7f16a468a640 1 -- 172.21.15.98:0/3254944095 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f16880a3240 msgr2=0x7f16880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.282+0000 7f16a468a640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f16880a3240 0x7f16880a5700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f168c009230 tx=0x7f168c0058d0 comp rx=0 tx=0).stop 2024-09-19T08:00:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.282+0000 7f16a468a640 1 -- 172.21.15.98:0/3254944095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f169c100e20 msgr2=0x7f169c0fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.282+0000 7f16a468a640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f169c100e20 0x7f169c0fcc80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f169800d950 tx=0x7f169800de10 comp rx=0 tx=0).stop 2024-09-19T08:00:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.282+0000 7f16a468a640 1 -- 172.21.15.98:0/3254944095 shutdown_connections 2024-09-19T08:00:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.282+0000 7f16a468a640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f16880a3240 0x7f16880a5700 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.282+0000 7f16a468a640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f169c102ef0 0x7f169c0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.282+0000 7f16a468a640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f169c102570 0x7f169c0fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.282+0000 7f16a468a640 1 --2- 172.21.15.98:0/3254944095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f169c100e20 0x7f169c0fcc80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.282+0000 7f16a468a640 1 -- 172.21.15.98:0/3254944095 >> 172.21.15.98:0/3254944095 conn(0x7f169c0fc7d0 msgr2=0x7f169c06a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.282+0000 7f16a468a640 1 -- 172.21.15.98:0/3254944095 shutdown_connections 2024-09-19T08:00:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:44.282+0000 7f16a468a640 1 -- 172.21.15.98:0/3254944095 wait complete. 2024-09-19T08:00:44.370 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:00:44 smithi162 bash[65279]: 8096b91bd9820b2ac1925d31a072992d2aadf4b2740a5f17e174531765473bcb 2024-09-19T08:00:44.370 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:00:44 smithi162 systemd[1]: Started Ceph mds.f for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:00:44.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:44 smithi162 ceph-mon[35260]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:44.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:44 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:44.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:44 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:44.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:44 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:44.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:44 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:44.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:44 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:44.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:44 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:44.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:44 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3254944095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:44.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:44 smithi139 ceph-mon[34874]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:44.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:44 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:44.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:44 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:44.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:44 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:44.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:44 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:44.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:44 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:44.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:44 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:44.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:44 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3254944095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:44.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:44 smithi098 ceph-mon[31333]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:44 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:44 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:44 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:44 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:44 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:44 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:44.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:44 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3254944095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:45.740 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:45 smithi162 ceph-mon[35260]: Deploying daemon mds.c on smithi162 2024-09-19T08:00:45.740 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:45 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] up:boot 2024-09-19T08:00:45.740 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:45 smithi162 ceph-mon[35260]: fsmap 6 up:standby 2024-09-19T08:00:45.740 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:45 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-19T08:00:45.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:45 smithi139 ceph-mon[34874]: Deploying daemon mds.c on smithi162 2024-09-19T08:00:45.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:45 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] up:boot 2024-09-19T08:00:45.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:45 smithi139 ceph-mon[34874]: fsmap 6 up:standby 2024-09-19T08:00:45.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:45 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-19T08:00:45.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:45 smithi098 ceph-mon[31333]: Deploying daemon mds.c on smithi162 2024-09-19T08:00:45.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:45 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] up:boot 2024-09-19T08:00:45.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:45 smithi098 ceph-mon[31333]: fsmap 6 up:standby 2024-09-19T08:00:45.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:45 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-19T08:00:46.120 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:00:45 smithi162 systemd[1]: Starting Ceph mds.c for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:00:46.397 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:00:46 smithi162 podman[65583]: 2024-09-19 08:00:46.049386829 +0000 UTC m=+0.015857673 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:00:46.397 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:00:46 smithi162 podman[65583]: 2024-09-19 08:00:46.170738503 +0000 UTC m=+0.137209344 container create 60526bf80aba8bd8fd67c5f5e5cdaffedd38a1ffc2c8fe1839f38161091ede4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-19T08:00:46.397 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:00:46 smithi162 podman[65583]: 2024-09-19 08:00:46.397922079 +0000 UTC m=+0.364392921 container init 60526bf80aba8bd8fd67c5f5e5cdaffedd38a1ffc2c8fe1839f38161091ede4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:00:46.680 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:00:46 smithi162 podman[65583]: 2024-09-19 08:00:46.40068829 +0000 UTC m=+0.367159134 container start 60526bf80aba8bd8fd67c5f5e5cdaffedd38a1ffc2c8fe1839f38161091ede4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-c, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-19T08:00:46.681 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:00:46 smithi162 ceph-mds[65602]: -- 172.21.15.162:0/4005950118 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e116c9b860 con 0x55e116c79400 2024-09-19T08:00:46.681 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:00:46 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-c[65598]: starting mds.c at 2024-09-19T08:00:46.681 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:00:46 smithi162 bash[65583]: 60526bf80aba8bd8fd67c5f5e5cdaffedd38a1ffc2c8fe1839f38161091ede4d 2024-09-19T08:00:46.681 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:00:46 smithi162 systemd[1]: Started Ceph mds.c for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:00:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:46 smithi098 ceph-mon[31333]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:46 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:46.974 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:46 smithi139 ceph-mon[34874]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:46.974 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:46.974 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:46.974 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:46.974 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:46.974 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:46.974 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:46 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:47.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:00:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:46 smithi162 ceph-mon[35260]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:46 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:47.260 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:00:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.732+0000 7ff83b5d3640 1 -- 172.21.15.98:0/2745817975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8341036e0 msgr2=0x7ff834107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.732+0000 7ff83b5d3640 1 --2- 172.21.15.98:0/2745817975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8341036e0 0x7ff834107b90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff828009930 tx=0x7ff82802a420 comp rx=0 tx=0).stop 2024-09-19T08:00:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 -- 172.21.15.98:0/2745817975 shutdown_connections 2024-09-19T08:00:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 --2- 172.21.15.98:0/2745817975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8341082a0 0x7ff83410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 --2- 172.21.15.98:0/2745817975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8341036e0 0x7ff834107b90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 --2- 172.21.15.98:0/2745817975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff834102d30 0x7ff834103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 -- 172.21.15.98:0/2745817975 >> 172.21.15.98:0/2745817975 conn(0x7ff8340fc7d0 msgr2=0x7ff8340febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 -- 172.21.15.98:0/2745817975 shutdown_connections 2024-09-19T08:00:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 -- 172.21.15.98:0/2745817975 wait complete. 2024-09-19T08:00:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 Processor -- start 2024-09-19T08:00:47.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 -- start start 2024-09-19T08:00:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff834102d30 0x7ff8341985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8341036e0 0x7ff834198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8341082a0 0x7ff83419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8341120c0 con 0x7ff8341036e0 2024-09-19T08:00:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff834111f40 con 0x7ff8341082a0 2024-09-19T08:00:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.733+0000 7ff83b5d3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff834112240 con 0x7ff834102d30 2024-09-19T08:00:47.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff838b6e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8341036e0 0x7ff834198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:47.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff838b6e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8341036e0 0x7ff834198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34528/0 (socket says 172.21.15.98:34528) 2024-09-19T08:00:47.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff838b6e640 1 -- 172.21.15.98:0/906179540 learned_addr learned my addr 172.21.15.98:0/906179540 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:00:47.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff839b70640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8341082a0 0x7ff83419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:47.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff83936f640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff834102d30 0x7ff8341985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:47.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff838b6e640 1 -- 172.21.15.98:0/906179540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff834102d30 msgr2=0x7ff8341985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:47.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff838b6e640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff834102d30 0x7ff8341985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:47.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff838b6e640 1 -- 172.21.15.98:0/906179540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8341082a0 msgr2=0x7ff83419ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:47.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff838b6e640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8341082a0 0x7ff83419ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:47.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff838b6e640 1 -- 172.21.15.98:0/906179540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff83419d570 con 0x7ff8341036e0 2024-09-19T08:00:47.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff839b70640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8341082a0 0x7ff83419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:00:47.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff83936f640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff834102d30 0x7ff8341985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:00:47.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff838b6e640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8341036e0 0x7ff834198b00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff828005940 tx=0x7ff8280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:47.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff8327fc640 1 -- 172.21.15.98:0/906179540 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82803f030 con 0x7ff8341036e0 2024-09-19T08:00:47.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff83b5d3640 1 -- 172.21.15.98:0/906179540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff83419d800 con 0x7ff8341036e0 2024-09-19T08:00:47.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff8327fc640 1 -- 172.21.15.98:0/906179540 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff828003ce0 con 0x7ff8341036e0 2024-09-19T08:00:47.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff8327fc640 1 -- 172.21.15.98:0/906179540 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828035070 con 0x7ff8341036e0 2024-09-19T08:00:47.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.734+0000 7ff83b5d3640 1 -- 172.21.15.98:0/906179540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8341a5140 con 0x7ff8341036e0 2024-09-19T08:00:47.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.735+0000 7ff83b5d3640 1 -- 172.21.15.98:0/906179540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff83406a850 con 0x7ff8341036e0 2024-09-19T08:00:47.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.736+0000 7ff8327fc640 1 -- 172.21.15.98:0/906179540 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ff828042030 con 0x7ff8341036e0 2024-09-19T08:00:47.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.737+0000 7ff8327fc640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff81c0a3240 0x7ff81c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:47.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.737+0000 7ff83936f640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff81c0a3240 0x7ff81c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:47.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.737+0000 7ff8327fc640 1 -- 172.21.15.98:0/906179540 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7ff8280efae0 con 0x7ff8341036e0 2024-09-19T08:00:47.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.737+0000 7ff83936f640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff81c0a3240 0x7ff81c0a5700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff824008e00 tx=0x7ff824008350 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:47.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.741+0000 7ff8327fc640 1 -- 172.21.15.98:0/906179540 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ff8280bcca0 con 0x7ff8341036e0 2024-09-19T08:00:47.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:47 smithi098 ceph-mon[31333]: Deploying daemon mds.h on smithi139 2024-09-19T08:00:47.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:47 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.162:6836/3969500353,v1:172.21.15.162:6837/3969500353] up:boot 2024-09-19T08:00:47.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:47 smithi098 ceph-mon[31333]: fsmap 7 up:standby 2024-09-19T08:00:47.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:47 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-19T08:00:47.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.943+0000 7ff83b5d3640 1 -- 172.21.15.98:0/906179540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff83410a2b0 con 0x7ff8341036e0 2024-09-19T08:00:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.943+0000 7ff8327fc640 1 -- 172.21.15.98:0/906179540 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6394 (secure 0 0 0) 0x7ff8280bc5a0 con 0x7ff8341036e0 2024-09-19T08:00:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:00:47.944 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"btime":"2024-09-19T08:00:46:679996+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24406,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-09-19T08:00:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-09-19T08:00:47.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.945+0000 7ff83b5d3640 1 -- 172.21.15.98:0/906179540 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff81c0a3240 msgr2=0x7ff81c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.945+0000 7ff83b5d3640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff81c0a3240 0x7ff81c0a5700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff824008e00 tx=0x7ff824008350 comp rx=0 tx=0).stop 2024-09-19T08:00:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.945+0000 7ff83b5d3640 1 -- 172.21.15.98:0/906179540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8341036e0 msgr2=0x7ff834198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.945+0000 7ff83b5d3640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8341036e0 0x7ff834198b00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff828005940 tx=0x7ff8280042f0 comp rx=0 tx=0).stop 2024-09-19T08:00:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.945+0000 7ff83b5d3640 1 -- 172.21.15.98:0/906179540 shutdown_connections 2024-09-19T08:00:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.945+0000 7ff83b5d3640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff81c0a3240 0x7ff81c0a5700 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.945+0000 7ff83b5d3640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8341082a0 0x7ff83419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.945+0000 7ff83b5d3640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8341036e0 0x7ff834198b00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.945+0000 7ff83b5d3640 1 --2- 172.21.15.98:0/906179540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff834102d30 0x7ff8341985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.946+0000 7ff83b5d3640 1 -- 172.21.15.98:0/906179540 >> 172.21.15.98:0/906179540 conn(0x7ff8340fc7d0 msgr2=0x7ff834106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.946+0000 7ff83b5d3640 1 -- 172.21.15.98:0/906179540 shutdown_connections 2024-09-19T08:00:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:47.946+0000 7ff83b5d3640 1 -- 172.21.15.98:0/906179540 wait complete. 2024-09-19T08:00:48.061 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:47 smithi139 ceph-mon[34874]: Deploying daemon mds.h on smithi139 2024-09-19T08:00:48.062 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:47 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.162:6836/3969500353,v1:172.21.15.162:6837/3969500353] up:boot 2024-09-19T08:00:48.062 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:47 smithi139 ceph-mon[34874]: fsmap 7 up:standby 2024-09-19T08:00:48.062 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:47 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-19T08:00:48.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:47 smithi162 ceph-mon[35260]: Deploying daemon mds.h on smithi139 2024-09-19T08:00:48.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:47 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.162:6836/3969500353,v1:172.21.15.162:6837/3969500353] up:boot 2024-09-19T08:00:48.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:47 smithi162 ceph-mon[35260]: fsmap 7 up:standby 2024-09-19T08:00:48.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:47 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-19T08:00:48.427 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:00:48 smithi139 systemd[1]: Starting Ceph mds.h for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:00:48.767 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:48 smithi139 ceph-mon[34874]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:48.767 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:48 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/906179540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:48.768 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:00:48 smithi139 podman[64859]: 2024-09-19 08:00:48.405376859 +0000 UTC m=+0.015647142 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:00:48.768 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:00:48 smithi139 podman[64859]: 2024-09-19 08:00:48.531508338 +0000 UTC m=+0.141778608 container create 438b5988f0ec603879b00e688da8901fa5236e1900c18dcc85baeb628f83c774 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-19T08:00:48.768 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:00:48 smithi139 podman[64859]: 2024-09-19 08:00:48.742085004 +0000 UTC m=+0.352355273 container init 438b5988f0ec603879b00e688da8901fa5236e1900c18dcc85baeb628f83c774 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-h, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=squid-2f599a4, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-19T08:00:48.768 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:00:48 smithi139 podman[64859]: 2024-09-19 08:00:48.745164517 +0000 UTC m=+0.355434801 container start 438b5988f0ec603879b00e688da8901fa5236e1900c18dcc85baeb628f83c774 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-h, RELEASE=squid-2f599a4, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-19T08:00:49.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:48 smithi162 ceph-mon[35260]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:49.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:48 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/906179540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:49.177 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:00:48 smithi139 ceph-mds[64878]: -- 172.21.15.139:0/2715728933 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559eb3ec5860 con 0x559eb3ea3000 2024-09-19T08:00:49.177 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:00:48 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-h[64874]: starting mds.h at 2024-09-19T08:00:49.178 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:00:48 smithi139 bash[64859]: 438b5988f0ec603879b00e688da8901fa5236e1900c18dcc85baeb628f83c774 2024-09-19T08:00:49.178 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:00:48 smithi139 systemd[1]: Started Ceph mds.h for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:00:49.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:48 smithi098 ceph-mon[31333]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:49.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:48 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/906179540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:50.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:50.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:50.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:50.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:50.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:50.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:50.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:49 smithi139 ceph-mon[34874]: Deploying daemon mds.i on smithi162 2024-09-19T08:00:50.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:49 smithi139 ceph-mon[34874]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:50.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:49 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] up:boot 2024-09-19T08:00:50.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:49 smithi139 ceph-mon[34874]: fsmap 8 up:standby 2024-09-19T08:00:50.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:49 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-19T08:00:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:49 smithi098 ceph-mon[31333]: Deploying daemon mds.i on smithi162 2024-09-19T08:00:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:49 smithi098 ceph-mon[31333]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:49 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] up:boot 2024-09-19T08:00:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:49 smithi098 ceph-mon[31333]: fsmap 8 up:standby 2024-09-19T08:00:50.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:49 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-19T08:00:50.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:50.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:50.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:50.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:50.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:50.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:50.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:49 smithi162 ceph-mon[35260]: Deploying daemon mds.i on smithi162 2024-09-19T08:00:50.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:49 smithi162 ceph-mon[35260]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:50.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:49 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] up:boot 2024-09-19T08:00:50.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:49 smithi162 ceph-mon[35260]: fsmap 8 up:standby 2024-09-19T08:00:50.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:49 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-19T08:00:50.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:00:50.870 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:00:50 smithi162 systemd[1]: Starting Ceph mds.i for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:00:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:00:51.299 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:00:51 smithi162 podman[65886]: 2024-09-19 08:00:50.90736986 +0000 UTC m=+0.016481954 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:00:51.299 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:00:51 smithi162 podman[65886]: 2024-09-19 08:00:51.014473497 +0000 UTC m=+0.123585624 container create a360c89cc7ff298384ef210724b077d605bd068f9a73a12f8b3dd1f7882aa43f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-19T08:00:51.300 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:00:51 smithi162 podman[65886]: 2024-09-19 08:00:51.208600597 +0000 UTC m=+0.317712682 container init a360c89cc7ff298384ef210724b077d605bd068f9a73a12f8b3dd1f7882aa43f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-19T08:00:51.300 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:00:51 smithi162 podman[65886]: 2024-09-19 08:00:51.21144381 +0000 UTC m=+0.320555902 container start a360c89cc7ff298384ef210724b077d605bd068f9a73a12f8b3dd1f7882aa43f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-19T08:00:51.300 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:00:51 smithi162 ceph-mds[65905]: -- 172.21.15.162:0/3002003835 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e687ebd860 con 0x55e687e9a800 2024-09-19T08:00:51.300 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:00:51 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-i[65901]: starting mds.i at 2024-09-19T08:00:51.300 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:00:51 smithi162 bash[65886]: a360c89cc7ff298384ef210724b077d605bd068f9a73a12f8b3dd1f7882aa43f 2024-09-19T08:00:51.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.512+0000 7fdcbce07640 1 -- 172.21.15.98:0/3651481249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb81027a0 msgr2=0x7fdcb81067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:51.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.512+0000 7fdcbce07640 1 --2- 172.21.15.98:0/3651481249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb81027a0 0x7fdcb81067f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009900 tx=0x7fdcac02a410 comp rx=0 tx=0).stop 2024-09-19T08:00:51.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.512+0000 7fdcbce07640 1 -- 172.21.15.98:0/3651481249 shutdown_connections 2024-09-19T08:00:51.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.512+0000 7fdcbce07640 1 --2- 172.21.15.98:0/3651481249 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb8106f20 0x7fdcb810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:51.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.512+0000 7fdcbce07640 1 --2- 172.21.15.98:0/3651481249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb81027a0 0x7fdcb81067f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:51.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.512+0000 7fdcbce07640 1 --2- 172.21.15.98:0/3651481249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb8101df0 0x7fdcb81021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:51.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.512+0000 7fdcbce07640 1 -- 172.21.15.98:0/3651481249 >> 172.21.15.98:0/3651481249 conn(0x7fdcb80fd8f0 msgr2=0x7fdcb80ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:51.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.512+0000 7fdcbce07640 1 -- 172.21.15.98:0/3651481249 shutdown_connections 2024-09-19T08:00:51.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.512+0000 7fdcbce07640 1 -- 172.21.15.98:0/3651481249 wait complete. 2024-09-19T08:00:51.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.513+0000 7fdcbce07640 1 Processor -- start 2024-09-19T08:00:51.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.513+0000 7fdcbce07640 1 -- start start 2024-09-19T08:00:51.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcbce07640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb8101df0 0x7fdcb80744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:51.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcbce07640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb81027a0 0x7fdcb8074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:51.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcbce07640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb8106f20 0x7fdcb8074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:51.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcbce07640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb8111f40 con 0x7fdcb8106f20 2024-09-19T08:00:51.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcbce07640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb8111dc0 con 0x7fdcb8101df0 2024-09-19T08:00:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcbce07640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb81120c0 con 0x7fdcb81027a0 2024-09-19T08:00:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcb6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb8106f20 0x7fdcb8074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcb6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb8106f20 0x7fdcb8074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34538/0 (socket says 172.21.15.98:34538) 2024-09-19T08:00:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcb6d9d640 1 -- 172.21.15.98:0/3473224224 learned_addr learned my addr 172.21.15.98:0/3473224224 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:00:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcb659c640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb8101df0 0x7fdcb80744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcb5d9b640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb81027a0 0x7fdcb8074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcb6d9d640 1 -- 172.21.15.98:0/3473224224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb81027a0 msgr2=0x7fdcb8074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcb6d9d640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb81027a0 0x7fdcb8074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcb6d9d640 1 -- 172.21.15.98:0/3473224224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb8101df0 msgr2=0x7fdcb80744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcb6d9d640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb8101df0 0x7fdcb80744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcb6d9d640 1 -- 172.21.15.98:0/3473224224 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb81a6930 con 0x7fdcb8106f20 2024-09-19T08:00:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcb5d9b640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb81027a0 0x7fdcb8074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:00:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.514+0000 7fdcb6d9d640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb8106f20 0x7fdcb8074f70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdcb000eaa0 tx=0x7fdcb000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.515+0000 7fdca77fe640 1 -- 172.21.15.98:0/3473224224 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0004d20 con 0x7fdcb8106f20 2024-09-19T08:00:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.515+0000 7fdcbce07640 1 -- 172.21.15.98:0/3473224224 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb81a6c20 con 0x7fdcb8106f20 2024-09-19T08:00:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.515+0000 7fdca77fe640 1 -- 172.21.15.98:0/3473224224 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcb0005020 con 0x7fdcb8106f20 2024-09-19T08:00:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.515+0000 7fdca77fe640 1 -- 172.21.15.98:0/3473224224 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb00116d0 con 0x7fdcb8106f20 2024-09-19T08:00:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.515+0000 7fdcbce07640 1 -- 172.21.15.98:0/3473224224 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb81a7110 con 0x7fdcb8106f20 2024-09-19T08:00:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.516+0000 7fdcbce07640 1 -- 172.21.15.98:0/3473224224 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcb81021d0 con 0x7fdcb8106f20 2024-09-19T08:00:51.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.517+0000 7fdca77fe640 1 -- 172.21.15.98:0/3473224224 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fdcb00051d0 con 0x7fdcb8106f20 2024-09-19T08:00:51.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.518+0000 7fdca77fe640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdca00a3240 0x7fdca00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:51.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.518+0000 7fdcb659c640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdca00a3240 0x7fdca00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:51.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.518+0000 7fdca77fe640 1 -- 172.21.15.98:0/3473224224 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fdcb00cf600 con 0x7fdcb8106f20 2024-09-19T08:00:51.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.519+0000 7fdcb659c640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdca00a3240 0x7fdca00a5700 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdca800abe0 tx=0x7fdca800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:51.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.523+0000 7fdca77fe640 1 -- 172.21.15.98:0/3473224224 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fdcb0016030 con 0x7fdcb8106f20 2024-09-19T08:00:51.619 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:00:51 smithi162 systemd[1]: Started Ceph mds.i for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:00:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.748+0000 7fdcbce07640 1 -- 172.21.15.98:0/3473224224 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcb810a100 con 0x7fdcb8106f20 2024-09-19T08:00:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.748+0000 7fdca77fe640 1 -- 172.21.15.98:0/3473224224 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8052 (secure 0 0 0) 0x7fdcb009c7c0 con 0x7fdcb8106f20 2024-09-19T08:00:51.748 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:00:51.749 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"btime":"2024-09-19T08:00:51:290186+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24406,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24424,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-09-19T08:00:51.749 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 10 2024-09-19T08:00:51.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.750+0000 7fdcbce07640 1 -- 172.21.15.98:0/3473224224 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdca00a3240 msgr2=0x7fdca00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.750+0000 7fdcbce07640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdca00a3240 0x7fdca00a5700 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdca800abe0 tx=0x7fdca800a2d0 comp rx=0 tx=0).stop 2024-09-19T08:00:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.750+0000 7fdcbce07640 1 -- 172.21.15.98:0/3473224224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb8106f20 msgr2=0x7fdcb8074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.750+0000 7fdcbce07640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb8106f20 0x7fdcb8074f70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdcb000eaa0 tx=0x7fdcb000eea0 comp rx=0 tx=0).stop 2024-09-19T08:00:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.750+0000 7fdcbce07640 1 -- 172.21.15.98:0/3473224224 shutdown_connections 2024-09-19T08:00:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.750+0000 7fdcbce07640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdca00a3240 0x7fdca00a5700 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.750+0000 7fdcbce07640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb8106f20 0x7fdcb8074f70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.750+0000 7fdcbce07640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb81027a0 0x7fdcb8074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.750+0000 7fdcbce07640 1 --2- 172.21.15.98:0/3473224224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb8101df0 0x7fdcb80744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.750+0000 7fdcbce07640 1 -- 172.21.15.98:0/3473224224 >> 172.21.15.98:0/3473224224 conn(0x7fdcb80fd8f0 msgr2=0x7fdcb80feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.750+0000 7fdcbce07640 1 -- 172.21.15.98:0/3473224224 shutdown_connections 2024-09-19T08:00:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:51.751+0000 7fdcbce07640 1 -- 172.21.15.98:0/3473224224 wait complete. 2024-09-19T08:00:52.390 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:52 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.162:6838/3810709211,v1:172.21.15.162:6839/3810709211] up:boot 2024-09-19T08:00:52.390 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:52 smithi098 ceph-mon[31333]: fsmap 9 up:standby 2024-09-19T08:00:52.390 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-19T08:00:52.390 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:52.390 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:52.390 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:52.390 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:52.390 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:52.390 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:52 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:52.390 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:52 smithi098 ceph-mon[31333]: Deploying daemon mds.g on smithi098 2024-09-19T08:00:52.390 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:52 smithi098 ceph-mon[31333]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:52.391 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:52 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3473224224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:52.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:52 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.162:6838/3810709211,v1:172.21.15.162:6839/3810709211] up:boot 2024-09-19T08:00:52.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:52 smithi162 ceph-mon[35260]: fsmap 9 up:standby 2024-09-19T08:00:52.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-19T08:00:52.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:52.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:52.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:52.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:52.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:52.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:52 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:52.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:52 smithi162 ceph-mon[35260]: Deploying daemon mds.g on smithi098 2024-09-19T08:00:52.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:52 smithi162 ceph-mon[35260]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:52.621 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:52 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3473224224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:52.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:52 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.162:6838/3810709211,v1:172.21.15.162:6839/3810709211] up:boot 2024-09-19T08:00:52.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:52 smithi139 ceph-mon[34874]: fsmap 9 up:standby 2024-09-19T08:00:52.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-19T08:00:52.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:52.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:52.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:52.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:52.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:52.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:52 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:52.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:52 smithi139 ceph-mon[34874]: Deploying daemon mds.g on smithi098 2024-09-19T08:00:52.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:52 smithi139 ceph-mon[34874]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:52.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:52 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3473224224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:54.182 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:00:53 smithi098 systemd[1]: Starting Ceph mds.g for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:00:54.472 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:00:54 smithi098 podman[71930]: 2024-09-19 08:00:54.092612571 +0000 UTC m=+0.016213260 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:00:54.472 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:00:54 smithi098 podman[71930]: 2024-09-19 08:00:54.220888267 +0000 UTC m=+0.144488954 container create 8009a4d16a39985e3a960c3c44b632be07b633058d066ae3052e831972e0a836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-g, RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-19T08:00:54.472 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:00:54 smithi098 podman[71930]: 2024-09-19 08:00:54.472198856 +0000 UTC m=+0.395799550 container init 8009a4d16a39985e3a960c3c44b632be07b633058d066ae3052e831972e0a836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-g, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:00:54.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:54 smithi162 ceph-mon[35260]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:54.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:54 smithi139 ceph-mon[34874]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:54.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:54 smithi098 ceph-mon[31333]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:54.932 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:00:54 smithi098 podman[71930]: 2024-09-19 08:00:54.475553532 +0000 UTC m=+0.399154211 container start 8009a4d16a39985e3a960c3c44b632be07b633058d066ae3052e831972e0a836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-g, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4) 2024-09-19T08:00:54.932 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:00:54 smithi098 ceph-mds[71949]: -- 172.21.15.98:0/2484129803 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ce9bc97860 con 0x55ce9bc75400 2024-09-19T08:00:54.932 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:00:54 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-g[71945]: starting mds.g at 2024-09-19T08:00:54.932 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:00:54 smithi098 bash[71930]: 8009a4d16a39985e3a960c3c44b632be07b633058d066ae3052e831972e0a836 2024-09-19T08:00:54.932 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:00:54 smithi098 systemd[1]: Started Ceph mds.g for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:00:55.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:00:55.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:55 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.98:6838/3417988465,v1:172.21.15.98:6839/3417988465] up:boot 2024-09-19T08:00:55.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:55 smithi162 ceph-mon[35260]: fsmap 10 up:standby 2024-09-19T08:00:55.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-19T08:00:55.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:55.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:55.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:55.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:55.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:55.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:55 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:55.871 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:55 smithi162 ceph-mon[35260]: Deploying daemon mds.a on smithi098 2024-09-19T08:00:55.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:55 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.98:6838/3417988465,v1:172.21.15.98:6839/3417988465] up:boot 2024-09-19T08:00:55.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:55 smithi139 ceph-mon[34874]: fsmap 10 up:standby 2024-09-19T08:00:55.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-19T08:00:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:55 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:55.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:55 smithi139 ceph-mon[34874]: Deploying daemon mds.a on smithi098 2024-09-19T08:00:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:55 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.98:6838/3417988465,v1:172.21.15.98:6839/3417988465] up:boot 2024-09-19T08:00:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:55 smithi098 ceph-mon[31333]: fsmap 10 up:standby 2024-09-19T08:00:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-19T08:00:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:55 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:55.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:55 smithi098 ceph-mon[31333]: Deploying daemon mds.a on smithi098 2024-09-19T08:00:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:00:56.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.750+0000 7f6820952640 1 -- 172.21.15.98:0/3442564107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c101df0 msgr2=0x7f681c1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:56.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.750+0000 7f6820952640 1 --2- 172.21.15.98:0/3442564107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c101df0 0x7f681c1021d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6814009ca0 tx=0x7f681402a8c0 comp rx=0 tx=0).stop 2024-09-19T08:00:56.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.750+0000 7f6820952640 1 -- 172.21.15.98:0/3442564107 shutdown_connections 2024-09-19T08:00:56.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.750+0000 7f6820952640 1 --2- 172.21.15.98:0/3442564107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f681c106f20 0x7f681c10eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:56.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.750+0000 7f6820952640 1 --2- 172.21.15.98:0/3442564107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f681c1027a0 0x7f681c1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:56.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.750+0000 7f6820952640 1 --2- 172.21.15.98:0/3442564107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c101df0 0x7f681c1021d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:56.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.750+0000 7f6820952640 1 -- 172.21.15.98:0/3442564107 >> 172.21.15.98:0/3442564107 conn(0x7f681c0fd8f0 msgr2=0x7f681c0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:56.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.750+0000 7f6820952640 1 -- 172.21.15.98:0/3442564107 shutdown_connections 2024-09-19T08:00:56.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.750+0000 7f6820952640 1 -- 172.21.15.98:0/3442564107 wait complete. 2024-09-19T08:00:56.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6820952640 1 Processor -- start 2024-09-19T08:00:56.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6820952640 1 -- start start 2024-09-19T08:00:56.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6820952640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f681c101df0 0x7f681c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:56.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6820952640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f681c1027a0 0x7f681c074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:56.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6820952640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c106f20 0x7f681c074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:56.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6820952640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f681c112020 con 0x7f681c106f20 2024-09-19T08:00:56.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6820952640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f681c111ea0 con 0x7f681c1027a0 2024-09-19T08:00:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6820952640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f681c1121a0 con 0x7f681c101df0 2024-09-19T08:00:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f681ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c106f20 0x7f681c074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6819d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f681c1027a0 0x7f681c074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f681ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c106f20 0x7f681c074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45048/0 (socket says 172.21.15.98:45048) 2024-09-19T08:00:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f681ad9d640 1 -- 172.21.15.98:0/1483349893 learned_addr learned my addr 172.21.15.98:0/1483349893 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:00:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6819d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f681c1027a0 0x7f681c074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:55854/0 (socket says 172.21.15.98:55854) 2024-09-19T08:00:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f681a59c640 1 --2- 172.21.15.98:0/1483349893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f681c101df0 0x7f681c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6819d9b640 1 -- 172.21.15.98:0/1483349893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f681c101df0 msgr2=0x7f681c0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6819d9b640 1 --2- 172.21.15.98:0/1483349893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f681c101df0 0x7f681c0744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6819d9b640 1 -- 172.21.15.98:0/1483349893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c106f20 msgr2=0x7f681c074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6819d9b640 1 --2- 172.21.15.98:0/1483349893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c106f20 0x7f681c074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f6819d9b640 1 -- 172.21.15.98:0/1483349893 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681c1a6a80 con 0x7f681c1027a0 2024-09-19T08:00:56.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.751+0000 7f681ad9d640 1 --2- 172.21.15.98:0/1483349893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c106f20 0x7f681c074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:00:56.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.752+0000 7f6819d9b640 1 --2- 172.21.15.98:0/1483349893 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f681c1027a0 0x7f681c074a30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f680400b3f0 tx=0x7f680400b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:56.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.752+0000 7f68137fe640 1 -- 172.21.15.98:0/1483349893 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6804015020 con 0x7f681c1027a0 2024-09-19T08:00:56.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.752+0000 7f6820952640 1 -- 172.21.15.98:0/1483349893 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f681c1a6d70 con 0x7f681c1027a0 2024-09-19T08:00:56.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.752+0000 7f68137fe640 1 -- 172.21.15.98:0/1483349893 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6804011d10 con 0x7f681c1027a0 2024-09-19T08:00:56.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.752+0000 7f68137fe640 1 -- 172.21.15.98:0/1483349893 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680400f2e0 con 0x7f681c1027a0 2024-09-19T08:00:56.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.752+0000 7f6820952640 1 -- 172.21.15.98:0/1483349893 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f681c1a7250 con 0x7f681c1027a0 2024-09-19T08:00:56.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.753+0000 7f6820952640 1 -- 172.21.15.98:0/1483349893 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f681c102250 con 0x7f681c1027a0 2024-09-19T08:00:56.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.754+0000 7f68137fe640 1 -- 172.21.15.98:0/1483349893 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f6804018020 con 0x7f681c1027a0 2024-09-19T08:00:56.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.755+0000 7f68137fe640 1 --2- 172.21.15.98:0/1483349893 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f68080a3240 0x7f68080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:00:56.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.755+0000 7f68137fe640 1 -- 172.21.15.98:0/1483349893 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f68040cef70 con 0x7f681c1027a0 2024-09-19T08:00:56.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.755+0000 7f681a59c640 1 --2- 172.21.15.98:0/1483349893 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f68080a3240 0x7f68080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:00:56.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.756+0000 7f681a59c640 1 --2- 172.21.15.98:0/1483349893 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f68080a3240 0x7f68080a5700 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6814006010 tx=0x7f6814009900 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:00:56.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.758+0000 7f68137fe640 1 -- 172.21.15.98:0/1483349893 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f680401f020 con 0x7f681c1027a0 2024-09-19T08:00:56.908 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:56 smithi098 ceph-mon[31333]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:56.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:56 smithi139 ceph-mon[34874]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.969+0000 7f6820952640 1 -- 172.21.15.98:0/1483349893 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f681c075a20 con 0x7f681c1027a0 2024-09-19T08:00:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.970+0000 7f68137fe640 1 -- 172.21.15.98:0/1483349893 <== mon.2 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8878 (secure 0 0 0) 0x7f680409c130 con 0x7f681c1027a0 2024-09-19T08:00:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:00:56.971 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":11,"btime":"2024-09-19T08:00:54:598985+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24406,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24424,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-09-19T08:00:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 11 2024-09-19T08:00:56.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.973+0000 7f6820952640 1 -- 172.21.15.98:0/1483349893 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f68080a3240 msgr2=0x7f68080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:56.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.973+0000 7f6820952640 1 --2- 172.21.15.98:0/1483349893 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f68080a3240 0x7f68080a5700 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6814006010 tx=0x7f6814009900 comp rx=0 tx=0).stop 2024-09-19T08:00:56.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.973+0000 7f6820952640 1 -- 172.21.15.98:0/1483349893 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f681c1027a0 msgr2=0x7f681c074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:00:56.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.973+0000 7f6820952640 1 --2- 172.21.15.98:0/1483349893 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f681c1027a0 0x7f681c074a30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f680400b3f0 tx=0x7f680400b8b0 comp rx=0 tx=0).stop 2024-09-19T08:00:56.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.973+0000 7f6820952640 1 -- 172.21.15.98:0/1483349893 shutdown_connections 2024-09-19T08:00:56.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.973+0000 7f6820952640 1 --2- 172.21.15.98:0/1483349893 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f68080a3240 0x7f68080a5700 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:56.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.973+0000 7f6820952640 1 --2- 172.21.15.98:0/1483349893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f681c106f20 0x7f681c074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:56.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.973+0000 7f6820952640 1 --2- 172.21.15.98:0/1483349893 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f681c1027a0 0x7f681c074a30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:56.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.973+0000 7f6820952640 1 --2- 172.21.15.98:0/1483349893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f681c101df0 0x7f681c0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:00:56.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.973+0000 7f6820952640 1 -- 172.21.15.98:0/1483349893 >> 172.21.15.98:0/1483349893 conn(0x7f681c0fd8f0 msgr2=0x7f681c0fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:00:56.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.973+0000 7f6820952640 1 -- 172.21.15.98:0/1483349893 shutdown_connections 2024-09-19T08:00:56.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:00:56.973+0000 7f6820952640 1 -- 172.21.15.98:0/1483349893 wait complete. 2024-09-19T08:00:57.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:56 smithi162 ceph-mon[35260]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:57.886 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:57 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1483349893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:57.886 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:00:57 smithi098 podman[72325]: 2024-09-19 08:00:57.863647752 +0000 UTC m=+1.068257309 container start 760d32eec551ea053ddf0f3ec586a1b0e1330a94bcc58594cebc23faeb0f69cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2) 2024-09-19T08:00:57.887 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:00:57 smithi098 ceph-mds[72361]: -- 172.21.15.98:0/2845206333 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562bf6d67860 con 0x562bf6d45400 2024-09-19T08:00:57.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:57 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1483349893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:58.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:57 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1483349893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:00:58.182 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:00:57 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-a[72355]: starting mds.a at 2024-09-19T08:00:58.182 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:00:57 smithi098 bash[72325]: 760d32eec551ea053ddf0f3ec586a1b0e1330a94bcc58594cebc23faeb0f69cb 2024-09-19T08:00:58.182 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:00:57 smithi098 systemd[1]: Started Ceph mds.a for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:00:58.752 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:58 smithi139 ceph-mon[34874]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:58.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:58 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:58.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:58 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:58.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:58 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:58.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:58 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:58.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:58 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:58.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:58 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:58 smithi098 ceph-mon[31333]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:58.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:58 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:58.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:58 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:58.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:58 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:58.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:58 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:58.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:58 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:58.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:58 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:59.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:58 smithi162 ceph-mon[35260]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:00:59.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:59.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:59.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:00:59.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T08:00:59.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T08:00:59.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:58 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:59.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:00:59.747 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:59 smithi139 ceph-mon[34874]: Deploying daemon mds.b on smithi139 2024-09-19T08:00:59.748 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:59 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.98:6840/1339458935,v1:172.21.15.98:6841/1339458935] up:boot 2024-09-19T08:00:59.748 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:59 smithi139 ceph-mon[34874]: fsmap 11 up:standby 2024-09-19T08:00:59.748 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:00:59 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-19T08:00:59.914 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:00:59.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:59 smithi098 ceph-mon[31333]: Deploying daemon mds.b on smithi139 2024-09-19T08:00:59.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:59 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.98:6840/1339458935,v1:172.21.15.98:6841/1339458935] up:boot 2024-09-19T08:00:59.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:59 smithi098 ceph-mon[31333]: fsmap 11 up:standby 2024-09-19T08:00:59.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:00:59 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-19T08:01:00.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:59 smithi162 ceph-mon[35260]: Deploying daemon mds.b on smithi139 2024-09-19T08:01:00.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:59 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.98:6840/1339458935,v1:172.21.15.98:6841/1339458935] up:boot 2024-09-19T08:01:00.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:59 smithi162 ceph-mon[35260]: fsmap 11 up:standby 2024-09-19T08:01:00.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:00:59 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-19T08:01:00.142 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:00:59 smithi139 systemd[1]: Starting Ceph mds.b for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:01:00.427 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:01:00 smithi139 podman[65154]: 2024-09-19 08:01:00.142411678 +0000 UTC m=+0.109885523 container create b05a28ba4813690ab7776b7d62c91b9db00a6b4e867c624fb80e4c7b65d85895 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-19T08:01:00.427 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:01:00 smithi139 podman[65154]: 2024-09-19 08:01:00.04836976 +0000 UTC m=+0.015843553 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:01:00.427 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:01:00 smithi139 podman[65154]: 2024-09-19 08:01:00.361473953 +0000 UTC m=+0.328947745 container init b05a28ba4813690ab7776b7d62c91b9db00a6b4e867c624fb80e4c7b65d85895 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-b, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:01:00.428 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:01:00 smithi139 podman[65154]: 2024-09-19 08:01:00.364219001 +0000 UTC m=+0.331692787 container start b05a28ba4813690ab7776b7d62c91b9db00a6b4e867c624fb80e4c7b65d85895 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-b, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-19T08:01:00.428 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:01:00 smithi139 ceph-mds[65173]: -- 172.21.15.139:0/3922528133 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56546f427860 con 0x56546f405000 2024-09-19T08:01:00.428 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:01:00 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-b[65169]: starting mds.b at 2024-09-19T08:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.427+0000 7f72414d6640 1 -- 172.21.15.98:0/1747291110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f723c106f00 msgr2=0x7f723c10ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.427+0000 7f72414d6640 1 --2- 172.21.15.98:0/1747291110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f723c106f00 0x7f723c10ea80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7234009900 tx=0x7f723402a410 comp rx=0 tx=0).stop 2024-09-19T08:01:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.427+0000 7f72414d6640 1 -- 172.21.15.98:0/1747291110 shutdown_connections 2024-09-19T08:01:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.427+0000 7f72414d6640 1 --2- 172.21.15.98:0/1747291110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f723c106f00 0x7f723c10ea80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.427+0000 7f72414d6640 1 --2- 172.21.15.98:0/1747291110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f723c102780 0x7f723c1067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.427+0000 7f72414d6640 1 --2- 172.21.15.98:0/1747291110 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f723c101dd0 0x7f723c1021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.427+0000 7f72414d6640 1 -- 172.21.15.98:0/1747291110 >> 172.21.15.98:0/1747291110 conn(0x7f723c0fd8b0 msgr2=0x7f723c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.427+0000 7f72414d6640 1 -- 172.21.15.98:0/1747291110 shutdown_connections 2024-09-19T08:01:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.427+0000 7f72414d6640 1 -- 172.21.15.98:0/1747291110 wait complete. 2024-09-19T08:01:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.427+0000 7f72414d6640 1 Processor -- start 2024-09-19T08:01:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f72414d6640 1 -- start start 2024-09-19T08:01:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f72414d6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f723c101dd0 0x7f723c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f72414d6640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f723c102780 0x7f723c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f72414d6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f723c106f00 0x7f723c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f72414d6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f723c111fb0 con 0x7f723c106f00 2024-09-19T08:01:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f72414d6640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f723c111e30 con 0x7f723c102780 2024-09-19T08:01:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f72414d6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f723c112130 con 0x7f723c101dd0 2024-09-19T08:01:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f723b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f723c106f00 0x7f723c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f723b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f723c106f00 0x7f723c19cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45086/0 (socket says 172.21.15.98:45086) 2024-09-19T08:01:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f723b7fe640 1 -- 172.21.15.98:0/1730844796 learned_addr learned my addr 172.21.15.98:0/1730844796 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f723a7fc640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f723c102780 0x7f723c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f723affd640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f723c101dd0 0x7f723c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f723b7fe640 1 -- 172.21.15.98:0/1730844796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f723c101dd0 msgr2=0x7f723c198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f723b7fe640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f723c101dd0 0x7f723c198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f723b7fe640 1 -- 172.21.15.98:0/1730844796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f723c102780 msgr2=0x7f723c198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f723b7fe640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f723c102780 0x7f723c198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f723b7fe640 1 -- 172.21.15.98:0/1730844796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f723c19d400 con 0x7f723c106f00 2024-09-19T08:01:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f723affd640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f723c101dd0 0x7f723c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:01:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.428+0000 7f723b7fe640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f723c106f00 0x7f723c19cd20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7234004770 tx=0x7f7234003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.429+0000 7f7223fff640 1 -- 172.21.15.98:0/1730844796 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723403f070 con 0x7f723c106f00 2024-09-19T08:01:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.429+0000 7f72414d6640 1 -- 172.21.15.98:0/1730844796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f723c19d690 con 0x7f723c106f00 2024-09-19T08:01:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.429+0000 7f7223fff640 1 -- 172.21.15.98:0/1730844796 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7234031c90 con 0x7f723c106f00 2024-09-19T08:01:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.429+0000 7f7223fff640 1 -- 172.21.15.98:0/1730844796 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7234035030 con 0x7f723c106f00 2024-09-19T08:01:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.429+0000 7f72414d6640 1 -- 172.21.15.98:0/1730844796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f723c1a4fd0 con 0x7f723c106f00 2024-09-19T08:01:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.430+0000 7f72414d6640 1 -- 172.21.15.98:0/1730844796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f723c1021b0 con 0x7f723c106f00 2024-09-19T08:01:00.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.431+0000 7f7223fff640 1 -- 172.21.15.98:0/1730844796 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f7234042070 con 0x7f723c106f00 2024-09-19T08:01:00.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.431+0000 7f7223fff640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f72240a3240 0x7f72240a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:00.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.432+0000 7f723affd640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f72240a3240 0x7f72240a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:00.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.432+0000 7f7223fff640 1 -- 172.21.15.98:0/1730844796 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f72340f0170 con 0x7f723c106f00 2024-09-19T08:01:00.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.432+0000 7f723affd640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f72240a3240 0x7f72240a5700 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f722c005fd0 tx=0x7f722c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:00.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.435+0000 7f7223fff640 1 -- 172.21.15.98:0/1730844796 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f72340bd310 con 0x7f723c106f00 2024-09-19T08:01:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.644+0000 7f72414d6640 1 -- 172.21.15.98:0/1730844796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f723c1997b0 con 0x7f723c106f00 2024-09-19T08:01:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.645+0000 7f7223fff640 1 -- 172.21.15.98:0/1730844796 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9704 (secure 0 0 0) 0x7f723c1997b0 con 0x7f723c106f00 2024-09-19T08:01:00.645 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:01:00.645 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":12,"btime":"2024-09-19T08:00:58:632041+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24406,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24424,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-09-19T08:01:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 12 2024-09-19T08:01:00.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.647+0000 7f72414d6640 1 -- 172.21.15.98:0/1730844796 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f72240a3240 msgr2=0x7f72240a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:00.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.647+0000 7f72414d6640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f72240a3240 0x7f72240a5700 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f722c005fd0 tx=0x7f722c009500 comp rx=0 tx=0).stop 2024-09-19T08:01:00.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.647+0000 7f72414d6640 1 -- 172.21.15.98:0/1730844796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f723c106f00 msgr2=0x7f723c19cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:00.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.647+0000 7f72414d6640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f723c106f00 0x7f723c19cd20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7234004770 tx=0x7f7234003b90 comp rx=0 tx=0).stop 2024-09-19T08:01:00.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.647+0000 7f72414d6640 1 -- 172.21.15.98:0/1730844796 shutdown_connections 2024-09-19T08:01:00.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.647+0000 7f72414d6640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f72240a3240 0x7f72240a5700 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:00.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.647+0000 7f72414d6640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f723c106f00 0x7f723c19cd20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:00.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.647+0000 7f72414d6640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f723c102780 0x7f723c198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:00.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.647+0000 7f72414d6640 1 --2- 172.21.15.98:0/1730844796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f723c101dd0 0x7f723c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:00.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.647+0000 7f72414d6640 1 -- 172.21.15.98:0/1730844796 >> 172.21.15.98:0/1730844796 conn(0x7f723c0fd8b0 msgr2=0x7f723c0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:00.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.647+0000 7f72414d6640 1 -- 172.21.15.98:0/1730844796 shutdown_connections 2024-09-19T08:01:00.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:00.647+0000 7f72414d6640 1 -- 172.21.15.98:0/1730844796 wait complete. 2024-09-19T08:01:00.927 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:01:00 smithi139 bash[65154]: b05a28ba4813690ab7776b7d62c91b9db00a6b4e867c624fb80e4c7b65d85895 2024-09-19T08:01:00.927 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:01:00 smithi139 systemd[1]: Started Ceph mds.b for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:01:00.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:00 smithi139 ceph-mon[34874]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:00.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:00 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:00.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:00 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:00.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:00 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:00.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:00 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:00.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:00 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:00.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:00 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:00.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:00 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:01:00.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:00 smithi098 ceph-mon[31333]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:00.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:00 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:00.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:00 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:00.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:00 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:00.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:00 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:00.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:00 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:00.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:00 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:00.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:00 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:01:01.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:00 smithi162 ceph-mon[35260]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:01.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:00 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:01.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:00 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:01.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:00 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:01.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:00 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:01.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:00 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:01.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:00 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:01.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:00 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:01:01.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:01 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1730844796' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:01:01.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:01 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] up:boot 2024-09-19T08:01:01.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:01 smithi139 ceph-mon[34874]: fsmap 12 up:standby 2024-09-19T08:01:01.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:01 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-19T08:01:01.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:01 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1730844796' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:01:01.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:01 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] up:boot 2024-09-19T08:01:01.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:01 smithi098 ceph-mon[31333]: fsmap 12 up:standby 2024-09-19T08:01:01.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:01 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-19T08:01:02.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:01 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1730844796' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:01:02.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:01 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] up:boot 2024-09-19T08:01:02.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:01 smithi162 ceph-mon[35260]: fsmap 12 up:standby 2024-09-19T08:01:02.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:01 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-19T08:01:02.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:02 smithi139 ceph-mon[34874]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:02.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:02.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:02.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:02.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:02.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:02.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:02.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:01:02.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:01:02.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:02.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:02 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:01:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:02 smithi098 ceph-mon[31333]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:02.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:02.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:01:02.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:01:02.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:02.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:02 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:01:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:02 smithi162 ceph-mon[35260]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:01:03.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:01:03.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:03.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:02 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:01:03.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:01:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:04.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.176+0000 7f9fee6f2640 1 -- 172.21.15.98:0/1832268513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8102ef0 msgr2=0x7f9fe810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:04.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.176+0000 7f9fee6f2640 1 --2- 172.21.15.98:0/1832268513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8102ef0 0x7f9fe810da40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0009ca0 tx=0x7f9fe002a950 comp rx=0 tx=0).stop 2024-09-19T08:01:04.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.176+0000 7f9fee6f2640 1 -- 172.21.15.98:0/1832268513 shutdown_connections 2024-09-19T08:01:04.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.176+0000 7f9fee6f2640 1 --2- 172.21.15.98:0/1832268513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8102ef0 0x7f9fe810da40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:04.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.176+0000 7f9fee6f2640 1 --2- 172.21.15.98:0/1832268513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9fe8102570 0x7f9fe81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.176+0000 7f9fee6f2640 1 --2- 172.21.15.98:0/1832268513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fe8100e20 0x7f9fe8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.176+0000 7f9fee6f2640 1 -- 172.21.15.98:0/1832268513 >> 172.21.15.98:0/1832268513 conn(0x7f9fe80fc7d0 msgr2=0x7f9fe80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.176+0000 7f9fee6f2640 1 -- 172.21.15.98:0/1832268513 shutdown_connections 2024-09-19T08:01:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.176+0000 7f9fee6f2640 1 -- 172.21.15.98:0/1832268513 wait complete. 2024-09-19T08:01:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.176+0000 7f9fee6f2640 1 Processor -- start 2024-09-19T08:01:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fee6f2640 1 -- start start 2024-09-19T08:01:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fee6f2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9fe8100e20 0x7f9fe80fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fee6f2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fe8102570 0x7f9fe80fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fee6f2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8102ef0 0x7f9fe80fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fee6f2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe8110f80 con 0x7f9fe8102ef0 2024-09-19T08:01:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fee6f2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe8110e00 con 0x7f9fe8100e20 2024-09-19T08:01:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fee6f2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe8111100 con 0x7f9fe8102570 2024-09-19T08:01:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fecc8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8102ef0 0x7f9fe80fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fecc8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8102ef0 0x7f9fe80fd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45096/0 (socket says 172.21.15.98:45096) 2024-09-19T08:01:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fecc8f640 1 -- 172.21.15.98:0/3343903065 learned_addr learned my addr 172.21.15.98:0/3343903065 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fe7fff640 1 --2- 172.21.15.98:0/3343903065 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9fe8100e20 0x7f9fe80fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fe77fe640 1 --2- 172.21.15.98:0/3343903065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fe8102570 0x7f9fe80fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fecc8f640 1 -- 172.21.15.98:0/3343903065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fe8102570 msgr2=0x7f9fe80fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fecc8f640 1 --2- 172.21.15.98:0/3343903065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fe8102570 0x7f9fe80fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fecc8f640 1 -- 172.21.15.98:0/3343903065 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9fe8100e20 msgr2=0x7f9fe80fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fecc8f640 1 --2- 172.21.15.98:0/3343903065 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9fe8100e20 0x7f9fe80fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fecc8f640 1 -- 172.21.15.98:0/3343903065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe81a6950 con 0x7f9fe8102ef0 2024-09-19T08:01:04.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.177+0000 7f9fecc8f640 1 --2- 172.21.15.98:0/3343903065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8102ef0 0x7f9fe80fd700 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9fe002ae70 tx=0x7f9fe002f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:04.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.178+0000 7f9fe57fa640 1 -- 172.21.15.98:0/3343903065 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0034040 con 0x7f9fe8102ef0 2024-09-19T08:01:04.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.178+0000 7f9fee6f2640 1 -- 172.21.15.98:0/3343903065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe81a6be0 con 0x7f9fe8102ef0 2024-09-19T08:01:04.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.178+0000 7f9fe57fa640 1 -- 172.21.15.98:0/3343903065 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe0030920 con 0x7f9fe8102ef0 2024-09-19T08:01:04.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.178+0000 7f9fe57fa640 1 -- 172.21.15.98:0/3343903065 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0045480 con 0x7f9fe8102ef0 2024-09-19T08:01:04.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.178+0000 7f9fee6f2640 1 -- 172.21.15.98:0/3343903065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe81a7110 con 0x7f9fe8102ef0 2024-09-19T08:01:04.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.179+0000 7f9fee6f2640 1 -- 172.21.15.98:0/3343903065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe8101200 con 0x7f9fe8102ef0 2024-09-19T08:01:04.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.180+0000 7f9fe57fa640 1 -- 172.21.15.98:0/3343903065 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f9fe0037020 con 0x7f9fe8102ef0 2024-09-19T08:01:04.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.180+0000 7f9fe57fa640 1 --2- 172.21.15.98:0/3343903065 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f9fd00a3240 0x7f9fd00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:04.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.181+0000 7f9fe7fff640 1 --2- 172.21.15.98:0/3343903065 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f9fd00a3240 0x7f9fd00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:04.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.181+0000 7f9fe57fa640 1 -- 172.21.15.98:0/3343903065 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f9fe00f0370 con 0x7f9fe8102ef0 2024-09-19T08:01:04.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.181+0000 7f9fe7fff640 1 --2- 172.21.15.98:0/3343903065 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f9fd00a3240 0x7f9fd00a5700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8005fd0 tx=0x7f9fd800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.185+0000 7f9fe57fa640 1 -- 172.21.15.98:0/3343903065 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f9fe00bd490 con 0x7f9fe8102ef0 2024-09-19T08:01:04.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.401+0000 7f9fee6f2640 1 -- 172.21.15.98:0/3343903065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fe80fe1b0 con 0x7f9fe8102ef0 2024-09-19T08:01:04.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.402+0000 7f9fe57fa640 1 -- 172.21.15.98:0/3343903065 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10533 (secure 0 0 0) 0x7f9fe80fe1b0 con 0x7f9fe8102ef0 2024-09-19T08:01:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:01:04.402 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"btime":"2024-09-19T08:01:00:644221+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24406,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24424,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24485,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-09-19T08:01:04.402 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-09-19T08:01:04.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.405+0000 7f9fee6f2640 1 -- 172.21.15.98:0/3343903065 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f9fd00a3240 msgr2=0x7f9fd00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:04.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.405+0000 7f9fee6f2640 1 --2- 172.21.15.98:0/3343903065 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f9fd00a3240 0x7f9fd00a5700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8005fd0 tx=0x7f9fd800b390 comp rx=0 tx=0).stop 2024-09-19T08:01:04.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.405+0000 7f9fee6f2640 1 -- 172.21.15.98:0/3343903065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8102ef0 msgr2=0x7f9fe80fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:04.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.405+0000 7f9fee6f2640 1 --2- 172.21.15.98:0/3343903065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8102ef0 0x7f9fe80fd700 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9fe002ae70 tx=0x7f9fe002f5d0 comp rx=0 tx=0).stop 2024-09-19T08:01:04.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.405+0000 7f9fee6f2640 1 -- 172.21.15.98:0/3343903065 shutdown_connections 2024-09-19T08:01:04.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.405+0000 7f9fee6f2640 1 --2- 172.21.15.98:0/3343903065 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f9fd00a3240 0x7f9fd00a5700 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:04.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.405+0000 7f9fee6f2640 1 --2- 172.21.15.98:0/3343903065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8102ef0 0x7f9fe80fd700 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:04.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.405+0000 7f9fee6f2640 1 --2- 172.21.15.98:0/3343903065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fe8102570 0x7f9fe80fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:04.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.405+0000 7f9fee6f2640 1 --2- 172.21.15.98:0/3343903065 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9fe8100e20 0x7f9fe80fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:04.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.405+0000 7f9fee6f2640 1 -- 172.21.15.98:0/3343903065 >> 172.21.15.98:0/3343903065 conn(0x7f9fe80fc7d0 msgr2=0x7f9fe806a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:04.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.405+0000 7f9fee6f2640 1 -- 172.21.15.98:0/3343903065 shutdown_connections 2024-09-19T08:01:04.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:04.405+0000 7f9fee6f2640 1 -- 172.21.15.98:0/3343903065 wait complete. 2024-09-19T08:01:04.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:04 smithi139 ceph-mon[34874]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:04.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:04 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:04.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:04 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:04.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:04 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:04.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:04 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:04.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:04 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3343903065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:01:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:04 smithi098 ceph-mon[31333]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:04 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:04 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3343903065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:01:04.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-09-19T08:01:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:04 smithi162 ceph-mon[35260]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:04 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:04 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3343903065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:01:05.258 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:05.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.776+0000 7f61985a7640 1 -- 172.21.15.98:0/1431497172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6190100a80 msgr2=0x7f6190100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:05.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.776+0000 7f61985a7640 1 --2- 172.21.15.98:0/1431497172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6190100a80 0x7f6190100ee0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f618c009930 tx=0x7f618c02a420 comp rx=0 tx=0).stop 2024-09-19T08:01:05.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.776+0000 7f61985a7640 1 -- 172.21.15.98:0/1431497172 shutdown_connections 2024-09-19T08:01:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.776+0000 7f61985a7640 1 --2- 172.21.15.98:0/1431497172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6190101420 0x7f619010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.776+0000 7f61985a7640 1 --2- 172.21.15.98:0/1431497172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6190100a80 0x7f6190100ee0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.776+0000 7f61985a7640 1 --2- 172.21.15.98:0/1431497172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6190106a80 0x7f6190106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.776+0000 7f61985a7640 1 -- 172.21.15.98:0/1431497172 >> 172.21.15.98:0/1431497172 conn(0x7f61900fc7a0 msgr2=0x7f61900febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.776+0000 7f61985a7640 1 -- 172.21.15.98:0/1431497172 shutdown_connections 2024-09-19T08:01:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.776+0000 7f61985a7640 1 -- 172.21.15.98:0/1431497172 wait complete. 2024-09-19T08:01:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f61985a7640 1 Processor -- start 2024-09-19T08:01:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f61985a7640 1 -- start start 2024-09-19T08:01:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f61985a7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6190100a80 0x7f61901992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f61985a7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6190101420 0x7f6190199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f61985a7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6190106a80 0x7f61901a4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f61985a7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6190110e40 con 0x7f6190106a80 2024-09-19T08:01:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f61985a7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6190110cc0 con 0x7f6190101420 2024-09-19T08:01:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f61985a7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6190110fc0 con 0x7f6190100a80 2024-09-19T08:01:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f6196b44640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6190106a80 0x7f61901a4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f6196b44640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6190106a80 0x7f61901a4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45106/0 (socket says 172.21.15.98:45106) 2024-09-19T08:01:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f6196b44640 1 -- 172.21.15.98:0/1199756936 learned_addr learned my addr 172.21.15.98:0/1199756936 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f6196343640 1 --2- 172.21.15.98:0/1199756936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6190100a80 0x7f61901992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f6195b42640 1 --2- 172.21.15.98:0/1199756936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6190101420 0x7f6190199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f6196b44640 1 -- 172.21.15.98:0/1199756936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6190100a80 msgr2=0x7f61901992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f6196b44640 1 --2- 172.21.15.98:0/1199756936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6190100a80 0x7f61901992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f6196b44640 1 -- 172.21.15.98:0/1199756936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6190101420 msgr2=0x7f6190199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f6196b44640 1 --2- 172.21.15.98:0/1199756936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6190101420 0x7f6190199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.777+0000 7f6196b44640 1 -- 172.21.15.98:0/1199756936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61901a48b0 con 0x7f6190106a80 2024-09-19T08:01:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.778+0000 7f6196b44640 1 --2- 172.21.15.98:0/1199756936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6190106a80 0x7f61901a4370 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6188009fd0 tx=0x7f618800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:05.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.778+0000 7f61837fe640 1 -- 172.21.15.98:0/1199756936 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6188017070 con 0x7f6190106a80 2024-09-19T08:01:05.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.778+0000 7f61837fe640 1 -- 172.21.15.98:0/1199756936 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6188003e50 con 0x7f6190106a80 2024-09-19T08:01:05.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.778+0000 7f61837fe640 1 -- 172.21.15.98:0/1199756936 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61880106c0 con 0x7f6190106a80 2024-09-19T08:01:05.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.778+0000 7f61985a7640 1 -- 172.21.15.98:0/1199756936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61901a4a80 con 0x7f6190106a80 2024-09-19T08:01:05.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.778+0000 7f61985a7640 1 -- 172.21.15.98:0/1199756936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61901a4f40 con 0x7f6190106a80 2024-09-19T08:01:05.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.779+0000 7f61985a7640 1 -- 172.21.15.98:0/1199756936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6190100ee0 con 0x7f6190106a80 2024-09-19T08:01:05.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.780+0000 7f61837fe640 1 -- 172.21.15.98:0/1199756936 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f6188003970 con 0x7f6190106a80 2024-09-19T08:01:05.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.781+0000 7f61837fe640 1 --2- 172.21.15.98:0/1199756936 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f617c0a32c0 0x7f617c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:05.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.781+0000 7f6196343640 1 --2- 172.21.15.98:0/1199756936 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f617c0a32c0 0x7f617c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:05.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.781+0000 7f61837fe640 1 -- 172.21.15.98:0/1199756936 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f61880cf0f0 con 0x7f6190106a80 2024-09-19T08:01:05.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.781+0000 7f6196343640 1 --2- 172.21.15.98:0/1199756936 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f617c0a32c0 0x7f617c0a5780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6184005d00 tx=0x7f618400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:05.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.785+0000 7f61837fe640 1 -- 172.21.15.98:0/1199756936 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f618809bc60 con 0x7f6190106a80 2024-09-19T08:01:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.952+0000 7f61985a7640 1 -- 172.21.15.98:0/1199756936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f6190109010 con 0x7f6190106a80 2024-09-19T08:01:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.952+0000 7f61837fe640 1 -- 172.21.15.98:0/1199756936 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v73) ==== 84+0+65 (secure 0 0 0) 0x7f618809ff60 con 0x7f6190106a80 2024-09-19T08:01:05.953 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:01:05.953 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2024-09-19T08:01:05.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:01:05.953 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2024-09-19T08:01:05.953 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2024-09-19T08:01:05.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:01:05.953 INFO:teuthology.orchestra.run.smithi098.stdout:] 2024-09-19T08:01:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.955+0000 7f61985a7640 1 -- 172.21.15.98:0/1199756936 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f617c0a32c0 msgr2=0x7f617c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.955+0000 7f61985a7640 1 --2- 172.21.15.98:0/1199756936 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f617c0a32c0 0x7f617c0a5780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6184005d00 tx=0x7f618400b410 comp rx=0 tx=0).stop 2024-09-19T08:01:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.955+0000 7f61985a7640 1 -- 172.21.15.98:0/1199756936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6190106a80 msgr2=0x7f61901a4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.955+0000 7f61985a7640 1 --2- 172.21.15.98:0/1199756936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6190106a80 0x7f61901a4370 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6188009fd0 tx=0x7f618800b260 comp rx=0 tx=0).stop 2024-09-19T08:01:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.955+0000 7f61985a7640 1 -- 172.21.15.98:0/1199756936 shutdown_connections 2024-09-19T08:01:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.955+0000 7f61985a7640 1 --2- 172.21.15.98:0/1199756936 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f617c0a32c0 0x7f617c0a5780 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.955+0000 7f61985a7640 1 --2- 172.21.15.98:0/1199756936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6190106a80 0x7f61901a4370 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.955+0000 7f61985a7640 1 --2- 172.21.15.98:0/1199756936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6190101420 0x7f6190199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.955+0000 7f61985a7640 1 --2- 172.21.15.98:0/1199756936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6190100a80 0x7f61901992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.955+0000 7f61985a7640 1 -- 172.21.15.98:0/1199756936 >> 172.21.15.98:0/1199756936 conn(0x7f61900fc7a0 msgr2=0x7f61900fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:05.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.955+0000 7f61985a7640 1 -- 172.21.15.98:0/1199756936 shutdown_connections 2024-09-19T08:01:05.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:05.955+0000 7f61985a7640 1 -- 172.21.15.98:0/1199756936 wait complete. 2024-09-19T08:01:06.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:06.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:06.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:01:06.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:01:06.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:06.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:06 smithi098 ceph-mon[31333]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:06.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:06 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1199756936' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-19T08:01:06.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:06.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:06.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:01:06.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:01:06.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:06.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:06 smithi162 ceph-mon[35260]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:06.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:06 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1199756936' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-19T08:01:06.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:06.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:06.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:01:06.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:01:06.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:01:06.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:06 smithi139 ceph-mon[34874]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:06.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:06 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1199756936' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-19T08:01:06.571 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-19T08:01:06.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-19T08:01:06.815 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:07.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.280+0000 7f4a90b62640 1 -- 172.21.15.98:0/847526834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c100970 msgr2=0x7f4a8c10ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.280+0000 7f4a90b62640 1 --2- 172.21.15.98:0/847526834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c100970 0x7f4a8c10ea50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4a84009ca0 tx=0x7f4a8402a8c0 comp rx=0 tx=0).stop 2024-09-19T08:01:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.280+0000 7f4a90b62640 1 -- 172.21.15.98:0/847526834 shutdown_connections 2024-09-19T08:01:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.280+0000 7f4a90b62640 1 --2- 172.21.15.98:0/847526834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c100970 0x7f4a8c10ea50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.280+0000 7f4a90b62640 1 --2- 172.21.15.98:0/847526834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a8c0fffd0 0x7f4a8c100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.280+0000 7f4a90b62640 1 --2- 172.21.15.98:0/847526834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a8c106bb0 0x7f4a8c106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.280+0000 7f4a90b62640 1 -- 172.21.15.98:0/847526834 >> 172.21.15.98:0/847526834 conn(0x7f4a8c0fbee0 msgr2=0x7f4a8c0fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.280+0000 7f4a90b62640 1 -- 172.21.15.98:0/847526834 shutdown_connections 2024-09-19T08:01:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.280+0000 7f4a90b62640 1 -- 172.21.15.98:0/847526834 wait complete. 2024-09-19T08:01:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.281+0000 7f4a90b62640 1 Processor -- start 2024-09-19T08:01:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.281+0000 7f4a90b62640 1 -- start start 2024-09-19T08:01:07.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.281+0000 7f4a90b62640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c0fffd0 0x7f4a8c1a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:07.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.281+0000 7f4a90b62640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a8c100970 0x7f4a8c1a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:07.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.281+0000 7f4a90b62640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a8c106bb0 0x7f4a8c19a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:07.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.281+0000 7f4a90b62640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c1120a0 con 0x7f4a8c0fffd0 2024-09-19T08:01:07.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.281+0000 7f4a90b62640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c111f20 con 0x7f4a8c106bb0 2024-09-19T08:01:07.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.281+0000 7f4a90b62640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c112220 con 0x7f4a8c100970 2024-09-19T08:01:07.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.281+0000 7f4a8a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c0fffd0 0x7f4a8c1a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:07.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.281+0000 7f4a8a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c0fffd0 0x7f4a8c1a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45840/0 (socket says 172.21.15.98:45840) 2024-09-19T08:01:07.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.281+0000 7f4a8a59c640 1 -- 172.21.15.98:0/1493531570 learned_addr learned my addr 172.21.15.98:0/1493531570 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:07.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.281+0000 7f4a8ad9d640 1 --2- 172.21.15.98:0/1493531570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a8c106bb0 0x7f4a8c19a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:07.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.282+0000 7f4a8a59c640 1 -- 172.21.15.98:0/1493531570 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a8c100970 msgr2=0x7f4a8c1a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:07.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.282+0000 7f4a8a59c640 1 --2- 172.21.15.98:0/1493531570 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a8c100970 0x7f4a8c1a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:07.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.282+0000 7f4a8a59c640 1 -- 172.21.15.98:0/1493531570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a8c106bb0 msgr2=0x7f4a8c19a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:07.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.282+0000 7f4a8a59c640 1 --2- 172.21.15.98:0/1493531570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a8c106bb0 0x7f4a8c19a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:07.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.282+0000 7f4a8a59c640 1 -- 172.21.15.98:0/1493531570 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c19acd0 con 0x7f4a8c0fffd0 2024-09-19T08:01:07.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.282+0000 7f4a8a59c640 1 --2- 172.21.15.98:0/1493531570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c0fffd0 0x7f4a8c1a03e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00c890 tx=0x7f4a7c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:07.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.282+0000 7f4a7b7fe640 1 -- 172.21.15.98:0/1493531570 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c004220 con 0x7f4a8c0fffd0 2024-09-19T08:01:07.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.282+0000 7f4a90b62640 1 -- 172.21.15.98:0/1493531570 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8c19afc0 con 0x7f4a8c0fffd0 2024-09-19T08:01:07.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.282+0000 7f4a7b7fe640 1 -- 172.21.15.98:0/1493531570 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a7c0043c0 con 0x7f4a8c0fffd0 2024-09-19T08:01:07.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.282+0000 7f4a7b7fe640 1 -- 172.21.15.98:0/1493531570 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c004a00 con 0x7f4a8c0fffd0 2024-09-19T08:01:07.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.282+0000 7f4a90b62640 1 -- 172.21.15.98:0/1493531570 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a8c1a72b0 con 0x7f4a8c0fffd0 2024-09-19T08:01:07.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.283+0000 7f4a90b62640 1 -- 172.21.15.98:0/1493531570 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a8c100430 con 0x7f4a8c0fffd0 2024-09-19T08:01:07.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.284+0000 7f4a7b7fe640 1 -- 172.21.15.98:0/1493531570 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f4a7c0079b0 con 0x7f4a8c0fffd0 2024-09-19T08:01:07.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.285+0000 7f4a7b7fe640 1 --2- 172.21.15.98:0/1493531570 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4a740a3240 0x7f4a740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:07.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.285+0000 7f4a89d9b640 1 --2- 172.21.15.98:0/1493531570 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4a740a3240 0x7f4a740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:07.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.285+0000 7f4a7b7fe640 1 -- 172.21.15.98:0/1493531570 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f4a7c0cf220 con 0x7f4a8c0fffd0 2024-09-19T08:01:07.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.285+0000 7f4a89d9b640 1 --2- 172.21.15.98:0/1493531570 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4a740a3240 0x7f4a740a5700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c103160 tx=0x7f4a80006c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:07.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.290+0000 7f4a7b7fe640 1 -- 172.21.15.98:0/1493531570 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f4a7c014030 con 0x7f4a8c0fffd0 2024-09-19T08:01:07.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.457+0000 7f4a90b62640 1 -- 172.21.15.98:0/1493531570 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4a8c1a74e0 con 0x7f4a8c0fffd0 2024-09-19T08:01:07.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.745+0000 7f4a7b7fe640 1 -- 172.21.15.98:0/1493531570 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v74) ==== 138+0+0 (secure 0 0 0) 0x7f4a7c09c3e0 con 0x7f4a8c0fffd0 2024-09-19T08:01:07.745 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2024-09-19T08:01:07.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.747+0000 7f4a90b62640 1 -- 172.21.15.98:0/1493531570 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4a740a3240 msgr2=0x7f4a740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:07.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.747+0000 7f4a90b62640 1 --2- 172.21.15.98:0/1493531570 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4a740a3240 0x7f4a740a5700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c103160 tx=0x7f4a80006c90 comp rx=0 tx=0).stop 2024-09-19T08:01:07.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.747+0000 7f4a90b62640 1 -- 172.21.15.98:0/1493531570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c0fffd0 msgr2=0x7f4a8c1a03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:07.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.747+0000 7f4a90b62640 1 --2- 172.21.15.98:0/1493531570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c0fffd0 0x7f4a8c1a03e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00c890 tx=0x7f4a7c00cd50 comp rx=0 tx=0).stop 2024-09-19T08:01:07.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.747+0000 7f4a90b62640 1 -- 172.21.15.98:0/1493531570 shutdown_connections 2024-09-19T08:01:07.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.747+0000 7f4a90b62640 1 --2- 172.21.15.98:0/1493531570 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4a740a3240 0x7f4a740a5700 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:07.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.747+0000 7f4a90b62640 1 --2- 172.21.15.98:0/1493531570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a8c106bb0 0x7f4a8c19a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:07.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.747+0000 7f4a90b62640 1 --2- 172.21.15.98:0/1493531570 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a8c100970 0x7f4a8c1a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:07.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.747+0000 7f4a90b62640 1 --2- 172.21.15.98:0/1493531570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c0fffd0 0x7f4a8c1a03e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:07.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.747+0000 7f4a90b62640 1 -- 172.21.15.98:0/1493531570 >> 172.21.15.98:0/1493531570 conn(0x7f4a8c0fbee0 msgr2=0x7f4a8c10d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:07.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.748+0000 7f4a90b62640 1 -- 172.21.15.98:0/1493531570 shutdown_connections 2024-09-19T08:01:07.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:07.748+0000 7f4a90b62640 1 -- 172.21.15.98:0/1493531570 wait complete. 2024-09-19T08:01:08.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:07 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1493531570' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-19T08:01:08.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:07 smithi098 ceph-mon[31333]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:07 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1493531570' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-19T08:01:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:07 smithi162 ceph-mon[35260]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:08.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:07 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1493531570' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-19T08:01:08.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:07 smithi139 ceph-mon[34874]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:08.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-19T08:01:08.637 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:09.020 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:08 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1493531570' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-19T08:01:09.021 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:08 smithi098 ceph-mon[31333]: osdmap e74: 12 total, 12 up, 12 in 2024-09-19T08:01:09.021 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:08 smithi098 ceph-mon[31333]: osdmap e75: 12 total, 12 up, 12 in 2024-09-19T08:01:09.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:08 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1493531570' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-19T08:01:09.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:08 smithi162 ceph-mon[35260]: osdmap e74: 12 total, 12 up, 12 in 2024-09-19T08:01:09.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:08 smithi162 ceph-mon[35260]: osdmap e75: 12 total, 12 up, 12 in 2024-09-19T08:01:09.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.106+0000 7f83ad9aa640 1 -- 172.21.15.98:0/2570886411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a8106a80 msgr2=0x7f83a8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:09.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.106+0000 7f83ad9aa640 1 --2- 172.21.15.98:0/2570886411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a8106a80 0x7f83a8106e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8398009930 tx=0x7f839802a420 comp rx=0 tx=0).stop 2024-09-19T08:01:09.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.106+0000 7f83ad9aa640 1 -- 172.21.15.98:0/2570886411 shutdown_connections 2024-09-19T08:01:09.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.106+0000 7f83ad9aa640 1 --2- 172.21.15.98:0/2570886411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83a8101420 0x7f83a810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:09.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.106+0000 7f83ad9aa640 1 --2- 172.21.15.98:0/2570886411 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83a8100a80 0x7f83a8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:09.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.106+0000 7f83ad9aa640 1 --2- 172.21.15.98:0/2570886411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a8106a80 0x7f83a8106e60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:09.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.106+0000 7f83ad9aa640 1 -- 172.21.15.98:0/2570886411 >> 172.21.15.98:0/2570886411 conn(0x7f83a80fc7a0 msgr2=0x7f83a80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:09.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.106+0000 7f83ad9aa640 1 -- 172.21.15.98:0/2570886411 shutdown_connections 2024-09-19T08:01:09.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.106+0000 7f83ad9aa640 1 -- 172.21.15.98:0/2570886411 wait complete. 2024-09-19T08:01:09.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83ad9aa640 1 Processor -- start 2024-09-19T08:01:09.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83ad9aa640 1 -- start start 2024-09-19T08:01:09.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83ad9aa640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83a8100a80 0x7f83a81985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:09.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83ad9aa640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83a8101420 0x7f83a8198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:09.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83ad9aa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a8106a80 0x7f83a819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:09.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83ad9aa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83a8110f20 con 0x7f83a8106a80 2024-09-19T08:01:09.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83ad9aa640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f83a8110da0 con 0x7f83a8101420 2024-09-19T08:01:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83ad9aa640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f83a81110a0 con 0x7f83a8100a80 2024-09-19T08:01:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a8106a80 0x7f83a819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a8106a80 0x7f83a819cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45856/0 (socket says 172.21.15.98:45856) 2024-09-19T08:01:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83a77fe640 1 -- 172.21.15.98:0/3481608652 learned_addr learned my addr 172.21.15.98:0/3481608652 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83a6ffd640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83a8100a80 0x7f83a81985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f839ffff640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83a8101420 0x7f83a8198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83a77fe640 1 -- 172.21.15.98:0/3481608652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83a8100a80 msgr2=0x7f83a81985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83a77fe640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83a8100a80 0x7f83a81985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83a77fe640 1 -- 172.21.15.98:0/3481608652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83a8101420 msgr2=0x7f83a8198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83a77fe640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83a8101420 0x7f83a8198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83a77fe640 1 -- 172.21.15.98:0/3481608652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a819d5a0 con 0x7f83a8106a80 2024-09-19T08:01:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f839ffff640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83a8101420 0x7f83a8198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:01:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.107+0000 7f83a77fe640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a8106a80 0x7f83a819cec0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f83a000b970 tx=0x7f83a000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.108+0000 7f83ac9a8640 1 -- 172.21.15.98:0/3481608652 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a0019070 con 0x7f83a8106a80 2024-09-19T08:01:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.108+0000 7f83ac9a8640 1 -- 172.21.15.98:0/3481608652 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83a000cb40 con 0x7f83a8106a80 2024-09-19T08:01:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.108+0000 7f83ad9aa640 1 -- 172.21.15.98:0/3481608652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a819d890 con 0x7f83a8106a80 2024-09-19T08:01:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.108+0000 7f83ac9a8640 1 -- 172.21.15.98:0/3481608652 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a0007650 con 0x7f83a8106a80 2024-09-19T08:01:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.108+0000 7f83ad9aa640 1 -- 172.21.15.98:0/3481608652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a81a5170 con 0x7f83a8106a80 2024-09-19T08:01:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.109+0000 7f83ad9aa640 1 -- 172.21.15.98:0/3481608652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83a8100f60 con 0x7f83a8106a80 2024-09-19T08:01:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.110+0000 7f83ac9a8640 1 -- 172.21.15.98:0/3481608652 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f83a001c070 con 0x7f83a8106a80 2024-09-19T08:01:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.111+0000 7f83ac9a8640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f83940a32c0 0x7f83940a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.111+0000 7f83a6ffd640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f83940a32c0 0x7f83940a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.111+0000 7f83ac9a8640 1 -- 172.21.15.98:0/3481608652 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7497+0+0 (secure 0 0 0) 0x7f83a00cf010 con 0x7f83a8106a80 2024-09-19T08:01:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.111+0000 7f83a6ffd640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f83940a32c0 0x7f83940a5780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8398000c00 tx=0x7f8398031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.114+0000 7f83ac9a8640 1 -- 172.21.15.98:0/3481608652 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f83a009c060 con 0x7f83a8106a80 2024-09-19T08:01:09.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:08 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1493531570' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-19T08:01:09.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:08 smithi139 ceph-mon[34874]: osdmap e74: 12 total, 12 up, 12 in 2024-09-19T08:01:09.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:08 smithi139 ceph-mon[34874]: osdmap e75: 12 total, 12 up, 12 in 2024-09-19T08:01:09.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.284+0000 7f83ad9aa640 1 -- 172.21.15.98:0/3481608652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f83a802d7c0 con 0x7f83a8106a80 2024-09-19T08:01:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.749+0000 7f83ac9a8640 1 -- 172.21.15.98:0/3481608652 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v76) ==== 170+0+0 (secure 0 0 0) 0x7f83a009b960 con 0x7f83a8106a80 2024-09-19T08:01:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2024-09-19T08:01:09.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.752+0000 7f83ad9aa640 1 -- 172.21.15.98:0/3481608652 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f83940a32c0 msgr2=0x7f83940a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:09.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.752+0000 7f83ad9aa640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f83940a32c0 0x7f83940a5780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8398000c00 tx=0x7f8398031490 comp rx=0 tx=0).stop 2024-09-19T08:01:09.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.752+0000 7f83ad9aa640 1 -- 172.21.15.98:0/3481608652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a8106a80 msgr2=0x7f83a819cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:09.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.752+0000 7f83ad9aa640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a8106a80 0x7f83a819cec0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f83a000b970 tx=0x7f83a000be30 comp rx=0 tx=0).stop 2024-09-19T08:01:09.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.752+0000 7f83ad9aa640 1 -- 172.21.15.98:0/3481608652 shutdown_connections 2024-09-19T08:01:09.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.752+0000 7f83ad9aa640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f83940a32c0 0x7f83940a5780 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:09.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.752+0000 7f83ad9aa640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a8106a80 0x7f83a819cec0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:09.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.752+0000 7f83ad9aa640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83a8101420 0x7f83a8198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:09.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.752+0000 7f83ad9aa640 1 --2- 172.21.15.98:0/3481608652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83a8100a80 0x7f83a81985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:09.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.752+0000 7f83ad9aa640 1 -- 172.21.15.98:0/3481608652 >> 172.21.15.98:0/3481608652 conn(0x7f83a80fc7a0 msgr2=0x7f83a806ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:09.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.752+0000 7f83ad9aa640 1 -- 172.21.15.98:0/3481608652 shutdown_connections 2024-09-19T08:01:09.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:09.752+0000 7f83ad9aa640 1 -- 172.21.15.98:0/3481608652 wait complete. 2024-09-19T08:01:10.057 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:09 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3481608652' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-19T08:01:10.057 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:09 smithi098 ceph-mon[31333]: pgmap v267: 33 pgs: 7 creating+peering, 25 unknown, 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:10.057 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:09 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3481608652' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-19T08:01:10.057 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:09 smithi098 ceph-mon[31333]: osdmap e76: 12 total, 12 up, 12 in 2024-09-19T08:01:10.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:09 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3481608652' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-19T08:01:10.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:09 smithi162 ceph-mon[35260]: pgmap v267: 33 pgs: 7 creating+peering, 25 unknown, 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:10.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:09 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3481608652' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-19T08:01:10.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:09 smithi162 ceph-mon[35260]: osdmap e76: 12 total, 12 up, 12 in 2024-09-19T08:01:10.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:09 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3481608652' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-19T08:01:10.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:09 smithi139 ceph-mon[34874]: pgmap v267: 33 pgs: 7 creating+peering, 25 unknown, 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:10.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:09 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3481608652' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-19T08:01:10.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:09 smithi139 ceph-mon[34874]: osdmap e76: 12 total, 12 up, 12 in 2024-09-19T08:01:10.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-19T08:01:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:11.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:10 smithi098 ceph-mon[31333]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-19T08:01:11.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:10 smithi098 ceph-mon[31333]: osdmap e77: 12 total, 12 up, 12 in 2024-09-19T08:01:11.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:10 smithi162 ceph-mon[35260]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-19T08:01:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:10 smithi162 ceph-mon[35260]: osdmap e77: 12 total, 12 up, 12 in 2024-09-19T08:01:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.095+0000 7f2f61b90640 1 -- 172.21.15.98:0/3383783647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f5c102d40 msgr2=0x7f2f5c103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.095+0000 7f2f61b90640 1 --2- 172.21.15.98:0/3383783647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f5c102d40 0x7f2f5c103120 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c009900 tx=0x7f2f4c02a480 comp rx=0 tx=0).stop 2024-09-19T08:01:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.095+0000 7f2f61b90640 1 -- 172.21.15.98:0/3383783647 shutdown_connections 2024-09-19T08:01:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.095+0000 7f2f61b90640 1 --2- 172.21.15.98:0/3383783647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f5c10c170 0x7f2f5c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.095+0000 7f2f61b90640 1 --2- 172.21.15.98:0/3383783647 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f5c103660 0x7f2f5c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.095+0000 7f2f61b90640 1 --2- 172.21.15.98:0/3383783647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f5c102d40 0x7f2f5c103120 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.095+0000 7f2f61b90640 1 -- 172.21.15.98:0/3383783647 >> 172.21.15.98:0/3383783647 conn(0x7f2f5c0fc7a0 msgr2=0x7f2f5c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.095+0000 7f2f61b90640 1 -- 172.21.15.98:0/3383783647 shutdown_connections 2024-09-19T08:01:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.095+0000 7f2f61b90640 1 -- 172.21.15.98:0/3383783647 wait complete. 2024-09-19T08:01:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.095+0000 7f2f61b90640 1 Processor -- start 2024-09-19T08:01:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f61b90640 1 -- start start 2024-09-19T08:01:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f61b90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f5c102d40 0x7f2f5c1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f61b90640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f5c103660 0x7f2f5c198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f61b90640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f5c10c170 0x7f2f5c19ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f61b90640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c111f70 con 0x7f2f5c102d40 2024-09-19T08:01:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f61b90640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c111df0 con 0x7f2f5c10c170 2024-09-19T08:01:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f61b90640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c1120f0 con 0x7f2f5c103660 2024-09-19T08:01:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f5b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f5c102d40 0x7f2f5c1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f5b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f5c102d40 0x7f2f5c1983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45872/0 (socket says 172.21.15.98:45872) 2024-09-19T08:01:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f5b7fe640 1 -- 172.21.15.98:0/471282379 learned_addr learned my addr 172.21.15.98:0/471282379 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f5affd640 1 --2- 172.21.15.98:0/471282379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f5c103660 0x7f2f5c198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f5bfff640 1 --2- 172.21.15.98:0/471282379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f5c10c170 0x7f2f5c19ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f5b7fe640 1 -- 172.21.15.98:0/471282379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f5c103660 msgr2=0x7f2f5c198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f5b7fe640 1 --2- 172.21.15.98:0/471282379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f5c103660 0x7f2f5c198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f5b7fe640 1 -- 172.21.15.98:0/471282379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f5c10c170 msgr2=0x7f2f5c19ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f5b7fe640 1 --2- 172.21.15.98:0/471282379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f5c10c170 0x7f2f5c19ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.096+0000 7f2f5b7fe640 1 -- 172.21.15.98:0/471282379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f5c19d3a0 con 0x7f2f5c102d40 2024-09-19T08:01:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.097+0000 7f2f5b7fe640 1 --2- 172.21.15.98:0/471282379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f5c102d40 0x7f2f5c1983f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c006010 tx=0x7f2f4c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.097+0000 7f2f58ff9640 1 -- 172.21.15.98:0/471282379 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c03f030 con 0x7f2f5c102d40 2024-09-19T08:01:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.097+0000 7f2f58ff9640 1 -- 172.21.15.98:0/471282379 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f4c003ce0 con 0x7f2f5c102d40 2024-09-19T08:01:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.097+0000 7f2f61b90640 1 -- 172.21.15.98:0/471282379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f5c19d630 con 0x7f2f5c102d40 2024-09-19T08:01:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.097+0000 7f2f58ff9640 1 -- 172.21.15.98:0/471282379 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c035070 con 0x7f2f5c102d40 2024-09-19T08:01:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.097+0000 7f2f61b90640 1 -- 172.21.15.98:0/471282379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f5c1a4f70 con 0x7f2f5c102d40 2024-09-19T08:01:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.098+0000 7f2f61b90640 1 -- 172.21.15.98:0/471282379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f5c06b070 con 0x7f2f5c102d40 2024-09-19T08:01:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.099+0000 7f2f58ff9640 1 -- 172.21.15.98:0/471282379 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2f4c042030 con 0x7f2f5c102d40 2024-09-19T08:01:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.100+0000 7f2f58ff9640 1 --2- 172.21.15.98:0/471282379 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2f440a3240 0x7f2f440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.100+0000 7f2f5affd640 1 --2- 172.21.15.98:0/471282379 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2f440a3240 0x7f2f440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.100+0000 7f2f58ff9640 1 -- 172.21.15.98:0/471282379 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7879+0+0 (secure 0 0 0) 0x7f2f4c0f0110 con 0x7f2f5c102d40 2024-09-19T08:01:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.101+0000 7f2f5affd640 1 --2- 172.21.15.98:0/471282379 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2f440a3240 0x7f2f440a5700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2f50005fd0 tx=0x7f2f50005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.104+0000 7f2f58ff9640 1 -- 172.21.15.98:0/471282379 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2f4c0bd000 con 0x7f2f5c102d40 2024-09-19T08:01:11.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:10 smithi139 ceph-mon[34874]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-19T08:01:11.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:10 smithi139 ceph-mon[34874]: osdmap e77: 12 total, 12 up, 12 in 2024-09-19T08:01:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.305+0000 7f2f61b90640 1 -- 172.21.15.98:0/471282379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f2f5c103120 con 0x7f2f5c102d40 2024-09-19T08:01:11.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.777+0000 7f2f58ff9640 1 -- 172.21.15.98:0/471282379 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f2f5c103120 con 0x7f2f5c102d40 2024-09-19T08:01:11.778 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 2 and data pool 3 2024-09-19T08:01:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.780+0000 7f2f61b90640 1 -- 172.21.15.98:0/471282379 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2f440a3240 msgr2=0x7f2f440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.780+0000 7f2f61b90640 1 --2- 172.21.15.98:0/471282379 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2f440a3240 0x7f2f440a5700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2f50005fd0 tx=0x7f2f50005f60 comp rx=0 tx=0).stop 2024-09-19T08:01:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.780+0000 7f2f61b90640 1 -- 172.21.15.98:0/471282379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f5c102d40 msgr2=0x7f2f5c1983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.780+0000 7f2f61b90640 1 --2- 172.21.15.98:0/471282379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f5c102d40 0x7f2f5c1983f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c006010 tx=0x7f2f4c0042f0 comp rx=0 tx=0).stop 2024-09-19T08:01:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.780+0000 7f2f61b90640 1 -- 172.21.15.98:0/471282379 shutdown_connections 2024-09-19T08:01:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.780+0000 7f2f61b90640 1 --2- 172.21.15.98:0/471282379 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f2f440a3240 0x7f2f440a5700 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.780+0000 7f2f61b90640 1 --2- 172.21.15.98:0/471282379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f5c10c170 0x7f2f5c19ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.780+0000 7f2f61b90640 1 --2- 172.21.15.98:0/471282379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f5c103660 0x7f2f5c198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.780+0000 7f2f61b90640 1 --2- 172.21.15.98:0/471282379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f5c102d40 0x7f2f5c1983f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.780+0000 7f2f61b90640 1 -- 172.21.15.98:0/471282379 >> 172.21.15.98:0/471282379 conn(0x7f2f5c0fc7a0 msgr2=0x7f2f5c0fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.780+0000 7f2f61b90640 1 -- 172.21.15.98:0/471282379 shutdown_connections 2024-09-19T08:01:11.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:11.780+0000 7f2f61b90640 1 -- 172.21.15.98:0/471282379 wait complete. 2024-09-19T08:01:12.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:11 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/471282379' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-19T08:01:12.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:11 smithi098 ceph-mon[31333]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-19T08:01:12.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:11 smithi098 ceph-mon[31333]: pgmap v270: 97 pgs: 7 active+clean, 13 creating+peering, 77 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:12.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:11 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/471282379' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-19T08:01:12.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:11 smithi162 ceph-mon[35260]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-19T08:01:12.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:11 smithi162 ceph-mon[35260]: pgmap v270: 97 pgs: 7 active+clean, 13 creating+peering, 77 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:12.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:11 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/471282379' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-19T08:01:12.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:11 smithi139 ceph-mon[34874]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-19T08:01:12.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:11 smithi139 ceph-mon[34874]: pgmap v270: 97 pgs: 7 active+clean, 13 creating+peering, 77 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:01:12.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T08:01:12.726 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:13.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:12 smithi162 ceph-mon[35260]: osdmap e78: 12 total, 12 up, 12 in 2024-09-19T08:01:13.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:12 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/471282379' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-19T08:01:13.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:12 smithi162 ceph-mon[35260]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-19T08:01:13.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:12 smithi162 ceph-mon[35260]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-19T08:01:13.120 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:12 smithi098 ceph-mon[31333]: osdmap e78: 12 total, 12 up, 12 in 2024-09-19T08:01:13.121 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:12 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/471282379' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-19T08:01:13.121 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:12 smithi098 ceph-mon[31333]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-19T08:01:13.121 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:12 smithi098 ceph-mon[31333]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-19T08:01:13.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:12 smithi139 ceph-mon[34874]: osdmap e78: 12 total, 12 up, 12 in 2024-09-19T08:01:13.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:12 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/471282379' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-19T08:01:13.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:12 smithi139 ceph-mon[34874]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-19T08:01:13.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:12 smithi139 ceph-mon[34874]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-19T08:01:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.200+0000 7f5f65c24640 1 -- 172.21.15.98:0/1343150939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f60108300 msgr2=0x7f5f6010ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.200+0000 7f5f65c24640 1 --2- 172.21.15.98:0/1343150939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f60108300 0x7f5f6010ec70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009cd0 tx=0x7f5f5802a8d0 comp rx=0 tx=0).stop 2024-09-19T08:01:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.200+0000 7f5f65c24640 1 -- 172.21.15.98:0/1343150939 shutdown_connections 2024-09-19T08:01:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.200+0000 7f5f65c24640 1 --2- 172.21.15.98:0/1343150939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f60108300 0x7f5f6010ec70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.200+0000 7f5f65c24640 1 --2- 172.21.15.98:0/1343150939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f60103780 0x7f5f60107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.200+0000 7f5f65c24640 1 --2- 172.21.15.98:0/1343150939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f60102dd0 0x7f5f601031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.200+0000 7f5f65c24640 1 -- 172.21.15.98:0/1343150939 >> 172.21.15.98:0/1343150939 conn(0x7f5f600fc7d0 msgr2=0x7f5f600febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.200+0000 7f5f65c24640 1 -- 172.21.15.98:0/1343150939 shutdown_connections 2024-09-19T08:01:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.200+0000 7f5f65c24640 1 -- 172.21.15.98:0/1343150939 wait complete. 2024-09-19T08:01:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.200+0000 7f5f65c24640 1 Processor -- start 2024-09-19T08:01:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.200+0000 7f5f65c24640 1 -- start start 2024-09-19T08:01:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.200+0000 7f5f65c24640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f60102dd0 0x7f5f601985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f65c24640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f60103780 0x7f5f60198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f65c24640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f60108300 0x7f5f6019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f65c24640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5f60112110 con 0x7f5f60103780 2024-09-19T08:01:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f65c24640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5f60111f90 con 0x7f5f60108300 2024-09-19T08:01:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f65c24640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5f60112290 con 0x7f5f60102dd0 2024-09-19T08:01:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f57fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f60103780 0x7f5f60198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f57fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f60103780 0x7f5f60198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45892/0 (socket says 172.21.15.98:45892) 2024-09-19T08:01:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f57fff640 1 -- 172.21.15.98:0/3577329336 learned_addr learned my addr 172.21.15.98:0/3577329336 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f5f7fe640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f60102dd0 0x7f5f601985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f5ffff640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f60108300 0x7f5f6019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f57fff640 1 -- 172.21.15.98:0/3577329336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f60102dd0 msgr2=0x7f5f601985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f57fff640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f60102dd0 0x7f5f601985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f57fff640 1 -- 172.21.15.98:0/3577329336 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f60108300 msgr2=0x7f5f6019ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:13.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f57fff640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f60108300 0x7f5f6019ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:13.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f57fff640 1 -- 172.21.15.98:0/3577329336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6019d570 con 0x7f5f60103780 2024-09-19T08:01:13.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f5ffff640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f60108300 0x7f5f6019ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:01:13.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f57fff640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f60103780 0x7f5f60198b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800c970 tx=0x7f5f4800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:13.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f5d7fa640 1 -- 172.21.15.98:0/3577329336 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f48007bb0 con 0x7f5f60103780 2024-09-19T08:01:13.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f65c24640 1 -- 172.21.15.98:0/3577329336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f6019d860 con 0x7f5f60103780 2024-09-19T08:01:13.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f5d7fa640 1 -- 172.21.15.98:0/3577329336 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f48007d50 con 0x7f5f60103780 2024-09-19T08:01:13.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f5d7fa640 1 -- 172.21.15.98:0/3577329336 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4800f450 con 0x7f5f60103780 2024-09-19T08:01:13.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.201+0000 7f5f65c24640 1 -- 172.21.15.98:0/3577329336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f601a5140 con 0x7f5f60103780 2024-09-19T08:01:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.203+0000 7f5f65c24640 1 -- 172.21.15.98:0/3577329336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f601031b0 con 0x7f5f60103780 2024-09-19T08:01:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.203+0000 7f5f5d7fa640 1 -- 172.21.15.98:0/3577329336 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f5f48004eb0 con 0x7f5f60103780 2024-09-19T08:01:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.204+0000 7f5f5d7fa640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5f4c0a3240 0x7f5f4c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.204+0000 7f5f5f7fe640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5f4c0a3240 0x7f5f4c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.204+0000 7f5f5d7fa640 1 -- 172.21.15.98:0/3577329336 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7f5f480cf0e0 con 0x7f5f60103780 2024-09-19T08:01:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.204+0000 7f5f5f7fe640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5f4c0a3240 0x7f5f4c0a5700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5f50009990 tx=0x7f5f50009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.208+0000 7f5f5d7fa640 1 -- 172.21.15.98:0/3577329336 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f5f48014030 con 0x7f5f60103780 2024-09-19T08:01:13.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.374+0000 7f5f65c24640 1 -- 172.21.15.98:0/3577329336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f6010a290 con 0x7f5f60103780 2024-09-19T08:01:13.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.376+0000 7f5f5d7fa640 1 -- 172.21.15.98:0/3577329336 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22301 (secure 0 0 0) 0x7f5f4809bf50 con 0x7f5f60103780 2024-09-19T08:01:13.376 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:01:13.376 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":78,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","created":"2024-09-19T07:53:39.127398+0000","modified":"2024-09-19T08:01:11.306885+0000","last_up_change":"2024-09-19T08:00:22.467777+0000","last_in_change":"2024-09-19T07:59:58.538492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:56:43.773892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T08:01:07.693796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T08:01:09.518028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d01ce6-165d-493d-ae98-d8ad9e5d4134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6803","nonce":1719360533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6805","nonce":1719360533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6809","nonce":1719360533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6807","nonce":1719360533}]},"public_addr":"172.21.15.98:6803/1719360533","cluster_addr":"172.21.15.98:6805/1719360533","heartbeat_back_addr":"172.21.15.98:6809/1719360533","heartbeat_front_addr":"172.21.15.98:6807/1719360533","state":["exists","up"]},{"osd":1,"uuid":"34724bd5-0738-4473-b087-c09031c1ee2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036962626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036962626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036962626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036962626}]},"public_addr":"172.21.15.139:6801/4036962626","cluster_addr":"172.21.15.139:6803/4036962626","heartbeat_back_addr":"172.21.15.139:6807/4036962626","heartbeat_front_addr":"172.21.15.139:6805/4036962626","state":["exists","up"]},{"osd":2,"uuid":"16985ebe-0e32-42ee-bded-b7838e290afc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6801","nonce":2811412891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6803","nonce":2811412891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6807","nonce":2811412891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6805","nonce":2811412891}]},"public_addr":"172.21.15.162:6801/2811412891","cluster_addr":"172.21.15.162:6803/2811412891","heartbeat_back_addr":"172.21.15.162:6807/2811412891","heartbeat_front_addr":"172.21.15.162:6805/2811412891","state":["exists","up"]},{"osd":3,"uuid":"b140e466-3119-49cb-aec2-ba6ca4d10d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6811","nonce":1383392014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6813","nonce":1383392014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6817","nonce":1383392014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6815","nonce":1383392014}]},"public_addr":"172.21.15.98:6811/1383392014","cluster_addr":"172.21.15.98:6813/1383392014","heartbeat_back_addr":"172.21.15.98:6817/1383392014","heartbeat_front_addr":"172.21.15.98:6815/1383392014","state":["exists","up"]},{"osd":4,"uuid":"b4412813-df57-4ac6-8fb7-f57b69c74768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6809","nonce":116453520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6811","nonce":116453520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6815","nonce":116453520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6813","nonce":116453520}]},"public_addr":"172.21.15.139:6809/116453520","cluster_addr":"172.21.15.139:6811/116453520","heartbeat_back_addr":"172.21.15.139:6815/116453520","heartbeat_front_addr":"172.21.15.139:6813/116453520","state":["exists","up"]},{"osd":5,"uuid":"d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6809","nonce":3769530297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6811","nonce":3769530297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6815","nonce":3769530297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6813","nonce":3769530297}]},"public_addr":"172.21.15.162:6809/3769530297","cluster_addr":"172.21.15.162:6811/3769530297","heartbeat_back_addr":"172.21.15.162:6815/3769530297","heartbeat_front_addr":"172.21.15.162:6813/3769530297","state":["exists","up"]},{"osd":6,"uuid":"ba1c1590-6aea-40dc-90c8-e21b88896909","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6819","nonce":692373267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6821","nonce":692373267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6825","nonce":692373267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6823","nonce":692373267}]},"public_addr":"172.21.15.98:6819/692373267","cluster_addr":"172.21.15.98:6821/692373267","heartbeat_back_addr":"172.21.15.98:6825/692373267","heartbeat_front_addr":"172.21.15.98:6823/692373267","state":["exists","up"]},{"osd":7,"uuid":"eb9093ad-a068-4a63-ab87-0b50522cbc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6817","nonce":1963811650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6819","nonce":1963811650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6823","nonce":1963811650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6821","nonce":1963811650}]},"public_addr":"172.21.15.139:6817/1963811650","cluster_addr":"172.21.15.139:6819/1963811650","heartbeat_back_addr":"172.21.15.139:6823/1963811650","heartbeat_front_addr":"172.21.15.139:6821/1963811650","state":["exists","up"]},{"osd":8,"uuid":"31ca330a-fc45-466d-81c2-ef4386b3cc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6817","nonce":3607966458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6819","nonce":3607966458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6823","nonce":3607966458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6821","nonce":3607966458}]},"public_addr":"172.21.15.162:6817/3607966458","cluster_addr":"172.21.15.162:6819/3607966458","heartbeat_back_addr":"172.21.15.162:6823/3607966458","heartbeat_front_addr":"172.21.15.162:6821/3607966458","state":["exists","up"]},{"osd":9,"uuid":"9c4f216f-aef1-4367-9c58-b913b0c4f7db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6827","nonce":2050391244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6829","nonce":2050391244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6833","nonce":2050391244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6831","nonce":2050391244}]},"public_addr":"172.21.15.98:6827/2050391244","cluster_addr":"172.21.15.98:6829/2050391244","heartbeat_back_addr":"172.21.15.98:6833/2050391244","heartbeat_front_addr":"172.21.15.98:6831/2050391244","state":["exists","up"]},{"osd":10,"uuid":"03cc7df6-84ae-4d9a-a3d7-74c91e1744e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6825","nonce":2336761639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6827","nonce":2336761639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6831","nonce":2336761639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6829","nonce":2336761639}]},"public_addr":"172.21.15.139:6825/2336761639","cluster_addr":"172.21.15.139:6827/2336761639","heartbeat_back_addr":"172.21.15.139:6831/2336761639","heartbeat_front_addr":"172.21.15.139:6829/2336761639","state":["exists","up"]},{"osd":11,"uuid":"29e95498-4952-46d4-8fa6-6d8e2b862111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6825","nonce":1231011554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6827","nonce":1231011554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6831","nonce":1231011554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6829","nonce":1231011554}]},"public_addr":"172.21.15.162:6825/1231011554","cluster_addr":"172.21.15.162:6827/1231011554","heartbeat_back_addr":"172.21.15.162:6831/1231011554","heartbeat_front_addr":"172.21.15.162:6829/1231011554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:55:49.995244+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:15.655543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:40.331165+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:04.688881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:29.174966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:53.663161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:17.407551+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:42.584296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:06.317219+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:31.078625+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:55.689269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T08:00:19.659748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/3842627255":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3838590070":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/1238193420":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6800/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6800/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3194864054":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3866031018":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/1497156658":"2024-09-20T07:54:02.418785+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T08:01:13.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.378+0000 7f5f65c24640 1 -- 172.21.15.98:0/3577329336 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5f4c0a3240 msgr2=0x7f5f4c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:13.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.378+0000 7f5f65c24640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5f4c0a3240 0x7f5f4c0a5700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5f50009990 tx=0x7f5f50009110 comp rx=0 tx=0).stop 2024-09-19T08:01:13.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.378+0000 7f5f65c24640 1 -- 172.21.15.98:0/3577329336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f60103780 msgr2=0x7f5f60198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:13.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.378+0000 7f5f65c24640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f60103780 0x7f5f60198b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800c970 tx=0x7f5f4800cd70 comp rx=0 tx=0).stop 2024-09-19T08:01:13.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.378+0000 7f5f65c24640 1 -- 172.21.15.98:0/3577329336 shutdown_connections 2024-09-19T08:01:13.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.378+0000 7f5f65c24640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5f4c0a3240 0x7f5f4c0a5700 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:13.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.378+0000 7f5f65c24640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f60108300 0x7f5f6019ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:13.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.378+0000 7f5f65c24640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f60103780 0x7f5f60198b00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:13.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.378+0000 7f5f65c24640 1 --2- 172.21.15.98:0/3577329336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f60102dd0 0x7f5f601985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:13.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.378+0000 7f5f65c24640 1 -- 172.21.15.98:0/3577329336 >> 172.21.15.98:0/3577329336 conn(0x7f5f600fc7d0 msgr2=0x7f5f60105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:13.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.378+0000 7f5f65c24640 1 -- 172.21.15.98:0/3577329336 shutdown_connections 2024-09-19T08:01:13.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:13.378+0000 7f5f65c24640 1 -- 172.21.15.98:0/3577329336 wait complete. 2024-09-19T08:01:13.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:13 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] up:active 2024-09-19T08:01:13.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:13 smithi098 ceph-mon[31333]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-19T08:01:14.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:13 smithi098 ceph-mon[31333]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-19T08:01:14.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:13 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3577329336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:14.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:13 smithi098 ceph-mon[31333]: pgmap v272: 97 pgs: 64 active+clean, 13 creating+peering, 20 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 4 op/s 2024-09-19T08:01:14.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T08:01:14.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:13 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] up:active 2024-09-19T08:01:14.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:13 smithi162 ceph-mon[35260]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-19T08:01:14.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:13 smithi162 ceph-mon[35260]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-19T08:01:14.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:13 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3577329336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:14.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:13 smithi162 ceph-mon[35260]: pgmap v272: 97 pgs: 64 active+clean, 13 creating+peering, 20 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 4 op/s 2024-09-19T08:01:14.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:13 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] up:active 2024-09-19T08:01:14.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:13 smithi139 ceph-mon[34874]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-19T08:01:14.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:13 smithi139 ceph-mon[34874]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-19T08:01:14.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:13 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3577329336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:14.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:13 smithi139 ceph-mon[34874]: pgmap v272: 97 pgs: 64 active+clean, 13 creating+peering, 20 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 4 op/s 2024-09-19T08:01:14.270 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:14.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.724+0000 7f90e781d640 1 -- 172.21.15.98:0/4103928102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90e0102570 msgr2=0x7f90e01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:14.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.724+0000 7f90e781d640 1 --2- 172.21.15.98:0/4103928102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90e0102570 0x7f90e01029b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009900 tx=0x7f90d002a480 comp rx=0 tx=0).stop 2024-09-19T08:01:14.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.724+0000 7f90e781d640 1 -- 172.21.15.98:0/4103928102 shutdown_connections 2024-09-19T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.724+0000 7f90e781d640 1 --2- 172.21.15.98:0/4103928102 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90e0102ef0 0x7f90e010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.724+0000 7f90e781d640 1 --2- 172.21.15.98:0/4103928102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90e0102570 0x7f90e01029b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.724+0000 7f90e781d640 1 --2- 172.21.15.98:0/4103928102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90e0100e20 0x7f90e0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.724+0000 7f90e781d640 1 -- 172.21.15.98:0/4103928102 >> 172.21.15.98:0/4103928102 conn(0x7f90e00fc7d0 msgr2=0x7f90e00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.724+0000 7f90e781d640 1 -- 172.21.15.98:0/4103928102 shutdown_connections 2024-09-19T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.724+0000 7f90e781d640 1 -- 172.21.15.98:0/4103928102 wait complete. 2024-09-19T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.724+0000 7f90e781d640 1 Processor -- start 2024-09-19T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.724+0000 7f90e781d640 1 -- start start 2024-09-19T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.724+0000 7f90e781d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90e0100e20 0x7f90e01984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e781d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90e0102570 0x7f90e01989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e781d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90e0102ef0 0x7f90e019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e781d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f90e0110f80 con 0x7f90e0100e20 2024-09-19T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e781d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f90e0110e00 con 0x7f90e0102ef0 2024-09-19T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e781d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f90e0111100 con 0x7f90e0102570 2024-09-19T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e55b9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90e0100e20 0x7f90e01984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e55b9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90e0100e20 0x7f90e01984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45906/0 (socket says 172.21.15.98:45906) 2024-09-19T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e55b9640 1 -- 172.21.15.98:0/3767845748 learned_addr learned my addr 172.21.15.98:0/3767845748 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e5dba640 1 --2- 172.21.15.98:0/3767845748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90e0102ef0 0x7f90e019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e55b9640 1 -- 172.21.15.98:0/3767845748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90e0102570 msgr2=0x7f90e01989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e55b9640 1 --2- 172.21.15.98:0/3767845748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90e0102570 0x7f90e01989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e55b9640 1 -- 172.21.15.98:0/3767845748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90e0102ef0 msgr2=0x7f90e019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:14.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e55b9640 1 --2- 172.21.15.98:0/3767845748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90e0102ef0 0x7f90e019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:14.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e55b9640 1 -- 172.21.15.98:0/3767845748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e019d460 con 0x7f90e0100e20 2024-09-19T08:01:14.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e55b9640 1 --2- 172.21.15.98:0/3767845748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90e0100e20 0x7f90e01984b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f90dc00c920 tx=0x7f90dc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:14.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90d67fc640 1 -- 172.21.15.98:0/3767845748 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc007c70 con 0x7f90e0100e20 2024-09-19T08:01:14.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90d67fc640 1 -- 172.21.15.98:0/3767845748 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90dc007e10 con 0x7f90e0100e20 2024-09-19T08:01:14.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e781d640 1 -- 172.21.15.98:0/3767845748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e019d750 con 0x7f90e0100e20 2024-09-19T08:01:14.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90d67fc640 1 -- 172.21.15.98:0/3767845748 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc018890 con 0x7f90e0100e20 2024-09-19T08:01:14.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.725+0000 7f90e781d640 1 -- 172.21.15.98:0/3767845748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e01a5030 con 0x7f90e0100e20 2024-09-19T08:01:14.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.726+0000 7f90e781d640 1 -- 172.21.15.98:0/3767845748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90e0101200 con 0x7f90e0100e20 2024-09-19T08:01:14.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.728+0000 7f90d67fc640 1 -- 172.21.15.98:0/3767845748 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f90dc018a50 con 0x7f90e0100e20 2024-09-19T08:01:14.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.729+0000 7f90d67fc640 1 --2- 172.21.15.98:0/3767845748 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f90cc0a32c0 0x7f90cc0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.729+0000 7f90e4db8640 1 --2- 172.21.15.98:0/3767845748 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f90cc0a32c0 0x7f90cc0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.729+0000 7f90d67fc640 1 -- 172.21.15.98:0/3767845748 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7f90dc0cfbb0 con 0x7f90e0100e20 2024-09-19T08:01:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.729+0000 7f90e4db8640 1 --2- 172.21.15.98:0/3767845748 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f90cc0a32c0 0x7f90cc0a5780 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f90d00096c0 tx=0x7f90d0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.732+0000 7f90d67fc640 1 -- 172.21.15.98:0/3767845748 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f90dc014030 con 0x7f90e0100e20 2024-09-19T08:01:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.903+0000 7f90e781d640 1 -- 172.21.15.98:0/3767845748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90e010a0d0 con 0x7f90e0100e20 2024-09-19T08:01:14.903 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:14 smithi098 ceph-mon[31333]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-19T08:01:14.903 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:14 smithi098 ceph-mon[31333]: Cluster is now healthy 2024-09-19T08:01:14.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.904+0000 7f90d67fc640 1 -- 172.21.15.98:0/3767845748 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22301 (secure 0 0 0) 0x7f90dc09cb30 con 0x7f90e0100e20 2024-09-19T08:01:14.904 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:01:14.904 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":78,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","created":"2024-09-19T07:53:39.127398+0000","modified":"2024-09-19T08:01:11.306885+0000","last_up_change":"2024-09-19T08:00:22.467777+0000","last_in_change":"2024-09-19T07:59:58.538492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:56:43.773892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T08:01:07.693796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T08:01:09.518028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d01ce6-165d-493d-ae98-d8ad9e5d4134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6803","nonce":1719360533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6805","nonce":1719360533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6809","nonce":1719360533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6807","nonce":1719360533}]},"public_addr":"172.21.15.98:6803/1719360533","cluster_addr":"172.21.15.98:6805/1719360533","heartbeat_back_addr":"172.21.15.98:6809/1719360533","heartbeat_front_addr":"172.21.15.98:6807/1719360533","state":["exists","up"]},{"osd":1,"uuid":"34724bd5-0738-4473-b087-c09031c1ee2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036962626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036962626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036962626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036962626}]},"public_addr":"172.21.15.139:6801/4036962626","cluster_addr":"172.21.15.139:6803/4036962626","heartbeat_back_addr":"172.21.15.139:6807/4036962626","heartbeat_front_addr":"172.21.15.139:6805/4036962626","state":["exists","up"]},{"osd":2,"uuid":"16985ebe-0e32-42ee-bded-b7838e290afc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6801","nonce":2811412891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6803","nonce":2811412891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6807","nonce":2811412891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6805","nonce":2811412891}]},"public_addr":"172.21.15.162:6801/2811412891","cluster_addr":"172.21.15.162:6803/2811412891","heartbeat_back_addr":"172.21.15.162:6807/2811412891","heartbeat_front_addr":"172.21.15.162:6805/2811412891","state":["exists","up"]},{"osd":3,"uuid":"b140e466-3119-49cb-aec2-ba6ca4d10d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6811","nonce":1383392014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6813","nonce":1383392014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6817","nonce":1383392014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6815","nonce":1383392014}]},"public_addr":"172.21.15.98:6811/1383392014","cluster_addr":"172.21.15.98:6813/1383392014","heartbeat_back_addr":"172.21.15.98:6817/1383392014","heartbeat_front_addr":"172.21.15.98:6815/1383392014","state":["exists","up"]},{"osd":4,"uuid":"b4412813-df57-4ac6-8fb7-f57b69c74768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6809","nonce":116453520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6811","nonce":116453520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6815","nonce":116453520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6813","nonce":116453520}]},"public_addr":"172.21.15.139:6809/116453520","cluster_addr":"172.21.15.139:6811/116453520","heartbeat_back_addr":"172.21.15.139:6815/116453520","heartbeat_front_addr":"172.21.15.139:6813/116453520","state":["exists","up"]},{"osd":5,"uuid":"d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6809","nonce":3769530297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6811","nonce":3769530297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6815","nonce":3769530297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6813","nonce":3769530297}]},"public_addr":"172.21.15.162:6809/3769530297","cluster_addr":"172.21.15.162:6811/3769530297","heartbeat_back_addr":"172.21.15.162:6815/3769530297","heartbeat_front_addr":"172.21.15.162:6813/3769530297","state":["exists","up"]},{"osd":6,"uuid":"ba1c1590-6aea-40dc-90c8-e21b88896909","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6819","nonce":692373267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6821","nonce":692373267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6825","nonce":692373267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6823","nonce":692373267}]},"public_addr":"172.21.15.98:6819/692373267","cluster_addr":"172.21.15.98:6821/692373267","heartbeat_back_addr":"172.21.15.98:6825/692373267","heartbeat_front_addr":"172.21.15.98:6823/692373267","state":["exists","up"]},{"osd":7,"uuid":"eb9093ad-a068-4a63-ab87-0b50522cbc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6817","nonce":1963811650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6819","nonce":1963811650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6823","nonce":1963811650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6821","nonce":1963811650}]},"public_addr":"172.21.15.139:6817/1963811650","cluster_addr":"172.21.15.139:6819/1963811650","heartbeat_back_addr":"172.21.15.139:6823/1963811650","heartbeat_front_addr":"172.21.15.139:6821/1963811650","state":["exists","up"]},{"osd":8,"uuid":"31ca330a-fc45-466d-81c2-ef4386b3cc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6817","nonce":3607966458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6819","nonce":3607966458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6823","nonce":3607966458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6821","nonce":3607966458}]},"public_addr":"172.21.15.162:6817/3607966458","cluster_addr":"172.21.15.162:6819/3607966458","heartbeat_back_addr":"172.21.15.162:6823/3607966458","heartbeat_front_addr":"172.21.15.162:6821/3607966458","state":["exists","up"]},{"osd":9,"uuid":"9c4f216f-aef1-4367-9c58-b913b0c4f7db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6827","nonce":2050391244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6829","nonce":2050391244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6833","nonce":2050391244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6831","nonce":2050391244}]},"public_addr":"172.21.15.98:6827/2050391244","cluster_addr":"172.21.15.98:6829/2050391244","heartbeat_back_addr":"172.21.15.98:6833/2050391244","heartbeat_front_addr":"172.21.15.98:6831/2050391244","state":["exists","up"]},{"osd":10,"uuid":"03cc7df6-84ae-4d9a-a3d7-74c91e1744e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6825","nonce":2336761639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6827","nonce":2336761639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6831","nonce":2336761639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6829","nonce":2336761639}]},"public_addr":"172.21.15.139:6825/2336761639","cluster_addr":"172.21.15.139:6827/2336761639","heartbeat_back_addr":"172.21.15.139:6831/2336761639","heartbeat_front_addr":"172.21.15.139:6829/2336761639","state":["exists","up"]},{"osd":11,"uuid":"29e95498-4952-46d4-8fa6-6d8e2b862111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6825","nonce":1231011554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6827","nonce":1231011554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6831","nonce":1231011554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6829","nonce":1231011554}]},"public_addr":"172.21.15.162:6825/1231011554","cluster_addr":"172.21.15.162:6827/1231011554","heartbeat_back_addr":"172.21.15.162:6831/1231011554","heartbeat_front_addr":"172.21.15.162:6829/1231011554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:55:49.995244+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:15.655543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:40.331165+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:04.688881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:29.174966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:53.663161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:17.407551+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:42.584296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:06.317219+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:31.078625+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:55.689269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T08:00:19.659748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/3842627255":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3838590070":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/1238193420":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6800/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6800/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3194864054":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3866031018":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/1497156658":"2024-09-20T07:54:02.418785+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T08:01:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.906+0000 7f90e781d640 1 -- 172.21.15.98:0/3767845748 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f90cc0a32c0 msgr2=0x7f90cc0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.906+0000 7f90e781d640 1 --2- 172.21.15.98:0/3767845748 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f90cc0a32c0 0x7f90cc0a5780 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f90d00096c0 tx=0x7f90d0000bc0 comp rx=0 tx=0).stop 2024-09-19T08:01:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.906+0000 7f90e781d640 1 -- 172.21.15.98:0/3767845748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90e0100e20 msgr2=0x7f90e01984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.906+0000 7f90e781d640 1 --2- 172.21.15.98:0/3767845748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90e0100e20 0x7f90e01984b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f90dc00c920 tx=0x7f90dc00cde0 comp rx=0 tx=0).stop 2024-09-19T08:01:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.906+0000 7f90e781d640 1 -- 172.21.15.98:0/3767845748 shutdown_connections 2024-09-19T08:01:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.906+0000 7f90e781d640 1 --2- 172.21.15.98:0/3767845748 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f90cc0a32c0 0x7f90cc0a5780 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.906+0000 7f90e781d640 1 --2- 172.21.15.98:0/3767845748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90e0102ef0 0x7f90e019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.906+0000 7f90e781d640 1 --2- 172.21.15.98:0/3767845748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90e0102570 0x7f90e01989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.906+0000 7f90e781d640 1 --2- 172.21.15.98:0/3767845748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90e0100e20 0x7f90e01984b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:14.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.907+0000 7f90e781d640 1 -- 172.21.15.98:0/3767845748 >> 172.21.15.98:0/3767845748 conn(0x7f90e00fc7d0 msgr2=0x7f90e006a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:14.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.907+0000 7f90e781d640 1 -- 172.21.15.98:0/3767845748 shutdown_connections 2024-09-19T08:01:14.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:14.907+0000 7f90e781d640 1 -- 172.21.15.98:0/3767845748 wait complete. 2024-09-19T08:01:15.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:14 smithi162 ceph-mon[35260]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-19T08:01:15.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:14 smithi162 ceph-mon[35260]: Cluster is now healthy 2024-09-19T08:01:15.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:14 smithi139 ceph-mon[34874]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-19T08:01:15.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:14 smithi139 ceph-mon[34874]: Cluster is now healthy 2024-09-19T08:01:15.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-09-19T08:01:15.763 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:15.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:15 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3767845748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:15.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:15 smithi139 ceph-mon[34874]: pgmap v273: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2024-09-19T08:01:15.928 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:01:15 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-b[65169]: 2024-09-19T08:01:15.652+0000 7f2449f19640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T08:01:16.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:15 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3767845748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:16.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:15 smithi098 ceph-mon[31333]: pgmap v273: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2024-09-19T08:01:16.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:15 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3767845748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:16.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:15 smithi162 ceph-mon[35260]: pgmap v273: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2024-09-19T08:01:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.267+0000 7efcb6989640 1 -- 172.21.15.98:0/2315772920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0106a80 msgr2=0x7efcb0106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.267+0000 7efcb6989640 1 --2- 172.21.15.98:0/2315772920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0106a80 0x7efcb0106e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efca0009930 tx=0x7efca002a420 comp rx=0 tx=0).stop 2024-09-19T08:01:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.267+0000 7efcb6989640 1 -- 172.21.15.98:0/2315772920 shutdown_connections 2024-09-19T08:01:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.267+0000 7efcb6989640 1 --2- 172.21.15.98:0/2315772920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcb0101420 0x7efcb010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.267+0000 7efcb6989640 1 --2- 172.21.15.98:0/2315772920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efcb0100a80 0x7efcb0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.267+0000 7efcb6989640 1 --2- 172.21.15.98:0/2315772920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0106a80 0x7efcb0106e60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.267+0000 7efcb6989640 1 -- 172.21.15.98:0/2315772920 >> 172.21.15.98:0/2315772920 conn(0x7efcb00fc7a0 msgr2=0x7efcb00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.267+0000 7efcb6989640 1 -- 172.21.15.98:0/2315772920 shutdown_connections 2024-09-19T08:01:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.267+0000 7efcb6989640 1 -- 172.21.15.98:0/2315772920 wait complete. 2024-09-19T08:01:16.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcb6989640 1 Processor -- start 2024-09-19T08:01:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcb6989640 1 -- start start 2024-09-19T08:01:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcb6989640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0100a80 0x7efcb01970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcb6989640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efcb0101420 0x7efcb0197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcb6989640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcb0106a80 0x7efcb0197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcb6989640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efcb0110e40 con 0x7efcb0100a80 2024-09-19T08:01:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcb6989640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efcb0110cc0 con 0x7efcb0101420 2024-09-19T08:01:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcb6989640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efcb0110fc0 con 0x7efcb0106a80 2024-09-19T08:01:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcaffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0100a80 0x7efcb01970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcaffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0100a80 0x7efcb01970c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32950/0 (socket says 172.21.15.98:32950) 2024-09-19T08:01:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcaffff640 1 -- 172.21.15.98:0/4006579439 learned_addr learned my addr 172.21.15.98:0/4006579439 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcb4f26640 1 --2- 172.21.15.98:0/4006579439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcb0106a80 0x7efcb0197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:16.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efca7fff640 1 --2- 172.21.15.98:0/4006579439 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efcb0101420 0x7efcb0197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcaffff640 1 -- 172.21.15.98:0/4006579439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcb0106a80 msgr2=0x7efcb0197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcaffff640 1 --2- 172.21.15.98:0/4006579439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcb0106a80 0x7efcb0197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcaffff640 1 -- 172.21.15.98:0/4006579439 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efcb0101420 msgr2=0x7efcb0197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.268+0000 7efcaffff640 1 --2- 172.21.15.98:0/4006579439 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efcb0101420 0x7efcb0197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.269+0000 7efcaffff640 1 -- 172.21.15.98:0/4006579439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb01a6930 con 0x7efcb0100a80 2024-09-19T08:01:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.269+0000 7efcaffff640 1 --2- 172.21.15.98:0/4006579439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0100a80 0x7efcb01970c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efca0009230 tx=0x7efca0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.269+0000 7efcadffb640 1 -- 172.21.15.98:0/4006579439 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca003f030 con 0x7efcb0100a80 2024-09-19T08:01:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.269+0000 7efcadffb640 1 -- 172.21.15.98:0/4006579439 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efca0003ad0 con 0x7efcb0100a80 2024-09-19T08:01:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.269+0000 7efcb6989640 1 -- 172.21.15.98:0/4006579439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcb01a6bc0 con 0x7efcb0100a80 2024-09-19T08:01:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.269+0000 7efcadffb640 1 -- 172.21.15.98:0/4006579439 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca0035070 con 0x7efcb0100a80 2024-09-19T08:01:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.269+0000 7efcb6989640 1 -- 172.21.15.98:0/4006579439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcb01a70a0 con 0x7efcb0100a80 2024-09-19T08:01:16.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.270+0000 7efcb6989640 1 -- 172.21.15.98:0/4006579439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcb0100ee0 con 0x7efcb0100a80 2024-09-19T08:01:16.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.271+0000 7efcadffb640 1 -- 172.21.15.98:0/4006579439 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7efca0042030 con 0x7efcb0100a80 2024-09-19T08:01:16.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.272+0000 7efcadffb640 1 --2- 172.21.15.98:0/4006579439 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7efc9c0a3240 0x7efc9c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:16.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.272+0000 7efca7fff640 1 --2- 172.21.15.98:0/4006579439 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7efc9c0a3240 0x7efc9c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:16.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.272+0000 7efcadffb640 1 -- 172.21.15.98:0/4006579439 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7efca00f0070 con 0x7efcb0100a80 2024-09-19T08:01:16.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.272+0000 7efca7fff640 1 --2- 172.21.15.98:0/4006579439 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7efc9c0a3240 0x7efc9c0a5700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efcb0199220 tx=0x7efc9800a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:16.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.275+0000 7efcadffb640 1 -- 172.21.15.98:0/4006579439 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7efca00bcee0 con 0x7efcb0100a80 2024-09-19T08:01:16.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.477+0000 7efcb6989640 1 -- 172.21.15.98:0/4006579439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7efcb002d7c0 con 0x7efcb0100a80 2024-09-19T08:01:16.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.800+0000 7efcadffb640 1 -- 172.21.15.98:0/4006579439 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7efca00bc7e0 con 0x7efcb0100a80 2024-09-19T08:01:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.802+0000 7efcb6989640 1 -- 172.21.15.98:0/4006579439 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7efc9c0a3240 msgr2=0x7efc9c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.802+0000 7efcb6989640 1 --2- 172.21.15.98:0/4006579439 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7efc9c0a3240 0x7efc9c0a5700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efcb0199220 tx=0x7efc9800a4c0 comp rx=0 tx=0).stop 2024-09-19T08:01:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.802+0000 7efcb6989640 1 -- 172.21.15.98:0/4006579439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0100a80 msgr2=0x7efcb01970c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.802+0000 7efcb6989640 1 --2- 172.21.15.98:0/4006579439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0100a80 0x7efcb01970c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efca0009230 tx=0x7efca0004270 comp rx=0 tx=0).stop 2024-09-19T08:01:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.802+0000 7efcb6989640 1 -- 172.21.15.98:0/4006579439 shutdown_connections 2024-09-19T08:01:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.802+0000 7efcb6989640 1 --2- 172.21.15.98:0/4006579439 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7efc9c0a3240 0x7efc9c0a5700 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.802+0000 7efcb6989640 1 --2- 172.21.15.98:0/4006579439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcb0106a80 0x7efcb0197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.802+0000 7efcb6989640 1 --2- 172.21.15.98:0/4006579439 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efcb0101420 0x7efcb0197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.802+0000 7efcb6989640 1 --2- 172.21.15.98:0/4006579439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0100a80 0x7efcb01970c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.802+0000 7efcb6989640 1 -- 172.21.15.98:0/4006579439 >> 172.21.15.98:0/4006579439 conn(0x7efcb00fc7a0 msgr2=0x7efcb00fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.802+0000 7efcb6989640 1 -- 172.21.15.98:0/4006579439 shutdown_connections 2024-09-19T08:01:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:16.802+0000 7efcb6989640 1 -- 172.21.15.98:0/4006579439 wait complete. 2024-09-19T08:01:17.114 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:16 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/4006579439' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-19T08:01:17.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:16 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/4006579439' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-19T08:01:17.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:16 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/4006579439' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-19T08:01:17.425 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-09-19T08:01:17.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-09-19T08:01:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:18.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:17 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/4006579439' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-19T08:01:18.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:17 smithi098 ceph-mon[31333]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-19T08:01:18.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:17 smithi098 ceph-mon[31333]: pgmap v274: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-09-19T08:01:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:17 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/4006579439' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-19T08:01:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:17 smithi162 ceph-mon[35260]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-19T08:01:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:17 smithi162 ceph-mon[35260]: pgmap v274: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-09-19T08:01:18.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 -- 172.21.15.98:0/704430749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa30c100e20 msgr2=0x7fa30c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:18.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 --2- 172.21.15.98:0/704430749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa30c100e20 0x7fa30c101200 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009900 tx=0x7fa2fc02a480 comp rx=0 tx=0).stop 2024-09-19T08:01:18.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 -- 172.21.15.98:0/704430749 shutdown_connections 2024-09-19T08:01:18.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 --2- 172.21.15.98:0/704430749 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa30c102ef0 0x7fa30c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:18.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 --2- 172.21.15.98:0/704430749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa30c102570 0x7fa30c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:18.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 --2- 172.21.15.98:0/704430749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa30c100e20 0x7fa30c101200 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:18.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 -- 172.21.15.98:0/704430749 >> 172.21.15.98:0/704430749 conn(0x7fa30c0fc7d0 msgr2=0x7fa30c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:18.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 -- 172.21.15.98:0/704430749 shutdown_connections 2024-09-19T08:01:18.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 -- 172.21.15.98:0/704430749 wait complete. 2024-09-19T08:01:18.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 Processor -- start 2024-09-19T08:01:18.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 -- start start 2024-09-19T08:01:18.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa30c100e20 0x7fa30c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:18.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa30c102570 0x7fa30c198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:18.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa30c102ef0 0x7fa30c19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:18.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c111020 con 0x7fa30c102570 2024-09-19T08:01:18.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c110ea0 con 0x7fa30c102ef0 2024-09-19T08:01:18.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.160+0000 7fa3123fd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c1111a0 con 0x7fa30c100e20 2024-09-19T08:01:18.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa303fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa30c102570 0x7fa30c198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:18.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa303fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa30c102570 0x7fa30c198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32972/0 (socket says 172.21.15.98:32972) 2024-09-19T08:01:18.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa303fff640 1 -- 172.21.15.98:0/3807480626 learned_addr learned my addr 172.21.15.98:0/3807480626 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:18.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa31099a640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa30c102ef0 0x7fa30c19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:18.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa30bfff640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa30c100e20 0x7fa30c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:18.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa303fff640 1 -- 172.21.15.98:0/3807480626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa30c100e20 msgr2=0x7fa30c198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:18.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa303fff640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa30c100e20 0x7fa30c198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa303fff640 1 -- 172.21.15.98:0/3807480626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa30c102ef0 msgr2=0x7fa30c19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa303fff640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa30c102ef0 0x7fa30c19cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa303fff640 1 -- 172.21.15.98:0/3807480626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30c19d620 con 0x7fa30c102570 2024-09-19T08:01:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa31099a640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa30c102ef0 0x7fa30c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:01:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa303fff640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa30c102570 0x7fa30c198bb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400b4a0 tx=0x7fa2f400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa309ffb640 1 -- 172.21.15.98:0/3807480626 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4014070 con 0x7fa30c102570 2024-09-19T08:01:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa3123fd640 1 -- 172.21.15.98:0/3807480626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa30c19d910 con 0x7fa30c102570 2024-09-19T08:01:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa309ffb640 1 -- 172.21.15.98:0/3807480626 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2f4010af0 con 0x7fa30c102570 2024-09-19T08:01:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa309ffb640 1 -- 172.21.15.98:0/3807480626 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4017d10 con 0x7fa30c102570 2024-09-19T08:01:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.161+0000 7fa3123fd640 1 -- 172.21.15.98:0/3807480626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa30c1a51f0 con 0x7fa30c102570 2024-09-19T08:01:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.162+0000 7fa3123fd640 1 -- 172.21.15.98:0/3807480626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa30c101280 con 0x7fa30c102570 2024-09-19T08:01:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.163+0000 7fa309ffb640 1 -- 172.21.15.98:0/3807480626 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fa2f4017070 con 0x7fa30c102570 2024-09-19T08:01:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.164+0000 7fa309ffb640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa2f80a3240 0x7fa2f80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.164+0000 7fa30bfff640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa2f80a3240 0x7fa2f80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:18.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.164+0000 7fa309ffb640 1 -- 172.21.15.98:0/3807480626 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7fa2f40cf050 con 0x7fa30c102570 2024-09-19T08:01:18.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.165+0000 7fa30bfff640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa2f80a3240 0x7fa2f80a5700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc0096c0 tx=0x7fa2fc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:18.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.167+0000 7fa309ffb640 1 -- 172.21.15.98:0/3807480626 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fa2f409bec0 con 0x7fa30c102570 2024-09-19T08:01:18.184 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:17 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/4006579439' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-19T08:01:18.184 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:17 smithi139 ceph-mon[34874]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-19T08:01:18.184 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:17 smithi139 ceph-mon[34874]: pgmap v274: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-09-19T08:01:18.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.368+0000 7fa3123fd640 1 -- 172.21.15.98:0/3807480626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fa30c10a0a0 con 0x7fa30c102570 2024-09-19T08:01:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.812+0000 7fa309ffb640 1 -- 172.21.15.98:0/3807480626 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7fa2f409b7c0 con 0x7fa30c102570 2024-09-19T08:01:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.814+0000 7fa3123fd640 1 -- 172.21.15.98:0/3807480626 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa2f80a3240 msgr2=0x7fa2f80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.814+0000 7fa3123fd640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa2f80a3240 0x7fa2f80a5700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc0096c0 tx=0x7fa2fc000bc0 comp rx=0 tx=0).stop 2024-09-19T08:01:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.814+0000 7fa3123fd640 1 -- 172.21.15.98:0/3807480626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa30c102570 msgr2=0x7fa30c198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.814+0000 7fa3123fd640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa30c102570 0x7fa30c198bb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400b4a0 tx=0x7fa2f400b960 comp rx=0 tx=0).stop 2024-09-19T08:01:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.814+0000 7fa3123fd640 1 -- 172.21.15.98:0/3807480626 shutdown_connections 2024-09-19T08:01:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.814+0000 7fa3123fd640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa2f80a3240 0x7fa2f80a5700 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.814+0000 7fa3123fd640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa30c102ef0 0x7fa30c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.814+0000 7fa3123fd640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa30c102570 0x7fa30c198bb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.814+0000 7fa3123fd640 1 --2- 172.21.15.98:0/3807480626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa30c100e20 0x7fa30c198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:18.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.814+0000 7fa3123fd640 1 -- 172.21.15.98:0/3807480626 >> 172.21.15.98:0/3807480626 conn(0x7fa30c0fc7d0 msgr2=0x7fa30c06a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:18.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.814+0000 7fa3123fd640 1 -- 172.21.15.98:0/3807480626 shutdown_connections 2024-09-19T08:01:18.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:18.814+0000 7fa3123fd640 1 -- 172.21.15.98:0/3807480626 wait complete. 2024-09-19T08:01:19.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:18 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3807480626' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-19T08:01:19.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:18 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3807480626' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-19T08:01:19.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:18 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3807480626' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-19T08:01:19.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-09-19T08:01:19.921 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:19.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:19 smithi098 ceph-mon[31333]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-19T08:01:19.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:19 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3807480626' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-19T08:01:19.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:19 smithi098 ceph-mon[31333]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-19T08:01:19.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:19 smithi098 ceph-mon[31333]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-19T08:01:19.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:19 smithi098 ceph-mon[31333]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-09-19T08:01:19.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:19 smithi098 ceph-mon[31333]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-09-19T08:01:19.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:19 smithi098 ceph-mon[31333]: pgmap v275: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-09-19T08:01:20.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:19 smithi162 ceph-mon[35260]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-19T08:01:20.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:19 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3807480626' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-19T08:01:20.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:19 smithi162 ceph-mon[35260]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-19T08:01:20.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:19 smithi162 ceph-mon[35260]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-19T08:01:20.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:19 smithi162 ceph-mon[35260]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-09-19T08:01:20.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:19 smithi162 ceph-mon[35260]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-09-19T08:01:20.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:19 smithi162 ceph-mon[35260]: pgmap v275: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-09-19T08:01:20.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:19 smithi139 ceph-mon[34874]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-19T08:01:20.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:19 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3807480626' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-19T08:01:20.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:19 smithi139 ceph-mon[34874]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-19T08:01:20.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:19 smithi139 ceph-mon[34874]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-19T08:01:20.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:19 smithi139 ceph-mon[34874]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-09-19T08:01:20.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:19 smithi139 ceph-mon[34874]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-09-19T08:01:20.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:19 smithi139 ceph-mon[34874]: pgmap v275: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-09-19T08:01:20.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.436+0000 7faca0972640 1 -- 172.21.15.98:0/4087838667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac9c101fc0 msgr2=0x7fac9c1056d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:20.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.436+0000 7faca0972640 1 --2- 172.21.15.98:0/4087838667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac9c101fc0 0x7fac9c1056d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fac90009900 tx=0x7fac9002a410 comp rx=0 tx=0).stop 2024-09-19T08:01:20.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.436+0000 7faca0972640 1 -- 172.21.15.98:0/4087838667 shutdown_connections 2024-09-19T08:01:20.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.436+0000 7faca0972640 1 --2- 172.21.15.98:0/4087838667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac9c105e00 0x7fac9c10d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.436+0000 7faca0972640 1 --2- 172.21.15.98:0/4087838667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac9c101fc0 0x7fac9c1056d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.436+0000 7faca0972640 1 --2- 172.21.15.98:0/4087838667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac9c101610 0x7fac9c1019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.436+0000 7faca0972640 1 -- 172.21.15.98:0/4087838667 >> 172.21.15.98:0/4087838667 conn(0x7fac9c0fc5c0 msgr2=0x7fac9c0fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.436+0000 7faca0972640 1 -- 172.21.15.98:0/4087838667 shutdown_connections 2024-09-19T08:01:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.436+0000 7faca0972640 1 -- 172.21.15.98:0/4087838667 wait complete. 2024-09-19T08:01:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.437+0000 7faca0972640 1 Processor -- start 2024-09-19T08:01:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.437+0000 7faca0972640 1 -- start start 2024-09-19T08:01:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.437+0000 7faca0972640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac9c101610 0x7fac9c0fca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.437+0000 7faca0972640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac9c101fc0 0x7fac9c0fcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.437+0000 7faca0972640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac9c105e00 0x7fac9c0fd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.437+0000 7faca0972640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac9c110e80 con 0x7fac9c101fc0 2024-09-19T08:01:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.437+0000 7faca0972640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fac9c110d00 con 0x7fac9c101610 2024-09-19T08:01:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.437+0000 7faca0972640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fac9c111000 con 0x7fac9c105e00 2024-09-19T08:01:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.437+0000 7fac99d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac9c101fc0 0x7fac9c0fcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.437+0000 7fac99d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac9c101fc0 0x7fac9c0fcfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32996/0 (socket says 172.21.15.98:32996) 2024-09-19T08:01:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.437+0000 7fac99d9b640 1 -- 172.21.15.98:0/3850511580 learned_addr learned my addr 172.21.15.98:0/3850511580 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.438+0000 7fac9ad9d640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac9c105e00 0x7fac9c0fd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.438+0000 7fac9a59c640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac9c101610 0x7fac9c0fca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.438+0000 7fac99d9b640 1 -- 172.21.15.98:0/3850511580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac9c105e00 msgr2=0x7fac9c0fd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.438+0000 7fac99d9b640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac9c105e00 0x7fac9c0fd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.438+0000 7fac99d9b640 1 -- 172.21.15.98:0/3850511580 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac9c101610 msgr2=0x7fac9c0fca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.438+0000 7fac99d9b640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac9c101610 0x7fac9c0fca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.438+0000 7fac99d9b640 1 -- 172.21.15.98:0/3850511580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac9c1a6930 con 0x7fac9c101fc0 2024-09-19T08:01:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.438+0000 7fac9a59c640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac9c101610 0x7fac9c0fca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:01:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.438+0000 7fac99d9b640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac9c101fc0 0x7fac9c0fcfb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fac90006010 tx=0x7fac900042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.438+0000 7fac8b7fe640 1 -- 172.21.15.98:0/3850511580 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9003f030 con 0x7fac9c101fc0 2024-09-19T08:01:20.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.438+0000 7fac8b7fe640 1 -- 172.21.15.98:0/3850511580 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac90003ce0 con 0x7fac9c101fc0 2024-09-19T08:01:20.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.438+0000 7fac8b7fe640 1 -- 172.21.15.98:0/3850511580 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac90035070 con 0x7fac9c101fc0 2024-09-19T08:01:20.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.438+0000 7faca0972640 1 -- 172.21.15.98:0/3850511580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac9c1a6bc0 con 0x7fac9c101fc0 2024-09-19T08:01:20.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.438+0000 7faca0972640 1 -- 172.21.15.98:0/3850511580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac9c1a70f0 con 0x7fac9c101fc0 2024-09-19T08:01:20.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.439+0000 7faca0972640 1 -- 172.21.15.98:0/3850511580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac9c1019f0 con 0x7fac9c101fc0 2024-09-19T08:01:20.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.440+0000 7fac8b7fe640 1 -- 172.21.15.98:0/3850511580 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fac90042030 con 0x7fac9c101fc0 2024-09-19T08:01:20.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.441+0000 7fac8b7fe640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fac840a32c0 0x7fac840a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:20.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.441+0000 7fac9a59c640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fac840a32c0 0x7fac840a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:20.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.441+0000 7fac8b7fe640 1 -- 172.21.15.98:0/3850511580 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7fac900efd60 con 0x7fac9c101fc0 2024-09-19T08:01:20.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.441+0000 7fac9a59c640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fac840a32c0 0x7fac840a5780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fac8c006fd0 tx=0x7fac8c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:20.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.445+0000 7fac8b7fe640 1 -- 172.21.15.98:0/3850511580 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fac900bcbd0 con 0x7fac9c101fc0 2024-09-19T08:01:20.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.646+0000 7faca0972640 1 -- 172.21.15.98:0/3850511580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fac9c1a72c0 con 0x7fac9c101fc0 2024-09-19T08:01:20.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.676+0000 7fac8b7fe640 1 -- 172.21.15.98:0/3850511580 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v22) ==== 121+0+0 (secure 0 0 0) 0x7fac900bc4d0 con 0x7fac9c101fc0 2024-09-19T08:01:20.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.678+0000 7faca0972640 1 -- 172.21.15.98:0/3850511580 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fac840a32c0 msgr2=0x7fac840a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:20.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.678+0000 7faca0972640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fac840a32c0 0x7fac840a5780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fac8c006fd0 tx=0x7fac8c008040 comp rx=0 tx=0).stop 2024-09-19T08:01:20.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.678+0000 7faca0972640 1 -- 172.21.15.98:0/3850511580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac9c101fc0 msgr2=0x7fac9c0fcfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:20.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.678+0000 7faca0972640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac9c101fc0 0x7fac9c0fcfb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fac90006010 tx=0x7fac900042f0 comp rx=0 tx=0).stop 2024-09-19T08:01:20.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.679+0000 7faca0972640 1 -- 172.21.15.98:0/3850511580 shutdown_connections 2024-09-19T08:01:20.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.679+0000 7faca0972640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fac840a32c0 0x7fac840a5780 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:20.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.679+0000 7faca0972640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac9c105e00 0x7fac9c0fd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:20.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.679+0000 7faca0972640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac9c101fc0 0x7fac9c0fcfb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:20.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.679+0000 7faca0972640 1 --2- 172.21.15.98:0/3850511580 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac9c101610 0x7fac9c0fca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:20.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.679+0000 7faca0972640 1 -- 172.21.15.98:0/3850511580 >> 172.21.15.98:0/3850511580 conn(0x7fac9c0fc5c0 msgr2=0x7fac9c100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:20.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.679+0000 7faca0972640 1 -- 172.21.15.98:0/3850511580 shutdown_connections 2024-09-19T08:01:20.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:20.679+0000 7faca0972640 1 -- 172.21.15.98:0/3850511580 wait complete. 2024-09-19T08:01:20.927 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:01:20 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-b[65169]: 2024-09-19T08:01:20.653+0000 7f2449f19640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-19T08:01:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:20 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] up:active 2024-09-19T08:01:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:20 smithi139 ceph-mon[34874]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-19T08:01:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:20 smithi139 ceph-mon[34874]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-09-19T08:01:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:20 smithi139 ceph-mon[34874]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-09-19T08:01:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:20 smithi139 ceph-mon[34874]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-09-19T08:01:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:20 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3850511580' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-19T08:01:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:20 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3850511580' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-19T08:01:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:20 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] up:active 2024-09-19T08:01:20.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:20 smithi139 ceph-mon[34874]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-19T08:01:20.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:20 smithi139 ceph-mon[34874]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-09-19T08:01:20.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:20 smithi139 ceph-mon[34874]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:creating} 3 up:standby-replay 5 up:standby 2024-09-19T08:01:20.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:20 smithi139 ceph-mon[34874]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-09-19T08:01:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:20 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] up:active 2024-09-19T08:01:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:20 smithi098 ceph-mon[31333]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-19T08:01:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:20 smithi098 ceph-mon[31333]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-09-19T08:01:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:20 smithi098 ceph-mon[31333]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-09-19T08:01:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:20 smithi098 ceph-mon[31333]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-09-19T08:01:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:20 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3850511580' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-19T08:01:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:20 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3850511580' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-19T08:01:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:20 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] up:active 2024-09-19T08:01:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:20 smithi098 ceph-mon[31333]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-19T08:01:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:20 smithi098 ceph-mon[31333]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-09-19T08:01:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:20 smithi098 ceph-mon[31333]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:creating} 3 up:standby-replay 5 up:standby 2024-09-19T08:01:20.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:20 smithi098 ceph-mon[31333]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-09-19T08:01:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:20 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] up:active 2024-09-19T08:01:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:20 smithi162 ceph-mon[35260]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-19T08:01:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:20 smithi162 ceph-mon[35260]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-09-19T08:01:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:20 smithi162 ceph-mon[35260]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-09-19T08:01:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:20 smithi162 ceph-mon[35260]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-09-19T08:01:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:20 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3850511580' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-19T08:01:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:20 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3850511580' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-19T08:01:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:20 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] up:active 2024-09-19T08:01:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:20 smithi162 ceph-mon[35260]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-19T08:01:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:20 smithi162 ceph-mon[35260]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-09-19T08:01:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:20 smithi162 ceph-mon[35260]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:creating} 3 up:standby-replay 5 up:standby 2024-09-19T08:01:21.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:20 smithi162 ceph-mon[35260]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-09-19T08:01:21.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-09-19T08:01:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:22.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.042+0000 7f741a187640 1 -- 172.21.15.98:0/3150176555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f741410c170 msgr2=0x7f741410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.042+0000 7f741a187640 1 --2- 172.21.15.98:0/3150176555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f741410c170 0x7f741410e610 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f740c009ca0 tx=0x7f740c02a950 comp rx=0 tx=0).stop 2024-09-19T08:01:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.042+0000 7f741a187640 1 -- 172.21.15.98:0/3150176555 shutdown_connections 2024-09-19T08:01:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.042+0000 7f741a187640 1 --2- 172.21.15.98:0/3150176555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f741410c170 0x7f741410e610 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.042+0000 7f741a187640 1 --2- 172.21.15.98:0/3150176555 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7414103660 0x7f741410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.042+0000 7f741a187640 1 --2- 172.21.15.98:0/3150176555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7414102d40 0x7f7414103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.042+0000 7f741a187640 1 -- 172.21.15.98:0/3150176555 >> 172.21.15.98:0/3150176555 conn(0x7f74140fc7a0 msgr2=0x7f74140febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.042+0000 7f741a187640 1 -- 172.21.15.98:0/3150176555 shutdown_connections 2024-09-19T08:01:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.042+0000 7f741a187640 1 -- 172.21.15.98:0/3150176555 wait complete. 2024-09-19T08:01:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.043+0000 7f741a187640 1 Processor -- start 2024-09-19T08:01:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.043+0000 7f741a187640 1 -- start start 2024-09-19T08:01:22.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.043+0000 7f741a187640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7414102d40 0x7f74141a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.043+0000 7f741a187640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7414103660 0x7f74141a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.043+0000 7f741a187640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f741410c170 0x7f741419a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.043+0000 7f741a187640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7414112030 con 0x7f741410c170 2024-09-19T08:01:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.043+0000 7f741a187640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7414111eb0 con 0x7f7414102d40 2024-09-19T08:01:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.043+0000 7f741a187640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f74141121b0 con 0x7f7414103660 2024-09-19T08:01:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.043+0000 7f7413fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f741410c170 0x7f741419a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.043+0000 7f7413fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f741410c170 0x7f741419a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33020/0 (socket says 172.21.15.98:33020) 2024-09-19T08:01:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.043+0000 7f7413fff640 1 -- 172.21.15.98:0/1170805613 learned_addr learned my addr 172.21.15.98:0/1170805613 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.043+0000 7f7412ffd640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7414103660 0x7f74141a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.043+0000 7f74137fe640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7414102d40 0x7f74141a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.044+0000 7f7413fff640 1 -- 172.21.15.98:0/1170805613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7414103660 msgr2=0x7f74141a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.044+0000 7f7413fff640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7414103660 0x7f74141a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.044+0000 7f7413fff640 1 -- 172.21.15.98:0/1170805613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7414102d40 msgr2=0x7f74141a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.044+0000 7f7413fff640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7414102d40 0x7f74141a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.044+0000 7f7413fff640 1 -- 172.21.15.98:0/1170805613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741419acd0 con 0x7f741410c170 2024-09-19T08:01:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.044+0000 7f7412ffd640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7414103660 0x7f74141a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:01:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.044+0000 7f7413fff640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f741410c170 0x7f741419a5a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f740c006010 tx=0x7f740c02f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.044+0000 7f7410ff9640 1 -- 172.21.15.98:0/1170805613 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c034040 con 0x7f741410c170 2024-09-19T08:01:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.044+0000 7f7410ff9640 1 -- 172.21.15.98:0/1170805613 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f740c003e00 con 0x7f741410c170 2024-09-19T08:01:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.044+0000 7f741a187640 1 -- 172.21.15.98:0/1170805613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f741419af60 con 0x7f741410c170 2024-09-19T08:01:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.044+0000 7f7410ff9640 1 -- 172.21.15.98:0/1170805613 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c0456e0 con 0x7f741410c170 2024-09-19T08:01:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.044+0000 7f741a187640 1 -- 172.21.15.98:0/1170805613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74141a72b0 con 0x7f741410c170 2024-09-19T08:01:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.045+0000 7f741a187640 1 -- 172.21.15.98:0/1170805613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74141031a0 con 0x7f741410c170 2024-09-19T08:01:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.046+0000 7f7410ff9640 1 -- 172.21.15.98:0/1170805613 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f740c037020 con 0x7f741410c170 2024-09-19T08:01:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.047+0000 7f7410ff9640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f73fc0a3240 0x7f73fc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.047+0000 7f74137fe640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f73fc0a3240 0x7f73fc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.047+0000 7f7410ff9640 1 -- 172.21.15.98:0/1170805613 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7f740c0f0a30 con 0x7f741410c170 2024-09-19T08:01:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.047+0000 7f74137fe640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f73fc0a3240 0x7f73fc0a5700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7404009890 tx=0x7f7404008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.050+0000 7f7410ff9640 1 -- 172.21.15.98:0/1170805613 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f740c0bd930 con 0x7f741410c170 2024-09-19T08:01:22.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.254+0000 7f741a187640 1 -- 172.21.15.98:0/1170805613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f74141a7480 con 0x7f741410c170 2024-09-19T08:01:22.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.695+0000 7f7410ff9640 1 -- 172.21.15.98:0/1170805613 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v26) ==== 115+0+0 (secure 0 0 0) 0x7f740c0bd230 con 0x7f741410c170 2024-09-19T08:01:22.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.698+0000 7f741a187640 1 -- 172.21.15.98:0/1170805613 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f73fc0a3240 msgr2=0x7f73fc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:22.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.698+0000 7f741a187640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f73fc0a3240 0x7f73fc0a5700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7404009890 tx=0x7f7404008040 comp rx=0 tx=0).stop 2024-09-19T08:01:22.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.698+0000 7f741a187640 1 -- 172.21.15.98:0/1170805613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f741410c170 msgr2=0x7f741419a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:22.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.698+0000 7f741a187640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f741410c170 0x7f741419a5a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f740c006010 tx=0x7f740c02f710 comp rx=0 tx=0).stop 2024-09-19T08:01:22.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.698+0000 7f741a187640 1 -- 172.21.15.98:0/1170805613 shutdown_connections 2024-09-19T08:01:22.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.698+0000 7f741a187640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f73fc0a3240 0x7f73fc0a5700 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:22.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.698+0000 7f741a187640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f741410c170 0x7f741419a5a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:22.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.698+0000 7f741a187640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7414103660 0x7f74141a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:22.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.698+0000 7f741a187640 1 --2- 172.21.15.98:0/1170805613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7414102d40 0x7f74141a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:22.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.698+0000 7f741a187640 1 -- 172.21.15.98:0/1170805613 >> 172.21.15.98:0/1170805613 conn(0x7f74140fc7a0 msgr2=0x7f74141008d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:22.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.698+0000 7f741a187640 1 -- 172.21.15.98:0/1170805613 shutdown_connections 2024-09-19T08:01:22.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:22.698+0000 7f741a187640 1 -- 172.21.15.98:0/1170805613 wait complete. 2024-09-19T08:01:23.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:22 smithi098 ceph-mon[31333]: pgmap v276: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-09-19T08:01:23.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:22 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] up:active 2024-09-19T08:01:23.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:22 smithi098 ceph-mon[31333]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-19T08:01:23.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:22 smithi098 ceph-mon[31333]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-19T08:01:23.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:22 smithi098 ceph-mon[31333]: Cluster is now healthy 2024-09-19T08:01:23.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:22 smithi098 ceph-mon[31333]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:active} 3 up:standby-replay 5 up:standby 2024-09-19T08:01:23.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:22 smithi098 ceph-mon[31333]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-09-19T08:01:23.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:22 smithi098 ceph-mon[31333]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-09-19T08:01:23.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:22 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1170805613' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-19T08:01:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:22 smithi162 ceph-mon[35260]: pgmap v276: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-09-19T08:01:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:22 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] up:active 2024-09-19T08:01:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:22 smithi162 ceph-mon[35260]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-19T08:01:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:22 smithi162 ceph-mon[35260]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-19T08:01:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:22 smithi162 ceph-mon[35260]: Cluster is now healthy 2024-09-19T08:01:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:22 smithi162 ceph-mon[35260]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:active} 3 up:standby-replay 5 up:standby 2024-09-19T08:01:23.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:22 smithi162 ceph-mon[35260]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-09-19T08:01:23.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:22 smithi162 ceph-mon[35260]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-09-19T08:01:23.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:22 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1170805613' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-19T08:01:23.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:22 smithi139 ceph-mon[34874]: pgmap v276: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-09-19T08:01:23.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:22 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] up:active 2024-09-19T08:01:23.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:22 smithi139 ceph-mon[34874]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-19T08:01:23.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:22 smithi139 ceph-mon[34874]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-19T08:01:23.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:22 smithi139 ceph-mon[34874]: Cluster is now healthy 2024-09-19T08:01:23.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:22 smithi139 ceph-mon[34874]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:active} 3 up:standby-replay 5 up:standby 2024-09-19T08:01:23.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:22 smithi139 ceph-mon[34874]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-09-19T08:01:23.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:22 smithi139 ceph-mon[34874]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-09-19T08:01:23.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:22 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1170805613' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-19T08:01:23.311 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-09-19T08:01:23.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-09-19T08:01:23.559 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:23.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:23 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1170805613' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-19T08:01:23.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:23 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] up:active 2024-09-19T08:01:23.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:23 smithi098 ceph-mon[31333]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-09-19T08:01:23.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:23 smithi098 ceph-mon[31333]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.058+0000 7fbffcf22640 1 -- 172.21.15.98:0/1810850479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbff8100a80 msgr2=0x7fbff8100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.058+0000 7fbffcf22640 1 --2- 172.21.15.98:0/1810850479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbff8100a80 0x7fbff8100ee0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8009930 tx=0x7fbfe802a420 comp rx=0 tx=0).stop 2024-09-19T08:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.058+0000 7fbffcf22640 1 -- 172.21.15.98:0/1810850479 shutdown_connections 2024-09-19T08:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.058+0000 7fbffcf22640 1 --2- 172.21.15.98:0/1810850479 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbff8101420 0x7fbff810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.058+0000 7fbffcf22640 1 --2- 172.21.15.98:0/1810850479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbff8100a80 0x7fbff8100ee0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.058+0000 7fbffcf22640 1 --2- 172.21.15.98:0/1810850479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbff8106a80 0x7fbff8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.058+0000 7fbffcf22640 1 -- 172.21.15.98:0/1810850479 >> 172.21.15.98:0/1810850479 conn(0x7fbff80fc7a0 msgr2=0x7fbff80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.058+0000 7fbffcf22640 1 -- 172.21.15.98:0/1810850479 shutdown_connections 2024-09-19T08:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.058+0000 7fbffcf22640 1 -- 172.21.15.98:0/1810850479 wait complete. 2024-09-19T08:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.059+0000 7fbffcf22640 1 Processor -- start 2024-09-19T08:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.059+0000 7fbffcf22640 1 -- start start 2024-09-19T08:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.059+0000 7fbffcf22640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbff8100a80 0x7fbff8199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.059+0000 7fbffcf22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbff8101420 0x7fbff8199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.059+0000 7fbffcf22640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbff8106a80 0x7fbff81a43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.059+0000 7fbffcf22640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbff8110f10 con 0x7fbff8106a80 2024-09-19T08:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.059+0000 7fbffcf22640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbff8110d90 con 0x7fbff8100a80 2024-09-19T08:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.059+0000 7fbffcf22640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbff8111090 con 0x7fbff8101420 2024-09-19T08:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.059+0000 7fbff6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbff8106a80 0x7fbff81a43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.059+0000 7fbff6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbff8106a80 0x7fbff81a43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33032/0 (socket says 172.21.15.98:33032) 2024-09-19T08:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.059+0000 7fbff6d9d640 1 -- 172.21.15.98:0/2835036417 learned_addr learned my addr 172.21.15.98:0/2835036417 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.060+0000 7fbff5d9b640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbff8101420 0x7fbff8199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.060+0000 7fbff659c640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbff8100a80 0x7fbff8199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:24.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.060+0000 7fbff6d9d640 1 -- 172.21.15.98:0/2835036417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbff8101420 msgr2=0x7fbff8199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:24.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.060+0000 7fbff6d9d640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbff8101420 0x7fbff8199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:24.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.060+0000 7fbff6d9d640 1 -- 172.21.15.98:0/2835036417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbff8100a80 msgr2=0x7fbff8199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:24.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.060+0000 7fbff6d9d640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbff8100a80 0x7fbff8199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:24.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.060+0000 7fbff6d9d640 1 -- 172.21.15.98:0/2835036417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff81a4900 con 0x7fbff8106a80 2024-09-19T08:01:24.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.060+0000 7fbff659c640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbff8100a80 0x7fbff8199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:01:24.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.060+0000 7fbff6d9d640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbff8106a80 0x7fbff81a43c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbff000b970 tx=0x7fbff000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:24.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.060+0000 7fbfef7fe640 1 -- 172.21.15.98:0/2835036417 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff0019070 con 0x7fbff8106a80 2024-09-19T08:01:24.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.060+0000 7fbfef7fe640 1 -- 172.21.15.98:0/2835036417 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff000cb40 con 0x7fbff8106a80 2024-09-19T08:01:24.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.060+0000 7fbffcf22640 1 -- 172.21.15.98:0/2835036417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff81a4ad0 con 0x7fbff8106a80 2024-09-19T08:01:24.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.060+0000 7fbfef7fe640 1 -- 172.21.15.98:0/2835036417 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff0007650 con 0x7fbff8106a80 2024-09-19T08:01:24.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.060+0000 7fbffcf22640 1 -- 172.21.15.98:0/2835036417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff81a4f90 con 0x7fbff8106a80 2024-09-19T08:01:24.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.061+0000 7fbffcf22640 1 -- 172.21.15.98:0/2835036417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff8100ee0 con 0x7fbff8106a80 2024-09-19T08:01:24.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.062+0000 7fbfef7fe640 1 -- 172.21.15.98:0/2835036417 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fbff001c070 con 0x7fbff8106a80 2024-09-19T08:01:24.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.063+0000 7fbfef7fe640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbfe40a3240 0x7fbfe40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:24.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.063+0000 7fbff659c640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbfe40a3240 0x7fbfe40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:24.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.063+0000 7fbfef7fe640 1 -- 172.21.15.98:0/2835036417 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7fbff00ceda0 con 0x7fbff8106a80 2024-09-19T08:01:24.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.063+0000 7fbff659c640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbfe40a3240 0x7fbfe40a5700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0005fd0 tx=0x7fbfe0005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:24.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.067+0000 7fbfef7fe640 1 -- 172.21.15.98:0/2835036417 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fbff009b5c0 con 0x7fbff8106a80 2024-09-19T08:01:24.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:23 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1170805613' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-19T08:01:24.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:23 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] up:active 2024-09-19T08:01:24.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:23 smithi162 ceph-mon[35260]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-09-19T08:01:24.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:23 smithi162 ceph-mon[35260]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:01:24.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:23 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1170805613' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-19T08:01:24.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:23 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] up:active 2024-09-19T08:01:24.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:23 smithi139 ceph-mon[34874]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-09-19T08:01:24.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:23 smithi139 ceph-mon[34874]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:01:24.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.253+0000 7fbffcf22640 1 -- 172.21.15.98:0/2835036417 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fbff80fd550 con 0x7fbfe40a3240 2024-09-19T08:01:24.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.281+0000 7fbfef7fe640 1 -- 172.21.15.98:0/2835036417 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbff80fd550 con 0x7fbfe40a3240 2024-09-19T08:01:24.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.283+0000 7fbffcf22640 1 -- 172.21.15.98:0/2835036417 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbfe40a3240 msgr2=0x7fbfe40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:24.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.283+0000 7fbffcf22640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbfe40a3240 0x7fbfe40a5700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0005fd0 tx=0x7fbfe0005ea0 comp rx=0 tx=0).stop 2024-09-19T08:01:24.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.283+0000 7fbffcf22640 1 -- 172.21.15.98:0/2835036417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbff8106a80 msgr2=0x7fbff81a43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:24.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.283+0000 7fbffcf22640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbff8106a80 0x7fbff81a43c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbff000b970 tx=0x7fbff000be30 comp rx=0 tx=0).stop 2024-09-19T08:01:24.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.283+0000 7fbffcf22640 1 -- 172.21.15.98:0/2835036417 shutdown_connections 2024-09-19T08:01:24.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.283+0000 7fbffcf22640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbfe40a3240 0x7fbfe40a5700 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:24.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.283+0000 7fbffcf22640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbff8106a80 0x7fbff81a43c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:24.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.283+0000 7fbffcf22640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbff8101420 0x7fbff8199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:24.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.283+0000 7fbffcf22640 1 --2- 172.21.15.98:0/2835036417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbff8100a80 0x7fbff8199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:24.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.283+0000 7fbffcf22640 1 -- 172.21.15.98:0/2835036417 >> 172.21.15.98:0/2835036417 conn(0x7fbff80fc7a0 msgr2=0x7fbff80fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:24.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.283+0000 7fbffcf22640 1 -- 172.21.15.98:0/2835036417 shutdown_connections 2024-09-19T08:01:24.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:24.283+0000 7fbffcf22640 1 -- 172.21.15.98:0/2835036417 wait complete. 2024-09-19T08:01:24.682 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:01:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:01:24.258+0000 7f691ab26640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:01:24.682 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:01:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:01:24.258+0000 7f691ab26640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:01:24.682 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:01:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:01:24.258+0000 7f691ab26640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:01:24.682 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:01:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:01:24.258+0000 7f691ab26640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:01:24.682 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:01:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:01:24.258+0000 7f691ab26640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:01:24.940 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-09-19T08:01:24.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:24 smithi098 ceph-mon[31333]: pgmap v277: 97 pgs: 97 active+clean; 583 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 4.2 KiB/s wr, 13 op/s 2024-09-19T08:01:25.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:24 smithi162 ceph-mon[35260]: pgmap v277: 97 pgs: 97 active+clean; 583 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 4.2 KiB/s wr, 13 op/s 2024-09-19T08:01:25.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:24 smithi139 ceph-mon[34874]: pgmap v277: 97 pgs: 97 active+clean; 583 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 4.2 KiB/s wr, 13 op/s 2024-09-19T08:01:25.185 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.628+0000 7f6ef7c00640 1 -- 172.21.15.98:0/262316388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0074eb0 msgr2=0x7f6ef0112650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.628+0000 7f6ef7c00640 1 --2- 172.21.15.98:0/262316388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0074eb0 0x7f6ef0112650 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009900 tx=0x7f6ee002a480 comp rx=0 tx=0).stop 2024-09-19T08:01:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.628+0000 7f6ef7c00640 1 -- 172.21.15.98:0/262316388 shutdown_connections 2024-09-19T08:01:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.628+0000 7f6ef7c00640 1 --2- 172.21.15.98:0/262316388 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ef0112b90 0x7f6ef0114f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.628+0000 7f6ef7c00640 1 --2- 172.21.15.98:0/262316388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0074eb0 0x7f6ef0112650 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.628+0000 7f6ef7c00640 1 --2- 172.21.15.98:0/262316388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ef0074590 0x7f6ef0074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.628+0000 7f6ef7c00640 1 -- 172.21.15.98:0/262316388 >> 172.21.15.98:0/262316388 conn(0x7f6ef00fd8c0 msgr2=0x7f6ef00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.628+0000 7f6ef7c00640 1 -- 172.21.15.98:0/262316388 shutdown_connections 2024-09-19T08:01:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.628+0000 7f6ef7c00640 1 -- 172.21.15.98:0/262316388 wait complete. 2024-09-19T08:01:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.629+0000 7f6ef7c00640 1 Processor -- start 2024-09-19T08:01:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.629+0000 7f6ef7c00640 1 -- start start 2024-09-19T08:01:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.629+0000 7f6ef7c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0074590 0x7f6ef01a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.629+0000 7f6ef7c00640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ef0074eb0 0x7f6ef01a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.629+0000 7f6ef7c00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ef0112b90 0x7f6ef019e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.629+0000 7f6ef7c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef01185d0 con 0x7f6ef0074590 2024-09-19T08:01:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.629+0000 7f6ef7c00640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef0118450 con 0x7f6ef0074eb0 2024-09-19T08:01:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.629+0000 7f6ef7c00640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef0118750 con 0x7f6ef0112b90 2024-09-19T08:01:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.629+0000 7f6ef599c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0074590 0x7f6ef01a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.629+0000 7f6ef599c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0074590 0x7f6ef01a4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33066/0 (socket says 172.21.15.98:33066) 2024-09-19T08:01:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.629+0000 7f6ef599c640 1 -- 172.21.15.98:0/3648279465 learned_addr learned my addr 172.21.15.98:0/3648279465 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.629+0000 7f6ef519b640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ef0074eb0 0x7f6ef01a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.630+0000 7f6ef619d640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ef0112b90 0x7f6ef019e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.630+0000 7f6ef599c640 1 -- 172.21.15.98:0/3648279465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ef0112b90 msgr2=0x7f6ef019e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.630+0000 7f6ef599c640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ef0112b90 0x7f6ef019e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.630+0000 7f6ef599c640 1 -- 172.21.15.98:0/3648279465 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ef0074eb0 msgr2=0x7f6ef01a4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.630+0000 7f6ef599c640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ef0074eb0 0x7f6ef01a4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.630+0000 7f6ef599c640 1 -- 172.21.15.98:0/3648279465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef019f010 con 0x7f6ef0074590 2024-09-19T08:01:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.630+0000 7f6ef619d640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ef0112b90 0x7f6ef019e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:01:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.630+0000 7f6ef599c640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0074590 0x7f6ef01a4720 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6eec00c920 tx=0x7f6eec00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.630+0000 7f6ee6ffd640 1 -- 172.21.15.98:0/3648279465 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec007c70 con 0x7f6ef0074590 2024-09-19T08:01:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.630+0000 7f6ee6ffd640 1 -- 172.21.15.98:0/3648279465 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eec007e10 con 0x7f6ef0074590 2024-09-19T08:01:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.630+0000 7f6ef7c00640 1 -- 172.21.15.98:0/3648279465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef019f300 con 0x7f6ef0074590 2024-09-19T08:01:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.630+0000 7f6ee6ffd640 1 -- 172.21.15.98:0/3648279465 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec018890 con 0x7f6ef0074590 2024-09-19T08:01:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.630+0000 7f6ef7c00640 1 -- 172.21.15.98:0/3648279465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef01ab5f0 con 0x7f6ef0074590 2024-09-19T08:01:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.631+0000 7f6ef7c00640 1 -- 172.21.15.98:0/3648279465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ef0074970 con 0x7f6ef0074590 2024-09-19T08:01:25.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.632+0000 7f6ee6ffd640 1 -- 172.21.15.98:0/3648279465 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f6eec018a50 con 0x7f6ef0074590 2024-09-19T08:01:25.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.633+0000 7f6ee6ffd640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f6edc0a3240 0x7f6edc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:25.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.633+0000 7f6ef519b640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f6edc0a3240 0x7f6edc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:25.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.633+0000 7f6ee6ffd640 1 -- 172.21.15.98:0/3648279465 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7f6eec0cfbc0 con 0x7f6ef0074590 2024-09-19T08:01:25.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.633+0000 7f6ef519b640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f6edc0a3240 0x7f6edc0a5700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0000c00 tx=0x7f6ee00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:25.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.636+0000 7f6ee6ffd640 1 -- 172.21.15.98:0/3648279465 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f6eec014030 con 0x7f6ef0074590 2024-09-19T08:01:25.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.729+0000 7f6ee6ffd640 1 -- 172.21.15.98:0/3648279465 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f6eec098a10 con 0x7f6ef0074590 2024-09-19T08:01:25.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.838+0000 7f6ef7c00640 1 -- 172.21.15.98:0/3648279465 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f6ef00ff6c0 con 0x7f6edc0a3240 2024-09-19T08:01:25.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.866+0000 7f6ee6ffd640 1 -- 172.21.15.98:0/3648279465 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6ef00ff6c0 con 0x7f6edc0a3240 2024-09-19T08:01:25.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.868+0000 7f6ef7c00640 1 -- 172.21.15.98:0/3648279465 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f6edc0a3240 msgr2=0x7f6edc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:25.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.868+0000 7f6ef7c00640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f6edc0a3240 0x7f6edc0a5700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0000c00 tx=0x7f6ee00058d0 comp rx=0 tx=0).stop 2024-09-19T08:01:25.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.868+0000 7f6ef7c00640 1 -- 172.21.15.98:0/3648279465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0074590 msgr2=0x7f6ef01a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:25.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.868+0000 7f6ef7c00640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0074590 0x7f6ef01a4720 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6eec00c920 tx=0x7f6eec00cde0 comp rx=0 tx=0).stop 2024-09-19T08:01:25.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.868+0000 7f6ef7c00640 1 -- 172.21.15.98:0/3648279465 shutdown_connections 2024-09-19T08:01:25.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.868+0000 7f6ef7c00640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f6edc0a3240 0x7f6edc0a5700 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:25.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.868+0000 7f6ef7c00640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ef0112b90 0x7f6ef019e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:25.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.868+0000 7f6ef7c00640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ef0074eb0 0x7f6ef01a4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:25.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.868+0000 7f6ef7c00640 1 --2- 172.21.15.98:0/3648279465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0074590 0x7f6ef01a4720 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:25.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.868+0000 7f6ef7c00640 1 -- 172.21.15.98:0/3648279465 >> 172.21.15.98:0/3648279465 conn(0x7f6ef00fd8c0 msgr2=0x7f6ef00fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:25.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.868+0000 7f6ef7c00640 1 -- 172.21.15.98:0/3648279465 shutdown_connections 2024-09-19T08:01:25.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:25.868+0000 7f6ef7c00640 1 -- 172.21.15.98:0/3648279465 wait complete. 2024-09-19T08:01:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:25 smithi098 ceph-mon[31333]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-09-19T08:01:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-09-19T08:01:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-09-19T08:01:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1, 7, 11]}]: dispatch 2024-09-19T08:01:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-09-19T08:01:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-09-19T08:01:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-09-19T08:01:25.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:25 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-09-19T08:01:26.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:25 smithi162 ceph-mon[35260]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:26.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-09-19T08:01:26.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-09-19T08:01:26.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-09-19T08:01:26.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1, 7, 11]}]: dispatch 2024-09-19T08:01:26.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-09-19T08:01:26.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-09-19T08:01:26.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-09-19T08:01:26.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:25 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-09-19T08:01:26.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:25 smithi139 ceph-mon[34874]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:26.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-09-19T08:01:26.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-09-19T08:01:26.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-09-19T08:01:26.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1, 7, 11]}]: dispatch 2024-09-19T08:01:26.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-09-19T08:01:26.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-09-19T08:01:26.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-09-19T08:01:26.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:25 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-09-19T08:01:26.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-09-19T08:01:26.810 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:26.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:26 smithi098 ceph-mon[31333]: pgmap v278: 97 pgs: 97 active+clean; 584 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-09-19T08:01:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:26 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-09-19T08:01:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:26 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-09-19T08:01:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:26 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-09-19T08:01:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:26 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1, 7, 11]}]': finished 2024-09-19T08:01:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:26 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-09-19T08:01:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:26 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-09-19T08:01:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:26 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-09-19T08:01:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:26 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-19T08:01:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:26 smithi098 ceph-mon[31333]: osdmap e79: 12 total, 12 up, 12 in 2024-09-19T08:01:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:26 smithi098 ceph-mon[31333]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-19T08:01:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:26 smithi098 ceph-mon[31333]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:26 smithi162 ceph-mon[35260]: pgmap v278: 97 pgs: 97 active+clean; 584 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-09-19T08:01:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:26 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-09-19T08:01:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:26 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-09-19T08:01:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:26 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-09-19T08:01:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:26 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1, 7, 11]}]': finished 2024-09-19T08:01:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:26 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-09-19T08:01:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:26 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-09-19T08:01:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:26 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-09-19T08:01:27.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:26 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-19T08:01:27.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:26 smithi162 ceph-mon[35260]: osdmap e79: 12 total, 12 up, 12 in 2024-09-19T08:01:27.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:26 smithi162 ceph-mon[35260]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-19T08:01:27.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:26 smithi162 ceph-mon[35260]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:26 smithi139 ceph-mon[34874]: pgmap v278: 97 pgs: 97 active+clean; 584 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-09-19T08:01:27.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:26 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-09-19T08:01:27.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:26 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-09-19T08:01:27.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:26 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-09-19T08:01:27.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:26 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1, 7, 11]}]': finished 2024-09-19T08:01:27.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:26 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-09-19T08:01:27.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:26 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-09-19T08:01:27.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:26 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-09-19T08:01:27.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:26 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-19T08:01:27.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:26 smithi139 ceph-mon[34874]: osdmap e79: 12 total, 12 up, 12 in 2024-09-19T08:01:27.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:26 smithi139 ceph-mon[34874]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-19T08:01:27.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:26 smithi139 ceph-mon[34874]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.251+0000 7f8226150640 1 -- 172.21.15.98:0/704826700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8220106f50 msgr2=0x7f822010eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.251+0000 7f8226150640 1 --2- 172.21.15.98:0/704826700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8220106f50 0x7f822010eaf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8218009cd0 tx=0x7f821802a8e0 comp rx=0 tx=0).stop 2024-09-19T08:01:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.251+0000 7f8226150640 1 -- 172.21.15.98:0/704826700 shutdown_connections 2024-09-19T08:01:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.251+0000 7f8226150640 1 --2- 172.21.15.98:0/704826700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8220106f50 0x7f822010eaf0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.251+0000 7f8226150640 1 --2- 172.21.15.98:0/704826700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8220100ca0 0x7f8220106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.251+0000 7f8226150640 1 --2- 172.21.15.98:0/704826700 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82201002f0 0x7f82201006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.251+0000 7f8226150640 1 -- 172.21.15.98:0/704826700 >> 172.21.15.98:0/704826700 conn(0x7f8220077210 msgr2=0x7f82200fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.251+0000 7f8226150640 1 -- 172.21.15.98:0/704826700 shutdown_connections 2024-09-19T08:01:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.251+0000 7f8226150640 1 -- 172.21.15.98:0/704826700 wait complete. 2024-09-19T08:01:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.251+0000 7f8226150640 1 Processor -- start 2024-09-19T08:01:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.251+0000 7f8226150640 1 -- start start 2024-09-19T08:01:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f8226150640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82201002f0 0x7f82201985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f8226150640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8220100ca0 0x7f8220198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f8226150640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8220106f50 0x7f822019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f8226150640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8220112080 con 0x7f82201002f0 2024-09-19T08:01:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f8226150640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8220111f00 con 0x7f8220100ca0 2024-09-19T08:01:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f8226150640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8220112200 con 0x7f8220106f50 2024-09-19T08:01:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f821f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82201002f0 0x7f82201985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f821f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82201002f0 0x7f82201985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46208/0 (socket says 172.21.15.98:46208) 2024-09-19T08:01:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f821f7fe640 1 -- 172.21.15.98:0/4176241787 learned_addr learned my addr 172.21.15.98:0/4176241787 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f8217fff640 1 --2- 172.21.15.98:0/4176241787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8220100ca0 0x7f8220198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:27.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f821ffff640 1 --2- 172.21.15.98:0/4176241787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8220106f50 0x7f822019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:27.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f821f7fe640 1 -- 172.21.15.98:0/4176241787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8220106f50 msgr2=0x7f822019ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:27.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f821f7fe640 1 --2- 172.21.15.98:0/4176241787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8220106f50 0x7f822019ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:27.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f821f7fe640 1 -- 172.21.15.98:0/4176241787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8220100ca0 msgr2=0x7f8220198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:27.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f821f7fe640 1 --2- 172.21.15.98:0/4176241787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8220100ca0 0x7f8220198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:27.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f821f7fe640 1 -- 172.21.15.98:0/4176241787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f822019d570 con 0x7f82201002f0 2024-09-19T08:01:27.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f821f7fe640 1 --2- 172.21.15.98:0/4176241787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82201002f0 0x7f82201985c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f821000b4a0 tx=0x7f821000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:27.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f821d7fa640 1 -- 172.21.15.98:0/4176241787 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8210015020 con 0x7f82201002f0 2024-09-19T08:01:27.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f821d7fa640 1 -- 172.21.15.98:0/4176241787 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8210011d10 con 0x7f82201002f0 2024-09-19T08:01:27.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.252+0000 7f8226150640 1 -- 172.21.15.98:0/4176241787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f822019d860 con 0x7f82201002f0 2024-09-19T08:01:27.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.253+0000 7f821d7fa640 1 -- 172.21.15.98:0/4176241787 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821000f2e0 con 0x7f82201002f0 2024-09-19T08:01:27.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.253+0000 7f8226150640 1 -- 172.21.15.98:0/4176241787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82201a5140 con 0x7f82201002f0 2024-09-19T08:01:27.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.253+0000 7f8226150640 1 -- 172.21.15.98:0/4176241787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8220100750 con 0x7f82201002f0 2024-09-19T08:01:27.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.254+0000 7f821d7fa640 1 -- 172.21.15.98:0/4176241787 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f8210018020 con 0x7f82201002f0 2024-09-19T08:01:27.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.255+0000 7f821d7fa640 1 --2- 172.21.15.98:0/4176241787 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f820c0a32d0 0x7f820c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:27.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.255+0000 7f8217fff640 1 --2- 172.21.15.98:0/4176241787 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f820c0a32d0 0x7f820c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:27.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.255+0000 7f821d7fa640 1 -- 172.21.15.98:0/4176241787 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f82100cf250 con 0x7f82201002f0 2024-09-19T08:01:27.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.256+0000 7f8217fff640 1 --2- 172.21.15.98:0/4176241787 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f820c0a32d0 0x7f820c0a5790 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f82200ff880 tx=0x7f8208009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:27.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.260+0000 7f821d7fa640 1 -- 172.21.15.98:0/4176241787 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f821001f020 con 0x7f82201002f0 2024-09-19T08:01:27.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.452+0000 7f8226150640 1 -- 172.21.15.98:0/4176241787 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f8220062630 con 0x7f820c0a32d0 2024-09-19T08:01:27.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.457+0000 7f821d7fa640 1 -- 172.21.15.98:0/4176241787 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f8220062630 con 0x7f820c0a32d0 2024-09-19T08:01:27.457 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/qa/sv_0/7a360db1-e538-4aac-8976-f2855e71b623 2024-09-19T08:01:27.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.459+0000 7f8226150640 1 -- 172.21.15.98:0/4176241787 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f820c0a32d0 msgr2=0x7f820c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:27.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.459+0000 7f8226150640 1 --2- 172.21.15.98:0/4176241787 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f820c0a32d0 0x7f820c0a5790 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f82200ff880 tx=0x7f8208009110 comp rx=0 tx=0).stop 2024-09-19T08:01:27.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.459+0000 7f8226150640 1 -- 172.21.15.98:0/4176241787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82201002f0 msgr2=0x7f82201985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:27.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.459+0000 7f8226150640 1 --2- 172.21.15.98:0/4176241787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82201002f0 0x7f82201985c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f821000b4a0 tx=0x7f821000b8a0 comp rx=0 tx=0).stop 2024-09-19T08:01:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.459+0000 7f8226150640 1 -- 172.21.15.98:0/4176241787 shutdown_connections 2024-09-19T08:01:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.459+0000 7f8226150640 1 --2- 172.21.15.98:0/4176241787 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f820c0a32d0 0x7f820c0a5790 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.459+0000 7f8226150640 1 --2- 172.21.15.98:0/4176241787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8220106f50 0x7f822019ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.459+0000 7f8226150640 1 --2- 172.21.15.98:0/4176241787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8220100ca0 0x7f8220198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.459+0000 7f8226150640 1 --2- 172.21.15.98:0/4176241787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82201002f0 0x7f82201985c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.459+0000 7f8226150640 1 -- 172.21.15.98:0/4176241787 >> 172.21.15.98:0/4176241787 conn(0x7f8220077210 msgr2=0x7f82201051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.460+0000 7f8226150640 1 -- 172.21.15.98:0/4176241787 shutdown_connections 2024-09-19T08:01:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:27.460+0000 7f8226150640 1 -- 172.21.15.98:0/4176241787 wait complete. 2024-09-19T08:01:28.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:27 smithi098 ceph-mon[31333]: osdmap e80: 12 total, 12 up, 12 in 2024-09-19T08:01:28.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:27 smithi098 ceph-mon[31333]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:28.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:27 smithi098 ceph-mon[31333]: pgmap v281: 97 pgs: 97 active+clean; 585 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 6.1 KiB/s wr, 27 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-09-19T08:01:28.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-09-19T08:01:28.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:27 smithi162 ceph-mon[35260]: osdmap e80: 12 total, 12 up, 12 in 2024-09-19T08:01:28.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:27 smithi162 ceph-mon[35260]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:28.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:27 smithi162 ceph-mon[35260]: pgmap v281: 97 pgs: 97 active+clean; 585 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 6.1 KiB/s wr, 27 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-09-19T08:01:28.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:27 smithi139 ceph-mon[34874]: osdmap e80: 12 total, 12 up, 12 in 2024-09-19T08:01:28.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:27 smithi139 ceph-mon[34874]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:28.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:27 smithi139 ceph-mon[34874]: pgmap v281: 97 pgs: 97 active+clean; 585 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 6.1 KiB/s wr, 27 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-09-19T08:01:28.358 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:28.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.866+0000 7fc08703a640 1 -- 172.21.15.98:0/2994955692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc080100e20 msgr2=0x7fc080101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:28.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.866+0000 7fc08703a640 1 --2- 172.21.15.98:0/2994955692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc080100e20 0x7fc080101200 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc074009900 tx=0x7fc07402a480 comp rx=0 tx=0).stop 2024-09-19T08:01:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 -- 172.21.15.98:0/2994955692 shutdown_connections 2024-09-19T08:01:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 --2- 172.21.15.98:0/2994955692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc080102ef0 0x7fc08010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 --2- 172.21.15.98:0/2994955692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc080102570 0x7fc0801029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 --2- 172.21.15.98:0/2994955692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc080100e20 0x7fc080101200 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 -- 172.21.15.98:0/2994955692 >> 172.21.15.98:0/2994955692 conn(0x7fc0800fc7d0 msgr2=0x7fc0800febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 -- 172.21.15.98:0/2994955692 shutdown_connections 2024-09-19T08:01:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 -- 172.21.15.98:0/2994955692 wait complete. 2024-09-19T08:01:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 Processor -- start 2024-09-19T08:01:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 -- start start 2024-09-19T08:01:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc080100e20 0x7fc080197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc080102570 0x7fc080197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc080102ef0 0x7fc08019be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc080111020 con 0x7fc080102570 2024-09-19T08:01:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc080110ea0 con 0x7fc080100e20 2024-09-19T08:01:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc08703a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc0801111a0 con 0x7fc080102ef0 2024-09-19T08:01:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc07ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc080102570 0x7fc080197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.867+0000 7fc07ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc080102570 0x7fc080197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46240/0 (socket says 172.21.15.98:46240) 2024-09-19T08:01:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.868+0000 7fc07ffff640 1 -- 172.21.15.98:0/3416834203 learned_addr learned my addr 172.21.15.98:0/3416834203 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.868+0000 7fc07ffff640 1 -- 172.21.15.98:0/3416834203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc080102ef0 msgr2=0x7fc08019be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.868+0000 7fc07ffff640 1 --2- 172.21.15.98:0/3416834203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc080102ef0 0x7fc08019be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.868+0000 7fc07ffff640 1 -- 172.21.15.98:0/3416834203 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc080100e20 msgr2=0x7fc080197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.868+0000 7fc07ffff640 1 --2- 172.21.15.98:0/3416834203 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc080100e20 0x7fc080197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.868+0000 7fc07ffff640 1 -- 172.21.15.98:0/3416834203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc08019c510 con 0x7fc080102570 2024-09-19T08:01:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.868+0000 7fc07ffff640 1 --2- 172.21.15.98:0/3416834203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc080102570 0x7fc080197aa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00b4a0 tx=0x7fc06c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.868+0000 7fc07dffb640 1 -- 172.21.15.98:0/3416834203 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06c014070 con 0x7fc080102570 2024-09-19T08:01:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.868+0000 7fc08703a640 1 -- 172.21.15.98:0/3416834203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc08019c800 con 0x7fc080102570 2024-09-19T08:01:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.868+0000 7fc07dffb640 1 -- 172.21.15.98:0/3416834203 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc06c010af0 con 0x7fc080102570 2024-09-19T08:01:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.868+0000 7fc07dffb640 1 -- 172.21.15.98:0/3416834203 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06c017d10 con 0x7fc080102570 2024-09-19T08:01:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.868+0000 7fc08703a640 1 -- 172.21.15.98:0/3416834203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0801a40e0 con 0x7fc080102570 2024-09-19T08:01:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.869+0000 7fc08703a640 1 -- 172.21.15.98:0/3416834203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc080101280 con 0x7fc080102570 2024-09-19T08:01:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.870+0000 7fc07dffb640 1 -- 172.21.15.98:0/3416834203 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc06c017070 con 0x7fc080102570 2024-09-19T08:01:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.871+0000 7fc07dffb640 1 --2- 172.21.15.98:0/3416834203 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc0700a32d0 0x7fc0700a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.871+0000 7fc084dd6640 1 --2- 172.21.15.98:0/3416834203 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc0700a32d0 0x7fc0700a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.871+0000 7fc07dffb640 1 -- 172.21.15.98:0/3416834203 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fc06c0cee00 con 0x7fc080102570 2024-09-19T08:01:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.871+0000 7fc084dd6640 1 --2- 172.21.15.98:0/3416834203 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc0700a32d0 0x7fc0700a5790 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc0740096c0 tx=0x7fc074000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:28.875+0000 7fc07dffb640 1 -- 172.21.15.98:0/3416834203 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fc06c09bae0 con 0x7fc080102570 2024-09-19T08:01:29.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:29.064+0000 7fc08703a640 1 -- 172.21.15.98:0/3416834203 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc080062630 con 0x7fc0700a32d0 2024-09-19T08:01:29.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:29.090+0000 7fc07dffb640 1 -- 172.21.15.98:0/3416834203 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc080062630 con 0x7fc0700a32d0 2024-09-19T08:01:29.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:29.092+0000 7fc08703a640 1 -- 172.21.15.98:0/3416834203 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc0700a32d0 msgr2=0x7fc0700a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:29.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:29.092+0000 7fc08703a640 1 --2- 172.21.15.98:0/3416834203 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc0700a32d0 0x7fc0700a5790 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc0740096c0 tx=0x7fc074000bc0 comp rx=0 tx=0).stop 2024-09-19T08:01:29.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:29.092+0000 7fc08703a640 1 -- 172.21.15.98:0/3416834203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc080102570 msgr2=0x7fc080197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:29.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:29.092+0000 7fc08703a640 1 --2- 172.21.15.98:0/3416834203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc080102570 0x7fc080197aa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00b4a0 tx=0x7fc06c00b960 comp rx=0 tx=0).stop 2024-09-19T08:01:29.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:29.092+0000 7fc08703a640 1 -- 172.21.15.98:0/3416834203 shutdown_connections 2024-09-19T08:01:29.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:29.092+0000 7fc08703a640 1 --2- 172.21.15.98:0/3416834203 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc0700a32d0 0x7fc0700a5790 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:29.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:29.092+0000 7fc08703a640 1 --2- 172.21.15.98:0/3416834203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc080102ef0 0x7fc08019be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:29.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:29.092+0000 7fc08703a640 1 --2- 172.21.15.98:0/3416834203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc080102570 0x7fc080197aa0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:29.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:29.092+0000 7fc08703a640 1 --2- 172.21.15.98:0/3416834203 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc080100e20 0x7fc080197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:29.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:29.092+0000 7fc08703a640 1 -- 172.21.15.98:0/3416834203 >> 172.21.15.98:0/3416834203 conn(0x7fc0800fc7d0 msgr2=0x7fc0800febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:29.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:29.092+0000 7fc08703a640 1 -- 172.21.15.98:0/3416834203 shutdown_connections 2024-09-19T08:01:29.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:29.092+0000 7fc08703a640 1 -- 172.21.15.98:0/3416834203 wait complete. 2024-09-19T08:01:29.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-09-19T08:01:29.979 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:30.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.502+0000 7f27f50eb640 1 -- 172.21.15.98:0/2128057961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f0100b20 msgr2=0x7f27f0100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:30.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.502+0000 7f27f50eb640 1 --2- 172.21.15.98:0/2128057961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f0100b20 0x7f27f0100f00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009ca0 tx=0x7f27e002a8c0 comp rx=0 tx=0).stop 2024-09-19T08:01:30.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.502+0000 7f27f50eb640 1 -- 172.21.15.98:0/2128057961 shutdown_connections 2024-09-19T08:01:30.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.502+0000 7f27f50eb640 1 --2- 172.21.15.98:0/2128057961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27f010e500 0x7f27f0110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:30.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.502+0000 7f27f50eb640 1 --2- 172.21.15.98:0/2128057961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27f0101440 0x7f27f010df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:30.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.502+0000 7f27f50eb640 1 --2- 172.21.15.98:0/2128057961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f0100b20 0x7f27f0100f00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:30.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.502+0000 7f27f50eb640 1 -- 172.21.15.98:0/2128057961 >> 172.21.15.98:0/2128057961 conn(0x7f27f00fc7d0 msgr2=0x7f27f00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:30.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.502+0000 7f27f50eb640 1 -- 172.21.15.98:0/2128057961 shutdown_connections 2024-09-19T08:01:30.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.502+0000 7f27f50eb640 1 -- 172.21.15.98:0/2128057961 wait complete. 2024-09-19T08:01:30.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.503+0000 7f27f50eb640 1 Processor -- start 2024-09-19T08:01:30.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.503+0000 7f27f50eb640 1 -- start start 2024-09-19T08:01:30.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.503+0000 7f27f50eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f0100b20 0x7f27f019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:30.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.503+0000 7f27f50eb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27f0101440 0x7f27f019ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:30.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.503+0000 7f27f50eb640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27f010e500 0x7f27f019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:30.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27f50eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f27f01142d0 con 0x7f27f0100b20 2024-09-19T08:01:30.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27f50eb640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0114150 con 0x7f27f010e500 2024-09-19T08:01:30.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27f50eb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0114450 con 0x7f27f0101440 2024-09-19T08:01:30.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27eed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f0100b20 0x7f27f019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:30.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27eed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f0100b20 0x7f27f019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46268/0 (socket says 172.21.15.98:46268) 2024-09-19T08:01:30.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27eed9d640 1 -- 172.21.15.98:0/9222346 learned_addr learned my addr 172.21.15.98:0/9222346 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:30.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27ee59c640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27f0101440 0x7f27f019ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:30.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27ef59e640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27f010e500 0x7f27f019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:30.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27eed9d640 1 -- 172.21.15.98:0/9222346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27f0101440 msgr2=0x7f27f019ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:30.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27eed9d640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27f0101440 0x7f27f019ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:30.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27eed9d640 1 -- 172.21.15.98:0/9222346 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27f010e500 msgr2=0x7f27f019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:30.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27eed9d640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27f010e500 0x7f27f019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:30.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27eed9d640 1 -- 172.21.15.98:0/9222346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f019f6c0 con 0x7f27f0100b20 2024-09-19T08:01:30.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27ef59e640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27f010e500 0x7f27f019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:01:30.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27eed9d640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f0100b20 0x7f27f019a710 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f27e00097b0 tx=0x7f27e0031b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:30.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27e7fff640 1 -- 172.21.15.98:0/9222346 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e002fba0 con 0x7f27f0100b20 2024-09-19T08:01:30.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27f50eb640 1 -- 172.21.15.98:0/9222346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f019f950 con 0x7f27f0100b20 2024-09-19T08:01:30.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27e7fff640 1 -- 172.21.15.98:0/9222346 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e002fd40 con 0x7f27f0100b20 2024-09-19T08:01:30.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27e7fff640 1 -- 172.21.15.98:0/9222346 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e003b930 con 0x7f27f0100b20 2024-09-19T08:01:30.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.504+0000 7f27f50eb640 1 -- 172.21.15.98:0/9222346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f01a7290 con 0x7f27f0100b20 2024-09-19T08:01:30.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.506+0000 7f27f50eb640 1 -- 172.21.15.98:0/9222346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f006a850 con 0x7f27f0100b20 2024-09-19T08:01:30.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.506+0000 7f27e7fff640 1 -- 172.21.15.98:0/9222346 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f27e0030cc0 con 0x7f27f0100b20 2024-09-19T08:01:30.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.507+0000 7f27e7fff640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f27dc0a32d0 0x7f27dc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:30.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.507+0000 7f27ee59c640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f27dc0a32d0 0x7f27dc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:30.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.507+0000 7f27e7fff640 1 -- 172.21.15.98:0/9222346 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f27e00f1150 con 0x7f27f0100b20 2024-09-19T08:01:30.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.508+0000 7f27ee59c640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f27dc0a32d0 0x7f27dc0a5790 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f27d80058c0 tx=0x7f27d8005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:30.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.511+0000 7f27e7fff640 1 -- 172.21.15.98:0/9222346 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f27e0044380 con 0x7f27f0100b20 2024-09-19T08:01:30.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:30 smithi098 ceph-mon[31333]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:30.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:30 smithi098 ceph-mon[31333]: pgmap v282: 97 pgs: 97 active+clean; 605 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 KiB/s wr, 32 op/s; 156 B/s, 1 keys/s, 1 objects/s recovering 2024-09-19T08:01:30.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.705+0000 7f27f50eb640 1 -- 172.21.15.98:0/9222346 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f27f010f7e0 con 0x7f27dc0a32d0 2024-09-19T08:01:30.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.711+0000 7f27e7fff640 1 -- 172.21.15.98:0/9222346 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f27f010f7e0 con 0x7f27dc0a32d0 2024-09-19T08:01:30.711 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/qa/sv_1/a73a52f2-eaee-40cd-832f-432a14c0ec6e 2024-09-19T08:01:30.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.713+0000 7f27f50eb640 1 -- 172.21.15.98:0/9222346 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f27dc0a32d0 msgr2=0x7f27dc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:30.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.713+0000 7f27f50eb640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f27dc0a32d0 0x7f27dc0a5790 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f27d80058c0 tx=0x7f27d8005850 comp rx=0 tx=0).stop 2024-09-19T08:01:30.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.713+0000 7f27f50eb640 1 -- 172.21.15.98:0/9222346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f0100b20 msgr2=0x7f27f019a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:30.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.713+0000 7f27f50eb640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f0100b20 0x7f27f019a710 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f27e00097b0 tx=0x7f27e0031b10 comp rx=0 tx=0).stop 2024-09-19T08:01:30.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.713+0000 7f27f50eb640 1 -- 172.21.15.98:0/9222346 shutdown_connections 2024-09-19T08:01:30.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.713+0000 7f27f50eb640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f27dc0a32d0 0x7f27dc0a5790 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:30.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.713+0000 7f27f50eb640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27f010e500 0x7f27f019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:30.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.713+0000 7f27f50eb640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27f0101440 0x7f27f019ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:30.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.713+0000 7f27f50eb640 1 --2- 172.21.15.98:0/9222346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f0100b20 0x7f27f019a710 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:30.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.713+0000 7f27f50eb640 1 -- 172.21.15.98:0/9222346 >> 172.21.15.98:0/9222346 conn(0x7f27f00fc7d0 msgr2=0x7f27f00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:30.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.713+0000 7f27f50eb640 1 -- 172.21.15.98:0/9222346 shutdown_connections 2024-09-19T08:01:30.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:30.713+0000 7f27f50eb640 1 -- 172.21.15.98:0/9222346 wait complete. 2024-09-19T08:01:30.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:30 smithi162 ceph-mon[35260]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:30.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:30 smithi162 ceph-mon[35260]: pgmap v282: 97 pgs: 97 active+clean; 605 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 KiB/s wr, 32 op/s; 156 B/s, 1 keys/s, 1 objects/s recovering 2024-09-19T08:01:30.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:30 smithi139 ceph-mon[34874]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:30.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:30 smithi139 ceph-mon[34874]: pgmap v282: 97 pgs: 97 active+clean; 605 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 KiB/s wr, 32 op/s; 156 B/s, 1 keys/s, 1 objects/s recovering 2024-09-19T08:01:31.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:01:31.612 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:31.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:31 smithi098 ceph-mon[31333]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:31.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:31 smithi162 ceph-mon[35260]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:31.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:31 smithi139 ceph-mon[34874]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:01:32.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.072+0000 7ff859d3f640 1 -- 172.21.15.98:0/1433047682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff854100a80 msgr2=0x7ff854100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:32.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.072+0000 7ff859d3f640 1 --2- 172.21.15.98:0/1433047682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff854100a80 0x7ff854100ee0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff848009930 tx=0x7ff84802a420 comp rx=0 tx=0).stop 2024-09-19T08:01:32.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.072+0000 7ff859d3f640 1 -- 172.21.15.98:0/1433047682 shutdown_connections 2024-09-19T08:01:32.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.072+0000 7ff859d3f640 1 --2- 172.21.15.98:0/1433047682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff854101420 0x7ff85410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:32.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.072+0000 7ff859d3f640 1 --2- 172.21.15.98:0/1433047682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff854100a80 0x7ff854100ee0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:32.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.072+0000 7ff859d3f640 1 --2- 172.21.15.98:0/1433047682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff854106a80 0x7ff854106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:32.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.072+0000 7ff859d3f640 1 -- 172.21.15.98:0/1433047682 >> 172.21.15.98:0/1433047682 conn(0x7ff8540fc7a0 msgr2=0x7ff8540febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.072+0000 7ff859d3f640 1 -- 172.21.15.98:0/1433047682 shutdown_connections 2024-09-19T08:01:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.072+0000 7ff859d3f640 1 -- 172.21.15.98:0/1433047682 wait complete. 2024-09-19T08:01:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.073+0000 7ff859d3f640 1 Processor -- start 2024-09-19T08:01:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.073+0000 7ff859d3f640 1 -- start start 2024-09-19T08:01:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.073+0000 7ff859d3f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff854100a80 0x7ff8541973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.073+0000 7ff859d3f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff854101420 0x7ff854197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.073+0000 7ff859d3f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff854106a80 0x7ff85419bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.073+0000 7ff859d3f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff854110ea0 con 0x7ff854101420 2024-09-19T08:01:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.073+0000 7ff859d3f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff854110d20 con 0x7ff854100a80 2024-09-19T08:01:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.073+0000 7ff859d3f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff854111020 con 0x7ff854106a80 2024-09-19T08:01:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.073+0000 7ff852ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff854101420 0x7ff854197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff852ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff854101420 0x7ff854197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46290/0 (socket says 172.21.15.98:46290) 2024-09-19T08:01:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff852ffd640 1 -- 172.21.15.98:0/1456010282 learned_addr learned my addr 172.21.15.98:0/1456010282 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff8537fe640 1 --2- 172.21.15.98:0/1456010282 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff854100a80 0x7ff8541973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff852ffd640 1 -- 172.21.15.98:0/1456010282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff854106a80 msgr2=0x7ff85419bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff852ffd640 1 --2- 172.21.15.98:0/1456010282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff854106a80 0x7ff85419bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff852ffd640 1 -- 172.21.15.98:0/1456010282 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff854100a80 msgr2=0x7ff8541973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff852ffd640 1 --2- 172.21.15.98:0/1456010282 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff854100a80 0x7ff8541973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff852ffd640 1 -- 172.21.15.98:0/1456010282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85419c390 con 0x7ff854101420 2024-09-19T08:01:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff8537fe640 1 --2- 172.21.15.98:0/1456010282 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff854100a80 0x7ff8541973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:01:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff852ffd640 1 --2- 172.21.15.98:0/1456010282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff854101420 0x7ff854197920 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff848009a50 tx=0x7ff848004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff850ff9640 1 -- 172.21.15.98:0/1456010282 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84803f030 con 0x7ff854101420 2024-09-19T08:01:32.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff850ff9640 1 -- 172.21.15.98:0/1456010282 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff848003ce0 con 0x7ff854101420 2024-09-19T08:01:32.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff859d3f640 1 -- 172.21.15.98:0/1456010282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff85419c620 con 0x7ff854101420 2024-09-19T08:01:32.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff850ff9640 1 -- 172.21.15.98:0/1456010282 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff848035070 con 0x7ff854101420 2024-09-19T08:01:32.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.074+0000 7ff859d3f640 1 -- 172.21.15.98:0/1456010282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8541a3f60 con 0x7ff854101420 2024-09-19T08:01:32.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.075+0000 7ff859d3f640 1 -- 172.21.15.98:0/1456010282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8541021c0 con 0x7ff854101420 2024-09-19T08:01:32.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.076+0000 7ff850ff9640 1 -- 172.21.15.98:0/1456010282 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ff848042030 con 0x7ff854101420 2024-09-19T08:01:32.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.077+0000 7ff850ff9640 1 --2- 172.21.15.98:0/1456010282 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff83c0a32d0 0x7ff83c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:32.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.077+0000 7ff8537fe640 1 --2- 172.21.15.98:0/1456010282 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff83c0a32d0 0x7ff83c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:32.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.078+0000 7ff850ff9640 1 -- 172.21.15.98:0/1456010282 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7ff8480f0100 con 0x7ff854101420 2024-09-19T08:01:32.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.078+0000 7ff8537fe640 1 --2- 172.21.15.98:0/1456010282 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff83c0a32d0 0x7ff83c0a5790 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff844009fa0 tx=0x7ff844009570 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:32.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.082+0000 7ff850ff9640 1 -- 172.21.15.98:0/1456010282 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ff8480bccd0 con 0x7ff854101420 2024-09-19T08:01:32.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.284+0000 7ff859d3f640 1 -- 172.21.15.98:0/1456010282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff854109040 con 0x7ff854101420 2024-09-19T08:01:32.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.285+0000 7ff850ff9640 1 -- 172.21.15.98:0/1456010282 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11983 (secure 0 0 0) 0x7ff8480bc5d0 con 0x7ff854101420 2024-09-19T08:01:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:01:32.286 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":27,"btime":"2024-09-19T08:01:22:699589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:01:22.699569+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24461,24437,24406,24376]},"id":1}]} 2024-09-19T08:01:32.286 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 27 2024-09-19T08:01:32.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.287+0000 7ff859d3f640 1 -- 172.21.15.98:0/1456010282 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff83c0a32d0 msgr2=0x7ff83c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:32.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.287+0000 7ff859d3f640 1 --2- 172.21.15.98:0/1456010282 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff83c0a32d0 0x7ff83c0a5790 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff844009fa0 tx=0x7ff844009570 comp rx=0 tx=0).stop 2024-09-19T08:01:32.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.287+0000 7ff859d3f640 1 -- 172.21.15.98:0/1456010282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff854101420 msgr2=0x7ff854197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:32.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.287+0000 7ff859d3f640 1 --2- 172.21.15.98:0/1456010282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff854101420 0x7ff854197920 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff848009a50 tx=0x7ff848004270 comp rx=0 tx=0).stop 2024-09-19T08:01:32.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.287+0000 7ff859d3f640 1 -- 172.21.15.98:0/1456010282 shutdown_connections 2024-09-19T08:01:32.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.287+0000 7ff859d3f640 1 --2- 172.21.15.98:0/1456010282 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff83c0a32d0 0x7ff83c0a5790 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:32.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.287+0000 7ff859d3f640 1 --2- 172.21.15.98:0/1456010282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff854106a80 0x7ff85419bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:32.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.287+0000 7ff859d3f640 1 --2- 172.21.15.98:0/1456010282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff854101420 0x7ff854197920 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:32.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.287+0000 7ff859d3f640 1 --2- 172.21.15.98:0/1456010282 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff854100a80 0x7ff8541973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:32.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.287+0000 7ff859d3f640 1 -- 172.21.15.98:0/1456010282 >> 172.21.15.98:0/1456010282 conn(0x7ff8540fc7a0 msgr2=0x7ff8540fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:32.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.287+0000 7ff859d3f640 1 -- 172.21.15.98:0/1456010282 shutdown_connections 2024-09-19T08:01:32.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:32.287+0000 7ff859d3f640 1 -- 172.21.15.98:0/1456010282 wait complete. 2024-09-19T08:01:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:32 smithi162 ceph-mon[35260]: pgmap v283: 97 pgs: 97 active+clean; 606 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.0 KiB/s wr, 20 op/s; 156 B/s, 1 keys/s, 2 objects/s recovering 2024-09-19T08:01:32.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:32 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1456010282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:01:32.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:32 smithi139 ceph-mon[34874]: pgmap v283: 97 pgs: 97 active+clean; 606 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.0 KiB/s wr, 20 op/s; 156 B/s, 1 keys/s, 2 objects/s recovering 2024-09-19T08:01:32.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:32 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1456010282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:01:32.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:32 smithi098 ceph-mon[31333]: pgmap v283: 97 pgs: 97 active+clean; 606 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.0 KiB/s wr, 20 op/s; 156 B/s, 1 keys/s, 2 objects/s recovering 2024-09-19T08:01:32.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:32 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1456010282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:01:32.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T08:01:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:33.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.731+0000 7f4c9c947640 1 -- 172.21.15.98:0/2328568416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c98106a80 msgr2=0x7f4c98106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:33.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.731+0000 7f4c9c947640 1 --2- 172.21.15.98:0/2328568416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c98106a80 0x7f4c98106e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009930 tx=0x7f4c8802a420 comp rx=0 tx=0).stop 2024-09-19T08:01:33.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.731+0000 7f4c9c947640 1 -- 172.21.15.98:0/2328568416 shutdown_connections 2024-09-19T08:01:33.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.731+0000 7f4c9c947640 1 --2- 172.21.15.98:0/2328568416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c98101420 0x7f4c9810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:33.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.731+0000 7f4c9c947640 1 --2- 172.21.15.98:0/2328568416 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c98100a80 0x7f4c98100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:33.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.731+0000 7f4c9c947640 1 --2- 172.21.15.98:0/2328568416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c98106a80 0x7f4c98106e60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:33.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.731+0000 7f4c9c947640 1 -- 172.21.15.98:0/2328568416 >> 172.21.15.98:0/2328568416 conn(0x7f4c980fc7a0 msgr2=0x7f4c980febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:33.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.731+0000 7f4c9c947640 1 -- 172.21.15.98:0/2328568416 shutdown_connections 2024-09-19T08:01:33.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.731+0000 7f4c9c947640 1 -- 172.21.15.98:0/2328568416 wait complete. 2024-09-19T08:01:33.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c9c947640 1 Processor -- start 2024-09-19T08:01:33.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c9c947640 1 -- start start 2024-09-19T08:01:33.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c9c947640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c98100a80 0x7f4c98197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:33.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c9c947640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c98101420 0x7f4c98197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:33.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c9c947640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c98106a80 0x7f4c9819bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:33.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c9c947640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4c98110e80 con 0x7f4c98101420 2024-09-19T08:01:33.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c9c947640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4c98110d00 con 0x7f4c98106a80 2024-09-19T08:01:33.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c9c947640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4c98111000 con 0x7f4c98100a80 2024-09-19T08:01:33.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c95d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c98101420 0x7f4c98197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:33.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c95d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c98101420 0x7f4c98197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46306/0 (socket says 172.21.15.98:46306) 2024-09-19T08:01:33.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c95d9b640 1 -- 172.21.15.98:0/4060415556 learned_addr learned my addr 172.21.15.98:0/4060415556 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c96d9d640 1 --2- 172.21.15.98:0/4060415556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c98106a80 0x7f4c9819bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c9659c640 1 --2- 172.21.15.98:0/4060415556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c98100a80 0x7f4c98197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c95d9b640 1 -- 172.21.15.98:0/4060415556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c98100a80 msgr2=0x7f4c98197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c95d9b640 1 --2- 172.21.15.98:0/4060415556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c98100a80 0x7f4c98197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c95d9b640 1 -- 172.21.15.98:0/4060415556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c98106a80 msgr2=0x7f4c9819bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c95d9b640 1 --2- 172.21.15.98:0/4060415556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c98106a80 0x7f4c9819bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.732+0000 7f4c95d9b640 1 -- 172.21.15.98:0/4060415556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c9819c2b0 con 0x7f4c98101420 2024-09-19T08:01:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.733+0000 7f4c95d9b640 1 --2- 172.21.15.98:0/4060415556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c98101420 0x7f4c98197840 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c00d950 tx=0x7f4c8c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.733+0000 7f4c877fe640 1 -- 172.21.15.98:0/4060415556 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c017070 con 0x7f4c98101420 2024-09-19T08:01:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.733+0000 7f4c9c947640 1 -- 172.21.15.98:0/4060415556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c9819c5a0 con 0x7f4c98101420 2024-09-19T08:01:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.733+0000 7f4c877fe640 1 -- 172.21.15.98:0/4060415556 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c8c009c30 con 0x7f4c98101420 2024-09-19T08:01:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.733+0000 7f4c877fe640 1 -- 172.21.15.98:0/4060415556 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c00f6b0 con 0x7f4c98101420 2024-09-19T08:01:33.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.733+0000 7f4c9c947640 1 -- 172.21.15.98:0/4060415556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c981a3e80 con 0x7f4c98101420 2024-09-19T08:01:33.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.734+0000 7f4c9c947640 1 -- 172.21.15.98:0/4060415556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c98100ee0 con 0x7f4c98101420 2024-09-19T08:01:33.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.735+0000 7f4c877fe640 1 -- 172.21.15.98:0/4060415556 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4c8c0109d0 con 0x7f4c98101420 2024-09-19T08:01:33.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.736+0000 7f4c877fe640 1 --2- 172.21.15.98:0/4060415556 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4c800a32d0 0x7f4c800a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:33.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.736+0000 7f4c9659c640 1 --2- 172.21.15.98:0/4060415556 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4c800a32d0 0x7f4c800a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:33.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.736+0000 7f4c877fe640 1 -- 172.21.15.98:0/4060415556 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f4c8c0cf660 con 0x7f4c98101420 2024-09-19T08:01:33.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.736+0000 7f4c9659c640 1 --2- 172.21.15.98:0/4060415556 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4c800a32d0 0x7f4c800a5790 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009900 tx=0x7f4c880058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:33.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.740+0000 7f4c877fe640 1 -- 172.21.15.98:0/4060415556 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f4c8c009de0 con 0x7f4c98101420 2024-09-19T08:01:33.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.910+0000 7f4c9c947640 1 -- 172.21.15.98:0/4060415556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c98108ff0 con 0x7f4c98101420 2024-09-19T08:01:33.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.912+0000 7f4c877fe640 1 -- 172.21.15.98:0/4060415556 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22754 (secure 0 0 0) 0x7f4c8c09bbe0 con 0x7f4c98101420 2024-09-19T08:01:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:01:33.912 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":80,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","created":"2024-09-19T07:53:39.127398+0000","modified":"2024-09-19T08:01:26.714628+0000","last_up_change":"2024-09-19T08:00:22.467777+0000","last_in_change":"2024-09-19T07:59:58.538492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:56:43.773892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T08:01:07.693796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T08:01:09.518028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d01ce6-165d-493d-ae98-d8ad9e5d4134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6803","nonce":1719360533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6805","nonce":1719360533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6809","nonce":1719360533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6807","nonce":1719360533}]},"public_addr":"172.21.15.98:6803/1719360533","cluster_addr":"172.21.15.98:6805/1719360533","heartbeat_back_addr":"172.21.15.98:6809/1719360533","heartbeat_front_addr":"172.21.15.98:6807/1719360533","state":["exists","up"]},{"osd":1,"uuid":"34724bd5-0738-4473-b087-c09031c1ee2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036962626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036962626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036962626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036962626}]},"public_addr":"172.21.15.139:6801/4036962626","cluster_addr":"172.21.15.139:6803/4036962626","heartbeat_back_addr":"172.21.15.139:6807/4036962626","heartbeat_front_addr":"172.21.15.139:6805/4036962626","state":["exists","up"]},{"osd":2,"uuid":"16985ebe-0e32-42ee-bded-b7838e290afc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6801","nonce":2811412891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6803","nonce":2811412891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6807","nonce":2811412891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6805","nonce":2811412891}]},"public_addr":"172.21.15.162:6801/2811412891","cluster_addr":"172.21.15.162:6803/2811412891","heartbeat_back_addr":"172.21.15.162:6807/2811412891","heartbeat_front_addr":"172.21.15.162:6805/2811412891","state":["exists","up"]},{"osd":3,"uuid":"b140e466-3119-49cb-aec2-ba6ca4d10d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6811","nonce":1383392014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6813","nonce":1383392014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6817","nonce":1383392014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6815","nonce":1383392014}]},"public_addr":"172.21.15.98:6811/1383392014","cluster_addr":"172.21.15.98:6813/1383392014","heartbeat_back_addr":"172.21.15.98:6817/1383392014","heartbeat_front_addr":"172.21.15.98:6815/1383392014","state":["exists","up"]},{"osd":4,"uuid":"b4412813-df57-4ac6-8fb7-f57b69c74768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6809","nonce":116453520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6811","nonce":116453520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6815","nonce":116453520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6813","nonce":116453520}]},"public_addr":"172.21.15.139:6809/116453520","cluster_addr":"172.21.15.139:6811/116453520","heartbeat_back_addr":"172.21.15.139:6815/116453520","heartbeat_front_addr":"172.21.15.139:6813/116453520","state":["exists","up"]},{"osd":5,"uuid":"d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6809","nonce":3769530297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6811","nonce":3769530297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6815","nonce":3769530297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6813","nonce":3769530297}]},"public_addr":"172.21.15.162:6809/3769530297","cluster_addr":"172.21.15.162:6811/3769530297","heartbeat_back_addr":"172.21.15.162:6815/3769530297","heartbeat_front_addr":"172.21.15.162:6813/3769530297","state":["exists","up"]},{"osd":6,"uuid":"ba1c1590-6aea-40dc-90c8-e21b88896909","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6819","nonce":692373267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6821","nonce":692373267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6825","nonce":692373267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6823","nonce":692373267}]},"public_addr":"172.21.15.98:6819/692373267","cluster_addr":"172.21.15.98:6821/692373267","heartbeat_back_addr":"172.21.15.98:6825/692373267","heartbeat_front_addr":"172.21.15.98:6823/692373267","state":["exists","up"]},{"osd":7,"uuid":"eb9093ad-a068-4a63-ab87-0b50522cbc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6817","nonce":1963811650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6819","nonce":1963811650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6823","nonce":1963811650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6821","nonce":1963811650}]},"public_addr":"172.21.15.139:6817/1963811650","cluster_addr":"172.21.15.139:6819/1963811650","heartbeat_back_addr":"172.21.15.139:6823/1963811650","heartbeat_front_addr":"172.21.15.139:6821/1963811650","state":["exists","up"]},{"osd":8,"uuid":"31ca330a-fc45-466d-81c2-ef4386b3cc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6817","nonce":3607966458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6819","nonce":3607966458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6823","nonce":3607966458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6821","nonce":3607966458}]},"public_addr":"172.21.15.162:6817/3607966458","cluster_addr":"172.21.15.162:6819/3607966458","heartbeat_back_addr":"172.21.15.162:6823/3607966458","heartbeat_front_addr":"172.21.15.162:6821/3607966458","state":["exists","up"]},{"osd":9,"uuid":"9c4f216f-aef1-4367-9c58-b913b0c4f7db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6827","nonce":2050391244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6829","nonce":2050391244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6833","nonce":2050391244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6831","nonce":2050391244}]},"public_addr":"172.21.15.98:6827/2050391244","cluster_addr":"172.21.15.98:6829/2050391244","heartbeat_back_addr":"172.21.15.98:6833/2050391244","heartbeat_front_addr":"172.21.15.98:6831/2050391244","state":["exists","up"]},{"osd":10,"uuid":"03cc7df6-84ae-4d9a-a3d7-74c91e1744e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6825","nonce":2336761639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6827","nonce":2336761639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6831","nonce":2336761639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6829","nonce":2336761639}]},"public_addr":"172.21.15.139:6825/2336761639","cluster_addr":"172.21.15.139:6827/2336761639","heartbeat_back_addr":"172.21.15.139:6831/2336761639","heartbeat_front_addr":"172.21.15.139:6829/2336761639","state":["exists","up"]},{"osd":11,"uuid":"29e95498-4952-46d4-8fa6-6d8e2b862111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6825","nonce":1231011554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6827","nonce":1231011554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6831","nonce":1231011554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6829","nonce":1231011554}]},"public_addr":"172.21.15.162:6825/1231011554","cluster_addr":"172.21.15.162:6827/1231011554","heartbeat_back_addr":"172.21.15.162:6831/1231011554","heartbeat_front_addr":"172.21.15.162:6829/1231011554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:55:49.995244+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:15.655543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:40.331165+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:04.688881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:29.174966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:53.663161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:17.407551+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:42.584296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:06.317219+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:31.078625+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:55.689269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T08:00:19.659748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1},{"from":7,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/3842627255":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3838590070":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/1238193420":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6800/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6800/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3194864054":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3866031018":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/1497156658":"2024-09-20T07:54:02.418785+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T08:01:33.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.914+0000 7f4c9c947640 1 -- 172.21.15.98:0/4060415556 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4c800a32d0 msgr2=0x7f4c800a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:33.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.914+0000 7f4c9c947640 1 --2- 172.21.15.98:0/4060415556 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4c800a32d0 0x7f4c800a5790 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009900 tx=0x7f4c880058d0 comp rx=0 tx=0).stop 2024-09-19T08:01:33.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.914+0000 7f4c9c947640 1 -- 172.21.15.98:0/4060415556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c98101420 msgr2=0x7f4c98197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:33.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.914+0000 7f4c9c947640 1 --2- 172.21.15.98:0/4060415556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c98101420 0x7f4c98197840 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c00d950 tx=0x7f4c8c00de10 comp rx=0 tx=0).stop 2024-09-19T08:01:33.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.914+0000 7f4c9c947640 1 -- 172.21.15.98:0/4060415556 shutdown_connections 2024-09-19T08:01:33.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.914+0000 7f4c9c947640 1 --2- 172.21.15.98:0/4060415556 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4c800a32d0 0x7f4c800a5790 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:33.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.914+0000 7f4c9c947640 1 --2- 172.21.15.98:0/4060415556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c98106a80 0x7f4c9819bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:33.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.914+0000 7f4c9c947640 1 --2- 172.21.15.98:0/4060415556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c98101420 0x7f4c98197840 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:33.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.914+0000 7f4c9c947640 1 --2- 172.21.15.98:0/4060415556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c98100a80 0x7f4c98197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:33.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.914+0000 7f4c9c947640 1 -- 172.21.15.98:0/4060415556 >> 172.21.15.98:0/4060415556 conn(0x7f4c980fc7a0 msgr2=0x7f4c980fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:33.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.914+0000 7f4c9c947640 1 -- 172.21.15.98:0/4060415556 shutdown_connections 2024-09-19T08:01:33.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:33.914+0000 7f4c9c947640 1 -- 172.21.15.98:0/4060415556 wait complete. 2024-09-19T08:01:34.595 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-19T08:01:34.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-19T08:01:34.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:34 smithi098 ceph-mon[31333]: pgmap v284: 97 pgs: 97 active+clean; 606 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.6 KiB/s wr, 12 op/s; 156 B/s, 1 keys/s, 2 objects/s recovering 2024-09-19T08:01:34.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:34 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/4060415556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:34.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:34 smithi162 ceph-mon[35260]: pgmap v284: 97 pgs: 97 active+clean; 606 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.6 KiB/s wr, 12 op/s; 156 B/s, 1 keys/s, 2 objects/s recovering 2024-09-19T08:01:34.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:34 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/4060415556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:34.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:34 smithi139 ceph-mon[34874]: pgmap v284: 97 pgs: 97 active+clean; 606 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.6 KiB/s wr, 12 op/s; 156 B/s, 1 keys/s, 2 objects/s recovering 2024-09-19T08:01:34.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:34 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/4060415556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.346+0000 7fdfb113d640 1 -- 172.21.15.98:0/4196233371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfac102ef0 msgr2=0x7fdfac10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.346+0000 7fdfb113d640 1 --2- 172.21.15.98:0/4196233371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfac102ef0 0x7fdfac10da40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009900 tx=0x7fdf9c02a480 comp rx=0 tx=0).stop 2024-09-19T08:01:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.346+0000 7fdfb113d640 1 -- 172.21.15.98:0/4196233371 shutdown_connections 2024-09-19T08:01:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.346+0000 7fdfb113d640 1 --2- 172.21.15.98:0/4196233371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfac102ef0 0x7fdfac10da40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.346+0000 7fdfb113d640 1 --2- 172.21.15.98:0/4196233371 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdfac102570 0x7fdfac1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.346+0000 7fdfb113d640 1 --2- 172.21.15.98:0/4196233371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfac100e20 0x7fdfac101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.346+0000 7fdfb113d640 1 -- 172.21.15.98:0/4196233371 >> 172.21.15.98:0/4196233371 conn(0x7fdfac0fc7d0 msgr2=0x7fdfac0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.346+0000 7fdfb113d640 1 -- 172.21.15.98:0/4196233371 shutdown_connections 2024-09-19T08:01:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.346+0000 7fdfb113d640 1 -- 172.21.15.98:0/4196233371 wait complete. 2024-09-19T08:01:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.346+0000 7fdfb113d640 1 Processor -- start 2024-09-19T08:01:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfb113d640 1 -- start start 2024-09-19T08:01:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfb113d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdfac100e20 0x7fdfac199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfb113d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfac102570 0x7fdfac199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfb113d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfac102ef0 0x7fdfac1a4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfb113d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac111020 con 0x7fdfac102ef0 2024-09-19T08:01:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfb113d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac110ea0 con 0x7fdfac100e20 2024-09-19T08:01:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfb113d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac1111a0 con 0x7fdfac102570 2024-09-19T08:01:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfab59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfac102ef0 0x7fdfac1a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfab59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfac102ef0 0x7fdfac1a4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46326/0 (socket says 172.21.15.98:46326) 2024-09-19T08:01:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfab59e640 1 -- 172.21.15.98:0/4221793014 learned_addr learned my addr 172.21.15.98:0/4221793014 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfaa59c640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfac102570 0x7fdfac199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfaad9d640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdfac100e20 0x7fdfac199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfab59e640 1 -- 172.21.15.98:0/4221793014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfac102570 msgr2=0x7fdfac199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfab59e640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfac102570 0x7fdfac199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfab59e640 1 -- 172.21.15.98:0/4221793014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdfac100e20 msgr2=0x7fdfac199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfab59e640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdfac100e20 0x7fdfac199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfab59e640 1 -- 172.21.15.98:0/4221793014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfac1a49d0 con 0x7fdfac102ef0 2024-09-19T08:01:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfaa59c640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfac102570 0x7fdfac199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:01:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.347+0000 7fdfaad9d640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdfac100e20 0x7fdfac199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:01:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.348+0000 7fdfab59e640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfac102ef0 0x7fdfac1a4490 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c005ab0 tx=0x7fdf9c003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.348+0000 7fdfa3fff640 1 -- 172.21.15.98:0/4221793014 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c03f070 con 0x7fdfac102ef0 2024-09-19T08:01:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.348+0000 7fdfb113d640 1 -- 172.21.15.98:0/4221793014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfac1a4ba0 con 0x7fdfac102ef0 2024-09-19T08:01:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.348+0000 7fdfa3fff640 1 -- 172.21.15.98:0/4221793014 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf9c031e30 con 0x7fdfac102ef0 2024-09-19T08:01:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.348+0000 7fdfa3fff640 1 -- 172.21.15.98:0/4221793014 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c035030 con 0x7fdfac102ef0 2024-09-19T08:01:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.348+0000 7fdfb113d640 1 -- 172.21.15.98:0/4221793014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfac1a5030 con 0x7fdfac102ef0 2024-09-19T08:01:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.349+0000 7fdfb113d640 1 -- 172.21.15.98:0/4221793014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfac101280 con 0x7fdfac102ef0 2024-09-19T08:01:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.350+0000 7fdfa3fff640 1 -- 172.21.15.98:0/4221793014 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fdf9c042070 con 0x7fdfac102ef0 2024-09-19T08:01:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.351+0000 7fdfa3fff640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdf980a32d0 0x7fdf980a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.351+0000 7fdfaad9d640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdf980a32d0 0x7fdf980a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.351+0000 7fdfa3fff640 1 -- 172.21.15.98:0/4221793014 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fdf9c0f03d0 con 0x7fdfac102ef0 2024-09-19T08:01:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.351+0000 7fdfaad9d640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdf980a32d0 0x7fdf980a5790 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4007eb0 tx=0x7fdfa40073c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.355+0000 7fdfa3fff640 1 -- 172.21.15.98:0/4221793014 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fdf9c003e20 con 0x7fdfac102ef0 2024-09-19T08:01:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.521+0000 7fdfb113d640 1 -- 172.21.15.98:0/4221793014 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdfac109100 con 0x7fdf980a32d0 2024-09-19T08:01:35.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.527+0000 7fdfa3fff640 1 -- 172.21.15.98:0/4221793014 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258774 (secure 0 0 0) 0x7fdfa400cdd0 con 0x7fdf980a32d0 2024-09-19T08:01:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:01:35.535 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-19T08:01:35.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.530+0000 7fdfb113d640 1 -- 172.21.15.98:0/4221793014 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdf980a32d0 msgr2=0x7fdf980a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:35.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.530+0000 7fdfb113d640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdf980a32d0 0x7fdf980a5790 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4007eb0 tx=0x7fdfa40073c0 comp rx=0 tx=0).stop 2024-09-19T08:01:35.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.530+0000 7fdfb113d640 1 -- 172.21.15.98:0/4221793014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfac102ef0 msgr2=0x7fdfac1a4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:35.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.530+0000 7fdfb113d640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfac102ef0 0x7fdfac1a4490 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c005ab0 tx=0x7fdf9c003c30 comp rx=0 tx=0).stop 2024-09-19T08:01:35.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.530+0000 7fdfb113d640 1 -- 172.21.15.98:0/4221793014 shutdown_connections 2024-09-19T08:01:35.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.530+0000 7fdfb113d640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdf980a32d0 0x7fdf980a5790 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:35.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.530+0000 7fdfb113d640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfac102ef0 0x7fdfac1a4490 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:35.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.530+0000 7fdfb113d640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfac102570 0x7fdfac199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:35.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.530+0000 7fdfb113d640 1 --2- 172.21.15.98:0/4221793014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdfac100e20 0x7fdfac199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:35.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.530+0000 7fdfb113d640 1 -- 172.21.15.98:0/4221793014 >> 172.21.15.98:0/4221793014 conn(0x7fdfac0fc7d0 msgr2=0x7fdfac0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:35.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.530+0000 7fdfb113d640 1 -- 172.21.15.98:0/4221793014 shutdown_connections 2024-09-19T08:01:35.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:35.530+0000 7fdfb113d640 1 -- 172.21.15.98:0/4221793014 wait complete. 2024-09-19T08:01:36.306 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-09-19T08:01:33.554927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":620142,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":255,"num_read_kb":210,"num_write":401,"num_write_kb":3028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181990,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":329,"ondisk_log_size":329,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":290,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369976,"kb_used_data":12360,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571848,"statfs":{"total":1151940427776,"available":1151561572352,"internally_reserved":0,"allocated":12656640,"data_stored":6978620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21774,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":77,"num_read_kb":56,"num_write":21,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1254,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001799"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719055+0000","last_change":"2024-09-19T08:01:10.768950+0000","last_active":"2024-09-19T08:01:26.719055+0000","last_peered":"2024-09-19T08:01:26.719055+0000","last_clean":"2024-09-19T08:01:26.719055+0000","last_became_active":"2024-09-19T08:01:10.768047+0000","last_became_peered":"2024-09-19T08:01:10.768047+0000","last_unstale":"2024-09-19T08:01:26.719055+0000","last_undegraded":"2024-09-19T08:01:26.719055+0000","last_fullsized":"2024-09-19T08:01:26.719055+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:26.219079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757858+0000","last_change":"2024-09-19T08:01:10.783208+0000","last_active":"2024-09-19T08:01:26.757858+0000","last_peered":"2024-09-19T08:01:26.757858+0000","last_clean":"2024-09-19T08:01:26.757858+0000","last_became_active":"2024-09-19T08:01:10.782888+0000","last_became_peered":"2024-09-19T08:01:10.782888+0000","last_unstale":"2024-09-19T08:01:26.757858+0000","last_undegraded":"2024-09-19T08:01:26.757858+0000","last_fullsized":"2024-09-19T08:01:26.757858+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:51:27.603166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757759+0000","last_change":"2024-09-19T08:01:10.770395+0000","last_active":"2024-09-19T08:01:26.757759+0000","last_peered":"2024-09-19T08:01:26.757759+0000","last_clean":"2024-09-19T08:01:26.757759+0000","last_became_active":"2024-09-19T08:01:10.770073+0000","last_became_peered":"2024-09-19T08:01:10.770073+0000","last_unstale":"2024-09-19T08:01:26.757759+0000","last_undegraded":"2024-09-19T08:01:26.757759+0000","last_fullsized":"2024-09-19T08:01:26.757759+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:20.047023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758411+0000","last_change":"2024-09-19T08:01:10.777138+0000","last_active":"2024-09-19T08:01:26.758411+0000","last_peered":"2024-09-19T08:01:26.758411+0000","last_clean":"2024-09-19T08:01:26.758411+0000","last_became_active":"2024-09-19T08:01:10.776810+0000","last_became_peered":"2024-09-19T08:01:10.776810+0000","last_unstale":"2024-09-19T08:01:26.758411+0000","last_undegraded":"2024-09-19T08:01:26.758411+0000","last_fullsized":"2024-09-19T08:01:26.758411+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:39.983321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721379+0000","last_change":"2024-09-19T08:01:10.770435+0000","last_active":"2024-09-19T08:01:26.721379+0000","last_peered":"2024-09-19T08:01:26.721379+0000","last_clean":"2024-09-19T08:01:26.721379+0000","last_became_active":"2024-09-19T08:01:10.769686+0000","last_became_peered":"2024-09-19T08:01:10.769686+0000","last_unstale":"2024-09-19T08:01:26.721379+0000","last_undegraded":"2024-09-19T08:01:26.721379+0000","last_fullsized":"2024-09-19T08:01:26.721379+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:10:19.610925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718127+0000","last_change":"2024-09-19T08:01:10.782382+0000","last_active":"2024-09-19T08:01:26.718127+0000","last_peered":"2024-09-19T08:01:26.718127+0000","last_clean":"2024-09-19T08:01:26.718127+0000","last_became_active":"2024-09-19T08:01:10.781996+0000","last_became_peered":"2024-09-19T08:01:10.781996+0000","last_unstale":"2024-09-19T08:01:26.718127+0000","last_undegraded":"2024-09-19T08:01:26.718127+0000","last_fullsized":"2024-09-19T08:01:26.718127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:43:13.574336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722516+0000","last_change":"2024-09-19T08:01:10.773776+0000","last_active":"2024-09-19T08:01:26.722516+0000","last_peered":"2024-09-19T08:01:26.722516+0000","last_clean":"2024-09-19T08:01:26.722516+0000","last_became_active":"2024-09-19T08:01:10.772442+0000","last_became_peered":"2024-09-19T08:01:10.772442+0000","last_unstale":"2024-09-19T08:01:26.722516+0000","last_undegraded":"2024-09-19T08:01:26.722516+0000","last_fullsized":"2024-09-19T08:01:26.722516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:08:18.155806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.717962+0000","last_change":"2024-09-19T08:01:10.781443+0000","last_active":"2024-09-19T08:01:26.717962+0000","last_peered":"2024-09-19T08:01:26.717962+0000","last_clean":"2024-09-19T08:01:26.717962+0000","last_became_active":"2024-09-19T08:01:10.781083+0000","last_became_peered":"2024-09-19T08:01:10.781083+0000","last_unstale":"2024-09-19T08:01:26.717962+0000","last_undegraded":"2024-09-19T08:01:26.717962+0000","last_fullsized":"2024-09-19T08:01:26.717962+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:23:22.631865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721840+0000","last_change":"2024-09-19T08:01:10.767554+0000","last_active":"2024-09-19T08:01:26.721840+0000","last_peered":"2024-09-19T08:01:26.721840+0000","last_clean":"2024-09-19T08:01:26.721840+0000","last_became_active":"2024-09-19T08:01:10.767227+0000","last_became_peered":"2024-09-19T08:01:10.767227+0000","last_unstale":"2024-09-19T08:01:26.721840+0000","last_undegraded":"2024-09-19T08:01:26.721840+0000","last_fullsized":"2024-09-19T08:01:26.721840+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:30:13.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718514+0000","last_change":"2024-09-19T08:01:10.783174+0000","last_active":"2024-09-19T08:01:26.718514+0000","last_peered":"2024-09-19T08:01:26.718514+0000","last_clean":"2024-09-19T08:01:26.718514+0000","last_became_active":"2024-09-19T08:01:10.782794+0000","last_became_peered":"2024-09-19T08:01:10.782794+0000","last_unstale":"2024-09-19T08:01:26.718514+0000","last_undegraded":"2024-09-19T08:01:26.718514+0000","last_fullsized":"2024-09-19T08:01:26.718514+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:49:19.451107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723075+0000","last_change":"2024-09-19T08:01:10.779638+0000","last_active":"2024-09-19T08:01:26.723075+0000","last_peered":"2024-09-19T08:01:26.723075+0000","last_clean":"2024-09-19T08:01:26.723075+0000","last_became_active":"2024-09-19T08:01:10.779295+0000","last_became_peered":"2024-09-19T08:01:10.779295+0000","last_unstale":"2024-09-19T08:01:26.723075+0000","last_undegraded":"2024-09-19T08:01:26.723075+0000","last_fullsized":"2024-09-19T08:01:26.723075+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:35:10.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758492+0000","last_change":"2024-09-19T08:01:10.784487+0000","last_active":"2024-09-19T08:01:26.758492+0000","last_peered":"2024-09-19T08:01:26.758492+0000","last_clean":"2024-09-19T08:01:26.758492+0000","last_became_active":"2024-09-19T08:01:10.784114+0000","last_became_peered":"2024-09-19T08:01:10.784114+0000","last_unstale":"2024-09-19T08:01:26.758492+0000","last_undegraded":"2024-09-19T08:01:26.758492+0000","last_fullsized":"2024-09-19T08:01:26.758492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:09:54.819802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721613+0000","last_change":"2024-09-19T08:01:10.783186+0000","last_active":"2024-09-19T08:01:26.721613+0000","last_peered":"2024-09-19T08:01:26.721613+0000","last_clean":"2024-09-19T08:01:26.721613+0000","last_became_active":"2024-09-19T08:01:10.782876+0000","last_became_peered":"2024-09-19T08:01:10.782876+0000","last_unstale":"2024-09-19T08:01:26.721613+0000","last_undegraded":"2024-09-19T08:01:26.721613+0000","last_fullsized":"2024-09-19T08:01:26.721613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:26:15.252165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757647+0000","last_change":"2024-09-19T08:01:10.783237+0000","last_active":"2024-09-19T08:01:26.757647+0000","last_peered":"2024-09-19T08:01:26.757647+0000","last_clean":"2024-09-19T08:01:26.757647+0000","last_became_active":"2024-09-19T08:01:10.782966+0000","last_became_peered":"2024-09-19T08:01:10.782966+0000","last_unstale":"2024-09-19T08:01:26.757647+0000","last_undegraded":"2024-09-19T08:01:26.757647+0000","last_fullsized":"2024-09-19T08:01:26.757647+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:57:18.771737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757535+0000","last_change":"2024-09-19T08:01:10.771207+0000","last_active":"2024-09-19T08:01:26.757535+0000","last_peered":"2024-09-19T08:01:26.757535+0000","last_clean":"2024-09-19T08:01:26.757535+0000","last_became_active":"2024-09-19T08:01:10.770862+0000","last_became_peered":"2024-09-19T08:01:10.770862+0000","last_unstale":"2024-09-19T08:01:26.757535+0000","last_undegraded":"2024-09-19T08:01:26.757535+0000","last_fullsized":"2024-09-19T08:01:26.757535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:08:09.313329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.720764+0000","last_change":"2024-09-19T08:01:10.775191+0000","last_active":"2024-09-19T08:01:26.720764+0000","last_peered":"2024-09-19T08:01:26.720764+0000","last_clean":"2024-09-19T08:01:26.720764+0000","last_became_active":"2024-09-19T08:01:10.774832+0000","last_became_peered":"2024-09-19T08:01:10.774832+0000","last_unstale":"2024-09-19T08:01:26.720764+0000","last_undegraded":"2024-09-19T08:01:26.720764+0000","last_fullsized":"2024-09-19T08:01:26.720764+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:57:29.800787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719076+0000","last_change":"2024-09-19T08:01:10.769115+0000","last_active":"2024-09-19T08:01:26.719076+0000","last_peered":"2024-09-19T08:01:26.719076+0000","last_clean":"2024-09-19T08:01:26.719076+0000","last_became_active":"2024-09-19T08:01:10.768088+0000","last_became_peered":"2024-09-19T08:01:10.768088+0000","last_unstale":"2024-09-19T08:01:26.719076+0000","last_undegraded":"2024-09-19T08:01:26.719076+0000","last_fullsized":"2024-09-19T08:01:26.719076+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:27:16.042722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719916+0000","last_change":"2024-09-19T08:01:10.781426+0000","last_active":"2024-09-19T08:01:26.719916+0000","last_peered":"2024-09-19T08:01:26.719916+0000","last_clean":"2024-09-19T08:01:26.719916+0000","last_became_active":"2024-09-19T08:01:10.781094+0000","last_became_peered":"2024-09-19T08:01:10.781094+0000","last_unstale":"2024-09-19T08:01:26.719916+0000","last_undegraded":"2024-09-19T08:01:26.719916+0000","last_fullsized":"2024-09-19T08:01:26.719916+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:43:31.336610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725515+0000","last_change":"2024-09-19T08:01:10.766466+0000","last_active":"2024-09-19T08:01:26.725515+0000","last_peered":"2024-09-19T08:01:26.725515+0000","last_clean":"2024-09-19T08:01:26.725515+0000","last_became_active":"2024-09-19T08:01:10.766123+0000","last_became_peered":"2024-09-19T08:01:10.766123+0000","last_unstale":"2024-09-19T08:01:26.725515+0000","last_undegraded":"2024-09-19T08:01:26.725515+0000","last_fullsized":"2024-09-19T08:01:26.725515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:25.848603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718613+0000","last_change":"2024-09-19T08:01:10.772782+0000","last_active":"2024-09-19T08:01:26.718613+0000","last_peered":"2024-09-19T08:01:26.718613+0000","last_clean":"2024-09-19T08:01:26.718613+0000","last_became_active":"2024-09-19T08:01:10.772445+0000","last_became_peered":"2024-09-19T08:01:10.772445+0000","last_unstale":"2024-09-19T08:01:26.718613+0000","last_undegraded":"2024-09-19T08:01:26.718613+0000","last_fullsized":"2024-09-19T08:01:26.718613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:09.174005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721701+0000","last_change":"2024-09-19T08:01:10.771944+0000","last_active":"2024-09-19T08:01:26.721701+0000","last_peered":"2024-09-19T08:01:26.721701+0000","last_clean":"2024-09-19T08:01:26.721701+0000","last_became_active":"2024-09-19T08:01:10.771643+0000","last_became_peered":"2024-09-19T08:01:10.771643+0000","last_unstale":"2024-09-19T08:01:26.721701+0000","last_undegraded":"2024-09-19T08:01:26.721701+0000","last_fullsized":"2024-09-19T08:01:26.721701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:46:27.121722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721201+0000","last_change":"2024-09-19T08:01:10.775144+0000","last_active":"2024-09-19T08:01:26.721201+0000","last_peered":"2024-09-19T08:01:26.721201+0000","last_clean":"2024-09-19T08:01:26.721201+0000","last_became_active":"2024-09-19T08:01:10.774769+0000","last_became_peered":"2024-09-19T08:01:10.774769+0000","last_unstale":"2024-09-19T08:01:26.721201+0000","last_undegraded":"2024-09-19T08:01:26.721201+0000","last_fullsized":"2024-09-19T08:01:26.721201+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:59:47.053993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718014+0000","last_change":"2024-09-19T08:01:10.766750+0000","last_active":"2024-09-19T08:01:26.718014+0000","last_peered":"2024-09-19T08:01:26.718014+0000","last_clean":"2024-09-19T08:01:26.718014+0000","last_became_active":"2024-09-19T08:01:10.766450+0000","last_became_peered":"2024-09-19T08:01:10.766450+0000","last_unstale":"2024-09-19T08:01:26.718014+0000","last_undegraded":"2024-09-19T08:01:26.718014+0000","last_fullsized":"2024-09-19T08:01:26.718014+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:47:58.133815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721798+0000","last_change":"2024-09-19T08:01:10.768430+0000","last_active":"2024-09-19T08:01:26.721798+0000","last_peered":"2024-09-19T08:01:26.721798+0000","last_clean":"2024-09-19T08:01:26.721798+0000","last_became_active":"2024-09-19T08:01:10.768092+0000","last_became_peered":"2024-09-19T08:01:10.768092+0000","last_unstale":"2024-09-19T08:01:26.721798+0000","last_undegraded":"2024-09-19T08:01:26.721798+0000","last_fullsized":"2024-09-19T08:01:26.721798+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:16:42.343138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718376+0000","last_change":"2024-09-19T08:01:10.771131+0000","last_active":"2024-09-19T08:01:26.718376+0000","last_peered":"2024-09-19T08:01:26.718376+0000","last_clean":"2024-09-19T08:01:26.718376+0000","last_became_active":"2024-09-19T08:01:10.770851+0000","last_became_peered":"2024-09-19T08:01:10.770851+0000","last_unstale":"2024-09-19T08:01:26.718376+0000","last_undegraded":"2024-09-19T08:01:26.718376+0000","last_fullsized":"2024-09-19T08:01:26.718376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:07:15.830476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.726129+0000","last_change":"2024-09-19T08:01:10.776220+0000","last_active":"2024-09-19T08:01:26.726129+0000","last_peered":"2024-09-19T08:01:26.726129+0000","last_clean":"2024-09-19T08:01:26.726129+0000","last_became_active":"2024-09-19T08:01:10.775889+0000","last_became_peered":"2024-09-19T08:01:10.775889+0000","last_unstale":"2024-09-19T08:01:26.726129+0000","last_undegraded":"2024-09-19T08:01:26.726129+0000","last_fullsized":"2024-09-19T08:01:26.726129+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:23.714675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719003+0000","last_change":"2024-09-19T08:01:10.776370+0000","last_active":"2024-09-19T08:01:26.719003+0000","last_peered":"2024-09-19T08:01:26.719003+0000","last_clean":"2024-09-19T08:01:26.719003+0000","last_became_active":"2024-09-19T08:01:10.776027+0000","last_became_peered":"2024-09-19T08:01:10.776027+0000","last_unstale":"2024-09-19T08:01:26.719003+0000","last_undegraded":"2024-09-19T08:01:26.719003+0000","last_fullsized":"2024-09-19T08:01:26.719003+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:48:46.293404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758971+0000","last_change":"2024-09-19T08:01:26.758971+0000","last_active":"2024-09-19T08:01:26.758971+0000","last_peered":"2024-09-19T08:01:26.758971+0000","last_clean":"2024-09-19T08:01:26.758971+0000","last_became_active":"2024-09-19T08:01:26.758524+0000","last_became_peered":"2024-09-19T08:01:26.758524+0000","last_unstale":"2024-09-19T08:01:26.758971+0000","last_undegraded":"2024-09-19T08:01:26.758971+0000","last_fullsized":"2024-09-19T08:01:26.758971+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:34:22.439327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757760+0000","last_change":"2024-09-19T08:01:10.767445+0000","last_active":"2024-09-19T08:01:26.757760+0000","last_peered":"2024-09-19T08:01:26.757760+0000","last_clean":"2024-09-19T08:01:26.757760+0000","last_became_active":"2024-09-19T08:01:10.767128+0000","last_became_peered":"2024-09-19T08:01:10.767128+0000","last_unstale":"2024-09-19T08:01:26.757760+0000","last_undegraded":"2024-09-19T08:01:26.757760+0000","last_fullsized":"2024-09-19T08:01:26.757760+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:10.273948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725212+0000","last_change":"2024-09-19T08:01:10.768954+0000","last_active":"2024-09-19T08:01:26.725212+0000","last_peered":"2024-09-19T08:01:26.725212+0000","last_clean":"2024-09-19T08:01:26.725212+0000","last_became_active":"2024-09-19T08:01:10.768577+0000","last_became_peered":"2024-09-19T08:01:10.768577+0000","last_unstale":"2024-09-19T08:01:26.725212+0000","last_undegraded":"2024-09-19T08:01:26.725212+0000","last_fullsized":"2024-09-19T08:01:26.725212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:20:21.812531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725473+0000","last_change":"2024-09-19T08:01:10.774104+0000","last_active":"2024-09-19T08:01:26.725473+0000","last_peered":"2024-09-19T08:01:26.725473+0000","last_clean":"2024-09-19T08:01:26.725473+0000","last_became_active":"2024-09-19T08:01:10.773721+0000","last_became_peered":"2024-09-19T08:01:10.773721+0000","last_unstale":"2024-09-19T08:01:26.725473+0000","last_undegraded":"2024-09-19T08:01:26.725473+0000","last_fullsized":"2024-09-19T08:01:26.725473+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:45:34.945994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.727296+0000","last_change":"2024-09-19T08:01:26.727296+0000","last_active":"2024-09-19T08:01:26.727296+0000","last_peered":"2024-09-19T08:01:26.727296+0000","last_clean":"2024-09-19T08:01:26.727296+0000","last_became_active":"2024-09-19T08:01:26.726631+0000","last_became_peered":"2024-09-19T08:01:26.726631+0000","last_unstale":"2024-09-19T08:01:26.727296+0000","last_undegraded":"2024-09-19T08:01:26.727296+0000","last_fullsized":"2024-09-19T08:01:26.727296+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:19:24.807050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721044+0000","last_change":"2024-09-19T08:01:10.765282+0000","last_active":"2024-09-19T08:01:26.721044+0000","last_peered":"2024-09-19T08:01:26.721044+0000","last_clean":"2024-09-19T08:01:26.721044+0000","last_became_active":"2024-09-19T08:01:10.764928+0000","last_became_peered":"2024-09-19T08:01:10.764928+0000","last_unstale":"2024-09-19T08:01:26.721044+0000","last_undegraded":"2024-09-19T08:01:26.721044+0000","last_fullsized":"2024-09-19T08:01:26.721044+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:00:37.395110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"78'6","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:29.701406+0000","last_change":"2024-09-19T08:01:08.760686+0000","last_active":"2024-09-19T08:01:29.701406+0000","last_peered":"2024-09-19T08:01:29.701406+0000","last_clean":"2024-09-19T08:01:29.701406+0000","last_became_active":"2024-09-19T08:01:08.759912+0000","last_became_peered":"2024-09-19T08:01:08.759912+0000","last_unstale":"2024-09-19T08:01:29.701406+0000","last_undegraded":"2024-09-19T08:01:29.701406+0000","last_fullsized":"2024-09-19T08:01:29.701406+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:46:01.879611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721868+0000","last_change":"2024-09-19T08:01:10.779668+0000","last_active":"2024-09-19T08:01:26.721868+0000","last_peered":"2024-09-19T08:01:26.721868+0000","last_clean":"2024-09-19T08:01:26.721868+0000","last_became_active":"2024-09-19T08:01:10.779265+0000","last_became_peered":"2024-09-19T08:01:10.779265+0000","last_unstale":"2024-09-19T08:01:26.721868+0000","last_undegraded":"2024-09-19T08:01:26.721868+0000","last_fullsized":"2024-09-19T08:01:26.721868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:06.177700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718183+0000","last_change":"2024-09-19T08:01:08.762668+0000","last_active":"2024-09-19T08:01:26.718183+0000","last_peered":"2024-09-19T08:01:26.718183+0000","last_clean":"2024-09-19T08:01:26.718183+0000","last_became_active":"2024-09-19T08:01:08.762277+0000","last_became_peered":"2024-09-19T08:01:08.762277+0000","last_unstale":"2024-09-19T08:01:26.718183+0000","last_undegraded":"2024-09-19T08:01:26.718183+0000","last_fullsized":"2024-09-19T08:01:26.718183+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:36:59.316655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721452+0000","last_change":"2024-09-19T08:01:08.757401+0000","last_active":"2024-09-19T08:01:26.721452+0000","last_peered":"2024-09-19T08:01:26.721452+0000","last_clean":"2024-09-19T08:01:26.721452+0000","last_became_active":"2024-09-19T08:01:08.757067+0000","last_became_peered":"2024-09-19T08:01:08.757067+0000","last_unstale":"2024-09-19T08:01:26.721452+0000","last_undegraded":"2024-09-19T08:01:26.721452+0000","last_fullsized":"2024-09-19T08:01:26.721452+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:19.617830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718027+0000","last_change":"2024-09-19T08:01:10.765149+0000","last_active":"2024-09-19T08:01:26.718027+0000","last_peered":"2024-09-19T08:01:26.718027+0000","last_clean":"2024-09-19T08:01:26.718027+0000","last_became_active":"2024-09-19T08:01:10.764840+0000","last_became_peered":"2024-09-19T08:01:10.764840+0000","last_unstale":"2024-09-19T08:01:26.718027+0000","last_undegraded":"2024-09-19T08:01:26.718027+0000","last_fullsized":"2024-09-19T08:01:26.718027+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:15.539307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722074+0000","last_change":"2024-09-19T08:01:08.759435+0000","last_active":"2024-09-19T08:01:26.722074+0000","last_peered":"2024-09-19T08:01:26.722074+0000","last_clean":"2024-09-19T08:01:26.722074+0000","last_became_active":"2024-09-19T08:01:08.759046+0000","last_became_peered":"2024-09-19T08:01:08.759046+0000","last_unstale":"2024-09-19T08:01:26.722074+0000","last_undegraded":"2024-09-19T08:01:26.722074+0000","last_fullsized":"2024-09-19T08:01:26.722074+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:21:24.077626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719748+0000","last_change":"2024-09-19T08:01:10.778432+0000","last_active":"2024-09-19T08:01:26.719748+0000","last_peered":"2024-09-19T08:01:26.719748+0000","last_clean":"2024-09-19T08:01:26.719748+0000","last_became_active":"2024-09-19T08:01:10.778098+0000","last_became_peered":"2024-09-19T08:01:10.778098+0000","last_unstale":"2024-09-19T08:01:26.719748+0000","last_undegraded":"2024-09-19T08:01:26.719748+0000","last_fullsized":"2024-09-19T08:01:26.719748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:40:12.921165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"78'4","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:27.697969+0000","last_change":"2024-09-19T08:01:08.764117+0000","last_active":"2024-09-19T08:01:27.697969+0000","last_peered":"2024-09-19T08:01:27.697969+0000","last_clean":"2024-09-19T08:01:27.697969+0000","last_became_active":"2024-09-19T08:01:08.763308+0000","last_became_peered":"2024-09-19T08:01:08.763308+0000","last_unstale":"2024-09-19T08:01:27.697969+0000","last_undegraded":"2024-09-19T08:01:27.697969+0000","last_fullsized":"2024-09-19T08:01:27.697969+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:19:57.014628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721336+0000","last_change":"2024-09-19T08:01:10.781771+0000","last_active":"2024-09-19T08:01:26.721336+0000","last_peered":"2024-09-19T08:01:26.721336+0000","last_clean":"2024-09-19T08:01:26.721336+0000","last_became_active":"2024-09-19T08:01:10.781432+0000","last_became_peered":"2024-09-19T08:01:10.781432+0000","last_unstale":"2024-09-19T08:01:26.721336+0000","last_undegraded":"2024-09-19T08:01:26.721336+0000","last_fullsized":"2024-09-19T08:01:26.721336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:25:15.067850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.771171+0000","last_change":"2024-09-19T08:01:26.771171+0000","last_active":"2024-09-19T08:01:26.771171+0000","last_peered":"2024-09-19T08:01:26.771171+0000","last_clean":"2024-09-19T08:01:26.771171+0000","last_became_active":"2024-09-19T08:01:26.759274+0000","last_became_peered":"2024-09-19T08:01:26.759274+0000","last_unstale":"2024-09-19T08:01:26.771171+0000","last_undegraded":"2024-09-19T08:01:26.771171+0000","last_fullsized":"2024-09-19T08:01:26.771171+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:20:54.603420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757967+0000","last_change":"2024-09-19T08:01:10.768318+0000","last_active":"2024-09-19T08:01:26.757967+0000","last_peered":"2024-09-19T08:01:26.757967+0000","last_clean":"2024-09-19T08:01:26.757967+0000","last_became_active":"2024-09-19T08:01:10.768022+0000","last_became_peered":"2024-09-19T08:01:10.768022+0000","last_unstale":"2024-09-19T08:01:26.757967+0000","last_undegraded":"2024-09-19T08:01:26.757967+0000","last_fullsized":"2024-09-19T08:01:26.757967+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:00:45.328638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722058+0000","last_change":"2024-09-19T08:01:09.750530+0000","last_active":"2024-09-19T08:01:26.722058+0000","last_peered":"2024-09-19T08:01:26.722058+0000","last_clean":"2024-09-19T08:01:26.722058+0000","last_became_active":"2024-09-19T08:01:09.750029+0000","last_became_peered":"2024-09-19T08:01:09.750029+0000","last_unstale":"2024-09-19T08:01:26.722058+0000","last_undegraded":"2024-09-19T08:01:26.722058+0000","last_fullsized":"2024-09-19T08:01:26.722058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:01:16.062992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.724636+0000","last_change":"2024-09-19T08:01:26.724636+0000","last_active":"2024-09-19T08:01:26.724636+0000","last_peered":"2024-09-19T08:01:26.724636+0000","last_clean":"2024-09-19T08:01:26.724636+0000","last_became_active":"2024-09-19T08:01:26.724171+0000","last_became_peered":"2024-09-19T08:01:26.724171+0000","last_unstale":"2024-09-19T08:01:26.724636+0000","last_undegraded":"2024-09-19T08:01:26.724636+0000","last_fullsized":"2024-09-19T08:01:26.724636+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:54:05.998374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"79'17","reported_seq":59,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:27.698831+0000","last_change":"2024-09-19T08:01:08.761950+0000","last_active":"2024-09-19T08:01:27.698831+0000","last_peered":"2024-09-19T08:01:27.698831+0000","last_clean":"2024-09-19T08:01:27.698831+0000","last_became_active":"2024-09-19T08:01:08.761533+0000","last_became_peered":"2024-09-19T08:01:08.761533+0000","last_unstale":"2024-09-19T08:01:27.698831+0000","last_undegraded":"2024-09-19T08:01:27.698831+0000","last_fullsized":"2024-09-19T08:01:27.698831+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:07:26.668141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22332,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":31,"num_write":19,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723579+0000","last_change":"2024-09-19T08:01:10.774280+0000","last_active":"2024-09-19T08:01:26.723579+0000","last_peered":"2024-09-19T08:01:26.723579+0000","last_clean":"2024-09-19T08:01:26.723579+0000","last_became_active":"2024-09-19T08:01:10.773982+0000","last_became_peered":"2024-09-19T08:01:10.773982+0000","last_unstale":"2024-09-19T08:01:26.723579+0000","last_undegraded":"2024-09-19T08:01:26.723579+0000","last_fullsized":"2024-09-19T08:01:26.723579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:21:43.180057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725363+0000","last_change":"2024-09-19T08:01:08.761067+0000","last_active":"2024-09-19T08:01:26.725363+0000","last_peered":"2024-09-19T08:01:26.725363+0000","last_clean":"2024-09-19T08:01:26.725363+0000","last_became_active":"2024-09-19T08:01:08.760664+0000","last_became_peered":"2024-09-19T08:01:08.760664+0000","last_unstale":"2024-09-19T08:01:26.725363+0000","last_undegraded":"2024-09-19T08:01:26.725363+0000","last_fullsized":"2024-09-19T08:01:26.725363+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:22:38.086891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721462+0000","last_change":"2024-09-19T08:01:10.771845+0000","last_active":"2024-09-19T08:01:26.721462+0000","last_peered":"2024-09-19T08:01:26.721462+0000","last_clean":"2024-09-19T08:01:26.721462+0000","last_became_active":"2024-09-19T08:01:10.771533+0000","last_became_peered":"2024-09-19T08:01:10.771533+0000","last_unstale":"2024-09-19T08:01:26.721462+0000","last_undegraded":"2024-09-19T08:01:26.721462+0000","last_fullsized":"2024-09-19T08:01:26.721462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:29:35.180763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722806+0000","last_change":"2024-09-19T08:01:08.763869+0000","last_active":"2024-09-19T08:01:26.722806+0000","last_peered":"2024-09-19T08:01:26.722806+0000","last_clean":"2024-09-19T08:01:26.722806+0000","last_became_active":"2024-09-19T08:01:08.763451+0000","last_became_peered":"2024-09-19T08:01:08.763451+0000","last_unstale":"2024-09-19T08:01:26.722806+0000","last_undegraded":"2024-09-19T08:01:26.722806+0000","last_fullsized":"2024-09-19T08:01:26.722806+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:32:40.901049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722991+0000","last_change":"2024-09-19T08:01:10.770823+0000","last_active":"2024-09-19T08:01:26.722991+0000","last_peered":"2024-09-19T08:01:26.722991+0000","last_clean":"2024-09-19T08:01:26.722991+0000","last_became_active":"2024-09-19T08:01:10.770462+0000","last_became_peered":"2024-09-19T08:01:10.770462+0000","last_unstale":"2024-09-19T08:01:26.722991+0000","last_undegraded":"2024-09-19T08:01:26.722991+0000","last_fullsized":"2024-09-19T08:01:26.722991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:04:32.832009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"78'5","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:29.705807+0000","last_change":"2024-09-19T08:01:08.759429+0000","last_active":"2024-09-19T08:01:29.705807+0000","last_peered":"2024-09-19T08:01:29.705807+0000","last_clean":"2024-09-19T08:01:29.705807+0000","last_became_active":"2024-09-19T08:01:08.759062+0000","last_became_peered":"2024-09-19T08:01:08.759062+0000","last_unstale":"2024-09-19T08:01:29.705807+0000","last_undegraded":"2024-09-19T08:01:29.705807+0000","last_fullsized":"2024-09-19T08:01:29.705807+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:16:51.582435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722418+0000","last_change":"2024-09-19T08:01:10.784735+0000","last_active":"2024-09-19T08:01:26.722418+0000","last_peered":"2024-09-19T08:01:26.722418+0000","last_clean":"2024-09-19T08:01:26.722418+0000","last_became_active":"2024-09-19T08:01:10.784346+0000","last_became_peered":"2024-09-19T08:01:10.784346+0000","last_unstale":"2024-09-19T08:01:26.722418+0000","last_undegraded":"2024-09-19T08:01:26.722418+0000","last_fullsized":"2024-09-19T08:01:26.722418+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:27:54.580391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721469+0000","last_change":"2024-09-19T08:01:09.749498+0000","last_active":"2024-09-19T08:01:26.721469+0000","last_peered":"2024-09-19T08:01:26.721469+0000","last_clean":"2024-09-19T08:01:26.721469+0000","last_became_active":"2024-09-19T08:01:09.749091+0000","last_became_peered":"2024-09-19T08:01:09.749091+0000","last_unstale":"2024-09-19T08:01:26.721469+0000","last_undegraded":"2024-09-19T08:01:26.721469+0000","last_fullsized":"2024-09-19T08:01:26.721469+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:56:13.644931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721781+0000","last_change":"2024-09-19T08:01:10.779599+0000","last_active":"2024-09-19T08:01:26.721781+0000","last_peered":"2024-09-19T08:01:26.721781+0000","last_clean":"2024-09-19T08:01:26.721781+0000","last_became_active":"2024-09-19T08:01:10.779219+0000","last_became_peered":"2024-09-19T08:01:10.779219+0000","last_unstale":"2024-09-19T08:01:26.721781+0000","last_undegraded":"2024-09-19T08:01:26.721781+0000","last_fullsized":"2024-09-19T08:01:26.721781+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:20.193753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723508+0000","last_change":"2024-09-19T08:01:08.761432+0000","last_active":"2024-09-19T08:01:26.723508+0000","last_peered":"2024-09-19T08:01:26.723508+0000","last_clean":"2024-09-19T08:01:26.723508+0000","last_became_active":"2024-09-19T08:01:08.761044+0000","last_became_peered":"2024-09-19T08:01:08.761044+0000","last_unstale":"2024-09-19T08:01:26.723508+0000","last_undegraded":"2024-09-19T08:01:26.723508+0000","last_fullsized":"2024-09-19T08:01:26.723508+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:12.237334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718915+0000","last_change":"2024-09-19T08:01:10.772559+0000","last_active":"2024-09-19T08:01:26.718915+0000","last_peered":"2024-09-19T08:01:26.718915+0000","last_clean":"2024-09-19T08:01:26.718915+0000","last_became_active":"2024-09-19T08:01:10.772176+0000","last_became_peered":"2024-09-19T08:01:10.772176+0000","last_unstale":"2024-09-19T08:01:26.718915+0000","last_undegraded":"2024-09-19T08:01:26.718915+0000","last_fullsized":"2024-09-19T08:01:26.718915+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:27:16.296127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721816+0000","last_change":"2024-09-19T08:01:08.759989+0000","last_active":"2024-09-19T08:01:26.721816+0000","last_peered":"2024-09-19T08:01:26.721816+0000","last_clean":"2024-09-19T08:01:26.721816+0000","last_became_active":"2024-09-19T08:01:08.759705+0000","last_became_peered":"2024-09-19T08:01:08.759705+0000","last_unstale":"2024-09-19T08:01:26.721816+0000","last_undegraded":"2024-09-19T08:01:26.721816+0000","last_fullsized":"2024-09-19T08:01:26.721816+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:33:25.237567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723490+0000","last_change":"2024-09-19T08:01:10.776528+0000","last_active":"2024-09-19T08:01:26.723490+0000","last_peered":"2024-09-19T08:01:26.723490+0000","last_clean":"2024-09-19T08:01:26.723490+0000","last_became_active":"2024-09-19T08:01:10.776222+0000","last_became_peered":"2024-09-19T08:01:10.776222+0000","last_unstale":"2024-09-19T08:01:26.723490+0000","last_undegraded":"2024-09-19T08:01:26.723490+0000","last_fullsized":"2024-09-19T08:01:26.723490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.787492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722186+0000","last_change":"2024-09-19T08:01:08.759631+0000","last_active":"2024-09-19T08:01:26.722186+0000","last_peered":"2024-09-19T08:01:26.722186+0000","last_clean":"2024-09-19T08:01:26.722186+0000","last_became_active":"2024-09-19T08:01:08.759156+0000","last_became_peered":"2024-09-19T08:01:08.759156+0000","last_unstale":"2024-09-19T08:01:26.722186+0000","last_undegraded":"2024-09-19T08:01:26.722186+0000","last_fullsized":"2024-09-19T08:01:26.722186+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:48:50.910824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722951+0000","last_change":"2024-09-19T08:01:10.772660+0000","last_active":"2024-09-19T08:01:26.722951+0000","last_peered":"2024-09-19T08:01:26.722951+0000","last_clean":"2024-09-19T08:01:26.722951+0000","last_became_active":"2024-09-19T08:01:10.771816+0000","last_became_peered":"2024-09-19T08:01:10.771816+0000","last_unstale":"2024-09-19T08:01:26.722951+0000","last_undegraded":"2024-09-19T08:01:26.722951+0000","last_fullsized":"2024-09-19T08:01:26.722951+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:04:32.667107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'6","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:30.722236+0000","last_change":"2024-09-19T08:01:09.749844+0000","last_active":"2024-09-19T08:01:30.722236+0000","last_peered":"2024-09-19T08:01:30.722236+0000","last_clean":"2024-09-19T08:01:30.722236+0000","last_became_active":"2024-09-19T08:01:09.749410+0000","last_became_peered":"2024-09-19T08:01:09.749410+0000","last_unstale":"2024-09-19T08:01:30.722236+0000","last_undegraded":"2024-09-19T08:01:30.722236+0000","last_fullsized":"2024-09-19T08:01:30.722236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:03:14.989814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1563,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723119+0000","last_change":"2024-09-19T08:01:10.773782+0000","last_active":"2024-09-19T08:01:26.723119+0000","last_peered":"2024-09-19T08:01:26.723119+0000","last_clean":"2024-09-19T08:01:26.723119+0000","last_became_active":"2024-09-19T08:01:10.773471+0000","last_became_peered":"2024-09-19T08:01:10.773471+0000","last_unstale":"2024-09-19T08:01:26.723119+0000","last_undegraded":"2024-09-19T08:01:26.723119+0000","last_fullsized":"2024-09-19T08:01:26.723119+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:15.780259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":341,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722698+0000","last_change":"2024-09-19T08:00:01.677517+0000","last_active":"2024-09-19T08:01:26.722698+0000","last_peered":"2024-09-19T08:01:26.722698+0000","last_clean":"2024-09-19T08:01:26.722698+0000","last_became_active":"2024-09-19T08:00:01.677089+0000","last_became_peered":"2024-09-19T08:00:01.677089+0000","last_unstale":"2024-09-19T08:01:26.722698+0000","last_undegraded":"2024-09-19T08:01:26.722698+0000","last_fullsized":"2024-09-19T08:01:26.722698+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_clean_scrub_stamp":"2024-09-19T07:56:43.962142+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:10:36.158811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758444+0000","last_change":"2024-09-19T08:01:08.759819+0000","last_active":"2024-09-19T08:01:26.758444+0000","last_peered":"2024-09-19T08:01:26.758444+0000","last_clean":"2024-09-19T08:01:26.758444+0000","last_became_active":"2024-09-19T08:01:08.759420+0000","last_became_peered":"2024-09-19T08:01:08.759420+0000","last_unstale":"2024-09-19T08:01:26.758444+0000","last_undegraded":"2024-09-19T08:01:26.758444+0000","last_fullsized":"2024-09-19T08:01:26.758444+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:06:16.852323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.724656+0000","last_change":"2024-09-19T08:01:26.724260+0000","last_active":"2024-09-19T08:01:26.724656+0000","last_peered":"2024-09-19T08:01:26.724656+0000","last_clean":"2024-09-19T08:01:26.724656+0000","last_became_active":"2024-09-19T08:01:26.723851+0000","last_became_peered":"2024-09-19T08:01:26.723851+0000","last_unstale":"2024-09-19T08:01:26.724656+0000","last_undegraded":"2024-09-19T08:01:26.724656+0000","last_fullsized":"2024-09-19T08:01:26.724656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:18:46.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722601+0000","last_change":"2024-09-19T08:01:08.761291+0000","last_active":"2024-09-19T08:01:26.722601+0000","last_peered":"2024-09-19T08:01:26.722601+0000","last_clean":"2024-09-19T08:01:26.722601+0000","last_became_active":"2024-09-19T08:01:08.760238+0000","last_became_peered":"2024-09-19T08:01:08.760238+0000","last_unstale":"2024-09-19T08:01:26.722601+0000","last_undegraded":"2024-09-19T08:01:26.722601+0000","last_fullsized":"2024-09-19T08:01:26.722601+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:26:09.980573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721415+0000","last_change":"2024-09-19T08:01:10.773056+0000","last_active":"2024-09-19T08:01:26.721415+0000","last_peered":"2024-09-19T08:01:26.721415+0000","last_clean":"2024-09-19T08:01:26.721415+0000","last_became_active":"2024-09-19T08:01:10.772431+0000","last_became_peered":"2024-09-19T08:01:10.772431+0000","last_unstale":"2024-09-19T08:01:26.721415+0000","last_undegraded":"2024-09-19T08:01:26.721415+0000","last_fullsized":"2024-09-19T08:01:26.721415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:39:06.878086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"78'5","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:28.703936+0000","last_change":"2024-09-19T08:01:09.749707+0000","last_active":"2024-09-19T08:01:28.703936+0000","last_peered":"2024-09-19T08:01:28.703936+0000","last_clean":"2024-09-19T08:01:28.703936+0000","last_became_active":"2024-09-19T08:01:09.749013+0000","last_became_peered":"2024-09-19T08:01:09.749013+0000","last_unstale":"2024-09-19T08:01:28.703936+0000","last_undegraded":"2024-09-19T08:01:28.703936+0000","last_fullsized":"2024-09-19T08:01:28.703936+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:40:12.682158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721922+0000","last_change":"2024-09-19T08:01:10.774101+0000","last_active":"2024-09-19T08:01:26.721922+0000","last_peered":"2024-09-19T08:01:26.721922+0000","last_clean":"2024-09-19T08:01:26.721922+0000","last_became_active":"2024-09-19T08:01:10.773822+0000","last_became_peered":"2024-09-19T08:01:10.773822+0000","last_unstale":"2024-09-19T08:01:26.721922+0000","last_undegraded":"2024-09-19T08:01:26.721922+0000","last_fullsized":"2024-09-19T08:01:26.721922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:29.571118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"78'5","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:27.725998+0000","last_change":"2024-09-19T08:01:08.761675+0000","last_active":"2024-09-19T08:01:27.725998+0000","last_peered":"2024-09-19T08:01:27.725998+0000","last_clean":"2024-09-19T08:01:27.725998+0000","last_became_active":"2024-09-19T08:01:08.761346+0000","last_became_peered":"2024-09-19T08:01:08.761346+0000","last_unstale":"2024-09-19T08:01:27.725998+0000","last_undegraded":"2024-09-19T08:01:27.725998+0000","last_fullsized":"2024-09-19T08:01:27.725998+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:51:19.237458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718382+0000","last_change":"2024-09-19T08:01:10.781565+0000","last_active":"2024-09-19T08:01:26.718382+0000","last_peered":"2024-09-19T08:01:26.718382+0000","last_clean":"2024-09-19T08:01:26.718382+0000","last_became_active":"2024-09-19T08:01:10.781301+0000","last_became_peered":"2024-09-19T08:01:10.781301+0000","last_unstale":"2024-09-19T08:01:26.718382+0000","last_undegraded":"2024-09-19T08:01:26.718382+0000","last_fullsized":"2024-09-19T08:01:26.718382+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:55:52.932009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719945+0000","last_change":"2024-09-19T08:01:09.748918+0000","last_active":"2024-09-19T08:01:26.719945+0000","last_peered":"2024-09-19T08:01:26.719945+0000","last_clean":"2024-09-19T08:01:26.719945+0000","last_became_active":"2024-09-19T08:01:09.748497+0000","last_became_peered":"2024-09-19T08:01:09.748497+0000","last_unstale":"2024-09-19T08:01:26.719945+0000","last_undegraded":"2024-09-19T08:01:26.719945+0000","last_fullsized":"2024-09-19T08:01:26.719945+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:20:31.030133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718202+0000","last_change":"2024-09-19T08:01:10.781388+0000","last_active":"2024-09-19T08:01:26.718202+0000","last_peered":"2024-09-19T08:01:26.718202+0000","last_clean":"2024-09-19T08:01:26.718202+0000","last_became_active":"2024-09-19T08:01:10.781063+0000","last_became_peered":"2024-09-19T08:01:10.781063+0000","last_unstale":"2024-09-19T08:01:26.718202+0000","last_undegraded":"2024-09-19T08:01:26.718202+0000","last_fullsized":"2024-09-19T08:01:26.718202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:12:08.852717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723781+0000","last_change":"2024-09-19T08:01:09.750080+0000","last_active":"2024-09-19T08:01:26.723781+0000","last_peered":"2024-09-19T08:01:26.723781+0000","last_clean":"2024-09-19T08:01:26.723781+0000","last_became_active":"2024-09-19T08:01:09.749053+0000","last_became_peered":"2024-09-19T08:01:09.749053+0000","last_unstale":"2024-09-19T08:01:26.723781+0000","last_undegraded":"2024-09-19T08:01:26.723781+0000","last_fullsized":"2024-09-19T08:01:26.723781+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:15:45.292714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721665+0000","last_change":"2024-09-19T08:01:10.767070+0000","last_active":"2024-09-19T08:01:26.721665+0000","last_peered":"2024-09-19T08:01:26.721665+0000","last_clean":"2024-09-19T08:01:26.721665+0000","last_became_active":"2024-09-19T08:01:10.766760+0000","last_became_peered":"2024-09-19T08:01:10.766760+0000","last_unstale":"2024-09-19T08:01:26.721665+0000","last_undegraded":"2024-09-19T08:01:26.721665+0000","last_fullsized":"2024-09-19T08:01:26.721665+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:17.841135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'9","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.720138+0000","last_change":"2024-09-19T08:01:08.756143+0000","last_active":"2024-09-19T08:01:26.720138+0000","last_peered":"2024-09-19T08:01:26.720138+0000","last_clean":"2024-09-19T08:01:26.720138+0000","last_became_active":"2024-09-19T08:01:08.755777+0000","last_became_peered":"2024-09-19T08:01:08.755777+0000","last_unstale":"2024-09-19T08:01:26.720138+0000","last_undegraded":"2024-09-19T08:01:26.720138+0000","last_fullsized":"2024-09-19T08:01:26.720138+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:21.430673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718462+0000","last_change":"2024-09-19T08:01:10.774140+0000","last_active":"2024-09-19T08:01:26.718462+0000","last_peered":"2024-09-19T08:01:26.718462+0000","last_clean":"2024-09-19T08:01:26.718462+0000","last_became_active":"2024-09-19T08:01:10.773819+0000","last_became_peered":"2024-09-19T08:01:10.773819+0000","last_unstale":"2024-09-19T08:01:26.718462+0000","last_undegraded":"2024-09-19T08:01:26.718462+0000","last_fullsized":"2024-09-19T08:01:26.718462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:15:13.174013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758589+0000","last_change":"2024-09-19T08:01:09.169129+0000","last_active":"2024-09-19T08:01:26.758589+0000","last_peered":"2024-09-19T08:01:26.758589+0000","last_clean":"2024-09-19T08:01:26.758589+0000","last_became_active":"2024-09-19T08:01:09.168734+0000","last_became_peered":"2024-09-19T08:01:09.168734+0000","last_unstale":"2024-09-19T08:01:26.758589+0000","last_undegraded":"2024-09-19T08:01:26.758589+0000","last_fullsized":"2024-09-19T08:01:26.758589+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:01:52.232391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757822+0000","last_change":"2024-09-19T08:01:10.768592+0000","last_active":"2024-09-19T08:01:26.757822+0000","last_peered":"2024-09-19T08:01:26.757822+0000","last_clean":"2024-09-19T08:01:26.757822+0000","last_became_active":"2024-09-19T08:01:10.767645+0000","last_became_peered":"2024-09-19T08:01:10.767645+0000","last_unstale":"2024-09-19T08:01:26.757822+0000","last_undegraded":"2024-09-19T08:01:26.757822+0000","last_fullsized":"2024-09-19T08:01:26.757822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:08:46.227461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721037+0000","last_change":"2024-09-19T08:01:09.749240+0000","last_active":"2024-09-19T08:01:26.721037+0000","last_peered":"2024-09-19T08:01:26.721037+0000","last_clean":"2024-09-19T08:01:26.721037+0000","last_became_active":"2024-09-19T08:01:09.748945+0000","last_became_peered":"2024-09-19T08:01:09.748945+0000","last_unstale":"2024-09-19T08:01:26.721037+0000","last_undegraded":"2024-09-19T08:01:26.721037+0000","last_fullsized":"2024-09-19T08:01:26.721037+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:25:13.791290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.720802+0000","last_change":"2024-09-19T08:01:10.783133+0000","last_active":"2024-09-19T08:01:26.720802+0000","last_peered":"2024-09-19T08:01:26.720802+0000","last_clean":"2024-09-19T08:01:26.720802+0000","last_became_active":"2024-09-19T08:01:10.782797+0000","last_became_peered":"2024-09-19T08:01:10.782797+0000","last_unstale":"2024-09-19T08:01:26.720802+0000","last_undegraded":"2024-09-19T08:01:26.720802+0000","last_fullsized":"2024-09-19T08:01:26.720802+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:30:43.551877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718123+0000","last_change":"2024-09-19T08:01:08.757964+0000","last_active":"2024-09-19T08:01:26.718123+0000","last_peered":"2024-09-19T08:01:26.718123+0000","last_clean":"2024-09-19T08:01:26.718123+0000","last_became_active":"2024-09-19T08:01:08.757526+0000","last_became_peered":"2024-09-19T08:01:08.757526+0000","last_unstale":"2024-09-19T08:01:26.718123+0000","last_undegraded":"2024-09-19T08:01:26.718123+0000","last_fullsized":"2024-09-19T08:01:26.718123+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:03.565702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718652+0000","last_change":"2024-09-19T08:01:10.769974+0000","last_active":"2024-09-19T08:01:26.718652+0000","last_peered":"2024-09-19T08:01:26.718652+0000","last_clean":"2024-09-19T08:01:26.718652+0000","last_became_active":"2024-09-19T08:01:10.769650+0000","last_became_peered":"2024-09-19T08:01:10.769650+0000","last_unstale":"2024-09-19T08:01:26.718652+0000","last_undegraded":"2024-09-19T08:01:26.718652+0000","last_fullsized":"2024-09-19T08:01:26.718652+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:01.755198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725382+0000","last_change":"2024-09-19T08:01:08.763476+0000","last_active":"2024-09-19T08:01:26.725382+0000","last_peered":"2024-09-19T08:01:26.725382+0000","last_clean":"2024-09-19T08:01:26.725382+0000","last_became_active":"2024-09-19T08:01:08.763154+0000","last_became_peered":"2024-09-19T08:01:08.763154+0000","last_unstale":"2024-09-19T08:01:26.725382+0000","last_undegraded":"2024-09-19T08:01:26.725382+0000","last_fullsized":"2024-09-19T08:01:26.725382+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:06.202452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718164+0000","last_change":"2024-09-19T08:01:10.784201+0000","last_active":"2024-09-19T08:01:26.718164+0000","last_peered":"2024-09-19T08:01:26.718164+0000","last_clean":"2024-09-19T08:01:26.718164+0000","last_became_active":"2024-09-19T08:01:10.783867+0000","last_became_peered":"2024-09-19T08:01:10.783867+0000","last_unstale":"2024-09-19T08:01:26.718164+0000","last_undegraded":"2024-09-19T08:01:26.718164+0000","last_fullsized":"2024-09-19T08:01:26.718164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:43:50.082183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721581+0000","last_change":"2024-09-19T08:01:08.760479+0000","last_active":"2024-09-19T08:01:26.721581+0000","last_peered":"2024-09-19T08:01:26.721581+0000","last_clean":"2024-09-19T08:01:26.721581+0000","last_became_active":"2024-09-19T08:01:08.760004+0000","last_became_peered":"2024-09-19T08:01:08.760004+0000","last_unstale":"2024-09-19T08:01:26.721581+0000","last_undegraded":"2024-09-19T08:01:26.721581+0000","last_fullsized":"2024-09-19T08:01:26.721581+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:41:19.956623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722763+0000","last_change":"2024-09-19T08:01:10.772184+0000","last_active":"2024-09-19T08:01:26.722763+0000","last_peered":"2024-09-19T08:01:26.722763+0000","last_clean":"2024-09-19T08:01:26.722763+0000","last_became_active":"2024-09-19T08:01:10.771825+0000","last_became_peered":"2024-09-19T08:01:10.771825+0000","last_unstale":"2024-09-19T08:01:26.722763+0000","last_undegraded":"2024-09-19T08:01:26.722763+0000","last_fullsized":"2024-09-19T08:01:26.722763+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:36.764575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721944+0000","last_change":"2024-09-19T08:01:10.779368+0000","last_active":"2024-09-19T08:01:26.721944+0000","last_peered":"2024-09-19T08:01:26.721944+0000","last_clean":"2024-09-19T08:01:26.721944+0000","last_became_active":"2024-09-19T08:01:10.778884+0000","last_became_peered":"2024-09-19T08:01:10.778884+0000","last_unstale":"2024-09-19T08:01:26.721944+0000","last_undegraded":"2024-09-19T08:01:26.721944+0000","last_fullsized":"2024-09-19T08:01:26.721944+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:28:29.630300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.741496+0000","last_change":"2024-09-19T08:01:26.741053+0000","last_active":"2024-09-19T08:01:26.741496+0000","last_peered":"2024-09-19T08:01:26.741496+0000","last_clean":"2024-09-19T08:01:26.741496+0000","last_became_active":"2024-09-19T08:01:26.724714+0000","last_became_peered":"2024-09-19T08:01:26.724714+0000","last_unstale":"2024-09-19T08:01:26.741496+0000","last_undegraded":"2024-09-19T08:01:26.741496+0000","last_fullsized":"2024-09-19T08:01:26.741496+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:59.583797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721113+0000","last_change":"2024-09-19T08:01:10.781389+0000","last_active":"2024-09-19T08:01:26.721113+0000","last_peered":"2024-09-19T08:01:26.721113+0000","last_clean":"2024-09-19T08:01:26.721113+0000","last_became_active":"2024-09-19T08:01:10.781080+0000","last_became_peered":"2024-09-19T08:01:10.781080+0000","last_unstale":"2024-09-19T08:01:26.721113+0000","last_undegraded":"2024-09-19T08:01:26.721113+0000","last_fullsized":"2024-09-19T08:01:26.721113+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:40:45.225957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.768102+0000","last_change":"2024-09-19T08:01:26.768102+0000","last_active":"2024-09-19T08:01:26.768102+0000","last_peered":"2024-09-19T08:01:26.768102+0000","last_clean":"2024-09-19T08:01:26.768102+0000","last_became_active":"2024-09-19T08:01:26.759462+0000","last_became_peered":"2024-09-19T08:01:26.759462+0000","last_unstale":"2024-09-19T08:01:26.768102+0000","last_undegraded":"2024-09-19T08:01:26.768102+0000","last_fullsized":"2024-09-19T08:01:26.768102+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:02:06.815238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757966+0000","last_change":"2024-09-19T08:01:10.771031+0000","last_active":"2024-09-19T08:01:26.757966+0000","last_peered":"2024-09-19T08:01:26.757966+0000","last_clean":"2024-09-19T08:01:26.757966+0000","last_became_active":"2024-09-19T08:01:10.770675+0000","last_became_peered":"2024-09-19T08:01:10.770675+0000","last_unstale":"2024-09-19T08:01:26.757966+0000","last_undegraded":"2024-09-19T08:01:26.757966+0000","last_fullsized":"2024-09-19T08:01:26.757966+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:42.203848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718482+0000","last_change":"2024-09-19T08:01:09.748622+0000","last_active":"2024-09-19T08:01:26.718482+0000","last_peered":"2024-09-19T08:01:26.718482+0000","last_clean":"2024-09-19T08:01:26.718482+0000","last_became_active":"2024-09-19T08:01:09.748202+0000","last_became_peered":"2024-09-19T08:01:09.748202+0000","last_unstale":"2024-09-19T08:01:26.718482+0000","last_undegraded":"2024-09-19T08:01:26.718482+0000","last_fullsized":"2024-09-19T08:01:26.718482+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:17:39.861820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718292+0000","last_change":"2024-09-19T08:01:10.768949+0000","last_active":"2024-09-19T08:01:26.718292+0000","last_peered":"2024-09-19T08:01:26.718292+0000","last_clean":"2024-09-19T08:01:26.718292+0000","last_became_active":"2024-09-19T08:01:10.768159+0000","last_became_peered":"2024-09-19T08:01:10.768159+0000","last_unstale":"2024-09-19T08:01:26.718292+0000","last_undegraded":"2024-09-19T08:01:26.718292+0000","last_fullsized":"2024-09-19T08:01:26.718292+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:43.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"78'13","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.736383+0000","last_change":"2024-09-19T08:01:26.735877+0000","last_active":"2024-09-19T08:01:26.736383+0000","last_peered":"2024-09-19T08:01:26.736383+0000","last_clean":"2024-09-19T08:01:26.736383+0000","last_became_active":"2024-09-19T08:01:26.723597+0000","last_became_peered":"2024-09-19T08:01:26.723597+0000","last_unstale":"2024-09-19T08:01:26.736383+0000","last_undegraded":"2024-09-19T08:01:26.736383+0000","last_fullsized":"2024-09-19T08:01:26.736383+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:43:42.340832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,1],"acting":[9,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":29540,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":119,"num_read_kb":91,"num_write":218,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1254,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":479232,"data_stored":113090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190,"ondisk_log_size":190,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645329,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":888832,"data_stored":427316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874261,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32696,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712456,"statfs":{"total":95995035648,"available":95961554944,"internally_reserved":0,"allocated":1540096,"data_stored":1039030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005082,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":901120,"data_stored":425599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234015,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":884736,"data_stored":424723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462947,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1486848,"data_stored":1017825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593769,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":884736,"data_stored":425316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822702,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28500,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716652,"statfs":{"total":95995035648,"available":95965851648,"internally_reserved":0,"allocated":1507328,"data_stored":1018175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986227,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":884736,"data_stored":425177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215159,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":901120,"data_stored":425141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476797,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":474037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607617,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":917504,"data_stored":449233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":905216,"data_stored":427048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":24756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":49560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":24068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T08:01:36.310 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-19T08:01:36.555 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:36.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:36 smithi162 ceph-mon[35260]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:01:36.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:36 smithi162 ceph-mon[35260]: pgmap v285: 97 pgs: 97 active+clean; 630 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.9 KiB/s wr, 15 op/s; 127 B/s, 1 keys/s, 1 objects/s recovering 2024-09-19T08:01:36.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:36 smithi139 ceph-mon[34874]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:01:36.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:36 smithi139 ceph-mon[34874]: pgmap v285: 97 pgs: 97 active+clean; 630 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.9 KiB/s wr, 15 op/s; 127 B/s, 1 keys/s, 1 objects/s recovering 2024-09-19T08:01:36.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:36 smithi098 ceph-mon[31333]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:01:36.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:36 smithi098 ceph-mon[31333]: pgmap v285: 97 pgs: 97 active+clean; 630 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.9 KiB/s wr, 15 op/s; 127 B/s, 1 keys/s, 1 objects/s recovering 2024-09-19T08:01:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.060+0000 7f3ff89e9640 1 -- 172.21.15.98:0/1642027235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ff410c170 msgr2=0x7f3ff410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.060+0000 7f3ff89e9640 1 --2- 172.21.15.98:0/1642027235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ff410c170 0x7f3ff410e610 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3fec009ca0 tx=0x7f3fec02a950 comp rx=0 tx=0).stop 2024-09-19T08:01:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.060+0000 7f3ff89e9640 1 -- 172.21.15.98:0/1642027235 shutdown_connections 2024-09-19T08:01:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.060+0000 7f3ff89e9640 1 --2- 172.21.15.98:0/1642027235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ff410c170 0x7f3ff410e610 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.060+0000 7f3ff89e9640 1 --2- 172.21.15.98:0/1642027235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ff4103660 0x7f3ff410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.060+0000 7f3ff89e9640 1 --2- 172.21.15.98:0/1642027235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff4102d40 0x7f3ff4103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:37.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.060+0000 7f3ff89e9640 1 -- 172.21.15.98:0/1642027235 >> 172.21.15.98:0/1642027235 conn(0x7f3ff40fc7a0 msgr2=0x7f3ff40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:37.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.060+0000 7f3ff89e9640 1 -- 172.21.15.98:0/1642027235 shutdown_connections 2024-09-19T08:01:37.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.060+0000 7f3ff89e9640 1 -- 172.21.15.98:0/1642027235 wait complete. 2024-09-19T08:01:37.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.061+0000 7f3ff89e9640 1 Processor -- start 2024-09-19T08:01:37.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.061+0000 7f3ff89e9640 1 -- start start 2024-09-19T08:01:37.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.061+0000 7f3ff89e9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ff4102d40 0x7f3ff41984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:37.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.061+0000 7f3ff89e9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff4103660 0x7f3ff41989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:37.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.061+0000 7f3ff89e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ff410c170 0x7f3ff419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:37.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.061+0000 7f3ff89e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff4111fb0 con 0x7f3ff410c170 2024-09-19T08:01:37.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.061+0000 7f3ff89e9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff4111e30 con 0x7f3ff4102d40 2024-09-19T08:01:37.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.061+0000 7f3ff89e9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff4112130 con 0x7f3ff4103660 2024-09-19T08:01:37.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.061+0000 7f3ff2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ff410c170 0x7f3ff419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:37.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.061+0000 7f3ff2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ff410c170 0x7f3ff419cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50826/0 (socket says 172.21.15.98:50826) 2024-09-19T08:01:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.061+0000 7f3ff2d9d640 1 -- 172.21.15.98:0/3577616039 learned_addr learned my addr 172.21.15.98:0/3577616039 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.062+0000 7f3ff259c640 1 --2- 172.21.15.98:0/3577616039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ff4102d40 0x7f3ff41984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.062+0000 7f3ff1d9b640 1 --2- 172.21.15.98:0/3577616039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff4103660 0x7f3ff41989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.062+0000 7f3ff2d9d640 1 -- 172.21.15.98:0/3577616039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff4103660 msgr2=0x7f3ff41989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.062+0000 7f3ff2d9d640 1 --2- 172.21.15.98:0/3577616039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff4103660 0x7f3ff41989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.062+0000 7f3ff2d9d640 1 -- 172.21.15.98:0/3577616039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ff4102d40 msgr2=0x7f3ff41984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.062+0000 7f3ff2d9d640 1 --2- 172.21.15.98:0/3577616039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ff4102d40 0x7f3ff41984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.062+0000 7f3ff2d9d640 1 -- 172.21.15.98:0/3577616039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ff419d460 con 0x7f3ff410c170 2024-09-19T08:01:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.062+0000 7f3ff2d9d640 1 --2- 172.21.15.98:0/3577616039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ff410c170 0x7f3ff419cd80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3fec02ae70 tx=0x7f3fec02f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.062+0000 7f3feb7fe640 1 -- 172.21.15.98:0/3577616039 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec034040 con 0x7f3ff410c170 2024-09-19T08:01:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.062+0000 7f3ff89e9640 1 -- 172.21.15.98:0/3577616039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ff419d6f0 con 0x7f3ff410c170 2024-09-19T08:01:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.062+0000 7f3feb7fe640 1 -- 172.21.15.98:0/3577616039 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fec030920 con 0x7f3ff410c170 2024-09-19T08:01:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.062+0000 7f3feb7fe640 1 -- 172.21.15.98:0/3577616039 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec0047b0 con 0x7f3ff410c170 2024-09-19T08:01:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.062+0000 7f3ff89e9640 1 -- 172.21.15.98:0/3577616039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ff41a5030 con 0x7f3ff410c170 2024-09-19T08:01:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.063+0000 7f3ff89e9640 1 -- 172.21.15.98:0/3577616039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ff4103120 con 0x7f3ff410c170 2024-09-19T08:01:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.064+0000 7f3feb7fe640 1 -- 172.21.15.98:0/3577616039 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f3fec037020 con 0x7f3ff410c170 2024-09-19T08:01:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.065+0000 7f3feb7fe640 1 --2- 172.21.15.98:0/3577616039 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f3fe00a32d0 0x7f3fe00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.065+0000 7f3ff259c640 1 --2- 172.21.15.98:0/3577616039 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f3fe00a32d0 0x7f3fe00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.065+0000 7f3feb7fe640 1 -- 172.21.15.98:0/3577616039 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f3fec0f0800 con 0x7f3ff410c170 2024-09-19T08:01:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.066+0000 7f3ff259c640 1 --2- 172.21.15.98:0/3577616039 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f3fe00a32d0 0x7f3fe00a5790 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009990 tx=0x7f3fdc009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.069+0000 7f3feb7fe640 1 -- 172.21.15.98:0/3577616039 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f3fec0bd4e0 con 0x7f3ff410c170 2024-09-19T08:01:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.235+0000 7f3ff89e9640 1 -- 172.21.15.98:0/3577616039 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3ff4062630 con 0x7f3fe00a32d0 2024-09-19T08:01:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.241+0000 7f3feb7fe640 1 -- 172.21.15.98:0/3577616039 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258759 (secure 0 0 0) 0x7f3ff4062630 con 0x7f3fe00a32d0 2024-09-19T08:01:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:01:37.244 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-19T08:01:37.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.244+0000 7f3ff89e9640 1 -- 172.21.15.98:0/3577616039 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f3fe00a32d0 msgr2=0x7f3fe00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:37.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.244+0000 7f3ff89e9640 1 --2- 172.21.15.98:0/3577616039 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f3fe00a32d0 0x7f3fe00a5790 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009990 tx=0x7f3fdc009110 comp rx=0 tx=0).stop 2024-09-19T08:01:37.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.244+0000 7f3ff89e9640 1 -- 172.21.15.98:0/3577616039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ff410c170 msgr2=0x7f3ff419cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.244+0000 7f3ff89e9640 1 --2- 172.21.15.98:0/3577616039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ff410c170 0x7f3ff419cd80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3fec02ae70 tx=0x7f3fec02f5b0 comp rx=0 tx=0).stop 2024-09-19T08:01:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.244+0000 7f3ff89e9640 1 -- 172.21.15.98:0/3577616039 shutdown_connections 2024-09-19T08:01:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.244+0000 7f3ff89e9640 1 --2- 172.21.15.98:0/3577616039 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f3fe00a32d0 0x7f3fe00a5790 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.244+0000 7f3ff89e9640 1 --2- 172.21.15.98:0/3577616039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ff410c170 0x7f3ff419cd80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.244+0000 7f3ff89e9640 1 --2- 172.21.15.98:0/3577616039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff4103660 0x7f3ff41989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.244+0000 7f3ff89e9640 1 --2- 172.21.15.98:0/3577616039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ff4102d40 0x7f3ff41984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.244+0000 7f3ff89e9640 1 -- 172.21.15.98:0/3577616039 >> 172.21.15.98:0/3577616039 conn(0x7f3ff40fc7a0 msgr2=0x7f3ff40fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.244+0000 7f3ff89e9640 1 -- 172.21.15.98:0/3577616039 shutdown_connections 2024-09-19T08:01:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:37.244+0000 7f3ff89e9640 1 -- 172.21.15.98:0/3577616039 wait complete. 2024-09-19T08:01:37.936 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-19T08:01:35.555370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":644612,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":292,"num_read_kb":260,"num_write":414,"num_write_kb":3057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181990,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":339,"ondisk_log_size":339,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370008,"kb_used_data":12408,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571816,"statfs":{"total":1151940427776,"available":1151561539584,"internally_reserved":0,"allocated":12705792,"data_stored":7027560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46244,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":114,"num_read_kb":106,"num_write":34,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1254,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.839737"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719055+0000","last_change":"2024-09-19T08:01:10.768950+0000","last_active":"2024-09-19T08:01:26.719055+0000","last_peered":"2024-09-19T08:01:26.719055+0000","last_clean":"2024-09-19T08:01:26.719055+0000","last_became_active":"2024-09-19T08:01:10.768047+0000","last_became_peered":"2024-09-19T08:01:10.768047+0000","last_unstale":"2024-09-19T08:01:26.719055+0000","last_undegraded":"2024-09-19T08:01:26.719055+0000","last_fullsized":"2024-09-19T08:01:26.719055+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:26.219079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757858+0000","last_change":"2024-09-19T08:01:10.783208+0000","last_active":"2024-09-19T08:01:26.757858+0000","last_peered":"2024-09-19T08:01:26.757858+0000","last_clean":"2024-09-19T08:01:26.757858+0000","last_became_active":"2024-09-19T08:01:10.782888+0000","last_became_peered":"2024-09-19T08:01:10.782888+0000","last_unstale":"2024-09-19T08:01:26.757858+0000","last_undegraded":"2024-09-19T08:01:26.757858+0000","last_fullsized":"2024-09-19T08:01:26.757858+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:51:27.603166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757759+0000","last_change":"2024-09-19T08:01:10.770395+0000","last_active":"2024-09-19T08:01:26.757759+0000","last_peered":"2024-09-19T08:01:26.757759+0000","last_clean":"2024-09-19T08:01:26.757759+0000","last_became_active":"2024-09-19T08:01:10.770073+0000","last_became_peered":"2024-09-19T08:01:10.770073+0000","last_unstale":"2024-09-19T08:01:26.757759+0000","last_undegraded":"2024-09-19T08:01:26.757759+0000","last_fullsized":"2024-09-19T08:01:26.757759+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:20.047023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758411+0000","last_change":"2024-09-19T08:01:10.777138+0000","last_active":"2024-09-19T08:01:26.758411+0000","last_peered":"2024-09-19T08:01:26.758411+0000","last_clean":"2024-09-19T08:01:26.758411+0000","last_became_active":"2024-09-19T08:01:10.776810+0000","last_became_peered":"2024-09-19T08:01:10.776810+0000","last_unstale":"2024-09-19T08:01:26.758411+0000","last_undegraded":"2024-09-19T08:01:26.758411+0000","last_fullsized":"2024-09-19T08:01:26.758411+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:39.983321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721379+0000","last_change":"2024-09-19T08:01:10.770435+0000","last_active":"2024-09-19T08:01:26.721379+0000","last_peered":"2024-09-19T08:01:26.721379+0000","last_clean":"2024-09-19T08:01:26.721379+0000","last_became_active":"2024-09-19T08:01:10.769686+0000","last_became_peered":"2024-09-19T08:01:10.769686+0000","last_unstale":"2024-09-19T08:01:26.721379+0000","last_undegraded":"2024-09-19T08:01:26.721379+0000","last_fullsized":"2024-09-19T08:01:26.721379+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:10:19.610925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718127+0000","last_change":"2024-09-19T08:01:10.782382+0000","last_active":"2024-09-19T08:01:26.718127+0000","last_peered":"2024-09-19T08:01:26.718127+0000","last_clean":"2024-09-19T08:01:26.718127+0000","last_became_active":"2024-09-19T08:01:10.781996+0000","last_became_peered":"2024-09-19T08:01:10.781996+0000","last_unstale":"2024-09-19T08:01:26.718127+0000","last_undegraded":"2024-09-19T08:01:26.718127+0000","last_fullsized":"2024-09-19T08:01:26.718127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:43:13.574336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722516+0000","last_change":"2024-09-19T08:01:10.773776+0000","last_active":"2024-09-19T08:01:26.722516+0000","last_peered":"2024-09-19T08:01:26.722516+0000","last_clean":"2024-09-19T08:01:26.722516+0000","last_became_active":"2024-09-19T08:01:10.772442+0000","last_became_peered":"2024-09-19T08:01:10.772442+0000","last_unstale":"2024-09-19T08:01:26.722516+0000","last_undegraded":"2024-09-19T08:01:26.722516+0000","last_fullsized":"2024-09-19T08:01:26.722516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:08:18.155806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.717962+0000","last_change":"2024-09-19T08:01:10.781443+0000","last_active":"2024-09-19T08:01:26.717962+0000","last_peered":"2024-09-19T08:01:26.717962+0000","last_clean":"2024-09-19T08:01:26.717962+0000","last_became_active":"2024-09-19T08:01:10.781083+0000","last_became_peered":"2024-09-19T08:01:10.781083+0000","last_unstale":"2024-09-19T08:01:26.717962+0000","last_undegraded":"2024-09-19T08:01:26.717962+0000","last_fullsized":"2024-09-19T08:01:26.717962+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:23:22.631865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721840+0000","last_change":"2024-09-19T08:01:10.767554+0000","last_active":"2024-09-19T08:01:26.721840+0000","last_peered":"2024-09-19T08:01:26.721840+0000","last_clean":"2024-09-19T08:01:26.721840+0000","last_became_active":"2024-09-19T08:01:10.767227+0000","last_became_peered":"2024-09-19T08:01:10.767227+0000","last_unstale":"2024-09-19T08:01:26.721840+0000","last_undegraded":"2024-09-19T08:01:26.721840+0000","last_fullsized":"2024-09-19T08:01:26.721840+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:30:13.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718514+0000","last_change":"2024-09-19T08:01:10.783174+0000","last_active":"2024-09-19T08:01:26.718514+0000","last_peered":"2024-09-19T08:01:26.718514+0000","last_clean":"2024-09-19T08:01:26.718514+0000","last_became_active":"2024-09-19T08:01:10.782794+0000","last_became_peered":"2024-09-19T08:01:10.782794+0000","last_unstale":"2024-09-19T08:01:26.718514+0000","last_undegraded":"2024-09-19T08:01:26.718514+0000","last_fullsized":"2024-09-19T08:01:26.718514+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:49:19.451107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723075+0000","last_change":"2024-09-19T08:01:10.779638+0000","last_active":"2024-09-19T08:01:26.723075+0000","last_peered":"2024-09-19T08:01:26.723075+0000","last_clean":"2024-09-19T08:01:26.723075+0000","last_became_active":"2024-09-19T08:01:10.779295+0000","last_became_peered":"2024-09-19T08:01:10.779295+0000","last_unstale":"2024-09-19T08:01:26.723075+0000","last_undegraded":"2024-09-19T08:01:26.723075+0000","last_fullsized":"2024-09-19T08:01:26.723075+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:35:10.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758492+0000","last_change":"2024-09-19T08:01:10.784487+0000","last_active":"2024-09-19T08:01:26.758492+0000","last_peered":"2024-09-19T08:01:26.758492+0000","last_clean":"2024-09-19T08:01:26.758492+0000","last_became_active":"2024-09-19T08:01:10.784114+0000","last_became_peered":"2024-09-19T08:01:10.784114+0000","last_unstale":"2024-09-19T08:01:26.758492+0000","last_undegraded":"2024-09-19T08:01:26.758492+0000","last_fullsized":"2024-09-19T08:01:26.758492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:09:54.819802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721613+0000","last_change":"2024-09-19T08:01:10.783186+0000","last_active":"2024-09-19T08:01:26.721613+0000","last_peered":"2024-09-19T08:01:26.721613+0000","last_clean":"2024-09-19T08:01:26.721613+0000","last_became_active":"2024-09-19T08:01:10.782876+0000","last_became_peered":"2024-09-19T08:01:10.782876+0000","last_unstale":"2024-09-19T08:01:26.721613+0000","last_undegraded":"2024-09-19T08:01:26.721613+0000","last_fullsized":"2024-09-19T08:01:26.721613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:26:15.252165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757647+0000","last_change":"2024-09-19T08:01:10.783237+0000","last_active":"2024-09-19T08:01:26.757647+0000","last_peered":"2024-09-19T08:01:26.757647+0000","last_clean":"2024-09-19T08:01:26.757647+0000","last_became_active":"2024-09-19T08:01:10.782966+0000","last_became_peered":"2024-09-19T08:01:10.782966+0000","last_unstale":"2024-09-19T08:01:26.757647+0000","last_undegraded":"2024-09-19T08:01:26.757647+0000","last_fullsized":"2024-09-19T08:01:26.757647+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:57:18.771737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757535+0000","last_change":"2024-09-19T08:01:10.771207+0000","last_active":"2024-09-19T08:01:26.757535+0000","last_peered":"2024-09-19T08:01:26.757535+0000","last_clean":"2024-09-19T08:01:26.757535+0000","last_became_active":"2024-09-19T08:01:10.770862+0000","last_became_peered":"2024-09-19T08:01:10.770862+0000","last_unstale":"2024-09-19T08:01:26.757535+0000","last_undegraded":"2024-09-19T08:01:26.757535+0000","last_fullsized":"2024-09-19T08:01:26.757535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:08:09.313329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.720764+0000","last_change":"2024-09-19T08:01:10.775191+0000","last_active":"2024-09-19T08:01:26.720764+0000","last_peered":"2024-09-19T08:01:26.720764+0000","last_clean":"2024-09-19T08:01:26.720764+0000","last_became_active":"2024-09-19T08:01:10.774832+0000","last_became_peered":"2024-09-19T08:01:10.774832+0000","last_unstale":"2024-09-19T08:01:26.720764+0000","last_undegraded":"2024-09-19T08:01:26.720764+0000","last_fullsized":"2024-09-19T08:01:26.720764+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:57:29.800787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719076+0000","last_change":"2024-09-19T08:01:10.769115+0000","last_active":"2024-09-19T08:01:26.719076+0000","last_peered":"2024-09-19T08:01:26.719076+0000","last_clean":"2024-09-19T08:01:26.719076+0000","last_became_active":"2024-09-19T08:01:10.768088+0000","last_became_peered":"2024-09-19T08:01:10.768088+0000","last_unstale":"2024-09-19T08:01:26.719076+0000","last_undegraded":"2024-09-19T08:01:26.719076+0000","last_fullsized":"2024-09-19T08:01:26.719076+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:27:16.042722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719916+0000","last_change":"2024-09-19T08:01:10.781426+0000","last_active":"2024-09-19T08:01:26.719916+0000","last_peered":"2024-09-19T08:01:26.719916+0000","last_clean":"2024-09-19T08:01:26.719916+0000","last_became_active":"2024-09-19T08:01:10.781094+0000","last_became_peered":"2024-09-19T08:01:10.781094+0000","last_unstale":"2024-09-19T08:01:26.719916+0000","last_undegraded":"2024-09-19T08:01:26.719916+0000","last_fullsized":"2024-09-19T08:01:26.719916+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:43:31.336610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725515+0000","last_change":"2024-09-19T08:01:10.766466+0000","last_active":"2024-09-19T08:01:26.725515+0000","last_peered":"2024-09-19T08:01:26.725515+0000","last_clean":"2024-09-19T08:01:26.725515+0000","last_became_active":"2024-09-19T08:01:10.766123+0000","last_became_peered":"2024-09-19T08:01:10.766123+0000","last_unstale":"2024-09-19T08:01:26.725515+0000","last_undegraded":"2024-09-19T08:01:26.725515+0000","last_fullsized":"2024-09-19T08:01:26.725515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:25.848603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718613+0000","last_change":"2024-09-19T08:01:10.772782+0000","last_active":"2024-09-19T08:01:26.718613+0000","last_peered":"2024-09-19T08:01:26.718613+0000","last_clean":"2024-09-19T08:01:26.718613+0000","last_became_active":"2024-09-19T08:01:10.772445+0000","last_became_peered":"2024-09-19T08:01:10.772445+0000","last_unstale":"2024-09-19T08:01:26.718613+0000","last_undegraded":"2024-09-19T08:01:26.718613+0000","last_fullsized":"2024-09-19T08:01:26.718613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:09.174005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721701+0000","last_change":"2024-09-19T08:01:10.771944+0000","last_active":"2024-09-19T08:01:26.721701+0000","last_peered":"2024-09-19T08:01:26.721701+0000","last_clean":"2024-09-19T08:01:26.721701+0000","last_became_active":"2024-09-19T08:01:10.771643+0000","last_became_peered":"2024-09-19T08:01:10.771643+0000","last_unstale":"2024-09-19T08:01:26.721701+0000","last_undegraded":"2024-09-19T08:01:26.721701+0000","last_fullsized":"2024-09-19T08:01:26.721701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:46:27.121722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721201+0000","last_change":"2024-09-19T08:01:10.775144+0000","last_active":"2024-09-19T08:01:26.721201+0000","last_peered":"2024-09-19T08:01:26.721201+0000","last_clean":"2024-09-19T08:01:26.721201+0000","last_became_active":"2024-09-19T08:01:10.774769+0000","last_became_peered":"2024-09-19T08:01:10.774769+0000","last_unstale":"2024-09-19T08:01:26.721201+0000","last_undegraded":"2024-09-19T08:01:26.721201+0000","last_fullsized":"2024-09-19T08:01:26.721201+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:59:47.053993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718014+0000","last_change":"2024-09-19T08:01:10.766750+0000","last_active":"2024-09-19T08:01:26.718014+0000","last_peered":"2024-09-19T08:01:26.718014+0000","last_clean":"2024-09-19T08:01:26.718014+0000","last_became_active":"2024-09-19T08:01:10.766450+0000","last_became_peered":"2024-09-19T08:01:10.766450+0000","last_unstale":"2024-09-19T08:01:26.718014+0000","last_undegraded":"2024-09-19T08:01:26.718014+0000","last_fullsized":"2024-09-19T08:01:26.718014+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:47:58.133815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721798+0000","last_change":"2024-09-19T08:01:10.768430+0000","last_active":"2024-09-19T08:01:26.721798+0000","last_peered":"2024-09-19T08:01:26.721798+0000","last_clean":"2024-09-19T08:01:26.721798+0000","last_became_active":"2024-09-19T08:01:10.768092+0000","last_became_peered":"2024-09-19T08:01:10.768092+0000","last_unstale":"2024-09-19T08:01:26.721798+0000","last_undegraded":"2024-09-19T08:01:26.721798+0000","last_fullsized":"2024-09-19T08:01:26.721798+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:16:42.343138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718376+0000","last_change":"2024-09-19T08:01:10.771131+0000","last_active":"2024-09-19T08:01:26.718376+0000","last_peered":"2024-09-19T08:01:26.718376+0000","last_clean":"2024-09-19T08:01:26.718376+0000","last_became_active":"2024-09-19T08:01:10.770851+0000","last_became_peered":"2024-09-19T08:01:10.770851+0000","last_unstale":"2024-09-19T08:01:26.718376+0000","last_undegraded":"2024-09-19T08:01:26.718376+0000","last_fullsized":"2024-09-19T08:01:26.718376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:07:15.830476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.726129+0000","last_change":"2024-09-19T08:01:10.776220+0000","last_active":"2024-09-19T08:01:26.726129+0000","last_peered":"2024-09-19T08:01:26.726129+0000","last_clean":"2024-09-19T08:01:26.726129+0000","last_became_active":"2024-09-19T08:01:10.775889+0000","last_became_peered":"2024-09-19T08:01:10.775889+0000","last_unstale":"2024-09-19T08:01:26.726129+0000","last_undegraded":"2024-09-19T08:01:26.726129+0000","last_fullsized":"2024-09-19T08:01:26.726129+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:23.714675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719003+0000","last_change":"2024-09-19T08:01:10.776370+0000","last_active":"2024-09-19T08:01:26.719003+0000","last_peered":"2024-09-19T08:01:26.719003+0000","last_clean":"2024-09-19T08:01:26.719003+0000","last_became_active":"2024-09-19T08:01:10.776027+0000","last_became_peered":"2024-09-19T08:01:10.776027+0000","last_unstale":"2024-09-19T08:01:26.719003+0000","last_undegraded":"2024-09-19T08:01:26.719003+0000","last_fullsized":"2024-09-19T08:01:26.719003+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:48:46.293404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758971+0000","last_change":"2024-09-19T08:01:26.758971+0000","last_active":"2024-09-19T08:01:26.758971+0000","last_peered":"2024-09-19T08:01:26.758971+0000","last_clean":"2024-09-19T08:01:26.758971+0000","last_became_active":"2024-09-19T08:01:26.758524+0000","last_became_peered":"2024-09-19T08:01:26.758524+0000","last_unstale":"2024-09-19T08:01:26.758971+0000","last_undegraded":"2024-09-19T08:01:26.758971+0000","last_fullsized":"2024-09-19T08:01:26.758971+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:34:22.439327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757760+0000","last_change":"2024-09-19T08:01:10.767445+0000","last_active":"2024-09-19T08:01:26.757760+0000","last_peered":"2024-09-19T08:01:26.757760+0000","last_clean":"2024-09-19T08:01:26.757760+0000","last_became_active":"2024-09-19T08:01:10.767128+0000","last_became_peered":"2024-09-19T08:01:10.767128+0000","last_unstale":"2024-09-19T08:01:26.757760+0000","last_undegraded":"2024-09-19T08:01:26.757760+0000","last_fullsized":"2024-09-19T08:01:26.757760+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:10.273948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725212+0000","last_change":"2024-09-19T08:01:10.768954+0000","last_active":"2024-09-19T08:01:26.725212+0000","last_peered":"2024-09-19T08:01:26.725212+0000","last_clean":"2024-09-19T08:01:26.725212+0000","last_became_active":"2024-09-19T08:01:10.768577+0000","last_became_peered":"2024-09-19T08:01:10.768577+0000","last_unstale":"2024-09-19T08:01:26.725212+0000","last_undegraded":"2024-09-19T08:01:26.725212+0000","last_fullsized":"2024-09-19T08:01:26.725212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:20:21.812531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725473+0000","last_change":"2024-09-19T08:01:10.774104+0000","last_active":"2024-09-19T08:01:26.725473+0000","last_peered":"2024-09-19T08:01:26.725473+0000","last_clean":"2024-09-19T08:01:26.725473+0000","last_became_active":"2024-09-19T08:01:10.773721+0000","last_became_peered":"2024-09-19T08:01:10.773721+0000","last_unstale":"2024-09-19T08:01:26.725473+0000","last_undegraded":"2024-09-19T08:01:26.725473+0000","last_fullsized":"2024-09-19T08:01:26.725473+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:45:34.945994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.727296+0000","last_change":"2024-09-19T08:01:26.727296+0000","last_active":"2024-09-19T08:01:26.727296+0000","last_peered":"2024-09-19T08:01:26.727296+0000","last_clean":"2024-09-19T08:01:26.727296+0000","last_became_active":"2024-09-19T08:01:26.726631+0000","last_became_peered":"2024-09-19T08:01:26.726631+0000","last_unstale":"2024-09-19T08:01:26.727296+0000","last_undegraded":"2024-09-19T08:01:26.727296+0000","last_fullsized":"2024-09-19T08:01:26.727296+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:19:24.807050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721044+0000","last_change":"2024-09-19T08:01:10.765282+0000","last_active":"2024-09-19T08:01:26.721044+0000","last_peered":"2024-09-19T08:01:26.721044+0000","last_clean":"2024-09-19T08:01:26.721044+0000","last_became_active":"2024-09-19T08:01:10.764928+0000","last_became_peered":"2024-09-19T08:01:10.764928+0000","last_unstale":"2024-09-19T08:01:26.721044+0000","last_undegraded":"2024-09-19T08:01:26.721044+0000","last_fullsized":"2024-09-19T08:01:26.721044+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:00:37.395110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"78'6","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:29.701406+0000","last_change":"2024-09-19T08:01:08.760686+0000","last_active":"2024-09-19T08:01:29.701406+0000","last_peered":"2024-09-19T08:01:29.701406+0000","last_clean":"2024-09-19T08:01:29.701406+0000","last_became_active":"2024-09-19T08:01:08.759912+0000","last_became_peered":"2024-09-19T08:01:08.759912+0000","last_unstale":"2024-09-19T08:01:29.701406+0000","last_undegraded":"2024-09-19T08:01:29.701406+0000","last_fullsized":"2024-09-19T08:01:29.701406+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:46:01.879611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721868+0000","last_change":"2024-09-19T08:01:10.779668+0000","last_active":"2024-09-19T08:01:26.721868+0000","last_peered":"2024-09-19T08:01:26.721868+0000","last_clean":"2024-09-19T08:01:26.721868+0000","last_became_active":"2024-09-19T08:01:10.779265+0000","last_became_peered":"2024-09-19T08:01:10.779265+0000","last_unstale":"2024-09-19T08:01:26.721868+0000","last_undegraded":"2024-09-19T08:01:26.721868+0000","last_fullsized":"2024-09-19T08:01:26.721868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:06.177700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718183+0000","last_change":"2024-09-19T08:01:08.762668+0000","last_active":"2024-09-19T08:01:26.718183+0000","last_peered":"2024-09-19T08:01:26.718183+0000","last_clean":"2024-09-19T08:01:26.718183+0000","last_became_active":"2024-09-19T08:01:08.762277+0000","last_became_peered":"2024-09-19T08:01:08.762277+0000","last_unstale":"2024-09-19T08:01:26.718183+0000","last_undegraded":"2024-09-19T08:01:26.718183+0000","last_fullsized":"2024-09-19T08:01:26.718183+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:36:59.316655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721452+0000","last_change":"2024-09-19T08:01:08.757401+0000","last_active":"2024-09-19T08:01:26.721452+0000","last_peered":"2024-09-19T08:01:26.721452+0000","last_clean":"2024-09-19T08:01:26.721452+0000","last_became_active":"2024-09-19T08:01:08.757067+0000","last_became_peered":"2024-09-19T08:01:08.757067+0000","last_unstale":"2024-09-19T08:01:26.721452+0000","last_undegraded":"2024-09-19T08:01:26.721452+0000","last_fullsized":"2024-09-19T08:01:26.721452+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:19.617830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718027+0000","last_change":"2024-09-19T08:01:10.765149+0000","last_active":"2024-09-19T08:01:26.718027+0000","last_peered":"2024-09-19T08:01:26.718027+0000","last_clean":"2024-09-19T08:01:26.718027+0000","last_became_active":"2024-09-19T08:01:10.764840+0000","last_became_peered":"2024-09-19T08:01:10.764840+0000","last_unstale":"2024-09-19T08:01:26.718027+0000","last_undegraded":"2024-09-19T08:01:26.718027+0000","last_fullsized":"2024-09-19T08:01:26.718027+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:15.539307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722074+0000","last_change":"2024-09-19T08:01:08.759435+0000","last_active":"2024-09-19T08:01:26.722074+0000","last_peered":"2024-09-19T08:01:26.722074+0000","last_clean":"2024-09-19T08:01:26.722074+0000","last_became_active":"2024-09-19T08:01:08.759046+0000","last_became_peered":"2024-09-19T08:01:08.759046+0000","last_unstale":"2024-09-19T08:01:26.722074+0000","last_undegraded":"2024-09-19T08:01:26.722074+0000","last_fullsized":"2024-09-19T08:01:26.722074+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:21:24.077626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719748+0000","last_change":"2024-09-19T08:01:10.778432+0000","last_active":"2024-09-19T08:01:26.719748+0000","last_peered":"2024-09-19T08:01:26.719748+0000","last_clean":"2024-09-19T08:01:26.719748+0000","last_became_active":"2024-09-19T08:01:10.778098+0000","last_became_peered":"2024-09-19T08:01:10.778098+0000","last_unstale":"2024-09-19T08:01:26.719748+0000","last_undegraded":"2024-09-19T08:01:26.719748+0000","last_fullsized":"2024-09-19T08:01:26.719748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:40:12.921165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"78'4","reported_seq":50,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:32.705427+0000","last_change":"2024-09-19T08:01:08.764117+0000","last_active":"2024-09-19T08:01:32.705427+0000","last_peered":"2024-09-19T08:01:32.705427+0000","last_clean":"2024-09-19T08:01:32.705427+0000","last_became_active":"2024-09-19T08:01:08.763308+0000","last_became_peered":"2024-09-19T08:01:08.763308+0000","last_unstale":"2024-09-19T08:01:32.705427+0000","last_undegraded":"2024-09-19T08:01:32.705427+0000","last_fullsized":"2024-09-19T08:01:32.705427+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:19:57.014628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721336+0000","last_change":"2024-09-19T08:01:10.781771+0000","last_active":"2024-09-19T08:01:26.721336+0000","last_peered":"2024-09-19T08:01:26.721336+0000","last_clean":"2024-09-19T08:01:26.721336+0000","last_became_active":"2024-09-19T08:01:10.781432+0000","last_became_peered":"2024-09-19T08:01:10.781432+0000","last_unstale":"2024-09-19T08:01:26.721336+0000","last_undegraded":"2024-09-19T08:01:26.721336+0000","last_fullsized":"2024-09-19T08:01:26.721336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:25:15.067850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.771171+0000","last_change":"2024-09-19T08:01:26.771171+0000","last_active":"2024-09-19T08:01:26.771171+0000","last_peered":"2024-09-19T08:01:26.771171+0000","last_clean":"2024-09-19T08:01:26.771171+0000","last_became_active":"2024-09-19T08:01:26.759274+0000","last_became_peered":"2024-09-19T08:01:26.759274+0000","last_unstale":"2024-09-19T08:01:26.771171+0000","last_undegraded":"2024-09-19T08:01:26.771171+0000","last_fullsized":"2024-09-19T08:01:26.771171+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:20:54.603420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757967+0000","last_change":"2024-09-19T08:01:10.768318+0000","last_active":"2024-09-19T08:01:26.757967+0000","last_peered":"2024-09-19T08:01:26.757967+0000","last_clean":"2024-09-19T08:01:26.757967+0000","last_became_active":"2024-09-19T08:01:10.768022+0000","last_became_peered":"2024-09-19T08:01:10.768022+0000","last_unstale":"2024-09-19T08:01:26.757967+0000","last_undegraded":"2024-09-19T08:01:26.757967+0000","last_fullsized":"2024-09-19T08:01:26.757967+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:00:45.328638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722058+0000","last_change":"2024-09-19T08:01:09.750530+0000","last_active":"2024-09-19T08:01:26.722058+0000","last_peered":"2024-09-19T08:01:26.722058+0000","last_clean":"2024-09-19T08:01:26.722058+0000","last_became_active":"2024-09-19T08:01:09.750029+0000","last_became_peered":"2024-09-19T08:01:09.750029+0000","last_unstale":"2024-09-19T08:01:26.722058+0000","last_undegraded":"2024-09-19T08:01:26.722058+0000","last_fullsized":"2024-09-19T08:01:26.722058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:01:16.062992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.724636+0000","last_change":"2024-09-19T08:01:26.724636+0000","last_active":"2024-09-19T08:01:26.724636+0000","last_peered":"2024-09-19T08:01:26.724636+0000","last_clean":"2024-09-19T08:01:26.724636+0000","last_became_active":"2024-09-19T08:01:26.724171+0000","last_became_peered":"2024-09-19T08:01:26.724171+0000","last_unstale":"2024-09-19T08:01:26.724636+0000","last_undegraded":"2024-09-19T08:01:26.724636+0000","last_fullsized":"2024-09-19T08:01:26.724636+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:54:05.998374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'24","reported_seq":78,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:32.707260+0000","last_change":"2024-09-19T08:01:08.761950+0000","last_active":"2024-09-19T08:01:32.707260+0000","last_peered":"2024-09-19T08:01:32.707260+0000","last_clean":"2024-09-19T08:01:32.707260+0000","last_became_active":"2024-09-19T08:01:08.761533+0000","last_became_peered":"2024-09-19T08:01:08.761533+0000","last_unstale":"2024-09-19T08:01:32.707260+0000","last_undegraded":"2024-09-19T08:01:32.707260+0000","last_fullsized":"2024-09-19T08:01:32.707260+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:07:26.668141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46802,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":61,"num_write":26,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723579+0000","last_change":"2024-09-19T08:01:10.774280+0000","last_active":"2024-09-19T08:01:26.723579+0000","last_peered":"2024-09-19T08:01:26.723579+0000","last_clean":"2024-09-19T08:01:26.723579+0000","last_became_active":"2024-09-19T08:01:10.773982+0000","last_became_peered":"2024-09-19T08:01:10.773982+0000","last_unstale":"2024-09-19T08:01:26.723579+0000","last_undegraded":"2024-09-19T08:01:26.723579+0000","last_fullsized":"2024-09-19T08:01:26.723579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:21:43.180057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725363+0000","last_change":"2024-09-19T08:01:08.761067+0000","last_active":"2024-09-19T08:01:26.725363+0000","last_peered":"2024-09-19T08:01:26.725363+0000","last_clean":"2024-09-19T08:01:26.725363+0000","last_became_active":"2024-09-19T08:01:08.760664+0000","last_became_peered":"2024-09-19T08:01:08.760664+0000","last_unstale":"2024-09-19T08:01:26.725363+0000","last_undegraded":"2024-09-19T08:01:26.725363+0000","last_fullsized":"2024-09-19T08:01:26.725363+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:22:38.086891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721462+0000","last_change":"2024-09-19T08:01:10.771845+0000","last_active":"2024-09-19T08:01:26.721462+0000","last_peered":"2024-09-19T08:01:26.721462+0000","last_clean":"2024-09-19T08:01:26.721462+0000","last_became_active":"2024-09-19T08:01:10.771533+0000","last_became_peered":"2024-09-19T08:01:10.771533+0000","last_unstale":"2024-09-19T08:01:26.721462+0000","last_undegraded":"2024-09-19T08:01:26.721462+0000","last_fullsized":"2024-09-19T08:01:26.721462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:29:35.180763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722806+0000","last_change":"2024-09-19T08:01:08.763869+0000","last_active":"2024-09-19T08:01:26.722806+0000","last_peered":"2024-09-19T08:01:26.722806+0000","last_clean":"2024-09-19T08:01:26.722806+0000","last_became_active":"2024-09-19T08:01:08.763451+0000","last_became_peered":"2024-09-19T08:01:08.763451+0000","last_unstale":"2024-09-19T08:01:26.722806+0000","last_undegraded":"2024-09-19T08:01:26.722806+0000","last_fullsized":"2024-09-19T08:01:26.722806+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:32:40.901049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722991+0000","last_change":"2024-09-19T08:01:10.770823+0000","last_active":"2024-09-19T08:01:26.722991+0000","last_peered":"2024-09-19T08:01:26.722991+0000","last_clean":"2024-09-19T08:01:26.722991+0000","last_became_active":"2024-09-19T08:01:10.770462+0000","last_became_peered":"2024-09-19T08:01:10.770462+0000","last_unstale":"2024-09-19T08:01:26.722991+0000","last_undegraded":"2024-09-19T08:01:26.722991+0000","last_fullsized":"2024-09-19T08:01:26.722991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:04:32.832009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"78'5","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:29.705807+0000","last_change":"2024-09-19T08:01:08.759429+0000","last_active":"2024-09-19T08:01:29.705807+0000","last_peered":"2024-09-19T08:01:29.705807+0000","last_clean":"2024-09-19T08:01:29.705807+0000","last_became_active":"2024-09-19T08:01:08.759062+0000","last_became_peered":"2024-09-19T08:01:08.759062+0000","last_unstale":"2024-09-19T08:01:29.705807+0000","last_undegraded":"2024-09-19T08:01:29.705807+0000","last_fullsized":"2024-09-19T08:01:29.705807+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:16:51.582435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722418+0000","last_change":"2024-09-19T08:01:10.784735+0000","last_active":"2024-09-19T08:01:26.722418+0000","last_peered":"2024-09-19T08:01:26.722418+0000","last_clean":"2024-09-19T08:01:26.722418+0000","last_became_active":"2024-09-19T08:01:10.784346+0000","last_became_peered":"2024-09-19T08:01:10.784346+0000","last_unstale":"2024-09-19T08:01:26.722418+0000","last_undegraded":"2024-09-19T08:01:26.722418+0000","last_fullsized":"2024-09-19T08:01:26.722418+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:27:54.580391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721469+0000","last_change":"2024-09-19T08:01:09.749498+0000","last_active":"2024-09-19T08:01:26.721469+0000","last_peered":"2024-09-19T08:01:26.721469+0000","last_clean":"2024-09-19T08:01:26.721469+0000","last_became_active":"2024-09-19T08:01:09.749091+0000","last_became_peered":"2024-09-19T08:01:09.749091+0000","last_unstale":"2024-09-19T08:01:26.721469+0000","last_undegraded":"2024-09-19T08:01:26.721469+0000","last_fullsized":"2024-09-19T08:01:26.721469+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:56:13.644931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721781+0000","last_change":"2024-09-19T08:01:10.779599+0000","last_active":"2024-09-19T08:01:26.721781+0000","last_peered":"2024-09-19T08:01:26.721781+0000","last_clean":"2024-09-19T08:01:26.721781+0000","last_became_active":"2024-09-19T08:01:10.779219+0000","last_became_peered":"2024-09-19T08:01:10.779219+0000","last_unstale":"2024-09-19T08:01:26.721781+0000","last_undegraded":"2024-09-19T08:01:26.721781+0000","last_fullsized":"2024-09-19T08:01:26.721781+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:20.193753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723508+0000","last_change":"2024-09-19T08:01:08.761432+0000","last_active":"2024-09-19T08:01:26.723508+0000","last_peered":"2024-09-19T08:01:26.723508+0000","last_clean":"2024-09-19T08:01:26.723508+0000","last_became_active":"2024-09-19T08:01:08.761044+0000","last_became_peered":"2024-09-19T08:01:08.761044+0000","last_unstale":"2024-09-19T08:01:26.723508+0000","last_undegraded":"2024-09-19T08:01:26.723508+0000","last_fullsized":"2024-09-19T08:01:26.723508+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:12.237334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718915+0000","last_change":"2024-09-19T08:01:10.772559+0000","last_active":"2024-09-19T08:01:26.718915+0000","last_peered":"2024-09-19T08:01:26.718915+0000","last_clean":"2024-09-19T08:01:26.718915+0000","last_became_active":"2024-09-19T08:01:10.772176+0000","last_became_peered":"2024-09-19T08:01:10.772176+0000","last_unstale":"2024-09-19T08:01:26.718915+0000","last_undegraded":"2024-09-19T08:01:26.718915+0000","last_fullsized":"2024-09-19T08:01:26.718915+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:27:16.296127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721816+0000","last_change":"2024-09-19T08:01:08.759989+0000","last_active":"2024-09-19T08:01:26.721816+0000","last_peered":"2024-09-19T08:01:26.721816+0000","last_clean":"2024-09-19T08:01:26.721816+0000","last_became_active":"2024-09-19T08:01:08.759705+0000","last_became_peered":"2024-09-19T08:01:08.759705+0000","last_unstale":"2024-09-19T08:01:26.721816+0000","last_undegraded":"2024-09-19T08:01:26.721816+0000","last_fullsized":"2024-09-19T08:01:26.721816+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:33:25.237567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723490+0000","last_change":"2024-09-19T08:01:10.776528+0000","last_active":"2024-09-19T08:01:26.723490+0000","last_peered":"2024-09-19T08:01:26.723490+0000","last_clean":"2024-09-19T08:01:26.723490+0000","last_became_active":"2024-09-19T08:01:10.776222+0000","last_became_peered":"2024-09-19T08:01:10.776222+0000","last_unstale":"2024-09-19T08:01:26.723490+0000","last_undegraded":"2024-09-19T08:01:26.723490+0000","last_fullsized":"2024-09-19T08:01:26.723490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.787492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722186+0000","last_change":"2024-09-19T08:01:08.759631+0000","last_active":"2024-09-19T08:01:26.722186+0000","last_peered":"2024-09-19T08:01:26.722186+0000","last_clean":"2024-09-19T08:01:26.722186+0000","last_became_active":"2024-09-19T08:01:08.759156+0000","last_became_peered":"2024-09-19T08:01:08.759156+0000","last_unstale":"2024-09-19T08:01:26.722186+0000","last_undegraded":"2024-09-19T08:01:26.722186+0000","last_fullsized":"2024-09-19T08:01:26.722186+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:48:50.910824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722951+0000","last_change":"2024-09-19T08:01:10.772660+0000","last_active":"2024-09-19T08:01:26.722951+0000","last_peered":"2024-09-19T08:01:26.722951+0000","last_clean":"2024-09-19T08:01:26.722951+0000","last_became_active":"2024-09-19T08:01:10.771816+0000","last_became_peered":"2024-09-19T08:01:10.771816+0000","last_unstale":"2024-09-19T08:01:26.722951+0000","last_undegraded":"2024-09-19T08:01:26.722951+0000","last_fullsized":"2024-09-19T08:01:26.722951+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:04:32.667107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'6","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:30.722236+0000","last_change":"2024-09-19T08:01:09.749844+0000","last_active":"2024-09-19T08:01:30.722236+0000","last_peered":"2024-09-19T08:01:30.722236+0000","last_clean":"2024-09-19T08:01:30.722236+0000","last_became_active":"2024-09-19T08:01:09.749410+0000","last_became_peered":"2024-09-19T08:01:09.749410+0000","last_unstale":"2024-09-19T08:01:30.722236+0000","last_undegraded":"2024-09-19T08:01:30.722236+0000","last_fullsized":"2024-09-19T08:01:30.722236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:03:14.989814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1563,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723119+0000","last_change":"2024-09-19T08:01:10.773782+0000","last_active":"2024-09-19T08:01:26.723119+0000","last_peered":"2024-09-19T08:01:26.723119+0000","last_clean":"2024-09-19T08:01:26.723119+0000","last_became_active":"2024-09-19T08:01:10.773471+0000","last_became_peered":"2024-09-19T08:01:10.773471+0000","last_unstale":"2024-09-19T08:01:26.723119+0000","last_undegraded":"2024-09-19T08:01:26.723119+0000","last_fullsized":"2024-09-19T08:01:26.723119+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:15.780259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":341,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722698+0000","last_change":"2024-09-19T08:00:01.677517+0000","last_active":"2024-09-19T08:01:26.722698+0000","last_peered":"2024-09-19T08:01:26.722698+0000","last_clean":"2024-09-19T08:01:26.722698+0000","last_became_active":"2024-09-19T08:00:01.677089+0000","last_became_peered":"2024-09-19T08:00:01.677089+0000","last_unstale":"2024-09-19T08:01:26.722698+0000","last_undegraded":"2024-09-19T08:01:26.722698+0000","last_fullsized":"2024-09-19T08:01:26.722698+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_clean_scrub_stamp":"2024-09-19T07:56:43.962142+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:10:36.158811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758444+0000","last_change":"2024-09-19T08:01:08.759819+0000","last_active":"2024-09-19T08:01:26.758444+0000","last_peered":"2024-09-19T08:01:26.758444+0000","last_clean":"2024-09-19T08:01:26.758444+0000","last_became_active":"2024-09-19T08:01:08.759420+0000","last_became_peered":"2024-09-19T08:01:08.759420+0000","last_unstale":"2024-09-19T08:01:26.758444+0000","last_undegraded":"2024-09-19T08:01:26.758444+0000","last_fullsized":"2024-09-19T08:01:26.758444+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:06:16.852323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.724656+0000","last_change":"2024-09-19T08:01:26.724260+0000","last_active":"2024-09-19T08:01:26.724656+0000","last_peered":"2024-09-19T08:01:26.724656+0000","last_clean":"2024-09-19T08:01:26.724656+0000","last_became_active":"2024-09-19T08:01:26.723851+0000","last_became_peered":"2024-09-19T08:01:26.723851+0000","last_unstale":"2024-09-19T08:01:26.724656+0000","last_undegraded":"2024-09-19T08:01:26.724656+0000","last_fullsized":"2024-09-19T08:01:26.724656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:18:46.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722601+0000","last_change":"2024-09-19T08:01:08.761291+0000","last_active":"2024-09-19T08:01:26.722601+0000","last_peered":"2024-09-19T08:01:26.722601+0000","last_clean":"2024-09-19T08:01:26.722601+0000","last_became_active":"2024-09-19T08:01:08.760238+0000","last_became_peered":"2024-09-19T08:01:08.760238+0000","last_unstale":"2024-09-19T08:01:26.722601+0000","last_undegraded":"2024-09-19T08:01:26.722601+0000","last_fullsized":"2024-09-19T08:01:26.722601+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:26:09.980573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721415+0000","last_change":"2024-09-19T08:01:10.773056+0000","last_active":"2024-09-19T08:01:26.721415+0000","last_peered":"2024-09-19T08:01:26.721415+0000","last_clean":"2024-09-19T08:01:26.721415+0000","last_became_active":"2024-09-19T08:01:10.772431+0000","last_became_peered":"2024-09-19T08:01:10.772431+0000","last_unstale":"2024-09-19T08:01:26.721415+0000","last_undegraded":"2024-09-19T08:01:26.721415+0000","last_fullsized":"2024-09-19T08:01:26.721415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:39:06.878086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"78'5","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:33.710275+0000","last_change":"2024-09-19T08:01:09.749707+0000","last_active":"2024-09-19T08:01:33.710275+0000","last_peered":"2024-09-19T08:01:33.710275+0000","last_clean":"2024-09-19T08:01:33.710275+0000","last_became_active":"2024-09-19T08:01:09.749013+0000","last_became_peered":"2024-09-19T08:01:09.749013+0000","last_unstale":"2024-09-19T08:01:33.710275+0000","last_undegraded":"2024-09-19T08:01:33.710275+0000","last_fullsized":"2024-09-19T08:01:33.710275+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:40:12.682158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721922+0000","last_change":"2024-09-19T08:01:10.774101+0000","last_active":"2024-09-19T08:01:26.721922+0000","last_peered":"2024-09-19T08:01:26.721922+0000","last_clean":"2024-09-19T08:01:26.721922+0000","last_became_active":"2024-09-19T08:01:10.773822+0000","last_became_peered":"2024-09-19T08:01:10.773822+0000","last_unstale":"2024-09-19T08:01:26.721922+0000","last_undegraded":"2024-09-19T08:01:26.721922+0000","last_fullsized":"2024-09-19T08:01:26.721922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:29.571118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"78'5","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:32.732237+0000","last_change":"2024-09-19T08:01:08.761675+0000","last_active":"2024-09-19T08:01:32.732237+0000","last_peered":"2024-09-19T08:01:32.732237+0000","last_clean":"2024-09-19T08:01:32.732237+0000","last_became_active":"2024-09-19T08:01:08.761346+0000","last_became_peered":"2024-09-19T08:01:08.761346+0000","last_unstale":"2024-09-19T08:01:32.732237+0000","last_undegraded":"2024-09-19T08:01:32.732237+0000","last_fullsized":"2024-09-19T08:01:32.732237+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:51:19.237458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718382+0000","last_change":"2024-09-19T08:01:10.781565+0000","last_active":"2024-09-19T08:01:26.718382+0000","last_peered":"2024-09-19T08:01:26.718382+0000","last_clean":"2024-09-19T08:01:26.718382+0000","last_became_active":"2024-09-19T08:01:10.781301+0000","last_became_peered":"2024-09-19T08:01:10.781301+0000","last_unstale":"2024-09-19T08:01:26.718382+0000","last_undegraded":"2024-09-19T08:01:26.718382+0000","last_fullsized":"2024-09-19T08:01:26.718382+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:55:52.932009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719945+0000","last_change":"2024-09-19T08:01:09.748918+0000","last_active":"2024-09-19T08:01:26.719945+0000","last_peered":"2024-09-19T08:01:26.719945+0000","last_clean":"2024-09-19T08:01:26.719945+0000","last_became_active":"2024-09-19T08:01:09.748497+0000","last_became_peered":"2024-09-19T08:01:09.748497+0000","last_unstale":"2024-09-19T08:01:26.719945+0000","last_undegraded":"2024-09-19T08:01:26.719945+0000","last_fullsized":"2024-09-19T08:01:26.719945+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:20:31.030133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718202+0000","last_change":"2024-09-19T08:01:10.781388+0000","last_active":"2024-09-19T08:01:26.718202+0000","last_peered":"2024-09-19T08:01:26.718202+0000","last_clean":"2024-09-19T08:01:26.718202+0000","last_became_active":"2024-09-19T08:01:10.781063+0000","last_became_peered":"2024-09-19T08:01:10.781063+0000","last_unstale":"2024-09-19T08:01:26.718202+0000","last_undegraded":"2024-09-19T08:01:26.718202+0000","last_fullsized":"2024-09-19T08:01:26.718202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:12:08.852717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723781+0000","last_change":"2024-09-19T08:01:09.750080+0000","last_active":"2024-09-19T08:01:26.723781+0000","last_peered":"2024-09-19T08:01:26.723781+0000","last_clean":"2024-09-19T08:01:26.723781+0000","last_became_active":"2024-09-19T08:01:09.749053+0000","last_became_peered":"2024-09-19T08:01:09.749053+0000","last_unstale":"2024-09-19T08:01:26.723781+0000","last_undegraded":"2024-09-19T08:01:26.723781+0000","last_fullsized":"2024-09-19T08:01:26.723781+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:15:45.292714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721665+0000","last_change":"2024-09-19T08:01:10.767070+0000","last_active":"2024-09-19T08:01:26.721665+0000","last_peered":"2024-09-19T08:01:26.721665+0000","last_clean":"2024-09-19T08:01:26.721665+0000","last_became_active":"2024-09-19T08:01:10.766760+0000","last_became_peered":"2024-09-19T08:01:10.766760+0000","last_unstale":"2024-09-19T08:01:26.721665+0000","last_undegraded":"2024-09-19T08:01:26.721665+0000","last_fullsized":"2024-09-19T08:01:26.721665+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:17.841135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'12","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:30.790310+0000","last_change":"2024-09-19T08:01:08.756143+0000","last_active":"2024-09-19T08:01:30.790310+0000","last_peered":"2024-09-19T08:01:30.790310+0000","last_clean":"2024-09-19T08:01:30.790310+0000","last_became_active":"2024-09-19T08:01:08.755777+0000","last_became_peered":"2024-09-19T08:01:08.755777+0000","last_unstale":"2024-09-19T08:01:30.790310+0000","last_undegraded":"2024-09-19T08:01:30.790310+0000","last_fullsized":"2024-09-19T08:01:30.790310+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:21.430673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718462+0000","last_change":"2024-09-19T08:01:10.774140+0000","last_active":"2024-09-19T08:01:26.718462+0000","last_peered":"2024-09-19T08:01:26.718462+0000","last_clean":"2024-09-19T08:01:26.718462+0000","last_became_active":"2024-09-19T08:01:10.773819+0000","last_became_peered":"2024-09-19T08:01:10.773819+0000","last_unstale":"2024-09-19T08:01:26.718462+0000","last_undegraded":"2024-09-19T08:01:26.718462+0000","last_fullsized":"2024-09-19T08:01:26.718462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:15:13.174013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758589+0000","last_change":"2024-09-19T08:01:09.169129+0000","last_active":"2024-09-19T08:01:26.758589+0000","last_peered":"2024-09-19T08:01:26.758589+0000","last_clean":"2024-09-19T08:01:26.758589+0000","last_became_active":"2024-09-19T08:01:09.168734+0000","last_became_peered":"2024-09-19T08:01:09.168734+0000","last_unstale":"2024-09-19T08:01:26.758589+0000","last_undegraded":"2024-09-19T08:01:26.758589+0000","last_fullsized":"2024-09-19T08:01:26.758589+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:01:52.232391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757822+0000","last_change":"2024-09-19T08:01:10.768592+0000","last_active":"2024-09-19T08:01:26.757822+0000","last_peered":"2024-09-19T08:01:26.757822+0000","last_clean":"2024-09-19T08:01:26.757822+0000","last_became_active":"2024-09-19T08:01:10.767645+0000","last_became_peered":"2024-09-19T08:01:10.767645+0000","last_unstale":"2024-09-19T08:01:26.757822+0000","last_undegraded":"2024-09-19T08:01:26.757822+0000","last_fullsized":"2024-09-19T08:01:26.757822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:08:46.227461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721037+0000","last_change":"2024-09-19T08:01:09.749240+0000","last_active":"2024-09-19T08:01:26.721037+0000","last_peered":"2024-09-19T08:01:26.721037+0000","last_clean":"2024-09-19T08:01:26.721037+0000","last_became_active":"2024-09-19T08:01:09.748945+0000","last_became_peered":"2024-09-19T08:01:09.748945+0000","last_unstale":"2024-09-19T08:01:26.721037+0000","last_undegraded":"2024-09-19T08:01:26.721037+0000","last_fullsized":"2024-09-19T08:01:26.721037+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:25:13.791290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.720802+0000","last_change":"2024-09-19T08:01:10.783133+0000","last_active":"2024-09-19T08:01:26.720802+0000","last_peered":"2024-09-19T08:01:26.720802+0000","last_clean":"2024-09-19T08:01:26.720802+0000","last_became_active":"2024-09-19T08:01:10.782797+0000","last_became_peered":"2024-09-19T08:01:10.782797+0000","last_unstale":"2024-09-19T08:01:26.720802+0000","last_undegraded":"2024-09-19T08:01:26.720802+0000","last_fullsized":"2024-09-19T08:01:26.720802+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:30:43.551877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718123+0000","last_change":"2024-09-19T08:01:08.757964+0000","last_active":"2024-09-19T08:01:26.718123+0000","last_peered":"2024-09-19T08:01:26.718123+0000","last_clean":"2024-09-19T08:01:26.718123+0000","last_became_active":"2024-09-19T08:01:08.757526+0000","last_became_peered":"2024-09-19T08:01:08.757526+0000","last_unstale":"2024-09-19T08:01:26.718123+0000","last_undegraded":"2024-09-19T08:01:26.718123+0000","last_fullsized":"2024-09-19T08:01:26.718123+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:03.565702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718652+0000","last_change":"2024-09-19T08:01:10.769974+0000","last_active":"2024-09-19T08:01:26.718652+0000","last_peered":"2024-09-19T08:01:26.718652+0000","last_clean":"2024-09-19T08:01:26.718652+0000","last_became_active":"2024-09-19T08:01:10.769650+0000","last_became_peered":"2024-09-19T08:01:10.769650+0000","last_unstale":"2024-09-19T08:01:26.718652+0000","last_undegraded":"2024-09-19T08:01:26.718652+0000","last_fullsized":"2024-09-19T08:01:26.718652+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:01.755198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725382+0000","last_change":"2024-09-19T08:01:08.763476+0000","last_active":"2024-09-19T08:01:26.725382+0000","last_peered":"2024-09-19T08:01:26.725382+0000","last_clean":"2024-09-19T08:01:26.725382+0000","last_became_active":"2024-09-19T08:01:08.763154+0000","last_became_peered":"2024-09-19T08:01:08.763154+0000","last_unstale":"2024-09-19T08:01:26.725382+0000","last_undegraded":"2024-09-19T08:01:26.725382+0000","last_fullsized":"2024-09-19T08:01:26.725382+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:06.202452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718164+0000","last_change":"2024-09-19T08:01:10.784201+0000","last_active":"2024-09-19T08:01:26.718164+0000","last_peered":"2024-09-19T08:01:26.718164+0000","last_clean":"2024-09-19T08:01:26.718164+0000","last_became_active":"2024-09-19T08:01:10.783867+0000","last_became_peered":"2024-09-19T08:01:10.783867+0000","last_unstale":"2024-09-19T08:01:26.718164+0000","last_undegraded":"2024-09-19T08:01:26.718164+0000","last_fullsized":"2024-09-19T08:01:26.718164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:43:50.082183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721581+0000","last_change":"2024-09-19T08:01:08.760479+0000","last_active":"2024-09-19T08:01:26.721581+0000","last_peered":"2024-09-19T08:01:26.721581+0000","last_clean":"2024-09-19T08:01:26.721581+0000","last_became_active":"2024-09-19T08:01:08.760004+0000","last_became_peered":"2024-09-19T08:01:08.760004+0000","last_unstale":"2024-09-19T08:01:26.721581+0000","last_undegraded":"2024-09-19T08:01:26.721581+0000","last_fullsized":"2024-09-19T08:01:26.721581+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:41:19.956623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722763+0000","last_change":"2024-09-19T08:01:10.772184+0000","last_active":"2024-09-19T08:01:26.722763+0000","last_peered":"2024-09-19T08:01:26.722763+0000","last_clean":"2024-09-19T08:01:26.722763+0000","last_became_active":"2024-09-19T08:01:10.771825+0000","last_became_peered":"2024-09-19T08:01:10.771825+0000","last_unstale":"2024-09-19T08:01:26.722763+0000","last_undegraded":"2024-09-19T08:01:26.722763+0000","last_fullsized":"2024-09-19T08:01:26.722763+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:36.764575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721944+0000","last_change":"2024-09-19T08:01:10.779368+0000","last_active":"2024-09-19T08:01:26.721944+0000","last_peered":"2024-09-19T08:01:26.721944+0000","last_clean":"2024-09-19T08:01:26.721944+0000","last_became_active":"2024-09-19T08:01:10.778884+0000","last_became_peered":"2024-09-19T08:01:10.778884+0000","last_unstale":"2024-09-19T08:01:26.721944+0000","last_undegraded":"2024-09-19T08:01:26.721944+0000","last_fullsized":"2024-09-19T08:01:26.721944+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:28:29.630300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.741496+0000","last_change":"2024-09-19T08:01:26.741053+0000","last_active":"2024-09-19T08:01:26.741496+0000","last_peered":"2024-09-19T08:01:26.741496+0000","last_clean":"2024-09-19T08:01:26.741496+0000","last_became_active":"2024-09-19T08:01:26.724714+0000","last_became_peered":"2024-09-19T08:01:26.724714+0000","last_unstale":"2024-09-19T08:01:26.741496+0000","last_undegraded":"2024-09-19T08:01:26.741496+0000","last_fullsized":"2024-09-19T08:01:26.741496+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:59.583797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721113+0000","last_change":"2024-09-19T08:01:10.781389+0000","last_active":"2024-09-19T08:01:26.721113+0000","last_peered":"2024-09-19T08:01:26.721113+0000","last_clean":"2024-09-19T08:01:26.721113+0000","last_became_active":"2024-09-19T08:01:10.781080+0000","last_became_peered":"2024-09-19T08:01:10.781080+0000","last_unstale":"2024-09-19T08:01:26.721113+0000","last_undegraded":"2024-09-19T08:01:26.721113+0000","last_fullsized":"2024-09-19T08:01:26.721113+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:40:45.225957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.768102+0000","last_change":"2024-09-19T08:01:26.768102+0000","last_active":"2024-09-19T08:01:26.768102+0000","last_peered":"2024-09-19T08:01:26.768102+0000","last_clean":"2024-09-19T08:01:26.768102+0000","last_became_active":"2024-09-19T08:01:26.759462+0000","last_became_peered":"2024-09-19T08:01:26.759462+0000","last_unstale":"2024-09-19T08:01:26.768102+0000","last_undegraded":"2024-09-19T08:01:26.768102+0000","last_fullsized":"2024-09-19T08:01:26.768102+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:02:06.815238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757966+0000","last_change":"2024-09-19T08:01:10.771031+0000","last_active":"2024-09-19T08:01:26.757966+0000","last_peered":"2024-09-19T08:01:26.757966+0000","last_clean":"2024-09-19T08:01:26.757966+0000","last_became_active":"2024-09-19T08:01:10.770675+0000","last_became_peered":"2024-09-19T08:01:10.770675+0000","last_unstale":"2024-09-19T08:01:26.757966+0000","last_undegraded":"2024-09-19T08:01:26.757966+0000","last_fullsized":"2024-09-19T08:01:26.757966+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:42.203848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718482+0000","last_change":"2024-09-19T08:01:09.748622+0000","last_active":"2024-09-19T08:01:26.718482+0000","last_peered":"2024-09-19T08:01:26.718482+0000","last_clean":"2024-09-19T08:01:26.718482+0000","last_became_active":"2024-09-19T08:01:09.748202+0000","last_became_peered":"2024-09-19T08:01:09.748202+0000","last_unstale":"2024-09-19T08:01:26.718482+0000","last_undegraded":"2024-09-19T08:01:26.718482+0000","last_fullsized":"2024-09-19T08:01:26.718482+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:17:39.861820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718292+0000","last_change":"2024-09-19T08:01:10.768949+0000","last_active":"2024-09-19T08:01:26.718292+0000","last_peered":"2024-09-19T08:01:26.718292+0000","last_clean":"2024-09-19T08:01:26.718292+0000","last_became_active":"2024-09-19T08:01:10.768159+0000","last_became_peered":"2024-09-19T08:01:10.768159+0000","last_unstale":"2024-09-19T08:01:26.718292+0000","last_undegraded":"2024-09-19T08:01:26.718292+0000","last_fullsized":"2024-09-19T08:01:26.718292+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:43.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"78'13","reported_seq":73,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:31.730231+0000","last_change":"2024-09-19T08:01:26.735877+0000","last_active":"2024-09-19T08:01:31.730231+0000","last_peered":"2024-09-19T08:01:31.730231+0000","last_clean":"2024-09-19T08:01:31.730231+0000","last_became_active":"2024-09-19T08:01:26.723597+0000","last_became_peered":"2024-09-19T08:01:26.723597+0000","last_unstale":"2024-09-19T08:01:31.730231+0000","last_undegraded":"2024-09-19T08:01:31.730231+0000","last_fullsized":"2024-09-19T08:01:31.730231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:43:42.340832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,1],"acting":[9,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":54010,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":156,"num_read_kb":141,"num_write":231,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1254,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":528384,"data_stored":162030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":200,"ondisk_log_size":200,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645329,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":888832,"data_stored":427316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874262,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32712,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712440,"statfs":{"total":95995035648,"available":95961538560,"internally_reserved":0,"allocated":1564672,"data_stored":1063500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005083,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":901120,"data_stored":425599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234015,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":884736,"data_stored":424723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1486848,"data_stored":1017825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593770,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":884736,"data_stored":425316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822702,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28500,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716652,"statfs":{"total":95995035648,"available":95965851648,"internally_reserved":0,"allocated":1507328,"data_stored":1018175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986227,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":884736,"data_stored":425177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215159,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":901120,"data_stored":425141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476797,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":474037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607618,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":942080,"data_stored":473703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":905216,"data_stored":427048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":49226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":49560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":48538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T08:01:37.941 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-19T08:01:37.941 INFO:tasks.cephadm:Setting up client nodes... 2024-09-19T08:01:37.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-19T08:01:38.187 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.679+0000 7f44186ca640 1 -- 172.21.15.98:0/3559803462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4410102e00 msgr2=0x7f4410103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.679+0000 7f44186ca640 1 --2- 172.21.15.98:0/3559803462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4410102e00 0x7f4410103280 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4400009900 tx=0x7f440002a410 comp rx=0 tx=0).stop 2024-09-19T08:01:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.679+0000 7f44186ca640 1 -- 172.21.15.98:0/3559803462 shutdown_connections 2024-09-19T08:01:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.679+0000 7f44186ca640 1 --2- 172.21.15.98:0/3559803462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44101037c0 0x7f441010a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.679+0000 7f44186ca640 1 --2- 172.21.15.98:0/3559803462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4410102e00 0x7f4410103280 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.679+0000 7f44186ca640 1 --2- 172.21.15.98:0/3559803462 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4410101c00 0x7f4410102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.679+0000 7f44186ca640 1 -- 172.21.15.98:0/3559803462 >> 172.21.15.98:0/3559803462 conn(0x7f44100fd390 msgr2=0x7f44100ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.679+0000 7f44186ca640 1 -- 172.21.15.98:0/3559803462 shutdown_connections 2024-09-19T08:01:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.679+0000 7f44186ca640 1 -- 172.21.15.98:0/3559803462 wait complete. 2024-09-19T08:01:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.679+0000 7f44186ca640 1 Processor -- start 2024-09-19T08:01:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.679+0000 7f44186ca640 1 -- start start 2024-09-19T08:01:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f44186ca640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4410101c00 0x7f4410197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f44186ca640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4410102e00 0x7f44101983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f44186ca640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44101037c0 0x7f441019f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f44186ca640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f441010d860 con 0x7f44101037c0 2024-09-19T08:01:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f44186ca640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f441010d6e0 con 0x7f4410101c00 2024-09-19T08:01:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f44186ca640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f441010d9e0 con 0x7f4410102e00 2024-09-19T08:01:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f4416c67640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44101037c0 0x7f441019f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f4416c67640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44101037c0 0x7f441019f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50834/0 (socket says 172.21.15.98:50834) 2024-09-19T08:01:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f4416c67640 1 -- 172.21.15.98:0/3936516711 learned_addr learned my addr 172.21.15.98:0/3936516711 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f4416466640 1 --2- 172.21.15.98:0/3936516711 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4410101c00 0x7f4410197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f4416c67640 1 -- 172.21.15.98:0/3936516711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4410102e00 msgr2=0x7f44101983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f4416c67640 1 --2- 172.21.15.98:0/3936516711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4410102e00 0x7f44101983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f4416c67640 1 -- 172.21.15.98:0/3936516711 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4410101c00 msgr2=0x7f4410197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f4416c67640 1 --2- 172.21.15.98:0/3936516711 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4410101c00 0x7f4410197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f4416c67640 1 -- 172.21.15.98:0/3936516711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f441019fab0 con 0x7f44101037c0 2024-09-19T08:01:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f4416c67640 1 --2- 172.21.15.98:0/3936516711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44101037c0 0x7f441019f440 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f440800cce0 tx=0x7f440800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f44077fe640 1 -- 172.21.15.98:0/3936516711 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440800f450 con 0x7f44101037c0 2024-09-19T08:01:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f44077fe640 1 -- 172.21.15.98:0/3936516711 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f440800fad0 con 0x7f44101037c0 2024-09-19T08:01:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f44186ca640 1 -- 172.21.15.98:0/3936516711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f441019fda0 con 0x7f44101037c0 2024-09-19T08:01:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f44077fe640 1 -- 172.21.15.98:0/3936516711 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408018c10 con 0x7f44101037c0 2024-09-19T08:01:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.680+0000 7f44186ca640 1 -- 172.21.15.98:0/3936516711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44101a02b0 con 0x7f44101037c0 2024-09-19T08:01:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.682+0000 7f44186ca640 1 -- 172.21.15.98:0/3936516711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4410108fd0 con 0x7f44101037c0 2024-09-19T08:01:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.682+0000 7f44077fe640 1 -- 172.21.15.98:0/3936516711 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f440800f5f0 con 0x7f44101037c0 2024-09-19T08:01:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.683+0000 7f44077fe640 1 --2- 172.21.15.98:0/3936516711 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43fc0a3380 0x7f43fc0a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.683+0000 7f4416466640 1 --2- 172.21.15.98:0/3936516711 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43fc0a3380 0x7f43fc0a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.683+0000 7f44077fe640 1 -- 172.21.15.98:0/3936516711 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f44080cffd0 con 0x7f44101037c0 2024-09-19T08:01:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.684+0000 7f4416466640 1 --2- 172.21.15.98:0/3936516711 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43fc0a3380 0x7f43fc0a5840 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4410102c60 tx=0x7f440c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:38.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.687+0000 7f44077fe640 1 -- 172.21.15.98:0/3936516711 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f4408014030 con 0x7f44101037c0 2024-09-19T08:01:38.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.916+0000 7f44186ca640 1 -- 172.21.15.98:0/3936516711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f441002d970 con 0x7f44101037c0 2024-09-19T08:01:38.917 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:38 smithi098 ceph-mon[31333]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:01:38.917 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:38 smithi098 ceph-mon[31333]: pgmap v286: 97 pgs: 97 active+clean; 630 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.2 KiB/s wr, 12 op/s; 115 B/s, 0 keys/s, 1 objects/s recovering 2024-09-19T08:01:38.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.919+0000 7f44077fe640 1 -- 172.21.15.98:0/3936516711 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) ==== 170+0+59 (secure 0 0 0) 0x7f440809cc40 con 0x7f44101037c0 2024-09-19T08:01:38.919 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-09-19T08:01:38.919 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBi2utml9m2NhAAkPUZmXaNzbUT0AWDWWqmHA== 2024-09-19T08:01:38.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.921+0000 7f44186ca640 1 -- 172.21.15.98:0/3936516711 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43fc0a3380 msgr2=0x7f43fc0a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:38.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.921+0000 7f44186ca640 1 --2- 172.21.15.98:0/3936516711 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43fc0a3380 0x7f43fc0a5840 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4410102c60 tx=0x7f440c00a2d0 comp rx=0 tx=0).stop 2024-09-19T08:01:38.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.921+0000 7f44186ca640 1 -- 172.21.15.98:0/3936516711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44101037c0 msgr2=0x7f441019f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:38.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.921+0000 7f44186ca640 1 --2- 172.21.15.98:0/3936516711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44101037c0 0x7f441019f440 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f440800cce0 tx=0x7f440800a620 comp rx=0 tx=0).stop 2024-09-19T08:01:38.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.921+0000 7f44186ca640 1 -- 172.21.15.98:0/3936516711 shutdown_connections 2024-09-19T08:01:38.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.921+0000 7f44186ca640 1 --2- 172.21.15.98:0/3936516711 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43fc0a3380 0x7f43fc0a5840 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:38.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.921+0000 7f44186ca640 1 --2- 172.21.15.98:0/3936516711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44101037c0 0x7f441019f440 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:38.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.921+0000 7f44186ca640 1 --2- 172.21.15.98:0/3936516711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4410102e00 0x7f44101983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:38.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.921+0000 7f44186ca640 1 --2- 172.21.15.98:0/3936516711 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4410101c00 0x7f4410197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:38.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.921+0000 7f44186ca640 1 -- 172.21.15.98:0/3936516711 >> 172.21.15.98:0/3936516711 conn(0x7f44100fd390 msgr2=0x7f44101061a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:38.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.921+0000 7f44186ca640 1 -- 172.21.15.98:0/3936516711 shutdown_connections 2024-09-19T08:01:38.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:38.921+0000 7f44186ca640 1 -- 172.21.15.98:0/3936516711 wait complete. 2024-09-19T08:01:38.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:38 smithi139 ceph-mon[34874]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:01:38.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:38 smithi139 ceph-mon[34874]: pgmap v286: 97 pgs: 97 active+clean; 630 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.2 KiB/s wr, 12 op/s; 115 B/s, 0 keys/s, 1 objects/s recovering 2024-09-19T08:01:39.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:38 smithi162 ceph-mon[35260]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:01:39.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:38 smithi162 ceph-mon[35260]: pgmap v286: 97 pgs: 97 active+clean; 630 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.2 KiB/s wr, 12 op/s; 115 B/s, 0 keys/s, 1 objects/s recovering 2024-09-19T08:01:39.652 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T08:01:39.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-19T08:01:39.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-19T08:01:39.691 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-19T08:01:39.692 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-19T08:01:39.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-19T08:01:39.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:39 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3936516711' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-19T08:01:39.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:39 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3936516711' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-19T08:01:39.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:39 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3936516711' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-19T08:01:39.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:39 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3936516711' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-19T08:01:39.977 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:40.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:39 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3936516711' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-19T08:01:40.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:39 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3936516711' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-19T08:01:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.432+0000 7f11fa44f640 1 -- 172.21.15.98:0/1871790906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11f4101de0 msgr2=0x7f11f41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.432+0000 7f11fa44f640 1 --2- 172.21.15.98:0/1871790906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11f4101de0 0x7f11f41021c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f11e4009900 tx=0x7f11e402a410 comp rx=0 tx=0).stop 2024-09-19T08:01:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.432+0000 7f11fa44f640 1 -- 172.21.15.98:0/1871790906 shutdown_connections 2024-09-19T08:01:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.432+0000 7f11fa44f640 1 --2- 172.21.15.98:0/1871790906 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11f4106f10 0x7f11f410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.432+0000 7f11fa44f640 1 --2- 172.21.15.98:0/1871790906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f4102790 0x7f11f41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.432+0000 7f11fa44f640 1 --2- 172.21.15.98:0/1871790906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11f4101de0 0x7f11f41021c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.432+0000 7f11fa44f640 1 -- 172.21.15.98:0/1871790906 >> 172.21.15.98:0/1871790906 conn(0x7f11f40fd8c0 msgr2=0x7f11f40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.432+0000 7f11fa44f640 1 -- 172.21.15.98:0/1871790906 shutdown_connections 2024-09-19T08:01:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.432+0000 7f11fa44f640 1 -- 172.21.15.98:0/1871790906 wait complete. 2024-09-19T08:01:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.433+0000 7f11fa44f640 1 Processor -- start 2024-09-19T08:01:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.433+0000 7f11fa44f640 1 -- start start 2024-09-19T08:01:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.433+0000 7f11fa44f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11f4101de0 0x7f11f41984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.433+0000 7f11fa44f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11f4102790 0x7f11f4198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.433+0000 7f11fa44f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f4106f10 0x7f11f419cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.433+0000 7f11fa44f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11f4111fb0 con 0x7f11f4101de0 2024-09-19T08:01:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.433+0000 7f11fa44f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f11f4111e30 con 0x7f11f4102790 2024-09-19T08:01:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.433+0000 7f11fa44f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f11f4112130 con 0x7f11f4106f10 2024-09-19T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.433+0000 7f11f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11f4101de0 0x7f11f41984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.433+0000 7f11f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11f4101de0 0x7f11f41984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50858/0 (socket says 172.21.15.98:50858) 2024-09-19T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.433+0000 7f11f3fff640 1 -- 172.21.15.98:0/2884171885 learned_addr learned my addr 172.21.15.98:0/2884171885 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.434+0000 7f11f37fe640 1 --2- 172.21.15.98:0/2884171885 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11f4102790 0x7f11f4198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.434+0000 7f11f3fff640 1 -- 172.21.15.98:0/2884171885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f4106f10 msgr2=0x7f11f419cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.434+0000 7f11f3fff640 1 --2- 172.21.15.98:0/2884171885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f4106f10 0x7f11f419cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.434+0000 7f11f3fff640 1 -- 172.21.15.98:0/2884171885 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11f4102790 msgr2=0x7f11f4198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.434+0000 7f11f3fff640 1 --2- 172.21.15.98:0/2884171885 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11f4102790 0x7f11f4198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.434+0000 7f11f3fff640 1 -- 172.21.15.98:0/2884171885 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f419d4a0 con 0x7f11f4101de0 2024-09-19T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.434+0000 7f11f3fff640 1 --2- 172.21.15.98:0/2884171885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11f4101de0 0x7f11f41984f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f11e4006010 tx=0x7f11e40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.434+0000 7f11f17fa640 1 -- 172.21.15.98:0/2884171885 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e403f030 con 0x7f11f4101de0 2024-09-19T08:01:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.434+0000 7f11f17fa640 1 -- 172.21.15.98:0/2884171885 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e4003ce0 con 0x7f11f4101de0 2024-09-19T08:01:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.434+0000 7f11fa44f640 1 -- 172.21.15.98:0/2884171885 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f419d730 con 0x7f11f4101de0 2024-09-19T08:01:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.434+0000 7f11f17fa640 1 -- 172.21.15.98:0/2884171885 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4035070 con 0x7f11f4101de0 2024-09-19T08:01:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.434+0000 7f11fa44f640 1 -- 172.21.15.98:0/2884171885 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f41a5070 con 0x7f11f4101de0 2024-09-19T08:01:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.435+0000 7f11fa44f640 1 -- 172.21.15.98:0/2884171885 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11f41032c0 con 0x7f11f4101de0 2024-09-19T08:01:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.436+0000 7f11f17fa640 1 -- 172.21.15.98:0/2884171885 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f11e4042030 con 0x7f11f4101de0 2024-09-19T08:01:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.437+0000 7f11f17fa640 1 --2- 172.21.15.98:0/2884171885 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f11dc0a32d0 0x7f11dc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.437+0000 7f11f37fe640 1 --2- 172.21.15.98:0/2884171885 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f11dc0a32d0 0x7f11dc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.437+0000 7f11f17fa640 1 -- 172.21.15.98:0/2884171885 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f11e40f0200 con 0x7f11f4101de0 2024-09-19T08:01:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.438+0000 7f11f37fe640 1 --2- 172.21.15.98:0/2884171885 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f11dc0a32d0 0x7f11dc0a5790 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f11e8005fd0 tx=0x7f11e8005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.442+0000 7f11f17fa640 1 -- 172.21.15.98:0/2884171885 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f11e40bce90 con 0x7f11f4101de0 2024-09-19T08:01:40.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.656+0000 7f11fa44f640 1 -- 172.21.15.98:0/2884171885 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f11f410a0f0 con 0x7f11f4101de0 2024-09-19T08:01:40.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.660+0000 7f11f17fa640 1 -- 172.21.15.98:0/2884171885 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264044 (secure 0 0 0) 0x7f11e40bc790 con 0x7f11f4101de0 2024-09-19T08:01:40.664 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:01:40.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.663+0000 7f11fa44f640 1 -- 172.21.15.98:0/2884171885 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f11dc0a32d0 msgr2=0x7f11dc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:40.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.663+0000 7f11fa44f640 1 --2- 172.21.15.98:0/2884171885 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f11dc0a32d0 0x7f11dc0a5790 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f11e8005fd0 tx=0x7f11e8005850 comp rx=0 tx=0).stop 2024-09-19T08:01:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.663+0000 7f11fa44f640 1 -- 172.21.15.98:0/2884171885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11f4101de0 msgr2=0x7f11f41984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.663+0000 7f11fa44f640 1 --2- 172.21.15.98:0/2884171885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11f4101de0 0x7f11f41984f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f11e4006010 tx=0x7f11e40042f0 comp rx=0 tx=0).stop 2024-09-19T08:01:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.663+0000 7f11fa44f640 1 -- 172.21.15.98:0/2884171885 shutdown_connections 2024-09-19T08:01:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.663+0000 7f11fa44f640 1 --2- 172.21.15.98:0/2884171885 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f11dc0a32d0 0x7f11dc0a5790 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.663+0000 7f11fa44f640 1 --2- 172.21.15.98:0/2884171885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f4106f10 0x7f11f419cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.663+0000 7f11fa44f640 1 --2- 172.21.15.98:0/2884171885 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11f4102790 0x7f11f4198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.663+0000 7f11fa44f640 1 --2- 172.21.15.98:0/2884171885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11f4101de0 0x7f11f41984f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.663+0000 7f11fa44f640 1 -- 172.21.15.98:0/2884171885 >> 172.21.15.98:0/2884171885 conn(0x7f11f40fd8c0 msgr2=0x7f11f40febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.663+0000 7f11fa44f640 1 -- 172.21.15.98:0/2884171885 shutdown_connections 2024-09-19T08:01:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:40.663+0000 7f11fa44f640 1 -- 172.21.15.98:0/2884171885 wait complete. 2024-09-19T08:01:40.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:40 smithi139 ceph-mon[34874]: pgmap v287: 97 pgs: 97 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.5 KiB/s wr, 14 op/s; 104 B/s, 0 objects/s recovering 2024-09-19T08:01:40.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:40 smithi098 ceph-mon[31333]: pgmap v287: 97 pgs: 97 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.5 KiB/s wr, 14 op/s; 104 B/s, 0 objects/s recovering 2024-09-19T08:01:41.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:40 smithi162 ceph-mon[35260]: pgmap v287: 97 pgs: 97 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.5 KiB/s wr, 14 op/s; 104 B/s, 0 objects/s recovering 2024-09-19T08:01:41.434 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":470997812},{"type":"v1","addr":"172.21.15.98:6801","nonce":470997812}]},"active_addr":"172.21.15.98:6801/470997812","active_change":"2024-09-19T07:54:25.475559+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.98:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":336479555}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1252058007}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3067331617}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":921776066}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":814801022}]}]} 2024-09-19T08:01:41.440 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-19T08:01:41.440 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-19T08:01:41.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T08:01:41.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:41 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2884171885' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-19T08:01:41.692 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:41.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:41 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2884171885' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-19T08:01:42.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:41 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2884171885' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-19T08:01:42.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.184+0000 7fa1d9364640 1 -- 172.21.15.98:0/105604991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1d4105590 msgr2=0x7fa1d410d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:42.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.184+0000 7fa1d9364640 1 --2- 172.21.15.98:0/105604991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1d4105590 0x7fa1d410d110 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc00adb0 tx=0x7fa1cc02b7a0 comp rx=0 tx=0).stop 2024-09-19T08:01:42.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.184+0000 7fa1d9364640 1 -- 172.21.15.98:0/105604991 shutdown_connections 2024-09-19T08:01:42.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.184+0000 7fa1d9364640 1 --2- 172.21.15.98:0/105604991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1d4105590 0x7fa1d410d110 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:42.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.184+0000 7fa1d9364640 1 --2- 172.21.15.98:0/105604991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1d4100e10 0x7fa1d4104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:42.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.184+0000 7fa1d9364640 1 --2- 172.21.15.98:0/105604991 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1d4100460 0x7fa1d4100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:42.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.184+0000 7fa1d9364640 1 -- 172.21.15.98:0/105604991 >> 172.21.15.98:0/105604991 conn(0x7fa1d40fbee0 msgr2=0x7fa1d40fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:42.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.184+0000 7fa1d9364640 1 -- 172.21.15.98:0/105604991 shutdown_connections 2024-09-19T08:01:42.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.184+0000 7fa1d9364640 1 -- 172.21.15.98:0/105604991 wait complete. 2024-09-19T08:01:42.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.184+0000 7fa1d9364640 1 Processor -- start 2024-09-19T08:01:42.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d9364640 1 -- start start 2024-09-19T08:01:42.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d9364640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1d4100460 0x7fa1d4196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:42.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d9364640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1d4100e10 0x7fa1d4196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:42.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d9364640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1d4105590 0x7fa1d419ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:42.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d9364640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d410fe10 con 0x7fa1d4105590 2024-09-19T08:01:42.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d9364640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d410fc90 con 0x7fa1d4100e10 2024-09-19T08:01:42.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d9364640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d410ff90 con 0x7fa1d4100460 2024-09-19T08:01:42.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1d4105590 0x7fa1d419ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:42.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1d4105590 0x7fa1d419ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50880/0 (socket says 172.21.15.98:50880) 2024-09-19T08:01:42.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d37fe640 1 -- 172.21.15.98:0/1538299776 learned_addr learned my addr 172.21.15.98:0/1538299776 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:42.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d2ffd640 1 --2- 172.21.15.98:0/1538299776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1d4100460 0x7fa1d4196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:42.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1ca7fc640 1 --2- 172.21.15.98:0/1538299776 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1d4100e10 0x7fa1d4196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:42.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d37fe640 1 -- 172.21.15.98:0/1538299776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1d4100460 msgr2=0x7fa1d4196330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:42.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d37fe640 1 --2- 172.21.15.98:0/1538299776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1d4100460 0x7fa1d4196330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:42.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d37fe640 1 -- 172.21.15.98:0/1538299776 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1d4100e10 msgr2=0x7fa1d4196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:42.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d37fe640 1 --2- 172.21.15.98:0/1538299776 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1d4100e10 0x7fa1d4196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:42.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.185+0000 7fa1d37fe640 1 -- 172.21.15.98:0/1538299776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d419b2e0 con 0x7fa1d4105590 2024-09-19T08:01:42.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.186+0000 7fa1d37fe640 1 --2- 172.21.15.98:0/1538299776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1d4105590 0x7fa1d419ac00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc005cc0 tx=0x7fa1cc032400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:42.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.186+0000 7fa1d0ff9640 1 -- 172.21.15.98:0/1538299776 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc041070 con 0x7fa1d4105590 2024-09-19T08:01:42.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.186+0000 7fa1d0ff9640 1 -- 172.21.15.98:0/1538299776 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1cc032a00 con 0x7fa1d4105590 2024-09-19T08:01:42.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.186+0000 7fa1d0ff9640 1 -- 172.21.15.98:0/1538299776 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc0047c0 con 0x7fa1d4105590 2024-09-19T08:01:42.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.186+0000 7fa1d9364640 1 -- 172.21.15.98:0/1538299776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d419b570 con 0x7fa1d4105590 2024-09-19T08:01:42.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.186+0000 7fa1d9364640 1 -- 172.21.15.98:0/1538299776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d41a2eb0 con 0x7fa1d4105590 2024-09-19T08:01:42.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.187+0000 7fa1d9364640 1 -- 172.21.15.98:0/1538299776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1d41008c0 con 0x7fa1d4105590 2024-09-19T08:01:42.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.188+0000 7fa1d0ff9640 1 -- 172.21.15.98:0/1538299776 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa1cc03a040 con 0x7fa1d4105590 2024-09-19T08:01:42.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.189+0000 7fa1d0ff9640 1 --2- 172.21.15.98:0/1538299776 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa1c00a3350 0x7fa1c00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:42.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.189+0000 7fa1d0ff9640 1 -- 172.21.15.98:0/1538299776 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fa1cc0f1f70 con 0x7fa1d4105590 2024-09-19T08:01:42.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.189+0000 7fa1d2ffd640 1 --2- 172.21.15.98:0/1538299776 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa1c00a3350 0x7fa1c00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:42.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.190+0000 7fa1d2ffd640 1 --2- 172.21.15.98:0/1538299776 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa1c00a3350 0x7fa1c00a5810 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4009990 tx=0x7fa1c4009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:42.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.194+0000 7fa1d0ff9640 1 -- 172.21.15.98:0/1538299776 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fa1cc0beae0 con 0x7fa1d4105590 2024-09-19T08:01:42.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.360+0000 7fa1d9364640 1 -- 172.21.15.98:0/1538299776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa1d4108770 con 0x7fa1d4105590 2024-09-19T08:01:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.362+0000 7fa1d0ff9640 1 -- 172.21.15.98:0/1538299776 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22754 (secure 0 0 0) 0x7fa1cc00a9c0 con 0x7fa1d4105590 2024-09-19T08:01:42.363 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:01:42.363 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":80,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","created":"2024-09-19T07:53:39.127398+0000","modified":"2024-09-19T08:01:26.714628+0000","last_up_change":"2024-09-19T08:00:22.467777+0000","last_in_change":"2024-09-19T07:59:58.538492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:56:43.773892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T08:01:07.693796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T08:01:09.518028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d01ce6-165d-493d-ae98-d8ad9e5d4134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6803","nonce":1719360533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6805","nonce":1719360533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6809","nonce":1719360533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6807","nonce":1719360533}]},"public_addr":"172.21.15.98:6803/1719360533","cluster_addr":"172.21.15.98:6805/1719360533","heartbeat_back_addr":"172.21.15.98:6809/1719360533","heartbeat_front_addr":"172.21.15.98:6807/1719360533","state":["exists","up"]},{"osd":1,"uuid":"34724bd5-0738-4473-b087-c09031c1ee2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036962626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036962626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036962626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036962626}]},"public_addr":"172.21.15.139:6801/4036962626","cluster_addr":"172.21.15.139:6803/4036962626","heartbeat_back_addr":"172.21.15.139:6807/4036962626","heartbeat_front_addr":"172.21.15.139:6805/4036962626","state":["exists","up"]},{"osd":2,"uuid":"16985ebe-0e32-42ee-bded-b7838e290afc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6801","nonce":2811412891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6803","nonce":2811412891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6807","nonce":2811412891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6805","nonce":2811412891}]},"public_addr":"172.21.15.162:6801/2811412891","cluster_addr":"172.21.15.162:6803/2811412891","heartbeat_back_addr":"172.21.15.162:6807/2811412891","heartbeat_front_addr":"172.21.15.162:6805/2811412891","state":["exists","up"]},{"osd":3,"uuid":"b140e466-3119-49cb-aec2-ba6ca4d10d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6811","nonce":1383392014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6813","nonce":1383392014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6817","nonce":1383392014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6815","nonce":1383392014}]},"public_addr":"172.21.15.98:6811/1383392014","cluster_addr":"172.21.15.98:6813/1383392014","heartbeat_back_addr":"172.21.15.98:6817/1383392014","heartbeat_front_addr":"172.21.15.98:6815/1383392014","state":["exists","up"]},{"osd":4,"uuid":"b4412813-df57-4ac6-8fb7-f57b69c74768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6809","nonce":116453520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6811","nonce":116453520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6815","nonce":116453520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6813","nonce":116453520}]},"public_addr":"172.21.15.139:6809/116453520","cluster_addr":"172.21.15.139:6811/116453520","heartbeat_back_addr":"172.21.15.139:6815/116453520","heartbeat_front_addr":"172.21.15.139:6813/116453520","state":["exists","up"]},{"osd":5,"uuid":"d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6809","nonce":3769530297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6811","nonce":3769530297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6815","nonce":3769530297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6813","nonce":3769530297}]},"public_addr":"172.21.15.162:6809/3769530297","cluster_addr":"172.21.15.162:6811/3769530297","heartbeat_back_addr":"172.21.15.162:6815/3769530297","heartbeat_front_addr":"172.21.15.162:6813/3769530297","state":["exists","up"]},{"osd":6,"uuid":"ba1c1590-6aea-40dc-90c8-e21b88896909","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6819","nonce":692373267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6821","nonce":692373267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6825","nonce":692373267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6823","nonce":692373267}]},"public_addr":"172.21.15.98:6819/692373267","cluster_addr":"172.21.15.98:6821/692373267","heartbeat_back_addr":"172.21.15.98:6825/692373267","heartbeat_front_addr":"172.21.15.98:6823/692373267","state":["exists","up"]},{"osd":7,"uuid":"eb9093ad-a068-4a63-ab87-0b50522cbc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6817","nonce":1963811650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6819","nonce":1963811650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6823","nonce":1963811650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6821","nonce":1963811650}]},"public_addr":"172.21.15.139:6817/1963811650","cluster_addr":"172.21.15.139:6819/1963811650","heartbeat_back_addr":"172.21.15.139:6823/1963811650","heartbeat_front_addr":"172.21.15.139:6821/1963811650","state":["exists","up"]},{"osd":8,"uuid":"31ca330a-fc45-466d-81c2-ef4386b3cc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6817","nonce":3607966458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6819","nonce":3607966458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6823","nonce":3607966458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6821","nonce":3607966458}]},"public_addr":"172.21.15.162:6817/3607966458","cluster_addr":"172.21.15.162:6819/3607966458","heartbeat_back_addr":"172.21.15.162:6823/3607966458","heartbeat_front_addr":"172.21.15.162:6821/3607966458","state":["exists","up"]},{"osd":9,"uuid":"9c4f216f-aef1-4367-9c58-b913b0c4f7db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6827","nonce":2050391244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6829","nonce":2050391244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6833","nonce":2050391244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6831","nonce":2050391244}]},"public_addr":"172.21.15.98:6827/2050391244","cluster_addr":"172.21.15.98:6829/2050391244","heartbeat_back_addr":"172.21.15.98:6833/2050391244","heartbeat_front_addr":"172.21.15.98:6831/2050391244","state":["exists","up"]},{"osd":10,"uuid":"03cc7df6-84ae-4d9a-a3d7-74c91e1744e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6825","nonce":2336761639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6827","nonce":2336761639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6831","nonce":2336761639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6829","nonce":2336761639}]},"public_addr":"172.21.15.139:6825/2336761639","cluster_addr":"172.21.15.139:6827/2336761639","heartbeat_back_addr":"172.21.15.139:6831/2336761639","heartbeat_front_addr":"172.21.15.139:6829/2336761639","state":["exists","up"]},{"osd":11,"uuid":"29e95498-4952-46d4-8fa6-6d8e2b862111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6825","nonce":1231011554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6827","nonce":1231011554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6831","nonce":1231011554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6829","nonce":1231011554}]},"public_addr":"172.21.15.162:6825/1231011554","cluster_addr":"172.21.15.162:6827/1231011554","heartbeat_back_addr":"172.21.15.162:6831/1231011554","heartbeat_front_addr":"172.21.15.162:6829/1231011554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:55:49.995244+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:15.655543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:40.331165+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:04.688881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:29.174966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:53.663161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:17.407551+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:42.584296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:06.317219+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:31.078625+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:55.689269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T08:00:19.659748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1},{"from":7,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/3842627255":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3838590070":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/1238193420":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6800/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6800/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3194864054":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3866031018":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/1497156658":"2024-09-20T07:54:02.418785+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T08:01:42.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.364+0000 7fa1d9364640 1 -- 172.21.15.98:0/1538299776 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa1c00a3350 msgr2=0x7fa1c00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:42.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.364+0000 7fa1d9364640 1 --2- 172.21.15.98:0/1538299776 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa1c00a3350 0x7fa1c00a5810 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4009990 tx=0x7fa1c4009110 comp rx=0 tx=0).stop 2024-09-19T08:01:42.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.364+0000 7fa1d9364640 1 -- 172.21.15.98:0/1538299776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1d4105590 msgr2=0x7fa1d419ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:42.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.364+0000 7fa1d9364640 1 --2- 172.21.15.98:0/1538299776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1d4105590 0x7fa1d419ac00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc005cc0 tx=0x7fa1cc032400 comp rx=0 tx=0).stop 2024-09-19T08:01:42.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.364+0000 7fa1d9364640 1 -- 172.21.15.98:0/1538299776 shutdown_connections 2024-09-19T08:01:42.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.364+0000 7fa1d9364640 1 --2- 172.21.15.98:0/1538299776 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa1c00a3350 0x7fa1c00a5810 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:42.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.364+0000 7fa1d9364640 1 --2- 172.21.15.98:0/1538299776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1d4105590 0x7fa1d419ac00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:42.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.364+0000 7fa1d9364640 1 --2- 172.21.15.98:0/1538299776 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1d4100e10 0x7fa1d4196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:42.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.364+0000 7fa1d9364640 1 --2- 172.21.15.98:0/1538299776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1d4100460 0x7fa1d4196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:42.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.364+0000 7fa1d9364640 1 -- 172.21.15.98:0/1538299776 >> 172.21.15.98:0/1538299776 conn(0x7fa1d40fbee0 msgr2=0x7fa1d40fd230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:42.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.364+0000 7fa1d9364640 1 -- 172.21.15.98:0/1538299776 shutdown_connections 2024-09-19T08:01:42.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:42.364+0000 7fa1d9364640 1 -- 172.21.15.98:0/1538299776 wait complete. 2024-09-19T08:01:42.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:42 smithi098 ceph-mon[31333]: pgmap v288: 97 pgs: 97 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 3.3 KiB/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T08:01:42.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:42 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1538299776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:43.004 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-19T08:01:43.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T08:01:43.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:42 smithi162 ceph-mon[35260]: pgmap v288: 97 pgs: 97 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 3.3 KiB/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T08:01:43.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:42 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1538299776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:43.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:42 smithi139 ceph-mon[34874]: pgmap v288: 97 pgs: 97 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 3.3 KiB/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T08:01:43.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:42 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1538299776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.718+0000 7fcdc0780640 1 -- 172.21.15.98:0/3826768756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8101de0 msgr2=0x7fcdb81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.718+0000 7fcdc0780640 1 --2- 172.21.15.98:0/3826768756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8101de0 0x7fcdb81021c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4009900 tx=0x7fcdb402a3f0 comp rx=0 tx=0).stop 2024-09-19T08:01:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.718+0000 7fcdc0780640 1 -- 172.21.15.98:0/3826768756 shutdown_connections 2024-09-19T08:01:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.718+0000 7fcdc0780640 1 --2- 172.21.15.98:0/3826768756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdb8106f10 0x7fcdb810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.718+0000 7fcdc0780640 1 --2- 172.21.15.98:0/3826768756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdb8102790 0x7fcdb81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.718+0000 7fcdc0780640 1 --2- 172.21.15.98:0/3826768756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8101de0 0x7fcdb81021c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.718+0000 7fcdc0780640 1 -- 172.21.15.98:0/3826768756 >> 172.21.15.98:0/3826768756 conn(0x7fcdb80fd8c0 msgr2=0x7fcdb80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.718+0000 7fcdc0780640 1 -- 172.21.15.98:0/3826768756 shutdown_connections 2024-09-19T08:01:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.718+0000 7fcdc0780640 1 -- 172.21.15.98:0/3826768756 wait complete. 2024-09-19T08:01:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdc0780640 1 Processor -- start 2024-09-19T08:01:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdc0780640 1 -- start start 2024-09-19T08:01:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdc0780640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8101de0 0x7fcdb8198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdc0780640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdb8102790 0x7fcdb8198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdc0780640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdb8106f10 0x7fcdb819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdc0780640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb8111f30 con 0x7fcdb8101de0 2024-09-19T08:01:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdc0780640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb8111db0 con 0x7fcdb8102790 2024-09-19T08:01:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdc0780640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb81120b0 con 0x7fcdb8106f10 2024-09-19T08:01:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdbe51c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8101de0 0x7fcdb8198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdbe51c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8101de0 0x7fcdb8198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50892/0 (socket says 172.21.15.98:50892) 2024-09-19T08:01:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdbe51c640 1 -- 172.21.15.98:0/568636460 learned_addr learned my addr 172.21.15.98:0/568636460 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdbed1d640 1 --2- 172.21.15.98:0/568636460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdb8106f10 0x7fcdb819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdbdd1b640 1 --2- 172.21.15.98:0/568636460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdb8102790 0x7fcdb8198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdbe51c640 1 -- 172.21.15.98:0/568636460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdb8106f10 msgr2=0x7fcdb819cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdbe51c640 1 --2- 172.21.15.98:0/568636460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdb8106f10 0x7fcdb819cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdbe51c640 1 -- 172.21.15.98:0/568636460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdb8102790 msgr2=0x7fcdb8198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdbe51c640 1 --2- 172.21.15.98:0/568636460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdb8102790 0x7fcdb8198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.719+0000 7fcdbe51c640 1 -- 172.21.15.98:0/568636460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb819d2c0 con 0x7fcdb8101de0 2024-09-19T08:01:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.720+0000 7fcdbe51c640 1 --2- 172.21.15.98:0/568636460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8101de0 0x7fcdb8198310 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4005ab0 tx=0x7fcdb40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.720+0000 7fcdaf7fe640 1 -- 172.21.15.98:0/568636460 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb403f030 con 0x7fcdb8101de0 2024-09-19T08:01:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.720+0000 7fcdc0780640 1 -- 172.21.15.98:0/568636460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb819d550 con 0x7fcdb8101de0 2024-09-19T08:01:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.720+0000 7fcdaf7fe640 1 -- 172.21.15.98:0/568636460 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdb4003ce0 con 0x7fcdb8101de0 2024-09-19T08:01:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.720+0000 7fcdaf7fe640 1 -- 172.21.15.98:0/568636460 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4035070 con 0x7fcdb8101de0 2024-09-19T08:01:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.720+0000 7fcdc0780640 1 -- 172.21.15.98:0/568636460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb81a4e90 con 0x7fcdb8101de0 2024-09-19T08:01:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.721+0000 7fcdc0780640 1 -- 172.21.15.98:0/568636460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdb81032c0 con 0x7fcdb8101de0 2024-09-19T08:01:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.722+0000 7fcdaf7fe640 1 -- 172.21.15.98:0/568636460 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fcdb4042030 con 0x7fcdb8101de0 2024-09-19T08:01:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.723+0000 7fcdaf7fe640 1 --2- 172.21.15.98:0/568636460 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fcda40a32d0 0x7fcda40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.723+0000 7fcdbdd1b640 1 --2- 172.21.15.98:0/568636460 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fcda40a32d0 0x7fcda40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.723+0000 7fcdaf7fe640 1 -- 172.21.15.98:0/568636460 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fcdb40f03c0 con 0x7fcdb8101de0 2024-09-19T08:01:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.723+0000 7fcdbdd1b640 1 --2- 172.21.15.98:0/568636460 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fcda40a32d0 0x7fcda40a5790 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcdb81088e0 tx=0x7fcda800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.727+0000 7fcdaf7fe640 1 -- 172.21.15.98:0/568636460 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fcdb40bcf30 con 0x7fcdb8101de0 2024-09-19T08:01:43.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.898+0000 7fcdc0780640 1 -- 172.21.15.98:0/568636460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcdb81a5060 con 0x7fcdb8101de0 2024-09-19T08:01:43.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.899+0000 7fcdaf7fe640 1 -- 172.21.15.98:0/568636460 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22754 (secure 0 0 0) 0x7fcdb40bc830 con 0x7fcdb8101de0 2024-09-19T08:01:43.900 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:01:43.901 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":80,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","created":"2024-09-19T07:53:39.127398+0000","modified":"2024-09-19T08:01:26.714628+0000","last_up_change":"2024-09-19T08:00:22.467777+0000","last_in_change":"2024-09-19T07:59:58.538492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:56:43.773892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T08:01:07.693796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T08:01:09.518028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d01ce6-165d-493d-ae98-d8ad9e5d4134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6803","nonce":1719360533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6805","nonce":1719360533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6809","nonce":1719360533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6807","nonce":1719360533}]},"public_addr":"172.21.15.98:6803/1719360533","cluster_addr":"172.21.15.98:6805/1719360533","heartbeat_back_addr":"172.21.15.98:6809/1719360533","heartbeat_front_addr":"172.21.15.98:6807/1719360533","state":["exists","up"]},{"osd":1,"uuid":"34724bd5-0738-4473-b087-c09031c1ee2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036962626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036962626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036962626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036962626}]},"public_addr":"172.21.15.139:6801/4036962626","cluster_addr":"172.21.15.139:6803/4036962626","heartbeat_back_addr":"172.21.15.139:6807/4036962626","heartbeat_front_addr":"172.21.15.139:6805/4036962626","state":["exists","up"]},{"osd":2,"uuid":"16985ebe-0e32-42ee-bded-b7838e290afc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6801","nonce":2811412891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6803","nonce":2811412891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6807","nonce":2811412891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6805","nonce":2811412891}]},"public_addr":"172.21.15.162:6801/2811412891","cluster_addr":"172.21.15.162:6803/2811412891","heartbeat_back_addr":"172.21.15.162:6807/2811412891","heartbeat_front_addr":"172.21.15.162:6805/2811412891","state":["exists","up"]},{"osd":3,"uuid":"b140e466-3119-49cb-aec2-ba6ca4d10d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6811","nonce":1383392014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6813","nonce":1383392014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6817","nonce":1383392014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6815","nonce":1383392014}]},"public_addr":"172.21.15.98:6811/1383392014","cluster_addr":"172.21.15.98:6813/1383392014","heartbeat_back_addr":"172.21.15.98:6817/1383392014","heartbeat_front_addr":"172.21.15.98:6815/1383392014","state":["exists","up"]},{"osd":4,"uuid":"b4412813-df57-4ac6-8fb7-f57b69c74768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6809","nonce":116453520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6811","nonce":116453520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6815","nonce":116453520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6813","nonce":116453520}]},"public_addr":"172.21.15.139:6809/116453520","cluster_addr":"172.21.15.139:6811/116453520","heartbeat_back_addr":"172.21.15.139:6815/116453520","heartbeat_front_addr":"172.21.15.139:6813/116453520","state":["exists","up"]},{"osd":5,"uuid":"d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6809","nonce":3769530297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6811","nonce":3769530297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6815","nonce":3769530297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6813","nonce":3769530297}]},"public_addr":"172.21.15.162:6809/3769530297","cluster_addr":"172.21.15.162:6811/3769530297","heartbeat_back_addr":"172.21.15.162:6815/3769530297","heartbeat_front_addr":"172.21.15.162:6813/3769530297","state":["exists","up"]},{"osd":6,"uuid":"ba1c1590-6aea-40dc-90c8-e21b88896909","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6819","nonce":692373267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6821","nonce":692373267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6825","nonce":692373267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6823","nonce":692373267}]},"public_addr":"172.21.15.98:6819/692373267","cluster_addr":"172.21.15.98:6821/692373267","heartbeat_back_addr":"172.21.15.98:6825/692373267","heartbeat_front_addr":"172.21.15.98:6823/692373267","state":["exists","up"]},{"osd":7,"uuid":"eb9093ad-a068-4a63-ab87-0b50522cbc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6817","nonce":1963811650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6819","nonce":1963811650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6823","nonce":1963811650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6821","nonce":1963811650}]},"public_addr":"172.21.15.139:6817/1963811650","cluster_addr":"172.21.15.139:6819/1963811650","heartbeat_back_addr":"172.21.15.139:6823/1963811650","heartbeat_front_addr":"172.21.15.139:6821/1963811650","state":["exists","up"]},{"osd":8,"uuid":"31ca330a-fc45-466d-81c2-ef4386b3cc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6817","nonce":3607966458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6819","nonce":3607966458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6823","nonce":3607966458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6821","nonce":3607966458}]},"public_addr":"172.21.15.162:6817/3607966458","cluster_addr":"172.21.15.162:6819/3607966458","heartbeat_back_addr":"172.21.15.162:6823/3607966458","heartbeat_front_addr":"172.21.15.162:6821/3607966458","state":["exists","up"]},{"osd":9,"uuid":"9c4f216f-aef1-4367-9c58-b913b0c4f7db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6827","nonce":2050391244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6829","nonce":2050391244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6833","nonce":2050391244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6831","nonce":2050391244}]},"public_addr":"172.21.15.98:6827/2050391244","cluster_addr":"172.21.15.98:6829/2050391244","heartbeat_back_addr":"172.21.15.98:6833/2050391244","heartbeat_front_addr":"172.21.15.98:6831/2050391244","state":["exists","up"]},{"osd":10,"uuid":"03cc7df6-84ae-4d9a-a3d7-74c91e1744e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6825","nonce":2336761639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6827","nonce":2336761639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6831","nonce":2336761639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6829","nonce":2336761639}]},"public_addr":"172.21.15.139:6825/2336761639","cluster_addr":"172.21.15.139:6827/2336761639","heartbeat_back_addr":"172.21.15.139:6831/2336761639","heartbeat_front_addr":"172.21.15.139:6829/2336761639","state":["exists","up"]},{"osd":11,"uuid":"29e95498-4952-46d4-8fa6-6d8e2b862111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6825","nonce":1231011554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6827","nonce":1231011554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6831","nonce":1231011554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6829","nonce":1231011554}]},"public_addr":"172.21.15.162:6825/1231011554","cluster_addr":"172.21.15.162:6827/1231011554","heartbeat_back_addr":"172.21.15.162:6831/1231011554","heartbeat_front_addr":"172.21.15.162:6829/1231011554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:55:49.995244+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:15.655543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:40.331165+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:04.688881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:29.174966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:53.663161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:17.407551+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:42.584296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:06.317219+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:31.078625+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:55.689269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T08:00:19.659748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1},{"from":7,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/3842627255":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3838590070":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/1238193420":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6800/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6800/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3194864054":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3866031018":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/1497156658":"2024-09-20T07:54:02.418785+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T08:01:43.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.901+0000 7fcdc0780640 1 -- 172.21.15.98:0/568636460 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fcda40a32d0 msgr2=0x7fcda40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:43.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.901+0000 7fcdc0780640 1 --2- 172.21.15.98:0/568636460 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fcda40a32d0 0x7fcda40a5790 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcdb81088e0 tx=0x7fcda800a2d0 comp rx=0 tx=0).stop 2024-09-19T08:01:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.901+0000 7fcdc0780640 1 -- 172.21.15.98:0/568636460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8101de0 msgr2=0x7fcdb8198310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.901+0000 7fcdc0780640 1 --2- 172.21.15.98:0/568636460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8101de0 0x7fcdb8198310 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4005ab0 tx=0x7fcdb40042f0 comp rx=0 tx=0).stop 2024-09-19T08:01:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.901+0000 7fcdc0780640 1 -- 172.21.15.98:0/568636460 shutdown_connections 2024-09-19T08:01:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.901+0000 7fcdc0780640 1 --2- 172.21.15.98:0/568636460 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fcda40a32d0 0x7fcda40a5790 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.901+0000 7fcdc0780640 1 --2- 172.21.15.98:0/568636460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdb8106f10 0x7fcdb819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.901+0000 7fcdc0780640 1 --2- 172.21.15.98:0/568636460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdb8102790 0x7fcdb8198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.901+0000 7fcdc0780640 1 --2- 172.21.15.98:0/568636460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8101de0 0x7fcdb8198310 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.901+0000 7fcdc0780640 1 -- 172.21.15.98:0/568636460 >> 172.21.15.98:0/568636460 conn(0x7fcdb80fd8c0 msgr2=0x7fcdb80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.902+0000 7fcdc0780640 1 -- 172.21.15.98:0/568636460 shutdown_connections 2024-09-19T08:01:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:43.902+0000 7fcdc0780640 1 -- 172.21.15.98:0/568636460 wait complete. 2024-09-19T08:01:44.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-19T08:01:44.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-19T08:01:44.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-19T08:01:44.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-19T08:01:44.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-19T08:01:44.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-19T08:01:44.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-19T08:01:44.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-19T08:01:44.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-19T08:01:44.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-19T08:01:44.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-19T08:01:44.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-19T08:01:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:45.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:44 smithi162 ceph-mon[35260]: pgmap v289: 97 pgs: 97 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-19T08:01:45.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:44 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/568636460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:45.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:44 smithi139 ceph-mon[34874]: pgmap v289: 97 pgs: 97 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-19T08:01:45.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:44 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/568636460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:45.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:44 smithi098 ceph-mon[31333]: pgmap v289: 97 pgs: 97 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-19T08:01:45.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:44 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/568636460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:01:45.215 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:45.218 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:45.221 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:45.222 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:45.224 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:45.503 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:45.505 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:45.505 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:45.506 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.166+0000 7fbe15036640 1 -- 172.21.15.98:0/3991472836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe10106f10 msgr2=0x7fbe1010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.166+0000 7fbe15036640 1 --2- 172.21.15.98:0/3991472836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe10106f10 0x7fbe1010ea90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009900 tx=0x7fbe0002a410 comp rx=0 tx=0).stop 2024-09-19T08:01:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.166+0000 7fbe15036640 1 -- 172.21.15.98:0/3991472836 shutdown_connections 2024-09-19T08:01:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.166+0000 7fbe15036640 1 --2- 172.21.15.98:0/3991472836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe10106f10 0x7fbe1010ea90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.166+0000 7fbe15036640 1 --2- 172.21.15.98:0/3991472836 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe10102790 0x7fbe101067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.166+0000 7fbe15036640 1 --2- 172.21.15.98:0/3991472836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe10101de0 0x7fbe101021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.166+0000 7fbe15036640 1 -- 172.21.15.98:0/3991472836 >> 172.21.15.98:0/3991472836 conn(0x7fbe100fd8c0 msgr2=0x7fbe100ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.166+0000 7fbe15036640 1 -- 172.21.15.98:0/3991472836 shutdown_connections 2024-09-19T08:01:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.166+0000 7fbe15036640 1 -- 172.21.15.98:0/3991472836 wait complete. 2024-09-19T08:01:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.167+0000 7fbe15036640 1 Processor -- start 2024-09-19T08:01:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.167+0000 7fbe15036640 1 -- start start 2024-09-19T08:01:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.167+0000 7fbe15036640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe10101de0 0x7fbe10198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.167+0000 7fbe15036640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe10102790 0x7fbe10198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.167+0000 7fbe15036640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe10106f10 0x7fbe1019ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.167+0000 7fbe15036640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbe10112010 con 0x7fbe10106f10 2024-09-19T08:01:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.167+0000 7fbe15036640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbe10111e90 con 0x7fbe10101de0 2024-09-19T08:01:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.167+0000 7fbe15036640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbe10112190 con 0x7fbe10102790 2024-09-19T08:01:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.167+0000 7fbe0f59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe10106f10 0x7fbe1019ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.167+0000 7fbe0f59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe10106f10 0x7fbe1019ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50908/0 (socket says 172.21.15.98:50908) 2024-09-19T08:01:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.167+0000 7fbe0f59e640 1 -- 172.21.15.98:0/2172581418 learned_addr learned my addr 172.21.15.98:0/2172581418 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.167+0000 7fbe0ed9d640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe10101de0 0x7fbe10198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.167+0000 7fbe0e59c640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe10102790 0x7fbe10198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.168+0000 7fbe0f59e640 1 -- 172.21.15.98:0/2172581418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe10102790 msgr2=0x7fbe10198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.168+0000 7fbe0f59e640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe10102790 0x7fbe10198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.168+0000 7fbe0f59e640 1 -- 172.21.15.98:0/2172581418 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe10101de0 msgr2=0x7fbe10198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.168+0000 7fbe0f59e640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe10101de0 0x7fbe10198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.168+0000 7fbe0f59e640 1 -- 172.21.15.98:0/2172581418 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe1019d520 con 0x7fbe10106f10 2024-09-19T08:01:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.168+0000 7fbe0e59c640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe10102790 0x7fbe10198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:01:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.168+0000 7fbe0ed9d640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe10101de0 0x7fbe10198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:01:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.168+0000 7fbe0f59e640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe10106f10 0x7fbe1019ce40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009230 tx=0x7fbe000042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.168+0000 7fbe07fff640 1 -- 172.21.15.98:0/2172581418 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0003f030 con 0x7fbe10106f10 2024-09-19T08:01:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.168+0000 7fbe15036640 1 -- 172.21.15.98:0/2172581418 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe1019d7b0 con 0x7fbe10106f10 2024-09-19T08:01:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.168+0000 7fbe07fff640 1 -- 172.21.15.98:0/2172581418 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe00003ce0 con 0x7fbe10106f10 2024-09-19T08:01:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.168+0000 7fbe07fff640 1 -- 172.21.15.98:0/2172581418 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe00035070 con 0x7fbe10106f10 2024-09-19T08:01:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.168+0000 7fbe15036640 1 -- 172.21.15.98:0/2172581418 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe101a50f0 con 0x7fbe10106f10 2024-09-19T08:01:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.169+0000 7fbe15036640 1 -- 172.21.15.98:0/2172581418 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbe10062980 con 0x7fbe10106f10 2024-09-19T08:01:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.170+0000 7fbe07fff640 1 -- 172.21.15.98:0/2172581418 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fbe00042030 con 0x7fbe10106f10 2024-09-19T08:01:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.171+0000 7fbe07fff640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbdfc0a32d0 0x7fbdfc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.171+0000 7fbe0ed9d640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbdfc0a32d0 0x7fbdfc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.171+0000 7fbe07fff640 1 -- 172.21.15.98:0/2172581418 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fbe000f0450 con 0x7fbe10106f10 2024-09-19T08:01:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.171+0000 7fbe07fff640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520] conn(0x7fbdfc0ace80 0x7fbdfc0af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.171+0000 7fbe07fff640 1 -- 172.21.15.98:0/2172581418 --> [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbe00001d30 con 0x7fbdfc0ace80 2024-09-19T08:01:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.171+0000 7fbe07fff640 1 -- 172.21.15.98:0/2172581418 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7fbe000f08a0 con 0x7fbe10106f10 2024-09-19T08:01:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.171+0000 7fbe0ed9d640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbdfc0a32d0 0x7fbdfc0a5790 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009fa0 tx=0x7fbe08009570 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.171+0000 7fbe0e59c640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520] conn(0x7fbdfc0ace80 0x7fbdfc0af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.172+0000 7fbe0e59c640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520] conn(0x7fbdfc0ace80 0x7fbdfc0af320 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.173+0000 7fbe07fff640 1 -- 172.21.15.98:0/2172581418 <== osd.4 v2:172.21.15.139:6808/116453520 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbe00001d30 con 0x7fbdfc0ace80 2024-09-19T08:01:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.186+0000 7fbe15036640 1 -- 172.21.15.98:0/2172581418 --> [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbe10062e00 con 0x7fbdfc0ace80 2024-09-19T08:01:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.186+0000 7fbe07fff640 1 -- 172.21.15.98:0/2172581418 <== osd.4 v2:172.21.15.139:6808/116453520 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbe10062e00 con 0x7fbdfc0ace80 2024-09-19T08:01:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.187+0000 7fbe15036640 1 -- 172.21.15.98:0/2172581418 >> [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520] conn(0x7fbdfc0ace80 msgr2=0x7fbdfc0af320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.187+0000 7fbe15036640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520] conn(0x7fbdfc0ace80 0x7fbdfc0af320 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.187+0000 7fbe15036640 1 -- 172.21.15.98:0/2172581418 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbdfc0a32d0 msgr2=0x7fbdfc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.187+0000 7fbe15036640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbdfc0a32d0 0x7fbdfc0a5790 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009fa0 tx=0x7fbe08009570 comp rx=0 tx=0).stop 2024-09-19T08:01:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.187+0000 7fbe15036640 1 -- 172.21.15.98:0/2172581418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe10106f10 msgr2=0x7fbe1019ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.187+0000 7fbe15036640 1 --2- 172.21.15.98:0/2172581418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe10106f10 0x7fbe1019ce40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009230 tx=0x7fbe000042f0 comp rx=0 tx=0).stop 2024-09-19T08:01:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.187+0000 7fbe0f59e640 1 -- 172.21.15.98:0/2172581418 reap_dead start 2024-09-19T08:01:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.187+0000 7fbe15036640 1 -- 172.21.15.98:0/2172581418 shutdown_connections 2024-09-19T08:01:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.187+0000 7fbe15036640 1 -- 172.21.15.98:0/2172581418 >> 172.21.15.98:0/2172581418 conn(0x7fbe100fd8c0 msgr2=0x7fbe100fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.187+0000 7fbe15036640 1 -- 172.21.15.98:0/2172581418 shutdown_connections 2024-09-19T08:01:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.187+0000 7fbe15036640 1 -- 172.21.15.98:0/2172581418 wait complete. 2024-09-19T08:01:46.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.358+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3537829811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6081002d0 msgr2=0x7fb6081006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:46.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.358+0000 7fb60d4b5640 1 --2- 172.21.15.98:0/3537829811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6081002d0 0x7fb6081006b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8009930 tx=0x7fb5f802a420 comp rx=0 tx=0).stop 2024-09-19T08:01:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.358+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3537829811 shutdown_connections 2024-09-19T08:01:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.358+0000 7fb60d4b5640 1 --2- 172.21.15.98:0/3537829811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb608106f30 0x7fb60810ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.358+0000 7fb60d4b5640 1 --2- 172.21.15.98:0/3537829811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb608100c80 0x7fb608106820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.358+0000 7fb60d4b5640 1 --2- 172.21.15.98:0/3537829811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6081002d0 0x7fb6081006b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.358+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3537829811 >> 172.21.15.98:0/3537829811 conn(0x7fb6080fbee0 msgr2=0x7fb6080fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.358+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3537829811 shutdown_connections 2024-09-19T08:01:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.358+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3537829811 wait complete. 2024-09-19T08:01:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.358+0000 7fb60d4b5640 1 Processor -- start 2024-09-19T08:01:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb60d4b5640 1 -- start start 2024-09-19T08:01:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb60d4b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6081002d0 0x7fb608198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb60d4b5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb608100c80 0x7fb608198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb60d4b5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb608106f30 0x7fb60819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb60d4b5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb608112060 con 0x7fb6081002d0 2024-09-19T08:01:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb60d4b5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb608111ee0 con 0x7fb608100c80 2024-09-19T08:01:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb60d4b5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb6081121e0 con 0x7fb608106f30 2024-09-19T08:01:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb606ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6081002d0 0x7fb608198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb606ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6081002d0 0x7fb608198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58648/0 (socket says 172.21.15.98:58648) 2024-09-19T08:01:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb606ffd640 1 -- 172.21.15.98:0/3507643963 learned_addr learned my addr 172.21.15.98:0/3507643963 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb6077fe640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb608106f30 0x7fb60819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb6067fc640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb608100c80 0x7fb608198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb606ffd640 1 -- 172.21.15.98:0/3507643963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb608106f30 msgr2=0x7fb60819cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb606ffd640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb608106f30 0x7fb60819cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb606ffd640 1 -- 172.21.15.98:0/3507643963 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb608100c80 msgr2=0x7fb608198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb606ffd640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb608100c80 0x7fb608198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb606ffd640 1 -- 172.21.15.98:0/3507643963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60819d600 con 0x7fb6081002d0 2024-09-19T08:01:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb6067fc640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb608100c80 0x7fb608198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:01:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.359+0000 7fb606ffd640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6081002d0 0x7fb608198650 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8009a50 tx=0x7fb5f8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.360+0000 7fb5effff640 1 -- 172.21.15.98:0/3507643963 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f803f030 con 0x7fb6081002d0 2024-09-19T08:01:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.360+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3507643963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb60819d890 con 0x7fb6081002d0 2024-09-19T08:01:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.360+0000 7fb5effff640 1 -- 172.21.15.98:0/3507643963 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5f8003ce0 con 0x7fb6081002d0 2024-09-19T08:01:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.360+0000 7fb5effff640 1 -- 172.21.15.98:0/3507643963 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f8035070 con 0x7fb6081002d0 2024-09-19T08:01:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.360+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3507643963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6081a51d0 con 0x7fb6081002d0 2024-09-19T08:01:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.360+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3507643963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb608062980 con 0x7fb6081002d0 2024-09-19T08:01:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.362+0000 7fb5effff640 1 -- 172.21.15.98:0/3507643963 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fb5f8042030 con 0x7fb6081002d0 2024-09-19T08:01:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.362+0000 7fb5effff640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb5f00a32d0 0x7fb5f00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.363+0000 7fb6067fc640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb5f00a32d0 0x7fb5f00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.363+0000 7fb5effff640 1 -- 172.21.15.98:0/3507643963 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fb5f80f0460 con 0x7fb6081002d0 2024-09-19T08:01:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.363+0000 7fb5effff640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267] conn(0x7fb5f00acd50 0x7fb5f00af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.363+0000 7fb5effff640 1 -- 172.21.15.98:0/3507643963 --> [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5f00af8c0 con 0x7fb5f00acd50 2024-09-19T08:01:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.363+0000 7fb5effff640 1 -- 172.21.15.98:0/3507643963 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7fb5f80f08b0 con 0x7fb6081002d0 2024-09-19T08:01:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.363+0000 7fb6077fe640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267] conn(0x7fb5f00acd50 0x7fb5f00af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.363+0000 7fb6067fc640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb5f00a32d0 0x7fb5f00a5790 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc006fd0 tx=0x7fb5fc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.363+0000 7fb6077fe640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267] conn(0x7fb5f00acd50 0x7fb5f00af1f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.364+0000 7fb5effff640 1 -- 172.21.15.98:0/3507643963 <== osd.6 v2:172.21.15.98:6818/692373267 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb5f00af8c0 con 0x7fb5f00acd50 2024-09-19T08:01:46.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.377+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3507643963 --> [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb608062e00 con 0x7fb5f00acd50 2024-09-19T08:01:46.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.378+0000 7fb5effff640 1 -- 172.21.15.98:0/3507643963 <== osd.6 v2:172.21.15.98:6818/692373267 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb608062e00 con 0x7fb5f00acd50 2024-09-19T08:01:46.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.378+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3507643963 >> [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267] conn(0x7fb5f00acd50 msgr2=0x7fb5f00af1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:46.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.378+0000 7fb60d4b5640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267] conn(0x7fb5f00acd50 0x7fb5f00af1f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:46.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.378+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3507643963 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb5f00a32d0 msgr2=0x7fb5f00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:46.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.378+0000 7fb60d4b5640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb5f00a32d0 0x7fb5f00a5790 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc006fd0 tx=0x7fb5fc008040 comp rx=0 tx=0).stop 2024-09-19T08:01:46.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.378+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3507643963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6081002d0 msgr2=0x7fb608198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:46.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.378+0000 7fb60d4b5640 1 --2- 172.21.15.98:0/3507643963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6081002d0 0x7fb608198650 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8009a50 tx=0x7fb5f8004270 comp rx=0 tx=0).stop 2024-09-19T08:01:46.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.378+0000 7fb6077fe640 1 -- 172.21.15.98:0/3507643963 reap_dead start 2024-09-19T08:01:46.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.378+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3507643963 shutdown_connections 2024-09-19T08:01:46.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.378+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3507643963 >> 172.21.15.98:0/3507643963 conn(0x7fb6080fbee0 msgr2=0x7fb6080fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:46.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.378+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3507643963 shutdown_connections 2024-09-19T08:01:46.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:46.378+0000 7fb60d4b5640 1 -- 172.21.15.98:0/3507643963 wait complete. 2024-09-19T08:01:46.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:46 smithi139 ceph-mon[34874]: pgmap v290: 97 pgs: 97 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-09-19T08:01:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:46 smithi162 ceph-mon[35260]: pgmap v290: 97 pgs: 97 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-09-19T08:01:47.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:46 smithi098 ceph-mon[31333]: pgmap v290: 97 pgs: 97 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-09-19T08:01:47.465 INFO:teuthology.orchestra.run.smithi098.stdout:133143986230 2024-09-19T08:01:47.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-19T08:01:48.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.103+0000 7f9115271640 1 -- 172.21.15.98:0/2461950439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f911010c2f0 msgr2=0x7f911010e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:48.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.103+0000 7f9115271640 1 --2- 172.21.15.98:0/2461950439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f911010c2f0 0x7f911010e790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9108009ca0 tx=0x7f910802a8c0 comp rx=0 tx=0).stop 2024-09-19T08:01:48.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.103+0000 7f9115271640 1 -- 172.21.15.98:0/2461950439 shutdown_connections 2024-09-19T08:01:48.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.103+0000 7f9115271640 1 --2- 172.21.15.98:0/2461950439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f911010c2f0 0x7f911010e790 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:48.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.103+0000 7f9115271640 1 --2- 172.21.15.98:0/2461950439 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f91101037a0 0x7f911010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:48.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.103+0000 7f9115271640 1 --2- 172.21.15.98:0/2461950439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9110102df0 0x7f91101031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:48.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.103+0000 7f9115271640 1 -- 172.21.15.98:0/2461950439 >> 172.21.15.98:0/2461950439 conn(0x7f91100fc7d0 msgr2=0x7f91100febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:48.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.103+0000 7f9115271640 1 -- 172.21.15.98:0/2461950439 shutdown_connections 2024-09-19T08:01:48.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.103+0000 7f9115271640 1 -- 172.21.15.98:0/2461950439 wait complete. 2024-09-19T08:01:48.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.103+0000 7f9115271640 1 Processor -- start 2024-09-19T08:01:48.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f9115271640 1 -- start start 2024-09-19T08:01:48.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f9115271640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9110102df0 0x7f9110198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:48.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f9115271640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91101037a0 0x7f9110198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:48.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f9115271640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f911010c2f0 0x7f911019cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:48.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f9115271640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9110112110 con 0x7f91101037a0 2024-09-19T08:01:48.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f9115271640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9110111f90 con 0x7f911010c2f0 2024-09-19T08:01:48.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f9115271640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9110112290 con 0x7f9110102df0 2024-09-19T08:01:48.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f910e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91101037a0 0x7f9110198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:48.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f910e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91101037a0 0x7f9110198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58686/0 (socket says 172.21.15.98:58686) 2024-09-19T08:01:48.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f910e7fc640 1 -- 172.21.15.98:0/3053684238 learned_addr learned my addr 172.21.15.98:0/3053684238 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:48.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f910effd640 1 --2- 172.21.15.98:0/3053684238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9110102df0 0x7f9110198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:48.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f910f7fe640 1 --2- 172.21.15.98:0/3053684238 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f911010c2f0 0x7f911019cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:48.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f910e7fc640 1 -- 172.21.15.98:0/3053684238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9110102df0 msgr2=0x7f9110198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:48.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f910e7fc640 1 --2- 172.21.15.98:0/3053684238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9110102df0 0x7f9110198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:48.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f910e7fc640 1 -- 172.21.15.98:0/3053684238 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f911010c2f0 msgr2=0x7f911019cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:48.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f910e7fc640 1 --2- 172.21.15.98:0/3053684238 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f911010c2f0 0x7f911019cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:48.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f910e7fc640 1 -- 172.21.15.98:0/3053684238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911019d4c0 con 0x7f91101037a0 2024-09-19T08:01:48.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f910e7fc640 1 --2- 172.21.15.98:0/3053684238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91101037a0 0x7f9110198a50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f910000c8c0 tx=0x7f910000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:48.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.104+0000 7f9107fff640 1 -- 172.21.15.98:0/3053684238 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9100007bb0 con 0x7f91101037a0 2024-09-19T08:01:48.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.105+0000 7f9115271640 1 -- 172.21.15.98:0/3053684238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f911019d7b0 con 0x7f91101037a0 2024-09-19T08:01:48.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.105+0000 7f9107fff640 1 -- 172.21.15.98:0/3053684238 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9100007d50 con 0x7f91101037a0 2024-09-19T08:01:48.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.105+0000 7f9107fff640 1 -- 172.21.15.98:0/3053684238 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9100018830 con 0x7f91101037a0 2024-09-19T08:01:48.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.105+0000 7f9115271640 1 -- 172.21.15.98:0/3053684238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9110100980 con 0x7f91101037a0 2024-09-19T08:01:48.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.106+0000 7f9115271640 1 -- 172.21.15.98:0/3053684238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f91100fedc0 con 0x7f91101037a0 2024-09-19T08:01:48.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.107+0000 7f9107fff640 1 -- 172.21.15.98:0/3053684238 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f91000189d0 con 0x7f91101037a0 2024-09-19T08:01:48.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.107+0000 7f9107fff640 1 --2- 172.21.15.98:0/3053684238 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f90fc0a76e0 0x7f90fc0a9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:48.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.107+0000 7f910effd640 1 --2- 172.21.15.98:0/3053684238 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f90fc0a76e0 0x7f90fc0a9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:48.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.107+0000 7f9107fff640 1 -- 172.21.15.98:0/3053684238 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f91000cff40 con 0x7f91101037a0 2024-09-19T08:01:48.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.108+0000 7f9107fff640 1 --2- 172.21.15.98:0/3053684238 >> [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533] conn(0x7f90fc0b1290 0x7f90fc0b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:48.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.108+0000 7f9107fff640 1 -- 172.21.15.98:0/3053684238 --> [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f90fc0b3e00 con 0x7f90fc0b1290 2024-09-19T08:01:48.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.108+0000 7f9107fff640 1 -- 172.21.15.98:0/3053684238 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f91000d0390 con 0x7f91101037a0 2024-09-19T08:01:48.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.108+0000 7f910f7fe640 1 --2- 172.21.15.98:0/3053684238 >> [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533] conn(0x7f90fc0b1290 0x7f90fc0b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:48.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.108+0000 7f910effd640 1 --2- 172.21.15.98:0/3053684238 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f90fc0a76e0 0x7f90fc0a9ba0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009bd0 tx=0x7f90f8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:48.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.108+0000 7f910f7fe640 1 --2- 172.21.15.98:0/3053684238 >> [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533] conn(0x7f90fc0b1290 0x7f90fc0b3730 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:48.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.109+0000 7f9107fff640 1 -- 172.21.15.98:0/3053684238 <== osd.0 v2:172.21.15.98:6802/1719360533 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f90fc0b3e00 con 0x7f90fc0b1290 2024-09-19T08:01:48.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.122+0000 7f9115271640 1 -- 172.21.15.98:0/3053684238 --> [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f911006a790 con 0x7f90fc0b1290 2024-09-19T08:01:48.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.123+0000 7f9107fff640 1 -- 172.21.15.98:0/3053684238 <== osd.0 v2:172.21.15.98:6802/1719360533 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f911006a790 con 0x7f90fc0b1290 2024-09-19T08:01:48.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.123+0000 7f9115271640 1 -- 172.21.15.98:0/3053684238 >> [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533] conn(0x7f90fc0b1290 msgr2=0x7f90fc0b3730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:48.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.123+0000 7f9115271640 1 --2- 172.21.15.98:0/3053684238 >> [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533] conn(0x7f90fc0b1290 0x7f90fc0b3730 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:48.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.123+0000 7f9115271640 1 -- 172.21.15.98:0/3053684238 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f90fc0a76e0 msgr2=0x7f90fc0a9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:48.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.123+0000 7f9115271640 1 --2- 172.21.15.98:0/3053684238 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f90fc0a76e0 0x7f90fc0a9ba0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009bd0 tx=0x7f90f8005f90 comp rx=0 tx=0).stop 2024-09-19T08:01:48.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.123+0000 7f9115271640 1 -- 172.21.15.98:0/3053684238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91101037a0 msgr2=0x7f9110198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:48.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.123+0000 7f9115271640 1 --2- 172.21.15.98:0/3053684238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91101037a0 0x7f9110198a50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f910000c8c0 tx=0x7f910000cd80 comp rx=0 tx=0).stop 2024-09-19T08:01:48.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.123+0000 7f910f7fe640 1 -- 172.21.15.98:0/3053684238 reap_dead start 2024-09-19T08:01:48.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.123+0000 7f9115271640 1 -- 172.21.15.98:0/3053684238 shutdown_connections 2024-09-19T08:01:48.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.123+0000 7f9115271640 1 -- 172.21.15.98:0/3053684238 >> 172.21.15.98:0/3053684238 conn(0x7f91100fc7d0 msgr2=0x7f91100fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:48.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.123+0000 7f9115271640 1 -- 172.21.15.98:0/3053684238 shutdown_connections 2024-09-19T08:01:48.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.123+0000 7f9115271640 1 -- 172.21.15.98:0/3053684238 wait complete. 2024-09-19T08:01:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.024+0000 7f1e40319640 1 -- 172.21.15.98:0/4019064689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e38102ef0 msgr2=0x7f1e3810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.024+0000 7f1e40319640 1 --2- 172.21.15.98:0/4019064689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e38102ef0 0x7f1e3810da40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1e30009ca0 tx=0x7f1e3002a950 comp rx=0 tx=0).stop 2024-09-19T08:01:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.024+0000 7f1e40319640 1 -- 172.21.15.98:0/4019064689 shutdown_connections 2024-09-19T08:01:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.024+0000 7f1e40319640 1 --2- 172.21.15.98:0/4019064689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e38102ef0 0x7f1e3810da40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.024+0000 7f1e40319640 1 --2- 172.21.15.98:0/4019064689 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e38102570 0x7f1e381029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.024+0000 7f1e40319640 1 --2- 172.21.15.98:0/4019064689 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e38100e20 0x7f1e38101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.024+0000 7f1e40319640 1 -- 172.21.15.98:0/4019064689 >> 172.21.15.98:0/4019064689 conn(0x7f1e380fc7d0 msgr2=0x7f1e380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.024+0000 7f1e40319640 1 -- 172.21.15.98:0/4019064689 shutdown_connections 2024-09-19T08:01:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.024+0000 7f1e40319640 1 -- 172.21.15.98:0/4019064689 wait complete. 2024-09-19T08:01:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.025+0000 7f1e40319640 1 Processor -- start 2024-09-19T08:01:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.025+0000 7f1e40319640 1 -- start start 2024-09-19T08:01:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.025+0000 7f1e40319640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e38100e20 0x7f1e38199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e40319640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e38102570 0x7f1e38199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e40319640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e38102ef0 0x7f1e381a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e40319640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e38110fd0 con 0x7f1e38102ef0 2024-09-19T08:01:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e40319640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1e38110e50 con 0x7f1e38100e20 2024-09-19T08:01:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e40319640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1e38111150 con 0x7f1e38102570 2024-09-19T08:01:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e3e8b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e38102ef0 0x7f1e381a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e3e8b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e38102ef0 0x7f1e381a42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58668/0 (socket says 172.21.15.98:58668) 2024-09-19T08:01:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e3e8b6640 1 -- 172.21.15.98:0/2597775402 learned_addr learned my addr 172.21.15.98:0/2597775402 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e3e0b5640 1 --2- 172.21.15.98:0/2597775402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e38100e20 0x7f1e38199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e3d8b4640 1 --2- 172.21.15.98:0/2597775402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e38102570 0x7f1e38199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e3e8b6640 1 -- 172.21.15.98:0/2597775402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e38102570 msgr2=0x7f1e38199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e3e8b6640 1 --2- 172.21.15.98:0/2597775402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e38102570 0x7f1e38199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e3e8b6640 1 -- 172.21.15.98:0/2597775402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e38100e20 msgr2=0x7f1e38199240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e3e8b6640 1 --2- 172.21.15.98:0/2597775402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e38100e20 0x7f1e38199240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e3e8b6640 1 -- 172.21.15.98:0/2597775402 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e381a4800 con 0x7f1e38102ef0 2024-09-19T08:01:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e3e8b6640 1 --2- 172.21.15.98:0/2597775402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e38102ef0 0x7f1e381a42c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1e300098d0 tx=0x7f1e30003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.026+0000 7f1e2f7fe640 1 -- 172.21.15.98:0/2597775402 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e30034040 con 0x7f1e38102ef0 2024-09-19T08:01:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.027+0000 7f1e2f7fe640 1 -- 172.21.15.98:0/2597775402 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e3002fe10 con 0x7f1e38102ef0 2024-09-19T08:01:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.027+0000 7f1e40319640 1 -- 172.21.15.98:0/2597775402 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e381a49d0 con 0x7f1e38102ef0 2024-09-19T08:01:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.027+0000 7f1e2f7fe640 1 -- 172.21.15.98:0/2597775402 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e30039450 con 0x7f1e38102ef0 2024-09-19T08:01:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.027+0000 7f1e40319640 1 -- 172.21.15.98:0/2597775402 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e381a4e20 con 0x7f1e38102ef0 2024-09-19T08:01:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.028+0000 7f1e40319640 1 -- 172.21.15.98:0/2597775402 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1e38107180 con 0x7f1e38102ef0 2024-09-19T08:01:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.029+0000 7f1e2f7fe640 1 -- 172.21.15.98:0/2597775402 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1e30037020 con 0x7f1e38102ef0 2024-09-19T08:01:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.030+0000 7f1e2f7fe640 1 --2- 172.21.15.98:0/2597775402 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e240a32d0 0x7f1e240a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.030+0000 7f1e3e0b5640 1 --2- 172.21.15.98:0/2597775402 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e240a32d0 0x7f1e240a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.030+0000 7f1e2f7fe640 1 -- 172.21.15.98:0/2597775402 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f1e300f0a80 con 0x7f1e38102ef0 2024-09-19T08:01:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.030+0000 7f1e2f7fe640 1 --2- 172.21.15.98:0/2597775402 >> [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297] conn(0x7f1e240ace80 0x7f1e240af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.030+0000 7f1e2f7fe640 1 -- 172.21.15.98:0/2597775402 --> [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e3002c030 con 0x7f1e240ace80 2024-09-19T08:01:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.030+0000 7f1e2f7fe640 1 -- 172.21.15.98:0/2597775402 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f1e300f52c0 con 0x7f1e38102ef0 2024-09-19T08:01:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.030+0000 7f1e3e0b5640 1 --2- 172.21.15.98:0/2597775402 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e240a32d0 0x7f1e240a5790 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1e340058c0 tx=0x7f1e3400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.030+0000 7f1e3d8b4640 1 --2- 172.21.15.98:0/2597775402 >> [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297] conn(0x7f1e240ace80 0x7f1e240af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.031+0000 7f1e3d8b4640 1 --2- 172.21.15.98:0/2597775402 >> [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297] conn(0x7f1e240ace80 0x7f1e240af320 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.032+0000 7f1e2f7fe640 1 -- 172.21.15.98:0/2597775402 <== osd.5 v2:172.21.15.162:6808/3769530297 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e3002c030 con 0x7f1e240ace80 2024-09-19T08:01:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.045+0000 7f1e40319640 1 -- 172.21.15.98:0/2597775402 --> [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1e381021b0 con 0x7f1e240ace80 2024-09-19T08:01:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.046+0000 7f1e2f7fe640 1 -- 172.21.15.98:0/2597775402 <== osd.5 v2:172.21.15.162:6808/3769530297 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1e381021b0 con 0x7f1e240ace80 2024-09-19T08:01:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.046+0000 7f1e40319640 1 -- 172.21.15.98:0/2597775402 >> [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297] conn(0x7f1e240ace80 msgr2=0x7f1e240af320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.046+0000 7f1e40319640 1 --2- 172.21.15.98:0/2597775402 >> [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297] conn(0x7f1e240ace80 0x7f1e240af320 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.046+0000 7f1e40319640 1 -- 172.21.15.98:0/2597775402 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e240a32d0 msgr2=0x7f1e240a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.046+0000 7f1e40319640 1 --2- 172.21.15.98:0/2597775402 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e240a32d0 0x7f1e240a5790 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1e340058c0 tx=0x7f1e3400a2d0 comp rx=0 tx=0).stop 2024-09-19T08:01:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.046+0000 7f1e40319640 1 -- 172.21.15.98:0/2597775402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e38102ef0 msgr2=0x7f1e381a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.046+0000 7f1e40319640 1 --2- 172.21.15.98:0/2597775402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e38102ef0 0x7f1e381a42c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1e300098d0 tx=0x7f1e30003f10 comp rx=0 tx=0).stop 2024-09-19T08:01:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.046+0000 7f1e3e8b6640 1 -- 172.21.15.98:0/2597775402 reap_dead start 2024-09-19T08:01:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.046+0000 7f1e40319640 1 -- 172.21.15.98:0/2597775402 shutdown_connections 2024-09-19T08:01:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.046+0000 7f1e40319640 1 -- 172.21.15.98:0/2597775402 >> 172.21.15.98:0/2597775402 conn(0x7f1e380fc7d0 msgr2=0x7f1e38106760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.046+0000 7f1e40319640 1 -- 172.21.15.98:0/2597775402 shutdown_connections 2024-09-19T08:01:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.046+0000 7f1e40319640 1 -- 172.21.15.98:0/2597775402 wait complete. 2024-09-19T08:01:48.774 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:48.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:48 smithi098 ceph-mon[31333]: pgmap v291: 97 pgs: 97 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-19T08:01:49.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:48 smithi162 ceph-mon[35260]: pgmap v291: 97 pgs: 97 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-19T08:01:49.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:48 smithi139 ceph-mon[34874]: pgmap v291: 97 pgs: 97 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-19T08:01:49.869 INFO:teuthology.orchestra.run.smithi098.stdout:34359738442 2024-09-19T08:01:49.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-19T08:01:50.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.298+0000 7f31048c5640 1 -- 172.21.15.98:0/1405555118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f310010e430 msgr2=0x7f3100110820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:50.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.298+0000 7f31048c5640 1 --2- 172.21.15.98:0/1405555118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f310010e430 0x7f3100110820 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f30f8009ca0 tx=0x7f30f802a950 comp rx=0 tx=0).stop 2024-09-19T08:01:50.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.298+0000 7f31048c5640 1 -- 172.21.15.98:0/1405555118 shutdown_connections 2024-09-19T08:01:50.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.298+0000 7f31048c5640 1 --2- 172.21.15.98:0/1405555118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f310010e430 0x7f3100110820 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:50.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.298+0000 7f31048c5640 1 --2- 172.21.15.98:0/1405555118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3100074ee0 0x7f310010def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:50.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.298+0000 7f31048c5640 1 --2- 172.21.15.98:0/1405555118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31000745c0 0x7f31000749a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:50.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.298+0000 7f31048c5640 1 -- 172.21.15.98:0/1405555118 >> 172.21.15.98:0/1405555118 conn(0x7f31000fd8f0 msgr2=0x7f31000ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:50.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.298+0000 7f31048c5640 1 -- 172.21.15.98:0/1405555118 shutdown_connections 2024-09-19T08:01:50.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.298+0000 7f31048c5640 1 -- 172.21.15.98:0/1405555118 wait complete. 2024-09-19T08:01:50.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.299+0000 7f31048c5640 1 Processor -- start 2024-09-19T08:01:50.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.299+0000 7f31048c5640 1 -- start start 2024-09-19T08:01:50.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.299+0000 7f31048c5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31000745c0 0x7f310019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:50.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.299+0000 7f31048c5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3100074ee0 0x7f310019ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:50.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.299+0000 7f31048c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f310010e430 0x7f310019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:50.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.299+0000 7f31048c5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31001141e0 con 0x7f310010e430 2024-09-19T08:01:50.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.299+0000 7f31048c5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3100114060 con 0x7f3100074ee0 2024-09-19T08:01:50.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.299+0000 7f31048c5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3100114360 con 0x7f31000745c0 2024-09-19T08:01:50.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30fed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f310010e430 0x7f310019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:50.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30fed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f310010e430 0x7f310019ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58718/0 (socket says 172.21.15.98:58718) 2024-09-19T08:01:50.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30fed9d640 1 -- 172.21.15.98:0/570449004 learned_addr learned my addr 172.21.15.98:0/570449004 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:50.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30fe59c640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31000745c0 0x7f310019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:50.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30fdd9b640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3100074ee0 0x7f310019ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:50.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30fed9d640 1 -- 172.21.15.98:0/570449004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31000745c0 msgr2=0x7f310019a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:50.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30fed9d640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31000745c0 0x7f310019a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:50.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30fed9d640 1 -- 172.21.15.98:0/570449004 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3100074ee0 msgr2=0x7f310019ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:50.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30fed9d640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3100074ee0 0x7f310019ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:50.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30fed9d640 1 -- 172.21.15.98:0/570449004 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f310019f670 con 0x7f310010e430 2024-09-19T08:01:50.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30fdd9b640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3100074ee0 0x7f310019ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:01:50.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30fed9d640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f310010e430 0x7f310019ef90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f30f802ae70 tx=0x7f30f802f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:50.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30ef7fe640 1 -- 172.21.15.98:0/570449004 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f8034040 con 0x7f310010e430 2024-09-19T08:01:50.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30ef7fe640 1 -- 172.21.15.98:0/570449004 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30f8030920 con 0x7f310010e430 2024-09-19T08:01:50.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f31048c5640 1 -- 172.21.15.98:0/570449004 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f310019f900 con 0x7f310010e430 2024-09-19T08:01:50.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f30ef7fe640 1 -- 172.21.15.98:0/570449004 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f8045480 con 0x7f310010e430 2024-09-19T08:01:50.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.300+0000 7f31048c5640 1 -- 172.21.15.98:0/570449004 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31001a7240 con 0x7f310010e430 2024-09-19T08:01:50.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.301+0000 7f31048c5640 1 -- 172.21.15.98:0/570449004 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3100062980 con 0x7f310010e430 2024-09-19T08:01:50.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.302+0000 7f30ef7fe640 1 -- 172.21.15.98:0/570449004 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f30f8037020 con 0x7f310010e430 2024-09-19T08:01:50.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.303+0000 7f30ef7fe640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30e80a32d0 0x7f30e80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:50.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.303+0000 7f30fe59c640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30e80a32d0 0x7f30e80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:50.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.303+0000 7f30ef7fe640 1 -- 172.21.15.98:0/570449004 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f30f80f0cb0 con 0x7f310010e430 2024-09-19T08:01:50.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.303+0000 7f30ef7fe640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891] conn(0x7f30e80ace80 0x7f30e80af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:50.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.303+0000 7f30ef7fe640 1 -- 172.21.15.98:0/570449004 --> [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f30e80af9f0 con 0x7f30e80ace80 2024-09-19T08:01:50.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.303+0000 7f30ef7fe640 1 -- 172.21.15.98:0/570449004 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f30f80f1100 con 0x7f310010e430 2024-09-19T08:01:50.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.303+0000 7f30fe59c640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30e80a32d0 0x7f30e80a5790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f30f0005fd0 tx=0x7f30f0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:50.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.303+0000 7f30fdd9b640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891] conn(0x7f30e80ace80 0x7f30e80af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:50.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.304+0000 7f30fdd9b640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891] conn(0x7f30e80ace80 0x7f30e80af320 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:50.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.305+0000 7f30ef7fe640 1 -- 172.21.15.98:0/570449004 <== osd.2 v2:172.21.15.162:6800/2811412891 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30e80af9f0 con 0x7f30e80ace80 2024-09-19T08:01:50.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.318+0000 7f31048c5640 1 -- 172.21.15.98:0/570449004 --> [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3100062e00 con 0x7f30e80ace80 2024-09-19T08:01:50.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.318+0000 7f30ef7fe640 1 -- 172.21.15.98:0/570449004 <== osd.2 v2:172.21.15.162:6800/2811412891 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3100062e00 con 0x7f30e80ace80 2024-09-19T08:01:50.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.319+0000 7f31048c5640 1 -- 172.21.15.98:0/570449004 >> [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891] conn(0x7f30e80ace80 msgr2=0x7f30e80af320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:50.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.319+0000 7f31048c5640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891] conn(0x7f30e80ace80 0x7f30e80af320 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:50.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.319+0000 7f31048c5640 1 -- 172.21.15.98:0/570449004 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30e80a32d0 msgr2=0x7f30e80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:50.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.319+0000 7f31048c5640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f30e80a32d0 0x7f30e80a5790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f30f0005fd0 tx=0x7f30f0005850 comp rx=0 tx=0).stop 2024-09-19T08:01:50.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.319+0000 7f31048c5640 1 -- 172.21.15.98:0/570449004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f310010e430 msgr2=0x7f310019ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:50.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.319+0000 7f31048c5640 1 --2- 172.21.15.98:0/570449004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f310010e430 0x7f310019ef90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f30f802ae70 tx=0x7f30f802f5b0 comp rx=0 tx=0).stop 2024-09-19T08:01:50.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.319+0000 7f30fed9d640 1 -- 172.21.15.98:0/570449004 reap_dead start 2024-09-19T08:01:50.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.319+0000 7f31048c5640 1 -- 172.21.15.98:0/570449004 shutdown_connections 2024-09-19T08:01:50.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.319+0000 7f31048c5640 1 -- 172.21.15.98:0/570449004 >> 172.21.15.98:0/570449004 conn(0x7f31000fd8f0 msgr2=0x7f31000fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:50.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.319+0000 7f31048c5640 1 -- 172.21.15.98:0/570449004 shutdown_connections 2024-09-19T08:01:50.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:50.319+0000 7f31048c5640 1 -- 172.21.15.98:0/570449004 wait complete. 2024-09-19T08:01:50.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:50 smithi098 ceph-mon[31333]: pgmap v292: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-19T08:01:51.086 INFO:teuthology.orchestra.run.smithi098.stdout:68719476801 2024-09-19T08:01:51.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-19T08:01:51.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:50 smithi162 ceph-mon[35260]: pgmap v292: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-19T08:01:51.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:50 smithi139 ceph-mon[34874]: pgmap v292: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-19T08:01:51.379 INFO:teuthology.orchestra.run.smithi098.stdout:184683593773 2024-09-19T08:01:51.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-19T08:01:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.563+0000 7f06b6d83640 1 -- 172.21.15.98:0/2752014148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b0103650 msgr2=0x7f06b010bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.563+0000 7f06b6d83640 1 --2- 172.21.15.98:0/2752014148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b0103650 0x7f06b010bb40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f069c009900 tx=0x7f069c02a480 comp rx=0 tx=0).stop 2024-09-19T08:01:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.563+0000 7f06b6d83640 1 -- 172.21.15.98:0/2752014148 shutdown_connections 2024-09-19T08:01:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.563+0000 7f06b6d83640 1 --2- 172.21.15.98:0/2752014148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06b010c160 0x7f06b010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.563+0000 7f06b6d83640 1 --2- 172.21.15.98:0/2752014148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b0103650 0x7f06b010bb40 secure :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f069c009900 tx=0x7f069c02a480 comp rx=0 tx=0).stop 2024-09-19T08:01:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.563+0000 7f06b6d83640 1 --2- 172.21.15.98:0/2752014148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f06b0102d30 0x7f06b0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.563+0000 7f06b6d83640 1 -- 172.21.15.98:0/2752014148 >> 172.21.15.98:0/2752014148 conn(0x7f06b00fc7d0 msgr2=0x7f06b00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.563+0000 7f06b6d83640 1 -- 172.21.15.98:0/2752014148 shutdown_connections 2024-09-19T08:01:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.563+0000 7f06b6d83640 1 -- 172.21.15.98:0/2752014148 wait complete. 2024-09-19T08:01:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.564+0000 7f06b6d83640 1 Processor -- start 2024-09-19T08:01:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.564+0000 7f06b6d83640 1 -- start start 2024-09-19T08:01:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.564+0000 7f06b6d83640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b0102d30 0x7f06b01985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.564+0000 7f06b6d83640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06b010c160 0x7f06b0198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.564+0000 7f06b6d83640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f06b019cea0 0x7f06b019d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.564+0000 7f06b6d83640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f06b0111f60 con 0x7f06b0102d30 2024-09-19T08:01:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.564+0000 7f06b4b1f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b0102d30 0x7f06b01985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.564+0000 7f06b4b1f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b0102d30 0x7f06b01985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58740/0 (socket says 172.21.15.98:58740) 2024-09-19T08:01:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.564+0000 7f06b4b1f640 1 -- 172.21.15.98:0/1017089519 learned_addr learned my addr 172.21.15.98:0/1017089519 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.564+0000 7f06b6d83640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f06b0111de0 con 0x7f06b019cea0 2024-09-19T08:01:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.564+0000 7f06b6d83640 1 -- 172.21.15.98:0/1017089519 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f06b01120e0 con 0x7f06b010c160 2024-09-19T08:01:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.564+0000 7f06affff640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06b010c160 0x7f06b0198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.564+0000 7f06b5320640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f06b019cea0 0x7f06b019d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.565+0000 7f06b4b1f640 1 -- 172.21.15.98:0/1017089519 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06b010c160 msgr2=0x7f06b0198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.565+0000 7f06b4b1f640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06b010c160 0x7f06b0198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.565+0000 7f06b4b1f640 1 -- 172.21.15.98:0/1017089519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f06b019cea0 msgr2=0x7f06b019d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.565+0000 7f06b4b1f640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f06b019cea0 0x7f06b019d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.565+0000 7f06b4b1f640 1 -- 172.21.15.98:0/1017089519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b019d960 con 0x7f06b0102d30 2024-09-19T08:01:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.565+0000 7f06b5320640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f06b019cea0 0x7f06b019d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:01:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.565+0000 7f06b4b1f640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b0102d30 0x7f06b01985d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f06a400c920 tx=0x7f06a400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.565+0000 7f06adffb640 1 -- 172.21.15.98:0/1017089519 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4007c70 con 0x7f06b0102d30 2024-09-19T08:01:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.565+0000 7f06b6d83640 1 -- 172.21.15.98:0/1017089519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b019dc50 con 0x7f06b0102d30 2024-09-19T08:01:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.565+0000 7f06b6d83640 1 -- 172.21.15.98:0/1017089519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b01a5480 con 0x7f06b0102d30 2024-09-19T08:01:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.565+0000 7f06adffb640 1 -- 172.21.15.98:0/1017089519 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06a4007e10 con 0x7f06b0102d30 2024-09-19T08:01:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.565+0000 7f06adffb640 1 -- 172.21.15.98:0/1017089519 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4018890 con 0x7f06b0102d30 2024-09-19T08:01:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.566+0000 7f06b6d83640 1 -- 172.21.15.98:0/1017089519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f06b0062980 con 0x7f06b0102d30 2024-09-19T08:01:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.568+0000 7f06adffb640 1 -- 172.21.15.98:0/1017089519 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f06a400f450 con 0x7f06b0102d30 2024-09-19T08:01:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.568+0000 7f06adffb640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f06a00a3350 0x7f06a00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.568+0000 7f06affff640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f06a00a3350 0x7f06a00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.568+0000 7f06adffb640 1 -- 172.21.15.98:0/1017089519 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f06a40d01a0 con 0x7f06b0102d30 2024-09-19T08:01:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.568+0000 7f06adffb640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244] conn(0x7f06a00acf00 0x7f06a00af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.568+0000 7f06adffb640 1 -- 172.21.15.98:0/1017089519 --> [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06a00afa70 con 0x7f06a00acf00 2024-09-19T08:01:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.568+0000 7f06adffb640 1 -- 172.21.15.98:0/1017089519 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f06a40d05f0 con 0x7f06b0102d30 2024-09-19T08:01:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.569+0000 7f06b5320640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244] conn(0x7f06a00acf00 0x7f06a00af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.569+0000 7f06affff640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f06a00a3350 0x7f06a00a5810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f069c005940 tx=0x7f069c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.569+0000 7f06b5320640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244] conn(0x7f06a00acf00 0x7f06a00af3a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.569+0000 7f06adffb640 1 -- 172.21.15.98:0/1017089519 <== osd.9 v2:172.21.15.98:6826/2050391244 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06a00afa70 con 0x7f06a00acf00 2024-09-19T08:01:51.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.583+0000 7f06b6d83640 1 -- 172.21.15.98:0/1017089519 --> [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f06b0062e00 con 0x7f06a00acf00 2024-09-19T08:01:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.584+0000 7f06adffb640 1 -- 172.21.15.98:0/1017089519 <== osd.9 v2:172.21.15.98:6826/2050391244 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f06b0062e00 con 0x7f06a00acf00 2024-09-19T08:01:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.584+0000 7f06b6d83640 1 -- 172.21.15.98:0/1017089519 >> [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244] conn(0x7f06a00acf00 msgr2=0x7f06a00af3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.584+0000 7f06b6d83640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244] conn(0x7f06a00acf00 0x7f06a00af3a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.584+0000 7f06b6d83640 1 -- 172.21.15.98:0/1017089519 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f06a00a3350 msgr2=0x7f06a00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.584+0000 7f06b6d83640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f06a00a3350 0x7f06a00a5810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f069c005940 tx=0x7f069c0058d0 comp rx=0 tx=0).stop 2024-09-19T08:01:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.584+0000 7f06b6d83640 1 -- 172.21.15.98:0/1017089519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b0102d30 msgr2=0x7f06b01985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.584+0000 7f06b6d83640 1 --2- 172.21.15.98:0/1017089519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b0102d30 0x7f06b01985d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f06a400c920 tx=0x7f06a400cde0 comp rx=0 tx=0).stop 2024-09-19T08:01:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.584+0000 7f06b5320640 1 -- 172.21.15.98:0/1017089519 reap_dead start 2024-09-19T08:01:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.585+0000 7f06b6d83640 1 -- 172.21.15.98:0/1017089519 shutdown_connections 2024-09-19T08:01:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.585+0000 7f06b6d83640 1 -- 172.21.15.98:0/1017089519 >> 172.21.15.98:0/1017089519 conn(0x7f06b00fc7d0 msgr2=0x7f06b00fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.585+0000 7f06b6d83640 1 -- 172.21.15.98:0/1017089519 shutdown_connections 2024-09-19T08:01:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:51.585+0000 7f06b6d83640 1 -- 172.21.15.98:0/1017089519 wait complete. 2024-09-19T08:01:52.325 INFO:teuthology.orchestra.run.smithi098.stdout:261993005087 2024-09-19T08:01:52.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-19T08:01:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.470+0000 7f1e2e101640 1 -- 172.21.15.98:0/2079310894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e28102570 msgr2=0x7f1e281029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.470+0000 7f1e2e101640 1 --2- 172.21.15.98:0/2079310894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e28102570 0x7f1e281029b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c009900 tx=0x7f1e1c02a480 comp rx=0 tx=0).stop 2024-09-19T08:01:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.470+0000 7f1e2e101640 1 -- 172.21.15.98:0/2079310894 shutdown_connections 2024-09-19T08:01:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.470+0000 7f1e2e101640 1 --2- 172.21.15.98:0/2079310894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e28102ef0 0x7f1e2810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.470+0000 7f1e2e101640 1 --2- 172.21.15.98:0/2079310894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e28102570 0x7f1e281029b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.470+0000 7f1e2e101640 1 --2- 172.21.15.98:0/2079310894 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e28100e20 0x7f1e28101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.470+0000 7f1e2e101640 1 -- 172.21.15.98:0/2079310894 >> 172.21.15.98:0/2079310894 conn(0x7f1e280fc7d0 msgr2=0x7f1e280febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.470+0000 7f1e2e101640 1 -- 172.21.15.98:0/2079310894 shutdown_connections 2024-09-19T08:01:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.470+0000 7f1e2e101640 1 -- 172.21.15.98:0/2079310894 wait complete. 2024-09-19T08:01:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.471+0000 7f1e2e101640 1 Processor -- start 2024-09-19T08:01:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.471+0000 7f1e2e101640 1 -- start start 2024-09-19T08:01:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.471+0000 7f1e2e101640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e28100e20 0x7f1e28199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.471+0000 7f1e2e101640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e28102570 0x7f1e28199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.471+0000 7f1e2e101640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e28102ef0 0x7f1e281a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.471+0000 7f1e2e101640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e28110f80 con 0x7f1e28102570 2024-09-19T08:01:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.471+0000 7f1e2e101640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1e28110e00 con 0x7f1e28100e20 2024-09-19T08:01:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.471+0000 7f1e2e101640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1e28111100 con 0x7f1e28102ef0 2024-09-19T08:01:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.471+0000 7f1e26ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e28102570 0x7f1e28199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.471+0000 7f1e26ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e28102570 0x7f1e28199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58754/0 (socket says 172.21.15.98:58754) 2024-09-19T08:01:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.471+0000 7f1e26ffd640 1 -- 172.21.15.98:0/1011208500 learned_addr learned my addr 172.21.15.98:0/1011208500 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.471+0000 7f1e277fe640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e28100e20 0x7f1e28199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.471+0000 7f1e27fff640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e28102ef0 0x7f1e281a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.472+0000 7f1e26ffd640 1 -- 172.21.15.98:0/1011208500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e28102ef0 msgr2=0x7f1e281a43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.472+0000 7f1e26ffd640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e28102ef0 0x7f1e281a43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.472+0000 7f1e26ffd640 1 -- 172.21.15.98:0/1011208500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e28100e20 msgr2=0x7f1e28199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.472+0000 7f1e26ffd640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e28100e20 0x7f1e28199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.472+0000 7f1e26ffd640 1 -- 172.21.15.98:0/1011208500 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e281a48f0 con 0x7f1e28102570 2024-09-19T08:01:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.472+0000 7f1e27fff640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e28102ef0 0x7f1e281a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:01:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.472+0000 7f1e277fe640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e28100e20 0x7f1e28199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:01:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.472+0000 7f1e26ffd640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e28102570 0x7f1e28199870 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c006010 tx=0x7f1e1c003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.472+0000 7f1e24ff9640 1 -- 172.21.15.98:0/1011208500 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c03f070 con 0x7f1e28102570 2024-09-19T08:01:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.472+0000 7f1e2e101640 1 -- 172.21.15.98:0/1011208500 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e281a4ac0 con 0x7f1e28102570 2024-09-19T08:01:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.472+0000 7f1e24ff9640 1 -- 172.21.15.98:0/1011208500 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e1c031e30 con 0x7f1e28102570 2024-09-19T08:01:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.472+0000 7f1e24ff9640 1 -- 172.21.15.98:0/1011208500 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c035030 con 0x7f1e28102570 2024-09-19T08:01:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.472+0000 7f1e2e101640 1 -- 172.21.15.98:0/1011208500 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e281a4f70 con 0x7f1e28102570 2024-09-19T08:01:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.473+0000 7f1e2e101640 1 -- 172.21.15.98:0/1011208500 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1e28107200 con 0x7f1e28102570 2024-09-19T08:01:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.474+0000 7f1e24ff9640 1 -- 172.21.15.98:0/1011208500 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1e1c042070 con 0x7f1e28102570 2024-09-19T08:01:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.475+0000 7f1e24ff9640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e100a32d0 0x7f1e100a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.475+0000 7f1e277fe640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e100a32d0 0x7f1e100a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.475+0000 7f1e24ff9640 1 -- 172.21.15.98:0/1011208500 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f1e1c0f0aa0 con 0x7f1e28102570 2024-09-19T08:01:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.475+0000 7f1e24ff9640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650] conn(0x7f1e100ace80 0x7f1e100af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.475+0000 7f1e24ff9640 1 -- 172.21.15.98:0/1011208500 --> [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e1c038e90 con 0x7f1e100ace80 2024-09-19T08:01:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.475+0000 7f1e24ff9640 1 -- 172.21.15.98:0/1011208500 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f1e1c0f0ef0 con 0x7f1e28102570 2024-09-19T08:01:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.475+0000 7f1e277fe640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e100a32d0 0x7f1e100a5790 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1e180058c0 tx=0x7f1e18005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.475+0000 7f1e27fff640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650] conn(0x7f1e100ace80 0x7f1e100af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.476+0000 7f1e27fff640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650] conn(0x7f1e100ace80 0x7f1e100af320 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.477+0000 7f1e24ff9640 1 -- 172.21.15.98:0/1011208500 <== osd.7 v2:172.21.15.139:6816/1963811650 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e1c038e90 con 0x7f1e100ace80 2024-09-19T08:01:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.490+0000 7f1e2e101640 1 -- 172.21.15.98:0/1011208500 --> [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1e281021b0 con 0x7f1e100ace80 2024-09-19T08:01:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.491+0000 7f1e24ff9640 1 -- 172.21.15.98:0/1011208500 <== osd.7 v2:172.21.15.139:6816/1963811650 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1e281021b0 con 0x7f1e100ace80 2024-09-19T08:01:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.491+0000 7f1e2e101640 1 -- 172.21.15.98:0/1011208500 >> [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650] conn(0x7f1e100ace80 msgr2=0x7f1e100af320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.491+0000 7f1e2e101640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650] conn(0x7f1e100ace80 0x7f1e100af320 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.491+0000 7f1e2e101640 1 -- 172.21.15.98:0/1011208500 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e100a32d0 msgr2=0x7f1e100a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.491+0000 7f1e2e101640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e100a32d0 0x7f1e100a5790 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1e180058c0 tx=0x7f1e18005d30 comp rx=0 tx=0).stop 2024-09-19T08:01:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.491+0000 7f1e2e101640 1 -- 172.21.15.98:0/1011208500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e28102570 msgr2=0x7f1e28199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.491+0000 7f1e2e101640 1 --2- 172.21.15.98:0/1011208500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e28102570 0x7f1e28199870 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c006010 tx=0x7f1e1c003c80 comp rx=0 tx=0).stop 2024-09-19T08:01:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.491+0000 7f1e27fff640 1 -- 172.21.15.98:0/1011208500 reap_dead start 2024-09-19T08:01:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.491+0000 7f1e2e101640 1 -- 172.21.15.98:0/1011208500 shutdown_connections 2024-09-19T08:01:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.491+0000 7f1e2e101640 1 -- 172.21.15.98:0/1011208500 >> 172.21.15.98:0/1011208500 conn(0x7f1e280fc7d0 msgr2=0x7f1e280febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.492+0000 7f1e2e101640 1 -- 172.21.15.98:0/1011208500 shutdown_connections 2024-09-19T08:01:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:52.492+0000 7f1e2e101640 1 -- 172.21.15.98:0/1011208500 wait complete. 2024-09-19T08:01:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.568+0000 7f7cac054640 1 -- 172.21.15.98:0/1070423798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca4102d70 msgr2=0x7f7ca4103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.568+0000 7f7cac054640 1 --2- 172.21.15.98:0/1070423798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca4102d70 0x7f7ca4103150 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009930 tx=0x7f7ca002a420 comp rx=0 tx=0).stop 2024-09-19T08:01:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.568+0000 7f7cac054640 1 -- 172.21.15.98:0/1070423798 shutdown_connections 2024-09-19T08:01:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.568+0000 7f7cac054640 1 --2- 172.21.15.98:0/1070423798 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ca41082a0 0x7f7ca410ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.568+0000 7f7cac054640 1 --2- 172.21.15.98:0/1070423798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ca4103720 0x7f7ca4107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.568+0000 7f7cac054640 1 --2- 172.21.15.98:0/1070423798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca4102d70 0x7f7ca4103150 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.568+0000 7f7cac054640 1 -- 172.21.15.98:0/1070423798 >> 172.21.15.98:0/1070423798 conn(0x7f7ca40fc7d0 msgr2=0x7f7ca40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.569+0000 7f7cac054640 1 -- 172.21.15.98:0/1070423798 shutdown_connections 2024-09-19T08:01:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.569+0000 7f7cac054640 1 -- 172.21.15.98:0/1070423798 wait complete. 2024-09-19T08:01:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.569+0000 7f7cac054640 1 Processor -- start 2024-09-19T08:01:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.569+0000 7f7cac054640 1 -- start start 2024-09-19T08:01:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7cac054640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca4102d70 0x7f7ca4198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7cac054640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ca4103720 0x7f7ca4198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7cac054640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ca41082a0 0x7f7ca419ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7cac054640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca41120a0 con 0x7f7ca4102d70 2024-09-19T08:01:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7cac054640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca4111f20 con 0x7f7ca41082a0 2024-09-19T08:01:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7ca9df0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca4102d70 0x7f7ca4198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7cac054640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca4112220 con 0x7f7ca4103720 2024-09-19T08:01:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7ca9df0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca4102d70 0x7f7ca4198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58702/0 (socket says 172.21.15.98:58702) 2024-09-19T08:01:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7ca9df0640 1 -- 172.21.15.98:0/465935121 learned_addr learned my addr 172.21.15.98:0/465935121 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7caa5f1640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ca41082a0 0x7f7ca419ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7ca95ef640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ca4103720 0x7f7ca4198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7ca9df0640 1 -- 172.21.15.98:0/465935121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ca4103720 msgr2=0x7f7ca4198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7ca9df0640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ca4103720 0x7f7ca4198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7ca9df0640 1 -- 172.21.15.98:0/465935121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ca41082a0 msgr2=0x7f7ca419ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7ca9df0640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ca41082a0 0x7f7ca419ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7ca9df0640 1 -- 172.21.15.98:0/465935121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca419d530 con 0x7f7ca4102d70 2024-09-19T08:01:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7ca95ef640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ca4103720 0x7f7ca4198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:01:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7ca9df0640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca4102d70 0x7f7ca4198580 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009230 tx=0x7f7ca0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7c9affd640 1 -- 172.21.15.98:0/465935121 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca003f030 con 0x7f7ca4102d70 2024-09-19T08:01:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7c9affd640 1 -- 172.21.15.98:0/465935121 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ca0003ad0 con 0x7f7ca4102d70 2024-09-19T08:01:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.570+0000 7f7cac054640 1 -- 172.21.15.98:0/465935121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca419d7c0 con 0x7f7ca4102d70 2024-09-19T08:01:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.571+0000 7f7c9affd640 1 -- 172.21.15.98:0/465935121 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0035070 con 0x7f7ca4102d70 2024-09-19T08:01:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.571+0000 7f7cac054640 1 -- 172.21.15.98:0/465935121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca41a5100 con 0x7f7ca4102d70 2024-09-19T08:01:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.571+0000 7f7cac054640 1 -- 172.21.15.98:0/465935121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7ca4062980 con 0x7f7ca4102d70 2024-09-19T08:01:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.573+0000 7f7c9affd640 1 -- 172.21.15.98:0/465935121 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7ca0042030 con 0x7f7ca4102d70 2024-09-19T08:01:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.574+0000 7f7c9affd640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7c900a32d0 0x7f7c900a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.574+0000 7f7ca95ef640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7c900a32d0 0x7f7c900a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.574+0000 7f7c9affd640 1 -- 172.21.15.98:0/465935121 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f7ca00f04c0 con 0x7f7ca4102d70 2024-09-19T08:01:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.574+0000 7f7c9affd640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639] conn(0x7f7c900ace80 0x7f7c900af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.574+0000 7f7c9affd640 1 -- 172.21.15.98:0/465935121 --> [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c900af9f0 con 0x7f7c900ace80 2024-09-19T08:01:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.574+0000 7f7c9affd640 1 -- 172.21.15.98:0/465935121 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f7ca00f0870 con 0x7f7ca4102d70 2024-09-19T08:01:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.574+0000 7f7ca95ef640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7c900a32d0 0x7f7c900a5790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40fee00 tx=0x7f7c94005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.574+0000 7f7caa5f1640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639] conn(0x7f7c900ace80 0x7f7c900af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.575+0000 7f7caa5f1640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639] conn(0x7f7c900ace80 0x7f7c900af320 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.576+0000 7f7c9affd640 1 -- 172.21.15.98:0/465935121 <== osd.10 v2:172.21.15.139:6824/2336761639 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7c900af9f0 con 0x7f7c900ace80 2024-09-19T08:01:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.589+0000 7f7cac054640 1 -- 172.21.15.98:0/465935121 --> [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7ca4062e00 con 0x7f7c900ace80 2024-09-19T08:01:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.590+0000 7f7c9affd640 1 -- 172.21.15.98:0/465935121 <== osd.10 v2:172.21.15.139:6824/2336761639 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7ca4062e00 con 0x7f7c900ace80 2024-09-19T08:01:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.590+0000 7f7cac054640 1 -- 172.21.15.98:0/465935121 >> [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639] conn(0x7f7c900ace80 msgr2=0x7f7c900af320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.590+0000 7f7cac054640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639] conn(0x7f7c900ace80 0x7f7c900af320 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.590+0000 7f7cac054640 1 -- 172.21.15.98:0/465935121 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7c900a32d0 msgr2=0x7f7c900a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.590+0000 7f7cac054640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7c900a32d0 0x7f7c900a5790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40fee00 tx=0x7f7c94005e50 comp rx=0 tx=0).stop 2024-09-19T08:01:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.590+0000 7f7cac054640 1 -- 172.21.15.98:0/465935121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca4102d70 msgr2=0x7f7ca4198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.590+0000 7f7cac054640 1 --2- 172.21.15.98:0/465935121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca4102d70 0x7f7ca4198580 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009230 tx=0x7f7ca0004270 comp rx=0 tx=0).stop 2024-09-19T08:01:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.590+0000 7f7caa5f1640 1 -- 172.21.15.98:0/465935121 reap_dead start 2024-09-19T08:01:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.590+0000 7f7cac054640 1 -- 172.21.15.98:0/465935121 shutdown_connections 2024-09-19T08:01:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.590+0000 7f7cac054640 1 -- 172.21.15.98:0/465935121 >> 172.21.15.98:0/465935121 conn(0x7f7ca40fc7d0 msgr2=0x7f7ca4106330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.590+0000 7f7cac054640 1 -- 172.21.15.98:0/465935121 shutdown_connections 2024-09-19T08:01:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:48.590+0000 7f7cac054640 1 -- 172.21.15.98:0/465935121 wait complete. 2024-09-19T08:01:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:52 smithi098 ceph-mon[31333]: pgmap v293: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-19T08:01:53.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:52 smithi162 ceph-mon[35260]: pgmap v293: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-19T08:01:53.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:52 smithi139 ceph-mon[34874]: pgmap v293: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-19T08:01:53.486 INFO:teuthology.orchestra.run.smithi098.stdout:279172874265 2024-09-19T08:01:53.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-19T08:01:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.669+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/3298801124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb4100e30 msgr2=0x7fdcb4104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.669+0000 7fdcb8e8e640 1 --2- 172.21.15.98:0/3298801124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb4100e30 0x7fdcb4104e80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c009900 tx=0x7fdc9c02a410 comp rx=0 tx=0).stop 2024-09-19T08:01:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.669+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/3298801124 shutdown_connections 2024-09-19T08:01:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.669+0000 7fdcb8e8e640 1 --2- 172.21.15.98:0/3298801124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb41055b0 0x7fdcb410d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.669+0000 7fdcb8e8e640 1 --2- 172.21.15.98:0/3298801124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb4100e30 0x7fdcb4104e80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.669+0000 7fdcb8e8e640 1 --2- 172.21.15.98:0/3298801124 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb4100480 0x7fdcb4100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.669+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/3298801124 >> 172.21.15.98:0/3298801124 conn(0x7fdcb4077210 msgr2=0x7fdcb40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.669+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/3298801124 shutdown_connections 2024-09-19T08:01:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.669+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/3298801124 wait complete. 2024-09-19T08:01:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.670+0000 7fdcb8e8e640 1 Processor -- start 2024-09-19T08:01:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.670+0000 7fdcb8e8e640 1 -- start start 2024-09-19T08:01:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.670+0000 7fdcb8e8e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb4100480 0x7fdcb41970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.670+0000 7fdcb8e8e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb4100e30 0x7fdcb4197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.670+0000 7fdcb8e8e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb41055b0 0x7fdcb4197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.670+0000 7fdcb8e8e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb4110eb0 con 0x7fdcb41055b0 2024-09-19T08:01:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.670+0000 7fdcb8e8e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb4110d30 con 0x7fdcb4100480 2024-09-19T08:01:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.670+0000 7fdcb8e8e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb4111030 con 0x7fdcb4100e30 2024-09-19T08:01:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcb2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb41055b0 0x7fdcb4197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcb2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb41055b0 0x7fdcb4197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58764/0 (socket says 172.21.15.98:58764) 2024-09-19T08:01:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcb2d9d640 1 -- 172.21.15.98:0/174725459 learned_addr learned my addr 172.21.15.98:0/174725459 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcabfff640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb4100e30 0x7fdcb4197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcb259c640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb4100480 0x7fdcb41970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcb2d9d640 1 -- 172.21.15.98:0/174725459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb4100e30 msgr2=0x7fdcb4197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcb2d9d640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb4100e30 0x7fdcb4197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcb2d9d640 1 -- 172.21.15.98:0/174725459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb4100480 msgr2=0x7fdcb41970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcb2d9d640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb4100480 0x7fdcb41970d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcb2d9d640 1 -- 172.21.15.98:0/174725459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb41a6940 con 0x7fdcb41055b0 2024-09-19T08:01:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcb259c640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb4100480 0x7fdcb41970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:01:53.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcb2d9d640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb41055b0 0x7fdcb4197b50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00cce0 tx=0x7fdcac00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcab7fe640 1 -- 172.21.15.98:0/174725459 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac00f450 con 0x7fdcb41055b0 2024-09-19T08:01:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/174725459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb41a6c30 con 0x7fdcb41055b0 2024-09-19T08:01:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcab7fe640 1 -- 172.21.15.98:0/174725459 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcac00fad0 con 0x7fdcb41055b0 2024-09-19T08:01:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcab7fe640 1 -- 172.21.15.98:0/174725459 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac018c10 con 0x7fdcb41055b0 2024-09-19T08:01:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.671+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/174725459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb41a70f0 con 0x7fdcb41055b0 2024-09-19T08:01:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.672+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/174725459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdcb4062940 con 0x7fdcb41055b0 2024-09-19T08:01:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.673+0000 7fdcab7fe640 1 -- 172.21.15.98:0/174725459 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fdcac00f5f0 con 0x7fdcb41055b0 2024-09-19T08:01:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.674+0000 7fdcab7fe640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdca00a32d0 0x7fdca00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.674+0000 7fdcab7fe640 1 -- 172.21.15.98:0/174725459 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fdcac0d0180 con 0x7fdcb41055b0 2024-09-19T08:01:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.674+0000 7fdcb259c640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdca00a32d0 0x7fdca00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.674+0000 7fdcab7fe640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458] conn(0x7fdca00ace80 0x7fdca00af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.674+0000 7fdcab7fe640 1 -- 172.21.15.98:0/174725459 --> [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdcac01ee50 con 0x7fdca00ace80 2024-09-19T08:01:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.674+0000 7fdcab7fe640 1 -- 172.21.15.98:0/174725459 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7fdcac0d7050 con 0x7fdcb41055b0 2024-09-19T08:01:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.675+0000 7fdcb259c640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdca00a32d0 0x7fdca00a5790 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdca4006fd0 tx=0x7fdca4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.675+0000 7fdcabfff640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458] conn(0x7fdca00ace80 0x7fdca00af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.675+0000 7fdcabfff640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458] conn(0x7fdca00ace80 0x7fdca00af320 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:53.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.676+0000 7fdcab7fe640 1 -- 172.21.15.98:0/174725459 <== osd.8 v2:172.21.15.162:6816/3607966458 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdcac01ee50 con 0x7fdca00ace80 2024-09-19T08:01:53.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.691+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/174725459 --> [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdcb4062df0 con 0x7fdca00ace80 2024-09-19T08:01:53.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.691+0000 7fdcab7fe640 1 -- 172.21.15.98:0/174725459 <== osd.8 v2:172.21.15.162:6816/3607966458 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fdcb4062df0 con 0x7fdca00ace80 2024-09-19T08:01:53.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.692+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/174725459 >> [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458] conn(0x7fdca00ace80 msgr2=0x7fdca00af320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:53.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.692+0000 7fdcb8e8e640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458] conn(0x7fdca00ace80 0x7fdca00af320 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:53.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.692+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/174725459 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdca00a32d0 msgr2=0x7fdca00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:53.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.692+0000 7fdcb8e8e640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdca00a32d0 0x7fdca00a5790 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdca4006fd0 tx=0x7fdca4008040 comp rx=0 tx=0).stop 2024-09-19T08:01:53.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.692+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/174725459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb41055b0 msgr2=0x7fdcb4197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:53.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.692+0000 7fdcb8e8e640 1 --2- 172.21.15.98:0/174725459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb41055b0 0x7fdcb4197b50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00cce0 tx=0x7fdcac00a620 comp rx=0 tx=0).stop 2024-09-19T08:01:53.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.692+0000 7fdcb2d9d640 1 -- 172.21.15.98:0/174725459 reap_dead start 2024-09-19T08:01:53.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.692+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/174725459 shutdown_connections 2024-09-19T08:01:53.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.692+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/174725459 >> 172.21.15.98:0/174725459 conn(0x7fdcb4077210 msgr2=0x7fdcb4103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:53.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.692+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/174725459 shutdown_connections 2024-09-19T08:01:53.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:53.692+0000 7fdcb8e8e640 1 -- 172.21.15.98:0/174725459 wait complete. 2024-09-19T08:01:54.432 INFO:teuthology.orchestra.run.smithi098.stdout:231928234020 2024-09-19T08:01:54.433 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-19T08:01:54.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:54 smithi098 ceph-mon[31333]: pgmap v294: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-19T08:01:55.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.056+0000 7f1e8c46b640 1 -- 172.21.15.98:0/372925936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e84102ef0 msgr2=0x7f1e8410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:55.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.056+0000 7f1e8c46b640 1 --2- 172.21.15.98:0/372925936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e84102ef0 0x7f1e8410da40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009900 tx=0x7f1e7402a480 comp rx=0 tx=0).stop 2024-09-19T08:01:55.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.056+0000 7f1e8c46b640 1 -- 172.21.15.98:0/372925936 shutdown_connections 2024-09-19T08:01:55.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.056+0000 7f1e8c46b640 1 --2- 172.21.15.98:0/372925936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e84102ef0 0x7f1e8410da40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:55.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.056+0000 7f1e8c46b640 1 --2- 172.21.15.98:0/372925936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e84102570 0x7f1e841029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.056+0000 7f1e8c46b640 1 --2- 172.21.15.98:0/372925936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e84100e20 0x7f1e84101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.056+0000 7f1e8c46b640 1 -- 172.21.15.98:0/372925936 >> 172.21.15.98:0/372925936 conn(0x7f1e840fc7d0 msgr2=0x7f1e840febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.056+0000 7f1e8c46b640 1 -- 172.21.15.98:0/372925936 shutdown_connections 2024-09-19T08:01:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.056+0000 7f1e8c46b640 1 -- 172.21.15.98:0/372925936 wait complete. 2024-09-19T08:01:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.056+0000 7f1e8c46b640 1 Processor -- start 2024-09-19T08:01:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8c46b640 1 -- start start 2024-09-19T08:01:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8c46b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e84100e20 0x7f1e84199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8c46b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e84102570 0x7f1e84199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8c46b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e84102ef0 0x7f1e841a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8c46b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e84110fa0 con 0x7f1e84100e20 2024-09-19T08:01:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8c46b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1e84110e20 con 0x7f1e84102570 2024-09-19T08:01:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8c46b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1e84111120 con 0x7f1e84102ef0 2024-09-19T08:01:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8a207640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e84100e20 0x7f1e84199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8a207640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e84100e20 0x7f1e84199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58782/0 (socket says 172.21.15.98:58782) 2024-09-19T08:01:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8a207640 1 -- 172.21.15.98:0/2584056664 learned_addr learned my addr 172.21.15.98:0/2584056664 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e89a06640 1 --2- 172.21.15.98:0/2584056664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e84102570 0x7f1e84199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8aa08640 1 --2- 172.21.15.98:0/2584056664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e84102ef0 0x7f1e841a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8a207640 1 -- 172.21.15.98:0/2584056664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e84102ef0 msgr2=0x7f1e841a4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8a207640 1 --2- 172.21.15.98:0/2584056664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e84102ef0 0x7f1e841a4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8a207640 1 -- 172.21.15.98:0/2584056664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e84102570 msgr2=0x7f1e84199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8a207640 1 --2- 172.21.15.98:0/2584056664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e84102570 0x7f1e84199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8a207640 1 -- 172.21.15.98:0/2584056664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e841a48d0 con 0x7f1e84100e20 2024-09-19T08:01:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8a207640 1 --2- 172.21.15.98:0/2584056664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e84100e20 0x7f1e84199310 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00c890 tx=0x7f1e7c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e837fe640 1 -- 172.21.15.98:0/2584056664 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c004220 con 0x7f1e84100e20 2024-09-19T08:01:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e837fe640 1 -- 172.21.15.98:0/2584056664 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e7c0043c0 con 0x7f1e84100e20 2024-09-19T08:01:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.057+0000 7f1e8c46b640 1 -- 172.21.15.98:0/2584056664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e841a4aa0 con 0x7f1e84100e20 2024-09-19T08:01:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.058+0000 7f1e837fe640 1 -- 172.21.15.98:0/2584056664 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c0049e0 con 0x7f1e84100e20 2024-09-19T08:01:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.058+0000 7f1e8c46b640 1 -- 172.21.15.98:0/2584056664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e841a4fe0 con 0x7f1e84100e20 2024-09-19T08:01:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.058+0000 7f1e8c46b640 1 -- 172.21.15.98:0/2584056664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1e841072a0 con 0x7f1e84100e20 2024-09-19T08:01:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.060+0000 7f1e837fe640 1 -- 172.21.15.98:0/2584056664 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1e7c0079b0 con 0x7f1e84100e20 2024-09-19T08:01:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.060+0000 7f1e837fe640 1 --2- 172.21.15.98:0/2584056664 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e6c0a32d0 0x7f1e6c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.060+0000 7f1e89a06640 1 --2- 172.21.15.98:0/2584056664 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e6c0a32d0 0x7f1e6c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.060+0000 7f1e837fe640 1 -- 172.21.15.98:0/2584056664 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f1e7c0cfc20 con 0x7f1e84100e20 2024-09-19T08:01:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.060+0000 7f1e837fe640 1 --2- 172.21.15.98:0/2584056664 >> [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626] conn(0x7f1e6c0ace80 0x7f1e6c0af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.060+0000 7f1e837fe640 1 -- 172.21.15.98:0/2584056664 --> [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e6c0af9f0 con 0x7f1e6c0ace80 2024-09-19T08:01:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.060+0000 7f1e837fe640 1 -- 172.21.15.98:0/2584056664 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f1e7c0d0070 con 0x7f1e84100e20 2024-09-19T08:01:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.061+0000 7f1e89a06640 1 --2- 172.21.15.98:0/2584056664 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e6c0a32d0 0x7f1e6c0a5790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1e8406ab00 tx=0x7f1e7800d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.061+0000 7f1e8aa08640 1 --2- 172.21.15.98:0/2584056664 >> [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626] conn(0x7f1e6c0ace80 0x7f1e6c0af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.061+0000 7f1e8aa08640 1 --2- 172.21.15.98:0/2584056664 >> [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626] conn(0x7f1e6c0ace80 0x7f1e6c0af320 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.062+0000 7f1e837fe640 1 -- 172.21.15.98:0/2584056664 <== osd.1 v2:172.21.15.139:6800/4036962626 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e6c0af9f0 con 0x7f1e6c0ace80 2024-09-19T08:01:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.075+0000 7f1e8c46b640 1 -- 172.21.15.98:0/2584056664 --> [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1e841021b0 con 0x7f1e6c0ace80 2024-09-19T08:01:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.076+0000 7f1e837fe640 1 -- 172.21.15.98:0/2584056664 <== osd.1 v2:172.21.15.139:6800/4036962626 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1e841021b0 con 0x7f1e6c0ace80 2024-09-19T08:01:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.076+0000 7f1e8c46b640 1 -- 172.21.15.98:0/2584056664 >> [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626] conn(0x7f1e6c0ace80 msgr2=0x7f1e6c0af320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.076+0000 7f1e8c46b640 1 --2- 172.21.15.98:0/2584056664 >> [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626] conn(0x7f1e6c0ace80 0x7f1e6c0af320 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.076+0000 7f1e8c46b640 1 -- 172.21.15.98:0/2584056664 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e6c0a32d0 msgr2=0x7f1e6c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.076+0000 7f1e8c46b640 1 --2- 172.21.15.98:0/2584056664 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f1e6c0a32d0 0x7f1e6c0a5790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1e8406ab00 tx=0x7f1e7800d040 comp rx=0 tx=0).stop 2024-09-19T08:01:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.076+0000 7f1e8c46b640 1 -- 172.21.15.98:0/2584056664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e84100e20 msgr2=0x7f1e84199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.076+0000 7f1e8c46b640 1 --2- 172.21.15.98:0/2584056664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e84100e20 0x7f1e84199310 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00c890 tx=0x7f1e7c00cd50 comp rx=0 tx=0).stop 2024-09-19T08:01:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.076+0000 7f1e8aa08640 1 -- 172.21.15.98:0/2584056664 reap_dead start 2024-09-19T08:01:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.076+0000 7f1e8c46b640 1 -- 172.21.15.98:0/2584056664 shutdown_connections 2024-09-19T08:01:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.076+0000 7f1e8c46b640 1 -- 172.21.15.98:0/2584056664 >> 172.21.15.98:0/2584056664 conn(0x7f1e840fc7d0 msgr2=0x7f1e84106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.077+0000 7f1e8c46b640 1 -- 172.21.15.98:0/2584056664 shutdown_connections 2024-09-19T08:01:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:55.077+0000 7f1e8c46b640 1 -- 172.21.15.98:0/2584056664 wait complete. 2024-09-19T08:01:55.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:54 smithi162 ceph-mon[35260]: pgmap v294: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-19T08:01:55.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:54 smithi139 ceph-mon[34874]: pgmap v294: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-19T08:01:55.937 INFO:teuthology.orchestra.run.smithi098.stdout:201863462952 2024-09-19T08:01:55.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-19T08:01:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.123+0000 7f8e3a55c640 1 -- 172.21.15.98:0/2180181093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e34106a90 msgr2=0x7f8e34106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.123+0000 7f8e3a55c640 1 --2- 172.21.15.98:0/2180181093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e34106a90 0x7f8e34106e70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8e24009930 tx=0x7f8e2402a420 comp rx=0 tx=0).stop 2024-09-19T08:01:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.124+0000 7f8e3a55c640 1 -- 172.21.15.98:0/2180181093 shutdown_connections 2024-09-19T08:01:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.124+0000 7f8e3a55c640 1 --2- 172.21.15.98:0/2180181093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e34101430 0x7f8e3410d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.124+0000 7f8e3a55c640 1 --2- 172.21.15.98:0/2180181093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e34100a90 0x7f8e34100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.124+0000 7f8e3a55c640 1 --2- 172.21.15.98:0/2180181093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e34106a90 0x7f8e34106e70 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.124+0000 7f8e3a55c640 1 -- 172.21.15.98:0/2180181093 >> 172.21.15.98:0/2180181093 conn(0x7f8e340fc7d0 msgr2=0x7f8e340febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.124+0000 7f8e3a55c640 1 -- 172.21.15.98:0/2180181093 shutdown_connections 2024-09-19T08:01:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.124+0000 7f8e3a55c640 1 -- 172.21.15.98:0/2180181093 wait complete. 2024-09-19T08:01:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.125+0000 7f8e3a55c640 1 Processor -- start 2024-09-19T08:01:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.125+0000 7f8e3a55c640 1 -- start start 2024-09-19T08:01:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.125+0000 7f8e3a55c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e34100a90 0x7f8e34198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.125+0000 7f8e3a55c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e34101430 0x7f8e34198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.125+0000 7f8e3a55c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e34106a90 0x7f8e3419cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.125+0000 7f8e3a55c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e34110e90 con 0x7f8e34106a90 2024-09-19T08:01:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.125+0000 7f8e3a55c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8e34110d10 con 0x7f8e34100a90 2024-09-19T08:01:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.125+0000 7f8e3a55c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8e34111010 con 0x7f8e34101430 2024-09-19T08:01:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.125+0000 7f8e38af9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e34106a90 0x7f8e3419cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.125+0000 7f8e38af9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e34106a90 0x7f8e3419cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58800/0 (socket says 172.21.15.98:58800) 2024-09-19T08:01:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.125+0000 7f8e38af9640 1 -- 172.21.15.98:0/1950186080 learned_addr learned my addr 172.21.15.98:0/1950186080 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.126+0000 7f8e337fe640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e34101430 0x7f8e34198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.126+0000 7f8e33fff640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e34100a90 0x7f8e34198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.126+0000 7f8e38af9640 1 -- 172.21.15.98:0/1950186080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e34101430 msgr2=0x7f8e34198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.126+0000 7f8e38af9640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e34101430 0x7f8e34198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.126+0000 7f8e38af9640 1 -- 172.21.15.98:0/1950186080 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e34100a90 msgr2=0x7f8e34198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.126+0000 7f8e38af9640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e34100a90 0x7f8e34198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.126+0000 7f8e38af9640 1 -- 172.21.15.98:0/1950186080 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3419d4b0 con 0x7f8e34106a90 2024-09-19T08:01:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.126+0000 7f8e33fff640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e34100a90 0x7f8e34198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:01:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.126+0000 7f8e38af9640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e34106a90 0x7f8e3419cdd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00eaa0 tx=0x7f8e2c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.127+0000 7f8e317fa640 1 -- 172.21.15.98:0/1950186080 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c00cdb0 con 0x7f8e34106a90 2024-09-19T08:01:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.127+0000 7f8e3a55c640 1 -- 172.21.15.98:0/1950186080 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e3419d7a0 con 0x7f8e34106a90 2024-09-19T08:01:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.127+0000 7f8e3a55c640 1 -- 172.21.15.98:0/1950186080 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e340fcdd0 con 0x7f8e34106a90 2024-09-19T08:01:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.127+0000 7f8e317fa640 1 -- 172.21.15.98:0/1950186080 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e2c005020 con 0x7f8e34106a90 2024-09-19T08:01:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.127+0000 7f8e317fa640 1 -- 172.21.15.98:0/1950186080 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c0116b0 con 0x7f8e34106a90 2024-09-19T08:01:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.128+0000 7f8e3a55c640 1 -- 172.21.15.98:0/1950186080 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8e20001030 con 0x7f8e34106a90 2024-09-19T08:01:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.129+0000 7f8e317fa640 1 -- 172.21.15.98:0/1950186080 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f8e2c004a40 con 0x7f8e34106a90 2024-09-19T08:01:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.129+0000 7f8e317fa640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f8e1c0a3350 0x7f8e1c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.130+0000 7f8e33fff640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f8e1c0a3350 0x7f8e1c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.130+0000 7f8e317fa640 1 -- 172.21.15.98:0/1950186080 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f8e2c0cfed0 con 0x7f8e34106a90 2024-09-19T08:01:56.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.130+0000 7f8e317fa640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554] conn(0x7f8e1c0acf00 0x7f8e1c0af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:56.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.130+0000 7f8e317fa640 1 -- 172.21.15.98:0/1950186080 --> [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e2c014ec0 con 0x7f8e1c0acf00 2024-09-19T08:01:56.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.130+0000 7f8e317fa640 1 -- 172.21.15.98:0/1950186080 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f8e2c0d0320 con 0x7f8e34106a90 2024-09-19T08:01:56.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.130+0000 7f8e33fff640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f8e1c0a3350 0x7f8e1c0a5810 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8e24000c00 tx=0x7f8e24005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:56.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.130+0000 7f8e337fe640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554] conn(0x7f8e1c0acf00 0x7f8e1c0af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:56.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.131+0000 7f8e337fe640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554] conn(0x7f8e1c0acf00 0x7f8e1c0af3a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:56.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.131+0000 7f8e317fa640 1 -- 172.21.15.98:0/1950186080 <== osd.11 v2:172.21.15.162:6824/1231011554 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8e2c014ec0 con 0x7f8e1c0acf00 2024-09-19T08:01:56.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.145+0000 7f8e3a55c640 1 -- 172.21.15.98:0/1950186080 --> [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8e20002dd0 con 0x7f8e1c0acf00 2024-09-19T08:01:56.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.146+0000 7f8e317fa640 1 -- 172.21.15.98:0/1950186080 <== osd.11 v2:172.21.15.162:6824/1231011554 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8e20002dd0 con 0x7f8e1c0acf00 2024-09-19T08:01:56.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.146+0000 7f8e3a55c640 1 -- 172.21.15.98:0/1950186080 >> [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554] conn(0x7f8e1c0acf00 msgr2=0x7f8e1c0af3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:56.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.146+0000 7f8e3a55c640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554] conn(0x7f8e1c0acf00 0x7f8e1c0af3a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:56.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.146+0000 7f8e3a55c640 1 -- 172.21.15.98:0/1950186080 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f8e1c0a3350 msgr2=0x7f8e1c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:56.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.146+0000 7f8e3a55c640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f8e1c0a3350 0x7f8e1c0a5810 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8e24000c00 tx=0x7f8e24005a20 comp rx=0 tx=0).stop 2024-09-19T08:01:56.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.146+0000 7f8e3a55c640 1 -- 172.21.15.98:0/1950186080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e34106a90 msgr2=0x7f8e3419cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:56.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.146+0000 7f8e3a55c640 1 --2- 172.21.15.98:0/1950186080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e34106a90 0x7f8e3419cdd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00eaa0 tx=0x7f8e2c00ef60 comp rx=0 tx=0).stop 2024-09-19T08:01:56.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.146+0000 7f8e38af9640 1 -- 172.21.15.98:0/1950186080 reap_dead start 2024-09-19T08:01:56.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.146+0000 7f8e3a55c640 1 -- 172.21.15.98:0/1950186080 shutdown_connections 2024-09-19T08:01:56.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.146+0000 7f8e3a55c640 1 -- 172.21.15.98:0/1950186080 >> 172.21.15.98:0/1950186080 conn(0x7f8e340fc7d0 msgr2=0x7f8e3406aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:56.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.147+0000 7f8e3a55c640 1 -- 172.21.15.98:0/1950186080 shutdown_connections 2024-09-19T08:01:56.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:56.147+0000 7f8e3a55c640 1 -- 172.21.15.98:0/1950186080 wait complete. 2024-09-19T08:01:56.664 INFO:teuthology.orchestra.run.smithi098.stdout:154618822706 2024-09-19T08:01:56.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-19T08:01:56.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:56 smithi098 ceph-mon[31333]: pgmap v295: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-19T08:01:57.076 INFO:teuthology.orchestra.run.smithi098.stdout:51539607623 2024-09-19T08:01:57.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-19T08:01:57.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:56 smithi162 ceph-mon[35260]: pgmap v295: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-19T08:01:57.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:56 smithi139 ceph-mon[34874]: pgmap v295: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-19T08:01:57.318 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:57.318 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:57.319 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:57.319 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:57.381 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.106+0000 7fdd834e0640 1 -- 172.21.15.98:0/1454204993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd7c102790 msgr2=0x7fdd7c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.106+0000 7fdd834e0640 1 --2- 172.21.15.98:0/1454204993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd7c102790 0x7fdd7c1067e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdd78009900 tx=0x7fdd7802a410 comp rx=0 tx=0).stop 2024-09-19T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.106+0000 7fdd834e0640 1 -- 172.21.15.98:0/1454204993 shutdown_connections 2024-09-19T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.106+0000 7fdd834e0640 1 --2- 172.21.15.98:0/1454204993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c106f10 0x7fdd7c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.106+0000 7fdd834e0640 1 --2- 172.21.15.98:0/1454204993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd7c102790 0x7fdd7c1067e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.106+0000 7fdd834e0640 1 --2- 172.21.15.98:0/1454204993 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd7c101de0 0x7fdd7c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.106+0000 7fdd834e0640 1 -- 172.21.15.98:0/1454204993 >> 172.21.15.98:0/1454204993 conn(0x7fdd7c0fd8c0 msgr2=0x7fdd7c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.106+0000 7fdd834e0640 1 -- 172.21.15.98:0/1454204993 shutdown_connections 2024-09-19T08:01:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.106+0000 7fdd834e0640 1 -- 172.21.15.98:0/1454204993 wait complete. 2024-09-19T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.106+0000 7fdd834e0640 1 Processor -- start 2024-09-19T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd834e0640 1 -- start start 2024-09-19T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd834e0640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd7c101de0 0x7fdd7c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd834e0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd7c102790 0x7fdd7c1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd834e0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c106f10 0x7fdd7c19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd834e0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c111f90 con 0x7fdd7c102790 2024-09-19T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd834e0640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c111e10 con 0x7fdd7c101de0 2024-09-19T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd834e0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c112110 con 0x7fdd7c106f10 2024-09-19T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd80a7b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd7c102790 0x7fdd7c1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd80a7b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd7c102790 0x7fdd7c1989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34732/0 (socket says 172.21.15.98:34732) 2024-09-19T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd80a7b640 1 -- 172.21.15.98:0/2065304819 learned_addr learned my addr 172.21.15.98:0/2065304819 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd8127c640 1 --2- 172.21.15.98:0/2065304819 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd7c101de0 0x7fdd7c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd81a7d640 1 --2- 172.21.15.98:0/2065304819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c106f10 0x7fdd7c19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd80a7b640 1 -- 172.21.15.98:0/2065304819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c106f10 msgr2=0x7fdd7c19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd80a7b640 1 --2- 172.21.15.98:0/2065304819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c106f10 0x7fdd7c19cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd80a7b640 1 -- 172.21.15.98:0/2065304819 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd7c101de0 msgr2=0x7fdd7c198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd80a7b640 1 --2- 172.21.15.98:0/2065304819 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd7c101de0 0x7fdd7c198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd80a7b640 1 -- 172.21.15.98:0/2065304819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd7c19d420 con 0x7fdd7c102790 2024-09-19T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd80a7b640 1 --2- 172.21.15.98:0/2065304819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd7c102790 0x7fdd7c1989b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdd78005ab0 tx=0x7fdd780042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.107+0000 7fdd6e7fc640 1 -- 172.21.15.98:0/2065304819 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7803f030 con 0x7fdd7c102790 2024-09-19T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.108+0000 7fdd6e7fc640 1 -- 172.21.15.98:0/2065304819 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd78003ce0 con 0x7fdd7c102790 2024-09-19T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.108+0000 7fdd834e0640 1 -- 172.21.15.98:0/2065304819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd7c19d6b0 con 0x7fdd7c102790 2024-09-19T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.108+0000 7fdd6e7fc640 1 -- 172.21.15.98:0/2065304819 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78035070 con 0x7fdd7c102790 2024-09-19T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.108+0000 7fdd834e0640 1 -- 172.21.15.98:0/2065304819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd7c1a4ff0 con 0x7fdd7c102790 2024-09-19T08:01:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.109+0000 7fdd834e0640 1 -- 172.21.15.98:0/2065304819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd7c1032c0 con 0x7fdd7c102790 2024-09-19T08:01:58.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.110+0000 7fdd6e7fc640 1 -- 172.21.15.98:0/2065304819 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fdd78042030 con 0x7fdd7c102790 2024-09-19T08:01:58.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.110+0000 7fdd6e7fc640 1 --2- 172.21.15.98:0/2065304819 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdd680a32d0 0x7fdd680a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:58.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.110+0000 7fdd8127c640 1 --2- 172.21.15.98:0/2065304819 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdd680a32d0 0x7fdd680a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.110+0000 7fdd6e7fc640 1 -- 172.21.15.98:0/2065304819 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fdd780f0380 con 0x7fdd7c102790 2024-09-19T08:01:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.111+0000 7fdd8127c640 1 --2- 172.21.15.98:0/2065304819 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdd680a32d0 0x7fdd680a5790 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdd700058c0 tx=0x7fdd7000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.115+0000 7fdd6e7fc640 1 -- 172.21.15.98:0/2065304819 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fdd780bcf50 con 0x7fdd7c102790 2024-09-19T08:01:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.285+0000 7fdd834e0640 1 -- 172.21.15.98:0/2065304819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fdd7c10a0f0 con 0x7fdd7c102790 2024-09-19T08:01:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.286+0000 7fdd6e7fc640 1 -- 172.21.15.98:0/2065304819 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdd780bc850 con 0x7fdd7c102790 2024-09-19T08:01:58.287 INFO:teuthology.orchestra.run.smithi098.stdout:34359738443 2024-09-19T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.288+0000 7fdd834e0640 1 -- 172.21.15.98:0/2065304819 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdd680a32d0 msgr2=0x7fdd680a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.288+0000 7fdd834e0640 1 --2- 172.21.15.98:0/2065304819 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdd680a32d0 0x7fdd680a5790 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdd700058c0 tx=0x7fdd7000a2d0 comp rx=0 tx=0).stop 2024-09-19T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.288+0000 7fdd834e0640 1 -- 172.21.15.98:0/2065304819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd7c102790 msgr2=0x7fdd7c1989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.288+0000 7fdd834e0640 1 --2- 172.21.15.98:0/2065304819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd7c102790 0x7fdd7c1989b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdd78005ab0 tx=0x7fdd780042f0 comp rx=0 tx=0).stop 2024-09-19T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.288+0000 7fdd834e0640 1 -- 172.21.15.98:0/2065304819 shutdown_connections 2024-09-19T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.288+0000 7fdd834e0640 1 --2- 172.21.15.98:0/2065304819 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdd680a32d0 0x7fdd680a5790 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.288+0000 7fdd834e0640 1 --2- 172.21.15.98:0/2065304819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c106f10 0x7fdd7c19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.288+0000 7fdd834e0640 1 --2- 172.21.15.98:0/2065304819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd7c102790 0x7fdd7c1989b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.288+0000 7fdd834e0640 1 --2- 172.21.15.98:0/2065304819 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd7c101de0 0x7fdd7c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.288+0000 7fdd834e0640 1 -- 172.21.15.98:0/2065304819 >> 172.21.15.98:0/2065304819 conn(0x7fdd7c0fd8c0 msgr2=0x7fdd7c0fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.288+0000 7fdd834e0640 1 -- 172.21.15.98:0/2065304819 shutdown_connections 2024-09-19T08:01:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.288+0000 7fdd834e0640 1 -- 172.21.15.98:0/2065304819 wait complete. 2024-09-19T08:01:58.828 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:58 smithi098 ceph-mon[31333]: pgmap v296: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-09-19T08:01:58.828 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:58 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2065304819' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-19T08:01:59.027 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:59.052 INFO:teuthology.orchestra.run.smithi098.stdout:309237645334 2024-09-19T08:01:59.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-19T08:01:59.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:58 smithi162 ceph-mon[35260]: pgmap v296: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-09-19T08:01:59.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:58 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2065304819' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-19T08:01:59.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:58 smithi139 ceph-mon[34874]: pgmap v296: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-09-19T08:01:59.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:58 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2065304819' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-19T08:01:59.228 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:59.228 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:01:59.593 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738442 got 34359738443 for osd.0 2024-09-19T08:01:59.593 DEBUG:teuthology.parallel:result is None 2024-09-19T08:01:59.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.612+0000 7fbdac818640 1 -- 172.21.15.98:0/2286718738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda8100e20 msgr2=0x7fbda8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:59.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.612+0000 7fbdac818640 1 --2- 172.21.15.98:0/2286718738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda8100e20 0x7fbda8101200 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbd98009900 tx=0x7fbd9802a480 comp rx=0 tx=0).stop 2024-09-19T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.612+0000 7fbdac818640 1 -- 172.21.15.98:0/2286718738 shutdown_connections 2024-09-19T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.612+0000 7fbdac818640 1 --2- 172.21.15.98:0/2286718738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbda8102ef0 0x7fbda810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.612+0000 7fbdac818640 1 --2- 172.21.15.98:0/2286718738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbda8102570 0x7fbda81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.612+0000 7fbdac818640 1 --2- 172.21.15.98:0/2286718738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda8100e20 0x7fbda8101200 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.612+0000 7fbdac818640 1 -- 172.21.15.98:0/2286718738 >> 172.21.15.98:0/2286718738 conn(0x7fbda80fc7d0 msgr2=0x7fbda80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.612+0000 7fbdac818640 1 -- 172.21.15.98:0/2286718738 shutdown_connections 2024-09-19T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.612+0000 7fbdac818640 1 -- 172.21.15.98:0/2286718738 wait complete. 2024-09-19T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.613+0000 7fbdac818640 1 Processor -- start 2024-09-19T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.613+0000 7fbdac818640 1 -- start start 2024-09-19T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.613+0000 7fbdac818640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbda8100e20 0x7fbda8199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.613+0000 7fbdac818640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbda8102570 0x7fbda8199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.613+0000 7fbdac818640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda8102ef0 0x7fbda81a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.613+0000 7fbdac818640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbda8110f80 con 0x7fbda8102ef0 2024-09-19T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.613+0000 7fbdac818640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbda8110e00 con 0x7fbda8102570 2024-09-19T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.613+0000 7fbdac818640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbda8111100 con 0x7fbda8100e20 2024-09-19T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.613+0000 7fbda6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda8102ef0 0x7fbda81a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.613+0000 7fbda6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda8102ef0 0x7fbda81a43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34760/0 (socket says 172.21.15.98:34760) 2024-09-19T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.613+0000 7fbda6d9d640 1 -- 172.21.15.98:0/114957656 learned_addr learned my addr 172.21.15.98:0/114957656 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.613+0000 7fbda659c640 1 --2- 172.21.15.98:0/114957656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbda8100e20 0x7fbda8199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:59.864 INFO:teuthology.orchestra.run.smithi098.stdout:68719476802 2024-09-19T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.614+0000 7fbda5d9b640 1 --2- 172.21.15.98:0/114957656 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbda8102570 0x7fbda8199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.614+0000 7fbda6d9d640 1 -- 172.21.15.98:0/114957656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbda8100e20 msgr2=0x7fbda8199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.614+0000 7fbda6d9d640 1 --2- 172.21.15.98:0/114957656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbda8100e20 0x7fbda8199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.614+0000 7fbda6d9d640 1 -- 172.21.15.98:0/114957656 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbda8102570 msgr2=0x7fbda8199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.614+0000 7fbda6d9d640 1 --2- 172.21.15.98:0/114957656 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbda8102570 0x7fbda8199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.614+0000 7fbda6d9d640 1 -- 172.21.15.98:0/114957656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda81a48f0 con 0x7fbda8102ef0 2024-09-19T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.614+0000 7fbda6d9d640 1 --2- 172.21.15.98:0/114957656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda8102ef0 0x7fbda81a43b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbda000eaa0 tx=0x7fbda000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.614+0000 7fbd977fe640 1 -- 172.21.15.98:0/114957656 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda000cdb0 con 0x7fbda8102ef0 2024-09-19T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.614+0000 7fbdac818640 1 -- 172.21.15.98:0/114957656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbda81a4ac0 con 0x7fbda8102ef0 2024-09-19T08:01:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.614+0000 7fbd977fe640 1 -- 172.21.15.98:0/114957656 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbda0005020 con 0x7fbda8102ef0 2024-09-19T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.614+0000 7fbd977fe640 1 -- 172.21.15.98:0/114957656 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda00116d0 con 0x7fbda8102ef0 2024-09-19T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.614+0000 7fbdac818640 1 -- 172.21.15.98:0/114957656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbda81a5000 con 0x7fbda8102ef0 2024-09-19T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.616+0000 7fbdac818640 1 -- 172.21.15.98:0/114957656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbda8101200 con 0x7fbda8102ef0 2024-09-19T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.616+0000 7fbd977fe640 1 -- 172.21.15.98:0/114957656 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fbda00054d0 con 0x7fbda8102ef0 2024-09-19T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.617+0000 7fbd977fe640 1 --2- 172.21.15.98:0/114957656 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbd900a32d0 0x7fbd900a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.617+0000 7fbda659c640 1 --2- 172.21.15.98:0/114957656 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbd900a32d0 0x7fbd900a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.617+0000 7fbd977fe640 1 -- 172.21.15.98:0/114957656 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fbda00cfc70 con 0x7fbda8102ef0 2024-09-19T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.617+0000 7fbda659c640 1 --2- 172.21.15.98:0/114957656 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbd900a32d0 0x7fbd900a5790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbd98009230 tx=0x7fbd980058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.622+0000 7fbd977fe640 1 -- 172.21.15.98:0/114957656 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fbda0016030 con 0x7fbda8102ef0 2024-09-19T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.827+0000 7fbdac818640 1 -- 172.21.15.98:0/114957656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fbda8109090 con 0x7fbda8102ef0 2024-09-19T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.827+0000 7fbd977fe640 1 -- 172.21.15.98:0/114957656 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbda009c840 con 0x7fbda8102ef0 2024-09-19T08:01:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.830+0000 7fbdac818640 1 -- 172.21.15.98:0/114957656 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbd900a32d0 msgr2=0x7fbd900a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.830+0000 7fbdac818640 1 --2- 172.21.15.98:0/114957656 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbd900a32d0 0x7fbd900a5790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbd98009230 tx=0x7fbd980058d0 comp rx=0 tx=0).stop 2024-09-19T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.830+0000 7fbdac818640 1 -- 172.21.15.98:0/114957656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda8102ef0 msgr2=0x7fbda81a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.830+0000 7fbdac818640 1 --2- 172.21.15.98:0/114957656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda8102ef0 0x7fbda81a43b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbda000eaa0 tx=0x7fbda000ef60 comp rx=0 tx=0).stop 2024-09-19T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.830+0000 7fbdac818640 1 -- 172.21.15.98:0/114957656 shutdown_connections 2024-09-19T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.830+0000 7fbdac818640 1 --2- 172.21.15.98:0/114957656 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbd900a32d0 0x7fbd900a5790 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.830+0000 7fbdac818640 1 --2- 172.21.15.98:0/114957656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda8102ef0 0x7fbda81a43b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.830+0000 7fbdac818640 1 --2- 172.21.15.98:0/114957656 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbda8102570 0x7fbda8199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.830+0000 7fbdac818640 1 --2- 172.21.15.98:0/114957656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbda8100e20 0x7fbda8199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.830+0000 7fbdac818640 1 -- 172.21.15.98:0/114957656 >> 172.21.15.98:0/114957656 conn(0x7fbda80fc7d0 msgr2=0x7fbda80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.830+0000 7fbdac818640 1 -- 172.21.15.98:0/114957656 shutdown_connections 2024-09-19T08:01:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.830+0000 7fbdac818640 1 -- 172.21.15.98:0/114957656 wait complete. 2024-09-19T08:01:59.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:59 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1885370206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-19T08:01:59.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:01:59 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/114957656' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-19T08:02:00.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:59 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1885370206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-19T08:02:00.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:01:59 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/114957656' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-19T08:02:00.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:59 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1885370206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-19T08:02:00.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:01:59 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/114957656' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-19T08:02:00.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.534+0000 7f178cb05640 1 -- 172.21.15.98:0/290955958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1788102570 msgr2=0x7f17881029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:00.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.534+0000 7f178cb05640 1 --2- 172.21.15.98:0/290955958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1788102570 0x7f17881029b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f177c009900 tx=0x7f177c02a480 comp rx=0 tx=0).stop 2024-09-19T08:02:00.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.534+0000 7f178cb05640 1 -- 172.21.15.98:0/290955958 shutdown_connections 2024-09-19T08:02:00.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.534+0000 7f178cb05640 1 --2- 172.21.15.98:0/290955958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1788102ef0 0x7f178810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:00.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.534+0000 7f178cb05640 1 --2- 172.21.15.98:0/290955958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1788102570 0x7f17881029b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:00.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.534+0000 7f178cb05640 1 --2- 172.21.15.98:0/290955958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1788100e20 0x7f1788101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.534+0000 7f178cb05640 1 -- 172.21.15.98:0/290955958 >> 172.21.15.98:0/290955958 conn(0x7f17880fc7d0 msgr2=0x7f17880febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.534+0000 7f178cb05640 1 -- 172.21.15.98:0/290955958 shutdown_connections 2024-09-19T08:02:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.534+0000 7f178cb05640 1 -- 172.21.15.98:0/290955958 wait complete. 2024-09-19T08:02:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.535+0000 7f178cb05640 1 Processor -- start 2024-09-19T08:02:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.535+0000 7f178cb05640 1 -- start start 2024-09-19T08:02:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.535+0000 7f178cb05640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1788100e20 0x7f1788198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.535+0000 7f178cb05640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1788102570 0x7f1788198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.535+0000 7f178cb05640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1788102ef0 0x7f178819ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.535+0000 7f178cb05640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1788110f80 con 0x7f1788102ef0 2024-09-19T08:02:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.535+0000 7f178cb05640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1788110e00 con 0x7f1788102570 2024-09-19T08:02:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.535+0000 7f178cb05640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1788111100 con 0x7f1788100e20 2024-09-19T08:02:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f1786d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1788102ef0 0x7f178819ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f1786d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1788102ef0 0x7f178819ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34754/0 (socket says 172.21.15.98:34754) 2024-09-19T08:02:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f1786d9d640 1 -- 172.21.15.98:0/1885370206 learned_addr learned my addr 172.21.15.98:0/1885370206 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f1785d9b640 1 --2- 172.21.15.98:0/1885370206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1788102570 0x7f1788198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f178659c640 1 --2- 172.21.15.98:0/1885370206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1788100e20 0x7f1788198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:00.200 INFO:teuthology.orchestra.run.smithi098.stdout:231928234021 2024-09-19T08:02:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f1786d9d640 1 -- 172.21.15.98:0/1885370206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1788100e20 msgr2=0x7f1788198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f1786d9d640 1 --2- 172.21.15.98:0/1885370206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1788100e20 0x7f1788198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f1786d9d640 1 -- 172.21.15.98:0/1885370206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1788102570 msgr2=0x7f1788198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f1786d9d640 1 --2- 172.21.15.98:0/1885370206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1788102570 0x7f1788198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f1786d9d640 1 -- 172.21.15.98:0/1885370206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f178819d4e0 con 0x7f1788102ef0 2024-09-19T08:02:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f1786d9d640 1 --2- 172.21.15.98:0/1885370206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1788102ef0 0x7f178819ce00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f178000eaa0 tx=0x7f178000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f17777fe640 1 -- 172.21.15.98:0/1885370206 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178000cdb0 con 0x7f1788102ef0 2024-09-19T08:02:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f178cb05640 1 -- 172.21.15.98:0/1885370206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f178819d7d0 con 0x7f1788102ef0 2024-09-19T08:02:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f17777fe640 1 -- 172.21.15.98:0/1885370206 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1780005020 con 0x7f1788102ef0 2024-09-19T08:02:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.536+0000 7f17777fe640 1 -- 172.21.15.98:0/1885370206 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17800116b0 con 0x7f1788102ef0 2024-09-19T08:02:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.537+0000 7f178cb05640 1 -- 172.21.15.98:0/1885370206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17880fcbb0 con 0x7f1788102ef0 2024-09-19T08:02:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.538+0000 7f178cb05640 1 -- 172.21.15.98:0/1885370206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17881021b0 con 0x7f1788102ef0 2024-09-19T08:02:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.539+0000 7f17777fe640 1 -- 172.21.15.98:0/1885370206 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1780011850 con 0x7f1788102ef0 2024-09-19T08:02:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.539+0000 7f17777fe640 1 --2- 172.21.15.98:0/1885370206 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f17700a32d0 0x7f17700a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.539+0000 7f17777fe640 1 -- 172.21.15.98:0/1885370206 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f17800cfc30 con 0x7f1788102ef0 2024-09-19T08:02:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.539+0000 7f178659c640 1 --2- 172.21.15.98:0/1885370206 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f17700a32d0 0x7f17700a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.540+0000 7f178659c640 1 --2- 172.21.15.98:0/1885370206 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f17700a32d0 0x7f17700a5790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1778009b60 tx=0x7f1778005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.544+0000 7f17777fe640 1 -- 172.21.15.98:0/1885370206 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f1780016030 con 0x7f1788102ef0 2024-09-19T08:02:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.756+0000 7f178cb05640 1 -- 172.21.15.98:0/1885370206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f178810a0a0 con 0x7f1788102ef0 2024-09-19T08:02:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.756+0000 7f17777fe640 1 -- 172.21.15.98:0/1885370206 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f178009c800 con 0x7f1788102ef0 2024-09-19T08:02:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.759+0000 7f178cb05640 1 -- 172.21.15.98:0/1885370206 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f17700a32d0 msgr2=0x7f17700a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.759+0000 7f178cb05640 1 --2- 172.21.15.98:0/1885370206 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f17700a32d0 0x7f17700a5790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1778009b60 tx=0x7f1778005f90 comp rx=0 tx=0).stop 2024-09-19T08:02:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.759+0000 7f178cb05640 1 -- 172.21.15.98:0/1885370206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1788102ef0 msgr2=0x7f178819ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.759+0000 7f178cb05640 1 --2- 172.21.15.98:0/1885370206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1788102ef0 0x7f178819ce00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f178000eaa0 tx=0x7f178000ef60 comp rx=0 tx=0).stop 2024-09-19T08:02:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.759+0000 7f178cb05640 1 -- 172.21.15.98:0/1885370206 shutdown_connections 2024-09-19T08:02:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.759+0000 7f178cb05640 1 --2- 172.21.15.98:0/1885370206 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f17700a32d0 0x7f17700a5790 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.759+0000 7f178cb05640 1 --2- 172.21.15.98:0/1885370206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1788102ef0 0x7f178819ce00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.759+0000 7f178cb05640 1 --2- 172.21.15.98:0/1885370206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1788102570 0x7f1788198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.759+0000 7f178cb05640 1 --2- 172.21.15.98:0/1885370206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1788100e20 0x7f1788198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.759+0000 7f178cb05640 1 -- 172.21.15.98:0/1885370206 >> 172.21.15.98:0/1885370206 conn(0x7f17880fc7d0 msgr2=0x7f178806a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.759+0000 7f178cb05640 1 -- 172.21.15.98:0/1885370206 shutdown_connections 2024-09-19T08:02:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:01:58.759+0000 7f178cb05640 1 -- 172.21.15.98:0/1885370206 wait complete. 2024-09-19T08:02:00.389 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:00.389 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:00.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:00 smithi098 ceph-mon[31333]: pgmap v297: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-09-19T08:02:01.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:00 smithi162 ceph-mon[35260]: pgmap v297: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-09-19T08:02:01.125 INFO:tasks.cephadm.ceph_manager.ceph:need seq 68719476801 got 68719476802 for osd.2 2024-09-19T08:02:01.126 DEBUG:teuthology.parallel:result is None 2024-09-19T08:02:01.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:00 smithi139 ceph-mon[34874]: pgmap v297: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-09-19T08:02:01.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.281+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/3223597745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec101430 msgr2=0x7ff3ec10d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:01.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.281+0000 7ff3f0dfe640 1 --2- 172.21.15.98:0/3223597745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec101430 0x7ff3ec10d960 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009cd0 tx=0x7ff3e402a8e0 comp rx=0 tx=0).stop 2024-09-19T08:02:01.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.281+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/3223597745 shutdown_connections 2024-09-19T08:02:01.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.281+0000 7ff3f0dfe640 1 --2- 172.21.15.98:0/3223597745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec101430 0x7ff3ec10d960 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:01.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.281+0000 7ff3f0dfe640 1 --2- 172.21.15.98:0/3223597745 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3ec100a90 0x7ff3ec100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:01.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.281+0000 7ff3f0dfe640 1 --2- 172.21.15.98:0/3223597745 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3ec106a90 0x7ff3ec106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:01.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.281+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/3223597745 >> 172.21.15.98:0/3223597745 conn(0x7ff3ec0fc7d0 msgr2=0x7ff3ec0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:01.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.281+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/3223597745 shutdown_connections 2024-09-19T08:02:01.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.281+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/3223597745 wait complete. 2024-09-19T08:02:01.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.281+0000 7ff3f0dfe640 1 Processor -- start 2024-09-19T08:02:01.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.281+0000 7ff3f0dfe640 1 -- start start 2024-09-19T08:02:01.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3f0dfe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3ec100a90 0x7ff3ec198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:01.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3f0dfe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec101430 0x7ff3ec198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:01.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3f0dfe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3ec106a90 0x7ff3ec19ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:01.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3f0dfe640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff3ec110e90 con 0x7ff3ec101430 2024-09-19T08:02:01.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3f0dfe640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff3ec110d10 con 0x7ff3ec106a90 2024-09-19T08:02:01.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3f0dfe640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff3ec111010 con 0x7ff3ec100a90 2024-09-19T08:02:01.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3e3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec101430 0x7ff3ec198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:01.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3e3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec101430 0x7ff3ec198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34768/0 (socket says 172.21.15.98:34768) 2024-09-19T08:02:01.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3e3fff640 1 -- 172.21.15.98:0/1440196379 learned_addr learned my addr 172.21.15.98:0/1440196379 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:01.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3ea59c640 1 --2- 172.21.15.98:0/1440196379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3ec100a90 0x7ff3ec198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3ead9d640 1 --2- 172.21.15.98:0/1440196379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3ec106a90 0x7ff3ec19ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3e3fff640 1 -- 172.21.15.98:0/1440196379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3ec100a90 msgr2=0x7ff3ec198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3e3fff640 1 --2- 172.21.15.98:0/1440196379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3ec100a90 0x7ff3ec198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3e3fff640 1 -- 172.21.15.98:0/1440196379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3ec106a90 msgr2=0x7ff3ec19ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3e3fff640 1 --2- 172.21.15.98:0/1440196379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3ec106a90 0x7ff3ec19ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3e3fff640 1 -- 172.21.15.98:0/1440196379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3ec19d3b0 con 0x7ff3ec101430 2024-09-19T08:02:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.282+0000 7ff3e3fff640 1 --2- 172.21.15.98:0/1440196379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec101430 0x7ff3ec198940 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400c890 tx=0x7ff3d400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.283+0000 7ff3e37fe640 1 -- 172.21.15.98:0/1440196379 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4004220 con 0x7ff3ec101430 2024-09-19T08:02:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.283+0000 7ff3e37fe640 1 -- 172.21.15.98:0/1440196379 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3d40043c0 con 0x7ff3ec101430 2024-09-19T08:02:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.283+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/1440196379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3ec19d6a0 con 0x7ff3ec101430 2024-09-19T08:02:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.283+0000 7ff3e37fe640 1 -- 172.21.15.98:0/1440196379 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d40049e0 con 0x7ff3ec101430 2024-09-19T08:02:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.283+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/1440196379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3ec1a4f80 con 0x7ff3ec101430 2024-09-19T08:02:01.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.284+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/1440196379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3ec100ef0 con 0x7ff3ec101430 2024-09-19T08:02:01.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.285+0000 7ff3e37fe640 1 -- 172.21.15.98:0/1440196379 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ff3d40079b0 con 0x7ff3ec101430 2024-09-19T08:02:01.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.285+0000 7ff3e37fe640 1 --2- 172.21.15.98:0/1440196379 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff3d80a32d0 0x7ff3d80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:01.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.285+0000 7ff3ea59c640 1 --2- 172.21.15.98:0/1440196379 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff3d80a32d0 0x7ff3d80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:01.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.285+0000 7ff3e37fe640 1 -- 172.21.15.98:0/1440196379 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7ff3d40cfaf0 con 0x7ff3ec101430 2024-09-19T08:02:01.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.286+0000 7ff3ea59c640 1 --2- 172.21.15.98:0/1440196379 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff3d80a32d0 0x7ff3d80a5790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc005fd0 tx=0x7ff3dc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:01.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.290+0000 7ff3e37fe640 1 -- 172.21.15.98:0/1440196379 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ff3d4014030 con 0x7ff3ec101430 2024-09-19T08:02:01.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.458+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/1440196379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7ff3ec1a5240 con 0x7ff3ec101430 2024-09-19T08:02:01.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.459+0000 7ff3e37fe640 1 -- 172.21.15.98:0/1440196379 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ff3d409c6c0 con 0x7ff3ec101430 2024-09-19T08:02:01.459 INFO:teuthology.orchestra.run.smithi098.stdout:279172874268 2024-09-19T08:02:01.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.461+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/1440196379 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff3d80a32d0 msgr2=0x7ff3d80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:01.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.461+0000 7ff3f0dfe640 1 --2- 172.21.15.98:0/1440196379 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff3d80a32d0 0x7ff3d80a5790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc005fd0 tx=0x7ff3dc00b390 comp rx=0 tx=0).stop 2024-09-19T08:02:01.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.461+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/1440196379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec101430 msgr2=0x7ff3ec198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:01.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.461+0000 7ff3f0dfe640 1 --2- 172.21.15.98:0/1440196379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec101430 0x7ff3ec198940 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400c890 tx=0x7ff3d400cd50 comp rx=0 tx=0).stop 2024-09-19T08:02:01.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.461+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/1440196379 shutdown_connections 2024-09-19T08:02:01.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.461+0000 7ff3f0dfe640 1 --2- 172.21.15.98:0/1440196379 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7ff3d80a32d0 0x7ff3d80a5790 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:01.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.461+0000 7ff3f0dfe640 1 --2- 172.21.15.98:0/1440196379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3ec106a90 0x7ff3ec19ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:01.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.461+0000 7ff3f0dfe640 1 --2- 172.21.15.98:0/1440196379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec101430 0x7ff3ec198940 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:01.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.461+0000 7ff3f0dfe640 1 --2- 172.21.15.98:0/1440196379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3ec100a90 0x7ff3ec198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:01.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.461+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/1440196379 >> 172.21.15.98:0/1440196379 conn(0x7ff3ec0fc7d0 msgr2=0x7ff3ec104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:01.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.461+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/1440196379 shutdown_connections 2024-09-19T08:02:01.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:01.461+0000 7ff3f0dfe640 1 -- 172.21.15.98:0/1440196379 wait complete. 2024-09-19T08:02:02.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:01 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1440196379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-19T08:02:02.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:01 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1440196379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-19T08:02:02.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:01 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1440196379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-19T08:02:02.221 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874265 got 279172874268 for osd.10 2024-09-19T08:02:02.221 DEBUG:teuthology.parallel:result is None 2024-09-19T08:02:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.846+0000 7f701f65a640 1 -- 172.21.15.98:0/4250735007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7018101df0 msgr2=0x7f70181021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.846+0000 7f701f65a640 1 --2- 172.21.15.98:0/4250735007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7018101df0 0x7f70181021d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7014009900 tx=0x7f701402a410 comp rx=0 tx=0).stop 2024-09-19T08:02:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.846+0000 7f701f65a640 1 -- 172.21.15.98:0/4250735007 shutdown_connections 2024-09-19T08:02:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.846+0000 7f701f65a640 1 --2- 172.21.15.98:0/4250735007 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7018106f20 0x7f701810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.846+0000 7f701f65a640 1 --2- 172.21.15.98:0/4250735007 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70181027a0 0x7f70181067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.846+0000 7f701f65a640 1 --2- 172.21.15.98:0/4250735007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7018101df0 0x7f70181021d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.846+0000 7f701f65a640 1 -- 172.21.15.98:0/4250735007 >> 172.21.15.98:0/4250735007 conn(0x7f70180fd8b0 msgr2=0x7f70180ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.846+0000 7f701f65a640 1 -- 172.21.15.98:0/4250735007 shutdown_connections 2024-09-19T08:02:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.846+0000 7f701f65a640 1 -- 172.21.15.98:0/4250735007 wait complete. 2024-09-19T08:02:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.847+0000 7f701f65a640 1 Processor -- start 2024-09-19T08:02:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.847+0000 7f701f65a640 1 -- start start 2024-09-19T08:02:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.847+0000 7f701f65a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7018101df0 0x7f70181984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.847+0000 7f701f65a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70181027a0 0x7f70181989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.847+0000 7f701f65a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7018106f20 0x7f701819cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.847+0000 7f701f65a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7018111fa0 con 0x7f7018106f20 2024-09-19T08:02:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.847+0000 7f701f65a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7018111e20 con 0x7f70181027a0 2024-09-19T08:02:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.847+0000 7f701f65a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7018112120 con 0x7f7018101df0 2024-09-19T08:02:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.847+0000 7f701dbf7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7018106f20 0x7f701819cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.848+0000 7f701dbf7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7018106f20 0x7f701819cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34782/0 (socket says 172.21.15.98:34782) 2024-09-19T08:02:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.848+0000 7f701dbf7640 1 -- 172.21.15.98:0/993593349 learned_addr learned my addr 172.21.15.98:0/993593349 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.848+0000 7f701dbf7640 1 -- 172.21.15.98:0/993593349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7018101df0 msgr2=0x7f70181984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.848+0000 7f701dbf7640 1 --2- 172.21.15.98:0/993593349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7018101df0 0x7f70181984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.848+0000 7f701dbf7640 1 -- 172.21.15.98:0/993593349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70181027a0 msgr2=0x7f70181989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.848+0000 7f701dbf7640 1 --2- 172.21.15.98:0/993593349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70181027a0 0x7f70181989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.848+0000 7f701dbf7640 1 -- 172.21.15.98:0/993593349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701819d450 con 0x7f7018106f20 2024-09-19T08:02:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.848+0000 7f701dbf7640 1 --2- 172.21.15.98:0/993593349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7018106f20 0x7f701819cd70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f701000b970 tx=0x7f701000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.848+0000 7f700e7fc640 1 -- 172.21.15.98:0/993593349 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7010019070 con 0x7f7018106f20 2024-09-19T08:02:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.848+0000 7f701f65a640 1 -- 172.21.15.98:0/993593349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f701819d740 con 0x7f7018106f20 2024-09-19T08:02:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.848+0000 7f700e7fc640 1 -- 172.21.15.98:0/993593349 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f701000cb40 con 0x7f7018106f20 2024-09-19T08:02:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.848+0000 7f700e7fc640 1 -- 172.21.15.98:0/993593349 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7010007650 con 0x7f7018106f20 2024-09-19T08:02:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.848+0000 7f701f65a640 1 -- 172.21.15.98:0/993593349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70181a5020 con 0x7f7018106f20 2024-09-19T08:02:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.849+0000 7f701f65a640 1 -- 172.21.15.98:0/993593349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70181021d0 con 0x7f7018106f20 2024-09-19T08:02:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.850+0000 7f700e7fc640 1 -- 172.21.15.98:0/993593349 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f701001c070 con 0x7f7018106f20 2024-09-19T08:02:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.851+0000 7f700e7fc640 1 --2- 172.21.15.98:0/993593349 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f70040a32d0 0x7f70040a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.851+0000 7f701d3f6640 1 --2- 172.21.15.98:0/993593349 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f70040a32d0 0x7f70040a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.851+0000 7f700e7fc640 1 -- 172.21.15.98:0/993593349 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f70100cef90 con 0x7f7018106f20 2024-09-19T08:02:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.851+0000 7f701d3f6640 1 --2- 172.21.15.98:0/993593349 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f70040a32d0 0x7f70040a5790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f701402a930 tx=0x7f7014000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:02.855+0000 7f700e7fc640 1 -- 172.21.15.98:0/993593349 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f701009bd10 con 0x7f7018106f20 2024-09-19T08:02:03.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.027+0000 7f701f65a640 1 -- 172.21.15.98:0/993593349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f7018109b70 con 0x7f7018106f20 2024-09-19T08:02:03.028 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:02 smithi098 ceph-mon[31333]: pgmap v298: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-09-19T08:02:03.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.028+0000 7f700e7fc640 1 -- 172.21.15.98:0/993593349 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f701009b610 con 0x7f7018106f20 2024-09-19T08:02:03.028 INFO:teuthology.orchestra.run.smithi098.stdout:261993005089 2024-09-19T08:02:03.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.031+0000 7f701f65a640 1 -- 172.21.15.98:0/993593349 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f70040a32d0 msgr2=0x7f70040a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:03.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.031+0000 7f701f65a640 1 --2- 172.21.15.98:0/993593349 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f70040a32d0 0x7f70040a5790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f701402a930 tx=0x7f7014000bc0 comp rx=0 tx=0).stop 2024-09-19T08:02:03.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.031+0000 7f701f65a640 1 -- 172.21.15.98:0/993593349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7018106f20 msgr2=0x7f701819cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:03.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.031+0000 7f701f65a640 1 --2- 172.21.15.98:0/993593349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7018106f20 0x7f701819cd70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f701000b970 tx=0x7f701000be30 comp rx=0 tx=0).stop 2024-09-19T08:02:03.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.031+0000 7f701f65a640 1 -- 172.21.15.98:0/993593349 shutdown_connections 2024-09-19T08:02:03.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.031+0000 7f701f65a640 1 --2- 172.21.15.98:0/993593349 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f70040a32d0 0x7f70040a5790 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:03.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.031+0000 7f701f65a640 1 --2- 172.21.15.98:0/993593349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7018106f20 0x7f701819cd70 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:03.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.031+0000 7f701f65a640 1 --2- 172.21.15.98:0/993593349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70181027a0 0x7f70181989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:03.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.031+0000 7f701f65a640 1 --2- 172.21.15.98:0/993593349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7018101df0 0x7f70181984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:03.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.031+0000 7f701f65a640 1 -- 172.21.15.98:0/993593349 >> 172.21.15.98:0/993593349 conn(0x7f70180fd8b0 msgr2=0x7f70180feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:03.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.031+0000 7f701f65a640 1 -- 172.21.15.98:0/993593349 shutdown_connections 2024-09-19T08:02:03.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.031+0000 7f701f65a640 1 -- 172.21.15.98:0/993593349 wait complete. 2024-09-19T08:02:03.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.055+0000 7f7295a1e640 1 -- 172.21.15.98:0/4143042146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290106a90 msgr2=0x7f7290106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:03.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.055+0000 7f7295a1e640 1 --2- 172.21.15.98:0/4143042146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290106a90 0x7f7290106e70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7280009930 tx=0x7f728002a420 comp rx=0 tx=0).stop 2024-09-19T08:02:03.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.055+0000 7f7295a1e640 1 -- 172.21.15.98:0/4143042146 shutdown_connections 2024-09-19T08:02:03.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.055+0000 7f7295a1e640 1 --2- 172.21.15.98:0/4143042146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7290101430 0x7f729010d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:03.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.055+0000 7f7295a1e640 1 --2- 172.21.15.98:0/4143042146 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7290100a90 0x7f7290100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:03.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.055+0000 7f7295a1e640 1 --2- 172.21.15.98:0/4143042146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290106a90 0x7f7290106e70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:03.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.055+0000 7f7295a1e640 1 -- 172.21.15.98:0/4143042146 >> 172.21.15.98:0/4143042146 conn(0x7f72900fc7d0 msgr2=0x7f72900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:03.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.055+0000 7f7295a1e640 1 -- 172.21.15.98:0/4143042146 shutdown_connections 2024-09-19T08:02:03.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.055+0000 7f7295a1e640 1 -- 172.21.15.98:0/4143042146 wait complete. 2024-09-19T08:02:03.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.056+0000 7f7295a1e640 1 Processor -- start 2024-09-19T08:02:03.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.056+0000 7f7295a1e640 1 -- start start 2024-09-19T08:02:03.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.056+0000 7f7295a1e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7290100a90 0x7f72901984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:03.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.056+0000 7f7295a1e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7290101430 0x7f7290198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:03.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.056+0000 7f7295a1e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290106a90 0x7f729019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:03.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.056+0000 7f7295a1e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7290110eb0 con 0x7f7290106a90 2024-09-19T08:02:03.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.056+0000 7f7295a1e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7290110d30 con 0x7f7290101430 2024-09-19T08:02:03.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.056+0000 7f7295a1e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7290111030 con 0x7f7290100a90 2024-09-19T08:02:03.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.056+0000 7f728f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290106a90 0x7f729019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:03.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.056+0000 7f728f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290106a90 0x7f729019cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34794/0 (socket says 172.21.15.98:34794) 2024-09-19T08:02:03.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.056+0000 7f728f7fe640 1 -- 172.21.15.98:0/1183455305 learned_addr learned my addr 172.21.15.98:0/1183455305 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:03.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.057+0000 7f728effd640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7290100a90 0x7f72901984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:03.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.057+0000 7f728e7fc640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7290101430 0x7f7290198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:03.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.057+0000 7f728f7fe640 1 -- 172.21.15.98:0/1183455305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7290100a90 msgr2=0x7f72901984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:03.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.057+0000 7f728f7fe640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7290100a90 0x7f72901984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:03.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.057+0000 7f728f7fe640 1 -- 172.21.15.98:0/1183455305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7290101430 msgr2=0x7f7290198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:03.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.057+0000 7f728f7fe640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7290101430 0x7f7290198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:03.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.057+0000 7f728f7fe640 1 -- 172.21.15.98:0/1183455305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729019d480 con 0x7f7290106a90 2024-09-19T08:02:03.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.057+0000 7f728e7fc640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7290101430 0x7f7290198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:03.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.057+0000 7f728f7fe640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290106a90 0x7f729019cda0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f728800b970 tx=0x7f728800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:03.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.057+0000 7f7294a1c640 1 -- 172.21.15.98:0/1183455305 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7288019070 con 0x7f7290106a90 2024-09-19T08:02:03.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.057+0000 7f7294a1c640 1 -- 172.21.15.98:0/1183455305 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f728800cb40 con 0x7f7290106a90 2024-09-19T08:02:03.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.057+0000 7f7294a1c640 1 -- 172.21.15.98:0/1183455305 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7288007650 con 0x7f7290106a90 2024-09-19T08:02:03.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.058+0000 7f7295a1e640 1 -- 172.21.15.98:0/1183455305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f729019d770 con 0x7f7290106a90 2024-09-19T08:02:03.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.058+0000 7f7295a1e640 1 -- 172.21.15.98:0/1183455305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72901a5050 con 0x7f7290106a90 2024-09-19T08:02:03.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.059+0000 7f7295a1e640 1 -- 172.21.15.98:0/1183455305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7290100ef0 con 0x7f7290106a90 2024-09-19T08:02:03.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.060+0000 7f7294a1c640 1 -- 172.21.15.98:0/1183455305 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f728801c070 con 0x7f7290106a90 2024-09-19T08:02:03.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.061+0000 7f7294a1c640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f72780a32d0 0x7f72780a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:03.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.061+0000 7f728effd640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f72780a32d0 0x7f72780a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:03.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.061+0000 7f7294a1c640 1 -- 172.21.15.98:0/1183455305 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f72880cf020 con 0x7f7290106a90 2024-09-19T08:02:03.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.061+0000 7f728effd640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f72780a32d0 0x7f72780a5790 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7280000c00 tx=0x7f7280031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:03.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.067+0000 7f7294a1c640 1 -- 172.21.15.98:0/1183455305 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f728809bd00 con 0x7f7290106a90 2024-09-19T08:02:03.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:02 smithi162 ceph-mon[35260]: pgmap v298: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-09-19T08:02:03.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:02 smithi139 ceph-mon[34874]: pgmap v298: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-09-19T08:02:03.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.235+0000 7f7295a1e640 1 -- 172.21.15.98:0/1183455305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f7290109a80 con 0x7f7290106a90 2024-09-19T08:02:03.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.236+0000 7f7294a1c640 1 -- 172.21.15.98:0/1183455305 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f728809b600 con 0x7f7290106a90 2024-09-19T08:02:03.236 INFO:teuthology.orchestra.run.smithi098.stdout:51539607624 2024-09-19T08:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.238+0000 7f7295a1e640 1 -- 172.21.15.98:0/1183455305 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f72780a32d0 msgr2=0x7f72780a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.238+0000 7f7295a1e640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f72780a32d0 0x7f72780a5790 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7280000c00 tx=0x7f7280031490 comp rx=0 tx=0).stop 2024-09-19T08:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.238+0000 7f7295a1e640 1 -- 172.21.15.98:0/1183455305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290106a90 msgr2=0x7f729019cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.238+0000 7f7295a1e640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290106a90 0x7f729019cda0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f728800b970 tx=0x7f728800be30 comp rx=0 tx=0).stop 2024-09-19T08:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.238+0000 7f7295a1e640 1 -- 172.21.15.98:0/1183455305 shutdown_connections 2024-09-19T08:02:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.238+0000 7f7295a1e640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f72780a32d0 0x7f72780a5790 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.238+0000 7f7295a1e640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7290106a90 0x7f729019cda0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.238+0000 7f7295a1e640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7290101430 0x7f7290198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.238+0000 7f7295a1e640 1 --2- 172.21.15.98:0/1183455305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7290100a90 0x7f72901984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.238+0000 7f7295a1e640 1 -- 172.21.15.98:0/1183455305 >> 172.21.15.98:0/1183455305 conn(0x7f72900fc7d0 msgr2=0x7f729006aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.238+0000 7f7295a1e640 1 -- 172.21.15.98:0/1183455305 shutdown_connections 2024-09-19T08:02:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:03.238+0000 7f7295a1e640 1 -- 172.21.15.98:0/1183455305 wait complete. 2024-09-19T08:02:03.601 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234020 got 231928234021 for osd.8 2024-09-19T08:02:03.602 DEBUG:teuthology.parallel:result is None 2024-09-19T08:02:03.892 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:03 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/993593349' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-19T08:02:03.892 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:03 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1183455305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-19T08:02:03.892 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:03 smithi098 ceph-mon[31333]: pgmap v299: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:04.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:03 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/993593349' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-19T08:02:04.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:03 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1183455305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-19T08:02:04.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:03 smithi139 ceph-mon[34874]: pgmap v299: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:04.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:03 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/993593349' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-19T08:02:04.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:03 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1183455305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-19T08:02:04.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:03 smithi162 ceph-mon[35260]: pgmap v299: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:04.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.352+0000 7f1213e48640 1 -- 172.21.15.98:0/4172502014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c100e20 msgr2=0x7f120c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:04.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.352+0000 7f1213e48640 1 --2- 172.21.15.98:0/4172502014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c100e20 0x7f120c101200 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1200009900 tx=0x7f120002a480 comp rx=0 tx=0).stop 2024-09-19T08:02:04.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.352+0000 7f1213e48640 1 -- 172.21.15.98:0/4172502014 shutdown_connections 2024-09-19T08:02:04.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.352+0000 7f1213e48640 1 --2- 172.21.15.98:0/4172502014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f120c102ef0 0x7f120c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:04.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.352+0000 7f1213e48640 1 --2- 172.21.15.98:0/4172502014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f120c102570 0x7f120c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:04.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.352+0000 7f1213e48640 1 --2- 172.21.15.98:0/4172502014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c100e20 0x7f120c101200 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:04.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.352+0000 7f1213e48640 1 -- 172.21.15.98:0/4172502014 >> 172.21.15.98:0/4172502014 conn(0x7f120c0fc7d0 msgr2=0x7f120c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.352+0000 7f1213e48640 1 -- 172.21.15.98:0/4172502014 shutdown_connections 2024-09-19T08:02:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.352+0000 7f1213e48640 1 -- 172.21.15.98:0/4172502014 wait complete. 2024-09-19T08:02:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.352+0000 7f1213e48640 1 Processor -- start 2024-09-19T08:02:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.352+0000 7f1213e48640 1 -- start start 2024-09-19T08:02:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1213e48640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c100e20 0x7f120c199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1213e48640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f120c102570 0x7f120c199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1213e48640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f120c102ef0 0x7f120c1a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1213e48640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f120c110fa0 con 0x7f120c100e20 2024-09-19T08:02:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1213e48640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f120c110e20 con 0x7f120c102570 2024-09-19T08:02:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1213e48640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f120c111120 con 0x7f120c102ef0 2024-09-19T08:02:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1211be4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c100e20 0x7f120c199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1211be4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c100e20 0x7f120c199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34806/0 (socket says 172.21.15.98:34806) 2024-09-19T08:02:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1211be4640 1 -- 172.21.15.98:0/3490678318 learned_addr learned my addr 172.21.15.98:0/3490678318 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f12113e3640 1 --2- 172.21.15.98:0/3490678318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f120c102570 0x7f120c199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f12123e5640 1 --2- 172.21.15.98:0/3490678318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f120c102ef0 0x7f120c1a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1211be4640 1 -- 172.21.15.98:0/3490678318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f120c102ef0 msgr2=0x7f120c1a43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1211be4640 1 --2- 172.21.15.98:0/3490678318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f120c102ef0 0x7f120c1a43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1211be4640 1 -- 172.21.15.98:0/3490678318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f120c102570 msgr2=0x7f120c199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1211be4640 1 --2- 172.21.15.98:0/3490678318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f120c102570 0x7f120c199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1211be4640 1 -- 172.21.15.98:0/3490678318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120c1a48f0 con 0x7f120c100e20 2024-09-19T08:02:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1211be4640 1 --2- 172.21.15.98:0/3490678318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c100e20 0x7f120c199330 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1200006010 tx=0x7f1200003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f11feffd640 1 -- 172.21.15.98:0/3490678318 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120003f070 con 0x7f120c100e20 2024-09-19T08:02:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1213e48640 1 -- 172.21.15.98:0/3490678318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f120c1a4ac0 con 0x7f120c100e20 2024-09-19T08:02:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f11feffd640 1 -- 172.21.15.98:0/3490678318 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1200031e30 con 0x7f120c100e20 2024-09-19T08:02:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f11feffd640 1 -- 172.21.15.98:0/3490678318 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1200035030 con 0x7f120c100e20 2024-09-19T08:02:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.353+0000 7f1213e48640 1 -- 172.21.15.98:0/3490678318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f120c1a4f10 con 0x7f120c100e20 2024-09-19T08:02:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.354+0000 7f1213e48640 1 -- 172.21.15.98:0/3490678318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f120c107260 con 0x7f120c100e20 2024-09-19T08:02:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.355+0000 7f11feffd640 1 -- 172.21.15.98:0/3490678318 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1200042070 con 0x7f120c100e20 2024-09-19T08:02:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.356+0000 7f11feffd640 1 --2- 172.21.15.98:0/3490678318 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f11f80a32d0 0x7f11f80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.356+0000 7f12113e3640 1 --2- 172.21.15.98:0/3490678318 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f11f80a32d0 0x7f11f80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.356+0000 7f11feffd640 1 -- 172.21.15.98:0/3490678318 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f12000f0aa0 con 0x7f120c100e20 2024-09-19T08:02:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.356+0000 7f11feffd640 1 --2- 172.21.15.98:0/3490678318 >> [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014] conn(0x7f11f80ace80 0x7f11f80af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.356+0000 7f11feffd640 1 -- 172.21.15.98:0/3490678318 --> [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1200038e90 con 0x7f11f80ace80 2024-09-19T08:02:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.356+0000 7f11feffd640 1 -- 172.21.15.98:0/3490678318 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f12000f0ef0 con 0x7f120c100e20 2024-09-19T08:02:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.356+0000 7f12123e5640 1 --2- 172.21.15.98:0/3490678318 >> [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014] conn(0x7f11f80ace80 0x7f11f80af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.357+0000 7f12113e3640 1 --2- 172.21.15.98:0/3490678318 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f11f80a32d0 0x7f11f80a5790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f120c06aac0 tx=0x7f1208005cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.357+0000 7f12123e5640 1 --2- 172.21.15.98:0/3490678318 >> [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014] conn(0x7f11f80ace80 0x7f11f80af320 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.357+0000 7f11feffd640 1 -- 172.21.15.98:0/3490678318 <== osd.3 v2:172.21.15.98:6810/1383392014 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1200038e90 con 0x7f11f80ace80 2024-09-19T08:02:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.371+0000 7f1213e48640 1 -- 172.21.15.98:0/3490678318 --> [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f120c1021b0 con 0x7f11f80ace80 2024-09-19T08:02:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.371+0000 7f11feffd640 1 -- 172.21.15.98:0/3490678318 <== osd.3 v2:172.21.15.98:6810/1383392014 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f120c1021b0 con 0x7f11f80ace80 2024-09-19T08:02:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.371+0000 7f1213e48640 1 -- 172.21.15.98:0/3490678318 >> [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014] conn(0x7f11f80ace80 msgr2=0x7f11f80af320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.371+0000 7f1213e48640 1 --2- 172.21.15.98:0/3490678318 >> [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014] conn(0x7f11f80ace80 0x7f11f80af320 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.371+0000 7f1213e48640 1 -- 172.21.15.98:0/3490678318 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f11f80a32d0 msgr2=0x7f11f80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.371+0000 7f1213e48640 1 --2- 172.21.15.98:0/3490678318 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f11f80a32d0 0x7f11f80a5790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f120c06aac0 tx=0x7f1208005cc0 comp rx=0 tx=0).stop 2024-09-19T08:02:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.371+0000 7f1213e48640 1 -- 172.21.15.98:0/3490678318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c100e20 msgr2=0x7f120c199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.371+0000 7f1213e48640 1 --2- 172.21.15.98:0/3490678318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f120c100e20 0x7f120c199330 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1200006010 tx=0x7f1200003c80 comp rx=0 tx=0).stop 2024-09-19T08:02:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.372+0000 7f12123e5640 1 -- 172.21.15.98:0/3490678318 reap_dead start 2024-09-19T08:02:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.372+0000 7f1213e48640 1 -- 172.21.15.98:0/3490678318 shutdown_connections 2024-09-19T08:02:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.372+0000 7f1213e48640 1 -- 172.21.15.98:0/3490678318 >> 172.21.15.98:0/3490678318 conn(0x7f120c0fc7d0 msgr2=0x7f120c106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.372+0000 7f1213e48640 1 -- 172.21.15.98:0/3490678318 shutdown_connections 2024-09-19T08:02:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.372+0000 7f1213e48640 1 -- 172.21.15.98:0/3490678318 wait complete. 2024-09-19T08:02:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.066+0000 7f60108e8640 1 -- 172.21.15.98:0/490771174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c102db0 msgr2=0x7f600c103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.066+0000 7f60108e8640 1 --2- 172.21.15.98:0/490771174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c102db0 0x7f600c103190 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009930 tx=0x7f5ffc02a440 comp rx=0 tx=0).stop 2024-09-19T08:02:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.066+0000 7f60108e8640 1 -- 172.21.15.98:0/490771174 shutdown_connections 2024-09-19T08:02:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.066+0000 7f60108e8640 1 --2- 172.21.15.98:0/490771174 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f600c1082e0 0x7f600c10ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.066+0000 7f60108e8640 1 --2- 172.21.15.98:0/490771174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f600c103760 0x7f600c107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.066+0000 7f60108e8640 1 --2- 172.21.15.98:0/490771174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c102db0 0x7f600c103190 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.066+0000 7f60108e8640 1 -- 172.21.15.98:0/490771174 >> 172.21.15.98:0/490771174 conn(0x7f600c0fc7d0 msgr2=0x7f600c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.067+0000 7f60108e8640 1 -- 172.21.15.98:0/490771174 shutdown_connections 2024-09-19T08:02:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.067+0000 7f60108e8640 1 -- 172.21.15.98:0/490771174 wait complete. 2024-09-19T08:02:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.067+0000 7f60108e8640 1 Processor -- start 2024-09-19T08:02:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.067+0000 7f60108e8640 1 -- start start 2024-09-19T08:02:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.067+0000 7f60108e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c102db0 0x7f600c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f60108e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f600c103760 0x7f600c198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f60108e8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f600c1082e0 0x7f600c19cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f60108e8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f600c1120f0 con 0x7f600c102db0 2024-09-19T08:02:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f60108e8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f600c111f70 con 0x7f600c1082e0 2024-09-19T08:02:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f60108e8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f600c112270 con 0x7f600c103760 2024-09-19T08:02:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f600a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c102db0 0x7f600c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f600a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c102db0 0x7f600c198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34838/0 (socket says 172.21.15.98:34838) 2024-09-19T08:02:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f600a59c640 1 -- 172.21.15.98:0/1021837581 learned_addr learned my addr 172.21.15.98:0/1021837581 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f600ad9d640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f600c1082e0 0x7f600c19cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f6009d9b640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f600c103760 0x7f600c198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f600a59c640 1 -- 172.21.15.98:0/1021837581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f600c103760 msgr2=0x7f600c198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f600a59c640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f600c103760 0x7f600c198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f600a59c640 1 -- 172.21.15.98:0/1021837581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f600c1082e0 msgr2=0x7f600c19cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f600a59c640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f600c1082e0 0x7f600c19cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f600a59c640 1 -- 172.21.15.98:0/1021837581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600c19d5e0 con 0x7f600c102db0 2024-09-19T08:02:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f600ad9d640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f600c1082e0 0x7f600c19cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f600a59c640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c102db0 0x7f600c198630 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc004770 tx=0x7f5ffc003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f60037fe640 1 -- 172.21.15.98:0/1021837581 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc03f070 con 0x7f600c102db0 2024-09-19T08:02:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.068+0000 7f60037fe640 1 -- 172.21.15.98:0/1021837581 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ffc031c20 con 0x7f600c102db0 2024-09-19T08:02:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.069+0000 7f60108e8640 1 -- 172.21.15.98:0/1021837581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f600c19d870 con 0x7f600c102db0 2024-09-19T08:02:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.069+0000 7f60037fe640 1 -- 172.21.15.98:0/1021837581 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc035030 con 0x7f600c102db0 2024-09-19T08:02:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.069+0000 7f60108e8640 1 -- 172.21.15.98:0/1021837581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f600c1a51b0 con 0x7f600c102db0 2024-09-19T08:02:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.070+0000 7f60108e8640 1 -- 172.21.15.98:0/1021837581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f600c06a7d0 con 0x7f600c102db0 2024-09-19T08:02:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.072+0000 7f60037fe640 1 -- 172.21.15.98:0/1021837581 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f5ffc042070 con 0x7f600c102db0 2024-09-19T08:02:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.072+0000 7f60037fe640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5ff80a32d0 0x7f5ff80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.072+0000 7f6009d9b640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5ff80a32d0 0x7f5ff80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.072+0000 7f60037fe640 1 -- 172.21.15.98:0/1021837581 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f5ffc0f0350 con 0x7f600c102db0 2024-09-19T08:02:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.073+0000 7f6009d9b640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5ff80a32d0 0x7f5ff80a5790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f600c0fed50 tx=0x7f5ff4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.077+0000 7f60037fe640 1 -- 172.21.15.98:0/1021837581 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f5ffc0bcf20 con 0x7f600c102db0 2024-09-19T08:02:05.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.966+0000 7fdfcd14d640 1 -- 172.21.15.98:0/3841133275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfc8101420 msgr2=0x7fdfc810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:05.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.966+0000 7fdfcd14d640 1 --2- 172.21.15.98:0/3841133275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfc8101420 0x7fdfc810d950 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009930 tx=0x7fdfc002a420 comp rx=0 tx=0).stop 2024-09-19T08:02:05.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.966+0000 7fdfcd14d640 1 -- 172.21.15.98:0/3841133275 shutdown_connections 2024-09-19T08:02:05.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.966+0000 7fdfcd14d640 1 --2- 172.21.15.98:0/3841133275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfc8101420 0x7fdfc810d950 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.966+0000 7fdfcd14d640 1 --2- 172.21.15.98:0/3841133275 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdfc8100a80 0x7fdfc8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.966+0000 7fdfcd14d640 1 --2- 172.21.15.98:0/3841133275 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfc8106a80 0x7fdfc8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.966+0000 7fdfcd14d640 1 -- 172.21.15.98:0/3841133275 >> 172.21.15.98:0/3841133275 conn(0x7fdfc80fc7a0 msgr2=0x7fdfc80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:05.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.966+0000 7fdfcd14d640 1 -- 172.21.15.98:0/3841133275 shutdown_connections 2024-09-19T08:02:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.966+0000 7fdfcd14d640 1 -- 172.21.15.98:0/3841133275 wait complete. 2024-09-19T08:02:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.966+0000 7fdfcd14d640 1 Processor -- start 2024-09-19T08:02:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfcd14d640 1 -- start start 2024-09-19T08:02:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfcd14d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdfc8100a80 0x7fdfc81983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfcd14d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfc8101420 0x7fdfc81988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfcd14d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfc8106a80 0x7fdfc819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfcd14d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc8110e80 con 0x7fdfc8101420 2024-09-19T08:02:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfcd14d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc8110d00 con 0x7fdfc8100a80 2024-09-19T08:02:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfcd14d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc8111000 con 0x7fdfc8106a80 2024-09-19T08:02:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfc659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfc8101420 0x7fdfc81988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfc659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfc8101420 0x7fdfc81988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34816/0 (socket says 172.21.15.98:34816) 2024-09-19T08:02:05.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfc659c640 1 -- 172.21.15.98:0/1108871096 learned_addr learned my addr 172.21.15.98:0/1108871096 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:05.167 INFO:teuthology.orchestra.run.smithi098.stdout:184683593776 2024-09-19T08:02:05.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfc6d9d640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdfc8100a80 0x7fdfc81983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:05.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfc759e640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfc8106a80 0x7fdfc819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:05.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfc659c640 1 -- 172.21.15.98:0/1108871096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfc8106a80 msgr2=0x7fdfc819cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:05.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfc659c640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfc8106a80 0x7fdfc819cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfc659c640 1 -- 172.21.15.98:0/1108871096 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdfc8100a80 msgr2=0x7fdfc81983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:05.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfc659c640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdfc8100a80 0x7fdfc81983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfc659c640 1 -- 172.21.15.98:0/1108871096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc819d360 con 0x7fdfc8101420 2024-09-19T08:02:05.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.967+0000 7fdfc759e640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfc8106a80 0x7fdfc819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:05.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.968+0000 7fdfc659c640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfc8101420 0x7fdfc81988f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdfb800d950 tx=0x7fdfb800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:05.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.968+0000 7fdfaffff640 1 -- 172.21.15.98:0/1108871096 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb8017070 con 0x7fdfc8101420 2024-09-19T08:02:05.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.968+0000 7fdfcd14d640 1 -- 172.21.15.98:0/1108871096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfc819d650 con 0x7fdfc8101420 2024-09-19T08:02:05.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.968+0000 7fdfaffff640 1 -- 172.21.15.98:0/1108871096 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfb8009c30 con 0x7fdfc8101420 2024-09-19T08:02:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.968+0000 7fdfaffff640 1 -- 172.21.15.98:0/1108871096 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb800f6b0 con 0x7fdfc8101420 2024-09-19T08:02:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.968+0000 7fdfcd14d640 1 -- 172.21.15.98:0/1108871096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfc81a4f30 con 0x7fdfc8101420 2024-09-19T08:02:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.969+0000 7fdfcd14d640 1 -- 172.21.15.98:0/1108871096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfc8100ee0 con 0x7fdfc8101420 2024-09-19T08:02:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.970+0000 7fdfaffff640 1 -- 172.21.15.98:0/1108871096 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fdfb80109d0 con 0x7fdfc8101420 2024-09-19T08:02:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.971+0000 7fdfaffff640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdfb00a32d0 0x7fdfb00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.971+0000 7fdfc6d9d640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdfb00a32d0 0x7fdfb00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.971+0000 7fdfaffff640 1 -- 172.21.15.98:0/1108871096 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fdfb80cf720 con 0x7fdfc8101420 2024-09-19T08:02:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.971+0000 7fdfc6d9d640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdfb00a32d0 0x7fdfb00a5790 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc005fd0 tx=0x7fdfbc009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:04.975+0000 7fdfaffff640 1 -- 172.21.15.98:0/1108871096 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fdfb8009de0 con 0x7fdfc8101420 2024-09-19T08:02:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.149+0000 7fdfcd14d640 1 -- 172.21.15.98:0/1108871096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fdfc81a51f0 con 0x7fdfc8101420 2024-09-19T08:02:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.150+0000 7fdfaffff640 1 -- 172.21.15.98:0/1108871096 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fdfb809c2f0 con 0x7fdfc8101420 2024-09-19T08:02:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.152+0000 7fdfcd14d640 1 -- 172.21.15.98:0/1108871096 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdfb00a32d0 msgr2=0x7fdfb00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:05.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.152+0000 7fdfcd14d640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdfb00a32d0 0x7fdfb00a5790 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc005fd0 tx=0x7fdfbc009680 comp rx=0 tx=0).stop 2024-09-19T08:02:05.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.152+0000 7fdfcd14d640 1 -- 172.21.15.98:0/1108871096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfc8101420 msgr2=0x7fdfc81988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:05.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.152+0000 7fdfcd14d640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfc8101420 0x7fdfc81988f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdfb800d950 tx=0x7fdfb800de10 comp rx=0 tx=0).stop 2024-09-19T08:02:05.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.152+0000 7fdfcd14d640 1 -- 172.21.15.98:0/1108871096 shutdown_connections 2024-09-19T08:02:05.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.152+0000 7fdfcd14d640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdfb00a32d0 0x7fdfb00a5790 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.152+0000 7fdfcd14d640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfc8106a80 0x7fdfc819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.152+0000 7fdfcd14d640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfc8101420 0x7fdfc81988f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.152+0000 7fdfcd14d640 1 --2- 172.21.15.98:0/1108871096 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdfc8100a80 0x7fdfc81983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.152+0000 7fdfcd14d640 1 -- 172.21.15.98:0/1108871096 >> 172.21.15.98:0/1108871096 conn(0x7fdfc80fc7a0 msgr2=0x7fdfc806a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:05.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.152+0000 7fdfcd14d640 1 -- 172.21.15.98:0/1108871096 shutdown_connections 2024-09-19T08:02:05.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.152+0000 7fdfcd14d640 1 -- 172.21.15.98:0/1108871096 wait complete. 2024-09-19T08:02:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.252+0000 7f60108e8640 1 -- 172.21.15.98:0/1021837581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f600c10a2c0 con 0x7f600c102db0 2024-09-19T08:02:05.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:05 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1108871096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-19T08:02:05.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.253+0000 7f60037fe640 1 -- 172.21.15.98:0/1021837581 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5ffc0bc820 con 0x7f600c102db0 2024-09-19T08:02:05.254 INFO:teuthology.orchestra.run.smithi098.stdout:133143986234 2024-09-19T08:02:05.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.255+0000 7f60108e8640 1 -- 172.21.15.98:0/1021837581 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5ff80a32d0 msgr2=0x7f5ff80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:05.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.255+0000 7f60108e8640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5ff80a32d0 0x7f5ff80a5790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f600c0fed50 tx=0x7f5ff4008040 comp rx=0 tx=0).stop 2024-09-19T08:02:05.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.255+0000 7f60108e8640 1 -- 172.21.15.98:0/1021837581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c102db0 msgr2=0x7f600c198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:05.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.255+0000 7f60108e8640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c102db0 0x7f600c198630 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc004770 tx=0x7f5ffc003b90 comp rx=0 tx=0).stop 2024-09-19T08:02:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.256+0000 7f60108e8640 1 -- 172.21.15.98:0/1021837581 shutdown_connections 2024-09-19T08:02:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.256+0000 7f60108e8640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5ff80a32d0 0x7f5ff80a5790 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.256+0000 7f60108e8640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f600c1082e0 0x7f600c19cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.256+0000 7f60108e8640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f600c103760 0x7f600c198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.256+0000 7f60108e8640 1 --2- 172.21.15.98:0/1021837581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c102db0 0x7f600c198630 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.256+0000 7f60108e8640 1 -- 172.21.15.98:0/1021837581 >> 172.21.15.98:0/1021837581 conn(0x7f600c0fc7d0 msgr2=0x7f600c106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.256+0000 7f60108e8640 1 -- 172.21.15.98:0/1021837581 shutdown_connections 2024-09-19T08:02:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:05.256+0000 7f60108e8640 1 -- 172.21.15.98:0/1021837581 wait complete. 2024-09-19T08:02:05.619 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:05 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1108871096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-19T08:02:05.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:05 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1108871096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-19T08:02:06.004 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986230 got 133143986234 for osd.4 2024-09-19T08:02:06.004 DEBUG:teuthology.parallel:result is None 2024-09-19T08:02:06.354 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:06 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:02:06.354 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:06 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1021837581' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-19T08:02:06.354 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:06 smithi098 ceph-mon[31333]: pgmap v300: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-09-19T08:02:06.382 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005087 got 261993005089 for osd.9 2024-09-19T08:02:06.383 DEBUG:teuthology.parallel:result is None 2024-09-19T08:02:06.393 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:06 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:02:06.393 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:06 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1021837581' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-19T08:02:06.393 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:06 smithi162 ceph-mon[35260]: pgmap v300: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-09-19T08:02:06.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:06 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:02:06.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:06 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1021837581' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-19T08:02:06.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:06 smithi139 ceph-mon[34874]: pgmap v300: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-09-19T08:02:06.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.621+0000 7fc947666640 1 -- 172.21.15.98:0/2649567888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc940108300 msgr2=0x7fc94010ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:06.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.621+0000 7fc947666640 1 --2- 172.21.15.98:0/2649567888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc940108300 0x7fc94010ec70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc930009cd0 tx=0x7fc93002a8d0 comp rx=0 tx=0).stop 2024-09-19T08:02:06.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.621+0000 7fc947666640 1 -- 172.21.15.98:0/2649567888 shutdown_connections 2024-09-19T08:02:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.621+0000 7fc947666640 1 --2- 172.21.15.98:0/2649567888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc940108300 0x7fc94010ec70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.621+0000 7fc947666640 1 --2- 172.21.15.98:0/2649567888 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc940103780 0x7fc940107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.621+0000 7fc947666640 1 --2- 172.21.15.98:0/2649567888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc940102dd0 0x7fc9401031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.621+0000 7fc947666640 1 -- 172.21.15.98:0/2649567888 >> 172.21.15.98:0/2649567888 conn(0x7fc9400fc7d0 msgr2=0x7fc9400febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.621+0000 7fc947666640 1 -- 172.21.15.98:0/2649567888 shutdown_connections 2024-09-19T08:02:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.621+0000 7fc947666640 1 -- 172.21.15.98:0/2649567888 wait complete. 2024-09-19T08:02:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc947666640 1 Processor -- start 2024-09-19T08:02:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc947666640 1 -- start start 2024-09-19T08:02:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc947666640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc940102dd0 0x7fc940198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc947666640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc940103780 0x7fc940198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc947666640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc940108300 0x7fc94019ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc947666640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9401120d0 con 0x7fc940102dd0 2024-09-19T08:02:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc947666640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc940111f50 con 0x7fc940108300 2024-09-19T08:02:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc947666640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc940112250 con 0x7fc940103780 2024-09-19T08:02:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc945402640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc940102dd0 0x7fc940198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc945402640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc940102dd0 0x7fc940198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47922/0 (socket says 172.21.15.98:47922) 2024-09-19T08:02:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc945402640 1 -- 172.21.15.98:0/860317299 learned_addr learned my addr 172.21.15.98:0/860317299 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc944c01640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc940103780 0x7fc940198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc945c03640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc940108300 0x7fc94019ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc945402640 1 -- 172.21.15.98:0/860317299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc940103780 msgr2=0x7fc940198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc945402640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc940103780 0x7fc940198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc945402640 1 -- 172.21.15.98:0/860317299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc940108300 msgr2=0x7fc94019ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:06.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc945402640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc940108300 0x7fc94019ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:06.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc945402640 1 -- 172.21.15.98:0/860317299 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc94019d540 con 0x7fc940102dd0 2024-09-19T08:02:06.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.622+0000 7fc945c03640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc940108300 0x7fc94019ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:06.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.623+0000 7fc945402640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc940102dd0 0x7fc940198590 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc93800e9f0 tx=0x7fc93800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:06.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.623+0000 7fc93e7fc640 1 -- 172.21.15.98:0/860317299 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93800cd80 con 0x7fc940102dd0 2024-09-19T08:02:06.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.623+0000 7fc947666640 1 -- 172.21.15.98:0/860317299 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc94019d830 con 0x7fc940102dd0 2024-09-19T08:02:06.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.623+0000 7fc93e7fc640 1 -- 172.21.15.98:0/860317299 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc938005500 con 0x7fc940102dd0 2024-09-19T08:02:06.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.623+0000 7fc93e7fc640 1 -- 172.21.15.98:0/860317299 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc938010690 con 0x7fc940102dd0 2024-09-19T08:02:06.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.623+0000 7fc947666640 1 -- 172.21.15.98:0/860317299 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9401a5110 con 0x7fc940102dd0 2024-09-19T08:02:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.624+0000 7fc947666640 1 -- 172.21.15.98:0/860317299 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9401031b0 con 0x7fc940102dd0 2024-09-19T08:02:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.625+0000 7fc93e7fc640 1 -- 172.21.15.98:0/860317299 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc9380046c0 con 0x7fc940102dd0 2024-09-19T08:02:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.626+0000 7fc93e7fc640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc9280a32d0 0x7fc9280a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.626+0000 7fc944c01640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc9280a32d0 0x7fc9280a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.626+0000 7fc93e7fc640 1 -- 172.21.15.98:0/860317299 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fc9380cff20 con 0x7fc940102dd0 2024-09-19T08:02:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.626+0000 7fc944c01640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc9280a32d0 0x7fc9280a5790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc9400fee50 tx=0x7fc934005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.630+0000 7fc93e7fc640 1 -- 172.21.15.98:0/860317299 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fc93809caf0 con 0x7fc940102dd0 2024-09-19T08:02:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.842+0000 7fc947666640 1 -- 172.21.15.98:0/860317299 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc94010a290 con 0x7fc940102dd0 2024-09-19T08:02:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.842+0000 7fc93e7fc640 1 -- 172.21.15.98:0/860317299 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc93809c3f0 con 0x7fc940102dd0 2024-09-19T08:02:06.842 INFO:teuthology.orchestra.run.smithi098.stdout:154618822709 2024-09-19T08:02:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.844+0000 7fc947666640 1 -- 172.21.15.98:0/860317299 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc9280a32d0 msgr2=0x7fc9280a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.844+0000 7fc947666640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc9280a32d0 0x7fc9280a5790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc9400fee50 tx=0x7fc934005f90 comp rx=0 tx=0).stop 2024-09-19T08:02:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.844+0000 7fc947666640 1 -- 172.21.15.98:0/860317299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc940102dd0 msgr2=0x7fc940198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.844+0000 7fc947666640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc940102dd0 0x7fc940198590 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc93800e9f0 tx=0x7fc93800edf0 comp rx=0 tx=0).stop 2024-09-19T08:02:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.844+0000 7fc947666640 1 -- 172.21.15.98:0/860317299 shutdown_connections 2024-09-19T08:02:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.844+0000 7fc947666640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc9280a32d0 0x7fc9280a5790 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.844+0000 7fc947666640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc940108300 0x7fc94019ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.844+0000 7fc947666640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc940103780 0x7fc940198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:06.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.844+0000 7fc947666640 1 --2- 172.21.15.98:0/860317299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc940102dd0 0x7fc940198590 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:06.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.844+0000 7fc947666640 1 -- 172.21.15.98:0/860317299 >> 172.21.15.98:0/860317299 conn(0x7fc9400fc7d0 msgr2=0x7fc940105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:06.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.844+0000 7fc947666640 1 -- 172.21.15.98:0/860317299 shutdown_connections 2024-09-19T08:02:06.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:06.844+0000 7fc947666640 1 -- 172.21.15.98:0/860317299 wait complete. 2024-09-19T08:02:07.094 INFO:tasks.cephadm.ceph_manager.ceph:need seq 51539607623 got 51539607624 for osd.1 2024-09-19T08:02:07.095 DEBUG:teuthology.parallel:result is None 2024-09-19T08:02:07.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.680+0000 7fa627311640 1 -- 172.21.15.98:0/824850096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa620102790 msgr2=0x7fa6201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:07.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.680+0000 7fa627311640 1 --2- 172.21.15.98:0/824850096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa620102790 0x7fa6201067e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa610009900 tx=0x7fa61002a410 comp rx=0 tx=0).stop 2024-09-19T08:02:07.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.680+0000 7fa627311640 1 -- 172.21.15.98:0/824850096 shutdown_connections 2024-09-19T08:02:07.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.680+0000 7fa627311640 1 --2- 172.21.15.98:0/824850096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa620106f10 0x7fa62010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:07.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.680+0000 7fa627311640 1 --2- 172.21.15.98:0/824850096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa620102790 0x7fa6201067e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:07.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.680+0000 7fa627311640 1 --2- 172.21.15.98:0/824850096 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa620101de0 0x7fa6201021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:07.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.680+0000 7fa627311640 1 -- 172.21.15.98:0/824850096 >> 172.21.15.98:0/824850096 conn(0x7fa6200fd8c0 msgr2=0x7fa6200ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:07.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.680+0000 7fa627311640 1 -- 172.21.15.98:0/824850096 shutdown_connections 2024-09-19T08:02:07.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.680+0000 7fa627311640 1 -- 172.21.15.98:0/824850096 wait complete. 2024-09-19T08:02:07.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.681+0000 7fa627311640 1 Processor -- start 2024-09-19T08:02:07.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.681+0000 7fa627311640 1 -- start start 2024-09-19T08:02:07.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.681+0000 7fa627311640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa620101de0 0x7fa620198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:07.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.681+0000 7fa627311640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa620102790 0x7fa620198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:07.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.681+0000 7fa627311640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa620106f10 0x7fa62019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:07.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.681+0000 7fa627311640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa620111f90 con 0x7fa620102790 2024-09-19T08:02:07.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.681+0000 7fa627311640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa620111e10 con 0x7fa620106f10 2024-09-19T08:02:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.681+0000 7fa627311640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa620112110 con 0x7fa620101de0 2024-09-19T08:02:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.681+0000 7fa6248ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa620102790 0x7fa620198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.681+0000 7fa6248ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa620102790 0x7fa620198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47950/0 (socket says 172.21.15.98:47950) 2024-09-19T08:02:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.681+0000 7fa6248ac640 1 -- 172.21.15.98:0/4235287819 learned_addr learned my addr 172.21.15.98:0/4235287819 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.681+0000 7fa6250ad640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa620101de0 0x7fa620198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.682+0000 7fa6258ae640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa620106f10 0x7fa62019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.682+0000 7fa6248ac640 1 -- 172.21.15.98:0/4235287819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa620101de0 msgr2=0x7fa620198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.682+0000 7fa6248ac640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa620101de0 0x7fa620198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.682+0000 7fa6248ac640 1 -- 172.21.15.98:0/4235287819 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa620106f10 msgr2=0x7fa62019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.682+0000 7fa6248ac640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa620106f10 0x7fa62019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.682+0000 7fa6248ac640 1 -- 172.21.15.98:0/4235287819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62019d3e0 con 0x7fa620102790 2024-09-19T08:02:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.682+0000 7fa6258ae640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa620106f10 0x7fa62019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.682+0000 7fa6248ac640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa620102790 0x7fa620198970 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa610005ab0 tx=0x7fa6100042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:07.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.682+0000 7fa6167fc640 1 -- 172.21.15.98:0/4235287819 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61003f030 con 0x7fa620102790 2024-09-19T08:02:07.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.682+0000 7fa6167fc640 1 -- 172.21.15.98:0/4235287819 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa610003ce0 con 0x7fa620102790 2024-09-19T08:02:07.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.682+0000 7fa627311640 1 -- 172.21.15.98:0/4235287819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62019d670 con 0x7fa620102790 2024-09-19T08:02:07.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.682+0000 7fa6167fc640 1 -- 172.21.15.98:0/4235287819 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610035070 con 0x7fa620102790 2024-09-19T08:02:07.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.682+0000 7fa627311640 1 -- 172.21.15.98:0/4235287819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6201a4fb0 con 0x7fa620102790 2024-09-19T08:02:07.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.683+0000 7fa627311640 1 -- 172.21.15.98:0/4235287819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6201021c0 con 0x7fa620102790 2024-09-19T08:02:07.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.684+0000 7fa6167fc640 1 -- 172.21.15.98:0/4235287819 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa610042030 con 0x7fa620102790 2024-09-19T08:02:07.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.685+0000 7fa6167fc640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa60c0a32d0 0x7fa60c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:07.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.685+0000 7fa6250ad640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa60c0a32d0 0x7fa60c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:07.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.685+0000 7fa6167fc640 1 -- 172.21.15.98:0/4235287819 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fa6100f0090 con 0x7fa620102790 2024-09-19T08:02:07.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.685+0000 7fa6250ad640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa60c0a32d0 0x7fa60c0a5790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa61c009990 tx=0x7fa61c009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:07.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.689+0000 7fa6167fc640 1 -- 172.21.15.98:0/4235287819 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fa6100bcc60 con 0x7fa620102790 2024-09-19T08:02:07.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.860+0000 7fa627311640 1 -- 172.21.15.98:0/4235287819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fa62010a0f0 con 0x7fa620102790 2024-09-19T08:02:07.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.861+0000 7fa6167fc640 1 -- 172.21.15.98:0/4235287819 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa6100bc560 con 0x7fa620102790 2024-09-19T08:02:07.861 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:07 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:07.861 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:07 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:07.861 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:07 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:07.861 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:07 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:07.861 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:07 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/860317299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-19T08:02:07.861 INFO:teuthology.orchestra.run.smithi098.stdout:201863462955 2024-09-19T08:02:07.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.863+0000 7fa627311640 1 -- 172.21.15.98:0/4235287819 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa60c0a32d0 msgr2=0x7fa60c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:07.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.863+0000 7fa627311640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa60c0a32d0 0x7fa60c0a5790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa61c009990 tx=0x7fa61c009110 comp rx=0 tx=0).stop 2024-09-19T08:02:07.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.863+0000 7fa627311640 1 -- 172.21.15.98:0/4235287819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa620102790 msgr2=0x7fa620198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:07.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.863+0000 7fa627311640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa620102790 0x7fa620198970 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa610005ab0 tx=0x7fa6100042f0 comp rx=0 tx=0).stop 2024-09-19T08:02:07.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.863+0000 7fa627311640 1 -- 172.21.15.98:0/4235287819 shutdown_connections 2024-09-19T08:02:07.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.863+0000 7fa627311640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa60c0a32d0 0x7fa60c0a5790 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:07.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.863+0000 7fa627311640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa620106f10 0x7fa62019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:07.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.863+0000 7fa627311640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa620102790 0x7fa620198970 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:07.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.863+0000 7fa627311640 1 --2- 172.21.15.98:0/4235287819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa620101de0 0x7fa620198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:07.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.863+0000 7fa627311640 1 -- 172.21.15.98:0/4235287819 >> 172.21.15.98:0/4235287819 conn(0x7fa6200fd8c0 msgr2=0x7fa6200fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:07.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.863+0000 7fa627311640 1 -- 172.21.15.98:0/4235287819 shutdown_connections 2024-09-19T08:02:07.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:07.863+0000 7fa627311640 1 -- 172.21.15.98:0/4235287819 wait complete. 2024-09-19T08:02:07.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:07 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:07.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:07 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:07.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:07 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:07.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:07 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:07.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:07 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/860317299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-19T08:02:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:07 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:07 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:07 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:07 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:07 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/860317299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-19T08:02:08.432 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593773 got 184683593776 for osd.6 2024-09-19T08:02:08.432 DEBUG:teuthology.parallel:result is None 2024-09-19T08:02:08.851 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462952 got 201863462955 for osd.7 2024-09-19T08:02:08.851 DEBUG:teuthology.parallel:result is None 2024-09-19T08:02:08.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:08 smithi098 ceph-mon[31333]: pgmap v301: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:08.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:08 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/4235287819' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-19T08:02:09.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:08 smithi162 ceph-mon[35260]: pgmap v301: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:09.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:08 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/4235287819' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-19T08:02:09.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:08 smithi139 ceph-mon[34874]: pgmap v301: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:09.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:08 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/4235287819' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-19T08:02:09.269 INFO:teuthology.orchestra.run.smithi098.stdout:103079215166 2024-09-19T08:02:09.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-19T08:02:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:09.725 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822706 got 154618822709 for osd.5 2024-09-19T08:02:09.725 DEBUG:teuthology.parallel:result is None 2024-09-19T08:02:10.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.164+0000 7f9778a82640 1 -- 172.21.15.98:0/3799163219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9774102ef0 msgr2=0x7f977410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:10.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.164+0000 7f9778a82640 1 --2- 172.21.15.98:0/3799163219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9774102ef0 0x7f977410da40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f976c009ca0 tx=0x7f976c02a950 comp rx=0 tx=0).stop 2024-09-19T08:02:10.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.164+0000 7f9778a82640 1 -- 172.21.15.98:0/3799163219 shutdown_connections 2024-09-19T08:02:10.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.164+0000 7f9778a82640 1 --2- 172.21.15.98:0/3799163219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9774102ef0 0x7f977410da40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.164+0000 7f9778a82640 1 --2- 172.21.15.98:0/3799163219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9774102570 0x7f97741029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.164+0000 7f9778a82640 1 --2- 172.21.15.98:0/3799163219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9774100e20 0x7f9774101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.164+0000 7f9778a82640 1 -- 172.21.15.98:0/3799163219 >> 172.21.15.98:0/3799163219 conn(0x7f97740fc7d0 msgr2=0x7f97740febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.164+0000 7f9778a82640 1 -- 172.21.15.98:0/3799163219 shutdown_connections 2024-09-19T08:02:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.164+0000 7f9778a82640 1 -- 172.21.15.98:0/3799163219 wait complete. 2024-09-19T08:02:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9778a82640 1 Processor -- start 2024-09-19T08:02:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9778a82640 1 -- start start 2024-09-19T08:02:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9778a82640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9774100e20 0x7f977410e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9778a82640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9774102570 0x7f977410ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9778a82640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9774102ef0 0x7f977410ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9778a82640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97741120a0 con 0x7f9774102ef0 2024-09-19T08:02:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9778a82640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9774111f20 con 0x7f9774102570 2024-09-19T08:02:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9778a82640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9774112220 con 0x7f9774100e20 2024-09-19T08:02:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9772d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9774102ef0 0x7f977410ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9772d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9774102ef0 0x7f977410ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47974/0 (socket says 172.21.15.98:47974) 2024-09-19T08:02:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9772d9d640 1 -- 172.21.15.98:0/1353736439 learned_addr learned my addr 172.21.15.98:0/1353736439 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9771d9b640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9774102570 0x7f977410ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f977259c640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9774100e20 0x7f977410e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9772d9d640 1 -- 172.21.15.98:0/1353736439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9774100e20 msgr2=0x7f977410e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9772d9d640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9774100e20 0x7f977410e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9772d9d640 1 -- 172.21.15.98:0/1353736439 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9774102570 msgr2=0x7f977410ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9772d9d640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9774102570 0x7f977410ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9772d9d640 1 -- 172.21.15.98:0/1353736439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97741a68f0 con 0x7f9774102ef0 2024-09-19T08:02:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9771d9b640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9774102570 0x7f977410ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.165+0000 7f9772d9d640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9774102ef0 0x7f977410ef50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f976c006010 tx=0x7f976c02f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.166+0000 7f97637fe640 1 -- 172.21.15.98:0/1353736439 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c034040 con 0x7f9774102ef0 2024-09-19T08:02:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.166+0000 7f9778a82640 1 -- 172.21.15.98:0/1353736439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97741a6b80 con 0x7f9774102ef0 2024-09-19T08:02:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.166+0000 7f97637fe640 1 -- 172.21.15.98:0/1353736439 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f976c003e00 con 0x7f9774102ef0 2024-09-19T08:02:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.166+0000 7f97637fe640 1 -- 172.21.15.98:0/1353736439 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c045480 con 0x7f9774102ef0 2024-09-19T08:02:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.166+0000 7f9778a82640 1 -- 172.21.15.98:0/1353736439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97741a7060 con 0x7f9774102ef0 2024-09-19T08:02:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.167+0000 7f9778a82640 1 -- 172.21.15.98:0/1353736439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9774101200 con 0x7f9774102ef0 2024-09-19T08:02:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.168+0000 7f97637fe640 1 -- 172.21.15.98:0/1353736439 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f976c037020 con 0x7f9774102ef0 2024-09-19T08:02:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.168+0000 7f97637fe640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f975c0a32d0 0x7f975c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.169+0000 7f977259c640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f975c0a32d0 0x7f975c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.169+0000 7f97637fe640 1 -- 172.21.15.98:0/1353736439 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f976c0f0b40 con 0x7f9774102ef0 2024-09-19T08:02:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.169+0000 7f977259c640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f975c0a32d0 0x7f975c0a5790 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9764005e10 tx=0x7f9764005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.172+0000 7f97637fe640 1 -- 172.21.15.98:0/1353736439 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f976c0bd8b0 con 0x7f9774102ef0 2024-09-19T08:02:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.345+0000 7f9778a82640 1 -- 172.21.15.98:0/1353736439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f97741a7290 con 0x7f9774102ef0 2024-09-19T08:02:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.345+0000 7f97637fe640 1 -- 172.21.15.98:0/1353736439 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f976c0bd1b0 con 0x7f9774102ef0 2024-09-19T08:02:10.345 INFO:teuthology.orchestra.run.smithi098.stdout:309237645337 2024-09-19T08:02:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.347+0000 7f9778a82640 1 -- 172.21.15.98:0/1353736439 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f975c0a32d0 msgr2=0x7f975c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.347+0000 7f9778a82640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f975c0a32d0 0x7f975c0a5790 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9764005e10 tx=0x7f9764005da0 comp rx=0 tx=0).stop 2024-09-19T08:02:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.347+0000 7f9778a82640 1 -- 172.21.15.98:0/1353736439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9774102ef0 msgr2=0x7f977410ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.347+0000 7f9778a82640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9774102ef0 0x7f977410ef50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f976c006010 tx=0x7f976c02f6f0 comp rx=0 tx=0).stop 2024-09-19T08:02:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.347+0000 7f9778a82640 1 -- 172.21.15.98:0/1353736439 shutdown_connections 2024-09-19T08:02:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.347+0000 7f9778a82640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f975c0a32d0 0x7f975c0a5790 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.347+0000 7f9778a82640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9774102ef0 0x7f977410ef50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.347+0000 7f9778a82640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9774102570 0x7f977410ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.347+0000 7f9778a82640 1 --2- 172.21.15.98:0/1353736439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9774100e20 0x7f977410e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.347+0000 7f9778a82640 1 -- 172.21.15.98:0/1353736439 >> 172.21.15.98:0/1353736439 conn(0x7f97740fc7d0 msgr2=0x7f9774106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.348+0000 7f9778a82640 1 -- 172.21.15.98:0/1353736439 shutdown_connections 2024-09-19T08:02:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:10.348+0000 7f9778a82640 1 -- 172.21.15.98:0/1353736439 wait complete. 2024-09-19T08:02:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:10 smithi162 ceph-mon[35260]: pgmap v302: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:10 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1353736439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-19T08:02:11.164 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:10 smithi098 ceph-mon[31333]: pgmap v302: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:11.164 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:10 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1353736439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-19T08:02:11.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:10 smithi139 ceph-mon[34874]: pgmap v302: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:11.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:10 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1353736439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-19T08:02:11.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.497+0000 7f41c087b640 1 -- 172.21.15.98:0/1478533434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41bc102ef0 msgr2=0x7f41bc10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:11.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.497+0000 7f41c087b640 1 --2- 172.21.15.98:0/1478533434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41bc102ef0 0x7f41bc10da40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f41b4009ca0 tx=0x7f41b402a950 comp rx=0 tx=0).stop 2024-09-19T08:02:11.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.497+0000 7f41c087b640 1 -- 172.21.15.98:0/1478533434 shutdown_connections 2024-09-19T08:02:11.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.497+0000 7f41c087b640 1 --2- 172.21.15.98:0/1478533434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41bc102ef0 0x7f41bc10da40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:11.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.497+0000 7f41c087b640 1 --2- 172.21.15.98:0/1478533434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41bc102570 0x7f41bc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:11.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.497+0000 7f41c087b640 1 --2- 172.21.15.98:0/1478533434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41bc100e20 0x7f41bc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:11.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.497+0000 7f41c087b640 1 -- 172.21.15.98:0/1478533434 >> 172.21.15.98:0/1478533434 conn(0x7f41bc0fc7d0 msgr2=0x7f41bc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:11.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.497+0000 7f41c087b640 1 -- 172.21.15.98:0/1478533434 shutdown_connections 2024-09-19T08:02:11.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.497+0000 7f41c087b640 1 -- 172.21.15.98:0/1478533434 wait complete. 2024-09-19T08:02:11.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.498+0000 7f41c087b640 1 Processor -- start 2024-09-19T08:02:11.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.498+0000 7f41c087b640 1 -- start start 2024-09-19T08:02:11.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.498+0000 7f41c087b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41bc100e20 0x7f41bc19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:11.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.498+0000 7f41c087b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41bc102570 0x7f41bc19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:11.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.498+0000 7f41c087b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41bc102ef0 0x7f41bc19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:11.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.498+0000 7f41c087b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc110f80 con 0x7f41bc102570 2024-09-19T08:02:11.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.498+0000 7f41c087b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc110e00 con 0x7f41bc100e20 2024-09-19T08:02:11.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.498+0000 7f41c087b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc111100 con 0x7f41bc102ef0 2024-09-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.498+0000 7f41b9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41bc102570 0x7f41bc19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.498+0000 7f41b9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41bc102570 0x7f41bc19aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47994/0 (socket says 172.21.15.98:47994) 2024-09-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.498+0000 7f41b9d9b640 1 -- 172.21.15.98:0/2711172883 learned_addr learned my addr 172.21.15.98:0/2711172883 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.499+0000 7f41bad9d640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41bc102ef0 0x7f41bc19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.499+0000 7f41ba59c640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41bc100e20 0x7f41bc19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.499+0000 7f41b9d9b640 1 -- 172.21.15.98:0/2711172883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41bc102ef0 msgr2=0x7f41bc19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.499+0000 7f41b9d9b640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41bc102ef0 0x7f41bc19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.499+0000 7f41b9d9b640 1 -- 172.21.15.98:0/2711172883 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41bc100e20 msgr2=0x7f41bc19a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.499+0000 7f41b9d9b640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41bc100e20 0x7f41bc19a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.499+0000 7f41b9d9b640 1 -- 172.21.15.98:0/2711172883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41bc19f610 con 0x7f41bc102570 2024-09-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.499+0000 7f41ba59c640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41bc100e20 0x7f41bc19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.499+0000 7f41b9d9b640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41bc102570 0x7f41bc19aba0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f41b000c920 tx=0x7f41b000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:11.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.499+0000 7f41ab7fe640 1 -- 172.21.15.98:0/2711172883 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b0007c70 con 0x7f41bc102570 2024-09-19T08:02:11.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.499+0000 7f41ab7fe640 1 -- 172.21.15.98:0/2711172883 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41b0007e10 con 0x7f41bc102570 2024-09-19T08:02:11.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.499+0000 7f41c087b640 1 -- 172.21.15.98:0/2711172883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41bc19f900 con 0x7f41bc102570 2024-09-19T08:02:11.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.499+0000 7f41ab7fe640 1 -- 172.21.15.98:0/2711172883 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b0018890 con 0x7f41bc102570 2024-09-19T08:02:11.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.499+0000 7f41c087b640 1 -- 172.21.15.98:0/2711172883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41bc1a71e0 con 0x7f41bc102570 2024-09-19T08:02:11.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.500+0000 7f41c087b640 1 -- 172.21.15.98:0/2711172883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41bc101200 con 0x7f41bc102570 2024-09-19T08:02:11.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.501+0000 7f41ab7fe640 1 -- 172.21.15.98:0/2711172883 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f41b0018a70 con 0x7f41bc102570 2024-09-19T08:02:11.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.502+0000 7f41ab7fe640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f41a40a3350 0x7f41a40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:11.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.502+0000 7f41ab7fe640 1 -- 172.21.15.98:0/2711172883 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f41b00cfaf0 con 0x7f41bc102570 2024-09-19T08:02:11.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.502+0000 7f41ba59c640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f41a40a3350 0x7f41a40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:11.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.502+0000 7f41ba59c640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f41a40a3350 0x7f41a40a5810 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f41ac009c20 tx=0x7f41ac005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:11.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.506+0000 7f41ab7fe640 1 -- 172.21.15.98:0/2711172883 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f41b0014030 con 0x7f41bc102570 2024-09-19T08:02:11.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.679+0000 7f41c087b640 1 -- 172.21.15.98:0/2711172883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f41bc1a7440 con 0x7f41bc102570 2024-09-19T08:02:11.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.679+0000 7f41ab7fe640 1 -- 172.21.15.98:0/2711172883 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f41b009c6c0 con 0x7f41bc102570 2024-09-19T08:02:11.679 INFO:teuthology.orchestra.run.smithi098.stdout:103079215167 2024-09-19T08:02:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.682+0000 7f41c087b640 1 -- 172.21.15.98:0/2711172883 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f41a40a3350 msgr2=0x7f41a40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.682+0000 7f41c087b640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f41a40a3350 0x7f41a40a5810 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f41ac009c20 tx=0x7f41ac005f90 comp rx=0 tx=0).stop 2024-09-19T08:02:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.682+0000 7f41c087b640 1 -- 172.21.15.98:0/2711172883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41bc102570 msgr2=0x7f41bc19aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.682+0000 7f41c087b640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41bc102570 0x7f41bc19aba0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f41b000c920 tx=0x7f41b000cde0 comp rx=0 tx=0).stop 2024-09-19T08:02:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.682+0000 7f41c087b640 1 -- 172.21.15.98:0/2711172883 shutdown_connections 2024-09-19T08:02:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.682+0000 7f41c087b640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f41a40a3350 0x7f41a40a5810 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.682+0000 7f41c087b640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41bc102ef0 0x7f41bc19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.682+0000 7f41c087b640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41bc102570 0x7f41bc19aba0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:11.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.682+0000 7f41c087b640 1 --2- 172.21.15.98:0/2711172883 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41bc100e20 0x7f41bc19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:11.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.682+0000 7f41c087b640 1 -- 172.21.15.98:0/2711172883 >> 172.21.15.98:0/2711172883 conn(0x7f41bc0fc7d0 msgr2=0x7f41bc06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:11.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.682+0000 7f41c087b640 1 -- 172.21.15.98:0/2711172883 shutdown_connections 2024-09-19T08:02:11.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:11.682+0000 7f41c087b640 1 -- 172.21.15.98:0/2711172883 wait complete. 2024-09-19T08:02:11.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:11 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2711172883' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-19T08:02:11.932 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645334 got 309237645337 for osd.11 2024-09-19T08:02:11.933 DEBUG:teuthology.parallel:result is None 2024-09-19T08:02:12.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:11 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2711172883' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-19T08:02:12.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:11 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2711172883' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-19T08:02:12.543 INFO:tasks.cephadm.ceph_manager.ceph:need seq 103079215166 got 103079215167 for osd.3 2024-09-19T08:02:12.544 DEBUG:teuthology.parallel:result is None 2024-09-19T08:02:12.544 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-19T08:02:12.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-19T08:02:12.824 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:12.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:12 smithi098 ceph-mon[31333]: pgmap v303: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:13.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:12 smithi162 ceph-mon[35260]: pgmap v303: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:13.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:12 smithi139 ceph-mon[34874]: pgmap v303: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:13.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.352+0000 7fbe267b3640 1 -- 172.21.15.98:0/966079480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe20101400 msgr2=0x7fbe2010d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.352+0000 7fbe267b3640 1 --2- 172.21.15.98:0/966079480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe20101400 0x7fbe2010d930 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009cd0 tx=0x7fbe1802a8e0 comp rx=0 tx=0).stop 2024-09-19T08:02:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.352+0000 7fbe267b3640 1 -- 172.21.15.98:0/966079480 shutdown_connections 2024-09-19T08:02:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.352+0000 7fbe267b3640 1 --2- 172.21.15.98:0/966079480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe20101400 0x7fbe2010d930 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.352+0000 7fbe267b3640 1 --2- 172.21.15.98:0/966079480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe20100a60 0x7fbe20100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.352+0000 7fbe267b3640 1 --2- 172.21.15.98:0/966079480 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe20106a60 0x7fbe20106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.352+0000 7fbe267b3640 1 -- 172.21.15.98:0/966079480 >> 172.21.15.98:0/966079480 conn(0x7fbe200fc780 msgr2=0x7fbe200feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.352+0000 7fbe267b3640 1 -- 172.21.15.98:0/966079480 shutdown_connections 2024-09-19T08:02:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.352+0000 7fbe267b3640 1 -- 172.21.15.98:0/966079480 wait complete. 2024-09-19T08:02:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.353+0000 7fbe267b3640 1 Processor -- start 2024-09-19T08:02:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.353+0000 7fbe267b3640 1 -- start start 2024-09-19T08:02:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.353+0000 7fbe267b3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe20100a60 0x7fbe20198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.353+0000 7fbe267b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe20101400 0x7fbe201988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.353+0000 7fbe267b3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe20106a60 0x7fbe2019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.353+0000 7fbe267b3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbe20110e60 con 0x7fbe20100a60 2024-09-19T08:02:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.353+0000 7fbe267b3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbe20110ce0 con 0x7fbe20106a60 2024-09-19T08:02:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.353+0000 7fbe267b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbe20110fe0 con 0x7fbe20101400 2024-09-19T08:02:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.353+0000 7fbe1ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe20100a60 0x7fbe20198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.353+0000 7fbe1ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe20100a60 0x7fbe20198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48018/0 (socket says 172.21.15.98:48018) 2024-09-19T08:02:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.353+0000 7fbe1ffff640 1 -- 172.21.15.98:0/2567009485 learned_addr learned my addr 172.21.15.98:0/2567009485 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.353+0000 7fbe1f7fe640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe20101400 0x7fbe201988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.354+0000 7fbe24d50640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe20106a60 0x7fbe2019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.354+0000 7fbe1ffff640 1 -- 172.21.15.98:0/2567009485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe20101400 msgr2=0x7fbe201988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.354+0000 7fbe1ffff640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe20101400 0x7fbe201988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.354+0000 7fbe1ffff640 1 -- 172.21.15.98:0/2567009485 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe20106a60 msgr2=0x7fbe2019cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.354+0000 7fbe1ffff640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe20106a60 0x7fbe2019cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.354+0000 7fbe1ffff640 1 -- 172.21.15.98:0/2567009485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2019d320 con 0x7fbe20100a60 2024-09-19T08:02:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.354+0000 7fbe24d50640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe20106a60 0x7fbe2019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.354+0000 7fbe1ffff640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe20100a60 0x7fbe20198370 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbe1000c920 tx=0x7fbe1000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.354+0000 7fbe1d7fa640 1 -- 172.21.15.98:0/2567009485 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe10007c70 con 0x7fbe20100a60 2024-09-19T08:02:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.354+0000 7fbe1d7fa640 1 -- 172.21.15.98:0/2567009485 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe10007e10 con 0x7fbe20100a60 2024-09-19T08:02:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.354+0000 7fbe267b3640 1 -- 172.21.15.98:0/2567009485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe2019d610 con 0x7fbe20100a60 2024-09-19T08:02:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.354+0000 7fbe1d7fa640 1 -- 172.21.15.98:0/2567009485 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe10018890 con 0x7fbe20100a60 2024-09-19T08:02:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.354+0000 7fbe267b3640 1 -- 172.21.15.98:0/2567009485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe201a4ef0 con 0x7fbe20100a60 2024-09-19T08:02:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.355+0000 7fbe267b3640 1 -- 172.21.15.98:0/2567009485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe20100ec0 con 0x7fbe20100a60 2024-09-19T08:02:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.357+0000 7fbe1d7fa640 1 -- 172.21.15.98:0/2567009485 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fbe10018a50 con 0x7fbe20100a60 2024-09-19T08:02:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.358+0000 7fbe1d7fa640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbe080a32d0 0x7fbe080a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.358+0000 7fbe1f7fe640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbe080a32d0 0x7fbe080a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.358+0000 7fbe1d7fa640 1 -- 172.21.15.98:0/2567009485 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fbe100cfde0 con 0x7fbe20100a60 2024-09-19T08:02:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.358+0000 7fbe1f7fe640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbe080a32d0 0x7fbe080a5790 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbe20105110 tx=0x7fbe1400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.361+0000 7fbe1d7fa640 1 -- 172.21.15.98:0/2567009485 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fbe10014030 con 0x7fbe20100a60 2024-09-19T08:02:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.528+0000 7fbe267b3640 1 -- 172.21.15.98:0/2567009485 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbe20062630 con 0x7fbe080a32d0 2024-09-19T08:02:13.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.534+0000 7fbe1d7fa640 1 -- 172.21.15.98:0/2567009485 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258753 (secure 0 0 0) 0x7fbe20062630 con 0x7fbe080a32d0 2024-09-19T08:02:13.537 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:02:13.538 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-19T08:02:13.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.538+0000 7fbe267b3640 1 -- 172.21.15.98:0/2567009485 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbe080a32d0 msgr2=0x7fbe080a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:13.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.538+0000 7fbe267b3640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbe080a32d0 0x7fbe080a5790 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbe20105110 tx=0x7fbe1400b390 comp rx=0 tx=0).stop 2024-09-19T08:02:13.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.538+0000 7fbe267b3640 1 -- 172.21.15.98:0/2567009485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe20100a60 msgr2=0x7fbe20198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:13.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.538+0000 7fbe267b3640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe20100a60 0x7fbe20198370 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbe1000c920 tx=0x7fbe1000cde0 comp rx=0 tx=0).stop 2024-09-19T08:02:13.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.538+0000 7fbe267b3640 1 -- 172.21.15.98:0/2567009485 shutdown_connections 2024-09-19T08:02:13.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.538+0000 7fbe267b3640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fbe080a32d0 0x7fbe080a5790 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:13.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.538+0000 7fbe267b3640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe20106a60 0x7fbe2019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:13.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.538+0000 7fbe267b3640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe20101400 0x7fbe201988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:13.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.538+0000 7fbe267b3640 1 --2- 172.21.15.98:0/2567009485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe20100a60 0x7fbe20198370 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:13.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.538+0000 7fbe267b3640 1 -- 172.21.15.98:0/2567009485 >> 172.21.15.98:0/2567009485 conn(0x7fbe200fc780 msgr2=0x7fbe2006a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:13.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.538+0000 7fbe267b3640 1 -- 172.21.15.98:0/2567009485 shutdown_connections 2024-09-19T08:02:13.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:13.538+0000 7fbe267b3640 1 -- 172.21.15.98:0/2567009485 wait complete. 2024-09-19T08:02:14.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:13 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:14.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:13 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:14.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:13 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:02:14.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:13 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:02:14.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:13 smithi162 ceph-mon[35260]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:14.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:13 smithi162 ceph-mon[35260]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:02:14.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:13 smithi162 ceph-mon[35260]: pgmap v304: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:14.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:13 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:14.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:13 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:14.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:13 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:02:14.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:13 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:02:14.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:13 smithi139 ceph-mon[34874]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:14.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:13 smithi139 ceph-mon[34874]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:02:14.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:13 smithi139 ceph-mon[34874]: pgmap v304: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:14.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:13 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:14.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:13 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:14.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:13 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:02:14.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:13 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:02:14.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:13 smithi098 ceph-mon[31333]: from='mgr.14150 172.21.15.98:0/2068150341' entity='mgr.x' 2024-09-19T08:02:14.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:13 smithi098 ceph-mon[31333]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:02:14.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:13 smithi098 ceph-mon[31333]: pgmap v304: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:14.221 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-09-19T08:02:11.581685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657574,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":645,"num_read_kb":449,"num_write":426,"num_write_kb":3074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181990,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370044,"kb_used_data":12444,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571780,"statfs":{"total":1151940427776,"available":1151561502720,"internally_reserved":0,"allocated":12742656,"data_stored":7066446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.019902"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719055+0000","last_change":"2024-09-19T08:01:10.768950+0000","last_active":"2024-09-19T08:01:26.719055+0000","last_peered":"2024-09-19T08:01:26.719055+0000","last_clean":"2024-09-19T08:01:26.719055+0000","last_became_active":"2024-09-19T08:01:10.768047+0000","last_became_peered":"2024-09-19T08:01:10.768047+0000","last_unstale":"2024-09-19T08:01:26.719055+0000","last_undegraded":"2024-09-19T08:01:26.719055+0000","last_fullsized":"2024-09-19T08:01:26.719055+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:26.219079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757858+0000","last_change":"2024-09-19T08:01:10.783208+0000","last_active":"2024-09-19T08:01:26.757858+0000","last_peered":"2024-09-19T08:01:26.757858+0000","last_clean":"2024-09-19T08:01:26.757858+0000","last_became_active":"2024-09-19T08:01:10.782888+0000","last_became_peered":"2024-09-19T08:01:10.782888+0000","last_unstale":"2024-09-19T08:01:26.757858+0000","last_undegraded":"2024-09-19T08:01:26.757858+0000","last_fullsized":"2024-09-19T08:01:26.757858+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:51:27.603166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757759+0000","last_change":"2024-09-19T08:01:10.770395+0000","last_active":"2024-09-19T08:01:26.757759+0000","last_peered":"2024-09-19T08:01:26.757759+0000","last_clean":"2024-09-19T08:01:26.757759+0000","last_became_active":"2024-09-19T08:01:10.770073+0000","last_became_peered":"2024-09-19T08:01:10.770073+0000","last_unstale":"2024-09-19T08:01:26.757759+0000","last_undegraded":"2024-09-19T08:01:26.757759+0000","last_fullsized":"2024-09-19T08:01:26.757759+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:20.047023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758411+0000","last_change":"2024-09-19T08:01:10.777138+0000","last_active":"2024-09-19T08:01:26.758411+0000","last_peered":"2024-09-19T08:01:26.758411+0000","last_clean":"2024-09-19T08:01:26.758411+0000","last_became_active":"2024-09-19T08:01:10.776810+0000","last_became_peered":"2024-09-19T08:01:10.776810+0000","last_unstale":"2024-09-19T08:01:26.758411+0000","last_undegraded":"2024-09-19T08:01:26.758411+0000","last_fullsized":"2024-09-19T08:01:26.758411+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:39.983321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721379+0000","last_change":"2024-09-19T08:01:10.770435+0000","last_active":"2024-09-19T08:01:26.721379+0000","last_peered":"2024-09-19T08:01:26.721379+0000","last_clean":"2024-09-19T08:01:26.721379+0000","last_became_active":"2024-09-19T08:01:10.769686+0000","last_became_peered":"2024-09-19T08:01:10.769686+0000","last_unstale":"2024-09-19T08:01:26.721379+0000","last_undegraded":"2024-09-19T08:01:26.721379+0000","last_fullsized":"2024-09-19T08:01:26.721379+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:10:19.610925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718127+0000","last_change":"2024-09-19T08:01:10.782382+0000","last_active":"2024-09-19T08:01:26.718127+0000","last_peered":"2024-09-19T08:01:26.718127+0000","last_clean":"2024-09-19T08:01:26.718127+0000","last_became_active":"2024-09-19T08:01:10.781996+0000","last_became_peered":"2024-09-19T08:01:10.781996+0000","last_unstale":"2024-09-19T08:01:26.718127+0000","last_undegraded":"2024-09-19T08:01:26.718127+0000","last_fullsized":"2024-09-19T08:01:26.718127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:43:13.574336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722516+0000","last_change":"2024-09-19T08:01:10.773776+0000","last_active":"2024-09-19T08:01:26.722516+0000","last_peered":"2024-09-19T08:01:26.722516+0000","last_clean":"2024-09-19T08:01:26.722516+0000","last_became_active":"2024-09-19T08:01:10.772442+0000","last_became_peered":"2024-09-19T08:01:10.772442+0000","last_unstale":"2024-09-19T08:01:26.722516+0000","last_undegraded":"2024-09-19T08:01:26.722516+0000","last_fullsized":"2024-09-19T08:01:26.722516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:08:18.155806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.717962+0000","last_change":"2024-09-19T08:01:10.781443+0000","last_active":"2024-09-19T08:01:26.717962+0000","last_peered":"2024-09-19T08:01:26.717962+0000","last_clean":"2024-09-19T08:01:26.717962+0000","last_became_active":"2024-09-19T08:01:10.781083+0000","last_became_peered":"2024-09-19T08:01:10.781083+0000","last_unstale":"2024-09-19T08:01:26.717962+0000","last_undegraded":"2024-09-19T08:01:26.717962+0000","last_fullsized":"2024-09-19T08:01:26.717962+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:23:22.631865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721840+0000","last_change":"2024-09-19T08:01:10.767554+0000","last_active":"2024-09-19T08:01:26.721840+0000","last_peered":"2024-09-19T08:01:26.721840+0000","last_clean":"2024-09-19T08:01:26.721840+0000","last_became_active":"2024-09-19T08:01:10.767227+0000","last_became_peered":"2024-09-19T08:01:10.767227+0000","last_unstale":"2024-09-19T08:01:26.721840+0000","last_undegraded":"2024-09-19T08:01:26.721840+0000","last_fullsized":"2024-09-19T08:01:26.721840+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:30:13.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718514+0000","last_change":"2024-09-19T08:01:10.783174+0000","last_active":"2024-09-19T08:01:26.718514+0000","last_peered":"2024-09-19T08:01:26.718514+0000","last_clean":"2024-09-19T08:01:26.718514+0000","last_became_active":"2024-09-19T08:01:10.782794+0000","last_became_peered":"2024-09-19T08:01:10.782794+0000","last_unstale":"2024-09-19T08:01:26.718514+0000","last_undegraded":"2024-09-19T08:01:26.718514+0000","last_fullsized":"2024-09-19T08:01:26.718514+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:49:19.451107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723075+0000","last_change":"2024-09-19T08:01:10.779638+0000","last_active":"2024-09-19T08:01:26.723075+0000","last_peered":"2024-09-19T08:01:26.723075+0000","last_clean":"2024-09-19T08:01:26.723075+0000","last_became_active":"2024-09-19T08:01:10.779295+0000","last_became_peered":"2024-09-19T08:01:10.779295+0000","last_unstale":"2024-09-19T08:01:26.723075+0000","last_undegraded":"2024-09-19T08:01:26.723075+0000","last_fullsized":"2024-09-19T08:01:26.723075+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:35:10.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758492+0000","last_change":"2024-09-19T08:01:10.784487+0000","last_active":"2024-09-19T08:01:26.758492+0000","last_peered":"2024-09-19T08:01:26.758492+0000","last_clean":"2024-09-19T08:01:26.758492+0000","last_became_active":"2024-09-19T08:01:10.784114+0000","last_became_peered":"2024-09-19T08:01:10.784114+0000","last_unstale":"2024-09-19T08:01:26.758492+0000","last_undegraded":"2024-09-19T08:01:26.758492+0000","last_fullsized":"2024-09-19T08:01:26.758492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:09:54.819802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721613+0000","last_change":"2024-09-19T08:01:10.783186+0000","last_active":"2024-09-19T08:01:26.721613+0000","last_peered":"2024-09-19T08:01:26.721613+0000","last_clean":"2024-09-19T08:01:26.721613+0000","last_became_active":"2024-09-19T08:01:10.782876+0000","last_became_peered":"2024-09-19T08:01:10.782876+0000","last_unstale":"2024-09-19T08:01:26.721613+0000","last_undegraded":"2024-09-19T08:01:26.721613+0000","last_fullsized":"2024-09-19T08:01:26.721613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:26:15.252165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757647+0000","last_change":"2024-09-19T08:01:10.783237+0000","last_active":"2024-09-19T08:01:26.757647+0000","last_peered":"2024-09-19T08:01:26.757647+0000","last_clean":"2024-09-19T08:01:26.757647+0000","last_became_active":"2024-09-19T08:01:10.782966+0000","last_became_peered":"2024-09-19T08:01:10.782966+0000","last_unstale":"2024-09-19T08:01:26.757647+0000","last_undegraded":"2024-09-19T08:01:26.757647+0000","last_fullsized":"2024-09-19T08:01:26.757647+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:57:18.771737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757535+0000","last_change":"2024-09-19T08:01:10.771207+0000","last_active":"2024-09-19T08:01:26.757535+0000","last_peered":"2024-09-19T08:01:26.757535+0000","last_clean":"2024-09-19T08:01:26.757535+0000","last_became_active":"2024-09-19T08:01:10.770862+0000","last_became_peered":"2024-09-19T08:01:10.770862+0000","last_unstale":"2024-09-19T08:01:26.757535+0000","last_undegraded":"2024-09-19T08:01:26.757535+0000","last_fullsized":"2024-09-19T08:01:26.757535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:08:09.313329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.720764+0000","last_change":"2024-09-19T08:01:10.775191+0000","last_active":"2024-09-19T08:01:26.720764+0000","last_peered":"2024-09-19T08:01:26.720764+0000","last_clean":"2024-09-19T08:01:26.720764+0000","last_became_active":"2024-09-19T08:01:10.774832+0000","last_became_peered":"2024-09-19T08:01:10.774832+0000","last_unstale":"2024-09-19T08:01:26.720764+0000","last_undegraded":"2024-09-19T08:01:26.720764+0000","last_fullsized":"2024-09-19T08:01:26.720764+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:57:29.800787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719076+0000","last_change":"2024-09-19T08:01:10.769115+0000","last_active":"2024-09-19T08:01:26.719076+0000","last_peered":"2024-09-19T08:01:26.719076+0000","last_clean":"2024-09-19T08:01:26.719076+0000","last_became_active":"2024-09-19T08:01:10.768088+0000","last_became_peered":"2024-09-19T08:01:10.768088+0000","last_unstale":"2024-09-19T08:01:26.719076+0000","last_undegraded":"2024-09-19T08:01:26.719076+0000","last_fullsized":"2024-09-19T08:01:26.719076+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:27:16.042722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719916+0000","last_change":"2024-09-19T08:01:10.781426+0000","last_active":"2024-09-19T08:01:26.719916+0000","last_peered":"2024-09-19T08:01:26.719916+0000","last_clean":"2024-09-19T08:01:26.719916+0000","last_became_active":"2024-09-19T08:01:10.781094+0000","last_became_peered":"2024-09-19T08:01:10.781094+0000","last_unstale":"2024-09-19T08:01:26.719916+0000","last_undegraded":"2024-09-19T08:01:26.719916+0000","last_fullsized":"2024-09-19T08:01:26.719916+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:43:31.336610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725515+0000","last_change":"2024-09-19T08:01:10.766466+0000","last_active":"2024-09-19T08:01:26.725515+0000","last_peered":"2024-09-19T08:01:26.725515+0000","last_clean":"2024-09-19T08:01:26.725515+0000","last_became_active":"2024-09-19T08:01:10.766123+0000","last_became_peered":"2024-09-19T08:01:10.766123+0000","last_unstale":"2024-09-19T08:01:26.725515+0000","last_undegraded":"2024-09-19T08:01:26.725515+0000","last_fullsized":"2024-09-19T08:01:26.725515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:25.848603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718613+0000","last_change":"2024-09-19T08:01:10.772782+0000","last_active":"2024-09-19T08:01:26.718613+0000","last_peered":"2024-09-19T08:01:26.718613+0000","last_clean":"2024-09-19T08:01:26.718613+0000","last_became_active":"2024-09-19T08:01:10.772445+0000","last_became_peered":"2024-09-19T08:01:10.772445+0000","last_unstale":"2024-09-19T08:01:26.718613+0000","last_undegraded":"2024-09-19T08:01:26.718613+0000","last_fullsized":"2024-09-19T08:01:26.718613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:09.174005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721701+0000","last_change":"2024-09-19T08:01:10.771944+0000","last_active":"2024-09-19T08:01:26.721701+0000","last_peered":"2024-09-19T08:01:26.721701+0000","last_clean":"2024-09-19T08:01:26.721701+0000","last_became_active":"2024-09-19T08:01:10.771643+0000","last_became_peered":"2024-09-19T08:01:10.771643+0000","last_unstale":"2024-09-19T08:01:26.721701+0000","last_undegraded":"2024-09-19T08:01:26.721701+0000","last_fullsized":"2024-09-19T08:01:26.721701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:46:27.121722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721201+0000","last_change":"2024-09-19T08:01:10.775144+0000","last_active":"2024-09-19T08:01:26.721201+0000","last_peered":"2024-09-19T08:01:26.721201+0000","last_clean":"2024-09-19T08:01:26.721201+0000","last_became_active":"2024-09-19T08:01:10.774769+0000","last_became_peered":"2024-09-19T08:01:10.774769+0000","last_unstale":"2024-09-19T08:01:26.721201+0000","last_undegraded":"2024-09-19T08:01:26.721201+0000","last_fullsized":"2024-09-19T08:01:26.721201+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:59:47.053993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718014+0000","last_change":"2024-09-19T08:01:10.766750+0000","last_active":"2024-09-19T08:01:26.718014+0000","last_peered":"2024-09-19T08:01:26.718014+0000","last_clean":"2024-09-19T08:01:26.718014+0000","last_became_active":"2024-09-19T08:01:10.766450+0000","last_became_peered":"2024-09-19T08:01:10.766450+0000","last_unstale":"2024-09-19T08:01:26.718014+0000","last_undegraded":"2024-09-19T08:01:26.718014+0000","last_fullsized":"2024-09-19T08:01:26.718014+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:47:58.133815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721798+0000","last_change":"2024-09-19T08:01:10.768430+0000","last_active":"2024-09-19T08:01:26.721798+0000","last_peered":"2024-09-19T08:01:26.721798+0000","last_clean":"2024-09-19T08:01:26.721798+0000","last_became_active":"2024-09-19T08:01:10.768092+0000","last_became_peered":"2024-09-19T08:01:10.768092+0000","last_unstale":"2024-09-19T08:01:26.721798+0000","last_undegraded":"2024-09-19T08:01:26.721798+0000","last_fullsized":"2024-09-19T08:01:26.721798+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:16:42.343138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718376+0000","last_change":"2024-09-19T08:01:10.771131+0000","last_active":"2024-09-19T08:01:26.718376+0000","last_peered":"2024-09-19T08:01:26.718376+0000","last_clean":"2024-09-19T08:01:26.718376+0000","last_became_active":"2024-09-19T08:01:10.770851+0000","last_became_peered":"2024-09-19T08:01:10.770851+0000","last_unstale":"2024-09-19T08:01:26.718376+0000","last_undegraded":"2024-09-19T08:01:26.718376+0000","last_fullsized":"2024-09-19T08:01:26.718376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:07:15.830476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.726129+0000","last_change":"2024-09-19T08:01:10.776220+0000","last_active":"2024-09-19T08:01:26.726129+0000","last_peered":"2024-09-19T08:01:26.726129+0000","last_clean":"2024-09-19T08:01:26.726129+0000","last_became_active":"2024-09-19T08:01:10.775889+0000","last_became_peered":"2024-09-19T08:01:10.775889+0000","last_unstale":"2024-09-19T08:01:26.726129+0000","last_undegraded":"2024-09-19T08:01:26.726129+0000","last_fullsized":"2024-09-19T08:01:26.726129+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:23.714675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719003+0000","last_change":"2024-09-19T08:01:10.776370+0000","last_active":"2024-09-19T08:01:26.719003+0000","last_peered":"2024-09-19T08:01:26.719003+0000","last_clean":"2024-09-19T08:01:26.719003+0000","last_became_active":"2024-09-19T08:01:10.776027+0000","last_became_peered":"2024-09-19T08:01:10.776027+0000","last_unstale":"2024-09-19T08:01:26.719003+0000","last_undegraded":"2024-09-19T08:01:26.719003+0000","last_fullsized":"2024-09-19T08:01:26.719003+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:48:46.293404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758971+0000","last_change":"2024-09-19T08:01:26.758971+0000","last_active":"2024-09-19T08:01:26.758971+0000","last_peered":"2024-09-19T08:01:26.758971+0000","last_clean":"2024-09-19T08:01:26.758971+0000","last_became_active":"2024-09-19T08:01:26.758524+0000","last_became_peered":"2024-09-19T08:01:26.758524+0000","last_unstale":"2024-09-19T08:01:26.758971+0000","last_undegraded":"2024-09-19T08:01:26.758971+0000","last_fullsized":"2024-09-19T08:01:26.758971+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:34:22.439327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757760+0000","last_change":"2024-09-19T08:01:10.767445+0000","last_active":"2024-09-19T08:01:26.757760+0000","last_peered":"2024-09-19T08:01:26.757760+0000","last_clean":"2024-09-19T08:01:26.757760+0000","last_became_active":"2024-09-19T08:01:10.767128+0000","last_became_peered":"2024-09-19T08:01:10.767128+0000","last_unstale":"2024-09-19T08:01:26.757760+0000","last_undegraded":"2024-09-19T08:01:26.757760+0000","last_fullsized":"2024-09-19T08:01:26.757760+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:10.273948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725212+0000","last_change":"2024-09-19T08:01:10.768954+0000","last_active":"2024-09-19T08:01:26.725212+0000","last_peered":"2024-09-19T08:01:26.725212+0000","last_clean":"2024-09-19T08:01:26.725212+0000","last_became_active":"2024-09-19T08:01:10.768577+0000","last_became_peered":"2024-09-19T08:01:10.768577+0000","last_unstale":"2024-09-19T08:01:26.725212+0000","last_undegraded":"2024-09-19T08:01:26.725212+0000","last_fullsized":"2024-09-19T08:01:26.725212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:20:21.812531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725473+0000","last_change":"2024-09-19T08:01:10.774104+0000","last_active":"2024-09-19T08:01:26.725473+0000","last_peered":"2024-09-19T08:01:26.725473+0000","last_clean":"2024-09-19T08:01:26.725473+0000","last_became_active":"2024-09-19T08:01:10.773721+0000","last_became_peered":"2024-09-19T08:01:10.773721+0000","last_unstale":"2024-09-19T08:01:26.725473+0000","last_undegraded":"2024-09-19T08:01:26.725473+0000","last_fullsized":"2024-09-19T08:01:26.725473+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:45:34.945994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.727296+0000","last_change":"2024-09-19T08:01:26.727296+0000","last_active":"2024-09-19T08:01:26.727296+0000","last_peered":"2024-09-19T08:01:26.727296+0000","last_clean":"2024-09-19T08:01:26.727296+0000","last_became_active":"2024-09-19T08:01:26.726631+0000","last_became_peered":"2024-09-19T08:01:26.726631+0000","last_unstale":"2024-09-19T08:01:26.727296+0000","last_undegraded":"2024-09-19T08:01:26.727296+0000","last_fullsized":"2024-09-19T08:01:26.727296+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:19:24.807050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721044+0000","last_change":"2024-09-19T08:01:10.765282+0000","last_active":"2024-09-19T08:01:26.721044+0000","last_peered":"2024-09-19T08:01:26.721044+0000","last_clean":"2024-09-19T08:01:26.721044+0000","last_became_active":"2024-09-19T08:01:10.764928+0000","last_became_peered":"2024-09-19T08:01:10.764928+0000","last_unstale":"2024-09-19T08:01:26.721044+0000","last_undegraded":"2024-09-19T08:01:26.721044+0000","last_fullsized":"2024-09-19T08:01:26.721044+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:00:37.395110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"78'6","reported_seq":77,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:04.754064+0000","last_change":"2024-09-19T08:01:08.760686+0000","last_active":"2024-09-19T08:02:04.754064+0000","last_peered":"2024-09-19T08:02:04.754064+0000","last_clean":"2024-09-19T08:02:04.754064+0000","last_became_active":"2024-09-19T08:01:08.759912+0000","last_became_peered":"2024-09-19T08:01:08.759912+0000","last_unstale":"2024-09-19T08:02:04.754064+0000","last_undegraded":"2024-09-19T08:02:04.754064+0000","last_fullsized":"2024-09-19T08:02:04.754064+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:46:01.879611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721868+0000","last_change":"2024-09-19T08:01:10.779668+0000","last_active":"2024-09-19T08:01:26.721868+0000","last_peered":"2024-09-19T08:01:26.721868+0000","last_clean":"2024-09-19T08:01:26.721868+0000","last_became_active":"2024-09-19T08:01:10.779265+0000","last_became_peered":"2024-09-19T08:01:10.779265+0000","last_unstale":"2024-09-19T08:01:26.721868+0000","last_undegraded":"2024-09-19T08:01:26.721868+0000","last_fullsized":"2024-09-19T08:01:26.721868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:06.177700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718183+0000","last_change":"2024-09-19T08:01:08.762668+0000","last_active":"2024-09-19T08:01:26.718183+0000","last_peered":"2024-09-19T08:01:26.718183+0000","last_clean":"2024-09-19T08:01:26.718183+0000","last_became_active":"2024-09-19T08:01:08.762277+0000","last_became_peered":"2024-09-19T08:01:08.762277+0000","last_unstale":"2024-09-19T08:01:26.718183+0000","last_undegraded":"2024-09-19T08:01:26.718183+0000","last_fullsized":"2024-09-19T08:01:26.718183+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:36:59.316655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721452+0000","last_change":"2024-09-19T08:01:08.757401+0000","last_active":"2024-09-19T08:01:26.721452+0000","last_peered":"2024-09-19T08:01:26.721452+0000","last_clean":"2024-09-19T08:01:26.721452+0000","last_became_active":"2024-09-19T08:01:08.757067+0000","last_became_peered":"2024-09-19T08:01:08.757067+0000","last_unstale":"2024-09-19T08:01:26.721452+0000","last_undegraded":"2024-09-19T08:01:26.721452+0000","last_fullsized":"2024-09-19T08:01:26.721452+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:19.617830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718027+0000","last_change":"2024-09-19T08:01:10.765149+0000","last_active":"2024-09-19T08:01:26.718027+0000","last_peered":"2024-09-19T08:01:26.718027+0000","last_clean":"2024-09-19T08:01:26.718027+0000","last_became_active":"2024-09-19T08:01:10.764840+0000","last_became_peered":"2024-09-19T08:01:10.764840+0000","last_unstale":"2024-09-19T08:01:26.718027+0000","last_undegraded":"2024-09-19T08:01:26.718027+0000","last_fullsized":"2024-09-19T08:01:26.718027+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:15.539307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722074+0000","last_change":"2024-09-19T08:01:08.759435+0000","last_active":"2024-09-19T08:01:26.722074+0000","last_peered":"2024-09-19T08:01:26.722074+0000","last_clean":"2024-09-19T08:01:26.722074+0000","last_became_active":"2024-09-19T08:01:08.759046+0000","last_became_peered":"2024-09-19T08:01:08.759046+0000","last_unstale":"2024-09-19T08:01:26.722074+0000","last_undegraded":"2024-09-19T08:01:26.722074+0000","last_fullsized":"2024-09-19T08:01:26.722074+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:21:24.077626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719748+0000","last_change":"2024-09-19T08:01:10.778432+0000","last_active":"2024-09-19T08:01:26.719748+0000","last_peered":"2024-09-19T08:01:26.719748+0000","last_clean":"2024-09-19T08:01:26.719748+0000","last_became_active":"2024-09-19T08:01:10.778098+0000","last_became_peered":"2024-09-19T08:01:10.778098+0000","last_unstale":"2024-09-19T08:01:26.719748+0000","last_undegraded":"2024-09-19T08:01:26.719748+0000","last_fullsized":"2024-09-19T08:01:26.719748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:40:12.921165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"78'4","reported_seq":120,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:07.756756+0000","last_change":"2024-09-19T08:01:08.764117+0000","last_active":"2024-09-19T08:02:07.756756+0000","last_peered":"2024-09-19T08:02:07.756756+0000","last_clean":"2024-09-19T08:02:07.756756+0000","last_became_active":"2024-09-19T08:01:08.763308+0000","last_became_peered":"2024-09-19T08:01:08.763308+0000","last_unstale":"2024-09-19T08:02:07.756756+0000","last_undegraded":"2024-09-19T08:02:07.756756+0000","last_fullsized":"2024-09-19T08:02:07.756756+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:19:57.014628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":94,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721336+0000","last_change":"2024-09-19T08:01:10.781771+0000","last_active":"2024-09-19T08:01:26.721336+0000","last_peered":"2024-09-19T08:01:26.721336+0000","last_clean":"2024-09-19T08:01:26.721336+0000","last_became_active":"2024-09-19T08:01:10.781432+0000","last_became_peered":"2024-09-19T08:01:10.781432+0000","last_unstale":"2024-09-19T08:01:26.721336+0000","last_undegraded":"2024-09-19T08:01:26.721336+0000","last_fullsized":"2024-09-19T08:01:26.721336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:25:15.067850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.771171+0000","last_change":"2024-09-19T08:01:26.771171+0000","last_active":"2024-09-19T08:01:26.771171+0000","last_peered":"2024-09-19T08:01:26.771171+0000","last_clean":"2024-09-19T08:01:26.771171+0000","last_became_active":"2024-09-19T08:01:26.759274+0000","last_became_peered":"2024-09-19T08:01:26.759274+0000","last_unstale":"2024-09-19T08:01:26.771171+0000","last_undegraded":"2024-09-19T08:01:26.771171+0000","last_fullsized":"2024-09-19T08:01:26.771171+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:20:54.603420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757967+0000","last_change":"2024-09-19T08:01:10.768318+0000","last_active":"2024-09-19T08:01:26.757967+0000","last_peered":"2024-09-19T08:01:26.757967+0000","last_clean":"2024-09-19T08:01:26.757967+0000","last_became_active":"2024-09-19T08:01:10.768022+0000","last_became_peered":"2024-09-19T08:01:10.768022+0000","last_unstale":"2024-09-19T08:01:26.757967+0000","last_undegraded":"2024-09-19T08:01:26.757967+0000","last_fullsized":"2024-09-19T08:01:26.757967+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:00:45.328638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722058+0000","last_change":"2024-09-19T08:01:09.750530+0000","last_active":"2024-09-19T08:01:26.722058+0000","last_peered":"2024-09-19T08:01:26.722058+0000","last_clean":"2024-09-19T08:01:26.722058+0000","last_became_active":"2024-09-19T08:01:09.750029+0000","last_became_peered":"2024-09-19T08:01:09.750029+0000","last_unstale":"2024-09-19T08:01:26.722058+0000","last_undegraded":"2024-09-19T08:01:26.722058+0000","last_fullsized":"2024-09-19T08:01:26.722058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:01:16.062992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.724636+0000","last_change":"2024-09-19T08:01:26.724636+0000","last_active":"2024-09-19T08:01:26.724636+0000","last_peered":"2024-09-19T08:01:26.724636+0000","last_clean":"2024-09-19T08:01:26.724636+0000","last_became_active":"2024-09-19T08:01:26.724171+0000","last_became_peered":"2024-09-19T08:01:26.724171+0000","last_unstale":"2024-09-19T08:01:26.724636+0000","last_undegraded":"2024-09-19T08:01:26.724636+0000","last_fullsized":"2024-09-19T08:01:26.724636+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:54:05.998374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'30","reported_seq":157,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:07.756821+0000","last_change":"2024-09-19T08:01:08.761950+0000","last_active":"2024-09-19T08:02:07.756821+0000","last_peered":"2024-09-19T08:02:07.756821+0000","last_clean":"2024-09-19T08:02:07.756821+0000","last_became_active":"2024-09-19T08:01:08.761533+0000","last_became_peered":"2024-09-19T08:01:08.761533+0000","last_unstale":"2024-09-19T08:02:07.756821+0000","last_undegraded":"2024-09-19T08:02:07.756821+0000","last_fullsized":"2024-09-19T08:02:07.756821+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:07:26.668141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59764,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":105,"num_read_kb":110,"num_write":32,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723579+0000","last_change":"2024-09-19T08:01:10.774280+0000","last_active":"2024-09-19T08:01:26.723579+0000","last_peered":"2024-09-19T08:01:26.723579+0000","last_clean":"2024-09-19T08:01:26.723579+0000","last_became_active":"2024-09-19T08:01:10.773982+0000","last_became_peered":"2024-09-19T08:01:10.773982+0000","last_unstale":"2024-09-19T08:01:26.723579+0000","last_undegraded":"2024-09-19T08:01:26.723579+0000","last_fullsized":"2024-09-19T08:01:26.723579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:21:43.180057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725363+0000","last_change":"2024-09-19T08:01:08.761067+0000","last_active":"2024-09-19T08:01:26.725363+0000","last_peered":"2024-09-19T08:01:26.725363+0000","last_clean":"2024-09-19T08:01:26.725363+0000","last_became_active":"2024-09-19T08:01:08.760664+0000","last_became_peered":"2024-09-19T08:01:08.760664+0000","last_unstale":"2024-09-19T08:01:26.725363+0000","last_undegraded":"2024-09-19T08:01:26.725363+0000","last_fullsized":"2024-09-19T08:01:26.725363+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:22:38.086891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721462+0000","last_change":"2024-09-19T08:01:10.771845+0000","last_active":"2024-09-19T08:01:26.721462+0000","last_peered":"2024-09-19T08:01:26.721462+0000","last_clean":"2024-09-19T08:01:26.721462+0000","last_became_active":"2024-09-19T08:01:10.771533+0000","last_became_peered":"2024-09-19T08:01:10.771533+0000","last_unstale":"2024-09-19T08:01:26.721462+0000","last_undegraded":"2024-09-19T08:01:26.721462+0000","last_fullsized":"2024-09-19T08:01:26.721462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:29:35.180763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722806+0000","last_change":"2024-09-19T08:01:08.763869+0000","last_active":"2024-09-19T08:01:26.722806+0000","last_peered":"2024-09-19T08:01:26.722806+0000","last_clean":"2024-09-19T08:01:26.722806+0000","last_became_active":"2024-09-19T08:01:08.763451+0000","last_became_peered":"2024-09-19T08:01:08.763451+0000","last_unstale":"2024-09-19T08:01:26.722806+0000","last_undegraded":"2024-09-19T08:01:26.722806+0000","last_fullsized":"2024-09-19T08:01:26.722806+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:32:40.901049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722991+0000","last_change":"2024-09-19T08:01:10.770823+0000","last_active":"2024-09-19T08:01:26.722991+0000","last_peered":"2024-09-19T08:01:26.722991+0000","last_clean":"2024-09-19T08:01:26.722991+0000","last_became_active":"2024-09-19T08:01:10.770462+0000","last_became_peered":"2024-09-19T08:01:10.770462+0000","last_unstale":"2024-09-19T08:01:26.722991+0000","last_undegraded":"2024-09-19T08:01:26.722991+0000","last_fullsized":"2024-09-19T08:01:26.722991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:04:32.832009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"78'5","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:04.753190+0000","last_change":"2024-09-19T08:01:08.759429+0000","last_active":"2024-09-19T08:02:04.753190+0000","last_peered":"2024-09-19T08:02:04.753190+0000","last_clean":"2024-09-19T08:02:04.753190+0000","last_became_active":"2024-09-19T08:01:08.759062+0000","last_became_peered":"2024-09-19T08:01:08.759062+0000","last_unstale":"2024-09-19T08:02:04.753190+0000","last_undegraded":"2024-09-19T08:02:04.753190+0000","last_fullsized":"2024-09-19T08:02:04.753190+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:16:51.582435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":49,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722418+0000","last_change":"2024-09-19T08:01:10.784735+0000","last_active":"2024-09-19T08:01:26.722418+0000","last_peered":"2024-09-19T08:01:26.722418+0000","last_clean":"2024-09-19T08:01:26.722418+0000","last_became_active":"2024-09-19T08:01:10.784346+0000","last_became_peered":"2024-09-19T08:01:10.784346+0000","last_unstale":"2024-09-19T08:01:26.722418+0000","last_undegraded":"2024-09-19T08:01:26.722418+0000","last_fullsized":"2024-09-19T08:01:26.722418+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:27:54.580391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721469+0000","last_change":"2024-09-19T08:01:09.749498+0000","last_active":"2024-09-19T08:01:26.721469+0000","last_peered":"2024-09-19T08:01:26.721469+0000","last_clean":"2024-09-19T08:01:26.721469+0000","last_became_active":"2024-09-19T08:01:09.749091+0000","last_became_peered":"2024-09-19T08:01:09.749091+0000","last_unstale":"2024-09-19T08:01:26.721469+0000","last_undegraded":"2024-09-19T08:01:26.721469+0000","last_fullsized":"2024-09-19T08:01:26.721469+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:56:13.644931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721781+0000","last_change":"2024-09-19T08:01:10.779599+0000","last_active":"2024-09-19T08:01:26.721781+0000","last_peered":"2024-09-19T08:01:26.721781+0000","last_clean":"2024-09-19T08:01:26.721781+0000","last_became_active":"2024-09-19T08:01:10.779219+0000","last_became_peered":"2024-09-19T08:01:10.779219+0000","last_unstale":"2024-09-19T08:01:26.721781+0000","last_undegraded":"2024-09-19T08:01:26.721781+0000","last_fullsized":"2024-09-19T08:01:26.721781+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:20.193753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723508+0000","last_change":"2024-09-19T08:01:08.761432+0000","last_active":"2024-09-19T08:01:26.723508+0000","last_peered":"2024-09-19T08:01:26.723508+0000","last_clean":"2024-09-19T08:01:26.723508+0000","last_became_active":"2024-09-19T08:01:08.761044+0000","last_became_peered":"2024-09-19T08:01:08.761044+0000","last_unstale":"2024-09-19T08:01:26.723508+0000","last_undegraded":"2024-09-19T08:01:26.723508+0000","last_fullsized":"2024-09-19T08:01:26.723508+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:12.237334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718915+0000","last_change":"2024-09-19T08:01:10.772559+0000","last_active":"2024-09-19T08:01:26.718915+0000","last_peered":"2024-09-19T08:01:26.718915+0000","last_clean":"2024-09-19T08:01:26.718915+0000","last_became_active":"2024-09-19T08:01:10.772176+0000","last_became_peered":"2024-09-19T08:01:10.772176+0000","last_unstale":"2024-09-19T08:01:26.718915+0000","last_undegraded":"2024-09-19T08:01:26.718915+0000","last_fullsized":"2024-09-19T08:01:26.718915+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:27:16.296127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721816+0000","last_change":"2024-09-19T08:01:08.759989+0000","last_active":"2024-09-19T08:01:26.721816+0000","last_peered":"2024-09-19T08:01:26.721816+0000","last_clean":"2024-09-19T08:01:26.721816+0000","last_became_active":"2024-09-19T08:01:08.759705+0000","last_became_peered":"2024-09-19T08:01:08.759705+0000","last_unstale":"2024-09-19T08:01:26.721816+0000","last_undegraded":"2024-09-19T08:01:26.721816+0000","last_fullsized":"2024-09-19T08:01:26.721816+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:33:25.237567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723490+0000","last_change":"2024-09-19T08:01:10.776528+0000","last_active":"2024-09-19T08:01:26.723490+0000","last_peered":"2024-09-19T08:01:26.723490+0000","last_clean":"2024-09-19T08:01:26.723490+0000","last_became_active":"2024-09-19T08:01:10.776222+0000","last_became_peered":"2024-09-19T08:01:10.776222+0000","last_unstale":"2024-09-19T08:01:26.723490+0000","last_undegraded":"2024-09-19T08:01:26.723490+0000","last_fullsized":"2024-09-19T08:01:26.723490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.787492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722186+0000","last_change":"2024-09-19T08:01:08.759631+0000","last_active":"2024-09-19T08:01:26.722186+0000","last_peered":"2024-09-19T08:01:26.722186+0000","last_clean":"2024-09-19T08:01:26.722186+0000","last_became_active":"2024-09-19T08:01:08.759156+0000","last_became_peered":"2024-09-19T08:01:08.759156+0000","last_unstale":"2024-09-19T08:01:26.722186+0000","last_undegraded":"2024-09-19T08:01:26.722186+0000","last_fullsized":"2024-09-19T08:01:26.722186+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:48:50.910824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722951+0000","last_change":"2024-09-19T08:01:10.772660+0000","last_active":"2024-09-19T08:01:26.722951+0000","last_peered":"2024-09-19T08:01:26.722951+0000","last_clean":"2024-09-19T08:01:26.722951+0000","last_became_active":"2024-09-19T08:01:10.771816+0000","last_became_peered":"2024-09-19T08:01:10.771816+0000","last_unstale":"2024-09-19T08:01:26.722951+0000","last_undegraded":"2024-09-19T08:01:26.722951+0000","last_fullsized":"2024-09-19T08:01:26.722951+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:04:32.667107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'6","reported_seq":76,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:05.772985+0000","last_change":"2024-09-19T08:01:09.749844+0000","last_active":"2024-09-19T08:02:05.772985+0000","last_peered":"2024-09-19T08:02:05.772985+0000","last_clean":"2024-09-19T08:02:05.772985+0000","last_became_active":"2024-09-19T08:01:09.749410+0000","last_became_peered":"2024-09-19T08:01:09.749410+0000","last_unstale":"2024-09-19T08:02:05.772985+0000","last_undegraded":"2024-09-19T08:02:05.772985+0000","last_fullsized":"2024-09-19T08:02:05.772985+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:03:14.989814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1563,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":48,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723119+0000","last_change":"2024-09-19T08:01:10.773782+0000","last_active":"2024-09-19T08:01:26.723119+0000","last_peered":"2024-09-19T08:01:26.723119+0000","last_clean":"2024-09-19T08:01:26.723119+0000","last_became_active":"2024-09-19T08:01:10.773471+0000","last_became_peered":"2024-09-19T08:01:10.773471+0000","last_unstale":"2024-09-19T08:01:26.723119+0000","last_undegraded":"2024-09-19T08:01:26.723119+0000","last_fullsized":"2024-09-19T08:01:26.723119+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:15.780259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":341,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722698+0000","last_change":"2024-09-19T08:00:01.677517+0000","last_active":"2024-09-19T08:01:26.722698+0000","last_peered":"2024-09-19T08:01:26.722698+0000","last_clean":"2024-09-19T08:01:26.722698+0000","last_became_active":"2024-09-19T08:00:01.677089+0000","last_became_peered":"2024-09-19T08:00:01.677089+0000","last_unstale":"2024-09-19T08:01:26.722698+0000","last_undegraded":"2024-09-19T08:01:26.722698+0000","last_fullsized":"2024-09-19T08:01:26.722698+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_clean_scrub_stamp":"2024-09-19T07:56:43.962142+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:10:36.158811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758444+0000","last_change":"2024-09-19T08:01:08.759819+0000","last_active":"2024-09-19T08:01:26.758444+0000","last_peered":"2024-09-19T08:01:26.758444+0000","last_clean":"2024-09-19T08:01:26.758444+0000","last_became_active":"2024-09-19T08:01:08.759420+0000","last_became_peered":"2024-09-19T08:01:08.759420+0000","last_unstale":"2024-09-19T08:01:26.758444+0000","last_undegraded":"2024-09-19T08:01:26.758444+0000","last_fullsized":"2024-09-19T08:01:26.758444+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:06:16.852323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.724656+0000","last_change":"2024-09-19T08:01:26.724260+0000","last_active":"2024-09-19T08:01:26.724656+0000","last_peered":"2024-09-19T08:01:26.724656+0000","last_clean":"2024-09-19T08:01:26.724656+0000","last_became_active":"2024-09-19T08:01:26.723851+0000","last_became_peered":"2024-09-19T08:01:26.723851+0000","last_unstale":"2024-09-19T08:01:26.724656+0000","last_undegraded":"2024-09-19T08:01:26.724656+0000","last_fullsized":"2024-09-19T08:01:26.724656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:18:46.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722601+0000","last_change":"2024-09-19T08:01:08.761291+0000","last_active":"2024-09-19T08:01:26.722601+0000","last_peered":"2024-09-19T08:01:26.722601+0000","last_clean":"2024-09-19T08:01:26.722601+0000","last_became_active":"2024-09-19T08:01:08.760238+0000","last_became_peered":"2024-09-19T08:01:08.760238+0000","last_unstale":"2024-09-19T08:01:26.722601+0000","last_undegraded":"2024-09-19T08:01:26.722601+0000","last_fullsized":"2024-09-19T08:01:26.722601+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:26:09.980573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721415+0000","last_change":"2024-09-19T08:01:10.773056+0000","last_active":"2024-09-19T08:01:26.721415+0000","last_peered":"2024-09-19T08:01:26.721415+0000","last_clean":"2024-09-19T08:01:26.721415+0000","last_became_active":"2024-09-19T08:01:10.772431+0000","last_became_peered":"2024-09-19T08:01:10.772431+0000","last_unstale":"2024-09-19T08:01:26.721415+0000","last_undegraded":"2024-09-19T08:01:26.721415+0000","last_fullsized":"2024-09-19T08:01:26.721415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:39:06.878086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"78'5","reported_seq":78,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:08.757483+0000","last_change":"2024-09-19T08:01:09.749707+0000","last_active":"2024-09-19T08:02:08.757483+0000","last_peered":"2024-09-19T08:02:08.757483+0000","last_clean":"2024-09-19T08:02:08.757483+0000","last_became_active":"2024-09-19T08:01:09.749013+0000","last_became_peered":"2024-09-19T08:01:09.749013+0000","last_unstale":"2024-09-19T08:02:08.757483+0000","last_undegraded":"2024-09-19T08:02:08.757483+0000","last_fullsized":"2024-09-19T08:02:08.757483+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:40:12.682158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":51,"num_read_kb":51,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721922+0000","last_change":"2024-09-19T08:01:10.774101+0000","last_active":"2024-09-19T08:01:26.721922+0000","last_peered":"2024-09-19T08:01:26.721922+0000","last_clean":"2024-09-19T08:01:26.721922+0000","last_became_active":"2024-09-19T08:01:10.773822+0000","last_became_peered":"2024-09-19T08:01:10.773822+0000","last_unstale":"2024-09-19T08:01:26.721922+0000","last_undegraded":"2024-09-19T08:01:26.721922+0000","last_fullsized":"2024-09-19T08:01:26.721922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:29.571118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"78'5","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:07.773538+0000","last_change":"2024-09-19T08:01:08.761675+0000","last_active":"2024-09-19T08:02:07.773538+0000","last_peered":"2024-09-19T08:02:07.773538+0000","last_clean":"2024-09-19T08:02:07.773538+0000","last_became_active":"2024-09-19T08:01:08.761346+0000","last_became_peered":"2024-09-19T08:01:08.761346+0000","last_unstale":"2024-09-19T08:02:07.773538+0000","last_undegraded":"2024-09-19T08:02:07.773538+0000","last_fullsized":"2024-09-19T08:02:07.773538+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:51:19.237458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718382+0000","last_change":"2024-09-19T08:01:10.781565+0000","last_active":"2024-09-19T08:01:26.718382+0000","last_peered":"2024-09-19T08:01:26.718382+0000","last_clean":"2024-09-19T08:01:26.718382+0000","last_became_active":"2024-09-19T08:01:10.781301+0000","last_became_peered":"2024-09-19T08:01:10.781301+0000","last_unstale":"2024-09-19T08:01:26.718382+0000","last_undegraded":"2024-09-19T08:01:26.718382+0000","last_fullsized":"2024-09-19T08:01:26.718382+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:55:52.932009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719945+0000","last_change":"2024-09-19T08:01:09.748918+0000","last_active":"2024-09-19T08:01:26.719945+0000","last_peered":"2024-09-19T08:01:26.719945+0000","last_clean":"2024-09-19T08:01:26.719945+0000","last_became_active":"2024-09-19T08:01:09.748497+0000","last_became_peered":"2024-09-19T08:01:09.748497+0000","last_unstale":"2024-09-19T08:01:26.719945+0000","last_undegraded":"2024-09-19T08:01:26.719945+0000","last_fullsized":"2024-09-19T08:01:26.719945+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:20:31.030133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718202+0000","last_change":"2024-09-19T08:01:10.781388+0000","last_active":"2024-09-19T08:01:26.718202+0000","last_peered":"2024-09-19T08:01:26.718202+0000","last_clean":"2024-09-19T08:01:26.718202+0000","last_became_active":"2024-09-19T08:01:10.781063+0000","last_became_peered":"2024-09-19T08:01:10.781063+0000","last_unstale":"2024-09-19T08:01:26.718202+0000","last_undegraded":"2024-09-19T08:01:26.718202+0000","last_fullsized":"2024-09-19T08:01:26.718202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:12:08.852717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723781+0000","last_change":"2024-09-19T08:01:09.750080+0000","last_active":"2024-09-19T08:01:26.723781+0000","last_peered":"2024-09-19T08:01:26.723781+0000","last_clean":"2024-09-19T08:01:26.723781+0000","last_became_active":"2024-09-19T08:01:09.749053+0000","last_became_peered":"2024-09-19T08:01:09.749053+0000","last_unstale":"2024-09-19T08:01:26.723781+0000","last_undegraded":"2024-09-19T08:01:26.723781+0000","last_fullsized":"2024-09-19T08:01:26.723781+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:15:45.292714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721665+0000","last_change":"2024-09-19T08:01:10.767070+0000","last_active":"2024-09-19T08:01:26.721665+0000","last_peered":"2024-09-19T08:01:26.721665+0000","last_clean":"2024-09-19T08:01:26.721665+0000","last_became_active":"2024-09-19T08:01:10.766760+0000","last_became_peered":"2024-09-19T08:01:10.766760+0000","last_unstale":"2024-09-19T08:01:26.721665+0000","last_undegraded":"2024-09-19T08:01:26.721665+0000","last_fullsized":"2024-09-19T08:01:26.721665+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:17.841135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'15","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:34.790661+0000","last_change":"2024-09-19T08:01:08.756143+0000","last_active":"2024-09-19T08:01:34.790661+0000","last_peered":"2024-09-19T08:01:34.790661+0000","last_clean":"2024-09-19T08:01:34.790661+0000","last_became_active":"2024-09-19T08:01:08.755777+0000","last_became_peered":"2024-09-19T08:01:08.755777+0000","last_unstale":"2024-09-19T08:01:34.790661+0000","last_undegraded":"2024-09-19T08:01:34.790661+0000","last_fullsized":"2024-09-19T08:01:34.790661+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:21.430673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718462+0000","last_change":"2024-09-19T08:01:10.774140+0000","last_active":"2024-09-19T08:01:26.718462+0000","last_peered":"2024-09-19T08:01:26.718462+0000","last_clean":"2024-09-19T08:01:26.718462+0000","last_became_active":"2024-09-19T08:01:10.773819+0000","last_became_peered":"2024-09-19T08:01:10.773819+0000","last_unstale":"2024-09-19T08:01:26.718462+0000","last_undegraded":"2024-09-19T08:01:26.718462+0000","last_fullsized":"2024-09-19T08:01:26.718462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:15:13.174013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758589+0000","last_change":"2024-09-19T08:01:09.169129+0000","last_active":"2024-09-19T08:01:26.758589+0000","last_peered":"2024-09-19T08:01:26.758589+0000","last_clean":"2024-09-19T08:01:26.758589+0000","last_became_active":"2024-09-19T08:01:09.168734+0000","last_became_peered":"2024-09-19T08:01:09.168734+0000","last_unstale":"2024-09-19T08:01:26.758589+0000","last_undegraded":"2024-09-19T08:01:26.758589+0000","last_fullsized":"2024-09-19T08:01:26.758589+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:01:52.232391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757822+0000","last_change":"2024-09-19T08:01:10.768592+0000","last_active":"2024-09-19T08:01:26.757822+0000","last_peered":"2024-09-19T08:01:26.757822+0000","last_clean":"2024-09-19T08:01:26.757822+0000","last_became_active":"2024-09-19T08:01:10.767645+0000","last_became_peered":"2024-09-19T08:01:10.767645+0000","last_unstale":"2024-09-19T08:01:26.757822+0000","last_undegraded":"2024-09-19T08:01:26.757822+0000","last_fullsized":"2024-09-19T08:01:26.757822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:08:46.227461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721037+0000","last_change":"2024-09-19T08:01:09.749240+0000","last_active":"2024-09-19T08:01:26.721037+0000","last_peered":"2024-09-19T08:01:26.721037+0000","last_clean":"2024-09-19T08:01:26.721037+0000","last_became_active":"2024-09-19T08:01:09.748945+0000","last_became_peered":"2024-09-19T08:01:09.748945+0000","last_unstale":"2024-09-19T08:01:26.721037+0000","last_undegraded":"2024-09-19T08:01:26.721037+0000","last_fullsized":"2024-09-19T08:01:26.721037+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:25:13.791290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.720802+0000","last_change":"2024-09-19T08:01:10.783133+0000","last_active":"2024-09-19T08:01:26.720802+0000","last_peered":"2024-09-19T08:01:26.720802+0000","last_clean":"2024-09-19T08:01:26.720802+0000","last_became_active":"2024-09-19T08:01:10.782797+0000","last_became_peered":"2024-09-19T08:01:10.782797+0000","last_unstale":"2024-09-19T08:01:26.720802+0000","last_undegraded":"2024-09-19T08:01:26.720802+0000","last_fullsized":"2024-09-19T08:01:26.720802+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:30:43.551877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718123+0000","last_change":"2024-09-19T08:01:08.757964+0000","last_active":"2024-09-19T08:01:26.718123+0000","last_peered":"2024-09-19T08:01:26.718123+0000","last_clean":"2024-09-19T08:01:26.718123+0000","last_became_active":"2024-09-19T08:01:08.757526+0000","last_became_peered":"2024-09-19T08:01:08.757526+0000","last_unstale":"2024-09-19T08:01:26.718123+0000","last_undegraded":"2024-09-19T08:01:26.718123+0000","last_fullsized":"2024-09-19T08:01:26.718123+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:03.565702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718652+0000","last_change":"2024-09-19T08:01:10.769974+0000","last_active":"2024-09-19T08:01:26.718652+0000","last_peered":"2024-09-19T08:01:26.718652+0000","last_clean":"2024-09-19T08:01:26.718652+0000","last_became_active":"2024-09-19T08:01:10.769650+0000","last_became_peered":"2024-09-19T08:01:10.769650+0000","last_unstale":"2024-09-19T08:01:26.718652+0000","last_undegraded":"2024-09-19T08:01:26.718652+0000","last_fullsized":"2024-09-19T08:01:26.718652+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:01.755198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725382+0000","last_change":"2024-09-19T08:01:08.763476+0000","last_active":"2024-09-19T08:01:26.725382+0000","last_peered":"2024-09-19T08:01:26.725382+0000","last_clean":"2024-09-19T08:01:26.725382+0000","last_became_active":"2024-09-19T08:01:08.763154+0000","last_became_peered":"2024-09-19T08:01:08.763154+0000","last_unstale":"2024-09-19T08:01:26.725382+0000","last_undegraded":"2024-09-19T08:01:26.725382+0000","last_fullsized":"2024-09-19T08:01:26.725382+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:06.202452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718164+0000","last_change":"2024-09-19T08:01:10.784201+0000","last_active":"2024-09-19T08:01:26.718164+0000","last_peered":"2024-09-19T08:01:26.718164+0000","last_clean":"2024-09-19T08:01:26.718164+0000","last_became_active":"2024-09-19T08:01:10.783867+0000","last_became_peered":"2024-09-19T08:01:10.783867+0000","last_unstale":"2024-09-19T08:01:26.718164+0000","last_undegraded":"2024-09-19T08:01:26.718164+0000","last_fullsized":"2024-09-19T08:01:26.718164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:43:50.082183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721581+0000","last_change":"2024-09-19T08:01:08.760479+0000","last_active":"2024-09-19T08:01:26.721581+0000","last_peered":"2024-09-19T08:01:26.721581+0000","last_clean":"2024-09-19T08:01:26.721581+0000","last_became_active":"2024-09-19T08:01:08.760004+0000","last_became_peered":"2024-09-19T08:01:08.760004+0000","last_unstale":"2024-09-19T08:01:26.721581+0000","last_undegraded":"2024-09-19T08:01:26.721581+0000","last_fullsized":"2024-09-19T08:01:26.721581+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:41:19.956623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722763+0000","last_change":"2024-09-19T08:01:10.772184+0000","last_active":"2024-09-19T08:01:26.722763+0000","last_peered":"2024-09-19T08:01:26.722763+0000","last_clean":"2024-09-19T08:01:26.722763+0000","last_became_active":"2024-09-19T08:01:10.771825+0000","last_became_peered":"2024-09-19T08:01:10.771825+0000","last_unstale":"2024-09-19T08:01:26.722763+0000","last_undegraded":"2024-09-19T08:01:26.722763+0000","last_fullsized":"2024-09-19T08:01:26.722763+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:36.764575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721944+0000","last_change":"2024-09-19T08:01:10.779368+0000","last_active":"2024-09-19T08:01:26.721944+0000","last_peered":"2024-09-19T08:01:26.721944+0000","last_clean":"2024-09-19T08:01:26.721944+0000","last_became_active":"2024-09-19T08:01:10.778884+0000","last_became_peered":"2024-09-19T08:01:10.778884+0000","last_unstale":"2024-09-19T08:01:26.721944+0000","last_undegraded":"2024-09-19T08:01:26.721944+0000","last_fullsized":"2024-09-19T08:01:26.721944+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:28:29.630300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.741496+0000","last_change":"2024-09-19T08:01:26.741053+0000","last_active":"2024-09-19T08:01:26.741496+0000","last_peered":"2024-09-19T08:01:26.741496+0000","last_clean":"2024-09-19T08:01:26.741496+0000","last_became_active":"2024-09-19T08:01:26.724714+0000","last_became_peered":"2024-09-19T08:01:26.724714+0000","last_unstale":"2024-09-19T08:01:26.741496+0000","last_undegraded":"2024-09-19T08:01:26.741496+0000","last_fullsized":"2024-09-19T08:01:26.741496+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:59.583797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721113+0000","last_change":"2024-09-19T08:01:10.781389+0000","last_active":"2024-09-19T08:01:26.721113+0000","last_peered":"2024-09-19T08:01:26.721113+0000","last_clean":"2024-09-19T08:01:26.721113+0000","last_became_active":"2024-09-19T08:01:10.781080+0000","last_became_peered":"2024-09-19T08:01:10.781080+0000","last_unstale":"2024-09-19T08:01:26.721113+0000","last_undegraded":"2024-09-19T08:01:26.721113+0000","last_fullsized":"2024-09-19T08:01:26.721113+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:40:45.225957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.768102+0000","last_change":"2024-09-19T08:01:26.768102+0000","last_active":"2024-09-19T08:01:26.768102+0000","last_peered":"2024-09-19T08:01:26.768102+0000","last_clean":"2024-09-19T08:01:26.768102+0000","last_became_active":"2024-09-19T08:01:26.759462+0000","last_became_peered":"2024-09-19T08:01:26.759462+0000","last_unstale":"2024-09-19T08:01:26.768102+0000","last_undegraded":"2024-09-19T08:01:26.768102+0000","last_fullsized":"2024-09-19T08:01:26.768102+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:02:06.815238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757966+0000","last_change":"2024-09-19T08:01:10.771031+0000","last_active":"2024-09-19T08:01:26.757966+0000","last_peered":"2024-09-19T08:01:26.757966+0000","last_clean":"2024-09-19T08:01:26.757966+0000","last_became_active":"2024-09-19T08:01:10.770675+0000","last_became_peered":"2024-09-19T08:01:10.770675+0000","last_unstale":"2024-09-19T08:01:26.757966+0000","last_undegraded":"2024-09-19T08:01:26.757966+0000","last_fullsized":"2024-09-19T08:01:26.757966+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:42.203848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718482+0000","last_change":"2024-09-19T08:01:09.748622+0000","last_active":"2024-09-19T08:01:26.718482+0000","last_peered":"2024-09-19T08:01:26.718482+0000","last_clean":"2024-09-19T08:01:26.718482+0000","last_became_active":"2024-09-19T08:01:09.748202+0000","last_became_peered":"2024-09-19T08:01:09.748202+0000","last_unstale":"2024-09-19T08:01:26.718482+0000","last_undegraded":"2024-09-19T08:01:26.718482+0000","last_fullsized":"2024-09-19T08:01:26.718482+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:17:39.861820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718292+0000","last_change":"2024-09-19T08:01:10.768949+0000","last_active":"2024-09-19T08:01:26.718292+0000","last_peered":"2024-09-19T08:01:26.718292+0000","last_clean":"2024-09-19T08:01:26.718292+0000","last_became_active":"2024-09-19T08:01:10.768159+0000","last_became_peered":"2024-09-19T08:01:10.768159+0000","last_unstale":"2024-09-19T08:01:26.718292+0000","last_undegraded":"2024-09-19T08:01:26.718292+0000","last_fullsized":"2024-09-19T08:01:26.718292+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:43.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"78'13","reported_seq":108,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:06.772239+0000","last_change":"2024-09-19T08:01:26.735877+0000","last_active":"2024-09-19T08:02:06.772239+0000","last_peered":"2024-09-19T08:02:06.772239+0000","last_clean":"2024-09-19T08:02:06.772239+0000","last_became_active":"2024-09-19T08:01:26.723597+0000","last_became_peered":"2024-09-19T08:01:26.723597+0000","last_unstale":"2024-09-19T08:02:06.772239+0000","last_undegraded":"2024-09-19T08:02:06.772239+0000","last_fullsized":"2024-09-19T08:02:06.772239+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:43:42.340832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,1],"acting":[9,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66972,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":509,"num_read_kb":330,"num_write":243,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1254,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":200916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645337,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":888832,"data_stored":427316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874270,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32724,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712428,"statfs":{"total":95995035648,"available":95961526272,"internally_reserved":0,"allocated":1576960,"data_stored":1076462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005091,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":901120,"data_stored":425599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234023,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":884736,"data_stored":424723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462956,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1486848,"data_stored":1017825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593778,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":884736,"data_stored":425316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822710,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28500,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716652,"statfs":{"total":95995035648,"available":95965851648,"internally_reserved":0,"allocated":1507328,"data_stored":1018175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986235,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":884736,"data_stored":425177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215167,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":901120,"data_stored":425141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476805,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":486999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607626,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":486665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":905216,"data_stored":427048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":62188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T08:02:14.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-19T08:02:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:14.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.951+0000 7fd8e4caf640 1 -- 172.21.15.98:0/3712844432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e0106a80 msgr2=0x7fd8e0106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:14.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.951+0000 7fd8e4caf640 1 --2- 172.21.15.98:0/3712844432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e0106a80 0x7fd8e0106e60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009930 tx=0x7fd8d002a420 comp rx=0 tx=0).stop 2024-09-19T08:02:14.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.951+0000 7fd8e4caf640 1 -- 172.21.15.98:0/3712844432 shutdown_connections 2024-09-19T08:02:14.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.951+0000 7fd8e4caf640 1 --2- 172.21.15.98:0/3712844432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8e0101420 0x7fd8e010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:14.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.951+0000 7fd8e4caf640 1 --2- 172.21.15.98:0/3712844432 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8e0100a80 0x7fd8e0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:14.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.951+0000 7fd8e4caf640 1 --2- 172.21.15.98:0/3712844432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e0106a80 0x7fd8e0106e60 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:14.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.951+0000 7fd8e4caf640 1 -- 172.21.15.98:0/3712844432 >> 172.21.15.98:0/3712844432 conn(0x7fd8e00fc7a0 msgr2=0x7fd8e00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:14.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.951+0000 7fd8e4caf640 1 -- 172.21.15.98:0/3712844432 shutdown_connections 2024-09-19T08:02:14.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.951+0000 7fd8e4caf640 1 -- 172.21.15.98:0/3712844432 wait complete. 2024-09-19T08:02:14.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.952+0000 7fd8e4caf640 1 Processor -- start 2024-09-19T08:02:14.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.952+0000 7fd8e4caf640 1 -- start start 2024-09-19T08:02:14.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.952+0000 7fd8e4caf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8e0100a80 0x7fd8e010e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:14.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.952+0000 7fd8e4caf640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8e0101420 0x7fd8e010e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:14.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.952+0000 7fd8e4caf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e0106a80 0x7fd8e010ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:14.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.952+0000 7fd8e4caf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e0111fa0 con 0x7fd8e0106a80 2024-09-19T08:02:14.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.952+0000 7fd8e4caf640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e0111e20 con 0x7fd8e0101420 2024-09-19T08:02:14.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.952+0000 7fd8e4caf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e0112120 con 0x7fd8e0100a80 2024-09-19T08:02:14.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.952+0000 7fd8ded9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e0106a80 0x7fd8e010ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:14.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.952+0000 7fd8ded9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e0106a80 0x7fd8e010ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48030/0 (socket says 172.21.15.98:48030) 2024-09-19T08:02:14.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.952+0000 7fd8ded9d640 1 -- 172.21.15.98:0/2125443148 learned_addr learned my addr 172.21.15.98:0/2125443148 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:14.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.953+0000 7fd8de59c640 1 --2- 172.21.15.98:0/2125443148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8e0100a80 0x7fd8e010e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:14.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.953+0000 7fd8ddd9b640 1 --2- 172.21.15.98:0/2125443148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8e0101420 0x7fd8e010e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:14.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.953+0000 7fd8ded9d640 1 -- 172.21.15.98:0/2125443148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8e0100a80 msgr2=0x7fd8e010e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:14.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.953+0000 7fd8ded9d640 1 --2- 172.21.15.98:0/2125443148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8e0100a80 0x7fd8e010e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:14.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.953+0000 7fd8ded9d640 1 -- 172.21.15.98:0/2125443148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8e0101420 msgr2=0x7fd8e010e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:14.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.953+0000 7fd8ded9d640 1 --2- 172.21.15.98:0/2125443148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8e0101420 0x7fd8e010e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:14.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.953+0000 7fd8ded9d640 1 -- 172.21.15.98:0/2125443148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e01a6940 con 0x7fd8e0106a80 2024-09-19T08:02:14.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.953+0000 7fd8ded9d640 1 --2- 172.21.15.98:0/2125443148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e0106a80 0x7fd8e010ee60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd8d800cce0 tx=0x7fd8d800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:14.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.953+0000 7fd8d77fe640 1 -- 172.21.15.98:0/2125443148 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d800f450 con 0x7fd8e0106a80 2024-09-19T08:02:14.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.953+0000 7fd8e4caf640 1 -- 172.21.15.98:0/2125443148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e01a6c30 con 0x7fd8e0106a80 2024-09-19T08:02:14.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.953+0000 7fd8d77fe640 1 -- 172.21.15.98:0/2125443148 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8d800fad0 con 0x7fd8e0106a80 2024-09-19T08:02:14.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.953+0000 7fd8d77fe640 1 -- 172.21.15.98:0/2125443148 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8018c10 con 0x7fd8e0106a80 2024-09-19T08:02:14.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.953+0000 7fd8e4caf640 1 -- 172.21.15.98:0/2125443148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e01a7170 con 0x7fd8e0106a80 2024-09-19T08:02:14.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.954+0000 7fd8e4caf640 1 -- 172.21.15.98:0/2125443148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8e0100ee0 con 0x7fd8e0106a80 2024-09-19T08:02:14.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.955+0000 7fd8d77fe640 1 -- 172.21.15.98:0/2125443148 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fd8d800f5f0 con 0x7fd8e0106a80 2024-09-19T08:02:14.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.956+0000 7fd8d77fe640 1 --2- 172.21.15.98:0/2125443148 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd8cc0a32d0 0x7fd8cc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:14.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.956+0000 7fd8de59c640 1 --2- 172.21.15.98:0/2125443148 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd8cc0a32d0 0x7fd8cc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:14.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.956+0000 7fd8d77fe640 1 -- 172.21.15.98:0/2125443148 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fd8d80d0200 con 0x7fd8e0106a80 2024-09-19T08:02:14.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.956+0000 7fd8de59c640 1 --2- 172.21.15.98:0/2125443148 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd8cc0a32d0 0x7fd8cc0a5790 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0000c00 tx=0x7fd8d0031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:14.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:14.960+0000 7fd8d77fe640 1 -- 172.21.15.98:0/2125443148 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fd8d8014030 con 0x7fd8e0106a80 2024-09-19T08:02:15.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:15.124+0000 7fd8e4caf640 1 -- 172.21.15.98:0/2125443148 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd8e0062630 con 0x7fd8cc0a32d0 2024-09-19T08:02:15.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:15.130+0000 7fd8d77fe640 1 -- 172.21.15.98:0/2125443148 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258753 (secure 0 0 0) 0x7fd8e0062630 con 0x7fd8cc0a32d0 2024-09-19T08:02:15.132 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:02:15.133 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-19T08:02:15.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:15.133+0000 7fd8e4caf640 1 -- 172.21.15.98:0/2125443148 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd8cc0a32d0 msgr2=0x7fd8cc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:15.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:15.133+0000 7fd8e4caf640 1 --2- 172.21.15.98:0/2125443148 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd8cc0a32d0 0x7fd8cc0a5790 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0000c00 tx=0x7fd8d0031490 comp rx=0 tx=0).stop 2024-09-19T08:02:15.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:15.133+0000 7fd8e4caf640 1 -- 172.21.15.98:0/2125443148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e0106a80 msgr2=0x7fd8e010ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:15.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:15.133+0000 7fd8e4caf640 1 --2- 172.21.15.98:0/2125443148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e0106a80 0x7fd8e010ee60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd8d800cce0 tx=0x7fd8d800a620 comp rx=0 tx=0).stop 2024-09-19T08:02:15.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:15.133+0000 7fd8e4caf640 1 -- 172.21.15.98:0/2125443148 shutdown_connections 2024-09-19T08:02:15.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:15.133+0000 7fd8e4caf640 1 --2- 172.21.15.98:0/2125443148 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd8cc0a32d0 0x7fd8cc0a5790 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:15.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:15.133+0000 7fd8e4caf640 1 --2- 172.21.15.98:0/2125443148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e0106a80 0x7fd8e010ee60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:15.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:15.133+0000 7fd8e4caf640 1 --2- 172.21.15.98:0/2125443148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8e0101420 0x7fd8e010e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:15.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:15.133+0000 7fd8e4caf640 1 --2- 172.21.15.98:0/2125443148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8e0100a80 0x7fd8e010e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:15.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:15.133+0000 7fd8e4caf640 1 -- 172.21.15.98:0/2125443148 >> 172.21.15.98:0/2125443148 conn(0x7fd8e00fc7a0 msgr2=0x7fd8e00fcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:15.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:15.133+0000 7fd8e4caf640 1 -- 172.21.15.98:0/2125443148 shutdown_connections 2024-09-19T08:02:15.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:15.133+0000 7fd8e4caf640 1 -- 172.21.15.98:0/2125443148 wait complete. 2024-09-19T08:02:15.862 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-19T08:02:13.582092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657574,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":660,"num_read_kb":449,"num_write":426,"num_write_kb":3074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181990,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370044,"kb_used_data":12444,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571780,"statfs":{"total":1151940427776,"available":1151561502720,"internally_reserved":0,"allocated":12742656,"data_stored":7066446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002683"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719055+0000","last_change":"2024-09-19T08:01:10.768950+0000","last_active":"2024-09-19T08:01:26.719055+0000","last_peered":"2024-09-19T08:01:26.719055+0000","last_clean":"2024-09-19T08:01:26.719055+0000","last_became_active":"2024-09-19T08:01:10.768047+0000","last_became_peered":"2024-09-19T08:01:10.768047+0000","last_unstale":"2024-09-19T08:01:26.719055+0000","last_undegraded":"2024-09-19T08:01:26.719055+0000","last_fullsized":"2024-09-19T08:01:26.719055+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:26.219079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757858+0000","last_change":"2024-09-19T08:01:10.783208+0000","last_active":"2024-09-19T08:01:26.757858+0000","last_peered":"2024-09-19T08:01:26.757858+0000","last_clean":"2024-09-19T08:01:26.757858+0000","last_became_active":"2024-09-19T08:01:10.782888+0000","last_became_peered":"2024-09-19T08:01:10.782888+0000","last_unstale":"2024-09-19T08:01:26.757858+0000","last_undegraded":"2024-09-19T08:01:26.757858+0000","last_fullsized":"2024-09-19T08:01:26.757858+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:51:27.603166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757759+0000","last_change":"2024-09-19T08:01:10.770395+0000","last_active":"2024-09-19T08:01:26.757759+0000","last_peered":"2024-09-19T08:01:26.757759+0000","last_clean":"2024-09-19T08:01:26.757759+0000","last_became_active":"2024-09-19T08:01:10.770073+0000","last_became_peered":"2024-09-19T08:01:10.770073+0000","last_unstale":"2024-09-19T08:01:26.757759+0000","last_undegraded":"2024-09-19T08:01:26.757759+0000","last_fullsized":"2024-09-19T08:01:26.757759+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:20.047023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758411+0000","last_change":"2024-09-19T08:01:10.777138+0000","last_active":"2024-09-19T08:01:26.758411+0000","last_peered":"2024-09-19T08:01:26.758411+0000","last_clean":"2024-09-19T08:01:26.758411+0000","last_became_active":"2024-09-19T08:01:10.776810+0000","last_became_peered":"2024-09-19T08:01:10.776810+0000","last_unstale":"2024-09-19T08:01:26.758411+0000","last_undegraded":"2024-09-19T08:01:26.758411+0000","last_fullsized":"2024-09-19T08:01:26.758411+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:39.983321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721379+0000","last_change":"2024-09-19T08:01:10.770435+0000","last_active":"2024-09-19T08:01:26.721379+0000","last_peered":"2024-09-19T08:01:26.721379+0000","last_clean":"2024-09-19T08:01:26.721379+0000","last_became_active":"2024-09-19T08:01:10.769686+0000","last_became_peered":"2024-09-19T08:01:10.769686+0000","last_unstale":"2024-09-19T08:01:26.721379+0000","last_undegraded":"2024-09-19T08:01:26.721379+0000","last_fullsized":"2024-09-19T08:01:26.721379+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:10:19.610925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718127+0000","last_change":"2024-09-19T08:01:10.782382+0000","last_active":"2024-09-19T08:01:26.718127+0000","last_peered":"2024-09-19T08:01:26.718127+0000","last_clean":"2024-09-19T08:01:26.718127+0000","last_became_active":"2024-09-19T08:01:10.781996+0000","last_became_peered":"2024-09-19T08:01:10.781996+0000","last_unstale":"2024-09-19T08:01:26.718127+0000","last_undegraded":"2024-09-19T08:01:26.718127+0000","last_fullsized":"2024-09-19T08:01:26.718127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:43:13.574336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722516+0000","last_change":"2024-09-19T08:01:10.773776+0000","last_active":"2024-09-19T08:01:26.722516+0000","last_peered":"2024-09-19T08:01:26.722516+0000","last_clean":"2024-09-19T08:01:26.722516+0000","last_became_active":"2024-09-19T08:01:10.772442+0000","last_became_peered":"2024-09-19T08:01:10.772442+0000","last_unstale":"2024-09-19T08:01:26.722516+0000","last_undegraded":"2024-09-19T08:01:26.722516+0000","last_fullsized":"2024-09-19T08:01:26.722516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:08:18.155806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.717962+0000","last_change":"2024-09-19T08:01:10.781443+0000","last_active":"2024-09-19T08:01:26.717962+0000","last_peered":"2024-09-19T08:01:26.717962+0000","last_clean":"2024-09-19T08:01:26.717962+0000","last_became_active":"2024-09-19T08:01:10.781083+0000","last_became_peered":"2024-09-19T08:01:10.781083+0000","last_unstale":"2024-09-19T08:01:26.717962+0000","last_undegraded":"2024-09-19T08:01:26.717962+0000","last_fullsized":"2024-09-19T08:01:26.717962+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:23:22.631865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721840+0000","last_change":"2024-09-19T08:01:10.767554+0000","last_active":"2024-09-19T08:01:26.721840+0000","last_peered":"2024-09-19T08:01:26.721840+0000","last_clean":"2024-09-19T08:01:26.721840+0000","last_became_active":"2024-09-19T08:01:10.767227+0000","last_became_peered":"2024-09-19T08:01:10.767227+0000","last_unstale":"2024-09-19T08:01:26.721840+0000","last_undegraded":"2024-09-19T08:01:26.721840+0000","last_fullsized":"2024-09-19T08:01:26.721840+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:30:13.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718514+0000","last_change":"2024-09-19T08:01:10.783174+0000","last_active":"2024-09-19T08:01:26.718514+0000","last_peered":"2024-09-19T08:01:26.718514+0000","last_clean":"2024-09-19T08:01:26.718514+0000","last_became_active":"2024-09-19T08:01:10.782794+0000","last_became_peered":"2024-09-19T08:01:10.782794+0000","last_unstale":"2024-09-19T08:01:26.718514+0000","last_undegraded":"2024-09-19T08:01:26.718514+0000","last_fullsized":"2024-09-19T08:01:26.718514+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:49:19.451107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723075+0000","last_change":"2024-09-19T08:01:10.779638+0000","last_active":"2024-09-19T08:01:26.723075+0000","last_peered":"2024-09-19T08:01:26.723075+0000","last_clean":"2024-09-19T08:01:26.723075+0000","last_became_active":"2024-09-19T08:01:10.779295+0000","last_became_peered":"2024-09-19T08:01:10.779295+0000","last_unstale":"2024-09-19T08:01:26.723075+0000","last_undegraded":"2024-09-19T08:01:26.723075+0000","last_fullsized":"2024-09-19T08:01:26.723075+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:35:10.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758492+0000","last_change":"2024-09-19T08:01:10.784487+0000","last_active":"2024-09-19T08:01:26.758492+0000","last_peered":"2024-09-19T08:01:26.758492+0000","last_clean":"2024-09-19T08:01:26.758492+0000","last_became_active":"2024-09-19T08:01:10.784114+0000","last_became_peered":"2024-09-19T08:01:10.784114+0000","last_unstale":"2024-09-19T08:01:26.758492+0000","last_undegraded":"2024-09-19T08:01:26.758492+0000","last_fullsized":"2024-09-19T08:01:26.758492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:09:54.819802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721613+0000","last_change":"2024-09-19T08:01:10.783186+0000","last_active":"2024-09-19T08:01:26.721613+0000","last_peered":"2024-09-19T08:01:26.721613+0000","last_clean":"2024-09-19T08:01:26.721613+0000","last_became_active":"2024-09-19T08:01:10.782876+0000","last_became_peered":"2024-09-19T08:01:10.782876+0000","last_unstale":"2024-09-19T08:01:26.721613+0000","last_undegraded":"2024-09-19T08:01:26.721613+0000","last_fullsized":"2024-09-19T08:01:26.721613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:26:15.252165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757647+0000","last_change":"2024-09-19T08:01:10.783237+0000","last_active":"2024-09-19T08:01:26.757647+0000","last_peered":"2024-09-19T08:01:26.757647+0000","last_clean":"2024-09-19T08:01:26.757647+0000","last_became_active":"2024-09-19T08:01:10.782966+0000","last_became_peered":"2024-09-19T08:01:10.782966+0000","last_unstale":"2024-09-19T08:01:26.757647+0000","last_undegraded":"2024-09-19T08:01:26.757647+0000","last_fullsized":"2024-09-19T08:01:26.757647+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:57:18.771737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757535+0000","last_change":"2024-09-19T08:01:10.771207+0000","last_active":"2024-09-19T08:01:26.757535+0000","last_peered":"2024-09-19T08:01:26.757535+0000","last_clean":"2024-09-19T08:01:26.757535+0000","last_became_active":"2024-09-19T08:01:10.770862+0000","last_became_peered":"2024-09-19T08:01:10.770862+0000","last_unstale":"2024-09-19T08:01:26.757535+0000","last_undegraded":"2024-09-19T08:01:26.757535+0000","last_fullsized":"2024-09-19T08:01:26.757535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:08:09.313329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.720764+0000","last_change":"2024-09-19T08:01:10.775191+0000","last_active":"2024-09-19T08:01:26.720764+0000","last_peered":"2024-09-19T08:01:26.720764+0000","last_clean":"2024-09-19T08:01:26.720764+0000","last_became_active":"2024-09-19T08:01:10.774832+0000","last_became_peered":"2024-09-19T08:01:10.774832+0000","last_unstale":"2024-09-19T08:01:26.720764+0000","last_undegraded":"2024-09-19T08:01:26.720764+0000","last_fullsized":"2024-09-19T08:01:26.720764+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:57:29.800787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719076+0000","last_change":"2024-09-19T08:01:10.769115+0000","last_active":"2024-09-19T08:01:26.719076+0000","last_peered":"2024-09-19T08:01:26.719076+0000","last_clean":"2024-09-19T08:01:26.719076+0000","last_became_active":"2024-09-19T08:01:10.768088+0000","last_became_peered":"2024-09-19T08:01:10.768088+0000","last_unstale":"2024-09-19T08:01:26.719076+0000","last_undegraded":"2024-09-19T08:01:26.719076+0000","last_fullsized":"2024-09-19T08:01:26.719076+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:27:16.042722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719916+0000","last_change":"2024-09-19T08:01:10.781426+0000","last_active":"2024-09-19T08:01:26.719916+0000","last_peered":"2024-09-19T08:01:26.719916+0000","last_clean":"2024-09-19T08:01:26.719916+0000","last_became_active":"2024-09-19T08:01:10.781094+0000","last_became_peered":"2024-09-19T08:01:10.781094+0000","last_unstale":"2024-09-19T08:01:26.719916+0000","last_undegraded":"2024-09-19T08:01:26.719916+0000","last_fullsized":"2024-09-19T08:01:26.719916+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:43:31.336610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725515+0000","last_change":"2024-09-19T08:01:10.766466+0000","last_active":"2024-09-19T08:01:26.725515+0000","last_peered":"2024-09-19T08:01:26.725515+0000","last_clean":"2024-09-19T08:01:26.725515+0000","last_became_active":"2024-09-19T08:01:10.766123+0000","last_became_peered":"2024-09-19T08:01:10.766123+0000","last_unstale":"2024-09-19T08:01:26.725515+0000","last_undegraded":"2024-09-19T08:01:26.725515+0000","last_fullsized":"2024-09-19T08:01:26.725515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:25.848603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718613+0000","last_change":"2024-09-19T08:01:10.772782+0000","last_active":"2024-09-19T08:01:26.718613+0000","last_peered":"2024-09-19T08:01:26.718613+0000","last_clean":"2024-09-19T08:01:26.718613+0000","last_became_active":"2024-09-19T08:01:10.772445+0000","last_became_peered":"2024-09-19T08:01:10.772445+0000","last_unstale":"2024-09-19T08:01:26.718613+0000","last_undegraded":"2024-09-19T08:01:26.718613+0000","last_fullsized":"2024-09-19T08:01:26.718613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:09.174005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721701+0000","last_change":"2024-09-19T08:01:10.771944+0000","last_active":"2024-09-19T08:01:26.721701+0000","last_peered":"2024-09-19T08:01:26.721701+0000","last_clean":"2024-09-19T08:01:26.721701+0000","last_became_active":"2024-09-19T08:01:10.771643+0000","last_became_peered":"2024-09-19T08:01:10.771643+0000","last_unstale":"2024-09-19T08:01:26.721701+0000","last_undegraded":"2024-09-19T08:01:26.721701+0000","last_fullsized":"2024-09-19T08:01:26.721701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:46:27.121722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721201+0000","last_change":"2024-09-19T08:01:10.775144+0000","last_active":"2024-09-19T08:01:26.721201+0000","last_peered":"2024-09-19T08:01:26.721201+0000","last_clean":"2024-09-19T08:01:26.721201+0000","last_became_active":"2024-09-19T08:01:10.774769+0000","last_became_peered":"2024-09-19T08:01:10.774769+0000","last_unstale":"2024-09-19T08:01:26.721201+0000","last_undegraded":"2024-09-19T08:01:26.721201+0000","last_fullsized":"2024-09-19T08:01:26.721201+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:59:47.053993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718014+0000","last_change":"2024-09-19T08:01:10.766750+0000","last_active":"2024-09-19T08:01:26.718014+0000","last_peered":"2024-09-19T08:01:26.718014+0000","last_clean":"2024-09-19T08:01:26.718014+0000","last_became_active":"2024-09-19T08:01:10.766450+0000","last_became_peered":"2024-09-19T08:01:10.766450+0000","last_unstale":"2024-09-19T08:01:26.718014+0000","last_undegraded":"2024-09-19T08:01:26.718014+0000","last_fullsized":"2024-09-19T08:01:26.718014+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:47:58.133815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721798+0000","last_change":"2024-09-19T08:01:10.768430+0000","last_active":"2024-09-19T08:01:26.721798+0000","last_peered":"2024-09-19T08:01:26.721798+0000","last_clean":"2024-09-19T08:01:26.721798+0000","last_became_active":"2024-09-19T08:01:10.768092+0000","last_became_peered":"2024-09-19T08:01:10.768092+0000","last_unstale":"2024-09-19T08:01:26.721798+0000","last_undegraded":"2024-09-19T08:01:26.721798+0000","last_fullsized":"2024-09-19T08:01:26.721798+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:16:42.343138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718376+0000","last_change":"2024-09-19T08:01:10.771131+0000","last_active":"2024-09-19T08:01:26.718376+0000","last_peered":"2024-09-19T08:01:26.718376+0000","last_clean":"2024-09-19T08:01:26.718376+0000","last_became_active":"2024-09-19T08:01:10.770851+0000","last_became_peered":"2024-09-19T08:01:10.770851+0000","last_unstale":"2024-09-19T08:01:26.718376+0000","last_undegraded":"2024-09-19T08:01:26.718376+0000","last_fullsized":"2024-09-19T08:01:26.718376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:07:15.830476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.726129+0000","last_change":"2024-09-19T08:01:10.776220+0000","last_active":"2024-09-19T08:01:26.726129+0000","last_peered":"2024-09-19T08:01:26.726129+0000","last_clean":"2024-09-19T08:01:26.726129+0000","last_became_active":"2024-09-19T08:01:10.775889+0000","last_became_peered":"2024-09-19T08:01:10.775889+0000","last_unstale":"2024-09-19T08:01:26.726129+0000","last_undegraded":"2024-09-19T08:01:26.726129+0000","last_fullsized":"2024-09-19T08:01:26.726129+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:23.714675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719003+0000","last_change":"2024-09-19T08:01:10.776370+0000","last_active":"2024-09-19T08:01:26.719003+0000","last_peered":"2024-09-19T08:01:26.719003+0000","last_clean":"2024-09-19T08:01:26.719003+0000","last_became_active":"2024-09-19T08:01:10.776027+0000","last_became_peered":"2024-09-19T08:01:10.776027+0000","last_unstale":"2024-09-19T08:01:26.719003+0000","last_undegraded":"2024-09-19T08:01:26.719003+0000","last_fullsized":"2024-09-19T08:01:26.719003+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:48:46.293404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758971+0000","last_change":"2024-09-19T08:01:26.758971+0000","last_active":"2024-09-19T08:01:26.758971+0000","last_peered":"2024-09-19T08:01:26.758971+0000","last_clean":"2024-09-19T08:01:26.758971+0000","last_became_active":"2024-09-19T08:01:26.758524+0000","last_became_peered":"2024-09-19T08:01:26.758524+0000","last_unstale":"2024-09-19T08:01:26.758971+0000","last_undegraded":"2024-09-19T08:01:26.758971+0000","last_fullsized":"2024-09-19T08:01:26.758971+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:34:22.439327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757760+0000","last_change":"2024-09-19T08:01:10.767445+0000","last_active":"2024-09-19T08:01:26.757760+0000","last_peered":"2024-09-19T08:01:26.757760+0000","last_clean":"2024-09-19T08:01:26.757760+0000","last_became_active":"2024-09-19T08:01:10.767128+0000","last_became_peered":"2024-09-19T08:01:10.767128+0000","last_unstale":"2024-09-19T08:01:26.757760+0000","last_undegraded":"2024-09-19T08:01:26.757760+0000","last_fullsized":"2024-09-19T08:01:26.757760+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:10.273948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725212+0000","last_change":"2024-09-19T08:01:10.768954+0000","last_active":"2024-09-19T08:01:26.725212+0000","last_peered":"2024-09-19T08:01:26.725212+0000","last_clean":"2024-09-19T08:01:26.725212+0000","last_became_active":"2024-09-19T08:01:10.768577+0000","last_became_peered":"2024-09-19T08:01:10.768577+0000","last_unstale":"2024-09-19T08:01:26.725212+0000","last_undegraded":"2024-09-19T08:01:26.725212+0000","last_fullsized":"2024-09-19T08:01:26.725212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:20:21.812531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725473+0000","last_change":"2024-09-19T08:01:10.774104+0000","last_active":"2024-09-19T08:01:26.725473+0000","last_peered":"2024-09-19T08:01:26.725473+0000","last_clean":"2024-09-19T08:01:26.725473+0000","last_became_active":"2024-09-19T08:01:10.773721+0000","last_became_peered":"2024-09-19T08:01:10.773721+0000","last_unstale":"2024-09-19T08:01:26.725473+0000","last_undegraded":"2024-09-19T08:01:26.725473+0000","last_fullsized":"2024-09-19T08:01:26.725473+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:45:34.945994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.727296+0000","last_change":"2024-09-19T08:01:26.727296+0000","last_active":"2024-09-19T08:01:26.727296+0000","last_peered":"2024-09-19T08:01:26.727296+0000","last_clean":"2024-09-19T08:01:26.727296+0000","last_became_active":"2024-09-19T08:01:26.726631+0000","last_became_peered":"2024-09-19T08:01:26.726631+0000","last_unstale":"2024-09-19T08:01:26.727296+0000","last_undegraded":"2024-09-19T08:01:26.727296+0000","last_fullsized":"2024-09-19T08:01:26.727296+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:19:24.807050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721044+0000","last_change":"2024-09-19T08:01:10.765282+0000","last_active":"2024-09-19T08:01:26.721044+0000","last_peered":"2024-09-19T08:01:26.721044+0000","last_clean":"2024-09-19T08:01:26.721044+0000","last_became_active":"2024-09-19T08:01:10.764928+0000","last_became_peered":"2024-09-19T08:01:10.764928+0000","last_unstale":"2024-09-19T08:01:26.721044+0000","last_undegraded":"2024-09-19T08:01:26.721044+0000","last_fullsized":"2024-09-19T08:01:26.721044+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:00:37.395110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"78'6","reported_seq":82,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:09.760074+0000","last_change":"2024-09-19T08:01:08.760686+0000","last_active":"2024-09-19T08:02:09.760074+0000","last_peered":"2024-09-19T08:02:09.760074+0000","last_clean":"2024-09-19T08:02:09.760074+0000","last_became_active":"2024-09-19T08:01:08.759912+0000","last_became_peered":"2024-09-19T08:01:08.759912+0000","last_unstale":"2024-09-19T08:02:09.760074+0000","last_undegraded":"2024-09-19T08:02:09.760074+0000","last_fullsized":"2024-09-19T08:02:09.760074+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:46:01.879611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721868+0000","last_change":"2024-09-19T08:01:10.779668+0000","last_active":"2024-09-19T08:01:26.721868+0000","last_peered":"2024-09-19T08:01:26.721868+0000","last_clean":"2024-09-19T08:01:26.721868+0000","last_became_active":"2024-09-19T08:01:10.779265+0000","last_became_peered":"2024-09-19T08:01:10.779265+0000","last_unstale":"2024-09-19T08:01:26.721868+0000","last_undegraded":"2024-09-19T08:01:26.721868+0000","last_fullsized":"2024-09-19T08:01:26.721868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:06.177700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718183+0000","last_change":"2024-09-19T08:01:08.762668+0000","last_active":"2024-09-19T08:01:26.718183+0000","last_peered":"2024-09-19T08:01:26.718183+0000","last_clean":"2024-09-19T08:01:26.718183+0000","last_became_active":"2024-09-19T08:01:08.762277+0000","last_became_peered":"2024-09-19T08:01:08.762277+0000","last_unstale":"2024-09-19T08:01:26.718183+0000","last_undegraded":"2024-09-19T08:01:26.718183+0000","last_fullsized":"2024-09-19T08:01:26.718183+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:36:59.316655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721452+0000","last_change":"2024-09-19T08:01:08.757401+0000","last_active":"2024-09-19T08:01:26.721452+0000","last_peered":"2024-09-19T08:01:26.721452+0000","last_clean":"2024-09-19T08:01:26.721452+0000","last_became_active":"2024-09-19T08:01:08.757067+0000","last_became_peered":"2024-09-19T08:01:08.757067+0000","last_unstale":"2024-09-19T08:01:26.721452+0000","last_undegraded":"2024-09-19T08:01:26.721452+0000","last_fullsized":"2024-09-19T08:01:26.721452+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:19.617830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718027+0000","last_change":"2024-09-19T08:01:10.765149+0000","last_active":"2024-09-19T08:01:26.718027+0000","last_peered":"2024-09-19T08:01:26.718027+0000","last_clean":"2024-09-19T08:01:26.718027+0000","last_became_active":"2024-09-19T08:01:10.764840+0000","last_became_peered":"2024-09-19T08:01:10.764840+0000","last_unstale":"2024-09-19T08:01:26.718027+0000","last_undegraded":"2024-09-19T08:01:26.718027+0000","last_fullsized":"2024-09-19T08:01:26.718027+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:15.539307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722074+0000","last_change":"2024-09-19T08:01:08.759435+0000","last_active":"2024-09-19T08:01:26.722074+0000","last_peered":"2024-09-19T08:01:26.722074+0000","last_clean":"2024-09-19T08:01:26.722074+0000","last_became_active":"2024-09-19T08:01:08.759046+0000","last_became_peered":"2024-09-19T08:01:08.759046+0000","last_unstale":"2024-09-19T08:01:26.722074+0000","last_undegraded":"2024-09-19T08:01:26.722074+0000","last_fullsized":"2024-09-19T08:01:26.722074+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:21:24.077626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719748+0000","last_change":"2024-09-19T08:01:10.778432+0000","last_active":"2024-09-19T08:01:26.719748+0000","last_peered":"2024-09-19T08:01:26.719748+0000","last_clean":"2024-09-19T08:01:26.719748+0000","last_became_active":"2024-09-19T08:01:10.778098+0000","last_became_peered":"2024-09-19T08:01:10.778098+0000","last_unstale":"2024-09-19T08:01:26.719748+0000","last_undegraded":"2024-09-19T08:01:26.719748+0000","last_fullsized":"2024-09-19T08:01:26.719748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:40:12.921165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"78'4","reported_seq":120,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:07.756756+0000","last_change":"2024-09-19T08:01:08.764117+0000","last_active":"2024-09-19T08:02:07.756756+0000","last_peered":"2024-09-19T08:02:07.756756+0000","last_clean":"2024-09-19T08:02:07.756756+0000","last_became_active":"2024-09-19T08:01:08.763308+0000","last_became_peered":"2024-09-19T08:01:08.763308+0000","last_unstale":"2024-09-19T08:02:07.756756+0000","last_undegraded":"2024-09-19T08:02:07.756756+0000","last_fullsized":"2024-09-19T08:02:07.756756+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:19:57.014628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":94,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721336+0000","last_change":"2024-09-19T08:01:10.781771+0000","last_active":"2024-09-19T08:01:26.721336+0000","last_peered":"2024-09-19T08:01:26.721336+0000","last_clean":"2024-09-19T08:01:26.721336+0000","last_became_active":"2024-09-19T08:01:10.781432+0000","last_became_peered":"2024-09-19T08:01:10.781432+0000","last_unstale":"2024-09-19T08:01:26.721336+0000","last_undegraded":"2024-09-19T08:01:26.721336+0000","last_fullsized":"2024-09-19T08:01:26.721336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:25:15.067850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.771171+0000","last_change":"2024-09-19T08:01:26.771171+0000","last_active":"2024-09-19T08:01:26.771171+0000","last_peered":"2024-09-19T08:01:26.771171+0000","last_clean":"2024-09-19T08:01:26.771171+0000","last_became_active":"2024-09-19T08:01:26.759274+0000","last_became_peered":"2024-09-19T08:01:26.759274+0000","last_unstale":"2024-09-19T08:01:26.771171+0000","last_undegraded":"2024-09-19T08:01:26.771171+0000","last_fullsized":"2024-09-19T08:01:26.771171+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:20:54.603420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757967+0000","last_change":"2024-09-19T08:01:10.768318+0000","last_active":"2024-09-19T08:01:26.757967+0000","last_peered":"2024-09-19T08:01:26.757967+0000","last_clean":"2024-09-19T08:01:26.757967+0000","last_became_active":"2024-09-19T08:01:10.768022+0000","last_became_peered":"2024-09-19T08:01:10.768022+0000","last_unstale":"2024-09-19T08:01:26.757967+0000","last_undegraded":"2024-09-19T08:01:26.757967+0000","last_fullsized":"2024-09-19T08:01:26.757967+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:00:45.328638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722058+0000","last_change":"2024-09-19T08:01:09.750530+0000","last_active":"2024-09-19T08:01:26.722058+0000","last_peered":"2024-09-19T08:01:26.722058+0000","last_clean":"2024-09-19T08:01:26.722058+0000","last_became_active":"2024-09-19T08:01:09.750029+0000","last_became_peered":"2024-09-19T08:01:09.750029+0000","last_unstale":"2024-09-19T08:01:26.722058+0000","last_undegraded":"2024-09-19T08:01:26.722058+0000","last_fullsized":"2024-09-19T08:01:26.722058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:01:16.062992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.724636+0000","last_change":"2024-09-19T08:01:26.724636+0000","last_active":"2024-09-19T08:01:26.724636+0000","last_peered":"2024-09-19T08:01:26.724636+0000","last_clean":"2024-09-19T08:01:26.724636+0000","last_became_active":"2024-09-19T08:01:26.724171+0000","last_became_peered":"2024-09-19T08:01:26.724171+0000","last_unstale":"2024-09-19T08:01:26.724636+0000","last_undegraded":"2024-09-19T08:01:26.724636+0000","last_fullsized":"2024-09-19T08:01:26.724636+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:54:05.998374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'30","reported_seq":157,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:07.756821+0000","last_change":"2024-09-19T08:01:08.761950+0000","last_active":"2024-09-19T08:02:07.756821+0000","last_peered":"2024-09-19T08:02:07.756821+0000","last_clean":"2024-09-19T08:02:07.756821+0000","last_became_active":"2024-09-19T08:01:08.761533+0000","last_became_peered":"2024-09-19T08:01:08.761533+0000","last_unstale":"2024-09-19T08:02:07.756821+0000","last_undegraded":"2024-09-19T08:02:07.756821+0000","last_fullsized":"2024-09-19T08:02:07.756821+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:07:26.668141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59764,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":105,"num_read_kb":110,"num_write":32,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723579+0000","last_change":"2024-09-19T08:01:10.774280+0000","last_active":"2024-09-19T08:01:26.723579+0000","last_peered":"2024-09-19T08:01:26.723579+0000","last_clean":"2024-09-19T08:01:26.723579+0000","last_became_active":"2024-09-19T08:01:10.773982+0000","last_became_peered":"2024-09-19T08:01:10.773982+0000","last_unstale":"2024-09-19T08:01:26.723579+0000","last_undegraded":"2024-09-19T08:01:26.723579+0000","last_fullsized":"2024-09-19T08:01:26.723579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:21:43.180057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725363+0000","last_change":"2024-09-19T08:01:08.761067+0000","last_active":"2024-09-19T08:01:26.725363+0000","last_peered":"2024-09-19T08:01:26.725363+0000","last_clean":"2024-09-19T08:01:26.725363+0000","last_became_active":"2024-09-19T08:01:08.760664+0000","last_became_peered":"2024-09-19T08:01:08.760664+0000","last_unstale":"2024-09-19T08:01:26.725363+0000","last_undegraded":"2024-09-19T08:01:26.725363+0000","last_fullsized":"2024-09-19T08:01:26.725363+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:22:38.086891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721462+0000","last_change":"2024-09-19T08:01:10.771845+0000","last_active":"2024-09-19T08:01:26.721462+0000","last_peered":"2024-09-19T08:01:26.721462+0000","last_clean":"2024-09-19T08:01:26.721462+0000","last_became_active":"2024-09-19T08:01:10.771533+0000","last_became_peered":"2024-09-19T08:01:10.771533+0000","last_unstale":"2024-09-19T08:01:26.721462+0000","last_undegraded":"2024-09-19T08:01:26.721462+0000","last_fullsized":"2024-09-19T08:01:26.721462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:29:35.180763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722806+0000","last_change":"2024-09-19T08:01:08.763869+0000","last_active":"2024-09-19T08:01:26.722806+0000","last_peered":"2024-09-19T08:01:26.722806+0000","last_clean":"2024-09-19T08:01:26.722806+0000","last_became_active":"2024-09-19T08:01:08.763451+0000","last_became_peered":"2024-09-19T08:01:08.763451+0000","last_unstale":"2024-09-19T08:01:26.722806+0000","last_undegraded":"2024-09-19T08:01:26.722806+0000","last_fullsized":"2024-09-19T08:01:26.722806+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:32:40.901049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722991+0000","last_change":"2024-09-19T08:01:10.770823+0000","last_active":"2024-09-19T08:01:26.722991+0000","last_peered":"2024-09-19T08:01:26.722991+0000","last_clean":"2024-09-19T08:01:26.722991+0000","last_became_active":"2024-09-19T08:01:10.770462+0000","last_became_peered":"2024-09-19T08:01:10.770462+0000","last_unstale":"2024-09-19T08:01:26.722991+0000","last_undegraded":"2024-09-19T08:01:26.722991+0000","last_fullsized":"2024-09-19T08:01:26.722991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:04:32.832009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"78'5","reported_seq":80,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:09.759504+0000","last_change":"2024-09-19T08:01:08.759429+0000","last_active":"2024-09-19T08:02:09.759504+0000","last_peered":"2024-09-19T08:02:09.759504+0000","last_clean":"2024-09-19T08:02:09.759504+0000","last_became_active":"2024-09-19T08:01:08.759062+0000","last_became_peered":"2024-09-19T08:01:08.759062+0000","last_unstale":"2024-09-19T08:02:09.759504+0000","last_undegraded":"2024-09-19T08:02:09.759504+0000","last_fullsized":"2024-09-19T08:02:09.759504+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:16:51.582435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722418+0000","last_change":"2024-09-19T08:01:10.784735+0000","last_active":"2024-09-19T08:01:26.722418+0000","last_peered":"2024-09-19T08:01:26.722418+0000","last_clean":"2024-09-19T08:01:26.722418+0000","last_became_active":"2024-09-19T08:01:10.784346+0000","last_became_peered":"2024-09-19T08:01:10.784346+0000","last_unstale":"2024-09-19T08:01:26.722418+0000","last_undegraded":"2024-09-19T08:01:26.722418+0000","last_fullsized":"2024-09-19T08:01:26.722418+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:27:54.580391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721469+0000","last_change":"2024-09-19T08:01:09.749498+0000","last_active":"2024-09-19T08:01:26.721469+0000","last_peered":"2024-09-19T08:01:26.721469+0000","last_clean":"2024-09-19T08:01:26.721469+0000","last_became_active":"2024-09-19T08:01:09.749091+0000","last_became_peered":"2024-09-19T08:01:09.749091+0000","last_unstale":"2024-09-19T08:01:26.721469+0000","last_undegraded":"2024-09-19T08:01:26.721469+0000","last_fullsized":"2024-09-19T08:01:26.721469+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:56:13.644931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721781+0000","last_change":"2024-09-19T08:01:10.779599+0000","last_active":"2024-09-19T08:01:26.721781+0000","last_peered":"2024-09-19T08:01:26.721781+0000","last_clean":"2024-09-19T08:01:26.721781+0000","last_became_active":"2024-09-19T08:01:10.779219+0000","last_became_peered":"2024-09-19T08:01:10.779219+0000","last_unstale":"2024-09-19T08:01:26.721781+0000","last_undegraded":"2024-09-19T08:01:26.721781+0000","last_fullsized":"2024-09-19T08:01:26.721781+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:20.193753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723508+0000","last_change":"2024-09-19T08:01:08.761432+0000","last_active":"2024-09-19T08:01:26.723508+0000","last_peered":"2024-09-19T08:01:26.723508+0000","last_clean":"2024-09-19T08:01:26.723508+0000","last_became_active":"2024-09-19T08:01:08.761044+0000","last_became_peered":"2024-09-19T08:01:08.761044+0000","last_unstale":"2024-09-19T08:01:26.723508+0000","last_undegraded":"2024-09-19T08:01:26.723508+0000","last_fullsized":"2024-09-19T08:01:26.723508+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:12.237334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718915+0000","last_change":"2024-09-19T08:01:10.772559+0000","last_active":"2024-09-19T08:01:26.718915+0000","last_peered":"2024-09-19T08:01:26.718915+0000","last_clean":"2024-09-19T08:01:26.718915+0000","last_became_active":"2024-09-19T08:01:10.772176+0000","last_became_peered":"2024-09-19T08:01:10.772176+0000","last_unstale":"2024-09-19T08:01:26.718915+0000","last_undegraded":"2024-09-19T08:01:26.718915+0000","last_fullsized":"2024-09-19T08:01:26.718915+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:27:16.296127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721816+0000","last_change":"2024-09-19T08:01:08.759989+0000","last_active":"2024-09-19T08:01:26.721816+0000","last_peered":"2024-09-19T08:01:26.721816+0000","last_clean":"2024-09-19T08:01:26.721816+0000","last_became_active":"2024-09-19T08:01:08.759705+0000","last_became_peered":"2024-09-19T08:01:08.759705+0000","last_unstale":"2024-09-19T08:01:26.721816+0000","last_undegraded":"2024-09-19T08:01:26.721816+0000","last_fullsized":"2024-09-19T08:01:26.721816+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:33:25.237567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723490+0000","last_change":"2024-09-19T08:01:10.776528+0000","last_active":"2024-09-19T08:01:26.723490+0000","last_peered":"2024-09-19T08:01:26.723490+0000","last_clean":"2024-09-19T08:01:26.723490+0000","last_became_active":"2024-09-19T08:01:10.776222+0000","last_became_peered":"2024-09-19T08:01:10.776222+0000","last_unstale":"2024-09-19T08:01:26.723490+0000","last_undegraded":"2024-09-19T08:01:26.723490+0000","last_fullsized":"2024-09-19T08:01:26.723490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.787492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722186+0000","last_change":"2024-09-19T08:01:08.759631+0000","last_active":"2024-09-19T08:01:26.722186+0000","last_peered":"2024-09-19T08:01:26.722186+0000","last_clean":"2024-09-19T08:01:26.722186+0000","last_became_active":"2024-09-19T08:01:08.759156+0000","last_became_peered":"2024-09-19T08:01:08.759156+0000","last_unstale":"2024-09-19T08:01:26.722186+0000","last_undegraded":"2024-09-19T08:01:26.722186+0000","last_fullsized":"2024-09-19T08:01:26.722186+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:48:50.910824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722951+0000","last_change":"2024-09-19T08:01:10.772660+0000","last_active":"2024-09-19T08:01:26.722951+0000","last_peered":"2024-09-19T08:01:26.722951+0000","last_clean":"2024-09-19T08:01:26.722951+0000","last_became_active":"2024-09-19T08:01:10.771816+0000","last_became_peered":"2024-09-19T08:01:10.771816+0000","last_unstale":"2024-09-19T08:01:26.722951+0000","last_undegraded":"2024-09-19T08:01:26.722951+0000","last_fullsized":"2024-09-19T08:01:26.722951+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:04:32.667107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'6","reported_seq":81,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:10.778520+0000","last_change":"2024-09-19T08:01:09.749844+0000","last_active":"2024-09-19T08:02:10.778520+0000","last_peered":"2024-09-19T08:02:10.778520+0000","last_clean":"2024-09-19T08:02:10.778520+0000","last_became_active":"2024-09-19T08:01:09.749410+0000","last_became_peered":"2024-09-19T08:01:09.749410+0000","last_unstale":"2024-09-19T08:02:10.778520+0000","last_undegraded":"2024-09-19T08:02:10.778520+0000","last_fullsized":"2024-09-19T08:02:10.778520+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:03:14.989814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1563,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723119+0000","last_change":"2024-09-19T08:01:10.773782+0000","last_active":"2024-09-19T08:01:26.723119+0000","last_peered":"2024-09-19T08:01:26.723119+0000","last_clean":"2024-09-19T08:01:26.723119+0000","last_became_active":"2024-09-19T08:01:10.773471+0000","last_became_peered":"2024-09-19T08:01:10.773471+0000","last_unstale":"2024-09-19T08:01:26.723119+0000","last_undegraded":"2024-09-19T08:01:26.723119+0000","last_fullsized":"2024-09-19T08:01:26.723119+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:15.780259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":341,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722698+0000","last_change":"2024-09-19T08:00:01.677517+0000","last_active":"2024-09-19T08:01:26.722698+0000","last_peered":"2024-09-19T08:01:26.722698+0000","last_clean":"2024-09-19T08:01:26.722698+0000","last_became_active":"2024-09-19T08:00:01.677089+0000","last_became_peered":"2024-09-19T08:00:01.677089+0000","last_unstale":"2024-09-19T08:01:26.722698+0000","last_undegraded":"2024-09-19T08:01:26.722698+0000","last_fullsized":"2024-09-19T08:01:26.722698+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_clean_scrub_stamp":"2024-09-19T07:56:43.962142+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:10:36.158811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758444+0000","last_change":"2024-09-19T08:01:08.759819+0000","last_active":"2024-09-19T08:01:26.758444+0000","last_peered":"2024-09-19T08:01:26.758444+0000","last_clean":"2024-09-19T08:01:26.758444+0000","last_became_active":"2024-09-19T08:01:08.759420+0000","last_became_peered":"2024-09-19T08:01:08.759420+0000","last_unstale":"2024-09-19T08:01:26.758444+0000","last_undegraded":"2024-09-19T08:01:26.758444+0000","last_fullsized":"2024-09-19T08:01:26.758444+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:06:16.852323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.724656+0000","last_change":"2024-09-19T08:01:26.724260+0000","last_active":"2024-09-19T08:01:26.724656+0000","last_peered":"2024-09-19T08:01:26.724656+0000","last_clean":"2024-09-19T08:01:26.724656+0000","last_became_active":"2024-09-19T08:01:26.723851+0000","last_became_peered":"2024-09-19T08:01:26.723851+0000","last_unstale":"2024-09-19T08:01:26.724656+0000","last_undegraded":"2024-09-19T08:01:26.724656+0000","last_fullsized":"2024-09-19T08:01:26.724656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:18:46.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722601+0000","last_change":"2024-09-19T08:01:08.761291+0000","last_active":"2024-09-19T08:01:26.722601+0000","last_peered":"2024-09-19T08:01:26.722601+0000","last_clean":"2024-09-19T08:01:26.722601+0000","last_became_active":"2024-09-19T08:01:08.760238+0000","last_became_peered":"2024-09-19T08:01:08.760238+0000","last_unstale":"2024-09-19T08:01:26.722601+0000","last_undegraded":"2024-09-19T08:01:26.722601+0000","last_fullsized":"2024-09-19T08:01:26.722601+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:26:09.980573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721415+0000","last_change":"2024-09-19T08:01:10.773056+0000","last_active":"2024-09-19T08:01:26.721415+0000","last_peered":"2024-09-19T08:01:26.721415+0000","last_clean":"2024-09-19T08:01:26.721415+0000","last_became_active":"2024-09-19T08:01:10.772431+0000","last_became_peered":"2024-09-19T08:01:10.772431+0000","last_unstale":"2024-09-19T08:01:26.721415+0000","last_undegraded":"2024-09-19T08:01:26.721415+0000","last_fullsized":"2024-09-19T08:01:26.721415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:39:06.878086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"78'5","reported_seq":78,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:08.757483+0000","last_change":"2024-09-19T08:01:09.749707+0000","last_active":"2024-09-19T08:02:08.757483+0000","last_peered":"2024-09-19T08:02:08.757483+0000","last_clean":"2024-09-19T08:02:08.757483+0000","last_became_active":"2024-09-19T08:01:09.749013+0000","last_became_peered":"2024-09-19T08:01:09.749013+0000","last_unstale":"2024-09-19T08:02:08.757483+0000","last_undegraded":"2024-09-19T08:02:08.757483+0000","last_fullsized":"2024-09-19T08:02:08.757483+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:40:12.682158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":51,"num_read_kb":51,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721922+0000","last_change":"2024-09-19T08:01:10.774101+0000","last_active":"2024-09-19T08:01:26.721922+0000","last_peered":"2024-09-19T08:01:26.721922+0000","last_clean":"2024-09-19T08:01:26.721922+0000","last_became_active":"2024-09-19T08:01:10.773822+0000","last_became_peered":"2024-09-19T08:01:10.773822+0000","last_unstale":"2024-09-19T08:01:26.721922+0000","last_undegraded":"2024-09-19T08:01:26.721922+0000","last_fullsized":"2024-09-19T08:01:26.721922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:29.571118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"78'5","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:07.773538+0000","last_change":"2024-09-19T08:01:08.761675+0000","last_active":"2024-09-19T08:02:07.773538+0000","last_peered":"2024-09-19T08:02:07.773538+0000","last_clean":"2024-09-19T08:02:07.773538+0000","last_became_active":"2024-09-19T08:01:08.761346+0000","last_became_peered":"2024-09-19T08:01:08.761346+0000","last_unstale":"2024-09-19T08:02:07.773538+0000","last_undegraded":"2024-09-19T08:02:07.773538+0000","last_fullsized":"2024-09-19T08:02:07.773538+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:51:19.237458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718382+0000","last_change":"2024-09-19T08:01:10.781565+0000","last_active":"2024-09-19T08:01:26.718382+0000","last_peered":"2024-09-19T08:01:26.718382+0000","last_clean":"2024-09-19T08:01:26.718382+0000","last_became_active":"2024-09-19T08:01:10.781301+0000","last_became_peered":"2024-09-19T08:01:10.781301+0000","last_unstale":"2024-09-19T08:01:26.718382+0000","last_undegraded":"2024-09-19T08:01:26.718382+0000","last_fullsized":"2024-09-19T08:01:26.718382+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:55:52.932009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.719945+0000","last_change":"2024-09-19T08:01:09.748918+0000","last_active":"2024-09-19T08:01:26.719945+0000","last_peered":"2024-09-19T08:01:26.719945+0000","last_clean":"2024-09-19T08:01:26.719945+0000","last_became_active":"2024-09-19T08:01:09.748497+0000","last_became_peered":"2024-09-19T08:01:09.748497+0000","last_unstale":"2024-09-19T08:01:26.719945+0000","last_undegraded":"2024-09-19T08:01:26.719945+0000","last_fullsized":"2024-09-19T08:01:26.719945+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:20:31.030133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718202+0000","last_change":"2024-09-19T08:01:10.781388+0000","last_active":"2024-09-19T08:01:26.718202+0000","last_peered":"2024-09-19T08:01:26.718202+0000","last_clean":"2024-09-19T08:01:26.718202+0000","last_became_active":"2024-09-19T08:01:10.781063+0000","last_became_peered":"2024-09-19T08:01:10.781063+0000","last_unstale":"2024-09-19T08:01:26.718202+0000","last_undegraded":"2024-09-19T08:01:26.718202+0000","last_fullsized":"2024-09-19T08:01:26.718202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:12:08.852717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.723781+0000","last_change":"2024-09-19T08:01:09.750080+0000","last_active":"2024-09-19T08:01:26.723781+0000","last_peered":"2024-09-19T08:01:26.723781+0000","last_clean":"2024-09-19T08:01:26.723781+0000","last_became_active":"2024-09-19T08:01:09.749053+0000","last_became_peered":"2024-09-19T08:01:09.749053+0000","last_unstale":"2024-09-19T08:01:26.723781+0000","last_undegraded":"2024-09-19T08:01:26.723781+0000","last_fullsized":"2024-09-19T08:01:26.723781+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:15:45.292714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721665+0000","last_change":"2024-09-19T08:01:10.767070+0000","last_active":"2024-09-19T08:01:26.721665+0000","last_peered":"2024-09-19T08:01:26.721665+0000","last_clean":"2024-09-19T08:01:26.721665+0000","last_became_active":"2024-09-19T08:01:10.766760+0000","last_became_peered":"2024-09-19T08:01:10.766760+0000","last_unstale":"2024-09-19T08:01:26.721665+0000","last_undegraded":"2024-09-19T08:01:26.721665+0000","last_fullsized":"2024-09-19T08:01:26.721665+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:17.841135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'15","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:34.790661+0000","last_change":"2024-09-19T08:01:08.756143+0000","last_active":"2024-09-19T08:01:34.790661+0000","last_peered":"2024-09-19T08:01:34.790661+0000","last_clean":"2024-09-19T08:01:34.790661+0000","last_became_active":"2024-09-19T08:01:08.755777+0000","last_became_peered":"2024-09-19T08:01:08.755777+0000","last_unstale":"2024-09-19T08:01:34.790661+0000","last_undegraded":"2024-09-19T08:01:34.790661+0000","last_fullsized":"2024-09-19T08:01:34.790661+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:21.430673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718462+0000","last_change":"2024-09-19T08:01:10.774140+0000","last_active":"2024-09-19T08:01:26.718462+0000","last_peered":"2024-09-19T08:01:26.718462+0000","last_clean":"2024-09-19T08:01:26.718462+0000","last_became_active":"2024-09-19T08:01:10.773819+0000","last_became_peered":"2024-09-19T08:01:10.773819+0000","last_unstale":"2024-09-19T08:01:26.718462+0000","last_undegraded":"2024-09-19T08:01:26.718462+0000","last_fullsized":"2024-09-19T08:01:26.718462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:15:13.174013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.758589+0000","last_change":"2024-09-19T08:01:09.169129+0000","last_active":"2024-09-19T08:01:26.758589+0000","last_peered":"2024-09-19T08:01:26.758589+0000","last_clean":"2024-09-19T08:01:26.758589+0000","last_became_active":"2024-09-19T08:01:09.168734+0000","last_became_peered":"2024-09-19T08:01:09.168734+0000","last_unstale":"2024-09-19T08:01:26.758589+0000","last_undegraded":"2024-09-19T08:01:26.758589+0000","last_fullsized":"2024-09-19T08:01:26.758589+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:01:52.232391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757822+0000","last_change":"2024-09-19T08:01:10.768592+0000","last_active":"2024-09-19T08:01:26.757822+0000","last_peered":"2024-09-19T08:01:26.757822+0000","last_clean":"2024-09-19T08:01:26.757822+0000","last_became_active":"2024-09-19T08:01:10.767645+0000","last_became_peered":"2024-09-19T08:01:10.767645+0000","last_unstale":"2024-09-19T08:01:26.757822+0000","last_undegraded":"2024-09-19T08:01:26.757822+0000","last_fullsized":"2024-09-19T08:01:26.757822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:08:46.227461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721037+0000","last_change":"2024-09-19T08:01:09.749240+0000","last_active":"2024-09-19T08:01:26.721037+0000","last_peered":"2024-09-19T08:01:26.721037+0000","last_clean":"2024-09-19T08:01:26.721037+0000","last_became_active":"2024-09-19T08:01:09.748945+0000","last_became_peered":"2024-09-19T08:01:09.748945+0000","last_unstale":"2024-09-19T08:01:26.721037+0000","last_undegraded":"2024-09-19T08:01:26.721037+0000","last_fullsized":"2024-09-19T08:01:26.721037+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:25:13.791290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.720802+0000","last_change":"2024-09-19T08:01:10.783133+0000","last_active":"2024-09-19T08:01:26.720802+0000","last_peered":"2024-09-19T08:01:26.720802+0000","last_clean":"2024-09-19T08:01:26.720802+0000","last_became_active":"2024-09-19T08:01:10.782797+0000","last_became_peered":"2024-09-19T08:01:10.782797+0000","last_unstale":"2024-09-19T08:01:26.720802+0000","last_undegraded":"2024-09-19T08:01:26.720802+0000","last_fullsized":"2024-09-19T08:01:26.720802+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:30:43.551877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718123+0000","last_change":"2024-09-19T08:01:08.757964+0000","last_active":"2024-09-19T08:01:26.718123+0000","last_peered":"2024-09-19T08:01:26.718123+0000","last_clean":"2024-09-19T08:01:26.718123+0000","last_became_active":"2024-09-19T08:01:08.757526+0000","last_became_peered":"2024-09-19T08:01:08.757526+0000","last_unstale":"2024-09-19T08:01:26.718123+0000","last_undegraded":"2024-09-19T08:01:26.718123+0000","last_fullsized":"2024-09-19T08:01:26.718123+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:03.565702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718652+0000","last_change":"2024-09-19T08:01:10.769974+0000","last_active":"2024-09-19T08:01:26.718652+0000","last_peered":"2024-09-19T08:01:26.718652+0000","last_clean":"2024-09-19T08:01:26.718652+0000","last_became_active":"2024-09-19T08:01:10.769650+0000","last_became_peered":"2024-09-19T08:01:10.769650+0000","last_unstale":"2024-09-19T08:01:26.718652+0000","last_undegraded":"2024-09-19T08:01:26.718652+0000","last_fullsized":"2024-09-19T08:01:26.718652+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:01.755198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.725382+0000","last_change":"2024-09-19T08:01:08.763476+0000","last_active":"2024-09-19T08:01:26.725382+0000","last_peered":"2024-09-19T08:01:26.725382+0000","last_clean":"2024-09-19T08:01:26.725382+0000","last_became_active":"2024-09-19T08:01:08.763154+0000","last_became_peered":"2024-09-19T08:01:08.763154+0000","last_unstale":"2024-09-19T08:01:26.725382+0000","last_undegraded":"2024-09-19T08:01:26.725382+0000","last_fullsized":"2024-09-19T08:01:26.725382+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:06.202452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718164+0000","last_change":"2024-09-19T08:01:10.784201+0000","last_active":"2024-09-19T08:01:26.718164+0000","last_peered":"2024-09-19T08:01:26.718164+0000","last_clean":"2024-09-19T08:01:26.718164+0000","last_became_active":"2024-09-19T08:01:10.783867+0000","last_became_peered":"2024-09-19T08:01:10.783867+0000","last_unstale":"2024-09-19T08:01:26.718164+0000","last_undegraded":"2024-09-19T08:01:26.718164+0000","last_fullsized":"2024-09-19T08:01:26.718164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:43:50.082183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721581+0000","last_change":"2024-09-19T08:01:08.760479+0000","last_active":"2024-09-19T08:01:26.721581+0000","last_peered":"2024-09-19T08:01:26.721581+0000","last_clean":"2024-09-19T08:01:26.721581+0000","last_became_active":"2024-09-19T08:01:08.760004+0000","last_became_peered":"2024-09-19T08:01:08.760004+0000","last_unstale":"2024-09-19T08:01:26.721581+0000","last_undegraded":"2024-09-19T08:01:26.721581+0000","last_fullsized":"2024-09-19T08:01:26.721581+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:41:19.956623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.722763+0000","last_change":"2024-09-19T08:01:10.772184+0000","last_active":"2024-09-19T08:01:26.722763+0000","last_peered":"2024-09-19T08:01:26.722763+0000","last_clean":"2024-09-19T08:01:26.722763+0000","last_became_active":"2024-09-19T08:01:10.771825+0000","last_became_peered":"2024-09-19T08:01:10.771825+0000","last_unstale":"2024-09-19T08:01:26.722763+0000","last_undegraded":"2024-09-19T08:01:26.722763+0000","last_fullsized":"2024-09-19T08:01:26.722763+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:36.764575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721944+0000","last_change":"2024-09-19T08:01:10.779368+0000","last_active":"2024-09-19T08:01:26.721944+0000","last_peered":"2024-09-19T08:01:26.721944+0000","last_clean":"2024-09-19T08:01:26.721944+0000","last_became_active":"2024-09-19T08:01:10.778884+0000","last_became_peered":"2024-09-19T08:01:10.778884+0000","last_unstale":"2024-09-19T08:01:26.721944+0000","last_undegraded":"2024-09-19T08:01:26.721944+0000","last_fullsized":"2024-09-19T08:01:26.721944+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:28:29.630300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.741496+0000","last_change":"2024-09-19T08:01:26.741053+0000","last_active":"2024-09-19T08:01:26.741496+0000","last_peered":"2024-09-19T08:01:26.741496+0000","last_clean":"2024-09-19T08:01:26.741496+0000","last_became_active":"2024-09-19T08:01:26.724714+0000","last_became_peered":"2024-09-19T08:01:26.724714+0000","last_unstale":"2024-09-19T08:01:26.741496+0000","last_undegraded":"2024-09-19T08:01:26.741496+0000","last_fullsized":"2024-09-19T08:01:26.741496+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:59.583797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.721113+0000","last_change":"2024-09-19T08:01:10.781389+0000","last_active":"2024-09-19T08:01:26.721113+0000","last_peered":"2024-09-19T08:01:26.721113+0000","last_clean":"2024-09-19T08:01:26.721113+0000","last_became_active":"2024-09-19T08:01:10.781080+0000","last_became_peered":"2024-09-19T08:01:10.781080+0000","last_unstale":"2024-09-19T08:01:26.721113+0000","last_undegraded":"2024-09-19T08:01:26.721113+0000","last_fullsized":"2024-09-19T08:01:26.721113+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:40:45.225957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.768102+0000","last_change":"2024-09-19T08:01:26.768102+0000","last_active":"2024-09-19T08:01:26.768102+0000","last_peered":"2024-09-19T08:01:26.768102+0000","last_clean":"2024-09-19T08:01:26.768102+0000","last_became_active":"2024-09-19T08:01:26.759462+0000","last_became_peered":"2024-09-19T08:01:26.759462+0000","last_unstale":"2024-09-19T08:01:26.768102+0000","last_undegraded":"2024-09-19T08:01:26.768102+0000","last_fullsized":"2024-09-19T08:01:26.768102+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:02:06.815238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.757966+0000","last_change":"2024-09-19T08:01:10.771031+0000","last_active":"2024-09-19T08:01:26.757966+0000","last_peered":"2024-09-19T08:01:26.757966+0000","last_clean":"2024-09-19T08:01:26.757966+0000","last_became_active":"2024-09-19T08:01:10.770675+0000","last_became_peered":"2024-09-19T08:01:10.770675+0000","last_unstale":"2024-09-19T08:01:26.757966+0000","last_undegraded":"2024-09-19T08:01:26.757966+0000","last_fullsized":"2024-09-19T08:01:26.757966+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:42.203848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718482+0000","last_change":"2024-09-19T08:01:09.748622+0000","last_active":"2024-09-19T08:01:26.718482+0000","last_peered":"2024-09-19T08:01:26.718482+0000","last_clean":"2024-09-19T08:01:26.718482+0000","last_became_active":"2024-09-19T08:01:09.748202+0000","last_became_peered":"2024-09-19T08:01:09.748202+0000","last_unstale":"2024-09-19T08:01:26.718482+0000","last_undegraded":"2024-09-19T08:01:26.718482+0000","last_fullsized":"2024-09-19T08:01:26.718482+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:17:39.861820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:01:26.718292+0000","last_change":"2024-09-19T08:01:10.768949+0000","last_active":"2024-09-19T08:01:26.718292+0000","last_peered":"2024-09-19T08:01:26.718292+0000","last_clean":"2024-09-19T08:01:26.718292+0000","last_became_active":"2024-09-19T08:01:10.768159+0000","last_became_peered":"2024-09-19T08:01:10.768159+0000","last_unstale":"2024-09-19T08:01:26.718292+0000","last_undegraded":"2024-09-19T08:01:26.718292+0000","last_fullsized":"2024-09-19T08:01:26.718292+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:43.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"78'13","reported_seq":108,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-19T08:02:06.772239+0000","last_change":"2024-09-19T08:01:26.735877+0000","last_active":"2024-09-19T08:02:06.772239+0000","last_peered":"2024-09-19T08:02:06.772239+0000","last_clean":"2024-09-19T08:02:06.772239+0000","last_became_active":"2024-09-19T08:01:26.723597+0000","last_became_peered":"2024-09-19T08:01:26.723597+0000","last_unstale":"2024-09-19T08:02:06.772239+0000","last_undegraded":"2024-09-19T08:02:06.772239+0000","last_fullsized":"2024-09-19T08:02:06.772239+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:43:42.340832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,1],"acting":[9,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66972,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":524,"num_read_kb":330,"num_write":243,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1254,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":200916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645338,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":888832,"data_stored":427316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874270,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32724,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712428,"statfs":{"total":95995035648,"available":95961526272,"internally_reserved":0,"allocated":1576960,"data_stored":1076462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005091,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":901120,"data_stored":425599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234024,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":884736,"data_stored":424723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462956,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1486848,"data_stored":1017825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593778,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":884736,"data_stored":425316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822711,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28500,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716652,"statfs":{"total":95995035648,"available":95965851648,"internally_reserved":0,"allocated":1507328,"data_stored":1018175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986236,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":884736,"data_stored":425177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215168,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":901120,"data_stored":425141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476806,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":486999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607626,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":486665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":905216,"data_stored":427048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":62188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T08:02:15.866 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-19T08:02:15.866 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-19T08:02:15.866 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-19T08:02:15.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-19T08:02:16.109 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.551+0000 7f4029e8d640 1 -- 172.21.15.98:0/456313843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4024101b60 msgr2=0x7f402410d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.551+0000 7f4029e8d640 1 --2- 172.21.15.98:0/456313843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4024101b60 0x7f402410d970 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f401c009ca0 tx=0x7f401c02a8c0 comp rx=0 tx=0).stop 2024-09-19T08:02:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.551+0000 7f4029e8d640 1 -- 172.21.15.98:0/456313843 shutdown_connections 2024-09-19T08:02:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.551+0000 7f4029e8d640 1 --2- 172.21.15.98:0/456313843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4024101b60 0x7f402410d970 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.551+0000 7f4029e8d640 1 --2- 172.21.15.98:0/456313843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f40241011c0 0x7f4024101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.551+0000 7f4029e8d640 1 --2- 172.21.15.98:0/456313843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4024105ad0 0x7f4024105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.551+0000 7f4029e8d640 1 -- 172.21.15.98:0/456313843 >> 172.21.15.98:0/456313843 conn(0x7f4024077210 msgr2=0x7f40240fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.551+0000 7f4029e8d640 1 -- 172.21.15.98:0/456313843 shutdown_connections 2024-09-19T08:02:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.551+0000 7f4029e8d640 1 -- 172.21.15.98:0/456313843 wait complete. 2024-09-19T08:02:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4029e8d640 1 Processor -- start 2024-09-19T08:02:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4029e8d640 1 -- start start 2024-09-19T08:02:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4029e8d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f40241011c0 0x7f402419a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4029e8d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4024101b60 0x7f402419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4029e8d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4024105ad0 0x7f402419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4029e8d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4024111fe0 con 0x7f4024105ad0 2024-09-19T08:02:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4029e8d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4024111e60 con 0x7f40241011c0 2024-09-19T08:02:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4029e8d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4024112160 con 0x7f4024101b60 2024-09-19T08:02:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4023fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4024105ad0 0x7f402419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4023fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4024105ad0 0x7f402419ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42392/0 (socket says 172.21.15.98:42392) 2024-09-19T08:02:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4023fff640 1 -- 172.21.15.98:0/108255222 learned_addr learned my addr 172.21.15.98:0/108255222 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f401bfff640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4024101b60 0x7f402419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f40237fe640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f40241011c0 0x7f402419a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4023fff640 1 -- 172.21.15.98:0/108255222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4024101b60 msgr2=0x7f402419aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4023fff640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4024101b60 0x7f402419aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4023fff640 1 -- 172.21.15.98:0/108255222 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f40241011c0 msgr2=0x7f402419a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4023fff640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f40241011c0 0x7f402419a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f4023fff640 1 -- 172.21.15.98:0/108255222 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f402419f540 con 0x7f4024105ad0 2024-09-19T08:02:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f401bfff640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4024101b60 0x7f402419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.552+0000 7f40237fe640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f40241011c0 0x7f402419a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.553+0000 7f4023fff640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4024105ad0 0x7f402419ee60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f401c003f40 tx=0x7f401c003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.553+0000 7f40217fa640 1 -- 172.21.15.98:0/108255222 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c031790 con 0x7f4024105ad0 2024-09-19T08:02:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.553+0000 7f4029e8d640 1 -- 172.21.15.98:0/108255222 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f402419f7d0 con 0x7f4024105ad0 2024-09-19T08:02:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.553+0000 7f40217fa640 1 -- 172.21.15.98:0/108255222 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f401c031930 con 0x7f4024105ad0 2024-09-19T08:02:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.553+0000 7f40217fa640 1 -- 172.21.15.98:0/108255222 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c030bf0 con 0x7f4024105ad0 2024-09-19T08:02:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.553+0000 7f4029e8d640 1 -- 172.21.15.98:0/108255222 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40241a7110 con 0x7f4024105ad0 2024-09-19T08:02:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.554+0000 7f4029e8d640 1 -- 172.21.15.98:0/108255222 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4024101620 con 0x7f4024105ad0 2024-09-19T08:02:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.555+0000 7f40217fa640 1 -- 172.21.15.98:0/108255222 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f401c004b40 con 0x7f4024105ad0 2024-09-19T08:02:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.555+0000 7f40217fa640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f40100a32d0 0x7f40100a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.556+0000 7f40237fe640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f40100a32d0 0x7f40100a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.556+0000 7f40217fa640 1 -- 172.21.15.98:0/108255222 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f401c0f10d0 con 0x7f4024105ad0 2024-09-19T08:02:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.556+0000 7f40237fe640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f40100a32d0 0x7f40100a5790 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4014005e10 tx=0x7f4014005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.559+0000 7f40217fa640 1 -- 172.21.15.98:0/108255222 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f401c0bde40 con 0x7f4024105ad0 2024-09-19T08:02:16.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.765+0000 7f4029e8d640 1 -- 172.21.15.98:0/108255222 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f40241a7310 con 0x7f4024105ad0 2024-09-19T08:02:16.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.765+0000 7f40217fa640 1 -- 172.21.15.98:0/108255222 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f401c0bd740 con 0x7f4024105ad0 2024-09-19T08:02:16.765 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:02:16.765 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-19T08:02:16.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.767+0000 7f4029e8d640 1 -- 172.21.15.98:0/108255222 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f40100a32d0 msgr2=0x7f40100a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:16.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.767+0000 7f4029e8d640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f40100a32d0 0x7f40100a5790 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4014005e10 tx=0x7f4014005da0 comp rx=0 tx=0).stop 2024-09-19T08:02:16.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.767+0000 7f4029e8d640 1 -- 172.21.15.98:0/108255222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4024105ad0 msgr2=0x7f402419ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:16.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.767+0000 7f4029e8d640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4024105ad0 0x7f402419ee60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f401c003f40 tx=0x7f401c003f70 comp rx=0 tx=0).stop 2024-09-19T08:02:16.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.767+0000 7f4029e8d640 1 -- 172.21.15.98:0/108255222 shutdown_connections 2024-09-19T08:02:16.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.767+0000 7f4029e8d640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f40100a32d0 0x7f40100a5790 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:16.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.767+0000 7f4029e8d640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4024105ad0 0x7f402419ee60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:16.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.767+0000 7f4029e8d640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4024101b60 0x7f402419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:16.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.767+0000 7f4029e8d640 1 --2- 172.21.15.98:0/108255222 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f40241011c0 0x7f402419a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:16.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.767+0000 7f4029e8d640 1 -- 172.21.15.98:0/108255222 >> 172.21.15.98:0/108255222 conn(0x7f4024077210 msgr2=0x7f40241039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:16.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.767+0000 7f4029e8d640 1 -- 172.21.15.98:0/108255222 shutdown_connections 2024-09-19T08:02:16.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:16.767+0000 7f4029e8d640 1 -- 172.21.15.98:0/108255222 wait complete. 2024-09-19T08:02:16.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:16 smithi162 ceph-mon[35260]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:02:16.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:16 smithi162 ceph-mon[35260]: pgmap v305: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:16 smithi139 ceph-mon[34874]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:02:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:16 smithi139 ceph-mon[34874]: pgmap v305: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:16.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:16 smithi098 ceph-mon[31333]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:02:16.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:16 smithi098 ceph-mon[31333]: pgmap v305: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:17.499 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-19T08:02:17.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:02:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:17.869 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:17 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/108255222' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-19T08:02:17.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:17 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/108255222' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-19T08:02:17.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:17 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/108255222' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-19T08:02:18.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.201+0000 7fa110585640 1 -- 172.21.15.98:0/1396501554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa108102ef0 msgr2=0x7fa10810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:18.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.201+0000 7fa110585640 1 --2- 172.21.15.98:0/1396501554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa108102ef0 0x7fa10810da40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa100009ca0 tx=0x7fa10002a950 comp rx=0 tx=0).stop 2024-09-19T08:02:18.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 -- 172.21.15.98:0/1396501554 shutdown_connections 2024-09-19T08:02:18.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 --2- 172.21.15.98:0/1396501554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa108102ef0 0x7fa10810da40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:18.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 --2- 172.21.15.98:0/1396501554 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa108102570 0x7fa1081029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:18.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 --2- 172.21.15.98:0/1396501554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa108100e20 0x7fa108101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:18.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 -- 172.21.15.98:0/1396501554 >> 172.21.15.98:0/1396501554 conn(0x7fa1080fc7d0 msgr2=0x7fa1080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:18.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 -- 172.21.15.98:0/1396501554 shutdown_connections 2024-09-19T08:02:18.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 -- 172.21.15.98:0/1396501554 wait complete. 2024-09-19T08:02:18.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 Processor -- start 2024-09-19T08:02:18.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 -- start start 2024-09-19T08:02:18.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa108100e20 0x7fa108198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:18.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa108102570 0x7fa108198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:18.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa108102ef0 0x7fa10819ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:18.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa108111020 con 0x7fa108102570 2024-09-19T08:02:18.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa108110ea0 con 0x7fa108102ef0 2024-09-19T08:02:18.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.202+0000 7fa110585640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa1081111a0 con 0x7fa108100e20 2024-09-19T08:02:18.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa10db20640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa108102570 0x7fa108198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:18.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa10db20640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa108102570 0x7fa108198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42420/0 (socket says 172.21.15.98:42420) 2024-09-19T08:02:18.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa10db20640 1 -- 172.21.15.98:0/2887025590 learned_addr learned my addr 172.21.15.98:0/2887025590 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:18.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa10eb22640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa108102ef0 0x7fa10819ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:18.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa10e321640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa108100e20 0x7fa108198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa10db20640 1 -- 172.21.15.98:0/2887025590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa108100e20 msgr2=0x7fa108198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa10db20640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa108100e20 0x7fa108198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa10db20640 1 -- 172.21.15.98:0/2887025590 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa108102ef0 msgr2=0x7fa10819ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa10db20640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa108102ef0 0x7fa10819ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa10db20640 1 -- 172.21.15.98:0/2887025590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10819d5b0 con 0x7fa108102570 2024-09-19T08:02:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa10e321640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa108100e20 0x7fa108198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa10db20640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa108102570 0x7fa108198b40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800b4a0 tx=0x7fa0f800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa0ff7fe640 1 -- 172.21.15.98:0/2887025590 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8014070 con 0x7fa108102570 2024-09-19T08:02:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa0ff7fe640 1 -- 172.21.15.98:0/2887025590 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0f8010af0 con 0x7fa108102570 2024-09-19T08:02:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa110585640 1 -- 172.21.15.98:0/2887025590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa10819d8a0 con 0x7fa108102570 2024-09-19T08:02:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa0ff7fe640 1 -- 172.21.15.98:0/2887025590 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8017d10 con 0x7fa108102570 2024-09-19T08:02:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.203+0000 7fa110585640 1 -- 172.21.15.98:0/2887025590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1081a5180 con 0x7fa108102570 2024-09-19T08:02:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.204+0000 7fa110585640 1 -- 172.21.15.98:0/2887025590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa108101280 con 0x7fa108102570 2024-09-19T08:02:18.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.205+0000 7fa0ff7fe640 1 -- 172.21.15.98:0/2887025590 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa0f8017070 con 0x7fa108102570 2024-09-19T08:02:18.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.206+0000 7fa0ff7fe640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa0f40a3350 0x7fa0f40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:18.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.206+0000 7fa10e321640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa0f40a3350 0x7fa0f40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:18.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.206+0000 7fa0ff7fe640 1 -- 172.21.15.98:0/2887025590 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fa0f80cec70 con 0x7fa108102570 2024-09-19T08:02:18.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.207+0000 7fa10e321640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa0f40a3350 0x7fa0f40a5810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa104006fd0 tx=0x7fa104008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:18.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.211+0000 7fa0ff7fe640 1 -- 172.21.15.98:0/2887025590 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fa0f809b950 con 0x7fa108102570 2024-09-19T08:02:18.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.410+0000 7fa110585640 1 -- 172.21.15.98:0/2887025590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1081a5410 con 0x7fa108102570 2024-09-19T08:02:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.411+0000 7fa0ff7fe640 1 -- 172.21.15.98:0/2887025590 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11983 (secure 0 0 0) 0x7fa0f809b250 con 0x7fa108102570 2024-09-19T08:02:18.411 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:02:18.411 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":27,"btime":"2024-09-19T08:01:22:699589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:01:22.699569+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24461,24437,24406,24376]},"id":1}]} 2024-09-19T08:02:18.412 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 27 2024-09-19T08:02:18.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.413+0000 7fa110585640 1 -- 172.21.15.98:0/2887025590 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa0f40a3350 msgr2=0x7fa0f40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:18.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.413+0000 7fa110585640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa0f40a3350 0x7fa0f40a5810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa104006fd0 tx=0x7fa104008040 comp rx=0 tx=0).stop 2024-09-19T08:02:18.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.413+0000 7fa110585640 1 -- 172.21.15.98:0/2887025590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa108102570 msgr2=0x7fa108198b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:18.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.413+0000 7fa110585640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa108102570 0x7fa108198b40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800b4a0 tx=0x7fa0f800b960 comp rx=0 tx=0).stop 2024-09-19T08:02:18.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.413+0000 7fa110585640 1 -- 172.21.15.98:0/2887025590 shutdown_connections 2024-09-19T08:02:18.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.413+0000 7fa110585640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa0f40a3350 0x7fa0f40a5810 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:18.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.413+0000 7fa110585640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa108102ef0 0x7fa10819ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:18.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.413+0000 7fa110585640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa108102570 0x7fa108198b40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:18.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.413+0000 7fa110585640 1 --2- 172.21.15.98:0/2887025590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa108100e20 0x7fa108198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:18.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.413+0000 7fa110585640 1 -- 172.21.15.98:0/2887025590 >> 172.21.15.98:0/2887025590 conn(0x7fa1080fc7d0 msgr2=0x7fa108069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:18.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.414+0000 7fa110585640 1 -- 172.21.15.98:0/2887025590 shutdown_connections 2024-09-19T08:02:18.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:18.414+0000 7fa110585640 1 -- 172.21.15.98:0/2887025590 wait complete. 2024-09-19T08:02:18.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:18 smithi098 ceph-mon[31333]: pgmap v306: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:18.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:18 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2887025590' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:18.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:18 smithi139 ceph-mon[34874]: pgmap v306: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:18.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:18 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2887025590' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:19.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:02:19.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:18 smithi162 ceph-mon[35260]: pgmap v306: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:19.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:18 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2887025590' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:19.337 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:19.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.772+0000 7fc1e2dda640 1 -- 172.21.15.98:0/2358585422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1dc100a90 msgr2=0x7fc1dc100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:19.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.772+0000 7fc1e2dda640 1 --2- 172.21.15.98:0/2358585422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1dc100a90 0x7fc1dc100ef0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009cd0 tx=0x7fc1cc02a8e0 comp rx=0 tx=0).stop 2024-09-19T08:02:19.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.772+0000 7fc1e2dda640 1 -- 172.21.15.98:0/2358585422 shutdown_connections 2024-09-19T08:02:19.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.772+0000 7fc1e2dda640 1 --2- 172.21.15.98:0/2358585422 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc1dc101430 0x7fc1dc10d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:19.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.772+0000 7fc1e2dda640 1 --2- 172.21.15.98:0/2358585422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1dc100a90 0x7fc1dc100ef0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:19.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.772+0000 7fc1e2dda640 1 --2- 172.21.15.98:0/2358585422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1dc106a90 0x7fc1dc106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:19.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.772+0000 7fc1e2dda640 1 -- 172.21.15.98:0/2358585422 >> 172.21.15.98:0/2358585422 conn(0x7fc1dc0fc7d0 msgr2=0x7fc1dc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:19.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.772+0000 7fc1e2dda640 1 -- 172.21.15.98:0/2358585422 shutdown_connections 2024-09-19T08:02:19.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.772+0000 7fc1e2dda640 1 -- 172.21.15.98:0/2358585422 wait complete. 2024-09-19T08:02:19.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.773+0000 7fc1e2dda640 1 Processor -- start 2024-09-19T08:02:19.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.773+0000 7fc1e2dda640 1 -- start start 2024-09-19T08:02:19.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.773+0000 7fc1e2dda640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc1dc100a90 0x7fc1dc199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:19.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.773+0000 7fc1e2dda640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1dc101430 0x7fc1dc199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:19.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.773+0000 7fc1e2dda640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1dc106a90 0x7fc1dc1a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:19.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.773+0000 7fc1e2dda640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc110eb0 con 0x7fc1dc101430 2024-09-19T08:02:19.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.773+0000 7fc1dbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1dc101430 0x7fc1dc199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:19.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.773+0000 7fc1dbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1dc101430 0x7fc1dc199860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42452/0 (socket says 172.21.15.98:42452) 2024-09-19T08:02:19.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.773+0000 7fc1e2dda640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc110d30 con 0x7fc1dc100a90 2024-09-19T08:02:19.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.773+0000 7fc1dbfff640 1 -- 172.21.15.98:0/913928574 learned_addr learned my addr 172.21.15.98:0/913928574 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:19.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.773+0000 7fc1e2dda640 1 -- 172.21.15.98:0/913928574 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc111030 con 0x7fc1dc106a90 2024-09-19T08:02:19.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.773+0000 7fc1e0b76640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc1dc100a90 0x7fc1dc199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:19.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.774+0000 7fc1e1377640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1dc106a90 0x7fc1dc1a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:19.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.774+0000 7fc1dbfff640 1 -- 172.21.15.98:0/913928574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1dc106a90 msgr2=0x7fc1dc1a43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:19.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.774+0000 7fc1dbfff640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1dc106a90 0x7fc1dc1a43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:19.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.774+0000 7fc1dbfff640 1 -- 172.21.15.98:0/913928574 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc1dc100a90 msgr2=0x7fc1dc199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:19.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.774+0000 7fc1dbfff640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc1dc100a90 0x7fc1dc199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:19.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.774+0000 7fc1dbfff640 1 -- 172.21.15.98:0/913928574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1dc1a48e0 con 0x7fc1dc101430 2024-09-19T08:02:19.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.774+0000 7fc1e0b76640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc1dc100a90 0x7fc1dc199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:02:19.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.774+0000 7fc1dbfff640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1dc101430 0x7fc1dc199860 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc02ae00 tx=0x7fc1cc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:19.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.774+0000 7fc1d9ffb640 1 -- 172.21.15.98:0/913928574 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc004240 con 0x7fc1dc101430 2024-09-19T08:02:19.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.774+0000 7fc1e2dda640 1 -- 172.21.15.98:0/913928574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1dc1a4ab0 con 0x7fc1dc101430 2024-09-19T08:02:19.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.774+0000 7fc1d9ffb640 1 -- 172.21.15.98:0/913928574 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1cc0305a0 con 0x7fc1dc101430 2024-09-19T08:02:19.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.774+0000 7fc1d9ffb640 1 -- 172.21.15.98:0/913928574 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc03a660 con 0x7fc1dc101430 2024-09-19T08:02:19.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.774+0000 7fc1e2dda640 1 -- 172.21.15.98:0/913928574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1dc1a4fe0 con 0x7fc1dc101430 2024-09-19T08:02:19.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.776+0000 7fc1e2dda640 1 -- 172.21.15.98:0/913928574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1dc100ef0 con 0x7fc1dc101430 2024-09-19T08:02:19.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.776+0000 7fc1d9ffb640 1 -- 172.21.15.98:0/913928574 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc1cc030ac0 con 0x7fc1dc101430 2024-09-19T08:02:19.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.777+0000 7fc1d9ffb640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc1c80a32d0 0x7fc1c80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:19.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.777+0000 7fc1e0b76640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc1c80a32d0 0x7fc1c80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:19.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.777+0000 7fc1d9ffb640 1 -- 172.21.15.98:0/913928574 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fc1cc02e030 con 0x7fc1dc101430 2024-09-19T08:02:19.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.778+0000 7fc1e0b76640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc1c80a32d0 0x7fc1c80a5790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4009d60 tx=0x7fc1d4009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:19.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.781+0000 7fc1d9ffb640 1 -- 172.21.15.98:0/913928574 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fc1cc0098b0 con 0x7fc1dc101430 2024-09-19T08:02:19.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.985+0000 7fc1e2dda640 1 -- 172.21.15.98:0/913928574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1dc109020 con 0x7fc1dc101430 2024-09-19T08:02:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.986+0000 7fc1d9ffb640 1 -- 172.21.15.98:0/913928574 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11983 (secure 0 0 0) 0x7fc1cc0bd820 con 0x7fc1dc101430 2024-09-19T08:02:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:02:19.987 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":27,"btime":"2024-09-19T08:01:22:699589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:01:22.699569+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24461,24437,24406,24376]},"id":1}]} 2024-09-19T08:02:19.987 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 27 2024-09-19T08:02:19.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.988+0000 7fc1e2dda640 1 -- 172.21.15.98:0/913928574 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc1c80a32d0 msgr2=0x7fc1c80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:19.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.988+0000 7fc1e2dda640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc1c80a32d0 0x7fc1c80a5790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4009d60 tx=0x7fc1d4009110 comp rx=0 tx=0).stop 2024-09-19T08:02:19.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.988+0000 7fc1e2dda640 1 -- 172.21.15.98:0/913928574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1dc101430 msgr2=0x7fc1dc199860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:19.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.988+0000 7fc1e2dda640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1dc101430 0x7fc1dc199860 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc02ae00 tx=0x7fc1cc003ba0 comp rx=0 tx=0).stop 2024-09-19T08:02:19.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.988+0000 7fc1e2dda640 1 -- 172.21.15.98:0/913928574 shutdown_connections 2024-09-19T08:02:19.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.988+0000 7fc1e2dda640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc1c80a32d0 0x7fc1c80a5790 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:19.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.988+0000 7fc1e2dda640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1dc106a90 0x7fc1dc1a43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:19.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.988+0000 7fc1e2dda640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1dc101430 0x7fc1dc199860 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:19.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.988+0000 7fc1e2dda640 1 --2- 172.21.15.98:0/913928574 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc1dc100a90 0x7fc1dc199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:19.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.988+0000 7fc1e2dda640 1 -- 172.21.15.98:0/913928574 >> 172.21.15.98:0/913928574 conn(0x7fc1dc0fc7d0 msgr2=0x7fc1dc0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:19.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.988+0000 7fc1e2dda640 1 -- 172.21.15.98:0/913928574 shutdown_connections 2024-09-19T08:02:19.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:19.988+0000 7fc1e2dda640 1 -- 172.21.15.98:0/913928574 wait complete. 2024-09-19T08:02:20.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T08:02:20.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:20 smithi139 ceph-mon[34874]: pgmap v307: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:20.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:20 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/913928574' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:20.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:20 smithi098 ceph-mon[31333]: pgmap v307: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:20.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:20 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/913928574' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:20.944 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:20 smithi162 ceph-mon[35260]: pgmap v307: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:20 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/913928574' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.459+0000 7f75627b8640 1 -- 172.21.15.98:0/4260181559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f755c106f10 msgr2=0x7f755c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.459+0000 7f75627b8640 1 --2- 172.21.15.98:0/4260181559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f755c106f10 0x7f755c10ea90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7554009ca0 tx=0x7f755402a8c0 comp rx=0 tx=0).stop 2024-09-19T08:02:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.459+0000 7f75627b8640 1 -- 172.21.15.98:0/4260181559 shutdown_connections 2024-09-19T08:02:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.459+0000 7f75627b8640 1 --2- 172.21.15.98:0/4260181559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f755c106f10 0x7f755c10ea90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.459+0000 7f75627b8640 1 --2- 172.21.15.98:0/4260181559 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f755c102790 0x7f755c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.459+0000 7f75627b8640 1 --2- 172.21.15.98:0/4260181559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f755c101de0 0x7f755c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.459+0000 7f75627b8640 1 -- 172.21.15.98:0/4260181559 >> 172.21.15.98:0/4260181559 conn(0x7f755c0fd8c0 msgr2=0x7f755c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.459+0000 7f75627b8640 1 -- 172.21.15.98:0/4260181559 shutdown_connections 2024-09-19T08:02:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.459+0000 7f75627b8640 1 -- 172.21.15.98:0/4260181559 wait complete. 2024-09-19T08:02:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f75627b8640 1 Processor -- start 2024-09-19T08:02:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f75627b8640 1 -- start start 2024-09-19T08:02:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f75627b8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f755c101de0 0x7f755c198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f75627b8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f755c102790 0x7f755c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f75627b8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f755c106f10 0x7f755c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f75627b8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f755c111f70 con 0x7f755c101de0 2024-09-19T08:02:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f75627b8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f755c111df0 con 0x7f755c102790 2024-09-19T08:02:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f75627b8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f755c1120f0 con 0x7f755c106f10 2024-09-19T08:02:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f755bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f755c101de0 0x7f755c198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f755bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f755c101de0 0x7f755c198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42478/0 (socket says 172.21.15.98:42478) 2024-09-19T08:02:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f755bfff640 1 -- 172.21.15.98:0/997283856 learned_addr learned my addr 172.21.15.98:0/997283856 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f7560d55640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f755c106f10 0x7f755c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f755b7fe640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f755c102790 0x7f755c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f755bfff640 1 -- 172.21.15.98:0/997283856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f755c106f10 msgr2=0x7f755c19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f755bfff640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f755c106f10 0x7f755c19cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f755bfff640 1 -- 172.21.15.98:0/997283856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f755c102790 msgr2=0x7f755c198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f755bfff640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f755c102790 0x7f755c198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f755bfff640 1 -- 172.21.15.98:0/997283856 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f755c19d2e0 con 0x7f755c101de0 2024-09-19T08:02:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f7560d55640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f755c106f10 0x7f755c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.460+0000 7f755b7fe640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f755c102790 0x7f755c198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.461+0000 7f755bfff640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f755c101de0 0x7f755c198330 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f754c00c920 tx=0x7f754c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.461+0000 7f75597fa640 1 -- 172.21.15.98:0/997283856 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754c007c70 con 0x7f755c101de0 2024-09-19T08:02:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.461+0000 7f75597fa640 1 -- 172.21.15.98:0/997283856 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f754c007e10 con 0x7f755c101de0 2024-09-19T08:02:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.461+0000 7f75627b8640 1 -- 172.21.15.98:0/997283856 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f755c19d5d0 con 0x7f755c101de0 2024-09-19T08:02:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.461+0000 7f75597fa640 1 -- 172.21.15.98:0/997283856 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754c018890 con 0x7f755c101de0 2024-09-19T08:02:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.461+0000 7f75627b8640 1 -- 172.21.15.98:0/997283856 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f755c1a4eb0 con 0x7f755c101de0 2024-09-19T08:02:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.462+0000 7f75627b8640 1 -- 172.21.15.98:0/997283856 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f755c1021c0 con 0x7f755c101de0 2024-09-19T08:02:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.463+0000 7f75597fa640 1 -- 172.21.15.98:0/997283856 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f754c018a50 con 0x7f755c101de0 2024-09-19T08:02:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.463+0000 7f75597fa640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f75440a32d0 0x7f75440a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.464+0000 7f755b7fe640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f75440a32d0 0x7f75440a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:21.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.464+0000 7f75597fa640 1 -- 172.21.15.98:0/997283856 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f754c0cff30 con 0x7f755c101de0 2024-09-19T08:02:21.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.464+0000 7f755b7fe640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f75440a32d0 0x7f75440a5790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7550005d90 tx=0x7f7550005d20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:21.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.467+0000 7f75597fa640 1 -- 172.21.15.98:0/997283856 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f754c014030 con 0x7f755c101de0 2024-09-19T08:02:21.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.636+0000 7f75627b8640 1 -- 172.21.15.98:0/997283856 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f755c199690 con 0x7f755c101de0 2024-09-19T08:02:21.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.637+0000 7f75597fa640 1 -- 172.21.15.98:0/997283856 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22754 (secure 0 0 0) 0x7f754c09cb90 con 0x7f755c101de0 2024-09-19T08:02:21.638 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:02:21.638 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":80,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","created":"2024-09-19T07:53:39.127398+0000","modified":"2024-09-19T08:01:26.714628+0000","last_up_change":"2024-09-19T08:00:22.467777+0000","last_in_change":"2024-09-19T07:59:58.538492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:56:43.773892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T08:01:07.693796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T08:01:09.518028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d01ce6-165d-493d-ae98-d8ad9e5d4134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6803","nonce":1719360533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6805","nonce":1719360533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6809","nonce":1719360533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6807","nonce":1719360533}]},"public_addr":"172.21.15.98:6803/1719360533","cluster_addr":"172.21.15.98:6805/1719360533","heartbeat_back_addr":"172.21.15.98:6809/1719360533","heartbeat_front_addr":"172.21.15.98:6807/1719360533","state":["exists","up"]},{"osd":1,"uuid":"34724bd5-0738-4473-b087-c09031c1ee2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036962626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036962626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036962626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036962626}]},"public_addr":"172.21.15.139:6801/4036962626","cluster_addr":"172.21.15.139:6803/4036962626","heartbeat_back_addr":"172.21.15.139:6807/4036962626","heartbeat_front_addr":"172.21.15.139:6805/4036962626","state":["exists","up"]},{"osd":2,"uuid":"16985ebe-0e32-42ee-bded-b7838e290afc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6801","nonce":2811412891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6803","nonce":2811412891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6807","nonce":2811412891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6805","nonce":2811412891}]},"public_addr":"172.21.15.162:6801/2811412891","cluster_addr":"172.21.15.162:6803/2811412891","heartbeat_back_addr":"172.21.15.162:6807/2811412891","heartbeat_front_addr":"172.21.15.162:6805/2811412891","state":["exists","up"]},{"osd":3,"uuid":"b140e466-3119-49cb-aec2-ba6ca4d10d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6811","nonce":1383392014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6813","nonce":1383392014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6817","nonce":1383392014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6815","nonce":1383392014}]},"public_addr":"172.21.15.98:6811/1383392014","cluster_addr":"172.21.15.98:6813/1383392014","heartbeat_back_addr":"172.21.15.98:6817/1383392014","heartbeat_front_addr":"172.21.15.98:6815/1383392014","state":["exists","up"]},{"osd":4,"uuid":"b4412813-df57-4ac6-8fb7-f57b69c74768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6809","nonce":116453520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6811","nonce":116453520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6815","nonce":116453520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6813","nonce":116453520}]},"public_addr":"172.21.15.139:6809/116453520","cluster_addr":"172.21.15.139:6811/116453520","heartbeat_back_addr":"172.21.15.139:6815/116453520","heartbeat_front_addr":"172.21.15.139:6813/116453520","state":["exists","up"]},{"osd":5,"uuid":"d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6809","nonce":3769530297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6811","nonce":3769530297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6815","nonce":3769530297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6813","nonce":3769530297}]},"public_addr":"172.21.15.162:6809/3769530297","cluster_addr":"172.21.15.162:6811/3769530297","heartbeat_back_addr":"172.21.15.162:6815/3769530297","heartbeat_front_addr":"172.21.15.162:6813/3769530297","state":["exists","up"]},{"osd":6,"uuid":"ba1c1590-6aea-40dc-90c8-e21b88896909","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6819","nonce":692373267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6821","nonce":692373267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6825","nonce":692373267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6823","nonce":692373267}]},"public_addr":"172.21.15.98:6819/692373267","cluster_addr":"172.21.15.98:6821/692373267","heartbeat_back_addr":"172.21.15.98:6825/692373267","heartbeat_front_addr":"172.21.15.98:6823/692373267","state":["exists","up"]},{"osd":7,"uuid":"eb9093ad-a068-4a63-ab87-0b50522cbc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6817","nonce":1963811650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6819","nonce":1963811650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6823","nonce":1963811650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6821","nonce":1963811650}]},"public_addr":"172.21.15.139:6817/1963811650","cluster_addr":"172.21.15.139:6819/1963811650","heartbeat_back_addr":"172.21.15.139:6823/1963811650","heartbeat_front_addr":"172.21.15.139:6821/1963811650","state":["exists","up"]},{"osd":8,"uuid":"31ca330a-fc45-466d-81c2-ef4386b3cc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6817","nonce":3607966458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6819","nonce":3607966458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6823","nonce":3607966458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6821","nonce":3607966458}]},"public_addr":"172.21.15.162:6817/3607966458","cluster_addr":"172.21.15.162:6819/3607966458","heartbeat_back_addr":"172.21.15.162:6823/3607966458","heartbeat_front_addr":"172.21.15.162:6821/3607966458","state":["exists","up"]},{"osd":9,"uuid":"9c4f216f-aef1-4367-9c58-b913b0c4f7db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6827","nonce":2050391244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6829","nonce":2050391244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6833","nonce":2050391244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6831","nonce":2050391244}]},"public_addr":"172.21.15.98:6827/2050391244","cluster_addr":"172.21.15.98:6829/2050391244","heartbeat_back_addr":"172.21.15.98:6833/2050391244","heartbeat_front_addr":"172.21.15.98:6831/2050391244","state":["exists","up"]},{"osd":10,"uuid":"03cc7df6-84ae-4d9a-a3d7-74c91e1744e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6825","nonce":2336761639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6827","nonce":2336761639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6831","nonce":2336761639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6829","nonce":2336761639}]},"public_addr":"172.21.15.139:6825/2336761639","cluster_addr":"172.21.15.139:6827/2336761639","heartbeat_back_addr":"172.21.15.139:6831/2336761639","heartbeat_front_addr":"172.21.15.139:6829/2336761639","state":["exists","up"]},{"osd":11,"uuid":"29e95498-4952-46d4-8fa6-6d8e2b862111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6825","nonce":1231011554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6827","nonce":1231011554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6831","nonce":1231011554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6829","nonce":1231011554}]},"public_addr":"172.21.15.162:6825/1231011554","cluster_addr":"172.21.15.162:6827/1231011554","heartbeat_back_addr":"172.21.15.162:6831/1231011554","heartbeat_front_addr":"172.21.15.162:6829/1231011554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:55:49.995244+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:15.655543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:40.331165+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:04.688881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:29.174966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:53.663161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:17.407551+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:42.584296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:06.317219+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:31.078625+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:55.689269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T08:00:19.659748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1},{"from":7,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/3842627255":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3838590070":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/1238193420":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6800/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6800/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3194864054":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3866031018":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/1497156658":"2024-09-20T07:54:02.418785+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T08:02:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.639+0000 7f75627b8640 1 -- 172.21.15.98:0/997283856 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f75440a32d0 msgr2=0x7f75440a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.639+0000 7f75627b8640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f75440a32d0 0x7f75440a5790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7550005d90 tx=0x7f7550005d20 comp rx=0 tx=0).stop 2024-09-19T08:02:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.639+0000 7f75627b8640 1 -- 172.21.15.98:0/997283856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f755c101de0 msgr2=0x7f755c198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.639+0000 7f75627b8640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f755c101de0 0x7f755c198330 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f754c00c920 tx=0x7f754c00cde0 comp rx=0 tx=0).stop 2024-09-19T08:02:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.639+0000 7f75627b8640 1 -- 172.21.15.98:0/997283856 shutdown_connections 2024-09-19T08:02:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.639+0000 7f75627b8640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f75440a32d0 0x7f75440a5790 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.639+0000 7f75627b8640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f755c106f10 0x7f755c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.639+0000 7f75627b8640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f755c102790 0x7f755c198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:21.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.639+0000 7f75627b8640 1 --2- 172.21.15.98:0/997283856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f755c101de0 0x7f755c198330 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:21.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.639+0000 7f75627b8640 1 -- 172.21.15.98:0/997283856 >> 172.21.15.98:0/997283856 conn(0x7f755c0fd8c0 msgr2=0x7f755c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:21.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.640+0000 7f75627b8640 1 -- 172.21.15.98:0/997283856 shutdown_connections 2024-09-19T08:02:21.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:21.640+0000 7f75627b8640 1 -- 172.21.15.98:0/997283856 wait complete. 2024-09-19T08:02:21.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:21 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/997283856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:02:21.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:21 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/997283856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:02:22.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:21 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/997283856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:02:22.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:02:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:22.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:22 smithi098 ceph-mon[31333]: pgmap v308: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:22.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:22 smithi139 ceph-mon[34874]: pgmap v308: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.001+0000 7fa77732a640 1 -- 172.21.15.98:0/3936991340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa770101450 msgr2=0x7fa770105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.001+0000 7fa77732a640 1 --2- 172.21.15.98:0/3936991340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa770101450 0x7fa770105920 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa760009900 tx=0x7fa76002a470 comp rx=0 tx=0).stop 2024-09-19T08:02:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.001+0000 7fa77732a640 1 -- 172.21.15.98:0/3936991340 shutdown_connections 2024-09-19T08:02:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.001+0000 7fa77732a640 1 --2- 172.21.15.98:0/3936991340 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa770105f40 0x7fa7701063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.001+0000 7fa77732a640 1 --2- 172.21.15.98:0/3936991340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa770101450 0x7fa770105920 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.001+0000 7fa77732a640 1 --2- 172.21.15.98:0/3936991340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa770100b30 0x7fa770100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.001+0000 7fa77732a640 1 -- 172.21.15.98:0/3936991340 >> 172.21.15.98:0/3936991340 conn(0x7fa7700fc7a0 msgr2=0x7fa7700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.001+0000 7fa77732a640 1 -- 172.21.15.98:0/3936991340 shutdown_connections 2024-09-19T08:02:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.001+0000 7fa77732a640 1 -- 172.21.15.98:0/3936991340 wait complete. 2024-09-19T08:02:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.002+0000 7fa77732a640 1 Processor -- start 2024-09-19T08:02:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.002+0000 7fa77732a640 1 -- start start 2024-09-19T08:02:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.002+0000 7fa77732a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa770100b30 0x7fa7701a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.002+0000 7fa77732a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa770101450 0x7fa7701a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.002+0000 7fa77732a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa770105f40 0x7fa77019e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.002+0000 7fa77732a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa770118640 con 0x7fa770105f40 2024-09-19T08:02:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.002+0000 7fa77732a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa7701184c0 con 0x7fa770101450 2024-09-19T08:02:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.002+0000 7fa77732a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7701187c0 con 0x7fa770100b30 2024-09-19T08:02:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.002+0000 7fa7758c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa770105f40 0x7fa77019e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.002+0000 7fa7758c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa770105f40 0x7fa77019e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42500/0 (socket says 172.21.15.98:42500) 2024-09-19T08:02:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.002+0000 7fa7758c7640 1 -- 172.21.15.98:0/1348828185 learned_addr learned my addr 172.21.15.98:0/1348828185 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.002+0000 7fa7748c5640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa770101450 0x7fa7701a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.002+0000 7fa7750c6640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa770100b30 0x7fa7701a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.003+0000 7fa7758c7640 1 -- 172.21.15.98:0/1348828185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa770100b30 msgr2=0x7fa7701a4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.003+0000 7fa7758c7640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa770100b30 0x7fa7701a4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.003+0000 7fa7758c7640 1 -- 172.21.15.98:0/1348828185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa770101450 msgr2=0x7fa7701a4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.003+0000 7fa7758c7640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa770101450 0x7fa7701a4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.003+0000 7fa7758c7640 1 -- 172.21.15.98:0/1348828185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77019f100 con 0x7fa770105f40 2024-09-19T08:02:23.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.003+0000 7fa7750c6640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa770100b30 0x7fa7701a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:23.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.003+0000 7fa7758c7640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa770105f40 0x7fa77019e9d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa76800b970 tx=0x7fa76800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:23.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.003+0000 7fa7667fc640 1 -- 172.21.15.98:0/1348828185 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa768019070 con 0x7fa770105f40 2024-09-19T08:02:23.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.003+0000 7fa7667fc640 1 -- 172.21.15.98:0/1348828185 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa76800cb40 con 0x7fa770105f40 2024-09-19T08:02:23.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.003+0000 7fa7667fc640 1 -- 172.21.15.98:0/1348828185 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa768007650 con 0x7fa770105f40 2024-09-19T08:02:23.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.003+0000 7fa77732a640 1 -- 172.21.15.98:0/1348828185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa77019f3f0 con 0x7fa770105f40 2024-09-19T08:02:23.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.003+0000 7fa77732a640 1 -- 172.21.15.98:0/1348828185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7701ab6e0 con 0x7fa770105f40 2024-09-19T08:02:23.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.004+0000 7fa77732a640 1 -- 172.21.15.98:0/1348828185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7701059a0 con 0x7fa770105f40 2024-09-19T08:02:23.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.005+0000 7fa7667fc640 1 -- 172.21.15.98:0/1348828185 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa76801c070 con 0x7fa770105f40 2024-09-19T08:02:23.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.006+0000 7fa7667fc640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa75c0a3350 0x7fa75c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:23.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.006+0000 7fa7750c6640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa75c0a3350 0x7fa75c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:23.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.006+0000 7fa7667fc640 1 -- 172.21.15.98:0/1348828185 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fa7680cf420 con 0x7fa770105f40 2024-09-19T08:02:23.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.006+0000 7fa7750c6640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa75c0a3350 0x7fa75c0a5810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa76c006fd0 tx=0x7fa76c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:23.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.009+0000 7fa7667fc640 1 -- 172.21.15.98:0/1348828185 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fa76809c100 con 0x7fa770105f40 2024-09-19T08:02:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:22 smithi162 ceph-mon[35260]: pgmap v308: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.211+0000 7fa77732a640 1 -- 172.21.15.98:0/1348828185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa77019ff00 con 0x7fa770105f40 2024-09-19T08:02:23.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.212+0000 7fa7667fc640 1 -- 172.21.15.98:0/1348828185 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11983 (secure 0 0 0) 0x7fa77019ff00 con 0x7fa770105f40 2024-09-19T08:02:23.212 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:02:23.212 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":27,"btime":"2024-09-19T08:01:22:699589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:01:22.699569+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24461,24437,24406,24376]},"id":1}]} 2024-09-19T08:02:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 27 2024-09-19T08:02:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.214+0000 7fa77732a640 1 -- 172.21.15.98:0/1348828185 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa75c0a3350 msgr2=0x7fa75c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.214+0000 7fa77732a640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa75c0a3350 0x7fa75c0a5810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa76c006fd0 tx=0x7fa76c008040 comp rx=0 tx=0).stop 2024-09-19T08:02:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.214+0000 7fa77732a640 1 -- 172.21.15.98:0/1348828185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa770105f40 msgr2=0x7fa77019e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.214+0000 7fa77732a640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa770105f40 0x7fa77019e9d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa76800b970 tx=0x7fa76800be30 comp rx=0 tx=0).stop 2024-09-19T08:02:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.214+0000 7fa77732a640 1 -- 172.21.15.98:0/1348828185 shutdown_connections 2024-09-19T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.214+0000 7fa77732a640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fa75c0a3350 0x7fa75c0a5810 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.214+0000 7fa77732a640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa770105f40 0x7fa77019e9d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.214+0000 7fa77732a640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa770101450 0x7fa7701a4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.214+0000 7fa77732a640 1 --2- 172.21.15.98:0/1348828185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa770100b30 0x7fa7701a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.214+0000 7fa77732a640 1 -- 172.21.15.98:0/1348828185 >> 172.21.15.98:0/1348828185 conn(0x7fa7700fc7a0 msgr2=0x7fa77010f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.214+0000 7fa77732a640 1 -- 172.21.15.98:0/1348828185 shutdown_connections 2024-09-19T08:02:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:23.214+0000 7fa77732a640 1 -- 172.21.15.98:0/1348828185 wait complete. 2024-09-19T08:02:23.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-19T08:01:11.307169+0000', 'modified': '2024-09-19T08:01:22.699569+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24485, 'mds_1': 24461, 'mds_2': 24437, 'mds_3': 24406, 'mds_4': 24376}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14631': {'gid': 14631, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6839/3417988465', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 3417988465}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 3417988465}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14643': {'gid': 14643, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6841/1339458935', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6840', 'nonce': 1339458935}, {'type': 'v1', 'addr': '172.21.15.98:6841', 'nonce': 1339458935}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24376': {'gid': 24376, 'name': 'l', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.162:6833/3959364141', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6832', 'nonce': 3959364141}, {'type': 'v1', 'addr': '172.21.15.162:6833', 'nonce': 3959364141}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24406': {'gid': 24406, 'name': 'f', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.162:6835/1149502202', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6834', 'nonce': 1149502202}, {'type': 'v1', 'addr': '172.21.15.162:6835', 'nonce': 1149502202}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24413': {'gid': 24413, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6833/2267154969', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6832', 'nonce': 2267154969}, {'type': 'v1', 'addr': '172.21.15.139:6833', 'nonce': 2267154969}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24415': {'gid': 24415, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6837/3969500353', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6836', 'nonce': 3969500353}, {'type': 'v1', 'addr': '172.21.15.162:6837', 'nonce': 3969500353}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6839/3810709211', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6838', 'nonce': 3810709211}, {'type': 'v1', 'addr': '172.21.15.162:6839', 'nonce': 3810709211}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.139:6835/1275838513', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6834', 'nonce': 1275838513}, {'type': 'v1', 'addr': '172.21.15.139:6835', 'nonce': 1275838513}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.139:6837/1885641254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6836', 'nonce': 1885641254}, {'type': 'v1', 'addr': '172.21.15.139:6837', 'nonce': 1885641254}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.139:6839/1001534713', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6838', 'nonce': 1001534713}, {'type': 'v1', 'addr': '172.21.15.139:6839', 'nonce': 1001534713}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24485, 'qdb_cluster': [24485, 24461, 24437, 24406, 24376]} 2024-09-19T08:02:23.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-19T08:02:23.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.l status 2024-09-19T08:02:23.919 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:23 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1348828185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:23.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:23 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1348828185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:24.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:23 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1348828185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:24.142 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:24.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.645+0000 7f787b8eb640 1 -- 172.21.15.98:0/56476594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7874101420 msgr2=0x7f787410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:24.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.645+0000 7f787b8eb640 1 --2- 172.21.15.98:0/56476594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7874101420 0x7f787410d950 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f786c009cd0 tx=0x7f786c02a8e0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.645+0000 7f787b8eb640 1 -- 172.21.15.98:0/56476594 shutdown_connections 2024-09-19T08:02:24.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.645+0000 7f787b8eb640 1 --2- 172.21.15.98:0/56476594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7874101420 0x7f787410d950 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.645+0000 7f787b8eb640 1 --2- 172.21.15.98:0/56476594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7874100a80 0x7f7874100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.645+0000 7f787b8eb640 1 --2- 172.21.15.98:0/56476594 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7874106a80 0x7f7874106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.645+0000 7f787b8eb640 1 -- 172.21.15.98:0/56476594 >> 172.21.15.98:0/56476594 conn(0x7f78740fc7a0 msgr2=0x7f78740febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:24.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.645+0000 7f787b8eb640 1 -- 172.21.15.98:0/56476594 shutdown_connections 2024-09-19T08:02:24.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.645+0000 7f787b8eb640 1 -- 172.21.15.98:0/56476594 wait complete. 2024-09-19T08:02:24.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f787b8eb640 1 Processor -- start 2024-09-19T08:02:24.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f787b8eb640 1 -- start start 2024-09-19T08:02:24.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f787b8eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7874100a80 0x7f7874102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f787b8eb640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7874101420 0x7f7874102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f787b8eb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7874106a80 0x7f78741030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f787b8eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7874110e80 con 0x7f7874100a80 2024-09-19T08:02:24.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f787b8eb640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7874110d00 con 0x7f7874101420 2024-09-19T08:02:24.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f787b8eb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7874111000 con 0x7f7874106a80 2024-09-19T08:02:24.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f7879687640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7874100a80 0x7f7874102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f7879687640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7874100a80 0x7f7874102650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42524/0 (socket says 172.21.15.98:42524) 2024-09-19T08:02:24.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f7879687640 1 -- 172.21.15.98:0/1062225818 learned_addr learned my addr 172.21.15.98:0/1062225818 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:24.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f7879e88640 1 --2- 172.21.15.98:0/1062225818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7874106a80 0x7f78741030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f7878e86640 1 --2- 172.21.15.98:0/1062225818 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7874101420 0x7f7874102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f7879687640 1 -- 172.21.15.98:0/1062225818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7874106a80 msgr2=0x7f78741030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:24.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.646+0000 7f7879687640 1 --2- 172.21.15.98:0/1062225818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7874106a80 0x7f78741030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.647+0000 7f7879687640 1 -- 172.21.15.98:0/1062225818 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7874101420 msgr2=0x7f7874102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:24.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.647+0000 7f7879687640 1 --2- 172.21.15.98:0/1062225818 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7874101420 0x7f7874102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.647+0000 7f7879687640 1 -- 172.21.15.98:0/1062225818 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78741a6810 con 0x7f7874100a80 2024-09-19T08:02:24.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.647+0000 7f7879687640 1 --2- 172.21.15.98:0/1062225818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7874100a80 0x7f7874102650 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f786400c920 tx=0x7f786400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:24.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.647+0000 7f786a7fc640 1 -- 172.21.15.98:0/1062225818 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864007c70 con 0x7f7874100a80 2024-09-19T08:02:24.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.647+0000 7f787b8eb640 1 -- 172.21.15.98:0/1062225818 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78741a6b00 con 0x7f7874100a80 2024-09-19T08:02:24.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.647+0000 7f786a7fc640 1 -- 172.21.15.98:0/1062225818 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7864007e10 con 0x7f7874100a80 2024-09-19T08:02:24.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.647+0000 7f786a7fc640 1 -- 172.21.15.98:0/1062225818 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864018890 con 0x7f7874100a80 2024-09-19T08:02:24.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.647+0000 7f787b8eb640 1 -- 172.21.15.98:0/1062225818 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78741a7040 con 0x7f7874100a80 2024-09-19T08:02:24.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.649+0000 7f786a7fc640 1 -- 172.21.15.98:0/1062225818 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7864018a70 con 0x7f7874100a80 2024-09-19T08:02:24.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.649+0000 7f787dcc4740 1 Processor -- start 2024-09-19T08:02:24.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.649+0000 7f787dcc4740 1 -- start start 2024-09-19T08:02:24.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f787dcc4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac39d0 0x557fd1ac3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f787dcc4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1ac42f0 0x557fd1ace670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f787dcc4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1acebb0 0x557fd1ad0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f787dcc4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557fd1a2a090 con 0x557fd1ac42f0 2024-09-19T08:02:24.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f787dcc4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557fd19d1a70 con 0x557fd1ac39d0 2024-09-19T08:02:24.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f787dcc4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557fd1a26050 con 0x557fd1acebb0 2024-09-19T08:02:24.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f786a7fc640 1 --2- 172.21.15.98:0/1062225818 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f78600a32d0 0x7f78600a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f7879687640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1ac42f0 0x557fd1ace670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f7879e88640 1 --2- 172.21.15.98:0/1062225818 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f78600a32d0 0x7f78600a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f7879687640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1ac42f0 0x557fd1ace670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42532/0 (socket says 172.21.15.98:42532) 2024-09-19T08:02:24.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f7879687640 1 -- 172.21.15.98:0/3582426698 learned_addr learned my addr 172.21.15.98:0/3582426698 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:24.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f7879e88640 1 --2- 172.21.15.98:0/3582426698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac39d0 0x557fd1ac3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f7878e86640 1 --2- 172.21.15.98:0/3582426698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1acebb0 0x557fd1ad0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f786a7fc640 1 -- 172.21.15.98:0/1062225818 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f78640cfb60 con 0x7f7874100a80 2024-09-19T08:02:24.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f7878e86640 1 -- 172.21.15.98:0/3582426698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac39d0 msgr2=0x557fd1ac3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:24.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f7878e86640 1 --2- 172.21.15.98:0/3582426698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac39d0 0x557fd1ac3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f7878e86640 1 -- 172.21.15.98:0/3582426698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1ac42f0 msgr2=0x557fd1ace670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:24.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f7878e86640 1 --2- 172.21.15.98:0/3582426698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1ac42f0 0x557fd1ace670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f7878e86640 1 -- 172.21.15.98:0/3582426698 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fd19de130 con 0x557fd1acebb0 2024-09-19T08:02:24.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.650+0000 7f7879e88640 1 --2- 172.21.15.98:0/1062225818 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f78600a32d0 0x7f78600a5790 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f786c005fe0 tx=0x7f786c009920 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:24.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.651+0000 7f7878e86640 1 --2- 172.21.15.98:0/3582426698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1acebb0 0x557fd1ad0fa0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f78700058c0 tx=0x7f787000cee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee2ee40ed8472b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:24.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.651+0000 7f785f7fe640 1 -- 172.21.15.98:0/3582426698 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787000ee40 con 0x557fd1acebb0 2024-09-19T08:02:24.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.651+0000 7f785f7fe640 1 -- 172.21.15.98:0/3582426698 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78700035f0 con 0x557fd1acebb0 2024-09-19T08:02:24.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.651+0000 7f785f7fe640 1 -- 172.21.15.98:0/3582426698 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7870010690 con 0x557fd1acebb0 2024-09-19T08:02:24.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.651+0000 7f787dcc4740 1 -- 172.21.15.98:0/3582426698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1acebb0 msgr2=0x557fd1ad0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:24.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.651+0000 7f787dcc4740 1 --2- 172.21.15.98:0/3582426698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1acebb0 0x557fd1ad0fa0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f78700058c0 tx=0x7f787000cee0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.651+0000 7f787dcc4740 1 -- 172.21.15.98:0/3582426698 shutdown_connections 2024-09-19T08:02:24.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.651+0000 7f787dcc4740 1 --2- 172.21.15.98:0/3582426698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1acebb0 0x557fd1ad0fa0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.651+0000 7f787dcc4740 1 --2- 172.21.15.98:0/3582426698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1ac42f0 0x557fd1ace670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.651+0000 7f787dcc4740 1 --2- 172.21.15.98:0/3582426698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac39d0 0x557fd1ac3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f787dcc4740 1 -- 172.21.15.98:0/3582426698 >> 172.21.15.98:0/3582426698 conn(0x557fd1abff60 msgr2=0x557fd1ac0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:24.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f787dcc4740 1 -- 172.21.15.98:0/3582426698 shutdown_connections 2024-09-19T08:02:24.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f787dcc4740 1 -- 172.21.15.98:0/3582426698 wait complete. 2024-09-19T08:02:24.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f787dcc4740 1 -- start start 2024-09-19T08:02:24.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f787dcc4740 1 Processor -- start 2024-09-19T08:02:24.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f787dcc4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac39d0 0x557fd1ace300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f787dcc4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac42f0 0x557fd1acb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f787dcc4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1acebb0 0x557fd1acbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f787dcc4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557fd1ad1780 con 0x557fd1acebb0 2024-09-19T08:02:24.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f787dcc4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557fd19e4f20 con 0x557fd1ac42f0 2024-09-19T08:02:24.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f787dcc4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557fd1ad1900 con 0x557fd1ac39d0 2024-09-19T08:02:24.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f787dcc4740 10 client.0 _ll_register_callbacks cb 0x557fd19ae030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:24.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f7879e88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1acebb0 0x557fd1acbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f7879687640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac39d0 0x557fd1ace300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f7879e88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1acebb0 0x557fd1acbb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42540/0 (socket says 172.21.15.98:42540) 2024-09-19T08:02:24.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.652+0000 7f7879e88640 1 -- 172.21.15.98:0/1726371738 learned_addr learned my addr 172.21.15.98:0/1726371738 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:24.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f7879e88640 1 -- 172.21.15.98:0/1726371738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac39d0 msgr2=0x557fd1ace300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:24.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f7879e88640 1 --2- 172.21.15.98:0/1726371738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac39d0 0x557fd1ace300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f7879e88640 1 -- 172.21.15.98:0/1726371738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac42f0 msgr2=0x557fd1acb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:24.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f7878e86640 1 --2- 172.21.15.98:0/1726371738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac42f0 0x557fd1acb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f7879e88640 1 --2- 172.21.15.98:0/1726371738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac42f0 0x557fd1acb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f7879e88640 1 -- 172.21.15.98:0/1726371738 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fd179b2b0 con 0x557fd1acebb0 2024-09-19T08:02:24.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f7879e88640 1 --2- 172.21.15.98:0/1726371738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1acebb0 0x557fd1acbb90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f786c035000 tx=0x7f786c049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52243e524a5f9634 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:24.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f785cff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:24.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f785cff9640 1 -- 172.21.15.98:0/1726371738 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c003e00 con 0x557fd1acebb0 2024-09-19T08:02:24.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f787dcc4740 1 -- 172.21.15.98:0/1726371738 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78700035f0 con 0x557fd1acebb0 2024-09-19T08:02:24.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f785cff9640 1 -- 172.21.15.98:0/1726371738 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f786c0045c0 con 0x557fd1acebb0 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f785cff9640 1 -- 172.21.15.98:0/1726371738 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f786c0047c0 con 0x557fd1acebb0 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f787dcc4740 10 client.15003 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f787dcc4740 1 -- 172.21.15.98:0/1726371738 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557fd1adcf80 con 0x557fd1acebb0 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.653+0000 7f785cff9640 1 -- 172.21.15.98:0/1726371738 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78700035f0 con 0x557fd1acebb0 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.654+0000 7f785cff9640 1 -- 172.21.15.98:0/1726371738 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f786c049070 con 0x557fd1acebb0 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.654+0000 7f787dcc4740 10 client.15003 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.654+0000 7f787dcc4740 10 client.15003 resolve_mds: resolved name 'l' to daemon mds.l 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.654+0000 7f787dcc4740 20 client.15003 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.654+0000 7f787dcc4740 1 --2- 172.21.15.98:0/1726371738 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557fd1adf640 0x557fd1ae0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.654+0000 7f787dcc4740 4 client.15003 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.654+0000 7f787dcc4740 1 -- 172.21.15.98:0/1726371738 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fd1a37600 con 0x557fd1adf640 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.654+0000 7f7879687640 1 --2- 172.21.15.98:0/1726371738 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557fd1adf640 0x557fd1ae0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.654+0000 7f785cff9640 10 client.15003 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.654+0000 7f7879687640 1 --2- 172.21.15.98:0/1726371738 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557fd1adf640 0x557fd1ae0ab0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3ce6557ef855757a server_cookie=5e69d4c8727b82e6 in_seq=0 out_seq=0 2024-09-19T08:02:24.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f785cff9640 1 -- 172.21.15.98:0/1726371738 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557fd1a37600 con 0x557fd1adf640 2024-09-19T08:02:24.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f785cff9640 10 client.15003 handle_command_reply: tid=0 2024-09-19T08:02:24.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f787dcc4740 1 client.15003 shutdown 2024-09-19T08:02:24.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f787dcc4740 1 -- 172.21.15.98:0/1726371738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1acebb0 msgr2=0x557fd1acbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:24.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f787dcc4740 1 --2- 172.21.15.98:0/1726371738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1acebb0 0x557fd1acbb90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f786c035000 tx=0x7f786c049000 comp rx=0 tx=0).stop 2024-09-19T08:02:24.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f787dcc4740 1 -- 172.21.15.98:0/1726371738 shutdown_connections 2024-09-19T08:02:24.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f787dcc4740 1 --2- 172.21.15.98:0/1726371738 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557fd1adf640 0x557fd1ae0ab0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f787dcc4740 1 --2- 172.21.15.98:0/1726371738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1acebb0 0x557fd1acbb90 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f785cff9640 0 client.15003 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:02:24.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f787dcc4740 1 --2- 172.21.15.98:0/1726371738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac42f0 0x557fd1acb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f787dcc4740 1 --2- 172.21.15.98:0/1726371738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac39d0 0x557fd1ace300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f787dcc4740 1 -- 172.21.15.98:0/1726371738 >> 172.21.15.98:0/1726371738 conn(0x557fd1abff60 msgr2=0x557fd1ac5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:24.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f787dcc4740 1 -- 172.21.15.98:0/1726371738 shutdown_connections 2024-09-19T08:02:24.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f787dcc4740 1 -- 172.21.15.98:0/1726371738 wait complete. 2024-09-19T08:02:24.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.655+0000 7f787dcc4740 20 client.15003 trim_cache size 0 max 16384 2024-09-19T08:02:24.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f787dcc4740 1 Processor -- start 2024-09-19T08:02:24.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f787dcc4740 1 -- start start 2024-09-19T08:02:24.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f787dcc4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac39d0 0x557fd1ae0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f787dcc4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac42f0 0x557fd1abace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f787dcc4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1acebb0 0x557fd1ac6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f787dcc4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557fd1adedf0 con 0x557fd1acebb0 2024-09-19T08:02:24.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f787dcc4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557fd1adf4c0 con 0x557fd1ac42f0 2024-09-19T08:02:24.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f787dcc4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557fd1adfb90 con 0x557fd1ac39d0 2024-09-19T08:02:24.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f7879e88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1acebb0 0x557fd1ac6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f7879e88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1acebb0 0x557fd1ac6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42550/0 (socket says 172.21.15.98:42550) 2024-09-19T08:02:24.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f7879e88640 1 -- 172.21.15.98:0/3884108254 learned_addr learned my addr 172.21.15.98:0/3884108254 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:24.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f7879687640 1 --2- 172.21.15.98:0/3884108254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac39d0 0x557fd1ae0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f7878e86640 1 --2- 172.21.15.98:0/3884108254 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac42f0 0x557fd1abace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f7879e88640 1 -- 172.21.15.98:0/3884108254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac39d0 msgr2=0x557fd1ae0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:24.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f7879e88640 1 --2- 172.21.15.98:0/3884108254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac39d0 0x557fd1ae0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f7879e88640 1 -- 172.21.15.98:0/3884108254 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac42f0 msgr2=0x557fd1abace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:24.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f7879e88640 1 --2- 172.21.15.98:0/3884108254 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac42f0 0x557fd1abace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f7879e88640 1 -- 172.21.15.98:0/3884108254 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fd1ace670 con 0x557fd1acebb0 2024-09-19T08:02:24.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f7879687640 1 --2- 172.21.15.98:0/3884108254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac39d0 0x557fd1ae0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:24.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.674+0000 7f7878e86640 1 --2- 172.21.15.98:0/3884108254 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac42f0 0x557fd1abace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:24.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.675+0000 7f7879e88640 1 --2- 172.21.15.98:0/3884108254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1acebb0 0x557fd1ac6610 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f786c054a00 tx=0x7f786c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a280a5744ddce368 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:24.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.675+0000 7f785effd640 1 -- 172.21.15.98:0/3884108254 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c054ad0 con 0x557fd1acebb0 2024-09-19T08:02:24.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.675+0000 7f785effd640 1 -- 172.21.15.98:0/3884108254 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f786c05a030 con 0x557fd1acebb0 2024-09-19T08:02:24.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.675+0000 7f785effd640 1 -- 172.21.15.98:0/3884108254 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c030a80 con 0x557fd1acebb0 2024-09-19T08:02:24.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.675+0000 7f787dcc4740 1 -- 172.21.15.98:0/3884108254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1acebb0 msgr2=0x557fd1ac6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:24.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.675+0000 7f787dcc4740 1 --2- 172.21.15.98:0/3884108254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1acebb0 0x557fd1ac6610 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f786c054a00 tx=0x7f786c0095c0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.675+0000 7f787dcc4740 1 -- 172.21.15.98:0/3884108254 shutdown_connections 2024-09-19T08:02:24.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.675+0000 7f787dcc4740 1 --2- 172.21.15.98:0/3884108254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1acebb0 0x557fd1ac6610 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.675+0000 7f787dcc4740 1 --2- 172.21.15.98:0/3884108254 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1ac42f0 0x557fd1abace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.675+0000 7f787dcc4740 1 --2- 172.21.15.98:0/3884108254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac39d0 0x557fd1ae0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.675+0000 7f787dcc4740 1 -- 172.21.15.98:0/3884108254 >> 172.21.15.98:0/3884108254 conn(0x557fd1abff60 msgr2=0x557fd1acb610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:24.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.676+0000 7f787dcc4740 1 -- 172.21.15.98:0/3884108254 shutdown_connections 2024-09-19T08:02:24.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.676+0000 7f787dcc4740 1 -- 172.21.15.98:0/3884108254 wait complete. 2024-09-19T08:02:24.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.676+0000 7f787dcc4740 1 -- start start 2024-09-19T08:02:24.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.676+0000 7f787dcc4740 1 Processor -- start 2024-09-19T08:02:24.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.676+0000 7f787dcc4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1ac39d0 0x557fd1ac8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.676+0000 7f787dcc4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac42f0 0x557fd1ac8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.676+0000 7f787dcc4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1acebb0 0x557fd1ae7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.676+0000 7f787dcc4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557fd1ae0930 con 0x557fd1ac39d0 2024-09-19T08:02:24.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.676+0000 7f787dcc4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557fd1ae0260 con 0x557fd1acebb0 2024-09-19T08:02:24.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.676+0000 7f787dcc4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557fd1ae1000 con 0x557fd1ac42f0 2024-09-19T08:02:24.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.676+0000 7f787dcc4740 10 client.0 _ll_register_callbacks cb 0x7f786c031a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:24.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.676+0000 7f7879687640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1ac39d0 0x557fd1ac8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.676+0000 7f7879687640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1ac39d0 0x557fd1ac8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42562/0 (socket says 172.21.15.98:42562) 2024-09-19T08:02:24.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.676+0000 7f7879687640 1 -- 172.21.15.98:0/1319355544 learned_addr learned my addr 172.21.15.98:0/1319355544 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:24.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f7879e88640 1 --2- 172.21.15.98:0/1319355544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1acebb0 0x557fd1ae7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f7878e86640 1 --2- 172.21.15.98:0/1319355544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac42f0 0x557fd1ac8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f7879687640 1 -- 172.21.15.98:0/1319355544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac42f0 msgr2=0x557fd1ac8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:24.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f7879687640 1 --2- 172.21.15.98:0/1319355544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac42f0 0x557fd1ac8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f7879687640 1 -- 172.21.15.98:0/1319355544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1acebb0 msgr2=0x557fd1ae7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:24.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f7879687640 1 --2- 172.21.15.98:0/1319355544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1acebb0 0x557fd1ae7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f7879687640 1 -- 172.21.15.98:0/1319355544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fd1ae8240 con 0x557fd1ac39d0 2024-09-19T08:02:24.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f7879687640 1 --2- 172.21.15.98:0/1319355544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1ac39d0 0x557fd1ac8020 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f786400fee0 tx=0x7f78640c6e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471771950be76397 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:24.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f785cff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:24.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f785cff9640 1 -- 172.21.15.98:0/1319355544 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78640a03e0 con 0x557fd1ac39d0 2024-09-19T08:02:24.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f785cff9640 1 -- 172.21.15.98:0/1319355544 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f786409a330 con 0x557fd1ac39d0 2024-09-19T08:02:24.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f787dcc4740 1 -- 172.21.15.98:0/1319355544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f786c05a030 con 0x557fd1ac39d0 2024-09-19T08:02:24.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f785cff9640 1 -- 172.21.15.98:0/1319355544 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f78640a4ca0 con 0x557fd1ac39d0 2024-09-19T08:02:24.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f787dcc4740 10 client.15009 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:24.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f787dcc4740 1 -- 172.21.15.98:0/1319355544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557fd1ae94e0 con 0x557fd1ac39d0 2024-09-19T08:02:24.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f785cff9640 1 -- 172.21.15.98:0/1319355544 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f786c05a030 con 0x557fd1ac39d0 2024-09-19T08:02:24.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f785cff9640 1 -- 172.21.15.98:0/1319355544 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f78640986a0 con 0x557fd1ac39d0 2024-09-19T08:02:24.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.677+0000 7f787dcc4740 10 client.15009 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:24.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.678+0000 7f787dcc4740 10 client.15009 resolve_mds: resolved name 'l' to daemon mds.l 2024-09-19T08:02:24.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.678+0000 7f787dcc4740 20 client.15009 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:24.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.678+0000 7f787dcc4740 1 --2- 172.21.15.98:0/1319355544 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557fd1b4e1e0 0x557fd1b4cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:24.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.678+0000 7f787dcc4740 4 client.15009 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:02:24.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.678+0000 7f787dcc4740 1 -- 172.21.15.98:0/1319355544 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "status"}) -- 0x557fd1a31b10 con 0x557fd1b4e1e0 2024-09-19T08:02:24.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.678+0000 7f7878e86640 1 --2- 172.21.15.98:0/1319355544 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557fd1b4e1e0 0x557fd1b4cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:24.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.678+0000 7f7878e86640 1 --2- 172.21.15.98:0/1319355544 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557fd1b4e1e0 0x557fd1b4cef0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c28206bb430afb0f server_cookie=f6c5c8040a89ac6b in_seq=0 out_seq=0 2024-09-19T08:02:24.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.678+0000 7f785cff9640 10 client.15009 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:02:24.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.678+0000 7f785cff9640 1 -- 172.21.15.98:0/1319355544 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557fd1a31b10 con 0x557fd1b4e1e0 2024-09-19T08:02:24.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.678+0000 7f785cff9640 10 client.15009 handle_command_reply: tid=0 2024-09-19T08:02:24.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.678+0000 7f787dcc4740 1 client.15009 shutdown 2024-09-19T08:02:24.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.678+0000 7f787dcc4740 1 -- 172.21.15.98:0/1319355544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1ac39d0 msgr2=0x557fd1ac8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:24.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.678+0000 7f787dcc4740 1 --2- 172.21.15.98:0/1319355544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1ac39d0 0x557fd1ac8020 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f786400fee0 tx=0x7f78640c6e00 comp rx=0 tx=0).stop 2024-09-19T08:02:24.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787dcc4740 1 -- 172.21.15.98:0/1319355544 shutdown_connections 2024-09-19T08:02:24.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787dcc4740 1 --2- 172.21.15.98:0/1319355544 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557fd1b4e1e0 0x557fd1b4cef0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787dcc4740 1 --2- 172.21.15.98:0/1319355544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fd1acebb0 0x557fd1ae7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787dcc4740 1 --2- 172.21.15.98:0/1319355544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fd1ac42f0 0x557fd1ac8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787dcc4740 1 --2- 172.21.15.98:0/1319355544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557fd1ac39d0 0x557fd1ac8020 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787dcc4740 1 -- 172.21.15.98:0/1319355544 >> 172.21.15.98:0/1319355544 conn(0x557fd1abff60 msgr2=0x557fd1ac6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:24.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f785cff9640 0 client.15009 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:02:24.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787dcc4740 1 -- 172.21.15.98:0/1319355544 shutdown_connections 2024-09-19T08:02:24.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787dcc4740 1 -- 172.21.15.98:0/1319355544 wait complete. 2024-09-19T08:02:24.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787dcc4740 20 client.15009 trim_cache size 0 max 16384 2024-09-19T08:02:24.685 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:02:24.685 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:02:24.685 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 4, 2024-09-19T08:02:24.685 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24376, 2024-09-19T08:02:24.685 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:02:24.685 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:02:24.685 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:02:24.685 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 62.988103899000002, 2024-09-19T08:02:24.685 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 27, 2024-09-19T08:02:24.685 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 80, 2024-09-19T08:02:24.685 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 78, 2024-09-19T08:02:24.686 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 114.739569582 2024-09-19T08:02:24.686 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:02:24.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787b8eb640 1 -- 172.21.15.98:0/1062225818 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f78600a32d0 msgr2=0x7f78600a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:24.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787b8eb640 1 --2- 172.21.15.98:0/1062225818 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f78600a32d0 0x7f78600a5790 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f786c005fe0 tx=0x7f786c009920 comp rx=0 tx=0).stop 2024-09-19T08:02:24.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787b8eb640 1 -- 172.21.15.98:0/1062225818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7874100a80 msgr2=0x7f7874102650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:24.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787b8eb640 1 --2- 172.21.15.98:0/1062225818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7874100a80 0x7f7874102650 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f786400c920 tx=0x7f786400cde0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787b8eb640 1 -- 172.21.15.98:0/1062225818 shutdown_connections 2024-09-19T08:02:24.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787b8eb640 1 --2- 172.21.15.98:0/1062225818 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f78600a32d0 0x7f78600a5790 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787b8eb640 1 --2- 172.21.15.98:0/1062225818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7874106a80 0x7f78741030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787b8eb640 1 --2- 172.21.15.98:0/1062225818 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7874101420 0x7f7874102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787b8eb640 1 --2- 172.21.15.98:0/1062225818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7874100a80 0x7f7874102650 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:24.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787b8eb640 1 -- 172.21.15.98:0/1062225818 >> 172.21.15.98:0/1062225818 conn(0x7f78740fc7a0 msgr2=0x7f78740fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:24.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.679+0000 7f787b8eb640 1 -- 172.21.15.98:0/1062225818 shutdown_connections 2024-09-19T08:02:24.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:24.680+0000 7f787b8eb640 1 -- 172.21.15.98:0/1062225818 wait complete. 2024-09-19T08:02:24.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:24 smithi139 ceph-mon[34874]: pgmap v309: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:24.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:24 smithi098 ceph-mon[31333]: pgmap v309: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:25.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:24 smithi162 ceph-mon[35260]: pgmap v309: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:25.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.f status 2024-09-19T08:02:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.110+0000 7f035ac82640 1 -- 172.21.15.98:0/3925854607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0354106a80 msgr2=0x7f0354106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.110+0000 7f035ac82640 1 --2- 172.21.15.98:0/3925854607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0354106a80 0x7f0354106e60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0348009930 tx=0x7f034802a420 comp rx=0 tx=0).stop 2024-09-19T08:02:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.110+0000 7f035ac82640 1 -- 172.21.15.98:0/3925854607 shutdown_connections 2024-09-19T08:02:26.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.110+0000 7f035ac82640 1 --2- 172.21.15.98:0/3925854607 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0354101420 0x7f035410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.110+0000 7f035ac82640 1 --2- 172.21.15.98:0/3925854607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0354100a80 0x7f0354100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.110+0000 7f035ac82640 1 --2- 172.21.15.98:0/3925854607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0354106a80 0x7f0354106e60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.110+0000 7f035ac82640 1 -- 172.21.15.98:0/3925854607 >> 172.21.15.98:0/3925854607 conn(0x7f03540fc7a0 msgr2=0x7f03540febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:26.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.110+0000 7f035ac82640 1 -- 172.21.15.98:0/3925854607 shutdown_connections 2024-09-19T08:02:26.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.110+0000 7f035ac82640 1 -- 172.21.15.98:0/3925854607 wait complete. 2024-09-19T08:02:26.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.110+0000 7f035ac82640 1 Processor -- start 2024-09-19T08:02:26.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.110+0000 7f035ac82640 1 -- start start 2024-09-19T08:02:26.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f035ac82640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0354100a80 0x7f0354198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f035ac82640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0354101420 0x7f0354198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f035ac82640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0354106a80 0x7f035419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f035ac82640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0354110f20 con 0x7f0354101420 2024-09-19T08:02:26.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f035ac82640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0354110da0 con 0x7f0354100a80 2024-09-19T08:02:26.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f035ac82640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f03541110a0 con 0x7f0354106a80 2024-09-19T08:02:26.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f0353fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0354101420 0x7f0354198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f0353fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0354101420 0x7f0354198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42580/0 (socket says 172.21.15.98:42580) 2024-09-19T08:02:26.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f0353fff640 1 -- 172.21.15.98:0/409459121 learned_addr learned my addr 172.21.15.98:0/409459121 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:26.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f0358a1e640 1 --2- 172.21.15.98:0/409459121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0354100a80 0x7f0354198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f035921f640 1 --2- 172.21.15.98:0/409459121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0354106a80 0x7f035419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f0353fff640 1 -- 172.21.15.98:0/409459121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0354106a80 msgr2=0x7f035419ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:26.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f0353fff640 1 --2- 172.21.15.98:0/409459121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0354106a80 0x7f035419ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f0353fff640 1 -- 172.21.15.98:0/409459121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0354100a80 msgr2=0x7f0354198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:26.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f0353fff640 1 --2- 172.21.15.98:0/409459121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0354100a80 0x7f0354198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f0353fff640 1 -- 172.21.15.98:0/409459121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f035419d5b0 con 0x7f0354101420 2024-09-19T08:02:26.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f0353fff640 1 --2- 172.21.15.98:0/409459121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0354101420 0x7f0354198b40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f034000b4a0 tx=0x7f034000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:26.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f0351ffb640 1 -- 172.21.15.98:0/409459121 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0340014070 con 0x7f0354101420 2024-09-19T08:02:26.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f0351ffb640 1 -- 172.21.15.98:0/409459121 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0340010af0 con 0x7f0354101420 2024-09-19T08:02:26.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f035ac82640 1 -- 172.21.15.98:0/409459121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f035419d8a0 con 0x7f0354101420 2024-09-19T08:02:26.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.111+0000 7f0351ffb640 1 -- 172.21.15.98:0/409459121 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0340017d10 con 0x7f0354101420 2024-09-19T08:02:26.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.112+0000 7f035ac82640 1 -- 172.21.15.98:0/409459121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03541a5180 con 0x7f0354101420 2024-09-19T08:02:26.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.114+0000 7f0351ffb640 1 -- 172.21.15.98:0/409459121 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f0340017070 con 0x7f0354101420 2024-09-19T08:02:26.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.114+0000 7f035d05b740 1 Processor -- start 2024-09-19T08:02:26.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.114+0000 7f035d05b740 1 -- start start 2024-09-19T08:02:26.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.114+0000 7f0351ffb640 1 --2- 172.21.15.98:0/409459121 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f03440a32d0 0x7f03440a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.114+0000 7f035d05b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6cd9d0 0x55985d6cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.114+0000 7f035d05b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6ce2f0 0x55985d6d8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.114+0000 7f035d05b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6d8bb0 0x55985d6dafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.114+0000 7f035d05b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55985d634090 con 0x55985d6cd9d0 2024-09-19T08:02:26.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.114+0000 7f035d05b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55985d5dba70 con 0x55985d6d8bb0 2024-09-19T08:02:26.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.114+0000 7f035921f640 1 --2- 172.21.15.98:0/409459121 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f03440a32d0 0x7f03440a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.114+0000 7f035d05b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55985d630050 con 0x55985d6ce2f0 2024-09-19T08:02:26.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.114+0000 7f0351ffb640 1 -- 172.21.15.98:0/409459121 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f03400ceed0 con 0x7f0354101420 2024-09-19T08:02:26.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.115+0000 7f0358a1e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6cd9d0 0x55985d6cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.115+0000 7f035921f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6d8bb0 0x55985d6dafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.115+0000 7f035921f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6d8bb0 0x55985d6dafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:44608/0 (socket says 172.21.15.98:44608) 2024-09-19T08:02:26.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.115+0000 7f035921f640 1 -- 172.21.15.98:0/2602401057 learned_addr learned my addr 172.21.15.98:0/2602401057 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:26.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.115+0000 7f0353fff640 1 --2- 172.21.15.98:0/2602401057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6ce2f0 0x55985d6d8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.115+0000 7f035921f640 1 -- 172.21.15.98:0/2602401057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6ce2f0 msgr2=0x55985d6d8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.115+0000 7f035921f640 1 --2- 172.21.15.98:0/2602401057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6ce2f0 0x55985d6d8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.115+0000 7f035921f640 1 -- 172.21.15.98:0/2602401057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6cd9d0 msgr2=0x55985d6cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.116+0000 7f035921f640 1 --2- 172.21.15.98:0/2602401057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6cd9d0 0x55985d6cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.116+0000 7f035921f640 1 -- 172.21.15.98:0/2602401057 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55985d5e8130 con 0x55985d6d8bb0 2024-09-19T08:02:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.116+0000 7f0358a1e640 1 --2- 172.21.15.98:0/2602401057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6cd9d0 0x55985d6cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.116+0000 7f0353fff640 1 --2- 172.21.15.98:0/2602401057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6ce2f0 0x55985d6d8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.116+0000 7f035921f640 1 --2- 172.21.15.98:0/2602401057 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6d8bb0 0x55985d6dafa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f034c00d620 tx=0x7f034c00dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c07fb5277ea0e9dd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.116+0000 7f033effd640 1 -- 172.21.15.98:0/2602401057 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c0187d0 con 0x55985d6d8bb0 2024-09-19T08:02:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.116+0000 7f035921f640 1 --2- 172.21.15.98:0/409459121 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f03440a32d0 0x7f03440a5790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f034c01f000 tx=0x7f034c018000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.116+0000 7f033effd640 1 -- 172.21.15.98:0/2602401057 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f034c018e50 con 0x55985d6d8bb0 2024-09-19T08:02:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.116+0000 7f033effd640 1 -- 172.21.15.98:0/2602401057 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c00ad90 con 0x55985d6d8bb0 2024-09-19T08:02:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.117+0000 7f035d05b740 1 -- 172.21.15.98:0/2602401057 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6d8bb0 msgr2=0x55985d6dafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.117+0000 7f035d05b740 1 --2- 172.21.15.98:0/2602401057 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6d8bb0 0x55985d6dafa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f034c00d620 tx=0x7f034c00dae0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.117+0000 7f035d05b740 1 -- 172.21.15.98:0/2602401057 shutdown_connections 2024-09-19T08:02:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.117+0000 7f035d05b740 1 --2- 172.21.15.98:0/2602401057 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6d8bb0 0x55985d6dafa0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.117+0000 7f035d05b740 1 --2- 172.21.15.98:0/2602401057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6ce2f0 0x55985d6d8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.117+0000 7f035d05b740 1 --2- 172.21.15.98:0/2602401057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6cd9d0 0x55985d6cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.117+0000 7f035d05b740 1 -- 172.21.15.98:0/2602401057 >> 172.21.15.98:0/2602401057 conn(0x55985d6c9f60 msgr2=0x55985d6ca340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.117+0000 7f035d05b740 1 -- 172.21.15.98:0/2602401057 shutdown_connections 2024-09-19T08:02:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.117+0000 7f035d05b740 1 -- 172.21.15.98:0/2602401057 wait complete. 2024-09-19T08:02:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.117+0000 7f035d05b740 1 -- start start 2024-09-19T08:02:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.117+0000 7f035d05b740 1 Processor -- start 2024-09-19T08:02:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f035d05b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6cd9d0 0x55985d6d8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f035d05b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6ce2f0 0x55985d6d5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f035d05b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6d8bb0 0x55985d6d5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f035d05b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55985d6dfb90 con 0x55985d6ce2f0 2024-09-19T08:02:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f035d05b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55985d5eef20 con 0x55985d6d8bb0 2024-09-19T08:02:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f035d05b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55985d6dfd10 con 0x55985d6cd9d0 2024-09-19T08:02:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f035d05b740 10 client.0 _ll_register_callbacks cb 0x55985d5b8030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f0353fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6ce2f0 0x55985d6d5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f0353fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6ce2f0 0x55985d6d5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42600/0 (socket says 172.21.15.98:42600) 2024-09-19T08:02:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f0353fff640 1 -- 172.21.15.98:0/2384431533 learned_addr learned my addr 172.21.15.98:0/2384431533 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f035921f640 1 --2- 172.21.15.98:0/2384431533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6d8bb0 0x55985d6d5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f0358a1e640 1 --2- 172.21.15.98:0/2384431533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6cd9d0 0x55985d6d8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f0353fff640 1 -- 172.21.15.98:0/2384431533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6cd9d0 msgr2=0x55985d6d8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f0353fff640 1 --2- 172.21.15.98:0/2384431533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6cd9d0 0x55985d6d8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f0353fff640 1 -- 172.21.15.98:0/2384431533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6d8bb0 msgr2=0x55985d6d5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f0353fff640 1 --2- 172.21.15.98:0/2384431533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6d8bb0 0x55985d6d5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f0353fff640 1 -- 172.21.15.98:0/2384431533 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55985d3a52b0 con 0x55985d6ce2f0 2024-09-19T08:02:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f0353fff640 1 --2- 172.21.15.98:0/2384431533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6ce2f0 0x55985d6d5790 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f03400978b0 tx=0x7f0340009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5676dd45d1c4431 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f0333fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.118+0000 7f0333fff640 1 -- 172.21.15.98:0/2384431533 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034001e070 con 0x55985d6ce2f0 2024-09-19T08:02:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.119+0000 7f0333fff640 1 -- 172.21.15.98:0/2384431533 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f0340097ac0 con 0x55985d6ce2f0 2024-09-19T08:02:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.119+0000 7f035d05b740 1 -- 172.21.15.98:0/2384431533 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f034c018e50 con 0x55985d6ce2f0 2024-09-19T08:02:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.119+0000 7f0333fff640 1 -- 172.21.15.98:0/2384431533 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f034009f800 con 0x55985d6ce2f0 2024-09-19T08:02:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.119+0000 7f035d05b740 10 client.15018 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.119+0000 7f035d05b740 1 -- 172.21.15.98:0/2384431533 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55985d6eb470 con 0x55985d6ce2f0 2024-09-19T08:02:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.119+0000 7f0333fff640 1 -- 172.21.15.98:0/2384431533 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f034c018e50 con 0x55985d6ce2f0 2024-09-19T08:02:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.119+0000 7f0333fff640 1 -- 172.21.15.98:0/2384431533 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f034009a8a0 con 0x55985d6ce2f0 2024-09-19T08:02:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.119+0000 7f035d05b740 10 client.15018 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.119+0000 7f035d05b740 10 client.15018 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-19T08:02:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.119+0000 7f035d05b740 20 client.15018 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.119+0000 7f035d05b740 1 --2- 172.21.15.98:0/2384431533 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55985d6eda60 0x55985d6eeed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.119+0000 7f035d05b740 4 client.15018 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:02:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.119+0000 7f035d05b740 1 -- 172.21.15.98:0/2384431533 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55985d641600 con 0x55985d6eda60 2024-09-19T08:02:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.119+0000 7f0358a1e640 1 --2- 172.21.15.98:0/2384431533 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55985d6eda60 0x55985d6eeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.120+0000 7f0333fff640 10 client.15018 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:02:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.120+0000 7f0358a1e640 1 --2- 172.21.15.98:0/2384431533 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55985d6eda60 0x55985d6eeed0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5900d7c928abdea4 server_cookie=1d883e04c11edc64 in_seq=0 out_seq=0 2024-09-19T08:02:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f0333fff640 1 -- 172.21.15.98:0/2384431533 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55985d641600 con 0x55985d6eda60 2024-09-19T08:02:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f0333fff640 10 client.15018 handle_command_reply: tid=0 2024-09-19T08:02:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f035d05b740 1 client.15018 shutdown 2024-09-19T08:02:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f035d05b740 1 -- 172.21.15.98:0/2384431533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6ce2f0 msgr2=0x55985d6d5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f035d05b740 1 --2- 172.21.15.98:0/2384431533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6ce2f0 0x55985d6d5790 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f03400978b0 tx=0x7f0340009370 comp rx=0 tx=0).stop 2024-09-19T08:02:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f035d05b740 1 -- 172.21.15.98:0/2384431533 shutdown_connections 2024-09-19T08:02:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f035d05b740 1 --2- 172.21.15.98:0/2384431533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6d8bb0 0x55985d6d5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f035d05b740 1 --2- 172.21.15.98:0/2384431533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6ce2f0 0x55985d6d5790 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f035d05b740 1 --2- 172.21.15.98:0/2384431533 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55985d6eda60 0x55985d6eeed0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f035d05b740 1 --2- 172.21.15.98:0/2384431533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6cd9d0 0x55985d6d8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f0333fff640 0 client.15018 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:02:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f035d05b740 1 -- 172.21.15.98:0/2384431533 >> 172.21.15.98:0/2384431533 conn(0x55985d6c9f60 msgr2=0x55985d6cfc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f035d05b740 1 -- 172.21.15.98:0/2384431533 shutdown_connections 2024-09-19T08:02:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f035d05b740 1 -- 172.21.15.98:0/2384431533 wait complete. 2024-09-19T08:02:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.121+0000 7f035d05b740 20 client.15018 trim_cache size 0 max 16384 2024-09-19T08:02:26.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.138+0000 7f035d05b740 1 Processor -- start 2024-09-19T08:02:26.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.138+0000 7f035d05b740 1 -- start start 2024-09-19T08:02:26.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f035d05b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6cd9d0 0x55985d6ee800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f035d05b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6ce2f0 0x55985d6d5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f035d05b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6d8bb0 0x55985d6f7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f035d05b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55985d6ed210 con 0x55985d6ce2f0 2024-09-19T08:02:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f035d05b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55985d6ed8e0 con 0x55985d6cd9d0 2024-09-19T08:02:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f035d05b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55985d6edfb0 con 0x55985d6d8bb0 2024-09-19T08:02:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f0353fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6ce2f0 0x55985d6d5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f0353fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6ce2f0 0x55985d6d5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42614/0 (socket says 172.21.15.98:42614) 2024-09-19T08:02:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f0353fff640 1 -- 172.21.15.98:0/404604789 learned_addr learned my addr 172.21.15.98:0/404604789 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f0353fff640 1 -- 172.21.15.98:0/404604789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6d8bb0 msgr2=0x55985d6f7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f0353fff640 1 --2- 172.21.15.98:0/404604789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6d8bb0 0x55985d6f7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f0353fff640 1 -- 172.21.15.98:0/404604789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6cd9d0 msgr2=0x55985d6ee800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f0358a1e640 1 --2- 172.21.15.98:0/404604789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6cd9d0 0x55985d6ee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f0353fff640 1 --2- 172.21.15.98:0/404604789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6cd9d0 0x55985d6ee800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f0353fff640 1 -- 172.21.15.98:0/404604789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55985d6d6550 con 0x55985d6ce2f0 2024-09-19T08:02:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f0353fff640 1 --2- 172.21.15.98:0/404604789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6ce2f0 0x55985d6d5050 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f03400a3b90 tx=0x7f034009b350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d84c304a82bf8364 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f033e7fc640 1 -- 172.21.15.98:0/404604789 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034001e050 con 0x55985d6ce2f0 2024-09-19T08:02:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f033e7fc640 1 -- 172.21.15.98:0/404604789 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0340097ac0 con 0x55985d6ce2f0 2024-09-19T08:02:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.139+0000 7f033e7fc640 1 -- 172.21.15.98:0/404604789 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0340097310 con 0x55985d6ce2f0 2024-09-19T08:02:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.140+0000 7f035d05b740 1 -- 172.21.15.98:0/404604789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6ce2f0 msgr2=0x55985d6d5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.140+0000 7f035d05b740 1 --2- 172.21.15.98:0/404604789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6ce2f0 0x55985d6d5050 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f03400a3b90 tx=0x7f034009b350 comp rx=0 tx=0).stop 2024-09-19T08:02:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.140+0000 7f035d05b740 1 -- 172.21.15.98:0/404604789 shutdown_connections 2024-09-19T08:02:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.140+0000 7f035d05b740 1 --2- 172.21.15.98:0/404604789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6d8bb0 0x55985d6f7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.140+0000 7f035d05b740 1 --2- 172.21.15.98:0/404604789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6ce2f0 0x55985d6d5050 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.140+0000 7f035d05b740 1 --2- 172.21.15.98:0/404604789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6cd9d0 0x55985d6ee800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.140+0000 7f035d05b740 1 -- 172.21.15.98:0/404604789 >> 172.21.15.98:0/404604789 conn(0x55985d6c9f60 msgr2=0x55985d6ca340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.140+0000 7f035d05b740 1 -- 172.21.15.98:0/404604789 shutdown_connections 2024-09-19T08:02:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.140+0000 7f035d05b740 1 -- 172.21.15.98:0/404604789 wait complete. 2024-09-19T08:02:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.140+0000 7f035d05b740 1 -- start start 2024-09-19T08:02:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.140+0000 7f035d05b740 1 Processor -- start 2024-09-19T08:02:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035d05b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6cd9d0 0x55985d6f5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035d05b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6ce2f0 0x55985d6f6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035d05b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6d8bb0 0x55985d6f6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035d05b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55985d6eed50 con 0x55985d6d8bb0 2024-09-19T08:02:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035d05b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55985d6ee680 con 0x55985d6cd9d0 2024-09-19T08:02:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035d05b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55985d6ef420 con 0x55985d6ce2f0 2024-09-19T08:02:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035d05b740 10 client.0 _ll_register_callbacks cb 0x7f0340097770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035921f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6d8bb0 0x55985d6f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f0358a1e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6cd9d0 0x55985d6f5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035921f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6d8bb0 0x55985d6f6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42622/0 (socket says 172.21.15.98:42622) 2024-09-19T08:02:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035921f640 1 -- 172.21.15.98:0/2429081568 learned_addr learned my addr 172.21.15.98:0/2429081568 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f0353fff640 1 --2- 172.21.15.98:0/2429081568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6ce2f0 0x55985d6f6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035921f640 1 -- 172.21.15.98:0/2429081568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6ce2f0 msgr2=0x55985d6f6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035921f640 1 --2- 172.21.15.98:0/2429081568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6ce2f0 0x55985d6f6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035921f640 1 -- 172.21.15.98:0/2429081568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6cd9d0 msgr2=0x55985d6f5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035921f640 1 --2- 172.21.15.98:0/2429081568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6cd9d0 0x55985d6f5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035921f640 1 -- 172.21.15.98:0/2429081568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55985d6f0280 con 0x55985d6d8bb0 2024-09-19T08:02:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f0353fff640 1 --2- 172.21.15.98:0/2429081568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6ce2f0 0x55985d6f6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f0358a1e640 1 --2- 172.21.15.98:0/2429081568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6cd9d0 0x55985d6f5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:02:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f035921f640 1 --2- 172.21.15.98:0/2429081568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6d8bb0 0x55985d6f6570 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f034c00a9a0 tx=0x7f034c00a930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8bdd3370a11676 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.141+0000 7f0333fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f0333fff640 1 -- 172.21.15.98:0/2429081568 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c032060 con 0x55985d6d8bb0 2024-09-19T08:02:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f0333fff640 1 -- 172.21.15.98:0/2429081568 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f034c00a600 con 0x55985d6d8bb0 2024-09-19T08:02:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f035d05b740 1 -- 172.21.15.98:0/2429081568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0340097ac0 con 0x55985d6d8bb0 2024-09-19T08:02:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f0333fff640 1 -- 172.21.15.98:0/2429081568 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f034c0187d0 con 0x55985d6d8bb0 2024-09-19T08:02:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f035d05b740 10 client.15024 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f035d05b740 1 -- 172.21.15.98:0/2429081568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55985d6f4d10 con 0x55985d6d8bb0 2024-09-19T08:02:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f0333fff640 1 -- 172.21.15.98:0/2429081568 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0340097ac0 con 0x55985d6d8bb0 2024-09-19T08:02:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f0333fff640 1 -- 172.21.15.98:0/2429081568 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f034c023930 con 0x55985d6d8bb0 2024-09-19T08:02:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f035d05b740 10 client.15024 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f035d05b740 10 client.15024 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-19T08:02:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f035d05b740 20 client.15024 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f035d05b740 1 --2- 172.21.15.98:0/2429081568 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55985d75bfc0 0x55985d75acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f035d05b740 4 client.15024 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:02:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f035d05b740 1 -- 172.21.15.98:0/2429081568 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "status"}) -- 0x55985d63bb10 con 0x55985d75bfc0 2024-09-19T08:02:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.142+0000 7f0358a1e640 1 --2- 172.21.15.98:0/2429081568 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55985d75bfc0 0x55985d75acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f0333fff640 10 client.15024 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:02:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f0358a1e640 1 --2- 172.21.15.98:0/2429081568 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55985d75bfc0 0x55985d75acd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=64add5580e6d62f4 server_cookie=8e6753d309c5919b in_seq=0 out_seq=0 2024-09-19T08:02:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f0333fff640 1 -- 172.21.15.98:0/2429081568 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55985d63bb10 con 0x55985d75bfc0 2024-09-19T08:02:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f0333fff640 10 client.15024 handle_command_reply: tid=0 2024-09-19T08:02:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f035d05b740 1 client.15024 shutdown 2024-09-19T08:02:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f035d05b740 1 -- 172.21.15.98:0/2429081568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6d8bb0 msgr2=0x55985d6f6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f035d05b740 1 --2- 172.21.15.98:0/2429081568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6d8bb0 0x55985d6f6570 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f034c00a9a0 tx=0x7f034c00a930 comp rx=0 tx=0).stop 2024-09-19T08:02:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f035d05b740 1 -- 172.21.15.98:0/2429081568 shutdown_connections 2024-09-19T08:02:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f035d05b740 1 --2- 172.21.15.98:0/2429081568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55985d6d8bb0 0x55985d6f6570 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f035d05b740 1 --2- 172.21.15.98:0/2429081568 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55985d75bfc0 0x55985d75acd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f035d05b740 1 --2- 172.21.15.98:0/2429081568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55985d6ce2f0 0x55985d6f6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f035d05b740 1 --2- 172.21.15.98:0/2429081568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55985d6cd9d0 0x55985d6f5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f035d05b740 1 -- 172.21.15.98:0/2429081568 >> 172.21.15.98:0/2429081568 conn(0x55985d6c9f60 msgr2=0x55985d6c87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f0333fff640 0 client.15024 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:02:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f035d05b740 1 -- 172.21.15.98:0/2429081568 shutdown_connections 2024-09-19T08:02:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f035d05b740 1 -- 172.21.15.98:0/2429081568 wait complete. 2024-09-19T08:02:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.143+0000 7f035d05b740 20 client.15024 trim_cache size 0 max 16384 2024-09-19T08:02:26.150 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:02:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:02:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 3, 2024-09-19T08:02:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24406, 2024-09-19T08:02:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:02:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:02:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:02:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 65.459421176999996, 2024-09-19T08:02:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 27, 2024-09-19T08:02:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 80, 2024-09-19T08:02:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 78, 2024-09-19T08:02:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 102.118113991 2024-09-19T08:02:26.151 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:02:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.144+0000 7f035ac82640 1 -- 172.21.15.98:0/409459121 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f03440a32d0 msgr2=0x7f03440a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.144+0000 7f035ac82640 1 --2- 172.21.15.98:0/409459121 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f03440a32d0 0x7f03440a5790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f034c01f000 tx=0x7f034c018000 comp rx=0 tx=0).stop 2024-09-19T08:02:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.144+0000 7f035ac82640 1 -- 172.21.15.98:0/409459121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0354101420 msgr2=0x7f0354198b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.144+0000 7f035ac82640 1 --2- 172.21.15.98:0/409459121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0354101420 0x7f0354198b40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f034000b4a0 tx=0x7f034000b960 comp rx=0 tx=0).stop 2024-09-19T08:02:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.144+0000 7f035ac82640 1 -- 172.21.15.98:0/409459121 shutdown_connections 2024-09-19T08:02:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.144+0000 7f035ac82640 1 --2- 172.21.15.98:0/409459121 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f03440a32d0 0x7f03440a5790 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.144+0000 7f035ac82640 1 --2- 172.21.15.98:0/409459121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0354106a80 0x7f035419ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.144+0000 7f035ac82640 1 --2- 172.21.15.98:0/409459121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0354101420 0x7f0354198b40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.144+0000 7f035ac82640 1 --2- 172.21.15.98:0/409459121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0354100a80 0x7f0354198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.144+0000 7f035ac82640 1 -- 172.21.15.98:0/409459121 >> 172.21.15.98:0/409459121 conn(0x7f03540fc7a0 msgr2=0x7f035406ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.144+0000 7f035ac82640 1 -- 172.21.15.98:0/409459121 shutdown_connections 2024-09-19T08:02:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:26.144+0000 7f035ac82640 1 -- 172.21.15.98:0/409459121 wait complete. 2024-09-19T08:02:26.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.e status 2024-09-19T08:02:26.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:26 smithi139 ceph-mon[34874]: pgmap v310: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:26.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:26 smithi098 ceph-mon[31333]: pgmap v310: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:27.360 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:26 smithi162 ceph-mon[35260]: pgmap v310: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.604+0000 7f43fc263640 1 -- 172.21.15.98:0/1746110995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f4102d40 msgr2=0x7f43f4103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.604+0000 7f43fc263640 1 --2- 172.21.15.98:0/1746110995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f4102d40 0x7f43f4103120 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009ca0 tx=0x7f43f002a950 comp rx=0 tx=0).stop 2024-09-19T08:02:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.605+0000 7f43fc263640 1 -- 172.21.15.98:0/1746110995 shutdown_connections 2024-09-19T08:02:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.605+0000 7f43fc263640 1 --2- 172.21.15.98:0/1746110995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43f410c170 0x7f43f410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.605+0000 7f43fc263640 1 --2- 172.21.15.98:0/1746110995 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43f4103660 0x7f43f410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.605+0000 7f43fc263640 1 --2- 172.21.15.98:0/1746110995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f4102d40 0x7f43f4103120 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.605+0000 7f43fc263640 1 -- 172.21.15.98:0/1746110995 >> 172.21.15.98:0/1746110995 conn(0x7f43f40fc7a0 msgr2=0x7f43f40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.605+0000 7f43fc263640 1 -- 172.21.15.98:0/1746110995 shutdown_connections 2024-09-19T08:02:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.605+0000 7f43fc263640 1 -- 172.21.15.98:0/1746110995 wait complete. 2024-09-19T08:02:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.605+0000 7f43fc263640 1 Processor -- start 2024-09-19T08:02:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.605+0000 7f43fc263640 1 -- start start 2024-09-19T08:02:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fc263640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43f4102d40 0x7f43f41984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fc263640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43f4103660 0x7f43f41989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fc263640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f410c170 0x7f43f419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fc263640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f43f4111fb0 con 0x7f43f410c170 2024-09-19T08:02:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fc263640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f43f4111e30 con 0x7f43f4102d40 2024-09-19T08:02:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fc263640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f43f4112130 con 0x7f43f4103660 2024-09-19T08:02:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fa800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f410c170 0x7f43f419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fa800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f410c170 0x7f43f419cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38236/0 (socket says 172.21.15.98:38236) 2024-09-19T08:02:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fa800640 1 -- 172.21.15.98:0/2501425801 learned_addr learned my addr 172.21.15.98:0/2501425801 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43f9fff640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43f4102d40 0x7f43f41984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43f97fe640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43f4103660 0x7f43f41989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fa800640 1 -- 172.21.15.98:0/2501425801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43f4103660 msgr2=0x7f43f41989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fa800640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43f4103660 0x7f43f41989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fa800640 1 -- 172.21.15.98:0/2501425801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43f4102d40 msgr2=0x7f43f41984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fa800640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43f4102d40 0x7f43f41984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fa800640 1 -- 172.21.15.98:0/2501425801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f419d460 con 0x7f43f410c170 2024-09-19T08:02:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43f9fff640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43f4102d40 0x7f43f41984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fa800640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f410c170 0x7f43f419cd80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00c970 tx=0x7f43ec00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43eaffd640 1 -- 172.21.15.98:0/2501425801 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec007bb0 con 0x7f43f410c170 2024-09-19T08:02:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43eaffd640 1 -- 172.21.15.98:0/2501425801 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43ec007d50 con 0x7f43f410c170 2024-09-19T08:02:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fc263640 1 -- 172.21.15.98:0/2501425801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f419d750 con 0x7f43f410c170 2024-09-19T08:02:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43eaffd640 1 -- 172.21.15.98:0/2501425801 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec00f450 con 0x7f43f410c170 2024-09-19T08:02:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.606+0000 7f43fc263640 1 -- 172.21.15.98:0/2501425801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f41a5030 con 0x7f43f410c170 2024-09-19T08:02:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.609+0000 7f43eaffd640 1 -- 172.21.15.98:0/2501425801 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f43ec004f60 con 0x7f43f410c170 2024-09-19T08:02:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.609+0000 7f43fe63c740 1 Processor -- start 2024-09-19T08:02:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.609+0000 7f43fe63c740 1 -- start start 2024-09-19T08:02:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.609+0000 7f43fe63c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641549d0 0x55ac64154db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.609+0000 7f43fe63c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641552f0 0x55ac6415f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.609+0000 7f43fe63c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64161fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.609+0000 7f43fe63c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ac640bb090 con 0x55ac6415fbb0 2024-09-19T08:02:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.609+0000 7f43fe63c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ac64062a70 con 0x55ac641552f0 2024-09-19T08:02:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.609+0000 7f43fe63c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ac640b7050 con 0x55ac641549d0 2024-09-19T08:02:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43eaffd640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43e00a32d0 0x7f43e00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43f97fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64161fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43f97fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64161fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38242/0 (socket says 172.21.15.98:38242) 2024-09-19T08:02:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43f97fe640 1 -- 172.21.15.98:0/1945067198 learned_addr learned my addr 172.21.15.98:0/1945067198 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43f97fe640 1 --2- 172.21.15.98:0/1945067198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641549d0 0x55ac64154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43fa800640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43e00a32d0 0x7f43e00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43f9fff640 1 --2- 172.21.15.98:0/1945067198 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641552f0 0x55ac6415f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43f97fe640 1 -- 172.21.15.98:0/1945067198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641549d0 msgr2=0x55ac64154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43eaffd640 1 -- 172.21.15.98:0/2501425801 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f43ec0cf8b0 con 0x7f43f410c170 2024-09-19T08:02:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43f97fe640 1 --2- 172.21.15.98:0/1945067198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641549d0 0x55ac64154db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43f97fe640 1 -- 172.21.15.98:0/1945067198 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641552f0 msgr2=0x55ac6415f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43f97fe640 1 --2- 172.21.15.98:0/1945067198 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641552f0 0x55ac6415f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43f97fe640 1 -- 172.21.15.98:0/1945067198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac6406f130 con 0x55ac6415fbb0 2024-09-19T08:02:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43f9fff640 1 --2- 172.21.15.98:0/1945067198 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641552f0 0x55ac6415f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43f97fe640 1 --2- 172.21.15.98:0/1945067198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64161fa0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f43e4016530 tx=0x7f43e400a360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ddd0ec680769ba2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43dffff640 1 -- 172.21.15.98:0/1945067198 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e4018430 con 0x55ac6415fbb0 2024-09-19T08:02:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43fa800640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43e00a32d0 0x7f43e00a5790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0a44c0 tx=0x7f43ec06aac0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43dffff640 1 -- 172.21.15.98:0/1945067198 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43e4018ab0 con 0x55ac6415fbb0 2024-09-19T08:02:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.610+0000 7f43dffff640 1 -- 172.21.15.98:0/1945067198 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e4017970 con 0x55ac6415fbb0 2024-09-19T08:02:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.611+0000 7f43fe63c740 1 -- 172.21.15.98:0/1945067198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 msgr2=0x55ac64161fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.611+0000 7f43fe63c740 1 --2- 172.21.15.98:0/1945067198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64161fa0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f43e4016530 tx=0x7f43e400a360 comp rx=0 tx=0).stop 2024-09-19T08:02:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.611+0000 7f43fe63c740 1 -- 172.21.15.98:0/1945067198 shutdown_connections 2024-09-19T08:02:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.611+0000 7f43fe63c740 1 --2- 172.21.15.98:0/1945067198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64161fa0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.611+0000 7f43fe63c740 1 --2- 172.21.15.98:0/1945067198 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641552f0 0x55ac6415f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.611+0000 7f43fe63c740 1 --2- 172.21.15.98:0/1945067198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641549d0 0x55ac64154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.611+0000 7f43fe63c740 1 -- 172.21.15.98:0/1945067198 >> 172.21.15.98:0/1945067198 conn(0x55ac64150f60 msgr2=0x55ac64151340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.611+0000 7f43fe63c740 1 -- 172.21.15.98:0/1945067198 shutdown_connections 2024-09-19T08:02:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.611+0000 7f43fe63c740 1 -- 172.21.15.98:0/1945067198 wait complete. 2024-09-19T08:02:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.611+0000 7f43fe63c740 1 -- start start 2024-09-19T08:02:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43fe63c740 1 Processor -- start 2024-09-19T08:02:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43fe63c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641549d0 0x55ac6415f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43fe63c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac641552f0 0x55ac6415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43fe63c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac6415fbb0 0x55ac6415cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43fe63c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ac64162780 con 0x55ac641552f0 2024-09-19T08:02:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43fe63c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ac64075f20 con 0x55ac6415fbb0 2024-09-19T08:02:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43fe63c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ac64162900 con 0x55ac641549d0 2024-09-19T08:02:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43fe63c740 10 client.0 _ll_register_callbacks cb 0x55ac6403f030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43f9fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac641552f0 0x55ac6415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43f9fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac641552f0 0x55ac6415c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38248/0 (socket says 172.21.15.98:38248) 2024-09-19T08:02:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43f9fff640 1 -- 172.21.15.98:0/2844492091 learned_addr learned my addr 172.21.15.98:0/2844492091 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43f97fe640 1 --2- 172.21.15.98:0/2844492091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641549d0 0x55ac6415f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43f97fe640 1 --2- 172.21.15.98:0/2844492091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac6415fbb0 0x55ac6415cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43f9fff640 1 -- 172.21.15.98:0/2844492091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641549d0 msgr2=0x55ac6415f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43f9fff640 1 --2- 172.21.15.98:0/2844492091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641549d0 0x55ac6415f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43f9fff640 1 -- 172.21.15.98:0/2844492091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac6415fbb0 msgr2=0x55ac6415cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43f9fff640 1 --2- 172.21.15.98:0/2844492091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac6415fbb0 0x55ac6415cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43f9fff640 1 -- 172.21.15.98:0/2844492091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac63e2c2b0 con 0x55ac641552f0 2024-09-19T08:02:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.612+0000 7f43f97fe640 1 --2- 172.21.15.98:0/2844492091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641549d0 0x55ac6415f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43f9fff640 1 --2- 172.21.15.98:0/2844492091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac641552f0 0x55ac6415c7a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f43f0005cc0 tx=0x7f43f002f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b0153e33c8efc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43dd7fa640 1 -- 172.21.15.98:0/2844492091 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0034040 con 0x55ac641552f0 2024-09-19T08:02:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43dd7fa640 1 -- 172.21.15.98:0/2844492091 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f43f0003b80 con 0x55ac641552f0 2024-09-19T08:02:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43fe63c740 1 -- 172.21.15.98:0/2844492091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43e4018ab0 con 0x55ac641552f0 2024-09-19T08:02:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43dd7fa640 1 -- 172.21.15.98:0/2844492091 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f43f002f740 con 0x55ac641552f0 2024-09-19T08:02:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43fe63c740 10 client.15036 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43fe63c740 1 -- 172.21.15.98:0/2844492091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ac6416e0a0 con 0x55ac641552f0 2024-09-19T08:02:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43dd7fa640 1 -- 172.21.15.98:0/2844492091 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43e4018ab0 con 0x55ac641552f0 2024-09-19T08:02:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43dd7fa640 1 -- 172.21.15.98:0/2844492091 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f43f0030560 con 0x55ac641552f0 2024-09-19T08:02:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43fe63c740 10 client.15036 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43fe63c740 10 client.15036 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-19T08:02:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43fe63c740 20 client.15036 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43fe63c740 1 --2- 172.21.15.98:0/2844492091 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55ac64170710 0x55ac64171b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43fe63c740 4 client.15036 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:02:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.613+0000 7f43fe63c740 1 -- 172.21.15.98:0/2844492091 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac640c8600 con 0x55ac64170710 2024-09-19T08:02:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.614+0000 7f43f97fe640 1 --2- 172.21.15.98:0/2844492091 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55ac64170710 0x55ac64171b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.614+0000 7f43f97fe640 1 --2- 172.21.15.98:0/2844492091 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55ac64170710 0x55ac64171b80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c5aaa1bff41b5cbf server_cookie=5b0883ed2bfa0f21 in_seq=0 out_seq=0 2024-09-19T08:02:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.614+0000 7f43dd7fa640 10 client.15036 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:02:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.615+0000 7f43dd7fa640 1 -- 172.21.15.98:0/2844492091 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ac640c8600 con 0x55ac64170710 2024-09-19T08:02:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.615+0000 7f43dd7fa640 10 client.15036 handle_command_reply: tid=0 2024-09-19T08:02:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.615+0000 7f43fe63c740 1 client.15036 shutdown 2024-09-19T08:02:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.615+0000 7f43fe63c740 1 -- 172.21.15.98:0/2844492091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac641552f0 msgr2=0x55ac6415c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.615+0000 7f43fe63c740 1 --2- 172.21.15.98:0/2844492091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac641552f0 0x55ac6415c7a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f43f0005cc0 tx=0x7f43f002f5d0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.615+0000 7f43fe63c740 1 -- 172.21.15.98:0/2844492091 shutdown_connections 2024-09-19T08:02:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.615+0000 7f43fe63c740 1 --2- 172.21.15.98:0/2844492091 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55ac64170710 0x55ac64171b80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.615+0000 7f43fe63c740 1 --2- 172.21.15.98:0/2844492091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac6415fbb0 0x55ac6415cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.615+0000 7f43fe63c740 1 --2- 172.21.15.98:0/2844492091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac641552f0 0x55ac6415c7a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.615+0000 7f43dd7fa640 0 client.15036 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:02:27.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.615+0000 7f43fe63c740 1 --2- 172.21.15.98:0/2844492091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641549d0 0x55ac6415f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.615+0000 7f43fe63c740 1 -- 172.21.15.98:0/2844492091 >> 172.21.15.98:0/2844492091 conn(0x55ac64150f60 msgr2=0x55ac64156c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:27.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.616+0000 7f43fe63c740 1 -- 172.21.15.98:0/2844492091 shutdown_connections 2024-09-19T08:02:27.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.616+0000 7f43fe63c740 1 -- 172.21.15.98:0/2844492091 wait complete. 2024-09-19T08:02:27.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.616+0000 7f43fe63c740 20 client.15036 trim_cache size 0 max 16384 2024-09-19T08:02:27.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.633+0000 7f43fe63c740 1 Processor -- start 2024-09-19T08:02:27.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.633+0000 7f43fe63c740 1 -- start start 2024-09-19T08:02:27.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43fe63c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641549d0 0x55ac64171b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43fe63c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641552f0 0x55ac6415e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43fe63c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64159a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43fe63c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ac6416fec0 con 0x55ac6415fbb0 2024-09-19T08:02:27.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43fe63c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ac64170590 con 0x55ac641549d0 2024-09-19T08:02:27.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43fe63c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ac64170c60 con 0x55ac641552f0 2024-09-19T08:02:27.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43f97fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43f97fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38258/0 (socket says 172.21.15.98:38258) 2024-09-19T08:02:27.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43f97fe640 1 -- 172.21.15.98:0/2796872061 learned_addr learned my addr 172.21.15.98:0/2796872061 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:27.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43f9fff640 1 --2- 172.21.15.98:0/2796872061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641552f0 0x55ac6415e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43f97fe640 1 -- 172.21.15.98:0/2796872061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641552f0 msgr2=0x55ac6415e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:27.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43f97fe640 1 --2- 172.21.15.98:0/2796872061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641552f0 0x55ac6415e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43f97fe640 1 -- 172.21.15.98:0/2796872061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641549d0 msgr2=0x55ac64171b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:27.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43f97fe640 1 --2- 172.21.15.98:0/2796872061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641549d0 0x55ac64171b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43f97fe640 1 -- 172.21.15.98:0/2796872061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac64150d90 con 0x55ac6415fbb0 2024-09-19T08:02:27.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43f97fe640 1 --2- 172.21.15.98:0/2796872061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64159a00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f43e4016e80 tx=0x7f43e40047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf60d2307da7d8a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:27.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43df7fe640 1 -- 172.21.15.98:0/2796872061 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e4018630 con 0x55ac6415fbb0 2024-09-19T08:02:27.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43df7fe640 1 -- 172.21.15.98:0/2796872061 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43e40187d0 con 0x55ac6415fbb0 2024-09-19T08:02:27.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.634+0000 7f43df7fe640 1 -- 172.21.15.98:0/2796872061 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e4026460 con 0x55ac6415fbb0 2024-09-19T08:02:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.635+0000 7f43fe63c740 1 -- 172.21.15.98:0/2796872061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 msgr2=0x55ac64159a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.635+0000 7f43fe63c740 1 --2- 172.21.15.98:0/2796872061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64159a00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f43e4016e80 tx=0x7f43e40047d0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.635+0000 7f43fe63c740 1 -- 172.21.15.98:0/2796872061 shutdown_connections 2024-09-19T08:02:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.635+0000 7f43fe63c740 1 --2- 172.21.15.98:0/2796872061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64159a00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.635+0000 7f43fe63c740 1 --2- 172.21.15.98:0/2796872061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641552f0 0x55ac6415e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.635+0000 7f43fe63c740 1 --2- 172.21.15.98:0/2796872061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641549d0 0x55ac64171b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.635+0000 7f43fe63c740 1 -- 172.21.15.98:0/2796872061 >> 172.21.15.98:0/2796872061 conn(0x55ac64150f60 msgr2=0x55ac6415def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.635+0000 7f43fe63c740 1 -- 172.21.15.98:0/2796872061 shutdown_connections 2024-09-19T08:02:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.635+0000 7f43fe63c740 1 -- 172.21.15.98:0/2796872061 wait complete. 2024-09-19T08:02:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.635+0000 7f43fe63c740 1 -- start start 2024-09-19T08:02:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.635+0000 7f43fe63c740 1 Processor -- start 2024-09-19T08:02:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43fe63c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641549d0 0x55ac6417a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43fe63c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641552f0 0x55ac641755d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43fe63c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64175b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43fe63c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ac64171a00 con 0x55ac6415fbb0 2024-09-19T08:02:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43fe63c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ac64171330 con 0x55ac641549d0 2024-09-19T08:02:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43fe63c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ac641720d0 con 0x55ac641552f0 2024-09-19T08:02:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43fe63c740 10 client.0 _ll_register_callbacks cb 0x7f43f0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43f97fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64175b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43f97fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64175b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38266/0 (socket says 172.21.15.98:38266) 2024-09-19T08:02:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43f97fe640 1 -- 172.21.15.98:0/1123940676 learned_addr learned my addr 172.21.15.98:0/1123940676 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43f9fff640 1 --2- 172.21.15.98:0/1123940676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641552f0 0x55ac641755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43f97fe640 1 --2- 172.21.15.98:0/1123940676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641549d0 0x55ac6417a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43f97fe640 1 -- 172.21.15.98:0/1123940676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641552f0 msgr2=0x55ac641755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43f97fe640 1 --2- 172.21.15.98:0/1123940676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641552f0 0x55ac641755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43f97fe640 1 -- 172.21.15.98:0/1123940676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641549d0 msgr2=0x55ac6417a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43f97fe640 1 --2- 172.21.15.98:0/1123940676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641549d0 0x55ac6417a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43f97fe640 1 -- 172.21.15.98:0/1123940676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac64176050 con 0x55ac6415fbb0 2024-09-19T08:02:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43f97fe640 1 --2- 172.21.15.98:0/1123940676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64175b10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f43e4016530 tx=0x7f43e4004640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494a8d64d5e89e53 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.636+0000 7f43dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.637+0000 7f43dd7fa640 1 -- 172.21.15.98:0/1123940676 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e4018430 con 0x55ac6415fbb0 2024-09-19T08:02:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.637+0000 7f43dd7fa640 1 -- 172.21.15.98:0/1123940676 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f43e40185d0 con 0x55ac6415fbb0 2024-09-19T08:02:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.637+0000 7f43fe63c740 1 -- 172.21.15.98:0/1123940676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43e40187d0 con 0x55ac6415fbb0 2024-09-19T08:02:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.637+0000 7f43dd7fa640 1 -- 172.21.15.98:0/1123940676 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f43e4017890 con 0x55ac6415fbb0 2024-09-19T08:02:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.637+0000 7f43fe63c740 10 client.15042 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.637+0000 7f43fe63c740 1 -- 172.21.15.98:0/1123940676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ac641daaa0 con 0x55ac6415fbb0 2024-09-19T08:02:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.637+0000 7f43dd7fa640 1 -- 172.21.15.98:0/1123940676 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43e40187d0 con 0x55ac6415fbb0 2024-09-19T08:02:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.637+0000 7f43dd7fa640 1 -- 172.21.15.98:0/1123940676 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f43e4012050 con 0x55ac6415fbb0 2024-09-19T08:02:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.637+0000 7f43fe63c740 10 client.15042 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.637+0000 7f43fe63c740 10 client.15042 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-19T08:02:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.637+0000 7f43fe63c740 20 client.15042 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.637+0000 7f43fe63c740 1 --2- 172.21.15.98:0/1123940676 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55ac641deee0 0x55ac641ddbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.637+0000 7f43fe63c740 4 client.15042 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:02:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.637+0000 7f43fe63c740 1 -- 172.21.15.98:0/1123940676 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "status"}) -- 0x55ac640c2b10 con 0x55ac641deee0 2024-09-19T08:02:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.638+0000 7f43f9fff640 1 --2- 172.21.15.98:0/1123940676 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55ac641deee0 0x55ac641ddbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.638+0000 7f43dd7fa640 10 client.15042 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:02:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.638+0000 7f43f9fff640 1 --2- 172.21.15.98:0/1123940676 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55ac641deee0 0x55ac641ddbf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=635e8b259a8a1783 server_cookie=c7da440b75eceb50 in_seq=0 out_seq=0 2024-09-19T08:02:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.638+0000 7f43dd7fa640 1 -- 172.21.15.98:0/1123940676 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ac640c2b10 con 0x55ac641deee0 2024-09-19T08:02:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.638+0000 7f43dd7fa640 10 client.15042 handle_command_reply: tid=0 2024-09-19T08:02:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.638+0000 7f43fe63c740 1 client.15042 shutdown 2024-09-19T08:02:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.638+0000 7f43fe63c740 1 -- 172.21.15.98:0/1123940676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 msgr2=0x55ac64175b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.638+0000 7f43fe63c740 1 --2- 172.21.15.98:0/1123940676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64175b10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f43e4016530 tx=0x7f43e4004640 comp rx=0 tx=0).stop 2024-09-19T08:02:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.638+0000 7f43fe63c740 1 -- 172.21.15.98:0/1123940676 shutdown_connections 2024-09-19T08:02:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.638+0000 7f43fe63c740 1 --2- 172.21.15.98:0/1123940676 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55ac641deee0 0x55ac641ddbf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.638+0000 7f43fe63c740 1 --2- 172.21.15.98:0/1123940676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac6415fbb0 0x55ac64175b10 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.638+0000 7f43fe63c740 1 --2- 172.21.15.98:0/1123940676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac641552f0 0x55ac641755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.638+0000 7f43dd7fa640 0 client.15042 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:02:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.638+0000 7f43fe63c740 1 --2- 172.21.15.98:0/1123940676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac641549d0 0x55ac6417a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fe63c740 1 -- 172.21.15.98:0/1123940676 >> 172.21.15.98:0/1123940676 conn(0x55ac64150f60 msgr2=0x55ac6415e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fe63c740 1 -- 172.21.15.98:0/1123940676 shutdown_connections 2024-09-19T08:02:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fe63c740 1 -- 172.21.15.98:0/1123940676 wait complete. 2024-09-19T08:02:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fe63c740 20 client.15042 trim_cache size 0 max 16384 2024-09-19T08:02:27.644 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:02:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:02:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2024-09-19T08:02:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24437, 2024-09-19T08:02:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:02:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:02:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:02:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 67.810846495999996, 2024-09-19T08:02:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 27, 2024-09-19T08:02:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 80, 2024-09-19T08:02:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 78, 2024-09-19T08:02:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 106.095738164 2024-09-19T08:02:27.645 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:02:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fc263640 1 -- 172.21.15.98:0/2501425801 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43e00a32d0 msgr2=0x7f43e00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fc263640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43e00a32d0 0x7f43e00a5790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0a44c0 tx=0x7f43ec06aac0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fc263640 1 -- 172.21.15.98:0/2501425801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f410c170 msgr2=0x7f43f419cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fc263640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f410c170 0x7f43f419cd80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00c970 tx=0x7f43ec00ce30 comp rx=0 tx=0).stop 2024-09-19T08:02:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fc263640 1 -- 172.21.15.98:0/2501425801 shutdown_connections 2024-09-19T08:02:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fc263640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f43e00a32d0 0x7f43e00a5790 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fc263640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f410c170 0x7f43f419cd80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fc263640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43f4103660 0x7f43f41989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fc263640 1 --2- 172.21.15.98:0/2501425801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43f4102d40 0x7f43f41984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fc263640 1 -- 172.21.15.98:0/2501425801 >> 172.21.15.98:0/2501425801 conn(0x7f43f40fc7a0 msgr2=0x7f43f40fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fc263640 1 -- 172.21.15.98:0/2501425801 shutdown_connections 2024-09-19T08:02:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:27.639+0000 7f43fc263640 1 -- 172.21.15.98:0/2501425801 wait complete. 2024-09-19T08:02:28.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-19T08:02:28.630 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:28.918 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:28 smithi098 ceph-mon[31333]: pgmap v311: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:28.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:28 smithi139 ceph-mon[34874]: pgmap v311: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:29.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.115+0000 7fc79c23a640 1 -- 172.21.15.98:0/1570421824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc79410c170 msgr2=0x7fc79410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:29.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.115+0000 7fc79c23a640 1 --2- 172.21.15.98:0/1570421824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc79410c170 0x7fc79410e610 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc78c009ca0 tx=0x7fc78c02a950 comp rx=0 tx=0).stop 2024-09-19T08:02:29.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.115+0000 7fc79c23a640 1 -- 172.21.15.98:0/1570421824 shutdown_connections 2024-09-19T08:02:29.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.115+0000 7fc79c23a640 1 --2- 172.21.15.98:0/1570421824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc79410c170 0x7fc79410e610 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.115+0000 7fc79c23a640 1 --2- 172.21.15.98:0/1570421824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc794103660 0x7fc79410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.115+0000 7fc79c23a640 1 --2- 172.21.15.98:0/1570421824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc794102d40 0x7fc794103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.115+0000 7fc79c23a640 1 -- 172.21.15.98:0/1570421824 >> 172.21.15.98:0/1570421824 conn(0x7fc7940fc7a0 msgr2=0x7fc7940febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.115+0000 7fc79c23a640 1 -- 172.21.15.98:0/1570421824 shutdown_connections 2024-09-19T08:02:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.115+0000 7fc79c23a640 1 -- 172.21.15.98:0/1570421824 wait complete. 2024-09-19T08:02:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.116+0000 7fc79c23a640 1 Processor -- start 2024-09-19T08:02:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.116+0000 7fc79c23a640 1 -- start start 2024-09-19T08:02:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.116+0000 7fc79c23a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc794102d40 0x7fc7941009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.116+0000 7fc79c23a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc794103660 0x7fc794100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.116+0000 7fc79c23a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc79410c170 0x7fc794101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.116+0000 7fc79c23a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc794112030 con 0x7fc79410c170 2024-09-19T08:02:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.116+0000 7fc79c23a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc794111eb0 con 0x7fc794102d40 2024-09-19T08:02:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.116+0000 7fc79c23a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc7941121b0 con 0x7fc794103660 2024-09-19T08:02:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.116+0000 7fc79a7d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc79410c170 0x7fc794101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.116+0000 7fc79a7d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc79410c170 0x7fc794101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38276/0 (socket says 172.21.15.98:38276) 2024-09-19T08:02:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.116+0000 7fc79a7d7640 1 -- 172.21.15.98:0/4233875540 learned_addr learned my addr 172.21.15.98:0/4233875540 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.116+0000 7fc7997d5640 1 --2- 172.21.15.98:0/4233875540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc794103660 0x7fc794100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.117+0000 7fc799fd6640 1 --2- 172.21.15.98:0/4233875540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc794102d40 0x7fc7941009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.117+0000 7fc79a7d7640 1 -- 172.21.15.98:0/4233875540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc794103660 msgr2=0x7fc794100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.117+0000 7fc79a7d7640 1 --2- 172.21.15.98:0/4233875540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc794103660 0x7fc794100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.117+0000 7fc79a7d7640 1 -- 172.21.15.98:0/4233875540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc794102d40 msgr2=0x7fc7941009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.117+0000 7fc79a7d7640 1 --2- 172.21.15.98:0/4233875540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc794102d40 0x7fc7941009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.117+0000 7fc79a7d7640 1 -- 172.21.15.98:0/4233875540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7941a6a00 con 0x7fc79410c170 2024-09-19T08:02:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.117+0000 7fc79a7d7640 1 --2- 172.21.15.98:0/4233875540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc79410c170 0x7fc794101420 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc78c006010 tx=0x7fc78c02f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.117+0000 7fc78affd640 1 -- 172.21.15.98:0/4233875540 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c034040 con 0x7fc79410c170 2024-09-19T08:02:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.117+0000 7fc79c23a640 1 -- 172.21.15.98:0/4233875540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7941a6c90 con 0x7fc79410c170 2024-09-19T08:02:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.117+0000 7fc78affd640 1 -- 172.21.15.98:0/4233875540 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc78c003e00 con 0x7fc79410c170 2024-09-19T08:02:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.117+0000 7fc78affd640 1 -- 172.21.15.98:0/4233875540 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0456e0 con 0x7fc79410c170 2024-09-19T08:02:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.117+0000 7fc79c23a640 1 -- 172.21.15.98:0/4233875540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7941a71c0 con 0x7fc79410c170 2024-09-19T08:02:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.119+0000 7fc78affd640 1 -- 172.21.15.98:0/4233875540 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc78c037020 con 0x7fc79410c170 2024-09-19T08:02:29.122 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:28 smithi162 ceph-mon[35260]: pgmap v311: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc79e613740 1 Processor -- start 2024-09-19T08:02:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc79e613740 1 -- start start 2024-09-19T08:02:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc78affd640 1 --2- 172.21.15.98:0/4233875540 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc7800a32d0 0x7fc7800a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc79e613740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae85db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc79e613740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae862f0 0x55a5dae90670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc79e613740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae90bb0 0x55a5dae92fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc79e613740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a5dadec090 con 0x55a5dae859d0 2024-09-19T08:02:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc79e613740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a5dad93a70 con 0x55a5dae90bb0 2024-09-19T08:02:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc79e613740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a5dade8050 con 0x55a5dae862f0 2024-09-19T08:02:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc7997d5640 1 --2- 172.21.15.98:0/4233875540 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc7800a32d0 0x7fc7800a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc78affd640 1 -- 172.21.15.98:0/4233875540 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fc78c0f0c20 con 0x7fc79410c170 2024-09-19T08:02:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc799fd6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae85db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc799fd6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae85db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38278/0 (socket says 172.21.15.98:38278) 2024-09-19T08:02:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc799fd6640 1 -- 172.21.15.98:0/40626450 learned_addr learned my addr 172.21.15.98:0/40626450 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.120+0000 7fc7997d5640 1 --2- 172.21.15.98:0/40626450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae862f0 0x55a5dae90670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.121+0000 7fc79a7d7640 1 --2- 172.21.15.98:0/40626450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae90bb0 0x55a5dae92fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.121+0000 7fc799fd6640 1 -- 172.21.15.98:0/40626450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae862f0 msgr2=0x55a5dae90670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.121+0000 7fc799fd6640 1 --2- 172.21.15.98:0/40626450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae862f0 0x55a5dae90670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.121+0000 7fc799fd6640 1 -- 172.21.15.98:0/40626450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae90bb0 msgr2=0x55a5dae92fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.121+0000 7fc799fd6640 1 --2- 172.21.15.98:0/40626450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae90bb0 0x55a5dae92fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.121+0000 7fc799fd6640 1 -- 172.21.15.98:0/40626450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5dada0130 con 0x55a5dae859d0 2024-09-19T08:02:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.121+0000 7fc79a7d7640 1 --2- 172.21.15.98:0/40626450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae90bb0 0x55a5dae92fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.121+0000 7fc7997d5640 1 --2- 172.21.15.98:0/4233875540 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc7800a32d0 0x7fc7800a5790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc794102430 tx=0x7fc78400adc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.121+0000 7fc799fd6640 1 --2- 172.21.15.98:0/40626450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae85db0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc79000b7a0 tx=0x7fc79000bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51acb316fbccdf3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.121+0000 7fc77ffff640 1 -- 172.21.15.98:0/40626450 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79000d950 con 0x55a5dae859d0 2024-09-19T08:02:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.121+0000 7fc77ffff640 1 -- 172.21.15.98:0/40626450 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc790019070 con 0x55a5dae859d0 2024-09-19T08:02:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.121+0000 7fc77ffff640 1 -- 172.21.15.98:0/40626450 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7900123e0 con 0x55a5dae859d0 2024-09-19T08:02:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.122+0000 7fc79e613740 1 -- 172.21.15.98:0/40626450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 msgr2=0x55a5dae85db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.122+0000 7fc79e613740 1 --2- 172.21.15.98:0/40626450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae85db0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc79000b7a0 tx=0x7fc79000bc60 comp rx=0 tx=0).stop 2024-09-19T08:02:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.122+0000 7fc79e613740 1 -- 172.21.15.98:0/40626450 shutdown_connections 2024-09-19T08:02:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.122+0000 7fc79e613740 1 --2- 172.21.15.98:0/40626450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae90bb0 0x55a5dae92fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.122+0000 7fc79e613740 1 --2- 172.21.15.98:0/40626450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae862f0 0x55a5dae90670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.122+0000 7fc79e613740 1 --2- 172.21.15.98:0/40626450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae85db0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.122+0000 7fc79e613740 1 -- 172.21.15.98:0/40626450 >> 172.21.15.98:0/40626450 conn(0x55a5dae81f60 msgr2=0x55a5dae82340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.122+0000 7fc79e613740 1 -- 172.21.15.98:0/40626450 shutdown_connections 2024-09-19T08:02:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.122+0000 7fc79e613740 1 -- 172.21.15.98:0/40626450 wait complete. 2024-09-19T08:02:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.122+0000 7fc79e613740 1 -- start start 2024-09-19T08:02:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.122+0000 7fc79e613740 1 Processor -- start 2024-09-19T08:02:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc79e613740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae859d0 0x55a5dae902c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc79e613740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae862f0 0x55a5dae8d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc79e613740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae90bb0 0x55a5dae8db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc79e613740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a5dae93780 con 0x55a5dae862f0 2024-09-19T08:02:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc79e613740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a5dada6f20 con 0x55a5dae859d0 2024-09-19T08:02:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc79e613740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a5dae93900 con 0x55a5dae90bb0 2024-09-19T08:02:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc79e613740 10 client.0 _ll_register_callbacks cb 0x55a5dad70030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc7997d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae862f0 0x55a5dae8d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc7997d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae862f0 0x55a5dae8d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38290/0 (socket says 172.21.15.98:38290) 2024-09-19T08:02:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc7997d5640 1 -- 172.21.15.98:0/185295812 learned_addr learned my addr 172.21.15.98:0/185295812 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc79a7d7640 1 --2- 172.21.15.98:0/185295812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae90bb0 0x55a5dae8db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc799fd6640 1 --2- 172.21.15.98:0/185295812 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae859d0 0x55a5dae902c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc7997d5640 1 -- 172.21.15.98:0/185295812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae90bb0 msgr2=0x55a5dae8db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc7997d5640 1 --2- 172.21.15.98:0/185295812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae90bb0 0x55a5dae8db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc7997d5640 1 -- 172.21.15.98:0/185295812 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae859d0 msgr2=0x55a5dae902c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc7997d5640 1 --2- 172.21.15.98:0/185295812 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae859d0 0x55a5dae902c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc7997d5640 1 -- 172.21.15.98:0/185295812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5dab5d2b0 con 0x55a5dae862f0 2024-09-19T08:02:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc7997d5640 1 --2- 172.21.15.98:0/185295812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae862f0 0x55a5dae8d610 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc784018c10 tx=0x7fc784006da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a001da81337f412 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.123+0000 7fc77d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.124+0000 7fc77d7fa640 1 -- 172.21.15.98:0/185295812 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc784018e40 con 0x55a5dae862f0 2024-09-19T08:02:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.124+0000 7fc77d7fa640 1 -- 172.21.15.98:0/185295812 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fc784020440 con 0x55a5dae862f0 2024-09-19T08:02:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.124+0000 7fc79e613740 1 -- 172.21.15.98:0/185295812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc790019070 con 0x55a5dae862f0 2024-09-19T08:02:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.124+0000 7fc77d7fa640 1 -- 172.21.15.98:0/185295812 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc7840059c0 con 0x55a5dae862f0 2024-09-19T08:02:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.124+0000 7fc79e613740 10 client.15054 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.124+0000 7fc79e613740 1 -- 172.21.15.98:0/185295812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a5dae9eea0 con 0x55a5dae862f0 2024-09-19T08:02:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.124+0000 7fc77d7fa640 1 -- 172.21.15.98:0/185295812 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc790019070 con 0x55a5dae862f0 2024-09-19T08:02:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.124+0000 7fc77d7fa640 1 -- 172.21.15.98:0/185295812 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc784015040 con 0x55a5dae862f0 2024-09-19T08:02:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.124+0000 7fc79e613740 10 client.15054 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.124+0000 7fc79e613740 10 client.15054 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-19T08:02:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.124+0000 7fc79e613740 20 client.15054 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.124+0000 7fc79e613740 1 --2- 172.21.15.98:0/185295812 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55a5daea1580 0x55a5daea29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.124+0000 7fc79e613740 4 client.15054 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:02:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.124+0000 7fc79e613740 1 -- 172.21.15.98:0/185295812 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5dadf9600 con 0x55a5daea1580 2024-09-19T08:02:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.125+0000 7fc799fd6640 1 --2- 172.21.15.98:0/185295812 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55a5daea1580 0x55a5daea29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.125+0000 7fc799fd6640 1 --2- 172.21.15.98:0/185295812 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55a5daea1580 0x55a5daea29f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c5ad0fb23311fab5 server_cookie=416ae8d01e0b05d3 in_seq=0 out_seq=0 2024-09-19T08:02:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.125+0000 7fc77d7fa640 10 client.15054 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:02:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc77d7fa640 1 -- 172.21.15.98:0/185295812 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a5dadf9600 con 0x55a5daea1580 2024-09-19T08:02:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc77d7fa640 10 client.15054 handle_command_reply: tid=0 2024-09-19T08:02:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc79e613740 1 client.15054 shutdown 2024-09-19T08:02:29.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc79e613740 1 -- 172.21.15.98:0/185295812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae862f0 msgr2=0x55a5dae8d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc79e613740 1 --2- 172.21.15.98:0/185295812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae862f0 0x55a5dae8d610 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc784018c10 tx=0x7fc784006da0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc79e613740 1 -- 172.21.15.98:0/185295812 shutdown_connections 2024-09-19T08:02:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc79e613740 1 --2- 172.21.15.98:0/185295812 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55a5daea1580 0x55a5daea29f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc79e613740 1 --2- 172.21.15.98:0/185295812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae90bb0 0x55a5dae8db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc79e613740 1 --2- 172.21.15.98:0/185295812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae862f0 0x55a5dae8d610 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc77d7fa640 0 client.15054 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:02:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc79e613740 1 --2- 172.21.15.98:0/185295812 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae859d0 0x55a5dae902c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc79e613740 1 -- 172.21.15.98:0/185295812 >> 172.21.15.98:0/185295812 conn(0x55a5dae81f60 msgr2=0x55a5dae87c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc79e613740 1 -- 172.21.15.98:0/185295812 shutdown_connections 2024-09-19T08:02:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc79e613740 1 -- 172.21.15.98:0/185295812 wait complete. 2024-09-19T08:02:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.126+0000 7fc79e613740 20 client.15054 trim_cache size 0 max 16384 2024-09-19T08:02:29.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc79e613740 1 Processor -- start 2024-09-19T08:02:29.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc79e613740 1 -- start start 2024-09-19T08:02:29.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc79e613740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae8e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc79e613740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae862f0 0x55a5dae8d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc79e613740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae90bb0 0x55a5dae8db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc79e613740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a5daea0d30 con 0x55a5dae859d0 2024-09-19T08:02:29.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc79e613740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a5daea1400 con 0x55a5dae862f0 2024-09-19T08:02:29.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc79e613740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a5daea1ad0 con 0x55a5dae90bb0 2024-09-19T08:02:29.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc799fd6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae8e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc799fd6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae8e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38306/0 (socket says 172.21.15.98:38306) 2024-09-19T08:02:29.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc799fd6640 1 -- 172.21.15.98:0/2612284287 learned_addr learned my addr 172.21.15.98:0/2612284287 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:29.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc7997d5640 1 --2- 172.21.15.98:0/2612284287 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae862f0 0x55a5dae8d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc799fd6640 1 -- 172.21.15.98:0/2612284287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae90bb0 msgr2=0x55a5dae8db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:29.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc799fd6640 1 --2- 172.21.15.98:0/2612284287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae90bb0 0x55a5dae8db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc799fd6640 1 -- 172.21.15.98:0/2612284287 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae862f0 msgr2=0x55a5dae8d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:29.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc799fd6640 1 --2- 172.21.15.98:0/2612284287 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae862f0 0x55a5dae8d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc799fd6640 1 -- 172.21.15.98:0/2612284287 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5dae81d90 con 0x55a5dae859d0 2024-09-19T08:02:29.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc799fd6640 1 --2- 172.21.15.98:0/2612284287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae8e230 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc79000b740 tx=0x7fc79000bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c44cc702880b2811 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:29.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.144+0000 7fc77f7fe640 1 -- 172.21.15.98:0/2612284287 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc790023070 con 0x55a5dae859d0 2024-09-19T08:02:29.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.145+0000 7fc77f7fe640 1 -- 172.21.15.98:0/2612284287 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc79002a070 con 0x55a5dae859d0 2024-09-19T08:02:29.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.145+0000 7fc77f7fe640 1 -- 172.21.15.98:0/2612284287 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7900247a0 con 0x55a5dae859d0 2024-09-19T08:02:29.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.145+0000 7fc79e613740 1 -- 172.21.15.98:0/2612284287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 msgr2=0x55a5dae8e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:29.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.145+0000 7fc79e613740 1 --2- 172.21.15.98:0/2612284287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae8e230 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc79000b740 tx=0x7fc79000bc60 comp rx=0 tx=0).stop 2024-09-19T08:02:29.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.145+0000 7fc79e613740 1 -- 172.21.15.98:0/2612284287 shutdown_connections 2024-09-19T08:02:29.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.145+0000 7fc79e613740 1 --2- 172.21.15.98:0/2612284287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae90bb0 0x55a5dae8db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.145+0000 7fc79e613740 1 --2- 172.21.15.98:0/2612284287 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae862f0 0x55a5dae8d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.145+0000 7fc79e613740 1 --2- 172.21.15.98:0/2612284287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae8e230 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.145+0000 7fc79e613740 1 -- 172.21.15.98:0/2612284287 >> 172.21.15.98:0/2612284287 conn(0x55a5dae81f60 msgr2=0x55a5dae7cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:29.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.145+0000 7fc79e613740 1 -- 172.21.15.98:0/2612284287 shutdown_connections 2024-09-19T08:02:29.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.145+0000 7fc79e613740 1 -- 172.21.15.98:0/2612284287 wait complete. 2024-09-19T08:02:29.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.145+0000 7fc79e613740 1 -- start start 2024-09-19T08:02:29.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc79e613740 1 Processor -- start 2024-09-19T08:02:29.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc79e613740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae91710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc79e613740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae862f0 0x55a5dae91c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc79e613740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae90bb0 0x55a5daea9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc79e613740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a5daea2870 con 0x55a5dae859d0 2024-09-19T08:02:29.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc79e613740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a5daea21a0 con 0x55a5dae862f0 2024-09-19T08:02:29.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc79e613740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a5daea2f40 con 0x55a5dae90bb0 2024-09-19T08:02:29.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc79e613740 10 client.0 _ll_register_callbacks cb 0x7fc78400ac10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:29.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc799fd6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae91710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc799fd6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae91710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38318/0 (socket says 172.21.15.98:38318) 2024-09-19T08:02:29.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc799fd6640 1 -- 172.21.15.98:0/3838851281 learned_addr learned my addr 172.21.15.98:0/3838851281 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:29.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc7997d5640 1 --2- 172.21.15.98:0/3838851281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae862f0 0x55a5dae91c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc79a7d7640 1 --2- 172.21.15.98:0/3838851281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae90bb0 0x55a5daea9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc799fd6640 1 -- 172.21.15.98:0/3838851281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae90bb0 msgr2=0x55a5daea9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:29.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc799fd6640 1 --2- 172.21.15.98:0/3838851281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae90bb0 0x55a5daea9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc799fd6640 1 -- 172.21.15.98:0/3838851281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae862f0 msgr2=0x55a5dae91c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:29.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc799fd6640 1 --2- 172.21.15.98:0/3838851281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae862f0 0x55a5dae91c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc799fd6640 1 -- 172.21.15.98:0/3838851281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5daeaa350 con 0x55a5dae859d0 2024-09-19T08:02:29.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.146+0000 7fc7997d5640 1 --2- 172.21.15.98:0/3838851281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae862f0 0x55a5dae91c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:29.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc799fd6640 1 --2- 172.21.15.98:0/3838851281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae91710 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc790024480 tx=0x7fc79000bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a02f818e9eaf65 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:29.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc77d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:29.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc77d7fa640 1 -- 172.21.15.98:0/3838851281 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc790033060 con 0x55a5dae859d0 2024-09-19T08:02:29.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc77d7fa640 1 -- 172.21.15.98:0/3838851281 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fc790014730 con 0x55a5dae859d0 2024-09-19T08:02:29.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc79e613740 1 -- 172.21.15.98:0/3838851281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc79002a070 con 0x55a5dae859d0 2024-09-19T08:02:29.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc77d7fa640 1 -- 172.21.15.98:0/3838851281 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc790023040 con 0x55a5dae859d0 2024-09-19T08:02:29.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc79e613740 10 client.15060 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:29.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc79e613740 1 -- 172.21.15.98:0/3838851281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a5daeab610 con 0x55a5dae859d0 2024-09-19T08:02:29.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc77d7fa640 1 -- 172.21.15.98:0/3838851281 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc79002a070 con 0x55a5dae859d0 2024-09-19T08:02:29.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc77d7fa640 1 -- 172.21.15.98:0/3838851281 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc790013070 con 0x55a5dae859d0 2024-09-19T08:02:29.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc79e613740 10 client.15060 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:29.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc79e613740 10 client.15060 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-19T08:02:29.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc79e613740 20 client.15060 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:29.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc79e613740 1 --2- 172.21.15.98:0/3838851281 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55a5daf0fc00 0x55a5daf0e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:29.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc79e613740 4 client.15060 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:02:29.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.147+0000 7fc79e613740 1 -- 172.21.15.98:0/3838851281 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "status"}) -- 0x55a5dadf3b10 con 0x55a5daf0fc00 2024-09-19T08:02:29.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc7997d5640 1 --2- 172.21.15.98:0/3838851281 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55a5daf0fc00 0x55a5daf0e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:29.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc7997d5640 1 --2- 172.21.15.98:0/3838851281 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55a5daf0fc00 0x55a5daf0e910 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=97fdf2bbe2a0c5cb server_cookie=67987fcbfb7c077a in_seq=0 out_seq=0 2024-09-19T08:02:29.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc77d7fa640 10 client.15060 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:02:29.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc77d7fa640 1 -- 172.21.15.98:0/3838851281 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55a5dadf3b10 con 0x55a5daf0fc00 2024-09-19T08:02:29.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc77d7fa640 10 client.15060 handle_command_reply: tid=0 2024-09-19T08:02:29.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc79e613740 1 client.15060 shutdown 2024-09-19T08:02:29.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc79e613740 1 -- 172.21.15.98:0/3838851281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 msgr2=0x55a5dae91710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:29.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc79e613740 1 --2- 172.21.15.98:0/3838851281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae91710 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc790024480 tx=0x7fc79000bfc0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc79e613740 1 -- 172.21.15.98:0/3838851281 shutdown_connections 2024-09-19T08:02:29.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc79e613740 1 --2- 172.21.15.98:0/3838851281 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55a5daf0fc00 0x55a5daf0e910 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc79e613740 1 --2- 172.21.15.98:0/3838851281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5dae90bb0 0x55a5daea9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc79e613740 1 --2- 172.21.15.98:0/3838851281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a5dae862f0 0x55a5dae91c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc77d7fa640 0 client.15060 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:02:29.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc79e613740 1 --2- 172.21.15.98:0/3838851281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a5dae859d0 0x55a5dae91710 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.148+0000 7fc79e613740 1 -- 172.21.15.98:0/3838851281 >> 172.21.15.98:0/3838851281 conn(0x55a5dae81f60 msgr2=0x55a5dae8d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:29.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79e613740 1 -- 172.21.15.98:0/3838851281 shutdown_connections 2024-09-19T08:02:29.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79e613740 1 -- 172.21.15.98:0/3838851281 wait complete. 2024-09-19T08:02:29.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79e613740 20 client.15060 trim_cache size 0 max 16384 2024-09-19T08:02:29.154 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:02:29.155 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:02:29.155 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2024-09-19T08:02:29.155 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24461, 2024-09-19T08:02:29.155 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:02:29.155 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:02:29.155 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:02:29.155 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 70.331956047999995, 2024-09-19T08:02:29.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 27, 2024-09-19T08:02:29.155 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 80, 2024-09-19T08:02:29.155 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 78, 2024-09-19T08:02:29.155 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 100.37845693200001 2024-09-19T08:02:29.155 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:02:29.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79c23a640 1 -- 172.21.15.98:0/4233875540 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc7800a32d0 msgr2=0x7fc7800a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:29.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79c23a640 1 --2- 172.21.15.98:0/4233875540 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc7800a32d0 0x7fc7800a5790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc794102430 tx=0x7fc78400adc0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79c23a640 1 -- 172.21.15.98:0/4233875540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc79410c170 msgr2=0x7fc794101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:29.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79c23a640 1 --2- 172.21.15.98:0/4233875540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc79410c170 0x7fc794101420 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc78c006010 tx=0x7fc78c02f710 comp rx=0 tx=0).stop 2024-09-19T08:02:29.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79c23a640 1 -- 172.21.15.98:0/4233875540 shutdown_connections 2024-09-19T08:02:29.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79c23a640 1 --2- 172.21.15.98:0/4233875540 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fc7800a32d0 0x7fc7800a5790 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79c23a640 1 --2- 172.21.15.98:0/4233875540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc79410c170 0x7fc794101420 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79c23a640 1 --2- 172.21.15.98:0/4233875540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc794103660 0x7fc794100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79c23a640 1 --2- 172.21.15.98:0/4233875540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc794102d40 0x7fc7941009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:29.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79c23a640 1 -- 172.21.15.98:0/4233875540 >> 172.21.15.98:0/4233875540 conn(0x7fc7940fc7a0 msgr2=0x7fc7940fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:29.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79c23a640 1 -- 172.21.15.98:0/4233875540 shutdown_connections 2024-09-19T08:02:29.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:29.149+0000 7fc79c23a640 1 -- 172.21.15.98:0/4233875540 wait complete. 2024-09-19T08:02:29.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-19T08:02:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.710+0000 7fdf33b55640 1 -- 172.21.15.98:0/1184511257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c106f10 msgr2=0x7fdf2c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.710+0000 7fdf33b55640 1 --2- 172.21.15.98:0/1184511257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c106f10 0x7fdf2c10ea90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c009900 tx=0x7fdf1c02a410 comp rx=0 tx=0).stop 2024-09-19T08:02:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.710+0000 7fdf33b55640 1 -- 172.21.15.98:0/1184511257 shutdown_connections 2024-09-19T08:02:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.710+0000 7fdf33b55640 1 --2- 172.21.15.98:0/1184511257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c106f10 0x7fdf2c10ea90 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.710+0000 7fdf33b55640 1 --2- 172.21.15.98:0/1184511257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf2c102790 0x7fdf2c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.710+0000 7fdf33b55640 1 --2- 172.21.15.98:0/1184511257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdf2c101de0 0x7fdf2c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.710+0000 7fdf33b55640 1 -- 172.21.15.98:0/1184511257 >> 172.21.15.98:0/1184511257 conn(0x7fdf2c0fd8c0 msgr2=0x7fdf2c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.710+0000 7fdf33b55640 1 -- 172.21.15.98:0/1184511257 shutdown_connections 2024-09-19T08:02:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.710+0000 7fdf33b55640 1 -- 172.21.15.98:0/1184511257 wait complete. 2024-09-19T08:02:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf33b55640 1 Processor -- start 2024-09-19T08:02:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf33b55640 1 -- start start 2024-09-19T08:02:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf33b55640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c101de0 0x7fdf2c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf33b55640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdf2c102790 0x7fdf2c198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf33b55640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf2c106f10 0x7fdf2c19cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf33b55640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2c111f90 con 0x7fdf2c101de0 2024-09-19T08:02:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf33b55640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2c111e10 con 0x7fdf2c102790 2024-09-19T08:02:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf33b55640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2c112110 con 0x7fdf2c106f10 2024-09-19T08:02:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf318f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c101de0 0x7fdf2c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf318f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c101de0 0x7fdf2c1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38328/0 (socket says 172.21.15.98:38328) 2024-09-19T08:02:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf318f1640 1 -- 172.21.15.98:0/1026321449 learned_addr learned my addr 172.21.15.98:0/1026321449 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf320f2640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf2c106f10 0x7fdf2c19cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf310f0640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdf2c102790 0x7fdf2c198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf318f1640 1 -- 172.21.15.98:0/1026321449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf2c106f10 msgr2=0x7fdf2c19cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf318f1640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf2c106f10 0x7fdf2c19cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf318f1640 1 -- 172.21.15.98:0/1026321449 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdf2c102790 msgr2=0x7fdf2c198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf318f1640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdf2c102790 0x7fdf2c198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf318f1640 1 -- 172.21.15.98:0/1026321449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf2c19d380 con 0x7fdf2c101de0 2024-09-19T08:02:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.711+0000 7fdf320f2640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf2c106f10 0x7fdf2c19cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.712+0000 7fdf318f1640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c101de0 0x7fdf2c1983d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdf2400ba50 tx=0x7fdf2400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.712+0000 7fdf2affd640 1 -- 172.21.15.98:0/1026321449 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf24019070 con 0x7fdf2c101de0 2024-09-19T08:02:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.712+0000 7fdf33b55640 1 -- 172.21.15.98:0/1026321449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf2c19d670 con 0x7fdf2c101de0 2024-09-19T08:02:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.712+0000 7fdf2affd640 1 -- 172.21.15.98:0/1026321449 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf2400cbe0 con 0x7fdf2c101de0 2024-09-19T08:02:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.712+0000 7fdf2affd640 1 -- 172.21.15.98:0/1026321449 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf240076e0 con 0x7fdf2c101de0 2024-09-19T08:02:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.712+0000 7fdf33b55640 1 -- 172.21.15.98:0/1026321449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf2c1a4f50 con 0x7fdf2c101de0 2024-09-19T08:02:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.714+0000 7fdf2affd640 1 -- 172.21.15.98:0/1026321449 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fdf2401c070 con 0x7fdf2c101de0 2024-09-19T08:02:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.714+0000 7fdf35f2e740 1 Processor -- start 2024-09-19T08:02:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf35f2e740 1 -- start start 2024-09-19T08:02:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf35f2e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87b9d0 0x55ab9c87bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf35f2e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87c2f0 0x55ab9c886670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf2affd640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdf140a32d0 0x7fdf140a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf35f2e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c888fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf35f2e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ab9c7e2090 con 0x55ab9c886bb0 2024-09-19T08:02:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf35f2e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ab9c789a70 con 0x55ab9c87b9d0 2024-09-19T08:02:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf35f2e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ab9c7de050 con 0x55ab9c87c2f0 2024-09-19T08:02:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf320f2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c888fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf320f2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c888fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38342/0 (socket says 172.21.15.98:38342) 2024-09-19T08:02:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf320f2640 1 -- 172.21.15.98:0/640531828 learned_addr learned my addr 172.21.15.98:0/640531828 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf310f0640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdf140a32d0 0x7fdf140a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf2affd640 1 -- 172.21.15.98:0/1026321449 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fdf240cefe0 con 0x7fdf2c101de0 2024-09-19T08:02:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf318f1640 1 --2- 172.21.15.98:0/640531828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87c2f0 0x55ab9c886670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf320f2640 1 --2- 172.21.15.98:0/640531828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87b9d0 0x55ab9c87bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf320f2640 1 -- 172.21.15.98:0/640531828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87c2f0 msgr2=0x55ab9c886670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf320f2640 1 --2- 172.21.15.98:0/640531828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87c2f0 0x55ab9c886670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf320f2640 1 -- 172.21.15.98:0/640531828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87b9d0 msgr2=0x55ab9c87bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf320f2640 1 --2- 172.21.15.98:0/640531828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87b9d0 0x55ab9c87bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.715+0000 7fdf320f2640 1 -- 172.21.15.98:0/640531828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab9c796130 con 0x55ab9c886bb0 2024-09-19T08:02:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.716+0000 7fdf320f2640 1 --2- 172.21.15.98:0/640531828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c888fa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c005940 tx=0x7fdf1c031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c31c809b7c101813 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.716+0000 7fdf310f0640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdf140a32d0 0x7fdf140a5790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdf20005e10 tx=0x7fdf20005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.716+0000 7fdf13fff640 1 -- 172.21.15.98:0/640531828 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c035050 con 0x55ab9c886bb0 2024-09-19T08:02:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.716+0000 7fdf13fff640 1 -- 172.21.15.98:0/640531828 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf1c041030 con 0x55ab9c886bb0 2024-09-19T08:02:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.716+0000 7fdf13fff640 1 -- 172.21.15.98:0/640531828 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c004a00 con 0x55ab9c886bb0 2024-09-19T08:02:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.717+0000 7fdf35f2e740 1 -- 172.21.15.98:0/640531828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 msgr2=0x55ab9c888fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.717+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/640531828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c888fa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c005940 tx=0x7fdf1c031400 comp rx=0 tx=0).stop 2024-09-19T08:02:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.717+0000 7fdf35f2e740 1 -- 172.21.15.98:0/640531828 shutdown_connections 2024-09-19T08:02:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.717+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/640531828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c888fa0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.717+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/640531828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87c2f0 0x55ab9c886670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.717+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/640531828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87b9d0 0x55ab9c87bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.717+0000 7fdf35f2e740 1 -- 172.21.15.98:0/640531828 >> 172.21.15.98:0/640531828 conn(0x55ab9c877f60 msgr2=0x55ab9c878340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.717+0000 7fdf35f2e740 1 -- 172.21.15.98:0/640531828 shutdown_connections 2024-09-19T08:02:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.717+0000 7fdf35f2e740 1 -- 172.21.15.98:0/640531828 wait complete. 2024-09-19T08:02:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.717+0000 7fdf35f2e740 1 -- start start 2024-09-19T08:02:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.717+0000 7fdf35f2e740 1 Processor -- start 2024-09-19T08:02:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.717+0000 7fdf35f2e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c87b9d0 0x55ab9c886450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.717+0000 7fdf35f2e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87c2f0 0x55ab9c8837a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf35f2e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c886bb0 0x55ab9c883ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf35f2e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ab9c889780 con 0x55ab9c87b9d0 2024-09-19T08:02:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf35f2e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ab9c79cf20 con 0x55ab9c87c2f0 2024-09-19T08:02:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf35f2e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ab9c889900 con 0x55ab9c886bb0 2024-09-19T08:02:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf35f2e740 10 client.0 _ll_register_callbacks cb 0x55ab9c766030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf318f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c87b9d0 0x55ab9c886450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf318f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c87b9d0 0x55ab9c886450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38358/0 (socket says 172.21.15.98:38358) 2024-09-19T08:02:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf318f1640 1 -- 172.21.15.98:0/11376598 learned_addr learned my addr 172.21.15.98:0/11376598 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf318f1640 1 -- 172.21.15.98:0/11376598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c886bb0 msgr2=0x55ab9c883ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf310f0640 1 --2- 172.21.15.98:0/11376598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87c2f0 0x55ab9c8837a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf318f1640 1 --2- 172.21.15.98:0/11376598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c886bb0 0x55ab9c883ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf318f1640 1 -- 172.21.15.98:0/11376598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87c2f0 msgr2=0x55ab9c8837a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf318f1640 1 --2- 172.21.15.98:0/11376598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87c2f0 0x55ab9c8837a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf318f1640 1 -- 172.21.15.98:0/11376598 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab9c5532b0 con 0x55ab9c87b9d0 2024-09-19T08:02:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf310f0640 1 --2- 172.21.15.98:0/11376598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87c2f0 0x55ab9c8837a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf318f1640 1 --2- 172.21.15.98:0/11376598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c87b9d0 0x55ab9c886450 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdf24097f70 tx=0x7fdf24097970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23eaddb11156504e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf117fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf117fa640 1 -- 172.21.15.98:0/11376598 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2409f950 con 0x55ab9c87b9d0 2024-09-19T08:02:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf117fa640 1 -- 172.21.15.98:0/11376598 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fdf240ccbd0 con 0x55ab9c87b9d0 2024-09-19T08:02:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf35f2e740 1 -- 172.21.15.98:0/11376598 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf1c041030 con 0x55ab9c87b9d0 2024-09-19T08:02:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf117fa640 1 -- 172.21.15.98:0/11376598 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdf240a4210 con 0x55ab9c87b9d0 2024-09-19T08:02:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf35f2e740 10 client.15072 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf35f2e740 1 -- 172.21.15.98:0/11376598 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ab9c895100 con 0x55ab9c87b9d0 2024-09-19T08:02:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.718+0000 7fdf117fa640 1 -- 172.21.15.98:0/11376598 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf1c041030 con 0x55ab9c87b9d0 2024-09-19T08:02:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.719+0000 7fdf117fa640 1 -- 172.21.15.98:0/11376598 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fdf240a3d80 con 0x55ab9c87b9d0 2024-09-19T08:02:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.719+0000 7fdf35f2e740 10 client.15072 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.719+0000 7fdf35f2e740 10 client.15072 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-19T08:02:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.719+0000 7fdf35f2e740 20 client.15072 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.719+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/11376598 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ab9c8977e0 0x55ab9c898c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.719+0000 7fdf35f2e740 4 client.15072 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:02:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.719+0000 7fdf35f2e740 1 -- 172.21.15.98:0/11376598 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab9c7ef600 con 0x55ab9c8977e0 2024-09-19T08:02:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.719+0000 7fdf310f0640 1 --2- 172.21.15.98:0/11376598 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ab9c8977e0 0x55ab9c898c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.719+0000 7fdf117fa640 10 client.15072 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:02:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.719+0000 7fdf310f0640 1 --2- 172.21.15.98:0/11376598 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ab9c8977e0 0x55ab9c898c50 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b96454b9ee5e771f server_cookie=411bfe37252eea48 in_seq=0 out_seq=0 2024-09-19T08:02:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.720+0000 7fdf117fa640 1 -- 172.21.15.98:0/11376598 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ab9c7ef600 con 0x55ab9c8977e0 2024-09-19T08:02:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.720+0000 7fdf117fa640 10 client.15072 handle_command_reply: tid=0 2024-09-19T08:02:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.720+0000 7fdf35f2e740 1 client.15072 shutdown 2024-09-19T08:02:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.720+0000 7fdf35f2e740 1 -- 172.21.15.98:0/11376598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c87b9d0 msgr2=0x55ab9c886450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.720+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/11376598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c87b9d0 0x55ab9c886450 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdf24097f70 tx=0x7fdf24097970 comp rx=0 tx=0).stop 2024-09-19T08:02:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.720+0000 7fdf35f2e740 1 -- 172.21.15.98:0/11376598 shutdown_connections 2024-09-19T08:02:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.720+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/11376598 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ab9c8977e0 0x55ab9c898c50 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.721+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/11376598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c886bb0 0x55ab9c883ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.721+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/11376598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87c2f0 0x55ab9c8837a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.721+0000 7fdf117fa640 0 client.15072 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:02:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.721+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/11376598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c87b9d0 0x55ab9c886450 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.721+0000 7fdf35f2e740 1 -- 172.21.15.98:0/11376598 >> 172.21.15.98:0/11376598 conn(0x55ab9c877f60 msgr2=0x55ab9c87dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.721+0000 7fdf35f2e740 1 -- 172.21.15.98:0/11376598 shutdown_connections 2024-09-19T08:02:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.721+0000 7fdf35f2e740 1 -- 172.21.15.98:0/11376598 wait complete. 2024-09-19T08:02:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.721+0000 7fdf35f2e740 20 client.15072 trim_cache size 0 max 16384 2024-09-19T08:02:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf35f2e740 1 Processor -- start 2024-09-19T08:02:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf35f2e740 1 -- start start 2024-09-19T08:02:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf35f2e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87b9d0 0x55ab9c883060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf35f2e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87c2f0 0x55ab9c880a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf35f2e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c880f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf35f2e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ab9c896f90 con 0x55ab9c886bb0 2024-09-19T08:02:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf320f2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c880f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf35f2e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ab9c897660 con 0x55ab9c87c2f0 2024-09-19T08:02:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf35f2e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ab9c897d30 con 0x55ab9c87b9d0 2024-09-19T08:02:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf320f2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c880f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38362/0 (socket says 172.21.15.98:38362) 2024-09-19T08:02:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf320f2640 1 -- 172.21.15.98:0/2618312489 learned_addr learned my addr 172.21.15.98:0/2618312489 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf310f0640 1 --2- 172.21.15.98:0/2618312489 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87c2f0 0x55ab9c880a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf318f1640 1 --2- 172.21.15.98:0/2618312489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87b9d0 0x55ab9c883060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf320f2640 1 -- 172.21.15.98:0/2618312489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87b9d0 msgr2=0x55ab9c883060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf320f2640 1 --2- 172.21.15.98:0/2618312489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87b9d0 0x55ab9c883060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf320f2640 1 -- 172.21.15.98:0/2618312489 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87c2f0 msgr2=0x55ab9c880a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.739+0000 7fdf320f2640 1 --2- 172.21.15.98:0/2618312489 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87c2f0 0x55ab9c880a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.740+0000 7fdf320f2640 1 -- 172.21.15.98:0/2618312489 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab9c896d90 con 0x55ab9c886bb0 2024-09-19T08:02:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.740+0000 7fdf310f0640 1 --2- 172.21.15.98:0/2618312489 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87c2f0 0x55ab9c880a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.740+0000 7fdf320f2640 1 --2- 172.21.15.98:0/2618312489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c880f60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c042040 tx=0x7fdf1c031b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b048d955bc48553a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.740+0000 7fdf137fe640 1 -- 172.21.15.98:0/2618312489 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c035040 con 0x55ab9c886bb0 2024-09-19T08:02:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.740+0000 7fdf137fe640 1 -- 172.21.15.98:0/2618312489 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf1c005510 con 0x55ab9c886bb0 2024-09-19T08:02:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.740+0000 7fdf137fe640 1 -- 172.21.15.98:0/2618312489 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c004d60 con 0x55ab9c886bb0 2024-09-19T08:02:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.740+0000 7fdf35f2e740 1 -- 172.21.15.98:0/2618312489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 msgr2=0x55ab9c880f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.740+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/2618312489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c880f60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c042040 tx=0x7fdf1c031b50 comp rx=0 tx=0).stop 2024-09-19T08:02:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 -- 172.21.15.98:0/2618312489 shutdown_connections 2024-09-19T08:02:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/2618312489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c880f60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/2618312489 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87c2f0 0x55ab9c880a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/2618312489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87b9d0 0x55ab9c883060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 -- 172.21.15.98:0/2618312489 >> 172.21.15.98:0/2618312489 conn(0x55ab9c877f60 msgr2=0x55ab9c883760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 -- 172.21.15.98:0/2618312489 shutdown_connections 2024-09-19T08:02:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 -- 172.21.15.98:0/2618312489 wait complete. 2024-09-19T08:02:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 -- start start 2024-09-19T08:02:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 Processor -- start 2024-09-19T08:02:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87b9d0 0x55ab9c89be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87c2f0 0x55ab9c89c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c89c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ab9c898ad0 con 0x55ab9c886bb0 2024-09-19T08:02:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ab9c898400 con 0x55ab9c87b9d0 2024-09-19T08:02:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ab9c8991a0 con 0x55ab9c87c2f0 2024-09-19T08:02:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.741+0000 7fdf35f2e740 10 client.0 _ll_register_callbacks cb 0x7fdf240cf2a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf320f2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c89c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf320f2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c89c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38378/0 (socket says 172.21.15.98:38378) 2024-09-19T08:02:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf320f2640 1 -- 172.21.15.98:0/1173526482 learned_addr learned my addr 172.21.15.98:0/1173526482 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf318f1640 1 --2- 172.21.15.98:0/1173526482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87b9d0 0x55ab9c89be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf320f2640 1 -- 172.21.15.98:0/1173526482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87c2f0 msgr2=0x55ab9c89c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf310f0640 1 --2- 172.21.15.98:0/1173526482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87c2f0 0x55ab9c89c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf320f2640 1 --2- 172.21.15.98:0/1173526482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87c2f0 0x55ab9c89c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf320f2640 1 -- 172.21.15.98:0/1173526482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87b9d0 msgr2=0x55ab9c89be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf320f2640 1 --2- 172.21.15.98:0/1173526482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87b9d0 0x55ab9c89be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf320f2640 1 -- 172.21.15.98:0/1173526482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab9c9000b0 con 0x55ab9c886bb0 2024-09-19T08:02:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf318f1640 1 --2- 172.21.15.98:0/1173526482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87b9d0 0x55ab9c89be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf320f2640 1 --2- 172.21.15.98:0/1173526482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c89c8e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c005940 tx=0x7fdf1c004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab2652ee1599fce server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf117fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf117fa640 1 -- 172.21.15.98:0/1173526482 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c035040 con 0x55ab9c886bb0 2024-09-19T08:02:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf117fa640 1 -- 172.21.15.98:0/1173526482 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fdf1c003a00 con 0x55ab9c886bb0 2024-09-19T08:02:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf35f2e740 1 -- 172.21.15.98:0/1173526482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf1c005510 con 0x55ab9c886bb0 2024-09-19T08:02:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf117fa640 1 -- 172.21.15.98:0/1173526482 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdf1c003c00 con 0x55ab9c886bb0 2024-09-19T08:02:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf35f2e740 10 client.15078 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.742+0000 7fdf35f2e740 1 -- 172.21.15.98:0/1173526482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ab9c901300 con 0x55ab9c886bb0 2024-09-19T08:02:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.743+0000 7fdf117fa640 1 -- 172.21.15.98:0/1173526482 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf1c005510 con 0x55ab9c886bb0 2024-09-19T08:02:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.743+0000 7fdf117fa640 1 -- 172.21.15.98:0/1173526482 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fdf1c046480 con 0x55ab9c886bb0 2024-09-19T08:02:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.743+0000 7fdf35f2e740 10 client.15078 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.743+0000 7fdf35f2e740 10 client.15078 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-19T08:02:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.743+0000 7fdf35f2e740 20 client.15078 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.743+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/1173526482 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ab9c905780 0x55ab9c904490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.743+0000 7fdf35f2e740 4 client.15078 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:02:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.743+0000 7fdf35f2e740 1 -- 172.21.15.98:0/1173526482 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "status"}) -- 0x55ab9c7e9b10 con 0x55ab9c905780 2024-09-19T08:02:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.743+0000 7fdf318f1640 1 --2- 172.21.15.98:0/1173526482 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ab9c905780 0x55ab9c904490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.743+0000 7fdf318f1640 1 --2- 172.21.15.98:0/1173526482 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ab9c905780 0x55ab9c904490 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bf38b9054d1f9a7 server_cookie=28b010fddb9813d in_seq=0 out_seq=0 2024-09-19T08:02:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.743+0000 7fdf117fa640 10 client.15078 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:02:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf117fa640 1 -- 172.21.15.98:0/1173526482 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ab9c7e9b10 con 0x55ab9c905780 2024-09-19T08:02:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf117fa640 10 client.15078 handle_command_reply: tid=0 2024-09-19T08:02:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf35f2e740 1 client.15078 shutdown 2024-09-19T08:02:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf35f2e740 1 -- 172.21.15.98:0/1173526482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 msgr2=0x55ab9c89c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/1173526482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c89c8e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c005940 tx=0x7fdf1c004840 comp rx=0 tx=0).stop 2024-09-19T08:02:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf35f2e740 1 -- 172.21.15.98:0/1173526482 shutdown_connections 2024-09-19T08:02:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/1173526482 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ab9c905780 0x55ab9c904490 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/1173526482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab9c886bb0 0x55ab9c89c8e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/1173526482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab9c87c2f0 0x55ab9c89c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf117fa640 0 client.15078 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:02:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf35f2e740 1 --2- 172.21.15.98:0/1173526482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ab9c87b9d0 0x55ab9c89be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf35f2e740 1 -- 172.21.15.98:0/1173526482 >> 172.21.15.98:0/1173526482 conn(0x55ab9c877f60 msgr2=0x55ab9c882070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf35f2e740 1 -- 172.21.15.98:0/1173526482 shutdown_connections 2024-09-19T08:02:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf35f2e740 1 -- 172.21.15.98:0/1173526482 wait complete. 2024-09-19T08:02:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.744+0000 7fdf35f2e740 20 client.15078 trim_cache size 0 max 16384 2024-09-19T08:02:30.762 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:02:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:02:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-09-19T08:02:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24485, 2024-09-19T08:02:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:02:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:02:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:02:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 78.962321462999995, 2024-09-19T08:02:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 27, 2024-09-19T08:02:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 80, 2024-09-19T08:02:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 78, 2024-09-19T08:02:30.763 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 90.354552515999998 2024-09-19T08:02:30.763 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:02:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.745+0000 7fdf33b55640 1 -- 172.21.15.98:0/1026321449 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdf140a32d0 msgr2=0x7fdf140a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.745+0000 7fdf33b55640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdf140a32d0 0x7fdf140a5790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdf20005e10 tx=0x7fdf20005da0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.745+0000 7fdf33b55640 1 -- 172.21.15.98:0/1026321449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c101de0 msgr2=0x7fdf2c1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.745+0000 7fdf33b55640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c101de0 0x7fdf2c1983d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdf2400ba50 tx=0x7fdf2400bf10 comp rx=0 tx=0).stop 2024-09-19T08:02:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.745+0000 7fdf33b55640 1 -- 172.21.15.98:0/1026321449 shutdown_connections 2024-09-19T08:02:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.745+0000 7fdf33b55640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fdf140a32d0 0x7fdf140a5790 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.745+0000 7fdf33b55640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf2c106f10 0x7fdf2c19cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.745+0000 7fdf33b55640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdf2c102790 0x7fdf2c198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.745+0000 7fdf33b55640 1 --2- 172.21.15.98:0/1026321449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c101de0 0x7fdf2c1983d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.745+0000 7fdf33b55640 1 -- 172.21.15.98:0/1026321449 >> 172.21.15.98:0/1026321449 conn(0x7fdf2c0fd8c0 msgr2=0x7fdf2c0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.745+0000 7fdf33b55640 1 -- 172.21.15.98:0/1026321449 shutdown_connections 2024-09-19T08:02:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:30.745+0000 7fdf33b55640 1 -- 172.21.15.98:0/1026321449 wait complete. 2024-09-19T08:02:30.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:30 smithi139 ceph-mon[34874]: pgmap v312: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:30.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:30 smithi098 ceph-mon[31333]: pgmap v312: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:31.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:30 smithi162 ceph-mon[35260]: pgmap v312: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:31.411 INFO:tasks.cephadm:Setup complete, yielding 2024-09-19T08:02:31.411 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-19T08:02:31.421 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-09-19T08:02:31.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- bash -c 'ceph orch status' 2024-09-19T08:02:31.665 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:32.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.109+0000 7f4c3fe27640 1 -- 172.21.15.98:0/4006181770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c38102700 msgr2=0x7f4c38104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:32.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.109+0000 7f4c3fe27640 1 --2- 172.21.15.98:0/4006181770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c38102700 0x7f4c38104af0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4c28009900 tx=0x7f4c2802a410 comp rx=0 tx=0).stop 2024-09-19T08:02:32.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.109+0000 7f4c3fe27640 1 -- 172.21.15.98:0/4006181770 shutdown_connections 2024-09-19T08:02:32.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.109+0000 7f4c3fe27640 1 --2- 172.21.15.98:0/4006181770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c38107960 0x7f4c38109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:32.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.109+0000 7f4c3fe27640 1 --2- 172.21.15.98:0/4006181770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c38105030 0x7f4c38107420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:32.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.109+0000 7f4c3fe27640 1 --2- 172.21.15.98:0/4006181770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c38102700 0x7f4c38104af0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:32.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.109+0000 7f4c3fe27640 1 -- 172.21.15.98:0/4006181770 >> 172.21.15.98:0/4006181770 conn(0x7f4c380fc280 msgr2=0x7f4c380fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:32.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.109+0000 7f4c3fe27640 1 -- 172.21.15.98:0/4006181770 shutdown_connections 2024-09-19T08:02:32.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.109+0000 7f4c3fe27640 1 -- 172.21.15.98:0/4006181770 wait complete. 2024-09-19T08:02:32.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.110+0000 7f4c3fe27640 1 Processor -- start 2024-09-19T08:02:32.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.110+0000 7f4c3fe27640 1 -- start start 2024-09-19T08:02:32.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.110+0000 7f4c3fe27640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c38102700 0x7f4c38195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:32.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.110+0000 7f4c3fe27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c38105030 0x7f4c38196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:32.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.110+0000 7f4c3fe27640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c38107960 0x7f4c381a5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:32.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.110+0000 7f4c3fe27640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3810d900 con 0x7f4c38107960 2024-09-19T08:02:32.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.110+0000 7f4c3fe27640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3810d780 con 0x7f4c38102700 2024-09-19T08:02:32.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.110+0000 7f4c3fe27640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3810da80 con 0x7f4c38105030 2024-09-19T08:02:32.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.110+0000 7f4c3e3c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c38107960 0x7f4c381a5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:32.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.110+0000 7f4c3e3c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c38107960 0x7f4c381a5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38400/0 (socket says 172.21.15.98:38400) 2024-09-19T08:02:32.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.110+0000 7f4c3e3c4640 1 -- 172.21.15.98:0/1152479920 learned_addr learned my addr 172.21.15.98:0/1152479920 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:32.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.111+0000 7f4c3d3c2640 1 --2- 172.21.15.98:0/1152479920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c38105030 0x7f4c38196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:32.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.111+0000 7f4c3dbc3640 1 --2- 172.21.15.98:0/1152479920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c38102700 0x7f4c38195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:32.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.111+0000 7f4c3e3c4640 1 -- 172.21.15.98:0/1152479920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c38105030 msgr2=0x7f4c38196300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:32.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.111+0000 7f4c3e3c4640 1 --2- 172.21.15.98:0/1152479920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c38105030 0x7f4c38196300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:32.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.111+0000 7f4c3e3c4640 1 -- 172.21.15.98:0/1152479920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c38102700 msgr2=0x7f4c38195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:32.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.111+0000 7f4c3e3c4640 1 --2- 172.21.15.98:0/1152479920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c38102700 0x7f4c38195dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:32.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.111+0000 7f4c3e3c4640 1 -- 172.21.15.98:0/1152479920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c381a61f0 con 0x7f4c38107960 2024-09-19T08:02:32.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.111+0000 7f4c3e3c4640 1 --2- 172.21.15.98:0/1152479920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c38107960 0x7f4c381a5c30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000b970 tx=0x7f4c3000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:32.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.111+0000 7f4c36ffd640 1 -- 172.21.15.98:0/1152479920 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c30019070 con 0x7f4c38107960 2024-09-19T08:02:32.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.111+0000 7f4c36ffd640 1 -- 172.21.15.98:0/1152479920 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c3000cb40 con 0x7f4c38107960 2024-09-19T08:02:32.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.111+0000 7f4c36ffd640 1 -- 172.21.15.98:0/1152479920 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c30007650 con 0x7f4c38107960 2024-09-19T08:02:32.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.112+0000 7f4c3fe27640 1 -- 172.21.15.98:0/1152479920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c381a64e0 con 0x7f4c38107960 2024-09-19T08:02:32.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.112+0000 7f4c3fe27640 1 -- 172.21.15.98:0/1152479920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c381a6a50 con 0x7f4c38107960 2024-09-19T08:02:32.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.112+0000 7f4c3fe27640 1 -- 172.21.15.98:0/1152479920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c38109450 con 0x7f4c38107960 2024-09-19T08:02:32.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.114+0000 7f4c36ffd640 1 -- 172.21.15.98:0/1152479920 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4c3000ccf0 con 0x7f4c38107960 2024-09-19T08:02:32.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.114+0000 7f4c36ffd640 1 --2- 172.21.15.98:0/1152479920 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4c200a32d0 0x7f4c200a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:32.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.115+0000 7f4c3dbc3640 1 --2- 172.21.15.98:0/1152479920 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4c200a32d0 0x7f4c200a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:32.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.115+0000 7f4c36ffd640 1 -- 172.21.15.98:0/1152479920 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f4c300cf440 con 0x7f4c38107960 2024-09-19T08:02:32.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.115+0000 7f4c3dbc3640 1 --2- 172.21.15.98:0/1152479920 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4c200a32d0 0x7f4c200a5790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4c28000c00 tx=0x7f4c28031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:32.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.119+0000 7f4c36ffd640 1 -- 172.21.15.98:0/1152479920 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f4c3009c120 con 0x7f4c38107960 2024-09-19T08:02:32.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.324+0000 7f4c3fe27640 1 -- 172.21.15.98:0/1152479920 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f4c38062630 con 0x7f4c200a32d0 2024-09-19T08:02:32.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.326+0000 7f4c36ffd640 1 -- 172.21.15.98:0/1152479920 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f4c38062630 con 0x7f4c200a32d0 2024-09-19T08:02:32.326 INFO:teuthology.orchestra.run.smithi098.stdout:Backend: cephadm 2024-09-19T08:02:32.326 INFO:teuthology.orchestra.run.smithi098.stdout:Available: Yes 2024-09-19T08:02:32.326 INFO:teuthology.orchestra.run.smithi098.stdout:Paused: No 2024-09-19T08:02:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.328+0000 7f4c3fe27640 1 -- 172.21.15.98:0/1152479920 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4c200a32d0 msgr2=0x7f4c200a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.328+0000 7f4c3fe27640 1 --2- 172.21.15.98:0/1152479920 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4c200a32d0 0x7f4c200a5790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4c28000c00 tx=0x7f4c28031400 comp rx=0 tx=0).stop 2024-09-19T08:02:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.328+0000 7f4c3fe27640 1 -- 172.21.15.98:0/1152479920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c38107960 msgr2=0x7f4c381a5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.328+0000 7f4c3fe27640 1 --2- 172.21.15.98:0/1152479920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c38107960 0x7f4c381a5c30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000b970 tx=0x7f4c3000be30 comp rx=0 tx=0).stop 2024-09-19T08:02:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.328+0000 7f4c3fe27640 1 -- 172.21.15.98:0/1152479920 shutdown_connections 2024-09-19T08:02:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.328+0000 7f4c3fe27640 1 --2- 172.21.15.98:0/1152479920 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4c200a32d0 0x7f4c200a5790 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.328+0000 7f4c3fe27640 1 --2- 172.21.15.98:0/1152479920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c38107960 0x7f4c381a5c30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.328+0000 7f4c3fe27640 1 --2- 172.21.15.98:0/1152479920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c38105030 0x7f4c38196300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.328+0000 7f4c3fe27640 1 --2- 172.21.15.98:0/1152479920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c38102700 0x7f4c38195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.328+0000 7f4c3fe27640 1 -- 172.21.15.98:0/1152479920 >> 172.21.15.98:0/1152479920 conn(0x7f4c380fc280 msgr2=0x7f4c38101dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.328+0000 7f4c3fe27640 1 -- 172.21.15.98:0/1152479920 shutdown_connections 2024-09-19T08:02:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:32.328+0000 7f4c3fe27640 1 -- 172.21.15.98:0/1152479920 wait complete. 2024-09-19T08:02:32.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:32 smithi098 ceph-mon[31333]: pgmap v313: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:32.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:32 smithi139 ceph-mon[34874]: pgmap v313: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:33.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ps' 2024-09-19T08:02:33.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:32 smithi162 ceph-mon[35260]: pgmap v313: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.694+0000 7f553f9b9640 1 -- 172.21.15.98:0/630580852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538101d10 msgr2=0x7f5538102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.694+0000 7f553f9b9640 1 --2- 172.21.15.98:0/630580852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538101d10 0x7f5538102190 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5528009900 tx=0x7f552802a410 comp rx=0 tx=0).stop 2024-09-19T08:02:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.694+0000 7f553f9b9640 1 -- 172.21.15.98:0/630580852 shutdown_connections 2024-09-19T08:02:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.694+0000 7f553f9b9640 1 --2- 172.21.15.98:0/630580852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55381026d0 0x7f5538108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.694+0000 7f553f9b9640 1 --2- 172.21.15.98:0/630580852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538101d10 0x7f5538102190 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.694+0000 7f553f9b9640 1 --2- 172.21.15.98:0/630580852 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5538100b10 0x7f5538100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.694+0000 7f553f9b9640 1 -- 172.21.15.98:0/630580852 >> 172.21.15.98:0/630580852 conn(0x7f55380fc2a0 msgr2=0x7f55380fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.694+0000 7f553f9b9640 1 -- 172.21.15.98:0/630580852 shutdown_connections 2024-09-19T08:02:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.694+0000 7f553f9b9640 1 -- 172.21.15.98:0/630580852 wait complete. 2024-09-19T08:02:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553f9b9640 1 Processor -- start 2024-09-19T08:02:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553f9b9640 1 -- start start 2024-09-19T08:02:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553f9b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5538100b10 0x7f553806b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553f9b9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538101d10 0x7f553819cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553f9b9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55381026d0 0x7f553819f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553f9b9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f553810c7f0 con 0x7f5538101d10 2024-09-19T08:02:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553f9b9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f553810c670 con 0x7f55381026d0 2024-09-19T08:02:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553f9b9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f553810c970 con 0x7f5538100b10 2024-09-19T08:02:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553cf54640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538101d10 0x7f553819cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553cf54640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538101d10 0x7f553819cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38414/0 (socket says 172.21.15.98:38414) 2024-09-19T08:02:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553cf54640 1 -- 172.21.15.98:0/1208955272 learned_addr learned my addr 172.21.15.98:0/1208955272 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553d755640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5538100b10 0x7f553806b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553df56640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55381026d0 0x7f553819f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553cf54640 1 -- 172.21.15.98:0/1208955272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5538100b10 msgr2=0x7f553806b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553cf54640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5538100b10 0x7f553806b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553cf54640 1 -- 172.21.15.98:0/1208955272 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55381026d0 msgr2=0x7f553819f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553cf54640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55381026d0 0x7f553819f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553cf54640 1 -- 172.21.15.98:0/1208955272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f553819fb10 con 0x7f5538101d10 2024-09-19T08:02:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553d755640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5538100b10 0x7f553806b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553df56640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55381026d0 0x7f553819f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.695+0000 7f553cf54640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538101d10 0x7f553819cc00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5528009230 tx=0x7f5528004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.696+0000 7f552e7fc640 1 -- 172.21.15.98:0/1208955272 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552803f030 con 0x7f5538101d10 2024-09-19T08:02:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.696+0000 7f552e7fc640 1 -- 172.21.15.98:0/1208955272 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5528003ce0 con 0x7f5538101d10 2024-09-19T08:02:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.696+0000 7f552e7fc640 1 -- 172.21.15.98:0/1208955272 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5528035070 con 0x7f5538101d10 2024-09-19T08:02:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.696+0000 7f553f9b9640 1 -- 172.21.15.98:0/1208955272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f553819fda0 con 0x7f5538101d10 2024-09-19T08:02:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.696+0000 7f553f9b9640 1 -- 172.21.15.98:0/1208955272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55381a02a0 con 0x7f5538101d10 2024-09-19T08:02:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.697+0000 7f553f9b9640 1 -- 172.21.15.98:0/1208955272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5538107e40 con 0x7f5538101d10 2024-09-19T08:02:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.698+0000 7f552e7fc640 1 -- 172.21.15.98:0/1208955272 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f5528042030 con 0x7f5538101d10 2024-09-19T08:02:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.698+0000 7f552e7fc640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f55240a32d0 0x7f55240a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.698+0000 7f553d755640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f55240a32d0 0x7f55240a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.698+0000 7f552e7fc640 1 -- 172.21.15.98:0/1208955272 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f55280f02b0 con 0x7f5538101d10 2024-09-19T08:02:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.699+0000 7f553d755640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f55240a32d0 0x7f55240a5790 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f55340058c0 tx=0x7f5534005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.703+0000 7f552e7fc640 1 -- 172.21.15.98:0/1208955272 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f55280bce80 con 0x7f5538101d10 2024-09-19T08:02:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.880+0000 7f553f9b9640 1 -- 172.21.15.98:0/1208955272 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f5538062630 con 0x7f55240a32d0 2024-09-19T08:02:33.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.891+0000 7f552e7fc640 1 -- 172.21.15.98:0/1208955272 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4154 (secure 0 0 0) 0x7f5538062630 con 0x7f55240a32d0 2024-09-19T08:02:33.891 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-19T08:02:33.892 INFO:teuthology.orchestra.run.smithi098.stdout:mds.a smithi098 running (97s) 21s ago 95s 16.9M - 19.1.1-361-g2f599a46 d2baa855c067 760d32eec551 2024-09-19T08:02:33.892 INFO:teuthology.orchestra.run.smithi098.stdout:mds.b smithi139 running (93s) 27s ago 93s 26.7M - 19.1.1-361-g2f599a46 d2baa855c067 b05a28ba4813 2024-09-19T08:02:33.892 INFO:teuthology.orchestra.run.smithi098.stdout:mds.c smithi162 running (107s) 27s ago 107s 18.3M - 19.1.1-361-g2f599a46 d2baa855c067 60526bf80aba 2024-09-19T08:02:33.892 INFO:teuthology.orchestra.run.smithi098.stdout:mds.d smithi098 running (115s) 21s ago 114s 15.9M - 19.1.1-361-g2f599a46 d2baa855c067 0f7fe9582b6a 2024-09-19T08:02:33.892 INFO:teuthology.orchestra.run.smithi098.stdout:mds.e smithi139 running (112s) 27s ago 112s 18.0M - 19.1.1-361-g2f599a46 d2baa855c067 e6725063e110 2024-09-19T08:02:33.892 INFO:teuthology.orchestra.run.smithi098.stdout:mds.f smithi162 running (110s) 27s ago 109s 18.5M - 19.1.1-361-g2f599a46 d2baa855c067 8096b91bd982 2024-09-19T08:02:33.892 INFO:teuthology.orchestra.run.smithi098.stdout:mds.g smithi098 running (99s) 21s ago 99s 18.4M - 19.1.1-361-g2f599a46 d2baa855c067 8009a4d16a39 2024-09-19T08:02:33.892 INFO:teuthology.orchestra.run.smithi098.stdout:mds.h smithi139 running (105s) 27s ago 105s 19.5M - 19.1.1-361-g2f599a46 d2baa855c067 438b5988f0ec 2024-09-19T08:02:33.892 INFO:teuthology.orchestra.run.smithi098.stdout:mds.i smithi162 running (102s) 27s ago 102s 19.4M - 19.1.1-361-g2f599a46 d2baa855c067 a360c89cc7ff 2024-09-19T08:02:33.892 INFO:teuthology.orchestra.run.smithi098.stdout:mds.j smithi098 running (118s) 21s ago 118s 14.4M - 19.1.1-361-g2f599a46 d2baa855c067 d4b63495349a 2024-09-19T08:02:33.892 INFO:teuthology.orchestra.run.smithi098.stdout:mds.k smithi139 running (2m) 27s ago 2m 16.7M - 19.1.1-361-g2f599a46 d2baa855c067 6e318dfb5097 2024-09-19T08:02:33.892 INFO:teuthology.orchestra.run.smithi098.stdout:mds.l smithi162 running (2m) 27s ago 2m 18.0M - 19.1.1-361-g2f599a46 d2baa855c067 8d291b9421c5 2024-09-19T08:02:33.893 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi098 *:9283,8765 running (8m) 21s ago 8m 508M - 19.1.1-361-g2f599a46 d2baa855c067 9a5fff8af5c4 2024-09-19T08:02:33.893 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi139 *:8443,8765 running (7m) 27s ago 7m 443M - 19.1.1-361-g2f599a46 d2baa855c067 dcdc610cf1f4 2024-09-19T08:02:33.893 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.z smithi162 *:8443,8765 running (7m) 27s ago 7m 444M - 19.1.1-361-g2f599a46 d2baa855c067 cf1f1c2f4f8e 2024-09-19T08:02:33.893 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (8m) 21s ago 8m 50.5M 2048M 19.1.1-361-g2f599a46 d2baa855c067 44a037c430ef 2024-09-19T08:02:33.893 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi139 running (7m) 27s ago 7m 40.6M 2048M 19.1.1-361-g2f599a46 d2baa855c067 f39ae79d7bf0 2024-09-19T08:02:33.893 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi162 running (7m) 27s ago 7m 40.2M 2048M 19.1.1-361-g2f599a46 d2baa855c067 500b1c467f92 2024-09-19T08:02:33.893 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (6m) 21s ago 6m 56.3M 4272M 19.1.1-361-g2f599a46 d2baa855c067 128437643d5d 2024-09-19T08:02:33.893 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi139 running (6m) 27s ago 6m 59.3M 4272M 19.1.1-361-g2f599a46 d2baa855c067 1ff1cc20f0b9 2024-09-19T08:02:33.893 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi162 running (6m) 27s ago 6m 50.0M 4272M 19.1.1-361-g2f599a46 d2baa855c067 67d77ca425f0 2024-09-19T08:02:33.893 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (5m) 21s ago 5m 53.7M 4272M 19.1.1-361-g2f599a46 d2baa855c067 914ef2286c2e 2024-09-19T08:02:33.893 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi139 running (5m) 27s ago 5m 53.0M 4272M 19.1.1-361-g2f599a46 d2baa855c067 3ea83d98aff6 2024-09-19T08:02:33.893 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi162 running (4m) 27s ago 4m 47.4M 4272M 19.1.1-361-g2f599a46 d2baa855c067 f5628ebfd6be 2024-09-19T08:02:33.894 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi098 running (4m) 21s ago 4m 53.7M 4272M 19.1.1-361-g2f599a46 d2baa855c067 ab8411c2601c 2024-09-19T08:02:33.894 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi139 running (3m) 27s ago 3m 47.0M 4272M 19.1.1-361-g2f599a46 d2baa855c067 60c68ec52a2a 2024-09-19T08:02:33.894 INFO:teuthology.orchestra.run.smithi098.stdout:osd.8 smithi162 running (3m) 27s ago 3m 46.2M 4272M 19.1.1-361-g2f599a46 d2baa855c067 99abe786e3cf 2024-09-19T08:02:33.894 INFO:teuthology.orchestra.run.smithi098.stdout:osd.9 smithi098 running (3m) 21s ago 3m 50.9M 4272M 19.1.1-361-g2f599a46 d2baa855c067 06d5f705b6d5 2024-09-19T08:02:33.894 INFO:teuthology.orchestra.run.smithi098.stdout:osd.10 smithi139 running (2m) 27s ago 2m 57.9M 4272M 19.1.1-361-g2f599a46 d2baa855c067 199f7f0b9306 2024-09-19T08:02:33.894 INFO:teuthology.orchestra.run.smithi098.stdout:osd.11 smithi162 running (2m) 27s ago 2m 50.5M 4272M 19.1.1-361-g2f599a46 d2baa855c067 709b4600fa25 2024-09-19T08:02:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.893+0000 7f553f9b9640 1 -- 172.21.15.98:0/1208955272 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f55240a32d0 msgr2=0x7f55240a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.893+0000 7f553f9b9640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f55240a32d0 0x7f55240a5790 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f55340058c0 tx=0x7f5534005850 comp rx=0 tx=0).stop 2024-09-19T08:02:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.893+0000 7f553f9b9640 1 -- 172.21.15.98:0/1208955272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538101d10 msgr2=0x7f553819cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.893+0000 7f553f9b9640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538101d10 0x7f553819cc00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5528009230 tx=0x7f5528004430 comp rx=0 tx=0).stop 2024-09-19T08:02:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.894+0000 7f553f9b9640 1 -- 172.21.15.98:0/1208955272 shutdown_connections 2024-09-19T08:02:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.894+0000 7f553f9b9640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f55240a32d0 0x7f55240a5790 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.894+0000 7f553f9b9640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55381026d0 0x7f553819f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.894+0000 7f553f9b9640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538101d10 0x7f553819cc00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.894+0000 7f553f9b9640 1 --2- 172.21.15.98:0/1208955272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5538100b10 0x7f553806b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.894+0000 7f553f9b9640 1 -- 172.21.15.98:0/1208955272 >> 172.21.15.98:0/1208955272 conn(0x7f55380fc2a0 msgr2=0x7f5538105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.894+0000 7f553f9b9640 1 -- 172.21.15.98:0/1208955272 shutdown_connections 2024-09-19T08:02:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:33.894+0000 7f553f9b9640 1 -- 172.21.15.98:0/1208955272 wait complete. 2024-09-19T08:02:33.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:33 smithi098 ceph-mon[31333]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:34.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:33 smithi162 ceph-mon[35260]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:34.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:33 smithi139 ceph-mon[34874]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:34.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ls' 2024-09-19T08:02:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:34.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:34 smithi098 ceph-mon[31333]: pgmap v314: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:35.078 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:34 smithi139 ceph-mon[34874]: pgmap v314: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:35.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:34 smithi162 ceph-mon[35260]: pgmap v314: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:35.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.323+0000 7f9747025640 1 -- 172.21.15.98:0/240284073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9740101c00 msgr2=0x7f9740102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:35.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.323+0000 7f9747025640 1 --2- 172.21.15.98:0/240284073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9740101c00 0x7f9740102000 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9734009900 tx=0x7f973402a410 comp rx=0 tx=0).stop 2024-09-19T08:02:35.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.323+0000 7f9747025640 1 -- 172.21.15.98:0/240284073 shutdown_connections 2024-09-19T08:02:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.323+0000 7f9747025640 1 --2- 172.21.15.98:0/240284073 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f97401037c0 0x7f974010a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.323+0000 7f9747025640 1 --2- 172.21.15.98:0/240284073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9740102e00 0x7f9740103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.323+0000 7f9747025640 1 --2- 172.21.15.98:0/240284073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9740101c00 0x7f9740102000 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.323+0000 7f9747025640 1 -- 172.21.15.98:0/240284073 >> 172.21.15.98:0/240284073 conn(0x7f97400fd390 msgr2=0x7f97400ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.323+0000 7f9747025640 1 -- 172.21.15.98:0/240284073 shutdown_connections 2024-09-19T08:02:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.323+0000 7f9747025640 1 -- 172.21.15.98:0/240284073 wait complete. 2024-09-19T08:02:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.323+0000 7f9747025640 1 Processor -- start 2024-09-19T08:02:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.323+0000 7f9747025640 1 -- start start 2024-09-19T08:02:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f9747025640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9740101c00 0x7f974019a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f9747025640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9740102e00 0x7f974019a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f9747025640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97401037c0 0x7f97401a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f9747025640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f974010d960 con 0x7f97401037c0 2024-09-19T08:02:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f9747025640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f974010d7e0 con 0x7f9740102e00 2024-09-19T08:02:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f9747025640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f974010dae0 con 0x7f9740101c00 2024-09-19T08:02:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f97455c2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97401037c0 0x7f97401a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f97455c2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97401037c0 0x7f97401a1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38428/0 (socket says 172.21.15.98:38428) 2024-09-19T08:02:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f97455c2640 1 -- 172.21.15.98:0/1234010028 learned_addr learned my addr 172.21.15.98:0/1234010028 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f9744dc1640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9740101c00 0x7f974019a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f97455c2640 1 -- 172.21.15.98:0/1234010028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9740101c00 msgr2=0x7f974019a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f973ffff640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9740102e00 0x7f974019a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f97455c2640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9740101c00 0x7f974019a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f97455c2640 1 -- 172.21.15.98:0/1234010028 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9740102e00 msgr2=0x7f974019a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f97455c2640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9740102e00 0x7f974019a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f97455c2640 1 -- 172.21.15.98:0/1234010028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97401a1ed0 con 0x7f97401037c0 2024-09-19T08:02:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f9744dc1640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9740101c00 0x7f974019a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f973ffff640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9740102e00 0x7f974019a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f97455c2640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97401037c0 0x7f97401a1860 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f973800b970 tx=0x7f973800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f973dffb640 1 -- 172.21.15.98:0/1234010028 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9738019070 con 0x7f97401037c0 2024-09-19T08:02:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f973dffb640 1 -- 172.21.15.98:0/1234010028 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f973800cb40 con 0x7f97401037c0 2024-09-19T08:02:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f9747025640 1 -- 172.21.15.98:0/1234010028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97401a21c0 con 0x7f97401037c0 2024-09-19T08:02:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.324+0000 7f973dffb640 1 -- 172.21.15.98:0/1234010028 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9738007650 con 0x7f97401037c0 2024-09-19T08:02:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.325+0000 7f9747025640 1 -- 172.21.15.98:0/1234010028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97401a2780 con 0x7f97401037c0 2024-09-19T08:02:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.325+0000 7f9747025640 1 -- 172.21.15.98:0/1234010028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97401a2a40 con 0x7f97401037c0 2024-09-19T08:02:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.326+0000 7f973dffb640 1 -- 172.21.15.98:0/1234010028 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f973801c070 con 0x7f97401037c0 2024-09-19T08:02:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.327+0000 7f973dffb640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f97300a3380 0x7f97300a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.327+0000 7f9744dc1640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f97300a3380 0x7f97300a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.327+0000 7f973dffb640 1 -- 172.21.15.98:0/1234010028 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f97380cef80 con 0x7f97401037c0 2024-09-19T08:02:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.327+0000 7f9744dc1640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f97300a3380 0x7f97300a5840 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9734000c00 tx=0x7f9734031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:35.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.331+0000 7f973dffb640 1 -- 172.21.15.98:0/1234010028 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f973809bc60 con 0x7f97401037c0 2024-09-19T08:02:35.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.507+0000 7f9747025640 1 -- 172.21.15.98:0/1234010028 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f97401a2c10 con 0x7f97300a3380 2024-09-19T08:02:35.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.513+0000 7f973dffb640 1 -- 172.21.15.98:0/1234010028 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f97401a2c10 con 0x7f97300a3380 2024-09-19T08:02:35.513 INFO:teuthology.orchestra.run.smithi098.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-19T08:02:35.513 INFO:teuthology.orchestra.run.smithi098.stdout:mds.all 12/12 28s ago 2m smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-09-19T08:02:35.514 INFO:teuthology.orchestra.run.smithi098.stdout:mgr 3/3 28s ago 7m smithi098=x;smithi139=y;smithi162=z;count:3 2024-09-19T08:02:35.514 INFO:teuthology.orchestra.run.smithi098.stdout:mon 3/3 28s ago 7m smithi098:172.21.15.98=a;smithi139:172.21.15.139=b;smithi162:172.21.15.162=c;count:3 2024-09-19T08:02:35.514 INFO:teuthology.orchestra.run.smithi098.stdout:osd 12 28s ago - 2024-09-19T08:02:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.515+0000 7f9747025640 1 -- 172.21.15.98:0/1234010028 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f97300a3380 msgr2=0x7f97300a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.515+0000 7f9747025640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f97300a3380 0x7f97300a5840 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9734000c00 tx=0x7f9734031400 comp rx=0 tx=0).stop 2024-09-19T08:02:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.515+0000 7f9747025640 1 -- 172.21.15.98:0/1234010028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97401037c0 msgr2=0x7f97401a1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.515+0000 7f9747025640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97401037c0 0x7f97401a1860 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f973800b970 tx=0x7f973800be30 comp rx=0 tx=0).stop 2024-09-19T08:02:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.515+0000 7f9747025640 1 -- 172.21.15.98:0/1234010028 shutdown_connections 2024-09-19T08:02:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.515+0000 7f9747025640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f97300a3380 0x7f97300a5840 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.515+0000 7f9747025640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97401037c0 0x7f97401a1860 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.515+0000 7f9747025640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9740102e00 0x7f974019a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.515+0000 7f9747025640 1 --2- 172.21.15.98:0/1234010028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9740101c00 0x7f974019a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.515+0000 7f9747025640 1 -- 172.21.15.98:0/1234010028 >> 172.21.15.98:0/1234010028 conn(0x7f97400fd390 msgr2=0x7f97401085c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.516+0000 7f9747025640 1 -- 172.21.15.98:0/1234010028 shutdown_connections 2024-09-19T08:02:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:35.516+0000 7f9747025640 1 -- 172.21.15.98:0/1234010028 wait complete. 2024-09-19T08:02:35.898 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:35 smithi098 ceph-mon[31333]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:36.021 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:35 smithi162 ceph-mon[35260]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:36.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:35 smithi139 ceph-mon[34874]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:36.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- bash -c 'ceph orch host ls' 2024-09-19T08:02:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:36.889 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:36 smithi098 ceph-mon[31333]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:36.890 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:36 smithi098 ceph-mon[31333]: pgmap v315: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:36.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.970+0000 7f00ec4f2640 1 -- 172.21.15.98:0/3785157924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e4101d10 msgr2=0x7f00e4102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:36.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.970+0000 7f00ec4f2640 1 --2- 172.21.15.98:0/3785157924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e4101d10 0x7f00e4102190 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f00d8009900 tx=0x7f00d802a410 comp rx=0 tx=0).stop 2024-09-19T08:02:36.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.970+0000 7f00ec4f2640 1 -- 172.21.15.98:0/3785157924 shutdown_connections 2024-09-19T08:02:36.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.970+0000 7f00ec4f2640 1 --2- 172.21.15.98:0/3785157924 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00e41026d0 0x7f00e4108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:36.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.970+0000 7f00ec4f2640 1 --2- 172.21.15.98:0/3785157924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e4101d10 0x7f00e4102190 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:36.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.970+0000 7f00ec4f2640 1 --2- 172.21.15.98:0/3785157924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00e4100b10 0x7f00e4100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:36.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.970+0000 7f00ec4f2640 1 -- 172.21.15.98:0/3785157924 >> 172.21.15.98:0/3785157924 conn(0x7f00e40fc2a0 msgr2=0x7f00e40fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:36.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.970+0000 7f00ec4f2640 1 -- 172.21.15.98:0/3785157924 shutdown_connections 2024-09-19T08:02:36.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.970+0000 7f00ec4f2640 1 -- 172.21.15.98:0/3785157924 wait complete. 2024-09-19T08:02:36.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00ec4f2640 1 Processor -- start 2024-09-19T08:02:36.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00ec4f2640 1 -- start start 2024-09-19T08:02:36.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00ec4f2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00e4100b10 0x7f00e41980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00ec4f2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00e4101d10 0x7f00e41985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00ec4f2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e41026d0 0x7f00e419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00ec4f2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f00e410d9a0 con 0x7f00e41026d0 2024-09-19T08:02:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00ec4f2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f00e410d820 con 0x7f00e4100b10 2024-09-19T08:02:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00ec4f2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f00e410db20 con 0x7f00e4101d10 2024-09-19T08:02:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00eaa8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e41026d0 0x7f00e419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00eaa8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e41026d0 0x7f00e419f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52696/0 (socket says 172.21.15.98:52696) 2024-09-19T08:02:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00eaa8f640 1 -- 172.21.15.98:0/2602519230 learned_addr learned my addr 172.21.15.98:0/2602519230 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00ea28e640 1 --2- 172.21.15.98:0/2602519230 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00e4100b10 0x7f00e41980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00e9a8d640 1 --2- 172.21.15.98:0/2602519230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00e4101d10 0x7f00e41985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00eaa8f640 1 -- 172.21.15.98:0/2602519230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00e4101d10 msgr2=0x7f00e41985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00eaa8f640 1 --2- 172.21.15.98:0/2602519230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00e4101d10 0x7f00e41985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00eaa8f640 1 -- 172.21.15.98:0/2602519230 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00e4100b10 msgr2=0x7f00e41980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00eaa8f640 1 --2- 172.21.15.98:0/2602519230 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00e4100b10 0x7f00e41980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00eaa8f640 1 -- 172.21.15.98:0/2602519230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e419fce0 con 0x7f00e41026d0 2024-09-19T08:02:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.971+0000 7f00eaa8f640 1 --2- 172.21.15.98:0/2602519230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e41026d0 0x7f00e419f670 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00b970 tx=0x7f00dc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.972+0000 7f00d77fe640 1 -- 172.21.15.98:0/2602519230 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00dc019070 con 0x7f00e41026d0 2024-09-19T08:02:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.972+0000 7f00d77fe640 1 -- 172.21.15.98:0/2602519230 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00dc014440 con 0x7f00e41026d0 2024-09-19T08:02:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.972+0000 7f00ec4f2640 1 -- 172.21.15.98:0/2602519230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e419ffd0 con 0x7f00e41026d0 2024-09-19T08:02:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.972+0000 7f00d77fe640 1 -- 172.21.15.98:0/2602519230 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00dc0075d0 con 0x7f00e41026d0 2024-09-19T08:02:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.972+0000 7f00ec4f2640 1 -- 172.21.15.98:0/2602519230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e41a04b0 con 0x7f00e41026d0 2024-09-19T08:02:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.973+0000 7f00ec4f2640 1 -- 172.21.15.98:0/2602519230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00e4107fb0 con 0x7f00e41026d0 2024-09-19T08:02:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.974+0000 7f00d77fe640 1 -- 172.21.15.98:0/2602519230 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f00dc01c070 con 0x7f00e41026d0 2024-09-19T08:02:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.974+0000 7f00d77fe640 1 --2- 172.21.15.98:0/2602519230 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f00d00a3300 0x7f00d00a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.975+0000 7f00ea28e640 1 --2- 172.21.15.98:0/2602519230 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f00d00a3300 0x7f00d00a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.975+0000 7f00d77fe640 1 -- 172.21.15.98:0/2602519230 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f00dc0cef90 con 0x7f00e41026d0 2024-09-19T08:02:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.975+0000 7f00ea28e640 1 --2- 172.21.15.98:0/2602519230 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f00d00a3300 0x7f00d00a57c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f00e4101b70 tx=0x7f00e0009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:36.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:36.979+0000 7f00d77fe640 1 -- 172.21.15.98:0/2602519230 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f00dc09bc70 con 0x7f00e41026d0 2024-09-19T08:02:37.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:36 smithi162 ceph-mon[35260]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:37.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:36 smithi162 ceph-mon[35260]: pgmap v315: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:37.149+0000 7f00ec4f2640 1 -- 172.21.15.98:0/2602519230 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f00e4062630 con 0x7f00d00a3300 2024-09-19T08:02:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:37.151+0000 7f00d77fe640 1 -- 172.21.15.98:0/2602519230 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f00e4062630 con 0x7f00d00a3300 2024-09-19T08:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout:HOST ADDR LABELS STATUS 2024-09-19T08:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098 172.21.15.98 2024-09-19T08:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout:smithi139 172.21.15.139 2024-09-19T08:02:37.152 INFO:teuthology.orchestra.run.smithi098.stdout:smithi162 172.21.15.162 2024-09-19T08:02:37.152 INFO:teuthology.orchestra.run.smithi098.stdout:3 hosts in cluster 2024-09-19T08:02:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:37.154+0000 7f00ec4f2640 1 -- 172.21.15.98:0/2602519230 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f00d00a3300 msgr2=0x7f00d00a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:37.154+0000 7f00ec4f2640 1 --2- 172.21.15.98:0/2602519230 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f00d00a3300 0x7f00d00a57c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f00e4101b70 tx=0x7f00e0009110 comp rx=0 tx=0).stop 2024-09-19T08:02:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:37.154+0000 7f00ec4f2640 1 -- 172.21.15.98:0/2602519230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e41026d0 msgr2=0x7f00e419f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:37.154+0000 7f00ec4f2640 1 --2- 172.21.15.98:0/2602519230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e41026d0 0x7f00e419f670 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00b970 tx=0x7f00dc00bd70 comp rx=0 tx=0).stop 2024-09-19T08:02:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:37.154+0000 7f00ec4f2640 1 -- 172.21.15.98:0/2602519230 shutdown_connections 2024-09-19T08:02:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:37.154+0000 7f00ec4f2640 1 --2- 172.21.15.98:0/2602519230 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f00d00a3300 0x7f00d00a57c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:37.154+0000 7f00ec4f2640 1 --2- 172.21.15.98:0/2602519230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e41026d0 0x7f00e419f670 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:37.154+0000 7f00ec4f2640 1 --2- 172.21.15.98:0/2602519230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00e4101d10 0x7f00e41985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:37.154+0000 7f00ec4f2640 1 --2- 172.21.15.98:0/2602519230 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00e4100b10 0x7f00e41980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:37.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:37.154+0000 7f00ec4f2640 1 -- 172.21.15.98:0/2602519230 >> 172.21.15.98:0/2602519230 conn(0x7f00e40fc2a0 msgr2=0x7f00e4105180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:37.154+0000 7f00ec4f2640 1 -- 172.21.15.98:0/2602519230 shutdown_connections 2024-09-19T08:02:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:37.154+0000 7f00ec4f2640 1 -- 172.21.15.98:0/2602519230 wait complete. 2024-09-19T08:02:37.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:36 smithi139 ceph-mon[34874]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:37.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:36 smithi139 ceph-mon[34874]: pgmap v315: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:37.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- bash -c 'ceph orch device ls' 2024-09-19T08:02:37.985 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.465+0000 7f9947086640 1 -- 172.21.15.98:0/991721695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9940102700 msgr2=0x7f9940104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.465+0000 7f9947086640 1 --2- 172.21.15.98:0/991721695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9940102700 0x7f9940104af0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9928009900 tx=0x7f992802a410 comp rx=0 tx=0).stop 2024-09-19T08:02:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.465+0000 7f9947086640 1 -- 172.21.15.98:0/991721695 shutdown_connections 2024-09-19T08:02:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.465+0000 7f9947086640 1 --2- 172.21.15.98:0/991721695 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9940107960 0x7f9940109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.465+0000 7f9947086640 1 --2- 172.21.15.98:0/991721695 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9940105030 0x7f9940107420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.465+0000 7f9947086640 1 --2- 172.21.15.98:0/991721695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9940102700 0x7f9940104af0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.465+0000 7f9947086640 1 -- 172.21.15.98:0/991721695 >> 172.21.15.98:0/991721695 conn(0x7f99400fc280 msgr2=0x7f99400fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.465+0000 7f9947086640 1 -- 172.21.15.98:0/991721695 shutdown_connections 2024-09-19T08:02:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.465+0000 7f9947086640 1 -- 172.21.15.98:0/991721695 wait complete. 2024-09-19T08:02:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.466+0000 7f9947086640 1 Processor -- start 2024-09-19T08:02:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.466+0000 7f9947086640 1 -- start start 2024-09-19T08:02:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.466+0000 7f9947086640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9940102700 0x7f9940102060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.466+0000 7f9947086640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9940105030 0x7f99401006b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.466+0000 7f9947086640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9940107960 0x7f9940100bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.466+0000 7f9947086640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f994010d950 con 0x7f9940105030 2024-09-19T08:02:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.466+0000 7f9947086640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f994010d7d0 con 0x7f9940107960 2024-09-19T08:02:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.466+0000 7f9947086640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f994010dad0 con 0x7f9940102700 2024-09-19T08:02:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.466+0000 7f993ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9940105030 0x7f99401006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.466+0000 7f993ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9940105030 0x7f99401006b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52716/0 (socket says 172.21.15.98:52716) 2024-09-19T08:02:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.466+0000 7f993ffff640 1 -- 172.21.15.98:0/3286209292 learned_addr learned my addr 172.21.15.98:0/3286209292 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.466+0000 7f9944e22640 1 --2- 172.21.15.98:0/3286209292 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9940102700 0x7f9940102060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.466+0000 7f993ffff640 1 -- 172.21.15.98:0/3286209292 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9940102700 msgr2=0x7f9940102060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.467+0000 7f993ffff640 1 --2- 172.21.15.98:0/3286209292 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9940102700 0x7f9940102060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.467+0000 7f993ffff640 1 -- 172.21.15.98:0/3286209292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9940107960 msgr2=0x7f9940100bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.467+0000 7f9945623640 1 --2- 172.21.15.98:0/3286209292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9940107960 0x7f9940100bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.467+0000 7f993ffff640 1 --2- 172.21.15.98:0/3286209292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9940107960 0x7f9940100bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.467+0000 7f993ffff640 1 -- 172.21.15.98:0/3286209292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99401014b0 con 0x7f9940105030 2024-09-19T08:02:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.467+0000 7f993ffff640 1 --2- 172.21.15.98:0/3286209292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9940105030 0x7f99401006b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f993000b450 tx=0x7f993000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.467+0000 7f993dffb640 1 -- 172.21.15.98:0/3286209292 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9930015020 con 0x7f9940105030 2024-09-19T08:02:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.467+0000 7f993dffb640 1 -- 172.21.15.98:0/3286209292 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9930011d70 con 0x7f9940105030 2024-09-19T08:02:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.467+0000 7f9947086640 1 -- 172.21.15.98:0/3286209292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99401a21f0 con 0x7f9940105030 2024-09-19T08:02:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.467+0000 7f993dffb640 1 -- 172.21.15.98:0/3286209292 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993000f340 con 0x7f9940105030 2024-09-19T08:02:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.467+0000 7f9947086640 1 -- 172.21.15.98:0/3286209292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99401a27d0 con 0x7f9940105030 2024-09-19T08:02:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.468+0000 7f9947086640 1 -- 172.21.15.98:0/3286209292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9940105490 con 0x7f9940105030 2024-09-19T08:02:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.469+0000 7f993dffb640 1 -- 172.21.15.98:0/3286209292 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f9930018020 con 0x7f9940105030 2024-09-19T08:02:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.470+0000 7f993dffb640 1 --2- 172.21.15.98:0/3286209292 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f992c0a3350 0x7f992c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.470+0000 7f9944e22640 1 --2- 172.21.15.98:0/3286209292 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f992c0a3350 0x7f992c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.470+0000 7f993dffb640 1 -- 172.21.15.98:0/3286209292 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f99300cf440 con 0x7f9940105030 2024-09-19T08:02:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.470+0000 7f9944e22640 1 --2- 172.21.15.98:0/3286209292 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f992c0a3350 0x7f992c0a5810 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9928000c00 tx=0x7f9928031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.474+0000 7f993dffb640 1 -- 172.21.15.98:0/3286209292 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f993001f020 con 0x7f9940105030 2024-09-19T08:02:38.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.647+0000 7f9947086640 1 -- 172.21.15.98:0/3286209292 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f9940062630 con 0x7f992c0a3350 2024-09-19T08:02:38.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.649+0000 7f993dffb640 1 -- 172.21.15.98:0/3286209292 <== mgr.14150 v2:172.21.15.98:6800/470997812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f9940062630 con 0x7f992c0a3350 2024-09-19T08:02:38.648 INFO:teuthology.orchestra.run.smithi098.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-19T08:02:38.649 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010H400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-19T08:02:38.649 INFO:teuthology.orchestra.run.smithi098.stdout:smithi139 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500N6400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-09-19T08:02:38.649 INFO:teuthology.orchestra.run.smithi098.stdout:smithi162 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JU400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-09-19T08:02:38.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.651+0000 7f9947086640 1 -- 172.21.15.98:0/3286209292 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f992c0a3350 msgr2=0x7f992c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:38.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.651+0000 7f9947086640 1 --2- 172.21.15.98:0/3286209292 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f992c0a3350 0x7f992c0a5810 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9928000c00 tx=0x7f9928031400 comp rx=0 tx=0).stop 2024-09-19T08:02:38.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.651+0000 7f9947086640 1 -- 172.21.15.98:0/3286209292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9940105030 msgr2=0x7f99401006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:38.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.651+0000 7f9947086640 1 --2- 172.21.15.98:0/3286209292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9940105030 0x7f99401006b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f993000b450 tx=0x7f993000b910 comp rx=0 tx=0).stop 2024-09-19T08:02:38.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.651+0000 7f9947086640 1 -- 172.21.15.98:0/3286209292 shutdown_connections 2024-09-19T08:02:38.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.651+0000 7f9947086640 1 --2- 172.21.15.98:0/3286209292 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f992c0a3350 0x7f992c0a5810 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:38.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.651+0000 7f9947086640 1 --2- 172.21.15.98:0/3286209292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9940107960 0x7f9940100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:38.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.651+0000 7f9947086640 1 --2- 172.21.15.98:0/3286209292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9940105030 0x7f99401006b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:38.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.651+0000 7f9947086640 1 --2- 172.21.15.98:0/3286209292 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9940102700 0x7f9940102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:38.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.651+0000 7f9947086640 1 -- 172.21.15.98:0/3286209292 >> 172.21.15.98:0/3286209292 conn(0x7f99400fc280 msgr2=0x7f9940103180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:38.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.651+0000 7f9947086640 1 -- 172.21.15.98:0/3286209292 shutdown_connections 2024-09-19T08:02:38.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:38.651+0000 7f9947086640 1 -- 172.21.15.98:0/3286209292 wait complete. 2024-09-19T08:02:38.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:38 smithi098 ceph-mon[31333]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:38.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:38 smithi098 ceph-mon[31333]: pgmap v316: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:39.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:38 smithi162 ceph-mon[35260]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:39.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:38 smithi162 ceph-mon[35260]: pgmap v316: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:39.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:38 smithi139 ceph-mon[34874]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:39.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:38 smithi139 ceph-mon[34874]: pgmap v316: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:39.257 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-19T08:02:39.265 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-09-19T08:02:39.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- bash -c 'ceph fs dump' 2024-09-19T08:02:39.507 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:39.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:39 smithi098 ceph-mon[31333]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.015+0000 7f5fdf924640 1 -- 172.21.15.98:0/2859571187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd81078c0 msgr2=0x7f5fd8109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.015+0000 7f5fdf924640 1 --2- 172.21.15.98:0/2859571187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd81078c0 0x7f5fd8109d10 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8009ca0 tx=0x7f5fc802a8c0 comp rx=0 tx=0).stop 2024-09-19T08:02:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.015+0000 7f5fdf924640 1 -- 172.21.15.98:0/2859571187 shutdown_connections 2024-09-19T08:02:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.015+0000 7f5fdf924640 1 --2- 172.21.15.98:0/2859571187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd81078c0 0x7f5fd8109d10 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.015+0000 7f5fdf924640 1 --2- 172.21.15.98:0/2859571187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fd8104f90 0x7f5fd8107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.015+0000 7f5fdf924640 1 --2- 172.21.15.98:0/2859571187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fd8102660 0x7f5fd8104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.015+0000 7f5fdf924640 1 -- 172.21.15.98:0/2859571187 >> 172.21.15.98:0/2859571187 conn(0x7f5fd80fc280 msgr2=0x7f5fd80fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.015+0000 7f5fdf924640 1 -- 172.21.15.98:0/2859571187 shutdown_connections 2024-09-19T08:02:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.015+0000 7f5fdf924640 1 -- 172.21.15.98:0/2859571187 wait complete. 2024-09-19T08:02:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fdf924640 1 Processor -- start 2024-09-19T08:02:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fdf924640 1 -- start start 2024-09-19T08:02:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fdf924640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd8102660 0x7f5fd8197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fdf924640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fd8104f90 0x7f5fd8198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fdf924640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fd81078c0 0x7f5fd819f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fdf924640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd810d860 con 0x7f5fd8102660 2024-09-19T08:02:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fdf924640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd810d6e0 con 0x7f5fd81078c0 2024-09-19T08:02:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fdf924640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd810d9e0 con 0x7f5fd8104f90 2024-09-19T08:02:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fdd6c0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd8102660 0x7f5fd8197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fdd6c0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd8102660 0x7f5fd8197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52740/0 (socket says 172.21.15.98:52740) 2024-09-19T08:02:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fdd6c0640 1 -- 172.21.15.98:0/1104438172 learned_addr learned my addr 172.21.15.98:0/1104438172 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fdcebf640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fd8104f90 0x7f5fd8198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fddec1640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fd81078c0 0x7f5fd819f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fdd6c0640 1 -- 172.21.15.98:0/1104438172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fd8104f90 msgr2=0x7f5fd8198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.016+0000 7f5fdd6c0640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fd8104f90 0x7f5fd8198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.017+0000 7f5fdd6c0640 1 -- 172.21.15.98:0/1104438172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fd81078c0 msgr2=0x7f5fd819f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.017+0000 7f5fdd6c0640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fd81078c0 0x7f5fd819f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.017+0000 7f5fdd6c0640 1 -- 172.21.15.98:0/1104438172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd819fa60 con 0x7f5fd8102660 2024-09-19T08:02:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.017+0000 7f5fddec1640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fd81078c0 0x7f5fd819f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.017+0000 7f5fdd6c0640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd8102660 0x7f5fd8197e30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5fd000c920 tx=0x7f5fd000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.017+0000 7f5fd67fc640 1 -- 172.21.15.98:0/1104438172 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0007c70 con 0x7f5fd8102660 2024-09-19T08:02:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.017+0000 7f5fdf924640 1 -- 172.21.15.98:0/1104438172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fd819fd50 con 0x7f5fd8102660 2024-09-19T08:02:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.017+0000 7f5fd67fc640 1 -- 172.21.15.98:0/1104438172 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fd0007e10 con 0x7f5fd8102660 2024-09-19T08:02:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.017+0000 7f5fd67fc640 1 -- 172.21.15.98:0/1104438172 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0018890 con 0x7f5fd8102660 2024-09-19T08:02:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.017+0000 7f5fdf924640 1 -- 172.21.15.98:0/1104438172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fd81a0200 con 0x7f5fd8102660 2024-09-19T08:02:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.018+0000 7f5fdf924640 1 -- 172.21.15.98:0/1104438172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fd80fdd70 con 0x7f5fd8102660 2024-09-19T08:02:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.019+0000 7f5fd67fc640 1 -- 172.21.15.98:0/1104438172 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f5fd0018a30 con 0x7f5fd8102660 2024-09-19T08:02:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.020+0000 7f5fd67fc640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5fc00a3300 0x7f5fc00a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.020+0000 7f5fdcebf640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5fc00a3300 0x7f5fc00a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.020+0000 7f5fd67fc640 1 -- 172.21.15.98:0/1104438172 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f5fd00cfe40 con 0x7f5fd8102660 2024-09-19T08:02:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.020+0000 7f5fdcebf640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5fc00a3300 0x7f5fc00a57c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8199350 tx=0x7f5fcc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.023+0000 7f5fd67fc640 1 -- 172.21.15.98:0/1104438172 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f5fd0014030 con 0x7f5fd8102660 2024-09-19T08:02:40.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:39 smithi162 ceph-mon[35260]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:40.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:39 smithi139 ceph-mon[34874]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:02:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.226+0000 7f5fdf924640 1 -- 172.21.15.98:0/1104438172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f5fd802da20 con 0x7f5fd8102660 2024-09-19T08:02:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.227+0000 7f5fd67fc640 1 -- 172.21.15.98:0/1104438172 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3135 (secure 0 0 0) 0x7f5fd009caa0 con 0x7f5fd8102660 2024-09-19T08:02:40.227 INFO:teuthology.orchestra.run.smithi098.stdout:e27 2024-09-19T08:02:40.227 INFO:teuthology.orchestra.run.smithi098.stdout:btime 2024-09-19T08:01:22:699589+0000 2024-09-19T08:02:40.227 INFO:teuthology.orchestra.run.smithi098.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-19T08:02:40.227 INFO:teuthology.orchestra.run.smithi098.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-19T08:02:40.227 INFO:teuthology.orchestra.run.smithi098.stdout:legacy client fscid: 1 2024-09-19T08:02:40.228 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:02:40.228 INFO:teuthology.orchestra.run.smithi098.stdout:Filesystem 'cephfs' (1) 2024-09-19T08:02:40.228 INFO:teuthology.orchestra.run.smithi098.stdout:fs_name cephfs 2024-09-19T08:02:40.228 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 27 2024-09-19T08:02:40.228 INFO:teuthology.orchestra.run.smithi098.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-09-19T08:02:40.228 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-09-19T08:01:11.307169+0000 2024-09-19T08:02:40.228 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2024-09-19T08:01:22.699569+0000 2024-09-19T08:02:40.228 INFO:teuthology.orchestra.run.smithi098.stdout:tableserver 0 2024-09-19T08:02:40.228 INFO:teuthology.orchestra.run.smithi098.stdout:root 0 2024-09-19T08:02:40.228 INFO:teuthology.orchestra.run.smithi098.stdout:session_timeout 300 2024-09-19T08:02:40.228 INFO:teuthology.orchestra.run.smithi098.stdout:session_autoclose 300 2024-09-19T08:02:40.228 INFO:teuthology.orchestra.run.smithi098.stdout:max_file_size 1099511627776 2024-09-19T08:02:40.228 INFO:teuthology.orchestra.run.smithi098.stdout:max_xattr_size 65536 2024-09-19T08:02:40.228 INFO:teuthology.orchestra.run.smithi098.stdout:required_client_features {} 2024-09-19T08:02:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:last_failure 0 2024-09-19T08:02:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:last_failure_osd_epoch 0 2024-09-19T08:02:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-19T08:02:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:max_mds 5 2024-09-19T08:02:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:in 0,1,2,3,4 2024-09-19T08:02:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:up {0=24485,1=24461,2=24437,3=24406,4=24376} 2024-09-19T08:02:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:failed 2024-09-19T08:02:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:damaged 2024-09-19T08:02:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:stopped 2024-09-19T08:02:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:data_pools [3] 2024-09-19T08:02:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:metadata_pool 2 2024-09-19T08:02:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:inline_data disabled 2024-09-19T08:02:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:balancer 2024-09-19T08:02:40.230 INFO:teuthology.orchestra.run.smithi098.stdout:bal_rank_mask -1 2024-09-19T08:02:40.230 INFO:teuthology.orchestra.run.smithi098.stdout:standby_count_wanted 0 2024-09-19T08:02:40.230 INFO:teuthology.orchestra.run.smithi098.stdout:qdb_cluster leader: 24485 members: 24485,24461,24437,24406,24376 2024-09-19T08:02:40.230 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b{0:24485} state up:active seq 4 addr [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T08:02:40.230 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.i{0:24424} state up:standby-replay seq 1 addr [v2:172.21.15.162:6838/3810709211,v1:172.21.15.162:6839/3810709211] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T08:02:40.230 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.h{1:24461} state up:active seq 9 addr [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T08:02:40.230 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c{1:24415} state up:standby-replay seq 1 addr [v2:172.21.15.162:6836/3969500353,v1:172.21.15.162:6837/3969500353] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T08:02:40.230 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.e{2:24437} state up:active seq 11 addr [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T08:02:40.230 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.k{2:24413} state up:standby-replay seq 1 addr [v2:172.21.15.139:6832/2267154969,v1:172.21.15.139:6833/2267154969] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T08:02:40.230 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.f{3:24406} state up:active seq 11 addr [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T08:02:40.230 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a{3:14643} state up:standby-replay seq 1 addr [v2:172.21.15.98:6840/1339458935,v1:172.21.15.98:6841/1339458935] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T08:02:40.230 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.l{4:24376} state up:active seq 14 addr [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T08:02:40.231 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.g{4:14631} state up:standby-replay seq 1 addr [v2:172.21.15.98:6838/3417988465,v1:172.21.15.98:6839/3417988465] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T08:02:40.231 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:02:40.231 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:02:40.231 INFO:teuthology.orchestra.run.smithi098.stdout:Standby daemons: 2024-09-19T08:02:40.231 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:02:40.231 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.j{-1:14565} state up:standby seq 1 addr [v2:172.21.15.98:6834/85442870,v1:172.21.15.98:6835/85442870] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T08:02:40.231 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d{-1:14577} state up:standby seq 1 addr [v2:172.21.15.98:6836/3958359300,v1:172.21.15.98:6837/3958359300] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T08:02:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 27 2024-09-19T08:02:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.230+0000 7f5fdf924640 1 -- 172.21.15.98:0/1104438172 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5fc00a3300 msgr2=0x7f5fc00a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.230+0000 7f5fdf924640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5fc00a3300 0x7f5fc00a57c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8199350 tx=0x7f5fcc00b390 comp rx=0 tx=0).stop 2024-09-19T08:02:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.230+0000 7f5fdf924640 1 -- 172.21.15.98:0/1104438172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd8102660 msgr2=0x7f5fd8197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.230+0000 7f5fdf924640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd8102660 0x7f5fd8197e30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5fd000c920 tx=0x7f5fd000cde0 comp rx=0 tx=0).stop 2024-09-19T08:02:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.230+0000 7f5fdf924640 1 -- 172.21.15.98:0/1104438172 shutdown_connections 2024-09-19T08:02:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.230+0000 7f5fdf924640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f5fc00a3300 0x7f5fc00a57c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.230+0000 7f5fdf924640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fd81078c0 0x7f5fd819f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.230+0000 7f5fdf924640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fd8104f90 0x7f5fd8198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.230+0000 7f5fdf924640 1 --2- 172.21.15.98:0/1104438172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd8102660 0x7f5fd8197e30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.230+0000 7f5fdf924640 1 -- 172.21.15.98:0/1104438172 >> 172.21.15.98:0/1104438172 conn(0x7f5fd80fc280 msgr2=0x7f5fd8102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.230+0000 7f5fdf924640 1 -- 172.21.15.98:0/1104438172 shutdown_connections 2024-09-19T08:02:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:40.230+0000 7f5fdf924640 1 -- 172.21.15.98:0/1104438172 wait complete. 2024-09-19T08:02:40.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:40 smithi098 ceph-mon[31333]: pgmap v317: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:40.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:40 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1104438172' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-19T08:02:40.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- bash -c 'ceph osd dump' 2024-09-19T08:02:41.118 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:41.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:40 smithi162 ceph-mon[35260]: pgmap v317: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:41.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:40 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1104438172' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-19T08:02:41.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:40 smithi139 ceph-mon[34874]: pgmap v317: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:41.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:40 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1104438172' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-19T08:02:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.574+0000 7f912ce4d640 1 -- 172.21.15.98:0/4266807222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9128101c00 msgr2=0x7f9128102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.574+0000 7f912ce4d640 1 --2- 172.21.15.98:0/4266807222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9128101c00 0x7f9128102000 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9118009900 tx=0x7f911802a410 comp rx=0 tx=0).stop 2024-09-19T08:02:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.574+0000 7f912ce4d640 1 -- 172.21.15.98:0/4266807222 shutdown_connections 2024-09-19T08:02:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.574+0000 7f912ce4d640 1 --2- 172.21.15.98:0/4266807222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91281037c0 0x7f912810a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.574+0000 7f912ce4d640 1 --2- 172.21.15.98:0/4266807222 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9128102e00 0x7f9128103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.574+0000 7f912ce4d640 1 --2- 172.21.15.98:0/4266807222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9128101c00 0x7f9128102000 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.574+0000 7f912ce4d640 1 -- 172.21.15.98:0/4266807222 >> 172.21.15.98:0/4266807222 conn(0x7f91280fd390 msgr2=0x7f91280ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.574+0000 7f912ce4d640 1 -- 172.21.15.98:0/4266807222 shutdown_connections 2024-09-19T08:02:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.574+0000 7f912ce4d640 1 -- 172.21.15.98:0/4266807222 wait complete. 2024-09-19T08:02:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f912ce4d640 1 Processor -- start 2024-09-19T08:02:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f912ce4d640 1 -- start start 2024-09-19T08:02:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f912ce4d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9128101c00 0x7f9128197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f912ce4d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9128102e00 0x7f9128198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f912ce4d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91281037c0 0x7f912819f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f912ce4d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f912810d8e0 con 0x7f91281037c0 2024-09-19T08:02:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f912ce4d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f912810d760 con 0x7f9128102e00 2024-09-19T08:02:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f912ce4d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f912810da60 con 0x7f9128101c00 2024-09-19T08:02:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f9126d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91281037c0 0x7f912819f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f9126d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91281037c0 0x7f912819f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52762/0 (socket says 172.21.15.98:52762) 2024-09-19T08:02:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f9126d9d640 1 -- 172.21.15.98:0/3079851073 learned_addr learned my addr 172.21.15.98:0/3079851073 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f912659c640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9128101c00 0x7f9128197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f911ffff640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9128102e00 0x7f9128198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f9126d9d640 1 -- 172.21.15.98:0/3079851073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9128101c00 msgr2=0x7f9128197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f9126d9d640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9128101c00 0x7f9128197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f9126d9d640 1 -- 172.21.15.98:0/3079851073 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9128102e00 msgr2=0x7f9128198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f9126d9d640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9128102e00 0x7f9128198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f9126d9d640 1 -- 172.21.15.98:0/3079851073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f912819fb10 con 0x7f91281037c0 2024-09-19T08:02:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.575+0000 7f911ffff640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9128102e00 0x7f9128198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.576+0000 7f9126d9d640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91281037c0 0x7f912819f4a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f912000b970 tx=0x7f912000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.576+0000 7f911f7fe640 1 -- 172.21.15.98:0/3079851073 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120019070 con 0x7f91281037c0 2024-09-19T08:02:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.576+0000 7f911f7fe640 1 -- 172.21.15.98:0/3079851073 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f912000cb40 con 0x7f91281037c0 2024-09-19T08:02:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.576+0000 7f912ce4d640 1 -- 172.21.15.98:0/3079851073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f912819fe00 con 0x7f91281037c0 2024-09-19T08:02:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.576+0000 7f911f7fe640 1 -- 172.21.15.98:0/3079851073 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120007650 con 0x7f91281037c0 2024-09-19T08:02:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.576+0000 7f912ce4d640 1 -- 172.21.15.98:0/3079851073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91281a0330 con 0x7f91281037c0 2024-09-19T08:02:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.577+0000 7f912ce4d640 1 -- 172.21.15.98:0/3079851073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9128109000 con 0x7f91281037c0 2024-09-19T08:02:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.578+0000 7f911f7fe640 1 -- 172.21.15.98:0/3079851073 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f912001c070 con 0x7f91281037c0 2024-09-19T08:02:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.578+0000 7f911f7fe640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f91140a3300 0x7f91140a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.579+0000 7f912659c640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f91140a3300 0x7f91140a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.579+0000 7f911f7fe640 1 -- 172.21.15.98:0/3079851073 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f91200ceec0 con 0x7f91281037c0 2024-09-19T08:02:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.579+0000 7f912659c640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f91140a3300 0x7f91140a57c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9118005ab0 tx=0x7f9118000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.583+0000 7f911f7fe640 1 -- 172.21.15.98:0/3079851073 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f912009b550 con 0x7f91281037c0 2024-09-19T08:02:41.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.755+0000 7f912ce4d640 1 -- 172.21.15.98:0/3079851073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f912802d7c0 con 0x7f91281037c0 2024-09-19T08:02:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.756+0000 7f911f7fe640 1 -- 172.21.15.98:0/3079851073 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v80) ==== 56+0+5291 (secure 0 0 0) 0x7f912009f850 con 0x7f91281037c0 2024-09-19T08:02:41.756 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 80 2024-09-19T08:02:41.757 INFO:teuthology.orchestra.run.smithi098.stdout:fsid 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T08:02:41.757 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-09-19T07:53:39.127398+0000 2024-09-19T08:02:41.757 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2024-09-19T08:01:26.714628+0000 2024-09-19T08:02:41.757 INFO:teuthology.orchestra.run.smithi098.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-19T08:02:41.757 INFO:teuthology.orchestra.run.smithi098.stdout:crush_version 26 2024-09-19T08:02:41.757 INFO:teuthology.orchestra.run.smithi098.stdout:full_ratio 0.95 2024-09-19T08:02:41.757 INFO:teuthology.orchestra.run.smithi098.stdout:backfillfull_ratio 0.9 2024-09-19T08:02:41.757 INFO:teuthology.orchestra.run.smithi098.stdout:nearfull_ratio 0.85 2024-09-19T08:02:41.757 INFO:teuthology.orchestra.run.smithi098.stdout:require_min_compat_client luminous 2024-09-19T08:02:41.758 INFO:teuthology.orchestra.run.smithi098.stdout:min_compat_client luminous 2024-09-19T08:02:41.758 INFO:teuthology.orchestra.run.smithi098.stdout:require_osd_release squid 2024-09-19T08:02:41.758 INFO:teuthology.orchestra.run.smithi098.stdout:stretch_mode_enabled false 2024-09-19T08:02:41.758 INFO:teuthology.orchestra.run.smithi098.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-19T08:02:41.758 INFO:teuthology.orchestra.run.smithi098.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-09-19T08:02:41.758 INFO:teuthology.orchestra.run.smithi098.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-09-19T08:02:41.758 INFO:teuthology.orchestra.run.smithi098.stdout:max_osd 12 2024-09-19T08:02:41.758 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 up in weight 1 up_from 8 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6802/1719360533,v1:172.21.15.98:6803/1719360533] [v2:172.21.15.98:6804/1719360533,v1:172.21.15.98:6805/1719360533] exists,up a7d01ce6-165d-493d-ae98-d8ad9e5d4134 2024-09-19T08:02:41.758 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 up in weight 1 up_from 12 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6800/4036962626,v1:172.21.15.139:6801/4036962626] [v2:172.21.15.139:6802/4036962626,v1:172.21.15.139:6803/4036962626] exists,up 34724bd5-0738-4473-b087-c09031c1ee2e 2024-09-19T08:02:41.758 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 up in weight 1 up_from 16 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6800/2811412891,v1:172.21.15.162:6801/2811412891] [v2:172.21.15.162:6802/2811412891,v1:172.21.15.162:6803/2811412891] exists,up 16985ebe-0e32-42ee-bded-b7838e290afc 2024-09-19T08:02:41.758 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 up in weight 1 up_from 24 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6810/1383392014,v1:172.21.15.98:6811/1383392014] [v2:172.21.15.98:6812/1383392014,v1:172.21.15.98:6813/1383392014] exists,up b140e466-3119-49cb-aec2-ba6ca4d10d11 2024-09-19T08:02:41.758 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 up in weight 1 up_from 31 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6808/116453520,v1:172.21.15.139:6809/116453520] [v2:172.21.15.139:6810/116453520,v1:172.21.15.139:6811/116453520] exists,up b4412813-df57-4ac6-8fb7-f57b69c74768 2024-09-19T08:02:41.758 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 up in weight 1 up_from 36 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6808/3769530297,v1:172.21.15.162:6809/3769530297] [v2:172.21.15.162:6810/3769530297,v1:172.21.15.162:6811/3769530297] exists,up d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c 2024-09-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 up in weight 1 up_from 43 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6818/692373267,v1:172.21.15.98:6819/692373267] [v2:172.21.15.98:6820/692373267,v1:172.21.15.98:6821/692373267] exists,up ba1c1590-6aea-40dc-90c8-e21b88896909 2024-09-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 up in weight 1 up_from 47 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6816/1963811650,v1:172.21.15.139:6817/1963811650] [v2:172.21.15.139:6818/1963811650,v1:172.21.15.139:6819/1963811650] exists,up eb9093ad-a068-4a63-ab87-0b50522cbc29 2024-09-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi098.stdout:osd.8 up in weight 1 up_from 54 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6816/3607966458,v1:172.21.15.162:6817/3607966458] [v2:172.21.15.162:6818/3607966458,v1:172.21.15.162:6819/3607966458] exists,up 31ca330a-fc45-466d-81c2-ef4386b3cc6e 2024-09-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi098.stdout:osd.9 up in weight 1 up_from 61 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6826/2050391244,v1:172.21.15.98:6827/2050391244] [v2:172.21.15.98:6828/2050391244,v1:172.21.15.98:6829/2050391244] exists,up 9c4f216f-aef1-4367-9c58-b913b0c4f7db 2024-09-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi098.stdout:osd.10 up in weight 1 up_from 65 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6824/2336761639,v1:172.21.15.139:6825/2336761639] [v2:172.21.15.139:6826/2336761639,v1:172.21.15.139:6827/2336761639] exists,up 03cc7df6-84ae-4d9a-a3d7-74c91e1744e9 2024-09-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi098.stdout:osd.11 up in weight 1 up_from 72 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6824/1231011554,v1:172.21.15.162:6825/1231011554] [v2:172.21.15.162:6826/1231011554,v1:172.21.15.162:6827/1231011554] exists,up 29e95498-4952-46d4-8fa6-6d8e2b862111 2024-09-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.16 [2,4] 2024-09-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.1c [8,11] 2024-09-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.1d [2,4] 2024-09-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.1f [8,1,7,11] 2024-09-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.2 [3,9] 2024-09-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.14 [4,6] 2024-09-19T08:02:41.760 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.20 [5,9] 2024-09-19T08:02:41.760 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.24 [4,9] 2024-09-19T08:02:41.760 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3842627255 expires 2024-09-20T07:54:25.475454+0000 2024-09-19T08:02:41.760 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/16410793 expires 2024-09-20T07:54:25.475454+0000 2024-09-19T08:02:41.760 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3838590070 expires 2024-09-20T07:54:25.475454+0000 2024-09-19T08:02:41.760 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/1238193420 expires 2024-09-20T07:54:25.475454+0000 2024-09-19T08:02:41.760 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/1560445911 expires 2024-09-20T07:54:02.418785+0000 2024-09-19T08:02:41.760 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/16410793 expires 2024-09-20T07:54:25.475454+0000 2024-09-19T08:02:41.760 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/1560445911 expires 2024-09-20T07:54:02.418785+0000 2024-09-19T08:02:41.760 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3194864054 expires 2024-09-20T07:54:02.418785+0000 2024-09-19T08:02:41.760 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3866031018 expires 2024-09-20T07:54:02.418785+0000 2024-09-19T08:02:41.761 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/1497156658 expires 2024-09-20T07:54:02.418785+0000 2024-09-19T08:02:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.759+0000 7f912ce4d640 1 -- 172.21.15.98:0/3079851073 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f91140a3300 msgr2=0x7f91140a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.760+0000 7f912ce4d640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f91140a3300 0x7f91140a57c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9118005ab0 tx=0x7f9118000bc0 comp rx=0 tx=0).stop 2024-09-19T08:02:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.760+0000 7f912ce4d640 1 -- 172.21.15.98:0/3079851073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91281037c0 msgr2=0x7f912819f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.760+0000 7f912ce4d640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91281037c0 0x7f912819f4a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f912000b970 tx=0x7f912000be30 comp rx=0 tx=0).stop 2024-09-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.760+0000 7f912ce4d640 1 -- 172.21.15.98:0/3079851073 shutdown_connections 2024-09-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.760+0000 7f912ce4d640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f91140a3300 0x7f91140a57c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.760+0000 7f912ce4d640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91281037c0 0x7f912819f4a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.760+0000 7f912ce4d640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9128102e00 0x7f9128198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.760+0000 7f912ce4d640 1 --2- 172.21.15.98:0/3079851073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9128101c00 0x7f9128197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.760+0000 7f912ce4d640 1 -- 172.21.15.98:0/3079851073 >> 172.21.15.98:0/3079851073 conn(0x7f91280fd390 msgr2=0x7f91281061d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.760+0000 7f912ce4d640 1 -- 172.21.15.98:0/3079851073 shutdown_connections 2024-09-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:41.760+0000 7f912ce4d640 1 -- 172.21.15.98:0/3079851073 wait complete. 2024-09-19T08:02:42.447 INFO:teuthology.run_tasks:Running task fs.ready... 2024-09-19T08:02:42.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:02:42.713 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:42.958 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:42 smithi098 ceph-mon[31333]: pgmap v318: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:42.958 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:42 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3079851073' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-19T08:02:43.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:42 smithi162 ceph-mon[35260]: pgmap v318: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:43.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:42 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3079851073' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-19T08:02:43.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:42 smithi139 ceph-mon[34874]: pgmap v318: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:43.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:42 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3079851073' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-19T08:02:43.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.184+0000 7f80666a0640 1 -- 172.21.15.98:0/1975210124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8060100e20 msgr2=0x7f8060101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:43.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.184+0000 7f80666a0640 1 --2- 172.21.15.98:0/1975210124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8060100e20 0x7f8060101200 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8050009900 tx=0x7f805002a480 comp rx=0 tx=0).stop 2024-09-19T08:02:43.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.184+0000 7f80666a0640 1 -- 172.21.15.98:0/1975210124 shutdown_connections 2024-09-19T08:02:43.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.184+0000 7f80666a0640 1 --2- 172.21.15.98:0/1975210124 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8060102ef0 0x7f806010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:43.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.184+0000 7f80666a0640 1 --2- 172.21.15.98:0/1975210124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8060102570 0x7f80601029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:43.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.184+0000 7f80666a0640 1 --2- 172.21.15.98:0/1975210124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8060100e20 0x7f8060101200 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:43.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.184+0000 7f80666a0640 1 -- 172.21.15.98:0/1975210124 >> 172.21.15.98:0/1975210124 conn(0x7f80600fc7d0 msgr2=0x7f80600febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:43.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.184+0000 7f80666a0640 1 -- 172.21.15.98:0/1975210124 shutdown_connections 2024-09-19T08:02:43.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.184+0000 7f80666a0640 1 -- 172.21.15.98:0/1975210124 wait complete. 2024-09-19T08:02:43.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.184+0000 7f80666a0640 1 Processor -- start 2024-09-19T08:02:43.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.184+0000 7f80666a0640 1 -- start start 2024-09-19T08:02:43.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.184+0000 7f80666a0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8060100e20 0x7f80601984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:43.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.184+0000 7f80666a0640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8060102570 0x7f80601989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:43.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f80666a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8060102ef0 0x7f806019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:43.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f80666a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8060110f50 con 0x7f8060102ef0 2024-09-19T08:02:43.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f80666a0640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8060110dd0 con 0x7f8060102570 2024-09-19T08:02:43.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f80666a0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f80601110d0 con 0x7f8060100e20 2024-09-19T08:02:43.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f8064c3d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8060102ef0 0x7f806019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:43.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f8064c3d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8060102ef0 0x7f806019cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52776/0 (socket says 172.21.15.98:52776) 2024-09-19T08:02:43.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f8064c3d640 1 -- 172.21.15.98:0/3466639057 learned_addr learned my addr 172.21.15.98:0/3466639057 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:43.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f8057fff640 1 --2- 172.21.15.98:0/3466639057 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8060102570 0x7f80601989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:43.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f805ffff640 1 --2- 172.21.15.98:0/3466639057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8060100e20 0x7f80601984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:43.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f8064c3d640 1 -- 172.21.15.98:0/3466639057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8060100e20 msgr2=0x7f80601984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:43.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f8064c3d640 1 --2- 172.21.15.98:0/3466639057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8060100e20 0x7f80601984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:43.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f8064c3d640 1 -- 172.21.15.98:0/3466639057 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8060102570 msgr2=0x7f80601989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:43.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f8064c3d640 1 --2- 172.21.15.98:0/3466639057 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8060102570 0x7f80601989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:43.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f8064c3d640 1 -- 172.21.15.98:0/3466639057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806019d450 con 0x7f8060102ef0 2024-09-19T08:02:43.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f8064c3d640 1 --2- 172.21.15.98:0/3466639057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8060102ef0 0x7f806019cd70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f805800eb10 tx=0x7f805800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:43.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f805dffb640 1 -- 172.21.15.98:0/3466639057 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8058004b20 con 0x7f8060102ef0 2024-09-19T08:02:43.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f805dffb640 1 -- 172.21.15.98:0/3466639057 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8058004cc0 con 0x7f8060102ef0 2024-09-19T08:02:43.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f80666a0640 1 -- 172.21.15.98:0/3466639057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f806019d740 con 0x7f8060102ef0 2024-09-19T08:02:43.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f805dffb640 1 -- 172.21.15.98:0/3466639057 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8058010760 con 0x7f8060102ef0 2024-09-19T08:02:43.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.185+0000 7f80666a0640 1 -- 172.21.15.98:0/3466639057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80601a5020 con 0x7f8060102ef0 2024-09-19T08:02:43.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.186+0000 7f80666a0640 1 -- 172.21.15.98:0/3466639057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8060101200 con 0x7f8060102ef0 2024-09-19T08:02:43.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.187+0000 7f805dffb640 1 -- 172.21.15.98:0/3466639057 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f8058005020 con 0x7f8060102ef0 2024-09-19T08:02:43.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.188+0000 7f805dffb640 1 --2- 172.21.15.98:0/3466639057 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f804c0a3350 0x7f804c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:43.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.188+0000 7f805ffff640 1 --2- 172.21.15.98:0/3466639057 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f804c0a3350 0x7f804c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:43.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.188+0000 7f805dffb640 1 -- 172.21.15.98:0/3466639057 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f80580cffd0 con 0x7f8060102ef0 2024-09-19T08:02:43.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.189+0000 7f805ffff640 1 --2- 172.21.15.98:0/3466639057 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f804c0a3350 0x7f804c0a5810 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8050005ab0 tx=0x7f8050005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:43.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.192+0000 7f805dffb640 1 -- 172.21.15.98:0/3466639057 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f805809cba0 con 0x7f8060102ef0 2024-09-19T08:02:43.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.393+0000 7f80666a0640 1 -- 172.21.15.98:0/3466639057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80601a5250 con 0x7f8060102ef0 2024-09-19T08:02:43.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.394+0000 7f805dffb640 1 -- 172.21.15.98:0/3466639057 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11983 (secure 0 0 0) 0x7f805809c4a0 con 0x7f8060102ef0 2024-09-19T08:02:43.394 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:02:43.394 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":27,"btime":"2024-09-19T08:01:22:699589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:01:22.699569+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24461,24437,24406,24376]},"id":1}]} 2024-09-19T08:02:43.394 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 27 2024-09-19T08:02:43.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.396+0000 7f80666a0640 1 -- 172.21.15.98:0/3466639057 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f804c0a3350 msgr2=0x7f804c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:43.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.396+0000 7f80666a0640 1 --2- 172.21.15.98:0/3466639057 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f804c0a3350 0x7f804c0a5810 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8050005ab0 tx=0x7f8050005a20 comp rx=0 tx=0).stop 2024-09-19T08:02:43.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.396+0000 7f80666a0640 1 -- 172.21.15.98:0/3466639057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8060102ef0 msgr2=0x7f806019cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:43.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.397+0000 7f80666a0640 1 --2- 172.21.15.98:0/3466639057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8060102ef0 0x7f806019cd70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f805800eb10 tx=0x7f805800c2d0 comp rx=0 tx=0).stop 2024-09-19T08:02:43.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.397+0000 7f80666a0640 1 -- 172.21.15.98:0/3466639057 shutdown_connections 2024-09-19T08:02:43.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.397+0000 7f80666a0640 1 --2- 172.21.15.98:0/3466639057 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f804c0a3350 0x7f804c0a5810 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:43.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.397+0000 7f80666a0640 1 --2- 172.21.15.98:0/3466639057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8060102ef0 0x7f806019cd70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:43.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.397+0000 7f80666a0640 1 --2- 172.21.15.98:0/3466639057 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8060102570 0x7f80601989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:43.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.397+0000 7f80666a0640 1 --2- 172.21.15.98:0/3466639057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8060100e20 0x7f80601984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:43.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.397+0000 7f80666a0640 1 -- 172.21.15.98:0/3466639057 >> 172.21.15.98:0/3466639057 conn(0x7f80600fc7d0 msgr2=0x7f8060069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:43.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.397+0000 7f80666a0640 1 -- 172.21.15.98:0/3466639057 shutdown_connections 2024-09-19T08:02:43.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:43.397+0000 7f80666a0640 1 -- 172.21.15.98:0/3466639057 wait complete. 2024-09-19T08:02:44.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:43 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3466639057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:44.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:02:44.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:43 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3466639057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:44.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:43 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3466639057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:44.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.852+0000 7f5349457640 1 -- 172.21.15.98:0/1353641937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5344106a80 msgr2=0x7f5344106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.852+0000 7f5349457640 1 --2- 172.21.15.98:0/1353641937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5344106a80 0x7f5344106e60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5334009cd0 tx=0x7f533402a8e0 comp rx=0 tx=0).stop 2024-09-19T08:02:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.852+0000 7f5349457640 1 -- 172.21.15.98:0/1353641937 shutdown_connections 2024-09-19T08:02:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.852+0000 7f5349457640 1 --2- 172.21.15.98:0/1353641937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5344101420 0x7f534410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.852+0000 7f5349457640 1 --2- 172.21.15.98:0/1353641937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5344100a80 0x7f5344100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.852+0000 7f5349457640 1 --2- 172.21.15.98:0/1353641937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5344106a80 0x7f5344106e60 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.852+0000 7f5349457640 1 -- 172.21.15.98:0/1353641937 >> 172.21.15.98:0/1353641937 conn(0x7f53440fc7a0 msgr2=0x7f53440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.852+0000 7f5349457640 1 -- 172.21.15.98:0/1353641937 shutdown_connections 2024-09-19T08:02:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.852+0000 7f5349457640 1 -- 172.21.15.98:0/1353641937 wait complete. 2024-09-19T08:02:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.853+0000 7f5349457640 1 Processor -- start 2024-09-19T08:02:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.853+0000 7f5349457640 1 -- start start 2024-09-19T08:02:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.853+0000 7f5349457640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5344100a80 0x7f5344198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.853+0000 7f5349457640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5344101420 0x7f5344198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.853+0000 7f5349457640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5344106a80 0x7f534419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.853+0000 7f5349457640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5344110e80 con 0x7f5344101420 2024-09-19T08:02:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.853+0000 7f5349457640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5344110d00 con 0x7f5344106a80 2024-09-19T08:02:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.853+0000 7f5349457640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5344111000 con 0x7f5344100a80 2024-09-19T08:02:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.853+0000 7f53427fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5344101420 0x7f5344198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.853+0000 7f53427fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5344101420 0x7f5344198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52782/0 (socket says 172.21.15.98:52782) 2024-09-19T08:02:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f53427fc640 1 -- 172.21.15.98:0/1357070788 learned_addr learned my addr 172.21.15.98:0/1357070788 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f53437fe640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5344106a80 0x7f534419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f5342ffd640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5344100a80 0x7f5344198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f53427fc640 1 -- 172.21.15.98:0/1357070788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5344100a80 msgr2=0x7f5344198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f53427fc640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5344100a80 0x7f5344198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f53427fc640 1 -- 172.21.15.98:0/1357070788 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5344106a80 msgr2=0x7f534419cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f53427fc640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5344106a80 0x7f534419cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f53427fc640 1 -- 172.21.15.98:0/1357070788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534419d3e0 con 0x7f5344101420 2024-09-19T08:02:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f53437fe640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5344106a80 0x7f534419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f53427fc640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5344101420 0x7f5344198970 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f533800c920 tx=0x7f533800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f532bfff640 1 -- 172.21.15.98:0/1357070788 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5338007c70 con 0x7f5344101420 2024-09-19T08:02:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f5349457640 1 -- 172.21.15.98:0/1357070788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534419d6d0 con 0x7f5344101420 2024-09-19T08:02:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f532bfff640 1 -- 172.21.15.98:0/1357070788 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5338007e10 con 0x7f5344101420 2024-09-19T08:02:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f532bfff640 1 -- 172.21.15.98:0/1357070788 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5338018890 con 0x7f5344101420 2024-09-19T08:02:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.854+0000 7f5349457640 1 -- 172.21.15.98:0/1357070788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53441a4fb0 con 0x7f5344101420 2024-09-19T08:02:44.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.855+0000 7f5349457640 1 -- 172.21.15.98:0/1357070788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5344100ee0 con 0x7f5344101420 2024-09-19T08:02:44.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.856+0000 7f532bfff640 1 -- 172.21.15.98:0/1357070788 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f5338018a70 con 0x7f5344101420 2024-09-19T08:02:44.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.857+0000 7f532bfff640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f532c0a32d0 0x7f532c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:44.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.857+0000 7f5342ffd640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f532c0a32d0 0x7f532c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:44.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.857+0000 7f532bfff640 1 -- 172.21.15.98:0/1357070788 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f53380cfc90 con 0x7f5344101420 2024-09-19T08:02:44.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.857+0000 7f5342ffd640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f532c0a32d0 0x7f532c0a5790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5334006010 tx=0x7f53340093f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:44.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:44.861+0000 7f532bfff640 1 -- 172.21.15.98:0/1357070788 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f53380d5020 con 0x7f5344101420 2024-09-19T08:02:45.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:45.062+0000 7f5349457640 1 -- 172.21.15.98:0/1357070788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5344109a70 con 0x7f5344101420 2024-09-19T08:02:45.062 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:44 smithi098 ceph-mon[31333]: pgmap v319: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:45.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:45.063+0000 7f532bfff640 1 -- 172.21.15.98:0/1357070788 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11983 (secure 0 0 0) 0x7f5338014030 con 0x7f5344101420 2024-09-19T08:02:45.063 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:02:45.063 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":27,"btime":"2024-09-19T08:01:22:699589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:01:22.699569+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24461,24437,24406,24376]},"id":1}]} 2024-09-19T08:02:45.064 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 27 2024-09-19T08:02:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:45.065+0000 7f5349457640 1 -- 172.21.15.98:0/1357070788 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f532c0a32d0 msgr2=0x7f532c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:45.065+0000 7f5349457640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f532c0a32d0 0x7f532c0a5790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5334006010 tx=0x7f53340093f0 comp rx=0 tx=0).stop 2024-09-19T08:02:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:45.065+0000 7f5349457640 1 -- 172.21.15.98:0/1357070788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5344101420 msgr2=0x7f5344198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:45.065+0000 7f5349457640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5344101420 0x7f5344198970 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f533800c920 tx=0x7f533800cde0 comp rx=0 tx=0).stop 2024-09-19T08:02:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:45.065+0000 7f5349457640 1 -- 172.21.15.98:0/1357070788 shutdown_connections 2024-09-19T08:02:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:45.065+0000 7f5349457640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f532c0a32d0 0x7f532c0a5790 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:45.065+0000 7f5349457640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5344106a80 0x7f534419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:45.065+0000 7f5349457640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5344101420 0x7f5344198970 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:45.065+0000 7f5349457640 1 --2- 172.21.15.98:0/1357070788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5344100a80 0x7f5344198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:45.065+0000 7f5349457640 1 -- 172.21.15.98:0/1357070788 >> 172.21.15.98:0/1357070788 conn(0x7f53440fc7a0 msgr2=0x7f534406aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:45.065+0000 7f5349457640 1 -- 172.21.15.98:0/1357070788 shutdown_connections 2024-09-19T08:02:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:45.065+0000 7f5349457640 1 -- 172.21.15.98:0/1357070788 wait complete. 2024-09-19T08:02:45.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:44 smithi162 ceph-mon[35260]: pgmap v319: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:45.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:44 smithi139 ceph-mon[34874]: pgmap v319: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:45.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T08:02:45.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:45 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1357070788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:45.932 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:46.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:45 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1357070788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:46.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:45 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1357070788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:02:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.395+0000 7f29f68b3640 1 -- 172.21.15.98:0/4249822641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f0101de0 msgr2=0x7f29f01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.395+0000 7f29f68b3640 1 --2- 172.21.15.98:0/4249822641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f0101de0 0x7f29f01021c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009900 tx=0x7f29e002a410 comp rx=0 tx=0).stop 2024-09-19T08:02:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.395+0000 7f29f68b3640 1 -- 172.21.15.98:0/4249822641 shutdown_connections 2024-09-19T08:02:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.395+0000 7f29f68b3640 1 --2- 172.21.15.98:0/4249822641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29f0106f10 0x7f29f010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.395+0000 7f29f68b3640 1 --2- 172.21.15.98:0/4249822641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29f0102790 0x7f29f01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.395+0000 7f29f68b3640 1 --2- 172.21.15.98:0/4249822641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f0101de0 0x7f29f01021c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.395+0000 7f29f68b3640 1 -- 172.21.15.98:0/4249822641 >> 172.21.15.98:0/4249822641 conn(0x7f29f00fd8c0 msgr2=0x7f29f00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.395+0000 7f29f68b3640 1 -- 172.21.15.98:0/4249822641 shutdown_connections 2024-09-19T08:02:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.395+0000 7f29f68b3640 1 -- 172.21.15.98:0/4249822641 wait complete. 2024-09-19T08:02:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.395+0000 7f29f68b3640 1 Processor -- start 2024-09-19T08:02:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.395+0000 7f29f68b3640 1 -- start start 2024-09-19T08:02:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29f68b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29f0101de0 0x7f29f00744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29f68b3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f0102790 0x7f29f0074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29f68b3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29f0106f10 0x7f29f0074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29f68b3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f29f0112010 con 0x7f29f0102790 2024-09-19T08:02:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29f68b3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f29f0111e90 con 0x7f29f0106f10 2024-09-19T08:02:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29f68b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f29f0112190 con 0x7f29f0101de0 2024-09-19T08:02:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29e7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f0102790 0x7f29f0074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29e7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f0102790 0x7f29f0074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49658/0 (socket says 172.21.15.98:49658) 2024-09-19T08:02:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29e7fff640 1 -- 172.21.15.98:0/4084794661 learned_addr learned my addr 172.21.15.98:0/4084794661 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29f4e50640 1 --2- 172.21.15.98:0/4084794661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29f0106f10 0x7f29f0074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29effff640 1 --2- 172.21.15.98:0/4084794661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29f0101de0 0x7f29f00744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:46.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29e7fff640 1 -- 172.21.15.98:0/4084794661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29f0101de0 msgr2=0x7f29f00744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:46.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29e7fff640 1 --2- 172.21.15.98:0/4084794661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29f0101de0 0x7f29f00744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:46.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29e7fff640 1 -- 172.21.15.98:0/4084794661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29f0106f10 msgr2=0x7f29f0074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:46.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29e7fff640 1 --2- 172.21.15.98:0/4084794661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29f0106f10 0x7f29f0074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:46.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29e7fff640 1 -- 172.21.15.98:0/4084794661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f01a6a70 con 0x7f29f0102790 2024-09-19T08:02:46.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29e7fff640 1 --2- 172.21.15.98:0/4084794661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f0102790 0x7f29f0074a30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f29d800b4a0 tx=0x7f29d800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:46.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29edffb640 1 -- 172.21.15.98:0/4084794661 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d8014070 con 0x7f29f0102790 2024-09-19T08:02:46.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29edffb640 1 -- 172.21.15.98:0/4084794661 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29d8010af0 con 0x7f29f0102790 2024-09-19T08:02:46.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29f68b3640 1 -- 172.21.15.98:0/4084794661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29f01a6d60 con 0x7f29f0102790 2024-09-19T08:02:46.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29edffb640 1 -- 172.21.15.98:0/4084794661 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d8017d10 con 0x7f29f0102790 2024-09-19T08:02:46.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.396+0000 7f29f68b3640 1 -- 172.21.15.98:0/4084794661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29f01a71f0 con 0x7f29f0102790 2024-09-19T08:02:46.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.397+0000 7f29f68b3640 1 -- 172.21.15.98:0/4084794661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29f0102240 con 0x7f29f0102790 2024-09-19T08:02:46.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.399+0000 7f29edffb640 1 -- 172.21.15.98:0/4084794661 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f29d8017070 con 0x7f29f0102790 2024-09-19T08:02:46.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.399+0000 7f29edffb640 1 --2- 172.21.15.98:0/4084794661 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f29dc0a32d0 0x7f29dc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:46.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.399+0000 7f29effff640 1 --2- 172.21.15.98:0/4084794661 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f29dc0a32d0 0x7f29dc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:46.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.399+0000 7f29edffb640 1 -- 172.21.15.98:0/4084794661 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f29d80cf1a0 con 0x7f29f0102790 2024-09-19T08:02:46.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.400+0000 7f29effff640 1 --2- 172.21.15.98:0/4084794661 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f29dc0a32d0 0x7f29dc0a5790 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f29e00096c0 tx=0x7f29e0031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:46.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.403+0000 7f29edffb640 1 -- 172.21.15.98:0/4084794661 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f29d809be80 con 0x7f29f0102790 2024-09-19T08:02:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.571+0000 7f29f68b3640 1 -- 172.21.15.98:0/4084794661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29f0075a20 con 0x7f29f0102790 2024-09-19T08:02:46.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.572+0000 7f29edffb640 1 -- 172.21.15.98:0/4084794661 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22754 (secure 0 0 0) 0x7f29f0075a20 con 0x7f29f0102790 2024-09-19T08:02:46.572 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:02:46.572 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":80,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","created":"2024-09-19T07:53:39.127398+0000","modified":"2024-09-19T08:01:26.714628+0000","last_up_change":"2024-09-19T08:00:22.467777+0000","last_in_change":"2024-09-19T07:59:58.538492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:56:43.773892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T08:01:07.693796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T08:01:09.518028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d01ce6-165d-493d-ae98-d8ad9e5d4134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6803","nonce":1719360533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6805","nonce":1719360533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6809","nonce":1719360533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6807","nonce":1719360533}]},"public_addr":"172.21.15.98:6803/1719360533","cluster_addr":"172.21.15.98:6805/1719360533","heartbeat_back_addr":"172.21.15.98:6809/1719360533","heartbeat_front_addr":"172.21.15.98:6807/1719360533","state":["exists","up"]},{"osd":1,"uuid":"34724bd5-0738-4473-b087-c09031c1ee2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036962626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036962626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036962626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036962626}]},"public_addr":"172.21.15.139:6801/4036962626","cluster_addr":"172.21.15.139:6803/4036962626","heartbeat_back_addr":"172.21.15.139:6807/4036962626","heartbeat_front_addr":"172.21.15.139:6805/4036962626","state":["exists","up"]},{"osd":2,"uuid":"16985ebe-0e32-42ee-bded-b7838e290afc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6801","nonce":2811412891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6803","nonce":2811412891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6807","nonce":2811412891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6805","nonce":2811412891}]},"public_addr":"172.21.15.162:6801/2811412891","cluster_addr":"172.21.15.162:6803/2811412891","heartbeat_back_addr":"172.21.15.162:6807/2811412891","heartbeat_front_addr":"172.21.15.162:6805/2811412891","state":["exists","up"]},{"osd":3,"uuid":"b140e466-3119-49cb-aec2-ba6ca4d10d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6811","nonce":1383392014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6813","nonce":1383392014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6817","nonce":1383392014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6815","nonce":1383392014}]},"public_addr":"172.21.15.98:6811/1383392014","cluster_addr":"172.21.15.98:6813/1383392014","heartbeat_back_addr":"172.21.15.98:6817/1383392014","heartbeat_front_addr":"172.21.15.98:6815/1383392014","state":["exists","up"]},{"osd":4,"uuid":"b4412813-df57-4ac6-8fb7-f57b69c74768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6809","nonce":116453520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6811","nonce":116453520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6815","nonce":116453520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6813","nonce":116453520}]},"public_addr":"172.21.15.139:6809/116453520","cluster_addr":"172.21.15.139:6811/116453520","heartbeat_back_addr":"172.21.15.139:6815/116453520","heartbeat_front_addr":"172.21.15.139:6813/116453520","state":["exists","up"]},{"osd":5,"uuid":"d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6809","nonce":3769530297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6811","nonce":3769530297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6815","nonce":3769530297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6813","nonce":3769530297}]},"public_addr":"172.21.15.162:6809/3769530297","cluster_addr":"172.21.15.162:6811/3769530297","heartbeat_back_addr":"172.21.15.162:6815/3769530297","heartbeat_front_addr":"172.21.15.162:6813/3769530297","state":["exists","up"]},{"osd":6,"uuid":"ba1c1590-6aea-40dc-90c8-e21b88896909","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6819","nonce":692373267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6821","nonce":692373267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6825","nonce":692373267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6823","nonce":692373267}]},"public_addr":"172.21.15.98:6819/692373267","cluster_addr":"172.21.15.98:6821/692373267","heartbeat_back_addr":"172.21.15.98:6825/692373267","heartbeat_front_addr":"172.21.15.98:6823/692373267","state":["exists","up"]},{"osd":7,"uuid":"eb9093ad-a068-4a63-ab87-0b50522cbc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6817","nonce":1963811650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6819","nonce":1963811650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6823","nonce":1963811650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6821","nonce":1963811650}]},"public_addr":"172.21.15.139:6817/1963811650","cluster_addr":"172.21.15.139:6819/1963811650","heartbeat_back_addr":"172.21.15.139:6823/1963811650","heartbeat_front_addr":"172.21.15.139:6821/1963811650","state":["exists","up"]},{"osd":8,"uuid":"31ca330a-fc45-466d-81c2-ef4386b3cc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6817","nonce":3607966458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6819","nonce":3607966458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6823","nonce":3607966458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6821","nonce":3607966458}]},"public_addr":"172.21.15.162:6817/3607966458","cluster_addr":"172.21.15.162:6819/3607966458","heartbeat_back_addr":"172.21.15.162:6823/3607966458","heartbeat_front_addr":"172.21.15.162:6821/3607966458","state":["exists","up"]},{"osd":9,"uuid":"9c4f216f-aef1-4367-9c58-b913b0c4f7db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6827","nonce":2050391244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6829","nonce":2050391244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6833","nonce":2050391244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6831","nonce":2050391244}]},"public_addr":"172.21.15.98:6827/2050391244","cluster_addr":"172.21.15.98:6829/2050391244","heartbeat_back_addr":"172.21.15.98:6833/2050391244","heartbeat_front_addr":"172.21.15.98:6831/2050391244","state":["exists","up"]},{"osd":10,"uuid":"03cc7df6-84ae-4d9a-a3d7-74c91e1744e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6825","nonce":2336761639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6827","nonce":2336761639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6831","nonce":2336761639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6829","nonce":2336761639}]},"public_addr":"172.21.15.139:6825/2336761639","cluster_addr":"172.21.15.139:6827/2336761639","heartbeat_back_addr":"172.21.15.139:6831/2336761639","heartbeat_front_addr":"172.21.15.139:6829/2336761639","state":["exists","up"]},{"osd":11,"uuid":"29e95498-4952-46d4-8fa6-6d8e2b862111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6825","nonce":1231011554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6827","nonce":1231011554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6831","nonce":1231011554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6829","nonce":1231011554}]},"public_addr":"172.21.15.162:6825/1231011554","cluster_addr":"172.21.15.162:6827/1231011554","heartbeat_back_addr":"172.21.15.162:6831/1231011554","heartbeat_front_addr":"172.21.15.162:6829/1231011554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:55:49.995244+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:15.655543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:40.331165+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:04.688881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:29.174966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:53.663161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:17.407551+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:42.584296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:06.317219+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:31.078625+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:55.689269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T08:00:19.659748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1},{"from":7,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/3842627255":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3838590070":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/1238193420":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6800/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6800/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3194864054":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3866031018":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/1497156658":"2024-09-20T07:54:02.418785+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T08:02:46.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.574+0000 7f29f68b3640 1 -- 172.21.15.98:0/4084794661 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f29dc0a32d0 msgr2=0x7f29dc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:46.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.574+0000 7f29f68b3640 1 --2- 172.21.15.98:0/4084794661 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f29dc0a32d0 0x7f29dc0a5790 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f29e00096c0 tx=0x7f29e0031490 comp rx=0 tx=0).stop 2024-09-19T08:02:46.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.574+0000 7f29f68b3640 1 -- 172.21.15.98:0/4084794661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f0102790 msgr2=0x7f29f0074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:46.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.574+0000 7f29f68b3640 1 --2- 172.21.15.98:0/4084794661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f0102790 0x7f29f0074a30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f29d800b4a0 tx=0x7f29d800b960 comp rx=0 tx=0).stop 2024-09-19T08:02:46.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.574+0000 7f29f68b3640 1 -- 172.21.15.98:0/4084794661 shutdown_connections 2024-09-19T08:02:46.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.574+0000 7f29f68b3640 1 --2- 172.21.15.98:0/4084794661 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f29dc0a32d0 0x7f29dc0a5790 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:46.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.574+0000 7f29f68b3640 1 --2- 172.21.15.98:0/4084794661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29f0106f10 0x7f29f0074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:46.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.574+0000 7f29f68b3640 1 --2- 172.21.15.98:0/4084794661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f0102790 0x7f29f0074a30 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:46.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.574+0000 7f29f68b3640 1 --2- 172.21.15.98:0/4084794661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29f0101de0 0x7f29f00744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:46.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.574+0000 7f29f68b3640 1 -- 172.21.15.98:0/4084794661 >> 172.21.15.98:0/4084794661 conn(0x7f29f00fd8c0 msgr2=0x7f29f00fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:46.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.574+0000 7f29f68b3640 1 -- 172.21.15.98:0/4084794661 shutdown_connections 2024-09-19T08:02:46.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:46.574+0000 7f29f68b3640 1 -- 172.21.15.98:0/4084794661 wait complete. 2024-09-19T08:02:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:46 smithi098 ceph-mon[31333]: pgmap v320: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:46.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:46 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/4084794661' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:02:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:46 smithi162 ceph-mon[35260]: pgmap v320: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:46 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/4084794661' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:02:47.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:46 smithi139 ceph-mon[34874]: pgmap v320: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:47.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:46 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/4084794661' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:02:47.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-19T08:01:11.307169+0000', 'modified': '2024-09-19T08:01:22.699569+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24485, 'mds_1': 24461, 'mds_2': 24437, 'mds_3': 24406, 'mds_4': 24376}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14631': {'gid': 14631, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6839/3417988465', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 3417988465}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 3417988465}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14643': {'gid': 14643, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6841/1339458935', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6840', 'nonce': 1339458935}, {'type': 'v1', 'addr': '172.21.15.98:6841', 'nonce': 1339458935}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24376': {'gid': 24376, 'name': 'l', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.162:6833/3959364141', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6832', 'nonce': 3959364141}, {'type': 'v1', 'addr': '172.21.15.162:6833', 'nonce': 3959364141}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24406': {'gid': 24406, 'name': 'f', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.162:6835/1149502202', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6834', 'nonce': 1149502202}, {'type': 'v1', 'addr': '172.21.15.162:6835', 'nonce': 1149502202}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24413': {'gid': 24413, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6833/2267154969', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6832', 'nonce': 2267154969}, {'type': 'v1', 'addr': '172.21.15.139:6833', 'nonce': 2267154969}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24415': {'gid': 24415, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6837/3969500353', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6836', 'nonce': 3969500353}, {'type': 'v1', 'addr': '172.21.15.162:6837', 'nonce': 3969500353}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6839/3810709211', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6838', 'nonce': 3810709211}, {'type': 'v1', 'addr': '172.21.15.162:6839', 'nonce': 3810709211}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.139:6835/1275838513', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6834', 'nonce': 1275838513}, {'type': 'v1', 'addr': '172.21.15.139:6835', 'nonce': 1275838513}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.139:6837/1885641254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6836', 'nonce': 1885641254}, {'type': 'v1', 'addr': '172.21.15.139:6837', 'nonce': 1885641254}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.139:6839/1001534713', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6838', 'nonce': 1001534713}, {'type': 'v1', 'addr': '172.21.15.139:6839', 'nonce': 1001534713}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24485, 'qdb_cluster': [24485, 24461, 24437, 24406, 24376]} 2024-09-19T08:02:47.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-19T08:02:47.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.l status 2024-09-19T08:02:47.433 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:47.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.954+0000 7fb5882ce640 1 -- 172.21.15.98:0/1065685109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb58010c180 msgr2=0x7fb58010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:47.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.954+0000 7fb5882ce640 1 --2- 172.21.15.98:0/1065685109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb58010c180 0x7fb58010e610 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb578009ca0 tx=0x7fb57802a950 comp rx=0 tx=0).stop 2024-09-19T08:02:47.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.954+0000 7fb5882ce640 1 -- 172.21.15.98:0/1065685109 shutdown_connections 2024-09-19T08:02:47.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.954+0000 7fb5882ce640 1 --2- 172.21.15.98:0/1065685109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb58010c180 0x7fb58010e610 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:47.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.954+0000 7fb5882ce640 1 --2- 172.21.15.98:0/1065685109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb580103700 0x7fb58010bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:47.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.954+0000 7fb5882ce640 1 --2- 172.21.15.98:0/1065685109 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb580102de0 0x7fb5801031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:47.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.954+0000 7fb5882ce640 1 -- 172.21.15.98:0/1065685109 >> 172.21.15.98:0/1065685109 conn(0x7fb5800fc7a0 msgr2=0x7fb5800febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:47.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.954+0000 7fb5882ce640 1 -- 172.21.15.98:0/1065685109 shutdown_connections 2024-09-19T08:02:47.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.954+0000 7fb5882ce640 1 -- 172.21.15.98:0/1065685109 wait complete. 2024-09-19T08:02:47.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb5882ce640 1 Processor -- start 2024-09-19T08:02:47.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb5882ce640 1 -- start start 2024-09-19T08:02:47.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb5882ce640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb580102de0 0x7fb580100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:47.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb5882ce640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb580103700 0x7fb580100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:47.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb5882ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb58010c180 0x7fb5801014c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:47.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb5882ce640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb580111f70 con 0x7fb58010c180 2024-09-19T08:02:47.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb5882ce640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb580111df0 con 0x7fb580102de0 2024-09-19T08:02:47.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb5882ce640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb5801120f0 con 0x7fb580103700 2024-09-19T08:02:47.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb58686b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb58010c180 0x7fb5801014c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:47.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb58686b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb58010c180 0x7fb5801014c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49670/0 (socket says 172.21.15.98:49670) 2024-09-19T08:02:47.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb58686b640 1 -- 172.21.15.98:0/312553502 learned_addr learned my addr 172.21.15.98:0/312553502 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:47.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb58606a640 1 --2- 172.21.15.98:0/312553502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb580102de0 0x7fb580100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:47.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb58686b640 1 -- 172.21.15.98:0/312553502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb580103700 msgr2=0x7fb580100f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:47.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb58686b640 1 --2- 172.21.15.98:0/312553502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb580103700 0x7fb580100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:47.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb58686b640 1 -- 172.21.15.98:0/312553502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb580102de0 msgr2=0x7fb580100a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:47.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb58686b640 1 --2- 172.21.15.98:0/312553502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb580102de0 0x7fb580100a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:47.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb58686b640 1 -- 172.21.15.98:0/312553502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5801a6890 con 0x7fb58010c180 2024-09-19T08:02:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb58606a640 1 --2- 172.21.15.98:0/312553502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb580102de0 0x7fb580100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.955+0000 7fb58686b640 1 --2- 172.21.15.98:0/312553502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb58010c180 0x7fb5801014c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb57802ae70 tx=0x7fb57802f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.956+0000 7fb5737fe640 1 -- 172.21.15.98:0/312553502 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb578034040 con 0x7fb58010c180 2024-09-19T08:02:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.956+0000 7fb5737fe640 1 -- 172.21.15.98:0/312553502 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb578030920 con 0x7fb58010c180 2024-09-19T08:02:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.956+0000 7fb5882ce640 1 -- 172.21.15.98:0/312553502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5801a6b20 con 0x7fb58010c180 2024-09-19T08:02:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.956+0000 7fb5737fe640 1 -- 172.21.15.98:0/312553502 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb578045480 con 0x7fb58010c180 2024-09-19T08:02:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.956+0000 7fb5882ce640 1 -- 172.21.15.98:0/312553502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5801a7000 con 0x7fb58010c180 2024-09-19T08:02:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.958+0000 7fb5737fe640 1 -- 172.21.15.98:0/312553502 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fb578037020 con 0x7fb58010c180 2024-09-19T08:02:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.958+0000 7fb58a6a7740 1 Processor -- start 2024-09-19T08:02:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.958+0000 7fb58a6a7740 1 -- start start 2024-09-19T08:02:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb58a6a7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bc2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb58a6a7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 0x55e503bcd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:47.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb58a6a7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503bcffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb58a6a7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e503b29090 con 0x55e503bc29d0 2024-09-19T08:02:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb58a6a7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e503ad0a70 con 0x55e503bc32f0 2024-09-19T08:02:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb58a6a7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e503b25050 con 0x55e503bcdbb0 2024-09-19T08:02:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb585869640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bc2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb585869640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bc2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49678/0 (socket says 172.21.15.98:49678) 2024-09-19T08:02:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb585869640 1 -- 172.21.15.98:0/2416761208 learned_addr learned my addr 172.21.15.98:0/2416761208 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb5737fe640 1 --2- 172.21.15.98:0/312553502 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb56c0a3350 0x7fb56c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb58606a640 1 --2- 172.21.15.98:0/2416761208 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 0x55e503bcd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb58686b640 1 --2- 172.21.15.98:0/312553502 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb56c0a3350 0x7fb56c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb585869640 1 --2- 172.21.15.98:0/2416761208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503bcffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb585869640 1 -- 172.21.15.98:0/2416761208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 msgr2=0x55e503bcffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb585869640 1 --2- 172.21.15.98:0/2416761208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503bcffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:47.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb585869640 1 -- 172.21.15.98:0/2416761208 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 msgr2=0x55e503bcd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:47.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb585869640 1 --2- 172.21.15.98:0/2416761208 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 0x55e503bcd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:47.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb585869640 1 -- 172.21.15.98:0/2416761208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e503add130 con 0x55e503bc29d0 2024-09-19T08:02:47.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb5737fe640 1 -- 172.21.15.98:0/312553502 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fb5780f0a00 con 0x7fb58010c180 2024-09-19T08:02:47.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb585869640 1 --2- 172.21.15.98:0/2416761208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bc2db0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb57c005fd0 tx=0x7fb57c00b470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0dafc670e785cf6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:47.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb58686b640 1 --2- 172.21.15.98:0/312553502 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb56c0a3350 0x7fb56c0a5810 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb5780b9950 tx=0x7fb5780ee040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:47.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.959+0000 7fb570dea640 1 -- 172.21.15.98:0/2416761208 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c016030 con 0x55e503bc29d0 2024-09-19T08:02:47.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.960+0000 7fb570dea640 1 -- 172.21.15.98:0/2416761208 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb57c00fce0 con 0x55e503bc29d0 2024-09-19T08:02:47.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.960+0000 7fb570dea640 1 -- 172.21.15.98:0/2416761208 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c011600 con 0x55e503bc29d0 2024-09-19T08:02:47.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.960+0000 7fb58a6a7740 1 -- 172.21.15.98:0/2416761208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 msgr2=0x55e503bc2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:47.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.960+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/2416761208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bc2db0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb57c005fd0 tx=0x7fb57c00b470 comp rx=0 tx=0).stop 2024-09-19T08:02:47.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.960+0000 7fb58a6a7740 1 -- 172.21.15.98:0/2416761208 shutdown_connections 2024-09-19T08:02:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.960+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/2416761208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503bcffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.960+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/2416761208 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 0x55e503bcd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.960+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/2416761208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bc2db0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.960+0000 7fb58a6a7740 1 -- 172.21.15.98:0/2416761208 >> 172.21.15.98:0/2416761208 conn(0x55e503bbef60 msgr2=0x55e503bbf340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.960+0000 7fb58a6a7740 1 -- 172.21.15.98:0/2416761208 shutdown_connections 2024-09-19T08:02:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.960+0000 7fb58a6a7740 1 -- 172.21.15.98:0/2416761208 wait complete. 2024-09-19T08:02:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb58a6a7740 1 -- start start 2024-09-19T08:02:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb58a6a7740 1 Processor -- start 2024-09-19T08:02:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb58a6a7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bcd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb58a6a7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 0x55e503bca760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb58a6a7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503bcaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb58a6a7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e503bd0780 con 0x55e503bc29d0 2024-09-19T08:02:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb58a6a7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e503ae3f20 con 0x55e503bc32f0 2024-09-19T08:02:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb58a6a7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e503bd0900 con 0x55e503bcdbb0 2024-09-19T08:02:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb58a6a7740 10 client.0 _ll_register_callbacks cb 0x55e503aad030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb585869640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bcd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb585869640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bcd410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49684/0 (socket says 172.21.15.98:49684) 2024-09-19T08:02:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb585869640 1 -- 172.21.15.98:0/2588839652 learned_addr learned my addr 172.21.15.98:0/2588839652 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb585869640 1 -- 172.21.15.98:0/2588839652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 msgr2=0x55e503bcaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb585869640 1 --2- 172.21.15.98:0/2588839652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503bcaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb585869640 1 -- 172.21.15.98:0/2588839652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 msgr2=0x55e503bca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.961+0000 7fb585869640 1 --2- 172.21.15.98:0/2588839652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 0x55e503bca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb585869640 1 -- 172.21.15.98:0/2588839652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e50389a2b0 con 0x55e503bc29d0 2024-09-19T08:02:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb585869640 1 --2- 172.21.15.98:0/2588839652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bcd410 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb57c012ac0 tx=0x7fb57c012af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c2d602b1d5d04e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb569ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb569ffb640 1 -- 172.21.15.98:0/2588839652 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c016030 con 0x55e503bc29d0 2024-09-19T08:02:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb569ffb640 1 -- 172.21.15.98:0/2588839652 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fb57c005220 con 0x55e503bc29d0 2024-09-19T08:02:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb58a6a7740 1 -- 172.21.15.98:0/2588839652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb57c00fce0 con 0x55e503bc29d0 2024-09-19T08:02:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb569ffb640 1 -- 172.21.15.98:0/2588839652 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb57c00b830 con 0x55e503bc29d0 2024-09-19T08:02:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb58a6a7740 10 client.15150 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb58a6a7740 1 -- 172.21.15.98:0/2588839652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e503bdc0a0 con 0x55e503bc29d0 2024-09-19T08:02:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb569ffb640 1 -- 172.21.15.98:0/2588839652 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb57c00fce0 con 0x55e503bc29d0 2024-09-19T08:02:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb569ffb640 1 -- 172.21.15.98:0/2588839652 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fb57c023070 con 0x55e503bc29d0 2024-09-19T08:02:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb58a6a7740 10 client.15150 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb58a6a7740 10 client.15150 resolve_mds: resolved name 'l' to daemon mds.l 2024-09-19T08:02:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb58a6a7740 20 client.15150 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/2588839652 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e503bde500 0x55e503bdf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb58a6a7740 4 client.15150 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:02:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.962+0000 7fb58a6a7740 1 -- 172.21.15.98:0/2588839652 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e503b36600 con 0x55e503bde500 2024-09-19T08:02:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.963+0000 7fb58606a640 1 --2- 172.21.15.98:0/2588839652 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e503bde500 0x55e503bdf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.963+0000 7fb569ffb640 10 client.15150 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:02:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.963+0000 7fb58606a640 1 --2- 172.21.15.98:0/2588839652 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e503bde500 0x55e503bdf970 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=14659bb6e40dfe79 server_cookie=594de4132c3b4a57 in_seq=0 out_seq=0 2024-09-19T08:02:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb569ffb640 1 -- 172.21.15.98:0/2588839652 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e503b36600 con 0x55e503bde500 2024-09-19T08:02:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb569ffb640 10 client.15150 handle_command_reply: tid=0 2024-09-19T08:02:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb58a6a7740 1 client.15150 shutdown 2024-09-19T08:02:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb58a6a7740 1 -- 172.21.15.98:0/2588839652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 msgr2=0x55e503bcd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/2588839652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bcd410 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb57c012ac0 tx=0x7fb57c012af0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb58a6a7740 1 -- 172.21.15.98:0/2588839652 shutdown_connections 2024-09-19T08:02:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/2588839652 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e503bde500 0x55e503bdf970 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/2588839652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503bcaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/2588839652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 0x55e503bca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb569ffb640 0 client.15150 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:02:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/2588839652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bcd410 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb58a6a7740 1 -- 172.21.15.98:0/2588839652 >> 172.21.15.98:0/2588839652 conn(0x55e503bbef60 msgr2=0x55e503bc4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb58a6a7740 1 -- 172.21.15.98:0/2588839652 shutdown_connections 2024-09-19T08:02:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb58a6a7740 1 -- 172.21.15.98:0/2588839652 wait complete. 2024-09-19T08:02:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.964+0000 7fb58a6a7740 20 client.15150 trim_cache size 0 max 16384 2024-09-19T08:02:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb58a6a7740 1 Processor -- start 2024-09-19T08:02:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb58a6a7740 1 -- start start 2024-09-19T08:02:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb58a6a7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bdebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb58a6a7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 0x55e503bdd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb58a6a7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503bc9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb58a6a7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e503bddcb0 con 0x55e503bc29d0 2024-09-19T08:02:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb58a6a7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e503bde380 con 0x55e503bc32f0 2024-09-19T08:02:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb58a6a7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e503bdea50 con 0x55e503bcdbb0 2024-09-19T08:02:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb585869640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bdebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb585869640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bdebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49686/0 (socket says 172.21.15.98:49686) 2024-09-19T08:02:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb585869640 1 -- 172.21.15.98:0/931121158 learned_addr learned my addr 172.21.15.98:0/931121158 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb585869640 1 --2- 172.21.15.98:0/931121158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503bc9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb585869640 1 -- 172.21.15.98:0/931121158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 msgr2=0x55e503bc9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb585869640 1 --2- 172.21.15.98:0/931121158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503bc9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb58606a640 1 --2- 172.21.15.98:0/931121158 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 0x55e503bdd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb585869640 1 -- 172.21.15.98:0/931121158 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 msgr2=0x55e503bdd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb585869640 1 --2- 172.21.15.98:0/931121158 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 0x55e503bdd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb585869640 1 -- 172.21.15.98:0/931121158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e503bc3120 con 0x55e503bc29d0 2024-09-19T08:02:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb58606a640 1 --2- 172.21.15.98:0/931121158 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 0x55e503bdd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:48.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb585869640 1 --2- 172.21.15.98:0/931121158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bdebd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb57c004700 tx=0x7fb57c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f6046c9d6321d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:48.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.982+0000 7fb56bfff640 1 -- 172.21.15.98:0/931121158 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c00b3d0 con 0x55e503bc29d0 2024-09-19T08:02:48.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.983+0000 7fb56bfff640 1 -- 172.21.15.98:0/931121158 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb57c02a030 con 0x55e503bc29d0 2024-09-19T08:02:48.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.983+0000 7fb56bfff640 1 -- 172.21.15.98:0/931121158 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c02eb30 con 0x55e503bc29d0 2024-09-19T08:02:48.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.983+0000 7fb58a6a7740 1 -- 172.21.15.98:0/931121158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 msgr2=0x55e503bdebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:48.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.983+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/931121158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bdebd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb57c004700 tx=0x7fb57c000bc0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.983+0000 7fb58a6a7740 1 -- 172.21.15.98:0/931121158 shutdown_connections 2024-09-19T08:02:48.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.983+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/931121158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503bc9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.983+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/931121158 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc32f0 0x55e503bdd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.983+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/931121158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc29d0 0x55e503bdebd0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.983+0000 7fb58a6a7740 1 -- 172.21.15.98:0/931121158 >> 172.21.15.98:0/931121158 conn(0x55e503bbef60 msgr2=0x55e503bb9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:48.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.983+0000 7fb58a6a7740 1 -- 172.21.15.98:0/931121158 shutdown_connections 2024-09-19T08:02:48.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.983+0000 7fb58a6a7740 1 -- 172.21.15.98:0/931121158 wait complete. 2024-09-19T08:02:48.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.983+0000 7fb58a6a7740 1 -- start start 2024-09-19T08:02:48.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58a6a7740 1 Processor -- start 2024-09-19T08:02:48.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58a6a7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc29d0 0x55e503be3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:48.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58a6a7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc32f0 0x55e503be3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:48.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58a6a7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503c46c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:48.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58a6a7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e503bdf7f0 con 0x55e503bc32f0 2024-09-19T08:02:48.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58a6a7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e503bdf120 con 0x55e503bc29d0 2024-09-19T08:02:48.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58a6a7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e503bdfec0 con 0x55e503bcdbb0 2024-09-19T08:02:48.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58a6a7740 10 client.0 _ll_register_callbacks cb 0x7fb57400b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:48.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58606a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc32f0 0x55e503be3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:48.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58606a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc32f0 0x55e503be3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49688/0 (socket says 172.21.15.98:49688) 2024-09-19T08:02:48.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58606a640 1 -- 172.21.15.98:0/31627998 learned_addr learned my addr 172.21.15.98:0/31627998 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:48.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb585869640 1 --2- 172.21.15.98:0/31627998 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc29d0 0x55e503be3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:48.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb585869640 1 --2- 172.21.15.98:0/31627998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503c46c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:48.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58606a640 1 -- 172.21.15.98:0/31627998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 msgr2=0x55e503c46c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:48.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58606a640 1 --2- 172.21.15.98:0/31627998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503c46c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58606a640 1 -- 172.21.15.98:0/31627998 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc29d0 msgr2=0x55e503be3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:48.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58606a640 1 --2- 172.21.15.98:0/31627998 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc29d0 0x55e503be3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58606a640 1 -- 172.21.15.98:0/31627998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e503c47140 con 0x55e503bc32f0 2024-09-19T08:02:48.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb585869640 1 --2- 172.21.15.98:0/31627998 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc29d0 0x55e503be3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:48.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58606a640 1 --2- 172.21.15.98:0/31627998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc32f0 0x55e503be3660 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x55e503bbfdc0 tx=0x7fb574004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c03c090a4bb38d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:48.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb569ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:48.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb569ffb640 1 -- 172.21.15.98:0/31627998 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574011440 con 0x55e503bc32f0 2024-09-19T08:02:48.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb569ffb640 1 -- 172.21.15.98:0/31627998 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fb5740115e0 con 0x55e503bc32f0 2024-09-19T08:02:48.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.984+0000 7fb58a6a7740 1 -- 172.21.15.98:0/31627998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb57c02a030 con 0x55e503bc32f0 2024-09-19T08:02:48.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.985+0000 7fb569ffb640 1 -- 172.21.15.98:0/31627998 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb574029050 con 0x55e503bc32f0 2024-09-19T08:02:48.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.985+0000 7fb58a6a7740 10 client.15156 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:48.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.985+0000 7fb58a6a7740 1 -- 172.21.15.98:0/31627998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e503c483c0 con 0x55e503bc32f0 2024-09-19T08:02:48.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.985+0000 7fb569ffb640 1 -- 172.21.15.98:0/31627998 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb57c02a030 con 0x55e503bc32f0 2024-09-19T08:02:48.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.985+0000 7fb569ffb640 1 -- 172.21.15.98:0/31627998 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fb5740229a0 con 0x55e503bc32f0 2024-09-19T08:02:48.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.985+0000 7fb58a6a7740 10 client.15156 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:48.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.985+0000 7fb58a6a7740 10 client.15156 resolve_mds: resolved name 'l' to daemon mds.l 2024-09-19T08:02:48.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.985+0000 7fb58a6a7740 20 client.15156 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:48.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.985+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/31627998 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e503c4cfc0 0x55e503c4bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:48.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.985+0000 7fb58a6a7740 4 client.15156 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:02:48.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.985+0000 7fb58a6a7740 1 -- 172.21.15.98:0/31627998 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "status"}) -- 0x55e503b30b10 con 0x55e503c4cfc0 2024-09-19T08:02:48.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.985+0000 7fb585869640 1 --2- 172.21.15.98:0/31627998 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e503c4cfc0 0x55e503c4bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb569ffb640 10 client.15156 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:02:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb585869640 1 --2- 172.21.15.98:0/31627998 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e503c4cfc0 0x55e503c4bcd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a9cdc0df5f2eda79 server_cookie=271dd93d14cbae0e in_seq=0 out_seq=0 2024-09-19T08:02:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb569ffb640 1 -- 172.21.15.98:0/31627998 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55e503b30b10 con 0x55e503c4cfc0 2024-09-19T08:02:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb569ffb640 10 client.15156 handle_command_reply: tid=0 2024-09-19T08:02:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb58a6a7740 1 client.15156 shutdown 2024-09-19T08:02:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb58a6a7740 1 -- 172.21.15.98:0/31627998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc32f0 msgr2=0x55e503be3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/31627998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc32f0 0x55e503be3660 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x55e503bbfdc0 tx=0x7fb574004410 comp rx=0 tx=0).stop 2024-09-19T08:02:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb58a6a7740 1 -- 172.21.15.98:0/31627998 shutdown_connections 2024-09-19T08:02:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/31627998 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e503c4cfc0 0x55e503c4bcd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/31627998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e503bcdbb0 0x55e503c46c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/31627998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e503bc32f0 0x55e503be3660 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb58a6a7740 1 --2- 172.21.15.98:0/31627998 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e503bc29d0 0x55e503be3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb569ffb640 0 client.15156 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:02:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb58a6a7740 1 -- 172.21.15.98:0/31627998 >> 172.21.15.98:0/31627998 conn(0x55e503bbef60 msgr2=0x55e503bdd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:48.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb58a6a7740 1 -- 172.21.15.98:0/31627998 shutdown_connections 2024-09-19T08:02:48.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb58a6a7740 1 -- 172.21.15.98:0/31627998 wait complete. 2024-09-19T08:02:48.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.986+0000 7fb58a6a7740 20 client.15156 trim_cache size 0 max 16384 2024-09-19T08:02:48.008 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:02:48.008 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:02:48.008 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 4, 2024-09-19T08:02:48.008 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24376, 2024-09-19T08:02:48.009 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:02:48.009 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:02:48.009 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:02:48.009 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 86.295629753, 2024-09-19T08:02:48.009 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 27, 2024-09-19T08:02:48.009 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 80, 2024-09-19T08:02:48.009 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 78, 2024-09-19T08:02:48.009 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 138.04709436900001 2024-09-19T08:02:48.009 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:02:48.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.987+0000 7fb5882ce640 1 -- 172.21.15.98:0/312553502 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb56c0a3350 msgr2=0x7fb56c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:48.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.987+0000 7fb5882ce640 1 --2- 172.21.15.98:0/312553502 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb56c0a3350 0x7fb56c0a5810 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb5780b9950 tx=0x7fb5780ee040 comp rx=0 tx=0).stop 2024-09-19T08:02:48.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.987+0000 7fb5882ce640 1 -- 172.21.15.98:0/312553502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb58010c180 msgr2=0x7fb5801014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:48.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.987+0000 7fb5882ce640 1 --2- 172.21.15.98:0/312553502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb58010c180 0x7fb5801014c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb57802ae70 tx=0x7fb57802f5b0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.987+0000 7fb5882ce640 1 -- 172.21.15.98:0/312553502 shutdown_connections 2024-09-19T08:02:48.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.987+0000 7fb5882ce640 1 --2- 172.21.15.98:0/312553502 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fb56c0a3350 0x7fb56c0a5810 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.987+0000 7fb5882ce640 1 --2- 172.21.15.98:0/312553502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb58010c180 0x7fb5801014c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.987+0000 7fb5882ce640 1 --2- 172.21.15.98:0/312553502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb580103700 0x7fb580100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.987+0000 7fb5882ce640 1 --2- 172.21.15.98:0/312553502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb580102de0 0x7fb580100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:48.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.987+0000 7fb5882ce640 1 -- 172.21.15.98:0/312553502 >> 172.21.15.98:0/312553502 conn(0x7fb5800fc7a0 msgr2=0x7fb5800fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:48.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.987+0000 7fb5882ce640 1 -- 172.21.15.98:0/312553502 shutdown_connections 2024-09-19T08:02:48.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:47.987+0000 7fb5882ce640 1 -- 172.21.15.98:0/312553502 wait complete. 2024-09-19T08:02:48.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.f status 2024-09-19T08:02:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:48.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:48 smithi098 ceph-mon[31333]: pgmap v321: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:49.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:48 smithi162 ceph-mon[35260]: pgmap v321: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:49.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:48 smithi139 ceph-mon[34874]: pgmap v321: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.370+0000 7f6fab53e640 1 -- 172.21.15.98:0/271722853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fa4100a80 msgr2=0x7f6fa4100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.370+0000 7f6fab53e640 1 --2- 172.21.15.98:0/271722853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fa4100a80 0x7f6fa4100ee0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0009930 tx=0x7f6fa002a420 comp rx=0 tx=0).stop 2024-09-19T08:02:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.370+0000 7f6fab53e640 1 -- 172.21.15.98:0/271722853 shutdown_connections 2024-09-19T08:02:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.370+0000 7f6fab53e640 1 --2- 172.21.15.98:0/271722853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fa4101420 0x7f6fa410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.370+0000 7f6fab53e640 1 --2- 172.21.15.98:0/271722853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fa4100a80 0x7f6fa4100ee0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.370+0000 7f6fab53e640 1 --2- 172.21.15.98:0/271722853 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6fa4106a80 0x7f6fa4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.370+0000 7f6fab53e640 1 -- 172.21.15.98:0/271722853 >> 172.21.15.98:0/271722853 conn(0x7f6fa40fc7a0 msgr2=0x7f6fa40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.370+0000 7f6fab53e640 1 -- 172.21.15.98:0/271722853 shutdown_connections 2024-09-19T08:02:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.370+0000 7f6fab53e640 1 -- 172.21.15.98:0/271722853 wait complete. 2024-09-19T08:02:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fab53e640 1 Processor -- start 2024-09-19T08:02:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fab53e640 1 -- start start 2024-09-19T08:02:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fab53e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fa4100a80 0x7f6fa4199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fab53e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6fa4101420 0x7f6fa41997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fab53e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fa4106a80 0x7f6fa41a42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fab53e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa4110e40 con 0x7f6fa4100a80 2024-09-19T08:02:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fab53e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa4110cc0 con 0x7f6fa4101420 2024-09-19T08:02:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fab53e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa4110fc0 con 0x7f6fa4106a80 2024-09-19T08:02:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fa92da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fa4100a80 0x7f6fa4199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fa92da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fa4100a80 0x7f6fa4199270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49702/0 (socket says 172.21.15.98:49702) 2024-09-19T08:02:49.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fa92da640 1 -- 172.21.15.98:0/237924099 learned_addr learned my addr 172.21.15.98:0/237924099 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:49.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fa8ad9640 1 --2- 172.21.15.98:0/237924099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6fa4101420 0x7f6fa41997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fa92da640 1 -- 172.21.15.98:0/237924099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fa4106a80 msgr2=0x7f6fa41a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:49.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fa92da640 1 --2- 172.21.15.98:0/237924099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fa4106a80 0x7f6fa41a42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fa92da640 1 -- 172.21.15.98:0/237924099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6fa4101420 msgr2=0x7f6fa41997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:49.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fa92da640 1 --2- 172.21.15.98:0/237924099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6fa4101420 0x7f6fa41997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fa92da640 1 -- 172.21.15.98:0/237924099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa41a4830 con 0x7f6fa4100a80 2024-09-19T08:02:49.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fa8ad9640 1 --2- 172.21.15.98:0/237924099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6fa4101420 0x7f6fa41997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:49.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.371+0000 7f6fa92da640 1 --2- 172.21.15.98:0/237924099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fa4100a80 0x7f6fa4199270 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800c920 tx=0x7f6f9800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:49.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.372+0000 7f6f967fc640 1 -- 172.21.15.98:0/237924099 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f98007c70 con 0x7f6fa4100a80 2024-09-19T08:02:49.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.372+0000 7f6f967fc640 1 -- 172.21.15.98:0/237924099 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f98007e10 con 0x7f6fa4100a80 2024-09-19T08:02:49.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.372+0000 7f6fab53e640 1 -- 172.21.15.98:0/237924099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fa41a4a00 con 0x7f6fa4100a80 2024-09-19T08:02:49.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.372+0000 7f6f967fc640 1 -- 172.21.15.98:0/237924099 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f98018890 con 0x7f6fa4100a80 2024-09-19T08:02:49.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.372+0000 7f6fab53e640 1 -- 172.21.15.98:0/237924099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa41a4f10 con 0x7f6fa4100a80 2024-09-19T08:02:49.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.374+0000 7f6f967fc640 1 -- 172.21.15.98:0/237924099 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f6f98018a70 con 0x7f6fa4100a80 2024-09-19T08:02:49.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.374+0000 7f6fad917740 1 Processor -- start 2024-09-19T08:02:49.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.374+0000 7f6fad917740 1 -- start start 2024-09-19T08:02:49.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.374+0000 7f6fad917740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fa7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fad917740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fa82f0 0x55d2b8fb2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fad917740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fb4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fad917740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d2b8f0e090 con 0x55d2b8fb2bb0 2024-09-19T08:02:49.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fad917740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d2b8eb5a70 con 0x55d2b8fa79d0 2024-09-19T08:02:49.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fad917740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d2b8f0a050 con 0x55d2b8fa82f0 2024-09-19T08:02:49.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fa8ad9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fb4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6f967fc640 1 --2- 172.21.15.98:0/237924099 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f6f900a32d0 0x7f6f900a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fa8ad9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fb4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49712/0 (socket says 172.21.15.98:49712) 2024-09-19T08:02:49.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fa8ad9640 1 -- 172.21.15.98:0/2708565197 learned_addr learned my addr 172.21.15.98:0/2708565197 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:49.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fa9adb640 1 --2- 172.21.15.98:0/2708565197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fa7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fa92da640 1 --2- 172.21.15.98:0/2708565197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fa82f0 0x55d2b8fb2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fa9adb640 1 --2- 172.21.15.98:0/237924099 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f6f900a32d0 0x7f6f900a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fa8ad9640 1 -- 172.21.15.98:0/2708565197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fa82f0 msgr2=0x55d2b8fb2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:49.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fa8ad9640 1 --2- 172.21.15.98:0/2708565197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fa82f0 0x55d2b8fb2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fa8ad9640 1 -- 172.21.15.98:0/2708565197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fa79d0 msgr2=0x55d2b8fa7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:49.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fa8ad9640 1 --2- 172.21.15.98:0/2708565197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fa7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6f967fc640 1 -- 172.21.15.98:0/237924099 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f6f980cfc10 con 0x7f6fa4100a80 2024-09-19T08:02:49.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fa8ad9640 1 -- 172.21.15.98:0/2708565197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2b8ec2130 con 0x55d2b8fb2bb0 2024-09-19T08:02:49.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fa8ad9640 1 --2- 172.21.15.98:0/2708565197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fb4fa0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0006010 tx=0x7f6fa0009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83b0466f453cb88 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:49.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6fa9adb640 1 --2- 172.21.15.98:0/237924099 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f6f900a32d0 0x7f6f900a5790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009fd0 tx=0x7f6f9c005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:49.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6f8f7fe640 1 -- 172.21.15.98:0/2708565197 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa003f070 con 0x55d2b8fb2bb0 2024-09-19T08:02:49.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6f8f7fe640 1 -- 172.21.15.98:0/2708565197 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fa00317b0 con 0x55d2b8fb2bb0 2024-09-19T08:02:49.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.375+0000 7f6f8f7fe640 1 -- 172.21.15.98:0/2708565197 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0035070 con 0x55d2b8fb2bb0 2024-09-19T08:02:49.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.376+0000 7f6fad917740 1 -- 172.21.15.98:0/2708565197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fb2bb0 msgr2=0x55d2b8fb4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:49.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.376+0000 7f6fad917740 1 --2- 172.21.15.98:0/2708565197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fb4fa0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0006010 tx=0x7f6fa0009680 comp rx=0 tx=0).stop 2024-09-19T08:02:49.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.376+0000 7f6fad917740 1 -- 172.21.15.98:0/2708565197 shutdown_connections 2024-09-19T08:02:49.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.376+0000 7f6fad917740 1 --2- 172.21.15.98:0/2708565197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fb4fa0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.376+0000 7f6fad917740 1 --2- 172.21.15.98:0/2708565197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fa82f0 0x55d2b8fb2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.376+0000 7f6fad917740 1 --2- 172.21.15.98:0/2708565197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fa7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.376+0000 7f6fad917740 1 -- 172.21.15.98:0/2708565197 >> 172.21.15.98:0/2708565197 conn(0x55d2b8fa3f60 msgr2=0x55d2b8fa4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:49.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.376+0000 7f6fad917740 1 -- 172.21.15.98:0/2708565197 shutdown_connections 2024-09-19T08:02:49.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.376+0000 7f6fad917740 1 -- 172.21.15.98:0/2708565197 wait complete. 2024-09-19T08:02:49.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fad917740 1 -- start start 2024-09-19T08:02:49.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fad917740 1 Processor -- start 2024-09-19T08:02:49.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fad917740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fb2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fad917740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa82f0 0x55d2b8faf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fad917740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fafc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fad917740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d2b8fb5780 con 0x55d2b8fa82f0 2024-09-19T08:02:49.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fad917740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d2b8ec8f20 con 0x55d2b8fb2bb0 2024-09-19T08:02:49.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fad917740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d2b8fb5900 con 0x55d2b8fa79d0 2024-09-19T08:02:49.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fad917740 10 client.0 _ll_register_callbacks cb 0x55d2b8e92030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:49.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fa8ad9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa82f0 0x55d2b8faf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fa8ad9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa82f0 0x55d2b8faf6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49728/0 (socket says 172.21.15.98:49728) 2024-09-19T08:02:49.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fa8ad9640 1 -- 172.21.15.98:0/3885625612 learned_addr learned my addr 172.21.15.98:0/3885625612 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:49.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fa92da640 1 --2- 172.21.15.98:0/3885625612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fb2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fa8ad9640 1 -- 172.21.15.98:0/3885625612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fa79d0 msgr2=0x55d2b8fb2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:49.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fa8ad9640 1 --2- 172.21.15.98:0/3885625612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fb2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fa8ad9640 1 -- 172.21.15.98:0/3885625612 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fb2bb0 msgr2=0x55d2b8fafc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:49.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fa8ad9640 1 --2- 172.21.15.98:0/3885625612 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fafc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.377+0000 7f6fa8ad9640 1 -- 172.21.15.98:0/3885625612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2b8c7f2b0 con 0x55d2b8fa82f0 2024-09-19T08:02:49.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6fa8ad9640 1 --2- 172.21.15.98:0/3885625612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa82f0 0x55d2b8faf6d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0000c00 tx=0x7f6fa00052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec50dc1bb03b513 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:49.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6f8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:49.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6f8cff9640 1 -- 172.21.15.98:0/3885625612 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa003f070 con 0x55d2b8fa82f0 2024-09-19T08:02:49.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6f8cff9640 1 -- 172.21.15.98:0/3885625612 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f6fa0004050 con 0x55d2b8fa82f0 2024-09-19T08:02:49.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6fad917740 1 -- 172.21.15.98:0/3885625612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fa00317b0 con 0x55d2b8fa82f0 2024-09-19T08:02:49.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6f8cff9640 1 -- 172.21.15.98:0/3885625612 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6fa0035040 con 0x55d2b8fa82f0 2024-09-19T08:02:49.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6fad917740 10 client.15168 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:49.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6fad917740 1 -- 172.21.15.98:0/3885625612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d2b8fc1010 con 0x55d2b8fa82f0 2024-09-19T08:02:49.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6f8cff9640 1 -- 172.21.15.98:0/3885625612 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fa00317b0 con 0x55d2b8fa82f0 2024-09-19T08:02:49.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6f8cff9640 1 -- 172.21.15.98:0/3885625612 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6fa00046c0 con 0x55d2b8fa82f0 2024-09-19T08:02:49.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6fad917740 10 client.15168 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:49.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6fad917740 10 client.15168 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-19T08:02:49.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6fad917740 20 client.15168 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:49.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6fad917740 1 --2- 172.21.15.98:0/3885625612 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55d2b8fc3230 0x55d2b8fc46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6fad917740 4 client.15168 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:02:49.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.378+0000 7f6fad917740 1 -- 172.21.15.98:0/3885625612 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2b8f1b600 con 0x55d2b8fc3230 2024-09-19T08:02:49.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.379+0000 7f6fa92da640 1 --2- 172.21.15.98:0/3885625612 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55d2b8fc3230 0x55d2b8fc46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.379+0000 7f6fa92da640 1 --2- 172.21.15.98:0/3885625612 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55d2b8fc3230 0x55d2b8fc46a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3a6690fdddd8a2a0 server_cookie=2bd985b8fc4f02c1 in_seq=0 out_seq=0 2024-09-19T08:02:49.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.379+0000 7f6f8cff9640 10 client.15168 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:02:49.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6f8cff9640 1 -- 172.21.15.98:0/3885625612 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d2b8f1b600 con 0x55d2b8fc3230 2024-09-19T08:02:49.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6f8cff9640 10 client.15168 handle_command_reply: tid=0 2024-09-19T08:02:49.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6fad917740 1 client.15168 shutdown 2024-09-19T08:02:49.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6fad917740 1 -- 172.21.15.98:0/3885625612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa82f0 msgr2=0x55d2b8faf6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:49.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6fad917740 1 --2- 172.21.15.98:0/3885625612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa82f0 0x55d2b8faf6d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0000c00 tx=0x7f6fa00052d0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6fad917740 1 -- 172.21.15.98:0/3885625612 shutdown_connections 2024-09-19T08:02:49.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6fad917740 1 --2- 172.21.15.98:0/3885625612 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fafc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6fad917740 1 --2- 172.21.15.98:0/3885625612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa82f0 0x55d2b8faf6d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6fad917740 1 --2- 172.21.15.98:0/3885625612 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55d2b8fc3230 0x55d2b8fc46a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6fad917740 1 --2- 172.21.15.98:0/3885625612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fb2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6fad917740 1 -- 172.21.15.98:0/3885625612 >> 172.21.15.98:0/3885625612 conn(0x55d2b8fa3f60 msgr2=0x55d2b8fa9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:49.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6f8cff9640 0 client.15168 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:02:49.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6fad917740 1 -- 172.21.15.98:0/3885625612 shutdown_connections 2024-09-19T08:02:49.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6fad917740 1 -- 172.21.15.98:0/3885625612 wait complete. 2024-09-19T08:02:49.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.380+0000 7f6fad917740 20 client.15168 trim_cache size 0 max 16384 2024-09-19T08:02:49.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fad917740 1 Processor -- start 2024-09-19T08:02:49.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fad917740 1 -- start start 2024-09-19T08:02:49.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fad917740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fc3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fad917740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fa82f0 0x55d2b8fc46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fad917740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fc2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fad917740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d2b8fc29e0 con 0x55d2b8fa79d0 2024-09-19T08:02:49.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fad917740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d2b8fc30b0 con 0x55d2b8fb2bb0 2024-09-19T08:02:49.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fad917740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d2b8fc3780 con 0x55d2b8fa82f0 2024-09-19T08:02:49.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fa92da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fc3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fa92da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fc3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49738/0 (socket says 172.21.15.98:49738) 2024-09-19T08:02:49.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fa92da640 1 -- 172.21.15.98:0/3139861363 learned_addr learned my addr 172.21.15.98:0/3139861363 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:49.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fa92da640 1 -- 172.21.15.98:0/3139861363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fa82f0 msgr2=0x55d2b8fc46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:49.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fa92da640 1 --2- 172.21.15.98:0/3139861363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fa82f0 0x55d2b8fc46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fa92da640 1 -- 172.21.15.98:0/3139861363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fb2bb0 msgr2=0x55d2b8fc2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:49.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fa9adb640 1 --2- 172.21.15.98:0/3139861363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fc2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.398+0000 7f6fa92da640 1 --2- 172.21.15.98:0/3139861363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fc2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.399+0000 7f6fa92da640 1 -- 172.21.15.98:0/3139861363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2b8fb26f0 con 0x55d2b8fa79d0 2024-09-19T08:02:49.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.399+0000 7f6fa9adb640 1 --2- 172.21.15.98:0/3139861363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fc2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:49.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.399+0000 7f6fa92da640 1 --2- 172.21.15.98:0/3139861363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fc3fd0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6f98098ab0 tx=0x7f6f9806ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5dfd8e3fec4c07 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:49.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.399+0000 7f6f8effd640 1 -- 172.21.15.98:0/3139861363 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f980a0490 con 0x55d2b8fa79d0 2024-09-19T08:02:49.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.399+0000 7f6f8effd640 1 -- 172.21.15.98:0/3139861363 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f980a4d50 con 0x55d2b8fa79d0 2024-09-19T08:02:49.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.399+0000 7f6f8effd640 1 -- 172.21.15.98:0/3139861363 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f98098750 con 0x55d2b8fa79d0 2024-09-19T08:02:49.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.399+0000 7f6fad917740 1 -- 172.21.15.98:0/3139861363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa79d0 msgr2=0x55d2b8fc3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:49.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.399+0000 7f6fad917740 1 --2- 172.21.15.98:0/3139861363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fc3fd0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6f98098ab0 tx=0x7f6f9806ef60 comp rx=0 tx=0).stop 2024-09-19T08:02:49.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.399+0000 7f6fad917740 1 -- 172.21.15.98:0/3139861363 shutdown_connections 2024-09-19T08:02:49.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.399+0000 7f6fad917740 1 --2- 172.21.15.98:0/3139861363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fc2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.399+0000 7f6fad917740 1 --2- 172.21.15.98:0/3139861363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fa82f0 0x55d2b8fc46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.399+0000 7f6fad917740 1 --2- 172.21.15.98:0/3139861363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fc3fd0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.399+0000 7f6fad917740 1 -- 172.21.15.98:0/3139861363 >> 172.21.15.98:0/3139861363 conn(0x55d2b8fa3f60 msgr2=0x55d2b8fa4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:49.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.399+0000 7f6fad917740 1 -- 172.21.15.98:0/3139861363 shutdown_connections 2024-09-19T08:02:49.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.400+0000 7f6fad917740 1 -- 172.21.15.98:0/3139861363 wait complete. 2024-09-19T08:02:49.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.400+0000 7f6fad917740 1 -- start start 2024-09-19T08:02:49.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.400+0000 7f6fad917740 1 Processor -- start 2024-09-19T08:02:49.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.400+0000 7f6fad917740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.400+0000 7f6fad917740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa82f0 0x55d2b8fc9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.400+0000 7f6fad917740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.400+0000 7f6fad917740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d2b8fc4520 con 0x55d2b8fa82f0 2024-09-19T08:02:49.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.400+0000 7f6fad917740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d2b8fc3e50 con 0x55d2b8fa79d0 2024-09-19T08:02:49.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.400+0000 7f6fad917740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d2b8fc4bf0 con 0x55d2b8fb2bb0 2024-09-19T08:02:49.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.400+0000 7f6fad917740 10 client.0 _ll_register_callbacks cb 0x7f6f9c009d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:49.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.400+0000 7f6fa8ad9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa82f0 0x55d2b8fc9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.400+0000 7f6fa8ad9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa82f0 0x55d2b8fc9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49750/0 (socket says 172.21.15.98:49750) 2024-09-19T08:02:49.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.400+0000 7f6fa8ad9640 1 -- 172.21.15.98:0/1080767000 learned_addr learned my addr 172.21.15.98:0/1080767000 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:49.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6fa9adb640 1 --2- 172.21.15.98:0/1080767000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6fa8ad9640 1 -- 172.21.15.98:0/1080767000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fb2bb0 msgr2=0x55d2b8fca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:49.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6fa92da640 1 --2- 172.21.15.98:0/1080767000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6fa8ad9640 1 --2- 172.21.15.98:0/1080767000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6fa8ad9640 1 -- 172.21.15.98:0/1080767000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fa79d0 msgr2=0x55d2b8fccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:49.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6fa8ad9640 1 --2- 172.21.15.98:0/1080767000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6fa8ad9640 1 -- 172.21.15.98:0/1080767000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2b8fb0710 con 0x55d2b8fa82f0 2024-09-19T08:02:49.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6fa8ad9640 1 --2- 172.21.15.98:0/1080767000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa82f0 0x55d2b8fc9d30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0004a60 tx=0x7f6fa0005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1f42757e0ecbd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:49.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6f8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:49.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6f8cff9640 1 -- 172.21.15.98:0/1080767000 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa003f070 con 0x55d2b8fa82f0 2024-09-19T08:02:49.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6f8cff9640 1 -- 172.21.15.98:0/1080767000 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f6fa00315b0 con 0x55d2b8fa82f0 2024-09-19T08:02:49.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6fad917740 1 -- 172.21.15.98:0/1080767000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f980a4d50 con 0x55d2b8fa82f0 2024-09-19T08:02:49.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6f8cff9640 1 -- 172.21.15.98:0/1080767000 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6fa00317b0 con 0x55d2b8fa82f0 2024-09-19T08:02:49.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6fad917740 10 client.15174 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:49.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6fad917740 1 -- 172.21.15.98:0/1080767000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d2b8fb4710 con 0x55d2b8fa82f0 2024-09-19T08:02:49.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6f8cff9640 1 -- 172.21.15.98:0/1080767000 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f980a4d50 con 0x55d2b8fa82f0 2024-09-19T08:02:49.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6f8cff9640 1 -- 172.21.15.98:0/1080767000 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6fa00035f0 con 0x55d2b8fa82f0 2024-09-19T08:02:49.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.401+0000 7f6fad917740 10 client.15174 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:49.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.402+0000 7f6fad917740 10 client.15174 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-19T08:02:49.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.402+0000 7f6fad917740 20 client.15174 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:49.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.402+0000 7f6fad917740 1 --2- 172.21.15.98:0/1080767000 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55d2b9032100 0x55d2b9030e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:49.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.402+0000 7f6fad917740 4 client.15174 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:02:49.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.402+0000 7f6fad917740 1 -- 172.21.15.98:0/1080767000 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "status"}) -- 0x55d2b8f15b10 con 0x55d2b9032100 2024-09-19T08:02:49.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.402+0000 7f6fa92da640 1 --2- 172.21.15.98:0/1080767000 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55d2b9032100 0x55d2b9030e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:49.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.402+0000 7f6f8cff9640 10 client.15174 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:02:49.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.402+0000 7f6fa92da640 1 --2- 172.21.15.98:0/1080767000 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55d2b9032100 0x55d2b9030e10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=512fa16d57610c07 server_cookie=db04759a977bc0f4 in_seq=0 out_seq=0 2024-09-19T08:02:49.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.402+0000 7f6f8cff9640 1 -- 172.21.15.98:0/1080767000 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55d2b8f15b10 con 0x55d2b9032100 2024-09-19T08:02:49.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.402+0000 7f6f8cff9640 10 client.15174 handle_command_reply: tid=0 2024-09-19T08:02:49.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.402+0000 7f6fad917740 1 client.15174 shutdown 2024-09-19T08:02:49.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.402+0000 7f6fad917740 1 -- 172.21.15.98:0/1080767000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa82f0 msgr2=0x55d2b8fc9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:49.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.402+0000 7f6fad917740 1 --2- 172.21.15.98:0/1080767000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa82f0 0x55d2b8fc9d30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0004a60 tx=0x7f6fa0005780 comp rx=0 tx=0).stop 2024-09-19T08:02:49.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fad917740 1 -- 172.21.15.98:0/1080767000 shutdown_connections 2024-09-19T08:02:49.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fad917740 1 --2- 172.21.15.98:0/1080767000 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55d2b9032100 0x55d2b9030e10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fad917740 1 --2- 172.21.15.98:0/1080767000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2b8fb2bb0 0x55d2b8fca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fad917740 1 --2- 172.21.15.98:0/1080767000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2b8fa82f0 0x55d2b8fc9d30 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fad917740 1 --2- 172.21.15.98:0/1080767000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2b8fa79d0 0x55d2b8fccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fad917740 1 -- 172.21.15.98:0/1080767000 >> 172.21.15.98:0/1080767000 conn(0x55d2b8fa3f60 msgr2=0x55d2b8faacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:49.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6f8cff9640 0 client.15174 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:02:49.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fad917740 1 -- 172.21.15.98:0/1080767000 shutdown_connections 2024-09-19T08:02:49.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fad917740 1 -- 172.21.15.98:0/1080767000 wait complete. 2024-09-19T08:02:49.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fad917740 20 client.15174 trim_cache size 0 max 16384 2024-09-19T08:02:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:02:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:02:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 3, 2024-09-19T08:02:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24406, 2024-09-19T08:02:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:02:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:02:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:02:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 88.718759667, 2024-09-19T08:02:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 27, 2024-09-19T08:02:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 80, 2024-09-19T08:02:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 78, 2024-09-19T08:02:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 125.37745064000001 2024-09-19T08:02:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:02:49.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fab53e640 1 -- 172.21.15.98:0/237924099 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f6f900a32d0 msgr2=0x7f6f900a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:49.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fab53e640 1 --2- 172.21.15.98:0/237924099 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f6f900a32d0 0x7f6f900a5790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009fd0 tx=0x7f6f9c005bd0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fab53e640 1 -- 172.21.15.98:0/237924099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fa4100a80 msgr2=0x7f6fa4199270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:49.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fab53e640 1 --2- 172.21.15.98:0/237924099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fa4100a80 0x7f6fa4199270 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800c920 tx=0x7f6f9800cde0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fab53e640 1 -- 172.21.15.98:0/237924099 shutdown_connections 2024-09-19T08:02:49.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fab53e640 1 --2- 172.21.15.98:0/237924099 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f6f900a32d0 0x7f6f900a5790 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fab53e640 1 --2- 172.21.15.98:0/237924099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fa4106a80 0x7f6fa41a42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fab53e640 1 --2- 172.21.15.98:0/237924099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6fa4101420 0x7f6fa41997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fab53e640 1 --2- 172.21.15.98:0/237924099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fa4100a80 0x7f6fa4199270 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:49.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.403+0000 7f6fab53e640 1 -- 172.21.15.98:0/237924099 >> 172.21.15.98:0/237924099 conn(0x7f6fa40fc7a0 msgr2=0x7f6fa40fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:49.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.404+0000 7f6fab53e640 1 -- 172.21.15.98:0/237924099 shutdown_connections 2024-09-19T08:02:49.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:49.404+0000 7f6fab53e640 1 -- 172.21.15.98:0/237924099 wait complete. 2024-09-19T08:02:50.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.e status 2024-09-19T08:02:50.379 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.851+0000 7f4f37efc640 1 -- 172.21.15.98:0/4117623549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f301027a0 msgr2=0x7f4f301067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.851+0000 7f4f37efc640 1 --2- 172.21.15.98:0/4117623549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f301027a0 0x7f4f301067f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4f20009900 tx=0x7f4f2002a410 comp rx=0 tx=0).stop 2024-09-19T08:02:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.851+0000 7f4f37efc640 1 -- 172.21.15.98:0/4117623549 shutdown_connections 2024-09-19T08:02:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.851+0000 7f4f37efc640 1 --2- 172.21.15.98:0/4117623549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f30106f20 0x7f4f3010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.851+0000 7f4f37efc640 1 --2- 172.21.15.98:0/4117623549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f301027a0 0x7f4f301067f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.851+0000 7f4f37efc640 1 --2- 172.21.15.98:0/4117623549 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4f30101df0 0x7f4f301021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.851+0000 7f4f37efc640 1 -- 172.21.15.98:0/4117623549 >> 172.21.15.98:0/4117623549 conn(0x7f4f300fd8b0 msgr2=0x7f4f300ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.851+0000 7f4f37efc640 1 -- 172.21.15.98:0/4117623549 shutdown_connections 2024-09-19T08:02:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.851+0000 7f4f37efc640 1 -- 172.21.15.98:0/4117623549 wait complete. 2024-09-19T08:02:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.852+0000 7f4f37efc640 1 Processor -- start 2024-09-19T08:02:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.852+0000 7f4f37efc640 1 -- start start 2024-09-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.852+0000 7f4f37efc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f30101df0 0x7f4f30198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.852+0000 7f4f37efc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4f301027a0 0x7f4f301989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.852+0000 7f4f37efc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f30106f20 0x7f4f3019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.852+0000 7f4f37efc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4f30111fa0 con 0x7f4f30101df0 2024-09-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.852+0000 7f4f37efc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4f30111e20 con 0x7f4f301027a0 2024-09-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.852+0000 7f4f37efc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4f30112120 con 0x7f4f30106f20 2024-09-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f35c98640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f30101df0 0x7f4f30198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f35c98640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f30101df0 0x7f4f30198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49760/0 (socket says 172.21.15.98:49760) 2024-09-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f35c98640 1 -- 172.21.15.98:0/757497638 learned_addr learned my addr 172.21.15.98:0/757497638 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f35497640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4f301027a0 0x7f4f301989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f36499640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f30106f20 0x7f4f3019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f35c98640 1 -- 172.21.15.98:0/757497638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f30106f20 msgr2=0x7f4f3019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f35c98640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f30106f20 0x7f4f3019cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f35c98640 1 -- 172.21.15.98:0/757497638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4f301027a0 msgr2=0x7f4f301989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f35c98640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4f301027a0 0x7f4f301989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f35c98640 1 -- 172.21.15.98:0/757497638 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3019d420 con 0x7f4f30101df0 2024-09-19T08:02:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f35497640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4f301027a0 0x7f4f301989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f36499640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f30106f20 0x7f4f3019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f35c98640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f30101df0 0x7f4f30198470 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00e9f0 tx=0x7f4f2c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f26ffd640 1 -- 172.21.15.98:0/757497638 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c00cd80 con 0x7f4f30101df0 2024-09-19T08:02:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f26ffd640 1 -- 172.21.15.98:0/757497638 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f2c010910 con 0x7f4f30101df0 2024-09-19T08:02:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f37efc640 1 -- 172.21.15.98:0/757497638 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3019d710 con 0x7f4f30101df0 2024-09-19T08:02:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.853+0000 7f4f26ffd640 1 -- 172.21.15.98:0/757497638 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c018990 con 0x7f4f30101df0 2024-09-19T08:02:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.854+0000 7f4f37efc640 1 -- 172.21.15.98:0/757497638 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f301a4ff0 con 0x7f4f30101df0 2024-09-19T08:02:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.856+0000 7f4f26ffd640 1 -- 172.21.15.98:0/757497638 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4f2c010ac0 con 0x7f4f30101df0 2024-09-19T08:02:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.857+0000 7f4f26ffd640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4f1c0a3350 0x7f4f1c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.857+0000 7f4f35497640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4f1c0a3350 0x7f4f1c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.857+0000 7f4f26ffd640 1 -- 172.21.15.98:0/757497638 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f4f2c0cfe60 con 0x7f4f30101df0 2024-09-19T08:02:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.857+0000 7f4f35497640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4f1c0a3350 0x7f4f1c0a5810 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4f20000c00 tx=0x7f4f20032c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.857+0000 7f4f3a2d5740 1 Processor -- start 2024-09-19T08:02:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f3a2d5740 1 -- start start 2024-09-19T08:02:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f3a2d5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bd4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f3a2d5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 0x55fea5bdf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f3a2d5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5be1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f3a2d5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fea5b3b090 con 0x55fea5bd49d0 2024-09-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f3a2d5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fea5ae2a70 con 0x55fea5bdfbb0 2024-09-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f3a2d5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fea5b37050 con 0x55fea5bd52f0 2024-09-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f35c98640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bd4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f36499640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5be1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f35497640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 0x55fea5bdf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f36499640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5be1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:59692/0 (socket says 172.21.15.98:59692) 2024-09-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f35497640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 0x55fea5bdf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:36108/0 (socket says 172.21.15.98:36108) 2024-09-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f36499640 1 -- 172.21.15.98:0/1428117701 learned_addr learned my addr 172.21.15.98:0/1428117701 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f35c98640 1 -- 172.21.15.98:0/1428117701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 msgr2=0x55fea5bdf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f35c98640 1 --2- 172.21.15.98:0/1428117701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 0x55fea5bdf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f35c98640 1 -- 172.21.15.98:0/1428117701 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 msgr2=0x55fea5be1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f35c98640 1 --2- 172.21.15.98:0/1428117701 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5be1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.858+0000 7f4f35c98640 1 -- 172.21.15.98:0/1428117701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fea5aef130 con 0x55fea5bd49d0 2024-09-19T08:02:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.859+0000 7f4f35c98640 1 --2- 172.21.15.98:0/1428117701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bd4db0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0cd820 tx=0x7f4f2c09c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=380bef5c4ef04307 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.859+0000 7f4f1bfff640 1 -- 172.21.15.98:0/1428117701 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c0a4fa0 con 0x55fea5bd49d0 2024-09-19T08:02:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.859+0000 7f4f1bfff640 1 -- 172.21.15.98:0/1428117701 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f2c0cda70 con 0x55fea5bd49d0 2024-09-19T08:02:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.859+0000 7f4f1bfff640 1 -- 172.21.15.98:0/1428117701 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c09b9e0 con 0x55fea5bd49d0 2024-09-19T08:02:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.860+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/1428117701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 msgr2=0x55fea5bd4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.860+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/1428117701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bd4db0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0cd820 tx=0x7f4f2c09c230 comp rx=0 tx=0).stop 2024-09-19T08:02:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.860+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/1428117701 shutdown_connections 2024-09-19T08:02:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.860+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/1428117701 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5be1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.860+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/1428117701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 0x55fea5bdf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.860+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/1428117701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bd4db0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.860+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/1428117701 >> 172.21.15.98:0/1428117701 conn(0x55fea5bd0f60 msgr2=0x55fea5bd1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.860+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/1428117701 shutdown_connections 2024-09-19T08:02:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.860+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/1428117701 wait complete. 2024-09-19T08:02:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.860+0000 7f4f3a2d5740 1 -- start start 2024-09-19T08:02:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.860+0000 7f4f3a2d5740 1 Processor -- start 2024-09-19T08:02:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f3a2d5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bdf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f3a2d5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 0x55fea5bdc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f3a2d5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5bdccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f3a2d5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fea5be6b90 con 0x55fea5bd49d0 2024-09-19T08:02:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f3a2d5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fea5af5f20 con 0x55fea5bdfbb0 2024-09-19T08:02:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f3a2d5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fea5be6d10 con 0x55fea5bd52f0 2024-09-19T08:02:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f3a2d5740 10 client.0 _ll_register_callbacks cb 0x55fea5abf030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f35c98640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bdf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f35c98640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bdf460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49780/0 (socket says 172.21.15.98:49780) 2024-09-19T08:02:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f35c98640 1 -- 172.21.15.98:0/3808918608 learned_addr learned my addr 172.21.15.98:0/3808918608 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f35497640 1 --2- 172.21.15.98:0/3808918608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 0x55fea5bdc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f36499640 1 --2- 172.21.15.98:0/3808918608 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5bdccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f35c98640 1 -- 172.21.15.98:0/3808918608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 msgr2=0x55fea5bdc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f35c98640 1 --2- 172.21.15.98:0/3808918608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 0x55fea5bdc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f35c98640 1 -- 172.21.15.98:0/3808918608 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 msgr2=0x55fea5bdccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f35c98640 1 --2- 172.21.15.98:0/3808918608 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5bdccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f35c98640 1 -- 172.21.15.98:0/3808918608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fea58ac2b0 con 0x55fea5bd49d0 2024-09-19T08:02:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f35c98640 1 --2- 172.21.15.98:0/3808918608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bdf460 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00c6a0 tx=0x7f4f2c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159f3cc2bc1fb133 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f197fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.861+0000 7f4f197fa640 1 -- 172.21.15.98:0/3808918608 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c0b4070 con 0x55fea5bd49d0 2024-09-19T08:02:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.862+0000 7f4f197fa640 1 -- 172.21.15.98:0/3808918608 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f4f2c0b5260 con 0x55fea5bd49d0 2024-09-19T08:02:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.862+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/3808918608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f2c0cda70 con 0x55fea5bd49d0 2024-09-19T08:02:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.862+0000 7f4f197fa640 1 -- 172.21.15.98:0/3808918608 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4f2c0a4fa0 con 0x55fea5bd49d0 2024-09-19T08:02:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.862+0000 7f4f3a2d5740 10 client.15186 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.862+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/3808918608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fea5bf24b0 con 0x55fea5bd49d0 2024-09-19T08:02:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.862+0000 7f4f197fa640 1 -- 172.21.15.98:0/3808918608 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f2c0cda70 con 0x55fea5bd49d0 2024-09-19T08:02:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.862+0000 7f4f197fa640 1 -- 172.21.15.98:0/3808918608 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f4f2c098080 con 0x55fea5bd49d0 2024-09-19T08:02:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.862+0000 7f4f3a2d5740 10 client.15186 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.862+0000 7f4f3a2d5740 10 client.15186 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-19T08:02:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.862+0000 7f4f3a2d5740 20 client.15186 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.862+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/3808918608 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55fea5bf4720 0x55fea5bf5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.862+0000 7f4f3a2d5740 4 client.15186 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:02:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.862+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/3808918608 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fea5b48600 con 0x55fea5bf4720 2024-09-19T08:02:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.862+0000 7f4f35497640 1 --2- 172.21.15.98:0/3808918608 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55fea5bf4720 0x55fea5bf5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.863+0000 7f4f35497640 1 --2- 172.21.15.98:0/3808918608 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55fea5bf4720 0x55fea5bf5b90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8e1ffd51f3e28d08 server_cookie=bc8cc2f20874fe8c in_seq=0 out_seq=0 2024-09-19T08:02:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.863+0000 7f4f197fa640 10 client.15186 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:02:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.863+0000 7f4f197fa640 1 -- 172.21.15.98:0/3808918608 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fea5b48600 con 0x55fea5bf4720 2024-09-19T08:02:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.863+0000 7f4f197fa640 10 client.15186 handle_command_reply: tid=0 2024-09-19T08:02:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.864+0000 7f4f3a2d5740 1 client.15186 shutdown 2024-09-19T08:02:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.864+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/3808918608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 msgr2=0x55fea5bdf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.864+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/3808918608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bdf460 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00c6a0 tx=0x7f4f2c00c840 comp rx=0 tx=0).stop 2024-09-19T08:02:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.864+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/3808918608 shutdown_connections 2024-09-19T08:02:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.864+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/3808918608 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55fea5bf4720 0x55fea5bf5b90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.864+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/3808918608 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5bdccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.864+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/3808918608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 0x55fea5bdc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.864+0000 7f4f197fa640 0 client.15186 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:02:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.864+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/3808918608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bdf460 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.864+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/3808918608 >> 172.21.15.98:0/3808918608 conn(0x55fea5bd0f60 msgr2=0x55fea5bd6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.864+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/3808918608 shutdown_connections 2024-09-19T08:02:50.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.864+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/3808918608 wait complete. 2024-09-19T08:02:50.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.864+0000 7f4f3a2d5740 20 client.15186 trim_cache size 0 max 16384 2024-09-19T08:02:50.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.881+0000 7f4f3a2d5740 1 Processor -- start 2024-09-19T08:02:50.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.881+0000 7f4f3a2d5740 1 -- start start 2024-09-19T08:02:50.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f3a2d5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd49d0 0x55fea5bf4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f3a2d5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd52f0 0x55fea5bde610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f3a2d5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5bd9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f3a2d5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fea5bf3ed0 con 0x55fea5bd52f0 2024-09-19T08:02:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f3a2d5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fea5bf45a0 con 0x55fea5bdfbb0 2024-09-19T08:02:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f3a2d5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fea5bf4c70 con 0x55fea5bd49d0 2024-09-19T08:02:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f35497640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd52f0 0x55fea5bde610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f35497640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd52f0 0x55fea5bde610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49792/0 (socket says 172.21.15.98:49792) 2024-09-19T08:02:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f35497640 1 -- 172.21.15.98:0/2029873704 learned_addr learned my addr 172.21.15.98:0/2029873704 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f35c98640 1 --2- 172.21.15.98:0/2029873704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd49d0 0x55fea5bf4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f35497640 1 -- 172.21.15.98:0/2029873704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd49d0 msgr2=0x55fea5bf4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f35497640 1 --2- 172.21.15.98:0/2029873704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd49d0 0x55fea5bf4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f35497640 1 -- 172.21.15.98:0/2029873704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 msgr2=0x55fea5bd9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f35497640 1 --2- 172.21.15.98:0/2029873704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5bd9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f35497640 1 -- 172.21.15.98:0/2029873704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fea5bd5120 con 0x55fea5bd52f0 2024-09-19T08:02:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f35c98640 1 --2- 172.21.15.98:0/2029873704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd49d0 0x55fea5bf4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f35497640 1 --2- 172.21.15.98:0/2029873704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd52f0 0x55fea5bde610 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4f20009900 tx=0x7f4f2003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a9caad9dcb95b27 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f1b7fe640 1 -- 172.21.15.98:0/2029873704 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f20057070 con 0x55fea5bd52f0 2024-09-19T08:02:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f1b7fe640 1 -- 172.21.15.98:0/2029873704 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f20005520 con 0x55fea5bd52f0 2024-09-19T08:02:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.882+0000 7f4f1b7fe640 1 -- 172.21.15.98:0/2029873704 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f20035050 con 0x55fea5bd52f0 2024-09-19T08:02:50.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.883+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/2029873704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd52f0 msgr2=0x55fea5bde610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:50.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.883+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/2029873704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd52f0 0x55fea5bde610 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4f20009900 tx=0x7f4f2003e000 comp rx=0 tx=0).stop 2024-09-19T08:02:50.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.883+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/2029873704 shutdown_connections 2024-09-19T08:02:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.883+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/2029873704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5bd9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.883+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/2029873704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd52f0 0x55fea5bde610 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.883+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/2029873704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd49d0 0x55fea5bf4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.883+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/2029873704 >> 172.21.15.98:0/2029873704 conn(0x55fea5bd0f60 msgr2=0x55fea5bddf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.883+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/2029873704 shutdown_connections 2024-09-19T08:02:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.883+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/2029873704 wait complete. 2024-09-19T08:02:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.883+0000 7f4f3a2d5740 1 -- start start 2024-09-19T08:02:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f3a2d5740 1 Processor -- start 2024-09-19T08:02:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f3a2d5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bfe4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f3a2d5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 0x55fea5bf98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f3a2d5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5bf9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f3a2d5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fea5bf5a10 con 0x55fea5bd49d0 2024-09-19T08:02:50.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f3a2d5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fea5bf5340 con 0x55fea5bdfbb0 2024-09-19T08:02:50.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f3a2d5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fea5bf60e0 con 0x55fea5bd52f0 2024-09-19T08:02:50.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f3a2d5740 10 client.0 _ll_register_callbacks cb 0x7f4f280076a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:50.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f35c98640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bfe4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f35c98640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bfe4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49794/0 (socket says 172.21.15.98:49794) 2024-09-19T08:02:50.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f35c98640 1 -- 172.21.15.98:0/1224263866 learned_addr learned my addr 172.21.15.98:0/1224263866 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:50.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f36499640 1 --2- 172.21.15.98:0/1224263866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5bf9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f35497640 1 --2- 172.21.15.98:0/1224263866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 0x55fea5bf98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f35c98640 1 -- 172.21.15.98:0/1224263866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 msgr2=0x55fea5bf98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:50.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f35c98640 1 --2- 172.21.15.98:0/1224263866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 0x55fea5bf98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f35c98640 1 -- 172.21.15.98:0/1224263866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 msgr2=0x55fea5bf9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:50.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f35c98640 1 --2- 172.21.15.98:0/1224263866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5bf9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.884+0000 7f4f35c98640 1 -- 172.21.15.98:0/1224263866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fea5bfa630 con 0x55fea5bd49d0 2024-09-19T08:02:50.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f35c98640 1 --2- 172.21.15.98:0/1224263866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bfe4f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0a5160 tx=0x7f4f2c0a07a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b056dc0d6d5e18 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:50.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f197fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:50.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f197fa640 1 -- 172.21.15.98:0/1224263866 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c0b4070 con 0x55fea5bd49d0 2024-09-19T08:02:50.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f197fa640 1 -- 172.21.15.98:0/1224263866 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f4f2c09b990 con 0x55fea5bd49d0 2024-09-19T08:02:50.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/1224263866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f20005520 con 0x55fea5bd49d0 2024-09-19T08:02:50.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f197fa640 1 -- 172.21.15.98:0/1224263866 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4f2c0985d0 con 0x55fea5bd49d0 2024-09-19T08:02:50.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f3a2d5740 10 client.15192 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:50.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/1224263866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fea5bfa630 con 0x55fea5bd49d0 2024-09-19T08:02:50.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f197fa640 1 -- 172.21.15.98:0/1224263866 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f20005520 con 0x55fea5bd49d0 2024-09-19T08:02:50.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f197fa640 1 -- 172.21.15.98:0/1224263866 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f4f2c0a0120 con 0x55fea5bd49d0 2024-09-19T08:02:50.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f3a2d5740 10 client.15192 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:50.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f3a2d5740 10 client.15192 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-19T08:02:50.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f3a2d5740 20 client.15192 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:50.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/1224263866 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55fea5c633f0 0x55fea5c62100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:50.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f3a2d5740 4 client.15192 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:02:50.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.885+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/1224263866 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "status"}) -- 0x55fea5b42b10 con 0x55fea5c633f0 2024-09-19T08:02:50.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.886+0000 7f4f35497640 1 --2- 172.21.15.98:0/1224263866 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55fea5c633f0 0x55fea5c62100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:50.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.886+0000 7f4f197fa640 10 client.15192 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:02:50.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.886+0000 7f4f35497640 1 --2- 172.21.15.98:0/1224263866 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55fea5c633f0 0x55fea5c62100 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=90f1a9048f5a8905 server_cookie=52a0ac989ac548ef in_seq=0 out_seq=0 2024-09-19T08:02:50.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.886+0000 7f4f197fa640 1 -- 172.21.15.98:0/1224263866 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55fea5b42b10 con 0x55fea5c633f0 2024-09-19T08:02:50.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.886+0000 7f4f197fa640 10 client.15192 handle_command_reply: tid=0 2024-09-19T08:02:50.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f3a2d5740 1 client.15192 shutdown 2024-09-19T08:02:50.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/1224263866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 msgr2=0x55fea5bfe4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:50.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/1224263866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bfe4f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0a5160 tx=0x7f4f2c0a07a0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/1224263866 shutdown_connections 2024-09-19T08:02:50.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/1224263866 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55fea5c633f0 0x55fea5c62100 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/1224263866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fea5bdfbb0 0x55fea5bf9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/1224263866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fea5bd52f0 0x55fea5bf98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f197fa640 0 client.15192 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:02:50.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f3a2d5740 1 --2- 172.21.15.98:0/1224263866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fea5bd49d0 0x55fea5bfe4f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/1224263866 >> 172.21.15.98:0/1224263866 conn(0x55fea5bd0f60 msgr2=0x55fea5bde5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:50.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/1224263866 shutdown_connections 2024-09-19T08:02:50.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f3a2d5740 1 -- 172.21.15.98:0/1224263866 wait complete. 2024-09-19T08:02:50.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f3a2d5740 20 client.15192 trim_cache size 0 max 16384 2024-09-19T08:02:50.893 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:02:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:02:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2024-09-19T08:02:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24437, 2024-09-19T08:02:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:02:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:02:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:02:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 91.059041152000006, 2024-09-19T08:02:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 27, 2024-09-19T08:02:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 80, 2024-09-19T08:02:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 78, 2024-09-19T08:02:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 129.34394306199999 2024-09-19T08:02:50.894 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:02:50.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f37efc640 1 -- 172.21.15.98:0/757497638 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4f1c0a3350 msgr2=0x7f4f1c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:50.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f37efc640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4f1c0a3350 0x7f4f1c0a5810 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4f20000c00 tx=0x7f4f20032c70 comp rx=0 tx=0).stop 2024-09-19T08:02:50.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f37efc640 1 -- 172.21.15.98:0/757497638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f30101df0 msgr2=0x7f4f30198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:50.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.887+0000 7f4f37efc640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f30101df0 0x7f4f30198470 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00e9f0 tx=0x7f4f2c00eeb0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.888+0000 7f4f37efc640 1 -- 172.21.15.98:0/757497638 shutdown_connections 2024-09-19T08:02:50.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.888+0000 7f4f37efc640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f4f1c0a3350 0x7f4f1c0a5810 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.888+0000 7f4f37efc640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f30106f20 0x7f4f3019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.888+0000 7f4f37efc640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4f301027a0 0x7f4f301989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.888+0000 7f4f37efc640 1 --2- 172.21.15.98:0/757497638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f30101df0 0x7f4f30198470 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:50.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.888+0000 7f4f37efc640 1 -- 172.21.15.98:0/757497638 >> 172.21.15.98:0/757497638 conn(0x7f4f300fd8b0 msgr2=0x7f4f300feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:50.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.888+0000 7f4f37efc640 1 -- 172.21.15.98:0/757497638 shutdown_connections 2024-09-19T08:02:50.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:50.888+0000 7f4f37efc640 1 -- 172.21.15.98:0/757497638 wait complete. 2024-09-19T08:02:51.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:50 smithi162 ceph-mon[35260]: pgmap v322: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:51.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:50 smithi139 ceph-mon[34874]: pgmap v322: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:51.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:50 smithi098 ceph-mon[31333]: pgmap v322: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T08:02:51.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-19T08:02:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:52.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.584+0000 7f7c5fad3640 1 -- 172.21.15.98:0/3908862451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c58102dc0 msgr2=0x7f7c581031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:52.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.584+0000 7f7c5fad3640 1 --2- 172.21.15.98:0/3908862451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c58102dc0 0x7f7c581031a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009ca0 tx=0x7f7c5002a950 comp rx=0 tx=0).stop 2024-09-19T08:02:52.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.584+0000 7f7c5fad3640 1 -- 172.21.15.98:0/3908862451 shutdown_connections 2024-09-19T08:02:52.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.584+0000 7f7c5fad3640 1 --2- 172.21.15.98:0/3908862451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c5810c1a0 0x7f7c5810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.584+0000 7f7c5fad3640 1 --2- 172.21.15.98:0/3908862451 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c581036e0 0x7f7c5810bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.584+0000 7f7c5fad3640 1 --2- 172.21.15.98:0/3908862451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c58102dc0 0x7f7c581031a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.584+0000 7f7c5fad3640 1 -- 172.21.15.98:0/3908862451 >> 172.21.15.98:0/3908862451 conn(0x7f7c580fc7a0 msgr2=0x7f7c580febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:52.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.584+0000 7f7c5fad3640 1 -- 172.21.15.98:0/3908862451 shutdown_connections 2024-09-19T08:02:52.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.584+0000 7f7c5fad3640 1 -- 172.21.15.98:0/3908862451 wait complete. 2024-09-19T08:02:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5fad3640 1 Processor -- start 2024-09-19T08:02:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5fad3640 1 -- start start 2024-09-19T08:02:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5fad3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c58102dc0 0x7f7c58198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5fad3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c581036e0 0x7f7c58198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5fad3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c5810c1a0 0x7f7c5819cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5fad3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c58112080 con 0x7f7c581036e0 2024-09-19T08:02:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5fad3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7c58111f00 con 0x7f7c58102dc0 2024-09-19T08:02:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5fad3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7c58112200 con 0x7f7c5810c1a0 2024-09-19T08:02:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5d06e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c581036e0 0x7f7c58198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5d06e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c581036e0 0x7f7c58198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49810/0 (socket says 172.21.15.98:49810) 2024-09-19T08:02:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5d06e640 1 -- 172.21.15.98:0/3038012965 learned_addr learned my addr 172.21.15.98:0/3038012965 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/3038012965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c58102dc0 0x7f7c58198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5d06e640 1 -- 172.21.15.98:0/3038012965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c5810c1a0 msgr2=0x7f7c5819cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:52.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5d06e640 1 --2- 172.21.15.98:0/3038012965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c5810c1a0 0x7f7c5819cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5d06e640 1 -- 172.21.15.98:0/3038012965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c58102dc0 msgr2=0x7f7c58198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:52.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5d06e640 1 --2- 172.21.15.98:0/3038012965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c58102dc0 0x7f7c58198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5d06e640 1 -- 172.21.15.98:0/3038012965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c5819d640 con 0x7f7c581036e0 2024-09-19T08:02:52.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.585+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/3038012965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c58102dc0 0x7f7c58198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:52.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.586+0000 7f7c5d06e640 1 --2- 172.21.15.98:0/3038012965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c581036e0 0x7f7c58198bd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00b4a0 tx=0x7f7c4c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:52.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.586+0000 7f7c56ffd640 1 -- 172.21.15.98:0/3038012965 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c014070 con 0x7f7c581036e0 2024-09-19T08:02:52.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.586+0000 7f7c5fad3640 1 -- 172.21.15.98:0/3038012965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c5819d930 con 0x7f7c581036e0 2024-09-19T08:02:52.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.586+0000 7f7c56ffd640 1 -- 172.21.15.98:0/3038012965 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c4c010af0 con 0x7f7c581036e0 2024-09-19T08:02:52.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.586+0000 7f7c56ffd640 1 -- 172.21.15.98:0/3038012965 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c017d10 con 0x7f7c581036e0 2024-09-19T08:02:52.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.586+0000 7f7c5fad3640 1 -- 172.21.15.98:0/3038012965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c581a5210 con 0x7f7c581036e0 2024-09-19T08:02:52.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.588+0000 7f7c56ffd640 1 -- 172.21.15.98:0/3038012965 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7c4c017070 con 0x7f7c581036e0 2024-09-19T08:02:52.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.588+0000 7f7c61eac740 1 Processor -- start 2024-09-19T08:02:52.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c61eac740 1 -- start start 2024-09-19T08:02:52.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c61eac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf09d0 0x561c97cf0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c61eac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf12f0 0x561c97cfb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c61eac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cfbbb0 0x561c97cfdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c61eac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561c97c57090 con 0x561c97cf12f0 2024-09-19T08:02:52.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c61eac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561c97bfea70 con 0x561c97cf09d0 2024-09-19T08:02:52.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c61eac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561c97c53050 con 0x561c97cfbbb0 2024-09-19T08:02:52.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c5d86f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf12f0 0x561c97cfb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c5d86f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf12f0 0x561c97cfb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49814/0 (socket says 172.21.15.98:49814) 2024-09-19T08:02:52.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c5d86f640 1 -- 172.21.15.98:0/336752633 learned_addr learned my addr 172.21.15.98:0/336752633 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:52.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c5e070640 1 --2- 172.21.15.98:0/336752633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf09d0 0x561c97cf0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c5d06e640 1 --2- 172.21.15.98:0/336752633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cfbbb0 0x561c97cfdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c56ffd640 1 --2- 172.21.15.98:0/3038012965 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7c400a32d0 0x7f7c400a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c5d86f640 1 -- 172.21.15.98:0/336752633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cfbbb0 msgr2=0x561c97cfdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:52.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/336752633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cfbbb0 0x561c97cfdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c5d86f640 1 -- 172.21.15.98:0/336752633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf09d0 msgr2=0x561c97cf0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:52.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/336752633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf09d0 0x561c97cf0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c5d86f640 1 -- 172.21.15.98:0/336752633 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c97c0b130 con 0x561c97cf12f0 2024-09-19T08:02:52.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c5e070640 1 --2- 172.21.15.98:0/3038012965 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7c400a32d0 0x7f7c400a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c5d06e640 1 --2- 172.21.15.98:0/336752633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cfbbb0 0x561c97cfdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:52.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c56ffd640 1 -- 172.21.15.98:0/3038012965 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f7c4c0cee00 con 0x7f7c581036e0 2024-09-19T08:02:52.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.589+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/336752633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf12f0 0x561c97cfb670 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7c50035000 tx=0x7f7c5002f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b61819c2fa0160 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:52.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.590+0000 7f7c3ffff640 1 -- 172.21.15.98:0/336752633 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50034040 con 0x561c97cf12f0 2024-09-19T08:02:52.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.590+0000 7f7c3ffff640 1 -- 172.21.15.98:0/336752633 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c5002fe20 con 0x561c97cf12f0 2024-09-19T08:02:52.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.590+0000 7f7c5e070640 1 --2- 172.21.15.98:0/3038012965 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7c400a32d0 0x7f7c400a5790 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7c480099b0 tx=0x7f7c48008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:52.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.590+0000 7f7c3ffff640 1 -- 172.21.15.98:0/336752633 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c500045d0 con 0x561c97cf12f0 2024-09-19T08:02:52.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.590+0000 7f7c61eac740 1 -- 172.21.15.98:0/336752633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf12f0 msgr2=0x561c97cfb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:52.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.590+0000 7f7c61eac740 1 --2- 172.21.15.98:0/336752633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf12f0 0x561c97cfb670 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7c50035000 tx=0x7f7c5002f770 comp rx=0 tx=0).stop 2024-09-19T08:02:52.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.590+0000 7f7c61eac740 1 -- 172.21.15.98:0/336752633 shutdown_connections 2024-09-19T08:02:52.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.590+0000 7f7c61eac740 1 --2- 172.21.15.98:0/336752633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cfbbb0 0x561c97cfdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.590+0000 7f7c61eac740 1 --2- 172.21.15.98:0/336752633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf12f0 0x561c97cfb670 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.590+0000 7f7c61eac740 1 --2- 172.21.15.98:0/336752633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf09d0 0x561c97cf0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.590+0000 7f7c61eac740 1 -- 172.21.15.98:0/336752633 >> 172.21.15.98:0/336752633 conn(0x561c97cecf60 msgr2=0x561c97ced340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:52.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.591+0000 7f7c61eac740 1 -- 172.21.15.98:0/336752633 shutdown_connections 2024-09-19T08:02:52.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.591+0000 7f7c61eac740 1 -- 172.21.15.98:0/336752633 wait complete. 2024-09-19T08:02:52.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.591+0000 7f7c61eac740 1 -- start start 2024-09-19T08:02:52.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.591+0000 7f7c61eac740 1 Processor -- start 2024-09-19T08:02:52.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.591+0000 7f7c61eac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cf09d0 0x561c97cfb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.591+0000 7f7c61eac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf12f0 0x561c97cf86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.591+0000 7f7c61eac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cfbbb0 0x561c97cf8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.591+0000 7f7c61eac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561c97cfe780 con 0x561c97cfbbb0 2024-09-19T08:02:52.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.591+0000 7f7c61eac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561c97c11f20 con 0x561c97cf12f0 2024-09-19T08:02:52.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.591+0000 7f7c61eac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561c97cfe900 con 0x561c97cf09d0 2024-09-19T08:02:52.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.591+0000 7f7c61eac740 10 client.0 _ll_register_callbacks cb 0x561c97bdb030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:52.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.591+0000 7f7c5e070640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cfbbb0 0x561c97cf8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c5e070640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cfbbb0 0x561c97cf8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49820/0 (socket says 172.21.15.98:49820) 2024-09-19T08:02:52.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c5e070640 1 -- 172.21.15.98:0/5122731 learned_addr learned my addr 172.21.15.98:0/5122731 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:52.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/5122731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cf09d0 0x561c97cfb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c5d06e640 1 --2- 172.21.15.98:0/5122731 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf12f0 0x561c97cf86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c5e070640 1 -- 172.21.15.98:0/5122731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cf09d0 msgr2=0x561c97cfb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:52.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c5e070640 1 --2- 172.21.15.98:0/5122731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cf09d0 0x561c97cfb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c5e070640 1 -- 172.21.15.98:0/5122731 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf12f0 msgr2=0x561c97cf86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:52.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c5e070640 1 --2- 172.21.15.98:0/5122731 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf12f0 0x561c97cf86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c5e070640 1 -- 172.21.15.98:0/5122731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c979c82b0 con 0x561c97cfbbb0 2024-09-19T08:02:52.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c5e070640 1 --2- 172.21.15.98:0/5122731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cfbbb0 0x561c97cf8c10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7c48014680 tx=0x7f7c48006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf501ab8cad735e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:52.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:52.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c3d7fa640 1 -- 172.21.15.98:0/5122731 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48023020 con 0x561c97cfbbb0 2024-09-19T08:02:52.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c3d7fa640 1 -- 172.21.15.98:0/5122731 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f7c480136c0 con 0x561c97cfbbb0 2024-09-19T08:02:52.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c61eac740 1 -- 172.21.15.98:0/5122731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c5002fe20 con 0x561c97cfbbb0 2024-09-19T08:02:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c3d7fa640 1 -- 172.21.15.98:0/5122731 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7c480138c0 con 0x561c97cfbbb0 2024-09-19T08:02:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c61eac740 10 client.15204 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c61eac740 1 -- 172.21.15.98:0/5122731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561c97d0a010 con 0x561c97cfbbb0 2024-09-19T08:02:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.592+0000 7f7c3d7fa640 1 -- 172.21.15.98:0/5122731 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c5002fe20 con 0x561c97cfbbb0 2024-09-19T08:02:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.593+0000 7f7c3d7fa640 1 -- 172.21.15.98:0/5122731 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7c4801e3a0 con 0x561c97cfbbb0 2024-09-19T08:02:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.593+0000 7f7c61eac740 10 client.15204 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.593+0000 7f7c61eac740 10 client.15204 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-19T08:02:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.593+0000 7f7c61eac740 20 client.15204 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.593+0000 7f7c61eac740 1 --2- 172.21.15.98:0/5122731 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x561c97d0c640 0x561c97d0dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.593+0000 7f7c61eac740 4 client.15204 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:02:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.593+0000 7f7c61eac740 1 -- 172.21.15.98:0/5122731 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c97c64600 con 0x561c97d0c640 2024-09-19T08:02:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.593+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/5122731 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x561c97d0c640 0x561c97d0dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.593+0000 7f7c3d7fa640 10 client.15204 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:02:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.593+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/5122731 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x561c97d0c640 0x561c97d0dab0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a41764c3e755e084 server_cookie=a1079bcc70ef1f55 in_seq=0 out_seq=0 2024-09-19T08:02:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c3d7fa640 1 -- 172.21.15.98:0/5122731 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561c97c64600 con 0x561c97d0c640 2024-09-19T08:02:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c3d7fa640 10 client.15204 handle_command_reply: tid=0 2024-09-19T08:02:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c61eac740 1 client.15204 shutdown 2024-09-19T08:02:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c61eac740 1 -- 172.21.15.98:0/5122731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cfbbb0 msgr2=0x561c97cf8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c61eac740 1 --2- 172.21.15.98:0/5122731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cfbbb0 0x561c97cf8c10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7c48014680 tx=0x7f7c48006f90 comp rx=0 tx=0).stop 2024-09-19T08:02:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c61eac740 1 -- 172.21.15.98:0/5122731 shutdown_connections 2024-09-19T08:02:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c61eac740 1 --2- 172.21.15.98:0/5122731 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x561c97d0c640 0x561c97d0dab0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c61eac740 1 --2- 172.21.15.98:0/5122731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cfbbb0 0x561c97cf8c10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c61eac740 1 --2- 172.21.15.98:0/5122731 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf12f0 0x561c97cf86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c61eac740 1 --2- 172.21.15.98:0/5122731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cf09d0 0x561c97cfb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c3d7fa640 0 client.15204 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:02:52.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c61eac740 1 -- 172.21.15.98:0/5122731 >> 172.21.15.98:0/5122731 conn(0x561c97cecf60 msgr2=0x561c97cf2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:52.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c61eac740 1 -- 172.21.15.98:0/5122731 shutdown_connections 2024-09-19T08:02:52.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c61eac740 1 -- 172.21.15.98:0/5122731 wait complete. 2024-09-19T08:02:52.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.594+0000 7f7c61eac740 20 client.15204 trim_cache size 0 max 16384 2024-09-19T08:02:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.612+0000 7f7c61eac740 1 Processor -- start 2024-09-19T08:02:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.612+0000 7f7c61eac740 1 -- start start 2024-09-19T08:02:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.612+0000 7f7c61eac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf09d0 0x561c97d0bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.612+0000 7f7c61eac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cf12f0 0x561c97d0d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.612+0000 7f7c61eac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cfbbb0 0x561c97d16180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.612+0000 7f7c61eac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561c97d0bdf0 con 0x561c97cf09d0 2024-09-19T08:02:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.612+0000 7f7c61eac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561c97d0c4c0 con 0x561c97cfbbb0 2024-09-19T08:02:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.612+0000 7f7c61eac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561c97d0cb90 con 0x561c97cf12f0 2024-09-19T08:02:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.612+0000 7f7c5d86f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf09d0 0x561c97d0bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.612+0000 7f7c5d86f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf09d0 0x561c97d0bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49822/0 (socket says 172.21.15.98:49822) 2024-09-19T08:02:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.612+0000 7f7c5d86f640 1 -- 172.21.15.98:0/636863857 learned_addr learned my addr 172.21.15.98:0/636863857 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.613+0000 7f7c5d06e640 1 --2- 172.21.15.98:0/636863857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cf12f0 0x561c97d0d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.613+0000 7f7c5d86f640 1 -- 172.21.15.98:0/636863857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cf12f0 msgr2=0x561c97d0d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.613+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/636863857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cf12f0 0x561c97d0d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.613+0000 7f7c5d86f640 1 -- 172.21.15.98:0/636863857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cfbbb0 msgr2=0x561c97d16180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.613+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/636863857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cfbbb0 0x561c97d16180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.613+0000 7f7c5d86f640 1 -- 172.21.15.98:0/636863857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c97cecd90 con 0x561c97cf09d0 2024-09-19T08:02:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.613+0000 7f7c5d06e640 1 --2- 172.21.15.98:0/636863857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cf12f0 0x561c97d0d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.613+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/636863857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf09d0 0x561c97d0bf70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7c500041e0 tx=0x7f7c50004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b3867171b4ba58 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.613+0000 7f7c3f7fe640 1 -- 172.21.15.98:0/636863857 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50054070 con 0x561c97cf09d0 2024-09-19T08:02:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.613+0000 7f7c3f7fe640 1 -- 172.21.15.98:0/636863857 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c5002fe20 con 0x561c97cf09d0 2024-09-19T08:02:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.613+0000 7f7c3f7fe640 1 -- 172.21.15.98:0/636863857 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50034040 con 0x561c97cf09d0 2024-09-19T08:02:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 -- 172.21.15.98:0/636863857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf09d0 msgr2=0x561c97d0bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 --2- 172.21.15.98:0/636863857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf09d0 0x561c97d0bf70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7c500041e0 tx=0x7f7c50004170 comp rx=0 tx=0).stop 2024-09-19T08:02:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 -- 172.21.15.98:0/636863857 shutdown_connections 2024-09-19T08:02:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 --2- 172.21.15.98:0/636863857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cfbbb0 0x561c97d16180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 --2- 172.21.15.98:0/636863857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cf12f0 0x561c97d0d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 --2- 172.21.15.98:0/636863857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf09d0 0x561c97d0bf70 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 -- 172.21.15.98:0/636863857 >> 172.21.15.98:0/636863857 conn(0x561c97cecf60 msgr2=0x561c97cf7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 -- 172.21.15.98:0/636863857 shutdown_connections 2024-09-19T08:02:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 -- 172.21.15.98:0/636863857 wait complete. 2024-09-19T08:02:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 -- start start 2024-09-19T08:02:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 Processor -- start 2024-09-19T08:02:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf09d0 0x561c97d165f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf12f0 0x561c97cf5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cfbbb0 0x561c97cf6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561c97d0d930 con 0x561c97cf12f0 2024-09-19T08:02:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561c97d0d260 con 0x561c97cf09d0 2024-09-19T08:02:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561c97d0e000 con 0x561c97cfbbb0 2024-09-19T08:02:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.614+0000 7f7c61eac740 10 client.0 _ll_register_callbacks cb 0x7f7c48009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c5d06e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf12f0 0x561c97cf5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c5d06e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf12f0 0x561c97cf5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49830/0 (socket says 172.21.15.98:49830) 2024-09-19T08:02:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c5d06e640 1 -- 172.21.15.98:0/2087736640 learned_addr learned my addr 172.21.15.98:0/2087736640 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c5e070640 1 --2- 172.21.15.98:0/2087736640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cfbbb0 0x561c97cf6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/2087736640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf09d0 0x561c97d165f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c5d06e640 1 -- 172.21.15.98:0/2087736640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cfbbb0 msgr2=0x561c97cf6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c5d06e640 1 --2- 172.21.15.98:0/2087736640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cfbbb0 0x561c97cf6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c5d06e640 1 -- 172.21.15.98:0/2087736640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf09d0 msgr2=0x561c97d165f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c5d06e640 1 --2- 172.21.15.98:0/2087736640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf09d0 0x561c97d165f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c5d06e640 1 -- 172.21.15.98:0/2087736640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c97d15170 con 0x561c97cf12f0 2024-09-19T08:02:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/2087736640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf09d0 0x561c97d165f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c5d06e640 1 --2- 172.21.15.98:0/2087736640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf12f0 0x561c97cf5950 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009420 tx=0x7f7c4c09b020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7071fdbf55ce1371 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c3d7fa640 1 -- 172.21.15.98:0/2087736640 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c01e040 con 0x561c97cf12f0 2024-09-19T08:02:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c3d7fa640 1 -- 172.21.15.98:0/2087736640 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f7c4c010ca0 con 0x561c97cf12f0 2024-09-19T08:02:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c61eac740 1 -- 172.21.15.98:0/2087736640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c5002fe20 con 0x561c97cf12f0 2024-09-19T08:02:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c3d7fa640 1 -- 172.21.15.98:0/2087736640 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7c4c09b3e0 con 0x561c97cf12f0 2024-09-19T08:02:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c61eac740 10 client.15210 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.615+0000 7f7c61eac740 1 -- 172.21.15.98:0/2087736640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561c97d79e70 con 0x561c97cf12f0 2024-09-19T08:02:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.616+0000 7f7c3d7fa640 1 -- 172.21.15.98:0/2087736640 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c5002fe20 con 0x561c97cf12f0 2024-09-19T08:02:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.616+0000 7f7c3d7fa640 1 -- 172.21.15.98:0/2087736640 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7c4c0b4260 con 0x561c97cf12f0 2024-09-19T08:02:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.616+0000 7f7c61eac740 10 client.15210 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.616+0000 7f7c61eac740 10 client.15210 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-19T08:02:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.616+0000 7f7c61eac740 20 client.15210 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.616+0000 7f7c61eac740 1 --2- 172.21.15.98:0/2087736640 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x561c97d7ea80 0x561c97d7d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.616+0000 7f7c61eac740 4 client.15210 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:02:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.616+0000 7f7c61eac740 1 -- 172.21.15.98:0/2087736640 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "status"}) -- 0x561c97c5eb10 con 0x561c97d7ea80 2024-09-19T08:02:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.616+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/2087736640 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x561c97d7ea80 0x561c97d7d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c5d86f640 1 --2- 172.21.15.98:0/2087736640 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x561c97d7ea80 0x561c97d7d790 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a94d36a7734c3dc5 server_cookie=f77f9a7956f2a3a1 in_seq=0 out_seq=0 2024-09-19T08:02:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c3d7fa640 10 client.15210 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:02:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c3d7fa640 1 -- 172.21.15.98:0/2087736640 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561c97c5eb10 con 0x561c97d7ea80 2024-09-19T08:02:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c3d7fa640 10 client.15210 handle_command_reply: tid=0 2024-09-19T08:02:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c61eac740 1 client.15210 shutdown 2024-09-19T08:02:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c61eac740 1 -- 172.21.15.98:0/2087736640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf12f0 msgr2=0x561c97cf5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c61eac740 1 --2- 172.21.15.98:0/2087736640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf12f0 0x561c97cf5950 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009420 tx=0x7f7c4c09b020 comp rx=0 tx=0).stop 2024-09-19T08:02:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c61eac740 1 -- 172.21.15.98:0/2087736640 shutdown_connections 2024-09-19T08:02:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c61eac740 1 --2- 172.21.15.98:0/2087736640 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x561c97d7ea80 0x561c97d7d790 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c61eac740 1 --2- 172.21.15.98:0/2087736640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c97cfbbb0 0x561c97cf6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c61eac740 1 --2- 172.21.15.98:0/2087736640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c97cf12f0 0x561c97cf5950 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c3d7fa640 0 client.15210 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:02:52.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c61eac740 1 --2- 172.21.15.98:0/2087736640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c97cf09d0 0x561c97d165f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c61eac740 1 -- 172.21.15.98:0/2087736640 >> 172.21.15.98:0/2087736640 conn(0x561c97cecf60 msgr2=0x561c97ced340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:52.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c61eac740 1 -- 172.21.15.98:0/2087736640 shutdown_connections 2024-09-19T08:02:52.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c61eac740 1 -- 172.21.15.98:0/2087736640 wait complete. 2024-09-19T08:02:52.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.617+0000 7f7c61eac740 20 client.15210 trim_cache size 0 max 16384 2024-09-19T08:02:52.626 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:02:52.626 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:02:52.626 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2024-09-19T08:02:52.626 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24461, 2024-09-19T08:02:52.627 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:02:52.627 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:02:52.627 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:02:52.627 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 93.800531559000007, 2024-09-19T08:02:52.627 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 27, 2024-09-19T08:02:52.627 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 80, 2024-09-19T08:02:52.627 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 78, 2024-09-19T08:02:52.627 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 123.847047717 2024-09-19T08:02:52.627 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:02:52.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.618+0000 7f7c5fad3640 1 -- 172.21.15.98:0/3038012965 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7c400a32d0 msgr2=0x7f7c400a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:52.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.618+0000 7f7c5fad3640 1 --2- 172.21.15.98:0/3038012965 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7c400a32d0 0x7f7c400a5790 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7c480099b0 tx=0x7f7c48008040 comp rx=0 tx=0).stop 2024-09-19T08:02:52.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.618+0000 7f7c5fad3640 1 -- 172.21.15.98:0/3038012965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c581036e0 msgr2=0x7f7c58198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:52.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.618+0000 7f7c5fad3640 1 --2- 172.21.15.98:0/3038012965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c581036e0 0x7f7c58198bd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00b4a0 tx=0x7f7c4c00b960 comp rx=0 tx=0).stop 2024-09-19T08:02:52.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.618+0000 7f7c5fad3640 1 -- 172.21.15.98:0/3038012965 shutdown_connections 2024-09-19T08:02:52.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.618+0000 7f7c5fad3640 1 --2- 172.21.15.98:0/3038012965 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f7c400a32d0 0x7f7c400a5790 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.618+0000 7f7c5fad3640 1 --2- 172.21.15.98:0/3038012965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c5810c1a0 0x7f7c5819cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.618+0000 7f7c5fad3640 1 --2- 172.21.15.98:0/3038012965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c581036e0 0x7f7c58198bd0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.618+0000 7f7c5fad3640 1 --2- 172.21.15.98:0/3038012965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c58102dc0 0x7f7c58198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:52.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.618+0000 7f7c5fad3640 1 -- 172.21.15.98:0/3038012965 >> 172.21.15.98:0/3038012965 conn(0x7f7c580fc7a0 msgr2=0x7f7c580fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:52.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.618+0000 7f7c5fad3640 1 -- 172.21.15.98:0/3038012965 shutdown_connections 2024-09-19T08:02:52.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:52.618+0000 7f7c5fad3640 1 -- 172.21.15.98:0/3038012965 wait complete. 2024-09-19T08:02:52.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:52 smithi098 ceph-mon[31333]: pgmap v323: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:53.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:52 smithi162 ceph-mon[35260]: pgmap v323: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:53.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:52 smithi139 ceph-mon[34874]: pgmap v323: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-19T08:02:53.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-19T08:02:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:54.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.159+0000 7f79eb936640 1 -- 172.21.15.98:0/23936652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79e4102ef0 msgr2=0x7f79e410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:54.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.159+0000 7f79eb936640 1 --2- 172.21.15.98:0/23936652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79e4102ef0 0x7f79e410da40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009ca0 tx=0x7f79dc02a950 comp rx=0 tx=0).stop 2024-09-19T08:02:54.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.159+0000 7f79eb936640 1 -- 172.21.15.98:0/23936652 shutdown_connections 2024-09-19T08:02:54.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.159+0000 7f79eb936640 1 --2- 172.21.15.98:0/23936652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79e4102ef0 0x7f79e410da40 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.159+0000 7f79eb936640 1 --2- 172.21.15.98:0/23936652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f79e4102570 0x7f79e41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.159+0000 7f79eb936640 1 --2- 172.21.15.98:0/23936652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79e4100e20 0x7f79e4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.159+0000 7f79eb936640 1 -- 172.21.15.98:0/23936652 >> 172.21.15.98:0/23936652 conn(0x7f79e40fc7d0 msgr2=0x7f79e40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:54.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.159+0000 7f79eb936640 1 -- 172.21.15.98:0/23936652 shutdown_connections 2024-09-19T08:02:54.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.159+0000 7f79eb936640 1 -- 172.21.15.98:0/23936652 wait complete. 2024-09-19T08:02:54.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79eb936640 1 Processor -- start 2024-09-19T08:02:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79eb936640 1 -- start start 2024-09-19T08:02:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79eb936640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f79e4100e20 0x7f79e4199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79eb936640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79e4102570 0x7f79e4199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79eb936640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79e4102ef0 0x7f79e41a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79eb936640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79e4111020 con 0x7f79e4102570 2024-09-19T08:02:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79eb936640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f79e4110ea0 con 0x7f79e4100e20 2024-09-19T08:02:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79eb936640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f79e41111a0 con 0x7f79e4102ef0 2024-09-19T08:02:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79e8ed1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79e4102570 0x7f79e4199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79e8ed1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79e4102570 0x7f79e4199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49854/0 (socket says 172.21.15.98:49854) 2024-09-19T08:02:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79e8ed1640 1 -- 172.21.15.98:0/4013010806 learned_addr learned my addr 172.21.15.98:0/4013010806 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79e96d2640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f79e4100e20 0x7f79e4199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79e8ed1640 1 -- 172.21.15.98:0/4013010806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79e4102ef0 msgr2=0x7f79e41a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79e9ed3640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79e4102ef0 0x7f79e41a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79e8ed1640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79e4102ef0 0x7f79e41a44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79e8ed1640 1 -- 172.21.15.98:0/4013010806 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f79e4100e20 msgr2=0x7f79e4199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79e8ed1640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f79e4100e20 0x7f79e4199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79e8ed1640 1 -- 172.21.15.98:0/4013010806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e41a4a10 con 0x7f79e4102570 2024-09-19T08:02:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79e96d2640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f79e4100e20 0x7f79e4199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.160+0000 7f79e8ed1640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79e4102570 0x7f79e4199990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f79e000b4a0 tx=0x7f79e000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.161+0000 7f79da7fc640 1 -- 172.21.15.98:0/4013010806 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e0014070 con 0x7f79e4102570 2024-09-19T08:02:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.161+0000 7f79da7fc640 1 -- 172.21.15.98:0/4013010806 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e0010af0 con 0x7f79e4102570 2024-09-19T08:02:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.161+0000 7f79eb936640 1 -- 172.21.15.98:0/4013010806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e41a4be0 con 0x7f79e4102570 2024-09-19T08:02:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.161+0000 7f79da7fc640 1 -- 172.21.15.98:0/4013010806 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e0017d10 con 0x7f79e4102570 2024-09-19T08:02:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.161+0000 7f79eb936640 1 -- 172.21.15.98:0/4013010806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e41a5120 con 0x7f79e4102570 2024-09-19T08:02:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.163+0000 7f79da7fc640 1 -- 172.21.15.98:0/4013010806 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f79e0017070 con 0x7f79e4102570 2024-09-19T08:02:54.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.163+0000 7f79edd0f740 1 Processor -- start 2024-09-19T08:02:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79edd0f740 1 -- start start 2024-09-19T08:02:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79edd0f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e341db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79edd0f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e3422f0 0x55953e34c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79edd0f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e34cbb0 0x55953e34efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79edd0f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55953e2a8090 con 0x55953e3422f0 2024-09-19T08:02:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79edd0f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55953e24fa70 con 0x55953e3419d0 2024-09-19T08:02:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79edd0f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55953e2a4050 con 0x55953e34cbb0 2024-09-19T08:02:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79e96d2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e3422f0 0x55953e34c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79e96d2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e3422f0 0x55953e34c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49866/0 (socket says 172.21.15.98:49866) 2024-09-19T08:02:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79e96d2640 1 -- 172.21.15.98:0/3315830616 learned_addr learned my addr 172.21.15.98:0/3315830616 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79da7fc640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f79d00a3350 0x7f79d00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79e9ed3640 1 --2- 172.21.15.98:0/3315830616 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e341db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79e8ed1640 1 --2- 172.21.15.98:0/3315830616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e34cbb0 0x55953e34efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79e96d2640 1 -- 172.21.15.98:0/3315830616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e34cbb0 msgr2=0x55953e34efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79e96d2640 1 --2- 172.21.15.98:0/3315830616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e34cbb0 0x55953e34efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79e96d2640 1 -- 172.21.15.98:0/3315830616 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 msgr2=0x55953e341db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79e9ed3640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f79d00a3350 0x7f79d00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79e96d2640 1 --2- 172.21.15.98:0/3315830616 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e341db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79e96d2640 1 -- 172.21.15.98:0/3315830616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55953e25c130 con 0x55953e3422f0 2024-09-19T08:02:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79da7fc640 1 -- 172.21.15.98:0/4013010806 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f79e00ced80 con 0x7f79e4102570 2024-09-19T08:02:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.164+0000 7f79e8ed1640 1 --2- 172.21.15.98:0/3315830616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e34cbb0 0x55953e34efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.165+0000 7f79e96d2640 1 --2- 172.21.15.98:0/3315830616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e3422f0 0x55953e34c670 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f79d400aa10 tx=0x7f79d400aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a339e6fbd1a453 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.165+0000 7f79cf7fe640 1 -- 172.21.15.98:0/3315830616 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d4015070 con 0x55953e3422f0 2024-09-19T08:02:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.165+0000 7f79e9ed3640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f79d00a3350 0x7f79d00a5810 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f79dc004af0 tx=0x7f79dc0095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.165+0000 7f79cf7fe640 1 -- 172.21.15.98:0/3315830616 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79d400cc60 con 0x55953e3422f0 2024-09-19T08:02:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.165+0000 7f79cf7fe640 1 -- 172.21.15.98:0/3315830616 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d40106c0 con 0x55953e3422f0 2024-09-19T08:02:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.165+0000 7f79edd0f740 1 -- 172.21.15.98:0/3315830616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e3422f0 msgr2=0x55953e34c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.165+0000 7f79edd0f740 1 --2- 172.21.15.98:0/3315830616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e3422f0 0x55953e34c670 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f79d400aa10 tx=0x7f79d400aed0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.165+0000 7f79edd0f740 1 -- 172.21.15.98:0/3315830616 shutdown_connections 2024-09-19T08:02:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.165+0000 7f79edd0f740 1 --2- 172.21.15.98:0/3315830616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e34cbb0 0x55953e34efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.165+0000 7f79edd0f740 1 --2- 172.21.15.98:0/3315830616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e3422f0 0x55953e34c670 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.165+0000 7f79edd0f740 1 --2- 172.21.15.98:0/3315830616 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e341db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.165+0000 7f79edd0f740 1 -- 172.21.15.98:0/3315830616 >> 172.21.15.98:0/3315830616 conn(0x55953e33df60 msgr2=0x55953e33e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.166+0000 7f79edd0f740 1 -- 172.21.15.98:0/3315830616 shutdown_connections 2024-09-19T08:02:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.166+0000 7f79edd0f740 1 -- 172.21.15.98:0/3315830616 wait complete. 2024-09-19T08:02:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.166+0000 7f79edd0f740 1 -- start start 2024-09-19T08:02:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.166+0000 7f79edd0f740 1 Processor -- start 2024-09-19T08:02:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.166+0000 7f79edd0f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e34c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.166+0000 7f79edd0f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e3422f0 0x55953e3496f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.166+0000 7f79edd0f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e34cbb0 0x55953e349c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.166+0000 7f79edd0f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55953e34f780 con 0x55953e3422f0 2024-09-19T08:02:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.166+0000 7f79edd0f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55953e262f20 con 0x55953e3419d0 2024-09-19T08:02:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.166+0000 7f79edd0f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55953e34f900 con 0x55953e34cbb0 2024-09-19T08:02:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.166+0000 7f79edd0f740 10 client.0 _ll_register_callbacks cb 0x55953e22c030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.166+0000 7f79e8ed1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e3422f0 0x55953e3496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.166+0000 7f79e8ed1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e3422f0 0x55953e3496f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49880/0 (socket says 172.21.15.98:49880) 2024-09-19T08:02:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79e8ed1640 1 -- 172.21.15.98:0/1364032126 learned_addr learned my addr 172.21.15.98:0/1364032126 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79e96d2640 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e34c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79e8ed1640 1 -- 172.21.15.98:0/1364032126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e34cbb0 msgr2=0x55953e349c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79e9ed3640 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e34cbb0 0x55953e349c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79e8ed1640 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e34cbb0 0x55953e349c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79e8ed1640 1 -- 172.21.15.98:0/1364032126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 msgr2=0x55953e34c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79e8ed1640 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e34c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79e8ed1640 1 -- 172.21.15.98:0/1364032126 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55953e0192b0 con 0x55953e3422f0 2024-09-19T08:02:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79e9ed3640 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e34cbb0 0x55953e349c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:02:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79e96d2640 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e34c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79e8ed1640 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e3422f0 0x55953e3496f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f79e001f000 tx=0x7f79e0009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9aa5d3fd8fa86bb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79ccff9640 1 -- 172.21.15.98:0/1364032126 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e00d0070 con 0x55953e3422f0 2024-09-19T08:02:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79ccff9640 1 -- 172.21.15.98:0/1364032126 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f79e0097970 con 0x55953e3422f0 2024-09-19T08:02:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79edd0f740 1 -- 172.21.15.98:0/1364032126 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79d400cc60 con 0x55953e3422f0 2024-09-19T08:02:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79ccff9640 1 -- 172.21.15.98:0/1364032126 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f79e001e040 con 0x55953e3422f0 2024-09-19T08:02:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79edd0f740 10 client.15222 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79edd0f740 1 -- 172.21.15.98:0/1364032126 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55953e35aff0 con 0x55953e3422f0 2024-09-19T08:02:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.167+0000 7f79ccff9640 1 -- 172.21.15.98:0/1364032126 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79d400cc60 con 0x55953e3422f0 2024-09-19T08:02:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.168+0000 7f79ccff9640 1 -- 172.21.15.98:0/1364032126 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f79e00cc930 con 0x55953e3422f0 2024-09-19T08:02:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.168+0000 7f79edd0f740 10 client.15222 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.168+0000 7f79edd0f740 10 client.15222 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-19T08:02:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.168+0000 7f79edd0f740 20 client.15222 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.168+0000 7f79edd0f740 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55953e35d5c0 0x55953e35ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.168+0000 7f79edd0f740 4 client.15222 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:02:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.168+0000 7f79edd0f740 1 -- 172.21.15.98:0/1364032126 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55953e2b5600 con 0x55953e35d5c0 2024-09-19T08:02:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.168+0000 7f79e96d2640 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55953e35d5c0 0x55953e35ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.168+0000 7f79ccff9640 10 client.15222 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:02:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.168+0000 7f79e96d2640 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55953e35d5c0 0x55953e35ea30 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=887e9ba8be428fc2 server_cookie=10e68d85dd10cdb9 in_seq=0 out_seq=0 2024-09-19T08:02:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.169+0000 7f79ccff9640 1 -- 172.21.15.98:0/1364032126 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55953e2b5600 con 0x55953e35d5c0 2024-09-19T08:02:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.169+0000 7f79ccff9640 10 client.15222 handle_command_reply: tid=0 2024-09-19T08:02:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.169+0000 7f79edd0f740 1 client.15222 shutdown 2024-09-19T08:02:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.169+0000 7f79edd0f740 1 -- 172.21.15.98:0/1364032126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e3422f0 msgr2=0x55953e3496f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.169+0000 7f79edd0f740 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e3422f0 0x55953e3496f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f79e001f000 tx=0x7f79e0009390 comp rx=0 tx=0).stop 2024-09-19T08:02:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.169+0000 7f79edd0f740 1 -- 172.21.15.98:0/1364032126 shutdown_connections 2024-09-19T08:02:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.169+0000 7f79edd0f740 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55953e35d5c0 0x55953e35ea30 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.169+0000 7f79edd0f740 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e34cbb0 0x55953e349c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.169+0000 7f79ccff9640 0 client.15222 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:02:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.169+0000 7f79edd0f740 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e3422f0 0x55953e3496f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.169+0000 7f79edd0f740 1 --2- 172.21.15.98:0/1364032126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e34c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.169+0000 7f79edd0f740 1 -- 172.21.15.98:0/1364032126 >> 172.21.15.98:0/1364032126 conn(0x55953e33df60 msgr2=0x55953e343c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.169+0000 7f79edd0f740 1 -- 172.21.15.98:0/1364032126 shutdown_connections 2024-09-19T08:02:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.169+0000 7f79edd0f740 1 -- 172.21.15.98:0/1364032126 wait complete. 2024-09-19T08:02:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.170+0000 7f79edd0f740 20 client.15222 trim_cache size 0 max 16384 2024-09-19T08:02:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.187+0000 7f79edd0f740 1 Processor -- start 2024-09-19T08:02:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.187+0000 7f79edd0f740 1 -- start start 2024-09-19T08:02:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.187+0000 7f79edd0f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e359010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.187+0000 7f79edd0f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e3422f0 0x55953e348fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.187+0000 7f79edd0f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e34cbb0 0x55953e347fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.187+0000 7f79edd0f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55953e35cd70 con 0x55953e34cbb0 2024-09-19T08:02:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.187+0000 7f79edd0f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55953e35d440 con 0x55953e3419d0 2024-09-19T08:02:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.187+0000 7f79edd0f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55953e35db10 con 0x55953e3422f0 2024-09-19T08:02:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.187+0000 7f79e9ed3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e34cbb0 0x55953e347fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.187+0000 7f79e9ed3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e34cbb0 0x55953e347fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49892/0 (socket says 172.21.15.98:49892) 2024-09-19T08:02:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.187+0000 7f79e9ed3640 1 -- 172.21.15.98:0/585360124 learned_addr learned my addr 172.21.15.98:0/585360124 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.188+0000 7f79e8ed1640 1 --2- 172.21.15.98:0/585360124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e3422f0 0x55953e348fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.188+0000 7f79e96d2640 1 --2- 172.21.15.98:0/585360124 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e359010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.188+0000 7f79e9ed3640 1 -- 172.21.15.98:0/585360124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e3422f0 msgr2=0x55953e348fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.188+0000 7f79e9ed3640 1 --2- 172.21.15.98:0/585360124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e3422f0 0x55953e348fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.188+0000 7f79e9ed3640 1 -- 172.21.15.98:0/585360124 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 msgr2=0x55953e359010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.188+0000 7f79e9ed3640 1 --2- 172.21.15.98:0/585360124 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e359010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.188+0000 7f79e9ed3640 1 -- 172.21.15.98:0/585360124 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55953e33dd90 con 0x55953e34cbb0 2024-09-19T08:02:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.188+0000 7f79e9ed3640 1 --2- 172.21.15.98:0/585360124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e34cbb0 0x55953e347fc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009390 tx=0x7f79dc03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33ea9fba11c3e5f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.188+0000 7f79ceffd640 1 -- 172.21.15.98:0/585360124 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc034040 con 0x55953e34cbb0 2024-09-19T08:02:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.188+0000 7f79ceffd640 1 -- 172.21.15.98:0/585360124 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79dc003d70 con 0x55953e34cbb0 2024-09-19T08:02:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.188+0000 7f79ceffd640 1 -- 172.21.15.98:0/585360124 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc048a40 con 0x55953e34cbb0 2024-09-19T08:02:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 -- 172.21.15.98:0/585360124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e34cbb0 msgr2=0x55953e347fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 --2- 172.21.15.98:0/585360124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e34cbb0 0x55953e347fc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009390 tx=0x7f79dc03f000 comp rx=0 tx=0).stop 2024-09-19T08:02:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 -- 172.21.15.98:0/585360124 shutdown_connections 2024-09-19T08:02:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 --2- 172.21.15.98:0/585360124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e34cbb0 0x55953e347fc0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 --2- 172.21.15.98:0/585360124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e3422f0 0x55953e348fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 --2- 172.21.15.98:0/585360124 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e359010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 -- 172.21.15.98:0/585360124 >> 172.21.15.98:0/585360124 conn(0x55953e33df60 msgr2=0x55953e33e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 -- 172.21.15.98:0/585360124 shutdown_connections 2024-09-19T08:02:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 -- 172.21.15.98:0/585360124 wait complete. 2024-09-19T08:02:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 -- start start 2024-09-19T08:02:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 Processor -- start 2024-09-19T08:02:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e3639a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e3422f0 0x55953e363ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e34cbb0 0x55953e364880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55953e35e8b0 con 0x55953e34cbb0 2024-09-19T08:02:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55953e35e1e0 con 0x55953e3419d0 2024-09-19T08:02:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.189+0000 7f79edd0f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55953e35ef80 con 0x55953e3422f0 2024-09-19T08:02:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79edd0f740 10 client.0 _ll_register_callbacks cb 0x7f79e0017f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:02:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79e9ed3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e34cbb0 0x55953e364880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79e9ed3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e34cbb0 0x55953e364880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49906/0 (socket says 172.21.15.98:49906) 2024-09-19T08:02:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79e9ed3640 1 -- 172.21.15.98:0/729155052 learned_addr learned my addr 172.21.15.98:0/729155052 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79e96d2640 1 --2- 172.21.15.98:0/729155052 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e3639a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79e8ed1640 1 --2- 172.21.15.98:0/729155052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e3422f0 0x55953e363ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79e9ed3640 1 -- 172.21.15.98:0/729155052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e3422f0 msgr2=0x55953e363ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79e9ed3640 1 --2- 172.21.15.98:0/729155052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e3422f0 0x55953e363ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79e9ed3640 1 -- 172.21.15.98:0/729155052 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 msgr2=0x55953e3639a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:02:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79e9ed3640 1 --2- 172.21.15.98:0/729155052 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e3639a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79e9ed3640 1 -- 172.21.15.98:0/729155052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55953e364ec0 con 0x55953e34cbb0 2024-09-19T08:02:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79e9ed3640 1 --2- 172.21.15.98:0/729155052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e34cbb0 0x55953e364880 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009360 tx=0x7f79dc004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddfd428e44479ce3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:02:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79ccff9640 1 -- 172.21.15.98:0/729155052 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc057070 con 0x55953e34cbb0 2024-09-19T08:02:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79ccff9640 1 -- 172.21.15.98:0/729155052 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f79dc0489d0 con 0x55953e34cbb0 2024-09-19T08:02:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79edd0f740 1 -- 172.21.15.98:0/729155052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79dc003d70 con 0x55953e34cbb0 2024-09-19T08:02:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79ccff9640 1 -- 172.21.15.98:0/729155052 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f79dc063050 con 0x55953e34cbb0 2024-09-19T08:02:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79edd0f740 10 client.15228 fetch_fsmap learned FSMap version 27 2024-09-19T08:02:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.190+0000 7f79edd0f740 1 -- 172.21.15.98:0/729155052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55953e366250 con 0x55953e34cbb0 2024-09-19T08:02:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.191+0000 7f79ccff9640 1 -- 172.21.15.98:0/729155052 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79dc003d70 con 0x55953e34cbb0 2024-09-19T08:02:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.191+0000 7f79ccff9640 1 -- 172.21.15.98:0/729155052 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f79dc045040 con 0x55953e34cbb0 2024-09-19T08:02:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.191+0000 7f79edd0f740 10 client.15228 fetch_fsmap finished waiting for FSMap version 27 2024-09-19T08:02:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.191+0000 7f79edd0f740 10 client.15228 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-19T08:02:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.191+0000 7f79edd0f740 20 client.15228 populate_metadata read hostname 'smithi098' 2024-09-19T08:02:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.191+0000 7f79edd0f740 1 --2- 172.21.15.98:0/729155052 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55953e3cbcc0 0x55953e3ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.191+0000 7f79edd0f740 4 client.15228 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:02:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.191+0000 7f79edd0f740 1 -- 172.21.15.98:0/729155052 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "status"}) -- 0x55953e2afb10 con 0x55953e3cbcc0 2024-09-19T08:02:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.191+0000 7f79e96d2640 1 --2- 172.21.15.98:0/729155052 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55953e3cbcc0 0x55953e3ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.191+0000 7f79e96d2640 1 --2- 172.21.15.98:0/729155052 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55953e3cbcc0 0x55953e3ca9d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=476517a7de54dcbf server_cookie=a26eff9340e0cd4c in_seq=0 out_seq=0 2024-09-19T08:02:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.191+0000 7f79ccff9640 10 client.15228 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:02:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79ccff9640 1 -- 172.21.15.98:0/729155052 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55953e2afb10 con 0x55953e3cbcc0 2024-09-19T08:02:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79ccff9640 10 client.15228 handle_command_reply: tid=0 2024-09-19T08:02:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79edd0f740 1 client.15228 shutdown 2024-09-19T08:02:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79edd0f740 1 -- 172.21.15.98:0/729155052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e34cbb0 msgr2=0x55953e364880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79edd0f740 1 --2- 172.21.15.98:0/729155052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e34cbb0 0x55953e364880 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009360 tx=0x7f79dc004990 comp rx=0 tx=0).stop 2024-09-19T08:02:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79edd0f740 1 -- 172.21.15.98:0/729155052 shutdown_connections 2024-09-19T08:02:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79edd0f740 1 --2- 172.21.15.98:0/729155052 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55953e3cbcc0 0x55953e3ca9d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79edd0f740 1 --2- 172.21.15.98:0/729155052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55953e34cbb0 0x55953e364880 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79edd0f740 1 --2- 172.21.15.98:0/729155052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55953e3422f0 0x55953e363ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79edd0f740 1 --2- 172.21.15.98:0/729155052 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55953e3419d0 0x55953e3639a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79edd0f740 1 -- 172.21.15.98:0/729155052 >> 172.21.15.98:0/729155052 conn(0x55953e33df60 msgr2=0x55953e347bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79ccff9640 0 client.15228 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:02:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79edd0f740 1 -- 172.21.15.98:0/729155052 shutdown_connections 2024-09-19T08:02:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79edd0f740 1 -- 172.21.15.98:0/729155052 wait complete. 2024-09-19T08:02:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.192+0000 7f79edd0f740 20 client.15228 trim_cache size 0 max 16384 2024-09-19T08:02:54.203 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:02:54.203 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:02:54.203 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-09-19T08:02:54.204 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24485, 2024-09-19T08:02:54.204 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:02:54.204 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:02:54.204 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:02:54.204 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 102.41034663400001, 2024-09-19T08:02:54.204 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 27, 2024-09-19T08:02:54.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 80, 2024-09-19T08:02:54.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 78, 2024-09-19T08:02:54.204 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 113.802577328 2024-09-19T08:02:54.204 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:02:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.193+0000 7f79eb936640 1 -- 172.21.15.98:0/4013010806 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f79d00a3350 msgr2=0x7f79d00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.193+0000 7f79eb936640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f79d00a3350 0x7f79d00a5810 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f79dc004af0 tx=0x7f79dc0095c0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.193+0000 7f79eb936640 1 -- 172.21.15.98:0/4013010806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79e4102570 msgr2=0x7f79e4199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.193+0000 7f79eb936640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79e4102570 0x7f79e4199990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f79e000b4a0 tx=0x7f79e000b960 comp rx=0 tx=0).stop 2024-09-19T08:02:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.193+0000 7f79eb936640 1 -- 172.21.15.98:0/4013010806 shutdown_connections 2024-09-19T08:02:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.193+0000 7f79eb936640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f79d00a3350 0x7f79d00a5810 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.193+0000 7f79eb936640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79e4102ef0 0x7f79e41a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.193+0000 7f79eb936640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79e4102570 0x7f79e4199990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.193+0000 7f79eb936640 1 --2- 172.21.15.98:0/4013010806 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f79e4100e20 0x7f79e4199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:54.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.193+0000 7f79eb936640 1 -- 172.21.15.98:0/4013010806 >> 172.21.15.98:0/4013010806 conn(0x7f79e40fc7d0 msgr2=0x7f79e40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:54.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.193+0000 7f79eb936640 1 -- 172.21.15.98:0/4013010806 shutdown_connections 2024-09-19T08:02:54.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:54.193+0000 7f79eb936640 1 -- 172.21.15.98:0/4013010806 wait complete. 2024-09-19T08:02:54.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:54 smithi098 ceph-mon[31333]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:55.022 INFO:teuthology.run_tasks:Running task sequential... 2024-09-19T08:02:55.032 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-09-19T08:02:55.033 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-19T08:02:55.033 INFO:teuthology.task.print:Enabling mgr modules 2024-09-19T08:02:55.033 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-09-19T08:02:55.034 INFO:teuthology.task.exec:Executing custom commands... 2024-09-19T08:02:55.034 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-09-19T08:02:55.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-09-19T08:02:55.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:54 smithi162 ceph-mon[35260]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:55.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.135+0000 7faebae27640 1 -- 172.21.15.98:0/2282321462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb4101fe0 msgr2=0x7faeb41023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:55.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.135+0000 7faebae27640 1 --2- 172.21.15.98:0/2282321462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb4101fe0 0x7faeb41023e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7faeb0009900 tx=0x7faeb002a410 comp rx=0 tx=0).stop 2024-09-19T08:02:55.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.135+0000 7faebae27640 1 -- 172.21.15.98:0/2282321462 shutdown_connections 2024-09-19T08:02:55.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.135+0000 7faebae27640 1 --2- 172.21.15.98:0/2282321462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeb41016d0 0x7faeb4108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:55.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.135+0000 7faebae27640 1 --2- 172.21.15.98:0/2282321462 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faeb4100d10 0x7faeb4101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:55.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.135+0000 7faebae27640 1 --2- 172.21.15.98:0/2282321462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb4101fe0 0x7faeb41023e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:55.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.135+0000 7faebae27640 1 -- 172.21.15.98:0/2282321462 >> 172.21.15.98:0/2282321462 conn(0x7faeb40fb930 msgr2=0x7faeb40fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:55.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.135+0000 7faebae27640 1 -- 172.21.15.98:0/2282321462 shutdown_connections 2024-09-19T08:02:55.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.135+0000 7faebae27640 1 -- 172.21.15.98:0/2282321462 wait complete. 2024-09-19T08:02:55.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faebae27640 1 Processor -- start 2024-09-19T08:02:55.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faebae27640 1 -- start start 2024-09-19T08:02:55.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faebae27640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faeb4100d10 0x7faeb4197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:55.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faebae27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeb41016d0 0x7faeb4198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faebae27640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb4101fe0 0x7faeb419f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faebae27640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faeb410d870 con 0x7faeb4101fe0 2024-09-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faebae27640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7faeb410d6f0 con 0x7faeb4100d10 2024-09-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faebae27640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faeb410d9f0 con 0x7faeb41016d0 2024-09-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faeb93c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb4101fe0 0x7faeb419f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faeb93c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb4101fe0 0x7faeb419f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49924/0 (socket says 172.21.15.98:49924) 2024-09-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faeb93c4640 1 -- 172.21.15.98:0/195627314 learned_addr learned my addr 172.21.15.98:0/195627314 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faeb8bc3640 1 --2- 172.21.15.98:0/195627314 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faeb4100d10 0x7faeb4197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faeb93c4640 1 -- 172.21.15.98:0/195627314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeb41016d0 msgr2=0x7faeb4198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faeb93c4640 1 --2- 172.21.15.98:0/195627314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeb41016d0 0x7faeb4198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faeb93c4640 1 -- 172.21.15.98:0/195627314 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faeb4100d10 msgr2=0x7faeb4197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faeb93c4640 1 --2- 172.21.15.98:0/195627314 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faeb4100d10 0x7faeb4197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faeb93c4640 1 -- 172.21.15.98:0/195627314 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb419fae0 con 0x7faeb4101fe0 2024-09-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.136+0000 7faeb8bc3640 1 --2- 172.21.15.98:0/195627314 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faeb4100d10 0x7faeb4197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.137+0000 7faeb93c4640 1 --2- 172.21.15.98:0/195627314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb4101fe0 0x7faeb419f470 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7faeac00cad0 tx=0x7faeac00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.137+0000 7faea9ffb640 1 -- 172.21.15.98:0/195627314 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac00ae20 con 0x7faeb4101fe0 2024-09-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.137+0000 7faea9ffb640 1 -- 172.21.15.98:0/195627314 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faeac005020 con 0x7faeb4101fe0 2024-09-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.137+0000 7faebae27640 1 -- 172.21.15.98:0/195627314 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb419fdd0 con 0x7faeb4101fe0 2024-09-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.137+0000 7faea9ffb640 1 -- 172.21.15.98:0/195627314 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac00f6f0 con 0x7faeb4101fe0 2024-09-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.137+0000 7faebae27640 1 -- 172.21.15.98:0/195627314 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb41a0370 con 0x7faeb4101fe0 2024-09-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.138+0000 7faebae27640 1 -- 172.21.15.98:0/195627314 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeb4107ed0 con 0x7faeb4101fe0 2024-09-19T08:02:55.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.139+0000 7faea9ffb640 1 -- 172.21.15.98:0/195627314 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7faeac0051d0 con 0x7faeb4101fe0 2024-09-19T08:02:55.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.139+0000 7faea9ffb640 1 --2- 172.21.15.98:0/195627314 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fae9c0a3390 0x7fae9c0a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:55.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.140+0000 7faeb8bc3640 1 --2- 172.21.15.98:0/195627314 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fae9c0a3390 0x7fae9c0a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:55.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.140+0000 7faea9ffb640 1 -- 172.21.15.98:0/195627314 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7faeac0cfe50 con 0x7faeb4101fe0 2024-09-19T08:02:55.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.140+0000 7faeb8bc3640 1 --2- 172.21.15.98:0/195627314 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fae9c0a3390 0x7fae9c0a5850 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7faeb0006010 tx=0x7faeb0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:55.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.144+0000 7faea9ffb640 1 -- 172.21.15.98:0/195627314 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7faeac014030 con 0x7faeb4101fe0 2024-09-19T08:02:55.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:54 smithi139 ceph-mon[34874]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:02:55.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.358+0000 7faebae27640 1 -- 172.21.15.98:0/195627314 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7faeb402d970 con 0x7faeb4101fe0 2024-09-19T08:02:55.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.731+0000 7faea9ffb640 1 -- 172.21.15.98:0/195627314 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v18) ==== 92+0+0 (secure 0 0 0) 0x7faeac09ca20 con 0x7faeb4101fe0 2024-09-19T08:02:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.734+0000 7faebae27640 1 -- 172.21.15.98:0/195627314 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fae9c0a3390 msgr2=0x7fae9c0a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.734+0000 7faebae27640 1 --2- 172.21.15.98:0/195627314 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fae9c0a3390 0x7fae9c0a5850 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7faeb0006010 tx=0x7faeb0000bc0 comp rx=0 tx=0).stop 2024-09-19T08:02:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.734+0000 7faebae27640 1 -- 172.21.15.98:0/195627314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb4101fe0 msgr2=0x7faeb419f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.734+0000 7faebae27640 1 --2- 172.21.15.98:0/195627314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb4101fe0 0x7faeb419f470 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7faeac00cad0 tx=0x7faeac00cf90 comp rx=0 tx=0).stop 2024-09-19T08:02:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.734+0000 7faebae27640 1 -- 172.21.15.98:0/195627314 shutdown_connections 2024-09-19T08:02:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.734+0000 7faebae27640 1 --2- 172.21.15.98:0/195627314 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fae9c0a3390 0x7fae9c0a5850 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.734+0000 7faebae27640 1 --2- 172.21.15.98:0/195627314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb4101fe0 0x7faeb419f470 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.734+0000 7faebae27640 1 --2- 172.21.15.98:0/195627314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeb41016d0 0x7faeb4198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:55.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.734+0000 7faebae27640 1 --2- 172.21.15.98:0/195627314 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faeb4100d10 0x7faeb4197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:55.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.734+0000 7faebae27640 1 -- 172.21.15.98:0/195627314 >> 172.21.15.98:0/195627314 conn(0x7faeb40fb930 msgr2=0x7faeb41050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:55.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.734+0000 7faebae27640 1 -- 172.21.15.98:0/195627314 shutdown_connections 2024-09-19T08:02:55.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:55.734+0000 7faebae27640 1 -- 172.21.15.98:0/195627314 wait complete. 2024-09-19T08:02:55.751 INFO:teuthology.run_tasks:Running task kclient... 2024-09-19T08:02:55.776 INFO:tasks.kclient:Mounting kernel clients... 2024-09-19T08:02:55.776 INFO:tasks.kclient:config is {'client.0': {}} 2024-09-19T08:02:55.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T08:02:55.776 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-09-19T08:02:55.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T08:02:55.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-09-19T08:02:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:Cannot find device "ceph-brx" 2024-09-19T08:02:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T08:02:55.862 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-09-19T08:02:55.862 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-19T08:02:55.862 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-19T08:02:55.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-19T08:02:56.083 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:55 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/195627314' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-09-19T08:02:56.083 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:55 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: ignoring --setuser ceph since I am not root 2024-09-19T08:02:56.084 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:55 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: ignoring --setgroup ceph since I am not root 2024-09-19T08:02:56.084 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:55 smithi139 ceph-mgr[36278]: -- 172.21.15.139:0/2777508505 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e8f01bf0e0 con 0x55e8f019d000 2024-09-19T08:02:56.084 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:55 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:55.880+0000 7f90ff1d3040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T08:02:56.084 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:55 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:55.936+0000 7f90ff1d3040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T08:02:56.084 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:56.083+0000 7f90ff1d3040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T08:02:56.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:55 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/195627314' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-09-19T08:02:56.096 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:55 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: ignoring --setuser ceph since I am not root 2024-09-19T08:02:56.096 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:55 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: ignoring --setgroup ceph since I am not root 2024-09-19T08:02:56.096 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:55 smithi162 ceph-mgr[37040]: -- 172.21.15.162:0/2107198102 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56522fcf90e0 con 0x56522fcd6800 2024-09-19T08:02:56.096 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:55 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:55.887+0000 7f1ffb607040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T08:02:56.097 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:55 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:55.945+0000 7f1ffb607040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T08:02:56.097 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:56 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:56.096+0000 7f1ffb607040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T08:02:56.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:55 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/195627314' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-09-19T08:02:56.101 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:55 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: ignoring --setuser ceph since I am not root 2024-09-19T08:02:56.101 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:55 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: ignoring --setgroup ceph since I am not root 2024-09-19T08:02:56.101 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:55 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:55.886+0000 7f868b76d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T08:02:56.101 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:55 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:55.946+0000 7f868b76d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T08:02:56.101 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:56 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:56.101+0000 7f868b76d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T08:02:56.157 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:02:56.370 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:56 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:56.162+0000 7f1ffb607040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T08:02:56.370 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:56 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:56.308+0000 7f1ffb607040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T08:02:56.423 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:56 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:56.177+0000 7f868b76d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T08:02:56.423 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:56 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:56.325+0000 7f868b76d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T08:02:56.427 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:56.148+0000 7f90ff1d3040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T08:02:56.427 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:56.292+0000 7f90ff1d3040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T08:02:56.682 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:56 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:56.469+0000 7f868b76d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T08:02:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.689+0000 7fd041dcd640 1 -- 172.21.15.98:0/1632751805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd03c102570 msgr2=0x7fd03c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.689+0000 7fd041dcd640 1 --2- 172.21.15.98:0/1632751805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd03c102570 0x7fd03c1029b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd030009900 tx=0x7fd03002a480 comp rx=0 tx=0).stop 2024-09-19T08:02:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.689+0000 7fd041dcd640 1 -- 172.21.15.98:0/1632751805 shutdown_connections 2024-09-19T08:02:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.689+0000 7fd041dcd640 1 --2- 172.21.15.98:0/1632751805 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd03c102ef0 0x7fd03c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.689+0000 7fd041dcd640 1 --2- 172.21.15.98:0/1632751805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd03c102570 0x7fd03c1029b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.689+0000 7fd041dcd640 1 --2- 172.21.15.98:0/1632751805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd03c100e20 0x7fd03c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.689+0000 7fd041dcd640 1 -- 172.21.15.98:0/1632751805 >> 172.21.15.98:0/1632751805 conn(0x7fd03c0fc7d0 msgr2=0x7fd03c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.689+0000 7fd041dcd640 1 -- 172.21.15.98:0/1632751805 shutdown_connections 2024-09-19T08:02:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.689+0000 7fd041dcd640 1 -- 172.21.15.98:0/1632751805 wait complete. 2024-09-19T08:02:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.689+0000 7fd041dcd640 1 Processor -- start 2024-09-19T08:02:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.689+0000 7fd041dcd640 1 -- start start 2024-09-19T08:02:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd041dcd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd03c100e20 0x7fd03c199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd041dcd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd03c102570 0x7fd03c199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd041dcd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd03c102ef0 0x7fd03c1a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd041dcd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c110f80 con 0x7fd03c100e20 2024-09-19T08:02:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd041dcd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c110e00 con 0x7fd03c102570 2024-09-19T08:02:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd041dcd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c111100 con 0x7fd03c102ef0 2024-09-19T08:02:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd03b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd03c100e20 0x7fd03c199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd03b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd03c100e20 0x7fd03c199350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54858/0 (socket says 172.21.15.98:54858) 2024-09-19T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd03b7fe640 1 -- 172.21.15.98:0/4135928398 learned_addr learned my addr 172.21.15.98:0/4135928398 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd03affd640 1 --2- 172.21.15.98:0/4135928398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd03c102570 0x7fd03c199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd03bfff640 1 --2- 172.21.15.98:0/4135928398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd03c102ef0 0x7fd03c1a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd03b7fe640 1 -- 172.21.15.98:0/4135928398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd03c102ef0 msgr2=0x7fd03c1a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd03b7fe640 1 --2- 172.21.15.98:0/4135928398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd03c102ef0 0x7fd03c1a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd03b7fe640 1 -- 172.21.15.98:0/4135928398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd03c102570 msgr2=0x7fd03c199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd03b7fe640 1 --2- 172.21.15.98:0/4135928398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd03c102570 0x7fd03c199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd03b7fe640 1 -- 172.21.15.98:0/4135928398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03c1a4910 con 0x7fd03c100e20 2024-09-19T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd03b7fe640 1 --2- 172.21.15.98:0/4135928398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd03c100e20 0x7fd03c199350 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd02c00e9f0 tx=0x7fd02c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd038ff9640 1 -- 172.21.15.98:0/4135928398 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02c00cd80 con 0x7fd03c100e20 2024-09-19T08:02:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd041dcd640 1 -- 172.21.15.98:0/4135928398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd03c1a4ae0 con 0x7fd03c100e20 2024-09-19T08:02:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd038ff9640 1 -- 172.21.15.98:0/4135928398 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd02c005500 con 0x7fd03c100e20 2024-09-19T08:02:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.690+0000 7fd038ff9640 1 -- 172.21.15.98:0/4135928398 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02c010690 con 0x7fd03c100e20 2024-09-19T08:02:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.691+0000 7fd041dcd640 1 -- 172.21.15.98:0/4135928398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd03c1a4fa0 con 0x7fd03c100e20 2024-09-19T08:02:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.691+0000 7fd041dcd640 1 -- 172.21.15.98:0/4135928398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd03c101200 con 0x7fd03c100e20 2024-09-19T08:02:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.693+0000 7fd038ff9640 1 -- 172.21.15.98:0/4135928398 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138372+0+0 (secure 0 0 0) 0x7fd02c0046c0 con 0x7fd03c100e20 2024-09-19T08:02:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.693+0000 7fd038ff9640 1 --2- 172.21.15.98:0/4135928398 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd0240a3290 0x7fd0240a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.693+0000 7fd03affd640 1 -- 172.21.15.98:0/4135928398 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd0240a3290 msgr2=0x7fd0240a5750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/470997812 2024-09-19T08:02:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.693+0000 7fd038ff9640 1 -- 172.21.15.98:0/4135928398 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fd02c0cfdf0 con 0x7fd03c100e20 2024-09-19T08:02:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.693+0000 7fd03affd640 1 --2- 172.21.15.98:0/4135928398 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd0240a3290 0x7fd0240a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T08:02:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.702+0000 7fd038ff9640 1 -- 172.21.15.98:0/4135928398 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fd02c09cb20 con 0x7fd03c100e20 2024-09-19T08:02:56.732 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:56 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:56.450+0000 7f1ffb607040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T08:02:56.733 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:56.431+0000 7f90ff1d3040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T08:02:56.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.894+0000 7fd03affd640 1 -- 172.21.15.98:0/4135928398 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd0240a3290 msgr2=0x7fd0240a5750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/470997812 2024-09-19T08:02:56.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.894+0000 7fd03affd640 1 --2- 172.21.15.98:0/4135928398 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd0240a3290 0x7fd0240a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-19T08:02:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.910+0000 7fd041dcd640 1 -- 172.21.15.98:0/4135928398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd03c109090 con 0x7fd03c100e20 2024-09-19T08:02:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.910+0000 7fd038ff9640 1 -- 172.21.15.98:0/4135928398 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+73 (secure 0 0 0) 0x7fd02c09c420 con 0x7fd03c100e20 2024-09-19T08:02:56.910 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-19T08:02:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.912+0000 7fd041dcd640 1 -- 172.21.15.98:0/4135928398 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd0240a3290 msgr2=0x7fd0240a5750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T08:02:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.912+0000 7fd041dcd640 1 --2- 172.21.15.98:0/4135928398 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd0240a3290 0x7fd0240a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.912+0000 7fd041dcd640 1 -- 172.21.15.98:0/4135928398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd03c100e20 msgr2=0x7fd03c199350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.912+0000 7fd041dcd640 1 --2- 172.21.15.98:0/4135928398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd03c100e20 0x7fd03c199350 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd02c00e9f0 tx=0x7fd02c00edf0 comp rx=0 tx=0).stop 2024-09-19T08:02:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.912+0000 7fd041dcd640 1 -- 172.21.15.98:0/4135928398 shutdown_connections 2024-09-19T08:02:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.912+0000 7fd041dcd640 1 --2- 172.21.15.98:0/4135928398 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd0240a3290 0x7fd0240a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.912+0000 7fd041dcd640 1 --2- 172.21.15.98:0/4135928398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd03c102ef0 0x7fd03c1a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.912+0000 7fd041dcd640 1 --2- 172.21.15.98:0/4135928398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd03c102570 0x7fd03c199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.912+0000 7fd041dcd640 1 --2- 172.21.15.98:0/4135928398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd03c100e20 0x7fd03c199350 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.912+0000 7fd041dcd640 1 -- 172.21.15.98:0/4135928398 >> 172.21.15.98:0/4135928398 conn(0x7fd03c0fc7d0 msgr2=0x7fd03c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.913+0000 7fd041dcd640 1 -- 172.21.15.98:0/4135928398 shutdown_connections 2024-09-19T08:02:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:56.913+0000 7fd041dcd640 1 -- 172.21.15.98:0/4135928398 wait complete. 2024-09-19T08:02:56.983 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:56 smithi139 ceph-mon[34874]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:56.983 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:56 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/195627314' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-19T08:02:56.983 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:56 smithi139 ceph-mon[34874]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-09-19T08:02:56.983 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:56.740+0000 7f90ff1d3040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T08:02:56.984 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:56.799+0000 7f90ff1d3040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T08:02:56.984 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:56.983+0000 7f90ff1d3040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T08:02:57.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:56 smithi162 ceph-mon[35260]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:57.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:56 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/195627314' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-19T08:02:57.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:56 smithi162 ceph-mon[35260]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-09-19T08:02:57.014 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:56 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:56.768+0000 7f1ffb607040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T08:02:57.015 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:56 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:56.830+0000 7f1ffb607040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T08:02:57.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:56 smithi098 ceph-mon[31333]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T08:02:57.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:56 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/195627314' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-19T08:02:57.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:56 smithi098 ceph-mon[31333]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-09-19T08:02:57.046 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:56 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:56.792+0000 7f868b76d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T08:02:57.046 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:56 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:56.853+0000 7f868b76d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T08:02:57.046 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:57 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:57.045+0000 7f868b76d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T08:02:57.370 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:57 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:57.014+0000 7f1ffb607040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T08:02:57.683 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-19T08:02:57.683 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-19T08:02:57.683 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-09-19T08:02:57.683 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-19T08:02:57.683 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-19T08:02:57.683 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-19T08:02:57.683 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-19T08:02:57.683 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-19T08:02:57.683 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-19T08:02:57.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T08:02:57.684 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-09-19T08:02:57.700 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-19T08:02:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-19T08:02:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-09-19T08:02:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-09-19T08:02:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-09-19T08:02:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-09-19T08:02:57.700 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-19T08:02:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-09-19T08:02:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-09-19T08:02:57.701 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-19T08:02:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-09-19T08:02:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-09-19T08:02:57.701 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-19T08:02:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-09-19T08:02:57.701 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-19T08:02:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-09-19T08:02:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-19T08:02:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42190sec preferred_lft 42190sec 2024-09-19T08:02:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link noprefixroute 2024-09-19T08:02:57.702 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-09-19T08:02:57.702 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-19T08:02:57.702 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T08:02:57.702 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-09-19T08:02:57.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2024-09-19T08:02:57.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2024-09-19T08:02:57.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2024-09-19T08:02:57.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-19T08:02:57.703 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-09-19T08:02:57.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T08:02:57.909 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:57 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/4135928398' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-19T08:02:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T08:02:57.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T08:02:57.920 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-19T08:02:57.945 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-09-19T08:02:57.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T08:02:57.948 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-09-19T08:02:58.017 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-09-19T08:02:58.032 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-19T08:02:58.032 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-19T08:02:58.032 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-19T08:02:58.033 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-19T08:02:58.033 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T08:02:58.033 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-09-19T08:02:58.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-19T08:02:58.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-19T08:02:58.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-19T08:02:58.034 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-09-19T08:02:58.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T08:02:58.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:57 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/4135928398' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-19T08:02:58.120 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:57 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:57.906+0000 7f1ffb607040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T08:02:58.120 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:57 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:57.974+0000 7f1ffb607040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T08:02:58.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:57 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/4135928398' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-19T08:02:58.177 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:57 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:57.858+0000 7f90ff1d3040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T08:02:58.177 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:57 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:57.926+0000 7f90ff1d3040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T08:02:58.181 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:57 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:57.952+0000 7f868b76d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T08:02:58.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:58 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:58.022+0000 7f868b76d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T08:02:58.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T08:02:58.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T08:02:58.329 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-09-19T08:02:58.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T08:02:58.344 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-09-19T08:02:58.399 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T08:02:58.400 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-09-19T08:02:58.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-19T08:02:58.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-19T08:02:58.400 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-09-19T08:02:58.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T08:02:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T08:02:58.519 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-19T08:02:58.519 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T08:02:58.520 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-09-19T08:02:58.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-19T08:02:58.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-19T08:02:58.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-19T08:02:58.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-19T08:02:58.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-19T08:02:58.520 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-09-19T08:02:58.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T08:02:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T08:02:58.654 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T08:02:58.654 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-09-19T08:02:58.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-09-19T08:02:58.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2024-09-19T08:02:58.654 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-09-19T08:02:58.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T08:02:58.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T08:02:58.732 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-19T08:02:58.732 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-19T08:02:58.761 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-19T08:02:58.761 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-19T08:02:58.761 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-19T08:02:58.816 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-09-19T08:02:58.816 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-09-19T08:02:58.816 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/a73a52f2-eaee-40cd-832f-432a14c0ec6e 2024-09-19T08:02:58.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T08:02:58.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/a73a52f2-eaee-40cd-832f-432a14c0ec6e /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-09-19T08:02:58.895 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:58 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:58.634+0000 7f90ff1d3040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-19T08:02:58.895 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:58 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:58.750+0000 7f90ff1d3040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T08:02:58.918 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-09-19T08:02:58.918 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-09-19T08:02:58.918 INFO:teuthology.orchestra.run.smithi098.stdout:invalid new device string format 2024-09-19T08:02:58.955 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:58 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:58.692+0000 7f1ffb607040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-19T08:02:58.955 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:58 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:58.810+0000 7f1ffb607040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T08:02:59.048 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:58 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:58.794+0000 7f868b76d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-19T08:02:59.048 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:58 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:58.916+0000 7f868b76d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T08:02:59.059 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-09-19T08:02:59.059 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-09-19T08:02:59.059 INFO:teuthology.orchestra.run.smithi098.stdout:invalid new device string format 2024-09-19T08:02:59.059 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.139:3300,172.21.15.162:3300" 2024-09-19T08:02:59.059 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.98:3300,172.21.15.139:3300,172.21.15.162:3300:/volumes/qa/sv_1/a73a52f2-eaee-40cd-832f-432a14c0ec6e 2024-09-19T08:02:59.059 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync,key=0,fsid=1a02525a-765c-11ef-bceb-c7b262605968" will pass to kernel 2024-09-19T08:02:59.061 INFO:teuthology.orchestra.run.smithi098.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-19T08:02:59.061 INFO:teuthology.orchestra.run.smithi098.stdout: You just mounted a file system that supports labels which does not 2024-09-19T08:02:59.061 INFO:teuthology.orchestra.run.smithi098.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-19T08:02:59.061 INFO:teuthology.orchestra.run.smithi098.stdout: applications will generate AVC messages and not be allowed access to 2024-09-19T08:02:59.061 INFO:teuthology.orchestra.run.smithi098.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-19T08:02:59.063 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-19T08:02:59.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T08:02:59.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-19T08:02:59.090 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-19T08:02:59.177 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:58 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:58.894+0000 7f90ff1d3040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T08:02:59.177 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:58 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:58.957+0000 7f90ff1d3040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T08:02:59.177 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:59.109+0000 7f90ff1d3040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-19T08:02:59.177 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:59.177+0000 7f90ff1d3040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T08:02:59.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-19T08:02:59.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T08:02:59.188 INFO:teuthology.orchestra.run.smithi098.stdout:client24766 2024-09-19T08:02:59.188 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T08:02:59.189 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-19T08:02:59.206 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-19T08:02:59.239 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:58 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:58.955+0000 7f1ffb607040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T08:02:59.239 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:59.015+0000 7f1ffb607040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T08:02:59.239 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:59.168+0000 7f1ffb607040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-19T08:02:59.239 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:59.237+0000 7f1ffb607040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T08:02:59.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-19T08:02:59.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T08:02:59.302 INFO:teuthology.orchestra.run.smithi098.stdout:client24766 2024-09-19T08:02:59.302 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-19T08:02:59.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-19T08:02:59.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T08:02:59.359 INFO:teuthology.orchestra.run.smithi098.stdout:client24766 2024-09-19T08:02:59.359 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/1a02525a-765c-11ef-bceb-c7b262605968.client24766/status') 2024-09-19T08:02:59.365 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:59 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:59.067+0000 7f868b76d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T08:02:59.366 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:59 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:59.132+0000 7f868b76d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T08:02:59.366 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:59 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:59.291+0000 7f868b76d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-19T08:02:59.366 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:59 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:59.365+0000 7f868b76d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T08:02:59.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T08:02:59.418 INFO:teuthology.orchestra.run.smithi098.stdout:instance: client.24766 (3)172.21.15.98:0/428376014 2024-09-19T08:02:59.418 INFO:teuthology.orchestra.run.smithi098.stdout:blocklisted: false 2024-09-19T08:02:59.418 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records in 2024-09-19T08:02:59.418 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records out 2024-09-19T08:02:59.418 INFO:teuthology.orchestra.run.smithi098.stderr:70 bytes copied, 2.3008e-05 s, 3.0 MB/s 2024-09-19T08:02:59.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T08:02:59.425 INFO:teuthology.run_tasks:Running task exec... 2024-09-19T08:02:59.435 INFO:teuthology.task.exec:Executing custom commands... 2024-09-19T08:02:59.435 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-09-19T08:02:59.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-09-19T08:02:59.458 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:59.237+0000 7f90ff1d3040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T08:02:59.458 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:59.394+0000 7f90ff1d3040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T08:02:59.459 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:59.458+0000 7f90ff1d3040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T08:02:59.523 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:59.298+0000 7f1ffb607040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T08:02:59.523 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:59.456+0000 7f1ffb607040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T08:02:59.523 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:59.522+0000 7f1ffb607040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T08:02:59.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 -- 172.21.15.98:0/189252010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5fc100cd0 msgr2=0x7fd5fc101150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 --2- 172.21.15.98:0/189252010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5fc100cd0 0x7fd5fc101150 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009900 tx=0x7fd5ec02a410 comp rx=0 tx=0).stop 2024-09-19T08:02:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 -- 172.21.15.98:0/189252010 shutdown_connections 2024-09-19T08:02:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 --2- 172.21.15.98:0/189252010 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5fc101690 0x7fd5fc108f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 --2- 172.21.15.98:0/189252010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5fc100cd0 0x7fd5fc101150 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 --2- 172.21.15.98:0/189252010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5fc102030 0x7fd5fc102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 -- 172.21.15.98:0/189252010 >> 172.21.15.98:0/189252010 conn(0x7fd5fc0fb970 msgr2=0x7fd5fc0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 -- 172.21.15.98:0/189252010 shutdown_connections 2024-09-19T08:02:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 -- 172.21.15.98:0/189252010 wait complete. 2024-09-19T08:02:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 Processor -- start 2024-09-19T08:02:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 -- start start 2024-09-19T08:02:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5fc100cd0 0x7fd5fc197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5fc101690 0x7fd5fc198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:59.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5fc102030 0x7fd5fc19f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:59.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc10d810 con 0x7fd5fc101690 2024-09-19T08:02:59.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc10d690 con 0x7fd5fc102030 2024-09-19T08:02:59.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.567+0000 7fd603c53640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc10d990 con 0x7fd5fc100cd0 2024-09-19T08:02:59.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd6011ee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5fc101690 0x7fd5fc198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:59.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd6011ee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5fc101690 0x7fd5fc198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54872/0 (socket says 172.21.15.98:54872) 2024-09-19T08:02:59.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd6011ee640 1 -- 172.21.15.98:0/411586185 learned_addr learned my addr 172.21.15.98:0/411586185 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:59.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd6019ef640 1 --2- 172.21.15.98:0/411586185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5fc100cd0 0x7fd5fc197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:59.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd6021f0640 1 --2- 172.21.15.98:0/411586185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5fc102030 0x7fd5fc19f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:59.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd6011ee640 1 -- 172.21.15.98:0/411586185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5fc100cd0 msgr2=0x7fd5fc197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:59.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd6011ee640 1 --2- 172.21.15.98:0/411586185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5fc100cd0 0x7fd5fc197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd6011ee640 1 -- 172.21.15.98:0/411586185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5fc102030 msgr2=0x7fd5fc19f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:59.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd6011ee640 1 --2- 172.21.15.98:0/411586185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5fc102030 0x7fd5fc19f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd6011ee640 1 -- 172.21.15.98:0/411586185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5fc19f9d0 con 0x7fd5fc101690 2024-09-19T08:02:59.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd6011ee640 1 --2- 172.21.15.98:0/411586185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5fc101690 0x7fd5fc198300 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009230 tx=0x7fd5ec0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:59.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd5f2ffd640 1 -- 172.21.15.98:0/411586185 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec03f030 con 0x7fd5fc101690 2024-09-19T08:02:59.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd5f2ffd640 1 -- 172.21.15.98:0/411586185 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5ec003ce0 con 0x7fd5fc101690 2024-09-19T08:02:59.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd5f2ffd640 1 -- 172.21.15.98:0/411586185 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec035070 con 0x7fd5fc101690 2024-09-19T08:02:59.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd603c53640 1 -- 172.21.15.98:0/411586185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5fc19fc60 con 0x7fd5fc101690 2024-09-19T08:02:59.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.568+0000 7fd603c53640 1 -- 172.21.15.98:0/411586185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5fc1a0150 con 0x7fd5fc101690 2024-09-19T08:02:59.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.570+0000 7fd603c53640 1 -- 172.21.15.98:0/411586185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5fc107e10 con 0x7fd5fc101690 2024-09-19T08:02:59.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.571+0000 7fd5f2ffd640 1 -- 172.21.15.98:0/411586185 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138372+0+0 (secure 0 0 0) 0x7fd5ec042030 con 0x7fd5fc101690 2024-09-19T08:02:59.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.571+0000 7fd5f2ffd640 1 --2- 172.21.15.98:0/411586185 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd5e80a3340 0x7fd5e80a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:59.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.571+0000 7fd6019ef640 1 -- 172.21.15.98:0/411586185 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd5e80a3340 msgr2=0x7fd5e80a5800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/470997812 2024-09-19T08:02:59.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.571+0000 7fd5f2ffd640 1 -- 172.21.15.98:0/411586185 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7fd5ec0efe60 con 0x7fd5fc101690 2024-09-19T08:02:59.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.571+0000 7fd6019ef640 1 --2- 172.21.15.98:0/411586185 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd5e80a3340 0x7fd5e80a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T08:02:59.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.575+0000 7fd5f2ffd640 1 -- 172.21.15.98:0/411586185 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fd5ec0bc540 con 0x7fd5fc101690 2024-09-19T08:02:59.679 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:59 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:59.428+0000 7f868b76d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T08:02:59.679 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:59 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:59.610+0000 7f868b76d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T08:02:59.679 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:59 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:59.678+0000 7f868b76d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T08:02:59.724 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:59.524+0000 7f90ff1d3040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T08:02:59.724 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:59.607+0000 7f90ff1d3040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T08:02:59.725 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:59.665+0000 7f90ff1d3040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T08:02:59.725 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:02:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:02:59.724+0000 7f90ff1d3040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T08:02:59.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.772+0000 7fd6019ef640 1 -- 172.21.15.98:0/411586185 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd5e80a3340 msgr2=0x7fd5e80a5800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/470997812 2024-09-19T08:02:59.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.772+0000 7fd6019ef640 1 --2- 172.21.15.98:0/411586185 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd5e80a3340 0x7fd5e80a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-19T08:02:59.797 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:59.590+0000 7f1ffb607040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T08:02:59.797 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:59.673+0000 7f1ffb607040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T08:02:59.797 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:59.734+0000 7f1ffb607040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T08:02:59.797 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:02:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:02:59.795+0000 7f1ffb607040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T08:02:59.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.803+0000 7fd603c53640 1 -- 172.21.15.98:0/411586185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fd5fc062df0 con 0x7fd5fc101690 2024-09-19T08:02:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.855+0000 7fd5f2ffd640 1 -- 172.21.15.98:0/411586185 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v28) ==== 118+0+0 (secure 0 0 0) 0x7fd5ec0c0840 con 0x7fd5fc101690 2024-09-19T08:02:59.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.857+0000 7fd603c53640 1 -- 172.21.15.98:0/411586185 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd5e80a3340 msgr2=0x7fd5e80a5800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T08:02:59.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.857+0000 7fd603c53640 1 --2- 172.21.15.98:0/411586185 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd5e80a3340 0x7fd5e80a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.857+0000 7fd603c53640 1 -- 172.21.15.98:0/411586185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5fc101690 msgr2=0x7fd5fc198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.857+0000 7fd603c53640 1 --2- 172.21.15.98:0/411586185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5fc101690 0x7fd5fc198300 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009230 tx=0x7fd5ec0042f0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.857+0000 7fd603c53640 1 -- 172.21.15.98:0/411586185 shutdown_connections 2024-09-19T08:02:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.857+0000 7fd603c53640 1 --2- 172.21.15.98:0/411586185 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7fd5e80a3340 0x7fd5e80a5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.857+0000 7fd603c53640 1 --2- 172.21.15.98:0/411586185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5fc102030 0x7fd5fc19f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.857+0000 7fd603c53640 1 --2- 172.21.15.98:0/411586185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5fc101690 0x7fd5fc198300 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.857+0000 7fd603c53640 1 --2- 172.21.15.98:0/411586185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5fc100cd0 0x7fd5fc197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.857+0000 7fd603c53640 1 -- 172.21.15.98:0/411586185 >> 172.21.15.98:0/411586185 conn(0x7fd5fc0fb970 msgr2=0x7fd5fc104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.857+0000 7fd603c53640 1 -- 172.21.15.98:0/411586185 shutdown_connections 2024-09-19T08:02:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.857+0000 7fd603c53640 1 -- 172.21.15.98:0/411586185 wait complete. 2024-09-19T08:02:59.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-09-19T08:02:59.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:02:59 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/411586185' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-19T08:02:59.932 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:59 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:59.746+0000 7f868b76d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T08:02:59.932 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:59 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:59.831+0000 7f868b76d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T08:02:59.932 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:59 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:59.897+0000 7f868b76d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T08:02:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.973+0000 7f55b8d9f640 1 -- 172.21.15.98:0/948871641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b4101c30 msgr2=0x7f55b41020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.973+0000 7f55b8d9f640 1 --2- 172.21.15.98:0/948871641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b4101c30 0x7f55b41020b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f55a8009900 tx=0x7f55a802a410 comp rx=0 tx=0).stop 2024-09-19T08:02:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.973+0000 7f55b8d9f640 1 -- 172.21.15.98:0/948871641 shutdown_connections 2024-09-19T08:02:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.973+0000 7f55b8d9f640 1 --2- 172.21.15.98:0/948871641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55b41025f0 0x7f55b4108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.973+0000 7f55b8d9f640 1 --2- 172.21.15.98:0/948871641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b4101c30 0x7f55b41020b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.973+0000 7f55b8d9f640 1 --2- 172.21.15.98:0/948871641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55b4100a40 0x7f55b4100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.973+0000 7f55b8d9f640 1 -- 172.21.15.98:0/948871641 >> 172.21.15.98:0/948871641 conn(0x7f55b40fc1e0 msgr2=0x7f55b40fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:02:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.973+0000 7f55b8d9f640 1 -- 172.21.15.98:0/948871641 shutdown_connections 2024-09-19T08:02:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.973+0000 7f55b8d9f640 1 -- 172.21.15.98:0/948871641 wait complete. 2024-09-19T08:02:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.974+0000 7f55b8d9f640 1 Processor -- start 2024-09-19T08:02:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.974+0000 7f55b8d9f640 1 -- start start 2024-09-19T08:02:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.974+0000 7f55b8d9f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55b4100a40 0x7f55b419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.974+0000 7f55b8d9f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b4101c30 0x7f55b419a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.974+0000 7f55b8d9f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55b41025f0 0x7f55b41a1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.974+0000 7f55b8d9f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55b410c6c0 con 0x7f55b4101c30 2024-09-19T08:02:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.974+0000 7f55b8d9f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f55b410c540 con 0x7f55b41025f0 2024-09-19T08:02:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.974+0000 7f55b8d9f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f55b410c840 con 0x7f55b4100a40 2024-09-19T08:02:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.974+0000 7f55b1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b4101c30 0x7f55b419a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.974+0000 7f55b259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55b4100a40 0x7f55b419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.974+0000 7f55b259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55b4100a40 0x7f55b419a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:45552/0 (socket says 172.21.15.98:45552) 2024-09-19T08:02:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.974+0000 7f55b1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b4101c30 0x7f55b419a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54896/0 (socket says 172.21.15.98:54896) 2024-09-19T08:02:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.974+0000 7f55b2d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55b41025f0 0x7f55b41a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:02:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.974+0000 7f55b259c640 1 -- 172.21.15.98:0/2617969404 learned_addr learned my addr 172.21.15.98:0/2617969404 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:02:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.975+0000 7f55b259c640 1 -- 172.21.15.98:0/2617969404 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55b41025f0 msgr2=0x7f55b41a1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.975+0000 7f55b259c640 1 --2- 172.21.15.98:0/2617969404 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55b41025f0 0x7f55b41a1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.975+0000 7f55b259c640 1 -- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b4101c30 msgr2=0x7f55b419a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:02:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.975+0000 7f55b259c640 1 --2- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b4101c30 0x7f55b419a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:02:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.975+0000 7f55b259c640 1 -- 172.21.15.98:0/2617969404 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b41a1bf0 con 0x7f55b4100a40 2024-09-19T08:02:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.975+0000 7f55b259c640 1 --2- 172.21.15.98:0/2617969404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55b4100a40 0x7f55b419a040 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55a400d950 tx=0x7f55a400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:02:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.975+0000 7f55a37fe640 1 -- 172.21.15.98:0/2617969404 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a4017020 con 0x7f55b4100a40 2024-09-19T08:02:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.975+0000 7f55b8d9f640 1 -- 172.21.15.98:0/2617969404 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55b41a1ee0 con 0x7f55b4100a40 2024-09-19T08:02:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.975+0000 7f55a37fe640 1 -- 172.21.15.98:0/2617969404 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55a4009c30 con 0x7f55b4100a40 2024-09-19T08:02:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.975+0000 7f55a37fe640 1 -- 172.21.15.98:0/2617969404 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a400f6b0 con 0x7f55b4100a40 2024-09-19T08:02:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.975+0000 7f55b8d9f640 1 -- 172.21.15.98:0/2617969404 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55b41a2350 con 0x7f55b4100a40 2024-09-19T08:02:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.976+0000 7f55b8d9f640 1 -- 172.21.15.98:0/2617969404 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55b41a25e0 con 0x7f55b4100a40 2024-09-19T08:02:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.978+0000 7f55a37fe640 1 -- 172.21.15.98:0/2617969404 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 138372+0+0 (secure 0 0 0) 0x7f55a401a020 con 0x7f55b4100a40 2024-09-19T08:02:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.978+0000 7f55a37fe640 1 --2- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f559c0a3340 0x7f559c0a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:02:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.978+0000 7f55b1d9b640 1 -- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f559c0a3340 msgr2=0x7f559c0a5800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/470997812 2024-09-19T08:02:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.978+0000 7f55a37fe640 1 -- 172.21.15.98:0/2617969404 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8219+0+0 (secure 0 0 0) 0x7f55a40cfca0 con 0x7f55b4100a40 2024-09-19T08:02:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.978+0000 7f55b1d9b640 1 --2- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f559c0a3340 0x7f559c0a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T08:02:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:02:59.986+0000 7f55a37fe640 1 -- 172.21.15.98:0/2617969404 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f55a4021020 con 0x7f55b4100a40 2024-09-19T08:03:00.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:02:59 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/411586185' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-19T08:03:00.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:02:59 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/411586185' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-19T08:03:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:00.178+0000 7f55b8d9f640 1 -- 172.21.15.98:0/2617969404 --> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f55b40fe3a0 con 0x7f559c0a3340 2024-09-19T08:03:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:00.179+0000 7f55b1d9b640 1 -- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f559c0a3340 msgr2=0x7f559c0a5800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/470997812 2024-09-19T08:03:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:00.179+0000 7f55b1d9b640 1 --2- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f559c0a3340 0x7f559c0a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-19T08:03:00.431 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:02:59 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:02:59.960+0000 7f868b76d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T08:03:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:00.580+0000 7f55b1d9b640 1 -- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f559c0a3340 msgr2=0x7f559c0a5800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/470997812 2024-09-19T08:03:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:00.580+0000 7f55b1d9b640 1 --2- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f559c0a3340 0x7f559c0a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-19T08:03:00.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:00.763+0000 7f55a37fe640 1 -- 172.21.15.98:0/2617969404 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mgrmap(e 19) ==== 138040+0+0 (secure 0 0 0) 0x7f55a40988e0 con 0x7f55b4100a40 2024-09-19T08:03:00.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:00.764+0000 7f55a37fe640 1 -- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f559c0a3340 msgr2=0x7f559c0a5800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T08:03:00.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:00.764+0000 7f55a37fe640 1 --2- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:6800/470997812,v1:172.21.15.98:6801/470997812] conn(0x7f559c0a3340 0x7f559c0a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:00.865 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y[36274]: 2024-09-19T08:03:00.518+0000 7f90ff1d3040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T08:03:00.866 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z[37002]: 2024-09-19T08:03:00.605+0000 7f1ffb607040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T08:03:01.030 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:00.750+0000 7f868b76d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T08:03:01.030 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:00.988+0000 7f85f5884640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:01.030 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:00.988+0000 7f85f5884640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:01.030 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:00.988+0000 7f85f5884640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:01.030 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:00.988+0000 7f85f5884640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:01.030 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:00.988+0000 7f85f5884640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:01.030 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:00.990+0000 7f85f303f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:01.030 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:00.990+0000 7f85f303f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:01.031 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:00.990+0000 7f85f303f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:01.031 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:00.990+0000 7f85f303f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:01.031 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:00.990+0000 7f85f303f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:01.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/411586185' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-19T08:03:01.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:03:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: Standby manager daemon y restarted 2024-09-19T08:03:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: Standby manager daemon y started 2024-09-19T08:03:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.139:0/2762390102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-19T08:03:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.139:0/2762390102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T08:03:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.139:0/2762390102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-19T08:03:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.139:0/2762390102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T08:03:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: Standby manager daemon z restarted 2024-09-19T08:03:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: Standby manager daemon z started 2024-09-19T08:03:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.162:0/2747193904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-19T08:03:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.162:0/2747193904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T08:03:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.162:0/2747193904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-19T08:03:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.? 172.21.15.162:0/2747193904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T08:03:01.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: Active manager daemon x restarted 2024-09-19T08:03:01.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: Activating manager daemon x 2024-09-19T08:03:01.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: osdmap e81: 12 total, 12 up, 12 in 2024-09-19T08:03:01.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: mgrmap e19: x(active, starting, since 0.0129799s), standbys: z, y 2024-09-19T08:03:01.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T08:03:01.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T08:03:01.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T08:03:01.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-19T08:03:01.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-19T08:03:01.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-19T08:03:01.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-19T08:03:01.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-19T08:03:01.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-19T08:03:01.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-19T08:03:01.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-19T08:03:01.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-19T08:03:01.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-19T08:03:01.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-19T08:03:01.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-19T08:03:01.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-19T08:03:01.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-19T08:03:01.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-19T08:03:01.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T08:03:01.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T08:03:01.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T08:03:01.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T08:03:01.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T08:03:01.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T08:03:01.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T08:03:01.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T08:03:01.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T08:03:01.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T08:03:01.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T08:03:01.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:03:01.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-19T08:03:01.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-19T08:03:01.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-19T08:03:01.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: Manager daemon x is now available 2024-09-19T08:03:01.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:01.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-19T08:03:01.038 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:00 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-19T08:03:01.117 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/411586185' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-19T08:03:01.118 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:03:01.118 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: Standby manager daemon y restarted 2024-09-19T08:03:01.118 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: Standby manager daemon y started 2024-09-19T08:03:01.118 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.139:0/2762390102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-19T08:03:01.118 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.139:0/2762390102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T08:03:01.118 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.139:0/2762390102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-19T08:03:01.118 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.139:0/2762390102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T08:03:01.118 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: Standby manager daemon z restarted 2024-09-19T08:03:01.118 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: Standby manager daemon z started 2024-09-19T08:03:01.118 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.162:0/2747193904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-19T08:03:01.118 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.162:0/2747193904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T08:03:01.119 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.162:0/2747193904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-19T08:03:01.119 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.? 172.21.15.162:0/2747193904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T08:03:01.119 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: Active manager daemon x restarted 2024-09-19T08:03:01.119 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: Activating manager daemon x 2024-09-19T08:03:01.119 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: osdmap e81: 12 total, 12 up, 12 in 2024-09-19T08:03:01.119 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: mgrmap e19: x(active, starting, since 0.0129799s), standbys: z, y 2024-09-19T08:03:01.119 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T08:03:01.119 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T08:03:01.119 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T08:03:01.119 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-19T08:03:01.119 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-19T08:03:01.119 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-19T08:03:01.120 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-19T08:03:01.120 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-19T08:03:01.120 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-19T08:03:01.120 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-19T08:03:01.120 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-19T08:03:01.120 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-19T08:03:01.120 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-19T08:03:01.120 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-19T08:03:01.120 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-19T08:03:01.120 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-19T08:03:01.120 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-19T08:03:01.121 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-19T08:03:01.121 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T08:03:01.121 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T08:03:01.121 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T08:03:01.121 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T08:03:01.121 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T08:03:01.121 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T08:03:01.121 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T08:03:01.121 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T08:03:01.121 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T08:03:01.121 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T08:03:01.121 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T08:03:01.122 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:03:01.122 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-19T08:03:01.122 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-19T08:03:01.122 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-19T08:03:01.122 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: Manager daemon x is now available 2024-09-19T08:03:01.122 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:01.122 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-19T08:03:01.122 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:00 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-19T08:03:01.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/411586185' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-19T08:03:01.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:03:01.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: Standby manager daemon y restarted 2024-09-19T08:03:01.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: Standby manager daemon y started 2024-09-19T08:03:01.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.139:0/2762390102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-19T08:03:01.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.139:0/2762390102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T08:03:01.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.139:0/2762390102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-19T08:03:01.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.139:0/2762390102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T08:03:01.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: Standby manager daemon z restarted 2024-09-19T08:03:01.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: Standby manager daemon z started 2024-09-19T08:03:01.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.162:0/2747193904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-19T08:03:01.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.162:0/2747193904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T08:03:01.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.162:0/2747193904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-19T08:03:01.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.? 172.21.15.162:0/2747193904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T08:03:01.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: Active manager daemon x restarted 2024-09-19T08:03:01.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: Activating manager daemon x 2024-09-19T08:03:01.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: osdmap e81: 12 total, 12 up, 12 in 2024-09-19T08:03:01.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: mgrmap e19: x(active, starting, since 0.0129799s), standbys: z, y 2024-09-19T08:03:01.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T08:03:01.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T08:03:01.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T08:03:01.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-19T08:03:01.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-19T08:03:01.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-19T08:03:01.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-19T08:03:01.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-19T08:03:01.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-19T08:03:01.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-19T08:03:01.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-19T08:03:01.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-19T08:03:01.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-19T08:03:01.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-19T08:03:01.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-19T08:03:01.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-19T08:03:01.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-19T08:03:01.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-19T08:03:01.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T08:03:01.126 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T08:03:01.126 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T08:03:01.126 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T08:03:01.126 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T08:03:01.126 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T08:03:01.126 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T08:03:01.126 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T08:03:01.126 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T08:03:01.126 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T08:03:01.126 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T08:03:01.126 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T08:03:01.126 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-19T08:03:01.126 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-19T08:03:01.127 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-19T08:03:01.127 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: Manager daemon x is now available 2024-09-19T08:03:01.127 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:01.127 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-19T08:03:01.127 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:00 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-19T08:03:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.767+0000 7f55a37fe640 1 -- 172.21.15.98:0/2617969404 <== mon.1 v2:172.21.15.162:3300/0 8 ==== mgrmap(e 20) ==== 138167+0+0 (secure 0 0 0) 0x7f55a40cd7d0 con 0x7f55b4100a40 2024-09-19T08:03:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.768+0000 7f55a37fe640 1 --2- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f559c051dd0 0x7f559c0521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.768+0000 7f55a37fe640 1 -- 172.21.15.98:0/2617969404 --> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f559c034b90 con 0x7f559c051dd0 2024-09-19T08:03:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.770+0000 7f55b1d9b640 1 --2- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f559c051dd0 0x7f559c0521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.770+0000 7f55b1d9b640 1 --2- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f559c051dd0 0x7f559c0521b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f55a80314e0 tx=0x7f55a8000bc0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.790+0000 7f55a37fe640 1 -- 172.21.15.98:0/2617969404 <== mgr.15252 v2:172.21.15.98:6800/1961446727 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f559c034b90 con 0x7f559c051dd0 2024-09-19T08:03:01.791 INFO:teuthology.orchestra.run.smithi098.stdout:{} 2024-09-19T08:03:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.792+0000 7f55b8d9f640 1 -- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f559c051dd0 msgr2=0x7f559c0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.792+0000 7f55b8d9f640 1 --2- 172.21.15.98:0/2617969404 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f559c051dd0 0x7f559c0521b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f55a80314e0 tx=0x7f55a8000bc0 comp rx=0 tx=0).stop 2024-09-19T08:03:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.792+0000 7f55b8d9f640 1 -- 172.21.15.98:0/2617969404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55b4100a40 msgr2=0x7f55b419a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.792+0000 7f55b8d9f640 1 --2- 172.21.15.98:0/2617969404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55b4100a40 0x7f55b419a040 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55a400d950 tx=0x7f55a400de10 comp rx=0 tx=0).stop 2024-09-19T08:03:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.792+0000 7f55b2d9d640 1 -- 172.21.15.98:0/2617969404 reap_dead start 2024-09-19T08:03:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.792+0000 7f55b8d9f640 1 -- 172.21.15.98:0/2617969404 shutdown_connections 2024-09-19T08:03:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.792+0000 7f55b8d9f640 1 -- 172.21.15.98:0/2617969404 >> 172.21.15.98:0/2617969404 conn(0x7f55b40fc1e0 msgr2=0x7f55b40fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.792+0000 7f55b8d9f640 1 -- 172.21.15.98:0/2617969404 shutdown_connections 2024-09-19T08:03:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.792+0000 7f55b8d9f640 1 -- 172.21.15.98:0/2617969404 wait complete. 2024-09-19T08:03:01.804 INFO:teuthology.run_tasks:Running task check-counter... 2024-09-19T08:03:01.815 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-09-19T08:03:01.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T08:03:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.958+0000 7fd35710c640 1 -- 172.21.15.98:0/1566554373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd350102d20 msgr2=0x7fd3501031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.958+0000 7fd35710c640 1 --2- 172.21.15.98:0/1566554373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd350102d20 0x7fd3501031a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd33c009900 tx=0x7fd33c02a410 comp rx=0 tx=0).stop 2024-09-19T08:03:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.958+0000 7fd35710c640 1 -- 172.21.15.98:0/1566554373 shutdown_connections 2024-09-19T08:03:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.958+0000 7fd35710c640 1 --2- 172.21.15.98:0/1566554373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3501036e0 0x7fd350109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.958+0000 7fd35710c640 1 --2- 172.21.15.98:0/1566554373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd350102d20 0x7fd3501031a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.958+0000 7fd35710c640 1 --2- 172.21.15.98:0/1566554373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd350101b30 0x7fd350101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.958+0000 7fd35710c640 1 -- 172.21.15.98:0/1566554373 >> 172.21.15.98:0/1566554373 conn(0x7fd3500fd290 msgr2=0x7fd3500ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.959+0000 7fd35710c640 1 -- 172.21.15.98:0/1566554373 shutdown_connections 2024-09-19T08:03:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.959+0000 7fd35710c640 1 -- 172.21.15.98:0/1566554373 wait complete. 2024-09-19T08:03:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.959+0000 7fd35710c640 1 Processor -- start 2024-09-19T08:03:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.959+0000 7fd35710c640 1 -- start start 2024-09-19T08:03:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.959+0000 7fd35710c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd350101b30 0x7fd350075fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.959+0000 7fd35710c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd350102d20 0x7fd3500745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.959+0000 7fd35710c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3501036e0 0x7fd350074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.959+0000 7fd35710c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd35010d790 con 0x7fd3501036e0 2024-09-19T08:03:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.959+0000 7fd35710c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd35010d610 con 0x7fd350102d20 2024-09-19T08:03:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.959+0000 7fd35710c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd35010d910 con 0x7fd350101b30 2024-09-19T08:03:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.959+0000 7fd3556a9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3501036e0 0x7fd350074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.959+0000 7fd3556a9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3501036e0 0x7fd350074b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55032/0 (socket says 172.21.15.98:55032) 2024-09-19T08:03:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.959+0000 7fd3556a9640 1 -- 172.21.15.98:0/2653200192 learned_addr learned my addr 172.21.15.98:0/2653200192 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.959+0000 7fd354ea8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd350101b30 0x7fd350075fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.960+0000 7fd34ffff640 1 --2- 172.21.15.98:0/2653200192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd350102d20 0x7fd3500745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.960+0000 7fd3556a9640 1 -- 172.21.15.98:0/2653200192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd350101b30 msgr2=0x7fd350075fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.960+0000 7fd3556a9640 1 --2- 172.21.15.98:0/2653200192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd350101b30 0x7fd350075fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.960+0000 7fd3556a9640 1 -- 172.21.15.98:0/2653200192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd350102d20 msgr2=0x7fd3500745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.960+0000 7fd3556a9640 1 --2- 172.21.15.98:0/2653200192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd350102d20 0x7fd3500745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.960+0000 7fd3556a9640 1 -- 172.21.15.98:0/2653200192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3500753b0 con 0x7fd3501036e0 2024-09-19T08:03:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.960+0000 7fd3556a9640 1 --2- 172.21.15.98:0/2653200192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3501036e0 0x7fd350074b20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd34800eaa0 tx=0x7fd34800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.960+0000 7fd34dffb640 1 -- 172.21.15.98:0/2653200192 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34800cdb0 con 0x7fd3501036e0 2024-09-19T08:03:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.960+0000 7fd34dffb640 1 -- 172.21.15.98:0/2653200192 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd348005020 con 0x7fd3501036e0 2024-09-19T08:03:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.960+0000 7fd35710c640 1 -- 172.21.15.98:0/2653200192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3501a1ed0 con 0x7fd3501036e0 2024-09-19T08:03:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.960+0000 7fd34dffb640 1 -- 172.21.15.98:0/2653200192 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3480116b0 con 0x7fd3501036e0 2024-09-19T08:03:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.960+0000 7fd35710c640 1 -- 172.21.15.98:0/2653200192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3501a22e0 con 0x7fd3501036e0 2024-09-19T08:03:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.962+0000 7fd35710c640 1 -- 172.21.15.98:0/2653200192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd350108ed0 con 0x7fd3501036e0 2024-09-19T08:03:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.962+0000 7fd34dffb640 1 -- 172.21.15.98:0/2653200192 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 138167+0+0 (secure 0 0 0) 0x7fd348005560 con 0x7fd3501036e0 2024-09-19T08:03:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.963+0000 7fd34dffb640 1 --2- 172.21.15.98:0/2653200192 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd3400a3050 0x7fd3400a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.964+0000 7fd354ea8640 1 --2- 172.21.15.98:0/2653200192 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd3400a3050 0x7fd3400a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.964+0000 7fd34dffb640 1 -- 172.21.15.98:0/2653200192 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fd3480cf7c0 con 0x7fd3501036e0 2024-09-19T08:03:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.964+0000 7fd354ea8640 1 --2- 172.21.15.98:0/2653200192 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd3400a3050 0x7fd3400a5510 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd350102b80 tx=0x7fd344005f90 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:01.968+0000 7fd34dffb640 1 -- 172.21.15.98:0/2653200192 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fd348016030 con 0x7fd3501036e0 2024-09-19T08:03:02.094 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:01.777+0000 7f85f0039640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:02.095 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:01.777+0000 7f85f0039640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:02.095 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:01.777+0000 7f85f0039640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:02.095 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:01.777+0000 7f85f0039640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:02.095 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:01.777+0000 7f85f0039640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:03:02.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.141+0000 7fd35710c640 1 -- 172.21.15.98:0/2653200192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd350062e90 con 0x7fd3501036e0 2024-09-19T08:03:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.143+0000 7fd34dffb640 1 -- 172.21.15.98:0/2653200192 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+23128 (secure 0 0 0) 0x7fd34809c090 con 0x7fd3501036e0 2024-09-19T08:03:02.145 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:02.145 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","created":"2024-09-19T07:53:39.127398+0000","modified":"2024-09-19T08:03:00.753313+0000","last_up_change":"2024-09-19T08:00:22.467777+0000","last_in_change":"2024-09-19T07:59:58.538492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:56:43.773892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T08:01:07.693796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T08:01:09.518028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d01ce6-165d-493d-ae98-d8ad9e5d4134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6803","nonce":1719360533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6805","nonce":1719360533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6809","nonce":1719360533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6807","nonce":1719360533}]},"public_addr":"172.21.15.98:6803/1719360533","cluster_addr":"172.21.15.98:6805/1719360533","heartbeat_back_addr":"172.21.15.98:6809/1719360533","heartbeat_front_addr":"172.21.15.98:6807/1719360533","state":["exists","up"]},{"osd":1,"uuid":"34724bd5-0738-4473-b087-c09031c1ee2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036962626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036962626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036962626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036962626}]},"public_addr":"172.21.15.139:6801/4036962626","cluster_addr":"172.21.15.139:6803/4036962626","heartbeat_back_addr":"172.21.15.139:6807/4036962626","heartbeat_front_addr":"172.21.15.139:6805/4036962626","state":["exists","up"]},{"osd":2,"uuid":"16985ebe-0e32-42ee-bded-b7838e290afc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6801","nonce":2811412891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6803","nonce":2811412891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6807","nonce":2811412891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6805","nonce":2811412891}]},"public_addr":"172.21.15.162:6801/2811412891","cluster_addr":"172.21.15.162:6803/2811412891","heartbeat_back_addr":"172.21.15.162:6807/2811412891","heartbeat_front_addr":"172.21.15.162:6805/2811412891","state":["exists","up"]},{"osd":3,"uuid":"b140e466-3119-49cb-aec2-ba6ca4d10d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6811","nonce":1383392014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6813","nonce":1383392014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6817","nonce":1383392014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6815","nonce":1383392014}]},"public_addr":"172.21.15.98:6811/1383392014","cluster_addr":"172.21.15.98:6813/1383392014","heartbeat_back_addr":"172.21.15.98:6817/1383392014","heartbeat_front_addr":"172.21.15.98:6815/1383392014","state":["exists","up"]},{"osd":4,"uuid":"b4412813-df57-4ac6-8fb7-f57b69c74768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6809","nonce":116453520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6811","nonce":116453520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6815","nonce":116453520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6813","nonce":116453520}]},"public_addr":"172.21.15.139:6809/116453520","cluster_addr":"172.21.15.139:6811/116453520","heartbeat_back_addr":"172.21.15.139:6815/116453520","heartbeat_front_addr":"172.21.15.139:6813/116453520","state":["exists","up"]},{"osd":5,"uuid":"d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6809","nonce":3769530297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6811","nonce":3769530297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6815","nonce":3769530297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6813","nonce":3769530297}]},"public_addr":"172.21.15.162:6809/3769530297","cluster_addr":"172.21.15.162:6811/3769530297","heartbeat_back_addr":"172.21.15.162:6815/3769530297","heartbeat_front_addr":"172.21.15.162:6813/3769530297","state":["exists","up"]},{"osd":6,"uuid":"ba1c1590-6aea-40dc-90c8-e21b88896909","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6819","nonce":692373267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6821","nonce":692373267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6825","nonce":692373267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6823","nonce":692373267}]},"public_addr":"172.21.15.98:6819/692373267","cluster_addr":"172.21.15.98:6821/692373267","heartbeat_back_addr":"172.21.15.98:6825/692373267","heartbeat_front_addr":"172.21.15.98:6823/692373267","state":["exists","up"]},{"osd":7,"uuid":"eb9093ad-a068-4a63-ab87-0b50522cbc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6817","nonce":1963811650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6819","nonce":1963811650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6823","nonce":1963811650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6821","nonce":1963811650}]},"public_addr":"172.21.15.139:6817/1963811650","cluster_addr":"172.21.15.139:6819/1963811650","heartbeat_back_addr":"172.21.15.139:6823/1963811650","heartbeat_front_addr":"172.21.15.139:6821/1963811650","state":["exists","up"]},{"osd":8,"uuid":"31ca330a-fc45-466d-81c2-ef4386b3cc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6817","nonce":3607966458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6819","nonce":3607966458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6823","nonce":3607966458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6821","nonce":3607966458}]},"public_addr":"172.21.15.162:6817/3607966458","cluster_addr":"172.21.15.162:6819/3607966458","heartbeat_back_addr":"172.21.15.162:6823/3607966458","heartbeat_front_addr":"172.21.15.162:6821/3607966458","state":["exists","up"]},{"osd":9,"uuid":"9c4f216f-aef1-4367-9c58-b913b0c4f7db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6827","nonce":2050391244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6829","nonce":2050391244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6833","nonce":2050391244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6831","nonce":2050391244}]},"public_addr":"172.21.15.98:6827/2050391244","cluster_addr":"172.21.15.98:6829/2050391244","heartbeat_back_addr":"172.21.15.98:6833/2050391244","heartbeat_front_addr":"172.21.15.98:6831/2050391244","state":["exists","up"]},{"osd":10,"uuid":"03cc7df6-84ae-4d9a-a3d7-74c91e1744e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6825","nonce":2336761639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6827","nonce":2336761639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6831","nonce":2336761639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6829","nonce":2336761639}]},"public_addr":"172.21.15.139:6825/2336761639","cluster_addr":"172.21.15.139:6827/2336761639","heartbeat_back_addr":"172.21.15.139:6831/2336761639","heartbeat_front_addr":"172.21.15.139:6829/2336761639","state":["exists","up"]},{"osd":11,"uuid":"29e95498-4952-46d4-8fa6-6d8e2b862111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6825","nonce":1231011554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6827","nonce":1231011554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6831","nonce":1231011554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6829","nonce":1231011554}]},"public_addr":"172.21.15.162:6825/1231011554","cluster_addr":"172.21.15.162:6827/1231011554","heartbeat_back_addr":"172.21.15.162:6831/1231011554","heartbeat_front_addr":"172.21.15.162:6829/1231011554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:55:49.995244+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:15.655543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:40.331165+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:04.688881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:29.174966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:53.663161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:17.407551+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:42.584296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:06.317219+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:31.078625+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:55.689269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T08:00:19.659748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1},{"from":7,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1252058007":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/921776066":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/336479555":"2024-09-20T08:03:00.753287+0000","172.21.15.98:6800/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6800/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/1497156658":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6801/470997812":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/1238193420":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6800/470997812":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/3838590070":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3842627255":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3866031018":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3194864054":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3067331617":"2024-09-20T08:03:00.753287+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T08:03:02.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.145+0000 7fd35710c640 1 -- 172.21.15.98:0/2653200192 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd3400a3050 msgr2=0x7fd3400a5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.145+0000 7fd35710c640 1 --2- 172.21.15.98:0/2653200192 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd3400a3050 0x7fd3400a5510 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd350102b80 tx=0x7fd344005f90 comp rx=0 tx=0).stop 2024-09-19T08:03:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.145+0000 7fd35710c640 1 -- 172.21.15.98:0/2653200192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3501036e0 msgr2=0x7fd350074b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.145+0000 7fd35710c640 1 --2- 172.21.15.98:0/2653200192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3501036e0 0x7fd350074b20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd34800eaa0 tx=0x7fd34800ef60 comp rx=0 tx=0).stop 2024-09-19T08:03:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.145+0000 7fd35710c640 1 -- 172.21.15.98:0/2653200192 shutdown_connections 2024-09-19T08:03:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.145+0000 7fd35710c640 1 --2- 172.21.15.98:0/2653200192 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd3400a3050 0x7fd3400a5510 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.145+0000 7fd35710c640 1 --2- 172.21.15.98:0/2653200192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3501036e0 0x7fd350074b20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.145+0000 7fd35710c640 1 --2- 172.21.15.98:0/2653200192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd350102d20 0x7fd3500745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.145+0000 7fd35710c640 1 --2- 172.21.15.98:0/2653200192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd350101b30 0x7fd350075fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.145+0000 7fd35710c640 1 -- 172.21.15.98:0/2653200192 >> 172.21.15.98:0/2653200192 conn(0x7fd3500fd290 msgr2=0x7fd3501060a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.145+0000 7fd35710c640 1 -- 172.21.15.98:0/2653200192 shutdown_connections 2024-09-19T08:03:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.145+0000 7fd35710c640 1 -- 172.21.15.98:0/2653200192 wait complete. 2024-09-19T08:03:02.157 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-19T07:56:43.773892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-19T08:01:07.693796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-19T08:01:09.518028+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-19T08:03:02.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-19T08:03:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.261+0000 7f94da41e640 1 -- 172.21.15.98:0/214552646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d41025e0 msgr2=0x7f94d41049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.261+0000 7f94da41e640 1 --2- 172.21.15.98:0/214552646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d41025e0 0x7f94d41049d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f94c4009900 tx=0x7f94c402a410 comp rx=0 tx=0).stop 2024-09-19T08:03:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.261+0000 7f94da41e640 1 -- 172.21.15.98:0/214552646 shutdown_connections 2024-09-19T08:03:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.261+0000 7f94da41e640 1 --2- 172.21.15.98:0/214552646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d4107840 0x7f94d4109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.261+0000 7f94da41e640 1 --2- 172.21.15.98:0/214552646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94d4104f10 0x7f94d4107300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.261+0000 7f94da41e640 1 --2- 172.21.15.98:0/214552646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d41025e0 0x7f94d41049d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.261+0000 7f94da41e640 1 -- 172.21.15.98:0/214552646 >> 172.21.15.98:0/214552646 conn(0x7f94d40fc140 msgr2=0x7f94d40fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.262+0000 7f94da41e640 1 -- 172.21.15.98:0/214552646 shutdown_connections 2024-09-19T08:03:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.262+0000 7f94da41e640 1 -- 172.21.15.98:0/214552646 wait complete. 2024-09-19T08:03:02.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.262+0000 7f94da41e640 1 Processor -- start 2024-09-19T08:03:02.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.262+0000 7f94da41e640 1 -- start start 2024-09-19T08:03:02.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.262+0000 7f94da41e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d41025e0 0x7f94d4197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:02.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.262+0000 7f94da41e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94d4104f10 0x7f94d4198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:02.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.262+0000 7f94da41e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d4107840 0x7f94d419f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:02.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.262+0000 7f94da41e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94d410d840 con 0x7f94d4107840 2024-09-19T08:03:02.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.262+0000 7f94da41e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f94d410d6c0 con 0x7f94d41025e0 2024-09-19T08:03:02.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.262+0000 7f94da41e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94d410d9c0 con 0x7f94d4104f10 2024-09-19T08:03:02.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d3fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d41025e0 0x7f94d4197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:02.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d89bb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d4107840 0x7f94d419f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:02.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d3fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d41025e0 0x7f94d4197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:52302/0 (socket says 172.21.15.98:52302) 2024-09-19T08:03:02.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d3fff640 1 -- 172.21.15.98:0/2775166662 learned_addr learned my addr 172.21.15.98:0/2775166662 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:02.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d37fe640 1 --2- 172.21.15.98:0/2775166662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94d4104f10 0x7f94d4198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:02.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d89bb640 1 -- 172.21.15.98:0/2775166662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94d4104f10 msgr2=0x7f94d4198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d89bb640 1 --2- 172.21.15.98:0/2775166662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94d4104f10 0x7f94d4198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d89bb640 1 -- 172.21.15.98:0/2775166662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d41025e0 msgr2=0x7f94d4197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d89bb640 1 --2- 172.21.15.98:0/2775166662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d41025e0 0x7f94d4197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d89bb640 1 -- 172.21.15.98:0/2775166662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d419faf0 con 0x7f94d4107840 2024-09-19T08:03:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d3fff640 1 --2- 172.21.15.98:0/2775166662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d41025e0 0x7f94d4197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d89bb640 1 --2- 172.21.15.98:0/2775166662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d4107840 0x7f94d419f480 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f94cc005fd0 tx=0x7f94cc00edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d17fa640 1 -- 172.21.15.98:0/2775166662 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94cc004d20 con 0x7f94d4107840 2024-09-19T08:03:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d17fa640 1 -- 172.21.15.98:0/2775166662 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94cc005020 con 0x7f94d4107840 2024-09-19T08:03:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94da41e640 1 -- 172.21.15.98:0/2775166662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d419fde0 con 0x7f94d4107840 2024-09-19T08:03:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94d17fa640 1 -- 172.21.15.98:0/2775166662 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94cc0116d0 con 0x7f94d4107840 2024-09-19T08:03:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.263+0000 7f94da41e640 1 -- 172.21.15.98:0/2775166662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94d41a0250 con 0x7f94d4107840 2024-09-19T08:03:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.264+0000 7f94da41e640 1 -- 172.21.15.98:0/2775166662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94d40fdc30 con 0x7f94d4107840 2024-09-19T08:03:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.265+0000 7f94d17fa640 1 -- 172.21.15.98:0/2775166662 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 138167+0+0 (secure 0 0 0) 0x7f94cc005540 con 0x7f94d4107840 2024-09-19T08:03:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.266+0000 7f94d17fa640 1 --2- 172.21.15.98:0/2775166662 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94bc0a3050 0x7f94bc0a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.266+0000 7f94d3fff640 1 --2- 172.21.15.98:0/2775166662 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94bc0a3050 0x7f94bc0a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:02.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.266+0000 7f94d17fa640 1 -- 172.21.15.98:0/2775166662 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f94cc0cf690 con 0x7f94d4107840 2024-09-19T08:03:02.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.266+0000 7f94d3fff640 1 --2- 172.21.15.98:0/2775166662 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94bc0a3050 0x7f94bc0a5510 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f94c40096c0 tx=0x7f94c40315b0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:02.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.270+0000 7f94d17fa640 1 -- 172.21.15.98:0/2775166662 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f94cc016030 con 0x7f94d4107840 2024-09-19T08:03:02.432 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:02.119+0000 7f85d4f84640 -1 log_channel(cephadm) log [ERR] : [19/Sep/2024:08:03:02] ENGINE Error in HTTPServer.serve 2024-09-19T08:03:02.433 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: Traceback (most recent call last): 2024-09-19T08:03:02.433 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-19T08:03:02.433 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: self._connections.run(self.expiration_interval) 2024-09-19T08:03:02.433 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-19T08:03:02.433 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: self._run(expiration_interval) 2024-09-19T08:03:02.433 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-19T08:03:02.433 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: new_conn = self._from_server_socket(self.server.socket) 2024-09-19T08:03:02.433 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-19T08:03:02.433 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-19T08:03:02.433 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-19T08:03:02.433 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: s = self.context.wrap_socket( 2024-09-19T08:03:02.433 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-19T08:03:02.433 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: return self.sslsocket_class._create( 2024-09-19T08:03:02.434 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-19T08:03:02.434 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: self.do_handshake() 2024-09-19T08:03:02.434 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-19T08:03:02.434 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: self._sslobj.do_handshake() 2024-09-19T08:03:02.434 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-19T08:03:02.434 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:03:02.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.462+0000 7f94da41e640 1 -- 172.21.15.98:0/2775166662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f94d4062df0 con 0x7f94d4107840 2024-09-19T08:03:02.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.462+0000 7f94d17fa640 1 -- 172.21.15.98:0/2775166662 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f94cc09bf60 con 0x7f94d4107840 2024-09-19T08:03:02.464 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-09-19T08:03:02.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.465+0000 7f94da41e640 1 -- 172.21.15.98:0/2775166662 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94bc0a3050 msgr2=0x7f94bc0a5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.465+0000 7f94da41e640 1 --2- 172.21.15.98:0/2775166662 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94bc0a3050 0x7f94bc0a5510 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f94c40096c0 tx=0x7f94c40315b0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.465+0000 7f94da41e640 1 -- 172.21.15.98:0/2775166662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d4107840 msgr2=0x7f94d419f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.465+0000 7f94da41e640 1 --2- 172.21.15.98:0/2775166662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d4107840 0x7f94d419f480 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f94cc005fd0 tx=0x7f94cc00edd0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.465+0000 7f94da41e640 1 -- 172.21.15.98:0/2775166662 shutdown_connections 2024-09-19T08:03:02.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.465+0000 7f94da41e640 1 --2- 172.21.15.98:0/2775166662 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94bc0a3050 0x7f94bc0a5510 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.465+0000 7f94da41e640 1 --2- 172.21.15.98:0/2775166662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d4107840 0x7f94d419f480 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.465+0000 7f94da41e640 1 --2- 172.21.15.98:0/2775166662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94d4104f10 0x7f94d4198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.465+0000 7f94da41e640 1 --2- 172.21.15.98:0/2775166662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d41025e0 0x7f94d4197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.465+0000 7f94da41e640 1 -- 172.21.15.98:0/2775166662 >> 172.21.15.98:0/2775166662 conn(0x7f94d40fc140 msgr2=0x7f94d4102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:02.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.465+0000 7f94da41e640 1 -- 172.21.15.98:0/2775166662 shutdown_connections 2024-09-19T08:03:02.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.465+0000 7f94da41e640 1 -- 172.21.15.98:0/2775166662 wait complete. 2024-09-19T08:03:02.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-19T08:03:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.589+0000 7fcc1bc42640 1 -- 172.21.15.98:0/1900269861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc14102580 msgr2=0x7fcc14104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.589+0000 7fcc1bc42640 1 --2- 172.21.15.98:0/1900269861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc14102580 0x7fcc14104970 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009900 tx=0x7fcc1002a410 comp rx=0 tx=0).stop 2024-09-19T08:03:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 -- 172.21.15.98:0/1900269861 shutdown_connections 2024-09-19T08:03:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 --2- 172.21.15.98:0/1900269861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc141077e0 0x7fcc14109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 --2- 172.21.15.98:0/1900269861 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc14104eb0 0x7fcc141072a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 --2- 172.21.15.98:0/1900269861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc14102580 0x7fcc14104970 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 -- 172.21.15.98:0/1900269861 >> 172.21.15.98:0/1900269861 conn(0x7fcc140fc140 msgr2=0x7fcc140fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 -- 172.21.15.98:0/1900269861 shutdown_connections 2024-09-19T08:03:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 -- 172.21.15.98:0/1900269861 wait complete. 2024-09-19T08:03:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 Processor -- start 2024-09-19T08:03:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 -- start start 2024-09-19T08:03:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc14102580 0x7fcc1419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc14104eb0 0x7fcc1419a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc141077e0 0x7fcc141a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14101020 con 0x7fcc14102580 2024-09-19T08:03:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14100ea0 con 0x7fcc141077e0 2024-09-19T08:03:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.590+0000 7fcc1bc42640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141011a0 con 0x7fcc14104eb0 2024-09-19T08:03:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc199de640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc14102580 0x7fcc1419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc199de640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc14102580 0x7fcc1419a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55066/0 (socket says 172.21.15.98:55066) 2024-09-19T08:03:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc199de640 1 -- 172.21.15.98:0/2195507374 learned_addr learned my addr 172.21.15.98:0/2195507374 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc191dd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc14104eb0 0x7fcc1419a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc1a1df640 1 --2- 172.21.15.98:0/2195507374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc141077e0 0x7fcc141a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc199de640 1 -- 172.21.15.98:0/2195507374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc14104eb0 msgr2=0x7fcc1419a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc199de640 1 --2- 172.21.15.98:0/2195507374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc14104eb0 0x7fcc1419a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc199de640 1 -- 172.21.15.98:0/2195507374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc141077e0 msgr2=0x7fcc141a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc199de640 1 --2- 172.21.15.98:0/2195507374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc141077e0 0x7fcc141a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc199de640 1 -- 172.21.15.98:0/2195507374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc141a1cf0 con 0x7fcc14102580 2024-09-19T08:03:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc199de640 1 --2- 172.21.15.98:0/2195507374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc14102580 0x7fcc1419a0d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009230 tx=0x7fcc10004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:02.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc0affd640 1 -- 172.21.15.98:0/2195507374 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1003f030 con 0x7fcc14102580 2024-09-19T08:03:02.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc1bc42640 1 -- 172.21.15.98:0/2195507374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc141a1f80 con 0x7fcc14102580 2024-09-19T08:03:02.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc0affd640 1 -- 172.21.15.98:0/2195507374 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc10003ce0 con 0x7fcc14102580 2024-09-19T08:03:02.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc0affd640 1 -- 172.21.15.98:0/2195507374 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10035070 con 0x7fcc14102580 2024-09-19T08:03:02.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.591+0000 7fcc1bc42640 1 -- 172.21.15.98:0/2195507374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc141a2390 con 0x7fcc14102580 2024-09-19T08:03:02.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.593+0000 7fcc1bc42640 1 -- 172.21.15.98:0/2195507374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc141a25c0 con 0x7fcc14102580 2024-09-19T08:03:02.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.594+0000 7fcc0affd640 1 -- 172.21.15.98:0/2195507374 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 138167+0+0 (secure 0 0 0) 0x7fcc10042030 con 0x7fcc14102580 2024-09-19T08:03:02.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.594+0000 7fcc0affd640 1 --2- 172.21.15.98:0/2195507374 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcc000a3050 0x7fcc000a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:02.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.594+0000 7fcc191dd640 1 --2- 172.21.15.98:0/2195507374 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcc000a3050 0x7fcc000a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:02.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.594+0000 7fcc0affd640 1 -- 172.21.15.98:0/2195507374 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fcc100efbd0 con 0x7fcc14102580 2024-09-19T08:03:02.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.595+0000 7fcc191dd640 1 --2- 172.21.15.98:0/2195507374 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcc000a3050 0x7fcc000a5510 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcc1419b5e0 tx=0x7fcc0400b410 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:02.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.599+0000 7fcc0affd640 1 -- 172.21.15.98:0/2195507374 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fcc100bc460 con 0x7fcc14102580 2024-09-19T08:03:02.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.802+0000 7fcc1bc42640 1 -- 172.21.15.98:0/2195507374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcc141a2ea0 con 0x7fcc14102580 2024-09-19T08:03:02.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.802+0000 7fcc0affd640 1 -- 172.21.15.98:0/2195507374 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) ==== 104+0+11 (secure 0 0 0) 0x7fcc100c0760 con 0x7fcc14102580 2024-09-19T08:03:02.803 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-09-19T08:03:02.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.804+0000 7fcc1bc42640 1 -- 172.21.15.98:0/2195507374 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcc000a3050 msgr2=0x7fcc000a5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.804+0000 7fcc1bc42640 1 --2- 172.21.15.98:0/2195507374 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcc000a3050 0x7fcc000a5510 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcc1419b5e0 tx=0x7fcc0400b410 comp rx=0 tx=0).stop 2024-09-19T08:03:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.804+0000 7fcc1bc42640 1 -- 172.21.15.98:0/2195507374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc14102580 msgr2=0x7fcc1419a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.804+0000 7fcc1bc42640 1 --2- 172.21.15.98:0/2195507374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc14102580 0x7fcc1419a0d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009230 tx=0x7fcc10004360 comp rx=0 tx=0).stop 2024-09-19T08:03:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.805+0000 7fcc1bc42640 1 -- 172.21.15.98:0/2195507374 shutdown_connections 2024-09-19T08:03:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.805+0000 7fcc1bc42640 1 --2- 172.21.15.98:0/2195507374 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcc000a3050 0x7fcc000a5510 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.805+0000 7fcc1bc42640 1 --2- 172.21.15.98:0/2195507374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc141077e0 0x7fcc141a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.805+0000 7fcc1bc42640 1 --2- 172.21.15.98:0/2195507374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc14104eb0 0x7fcc1419a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.805+0000 7fcc1bc42640 1 --2- 172.21.15.98:0/2195507374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc14102580 0x7fcc1419a0d0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.805+0000 7fcc1bc42640 1 -- 172.21.15.98:0/2195507374 >> 172.21.15.98:0/2195507374 conn(0x7fcc140fc140 msgr2=0x7fcc141070d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.805+0000 7fcc1bc42640 1 -- 172.21.15.98:0/2195507374 shutdown_connections 2024-09-19T08:03:02.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.805+0000 7fcc1bc42640 1 -- 172.21.15.98:0/2195507374 wait complete. 2024-09-19T08:03:02.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-19T08:03:02.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.922+0000 7f5fea50a640 1 -- 172.21.15.98:0/1142655905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fe4102580 msgr2=0x7f5fe4104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.922+0000 7f5fea50a640 1 --2- 172.21.15.98:0/1142655905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fe4102580 0x7f5fe4104970 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009900 tx=0x7f5fd402a410 comp rx=0 tx=0).stop 2024-09-19T08:03:02.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.922+0000 7f5fea50a640 1 -- 172.21.15.98:0/1142655905 shutdown_connections 2024-09-19T08:03:02.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.922+0000 7f5fea50a640 1 --2- 172.21.15.98:0/1142655905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fe41077e0 0x7f5fe4109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.922+0000 7f5fea50a640 1 --2- 172.21.15.98:0/1142655905 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe4104eb0 0x7f5fe41072a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.922+0000 7f5fea50a640 1 --2- 172.21.15.98:0/1142655905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fe4102580 0x7f5fe4104970 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.922+0000 7f5fea50a640 1 -- 172.21.15.98:0/1142655905 >> 172.21.15.98:0/1142655905 conn(0x7f5fe40fc140 msgr2=0x7f5fe40fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:02.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.922+0000 7f5fea50a640 1 -- 172.21.15.98:0/1142655905 shutdown_connections 2024-09-19T08:03:02.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.922+0000 7f5fea50a640 1 -- 172.21.15.98:0/1142655905 wait complete. 2024-09-19T08:03:02.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.923+0000 7f5fea50a640 1 Processor -- start 2024-09-19T08:03:02.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.923+0000 7f5fea50a640 1 -- start start 2024-09-19T08:03:02.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.923+0000 7f5fea50a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fe4102580 0x7f5fe4197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:02.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.923+0000 7f5fea50a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe4104eb0 0x7f5fe4198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.923+0000 7f5fea50a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fe41077e0 0x7f5fe419f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.923+0000 7f5fea50a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe410d7e0 con 0x7f5fe4102580 2024-09-19T08:03:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.923+0000 7f5fea50a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe410d660 con 0x7f5fe41077e0 2024-09-19T08:03:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.923+0000 7f5fea50a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe410d960 con 0x7f5fe4104eb0 2024-09-19T08:03:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.923+0000 7f5fe3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fe4102580 0x7f5fe4197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.923+0000 7f5fe3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fe4102580 0x7f5fe4197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55112/0 (socket says 172.21.15.98:55112) 2024-09-19T08:03:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.923+0000 7f5fe3fff640 1 -- 172.21.15.98:0/1598525919 learned_addr learned my addr 172.21.15.98:0/1598525919 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.923+0000 7f5fe37fe640 1 --2- 172.21.15.98:0/1598525919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe4104eb0 0x7f5fe4198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.923+0000 7f5fe8aa7640 1 --2- 172.21.15.98:0/1598525919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fe41077e0 0x7f5fe419f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.924+0000 7f5fe3fff640 1 -- 172.21.15.98:0/1598525919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe4104eb0 msgr2=0x7f5fe4198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.924+0000 7f5fe3fff640 1 --2- 172.21.15.98:0/1598525919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe4104eb0 0x7f5fe4198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.924+0000 7f5fe3fff640 1 -- 172.21.15.98:0/1598525919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fe41077e0 msgr2=0x7f5fe419f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.924+0000 7f5fe3fff640 1 --2- 172.21.15.98:0/1598525919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fe41077e0 0x7f5fe419f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:02.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.924+0000 7f5fe3fff640 1 -- 172.21.15.98:0/1598525919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe419fb10 con 0x7f5fe4102580 2024-09-19T08:03:02.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.924+0000 7f5fe3fff640 1 --2- 172.21.15.98:0/1598525919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fe4102580 0x7f5fe4197ef0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009230 tx=0x7f5fd40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:02.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.924+0000 7f5fe17fa640 1 -- 172.21.15.98:0/1598525919 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd403f030 con 0x7f5fe4102580 2024-09-19T08:03:02.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.924+0000 7f5fe17fa640 1 -- 172.21.15.98:0/1598525919 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fd4003ce0 con 0x7f5fe4102580 2024-09-19T08:03:02.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.924+0000 7f5fe17fa640 1 -- 172.21.15.98:0/1598525919 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4035070 con 0x7f5fe4102580 2024-09-19T08:03:02.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.924+0000 7f5fea50a640 1 -- 172.21.15.98:0/1598525919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fe419fda0 con 0x7f5fe4102580 2024-09-19T08:03:02.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.924+0000 7f5fea50a640 1 -- 172.21.15.98:0/1598525919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fe41a0220 con 0x7f5fe4102580 2024-09-19T08:03:02.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.925+0000 7f5fea50a640 1 -- 172.21.15.98:0/1598525919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fe40fdc30 con 0x7f5fe4102580 2024-09-19T08:03:02.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.927+0000 7f5fe17fa640 1 -- 172.21.15.98:0/1598525919 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 138167+0+0 (secure 0 0 0) 0x7f5fd4042030 con 0x7f5fe4102580 2024-09-19T08:03:02.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.928+0000 7f5fe17fa640 1 --2- 172.21.15.98:0/1598525919 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5fcc0a30d0 0x7f5fcc0a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:02.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.928+0000 7f5fe37fe640 1 --2- 172.21.15.98:0/1598525919 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5fcc0a30d0 0x7f5fcc0a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:02.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.928+0000 7f5fe17fa640 1 -- 172.21.15.98:0/1598525919 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5fd40efe20 con 0x7f5fe4102580 2024-09-19T08:03:02.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.928+0000 7f5fe37fe640 1 --2- 172.21.15.98:0/1598525919 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5fcc0a30d0 0x7f5fcc0a5590 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4199400 tx=0x7f5fd8005f90 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:02.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:02.931+0000 7f5fe17fa640 1 -- 172.21.15.98:0/1598525919 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f5fd40bc6f0 con 0x7f5fe4102580 2024-09-19T08:03:03.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.107+0000 7f5fea50a640 1 -- 172.21.15.98:0/1598525919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5fe4062e90 con 0x7f5fe4102580 2024-09-19T08:03:03.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.107+0000 7f5fe17fa640 1 -- 172.21.15.98:0/1598525919 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) ==== 100+0+11 (secure 0 0 0) 0x7f5fd40c09f0 con 0x7f5fe4102580 2024-09-19T08:03:03.108 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-09-19T08:03:03.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.109+0000 7f5fea50a640 1 -- 172.21.15.98:0/1598525919 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5fcc0a30d0 msgr2=0x7f5fcc0a5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:03.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.109+0000 7f5fea50a640 1 --2- 172.21.15.98:0/1598525919 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5fcc0a30d0 0x7f5fcc0a5590 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4199400 tx=0x7f5fd8005f90 comp rx=0 tx=0).stop 2024-09-19T08:03:03.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.109+0000 7f5fea50a640 1 -- 172.21.15.98:0/1598525919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fe4102580 msgr2=0x7f5fe4197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:03.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.109+0000 7f5fea50a640 1 --2- 172.21.15.98:0/1598525919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fe4102580 0x7f5fe4197ef0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009230 tx=0x7f5fd40042f0 comp rx=0 tx=0).stop 2024-09-19T08:03:03.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.109+0000 7f5fea50a640 1 -- 172.21.15.98:0/1598525919 shutdown_connections 2024-09-19T08:03:03.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.109+0000 7f5fea50a640 1 --2- 172.21.15.98:0/1598525919 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5fcc0a30d0 0x7f5fcc0a5590 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:03.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.109+0000 7f5fea50a640 1 --2- 172.21.15.98:0/1598525919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fe41077e0 0x7f5fe419f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:03.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.109+0000 7f5fea50a640 1 --2- 172.21.15.98:0/1598525919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe4104eb0 0x7f5fe4198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:03.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.109+0000 7f5fea50a640 1 --2- 172.21.15.98:0/1598525919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fe4102580 0x7f5fe4197ef0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:03.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.109+0000 7f5fea50a640 1 -- 172.21.15.98:0/1598525919 >> 172.21.15.98:0/1598525919 conn(0x7f5fe40fc140 msgr2=0x7f5fe4102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:03.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.110+0000 7f5fea50a640 1 -- 172.21.15.98:0/1598525919 shutdown_connections 2024-09-19T08:03:03.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.110+0000 7f5fea50a640 1 -- 172.21.15.98:0/1598525919 wait complete. 2024-09-19T08:03:03.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:02 smithi162 ceph-mon[35260]: mgrmap e20: x(active, since 1.01884s), standbys: z, y 2024-09-19T08:03:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:02 smithi162 ceph-mon[35260]: from='client.24778 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:02 smithi162 ceph-mon[35260]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:03:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:02 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2653200192' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:03:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:02 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2775166662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-19T08:03:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:02 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:02 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:02 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:02 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:03.121 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-mon[31333]: mgrmap e20: x(active, since 1.01884s), standbys: z, y 2024-09-19T08:03:03.121 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-mon[31333]: from='client.24778 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:03.121 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-mon[31333]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:03:03.121 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2653200192' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:03:03.121 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2775166662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-19T08:03:03.121 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:03.121 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:03.121 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:03.121 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:02 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:03.122 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-09-19T08:03:03.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:03:03.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:02 smithi139 ceph-mon[34874]: mgrmap e20: x(active, since 1.01884s), standbys: z, y 2024-09-19T08:03:03.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:02 smithi139 ceph-mon[34874]: from='client.24778 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:03.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:02 smithi139 ceph-mon[34874]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:03:03.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:02 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2653200192' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:03:03.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:02 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2775166662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-19T08:03:03.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:02 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:03.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:02 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:03.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:02 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:03.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:02 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:03.381 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.911+0000 7f4717c3b640 1 -- 172.21.15.98:0/1470257831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4710108300 msgr2=0x7f471010ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.911+0000 7f4717c3b640 1 --2- 172.21.15.98:0/1470257831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4710108300 0x7f471010ec70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4708009cd0 tx=0x7f470802a8d0 comp rx=0 tx=0).stop 2024-09-19T08:03:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.912+0000 7f4717c3b640 1 -- 172.21.15.98:0/1470257831 shutdown_connections 2024-09-19T08:03:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.912+0000 7f4717c3b640 1 --2- 172.21.15.98:0/1470257831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4710108300 0x7f471010ec70 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.912+0000 7f4717c3b640 1 --2- 172.21.15.98:0/1470257831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4710103780 0x7f4710107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.912+0000 7f4717c3b640 1 --2- 172.21.15.98:0/1470257831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4710102dd0 0x7f47101031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.912+0000 7f4717c3b640 1 -- 172.21.15.98:0/1470257831 >> 172.21.15.98:0/1470257831 conn(0x7f47100fc7d0 msgr2=0x7f47100febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.912+0000 7f4717c3b640 1 -- 172.21.15.98:0/1470257831 shutdown_connections 2024-09-19T08:03:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.912+0000 7f4717c3b640 1 -- 172.21.15.98:0/1470257831 wait complete. 2024-09-19T08:03:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.913+0000 7f4717c3b640 1 Processor -- start 2024-09-19T08:03:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.913+0000 7f4717c3b640 1 -- start start 2024-09-19T08:03:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.913+0000 7f4717c3b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4710102dd0 0x7f4710198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.913+0000 7f4717c3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4710103780 0x7f4710198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.913+0000 7f4717c3b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4710108300 0x7f471019ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.913+0000 7f4717c3b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47101120d0 con 0x7f4710102dd0 2024-09-19T08:03:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.913+0000 7f4717c3b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4710111f50 con 0x7f4710108300 2024-09-19T08:03:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.913+0000 7f4717c3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4710112250 con 0x7f4710103780 2024-09-19T08:03:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.913+0000 7f47159d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4710102dd0 0x7f4710198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:03.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.913+0000 7f47151d6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4710103780 0x7f4710198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:03.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.913+0000 7f47159d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4710102dd0 0x7f4710198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55120/0 (socket says 172.21.15.98:55120) 2024-09-19T08:03:03.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.913+0000 7f47159d7640 1 -- 172.21.15.98:0/1698374312 learned_addr learned my addr 172.21.15.98:0/1698374312 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:03.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.914+0000 7f47161d8640 1 --2- 172.21.15.98:0/1698374312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4710108300 0x7f471019ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:03.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.914+0000 7f47151d6640 1 -- 172.21.15.98:0/1698374312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4710108300 msgr2=0x7f471019ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:03.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.914+0000 7f47151d6640 1 --2- 172.21.15.98:0/1698374312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4710108300 0x7f471019ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:03.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.914+0000 7f47151d6640 1 -- 172.21.15.98:0/1698374312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4710102dd0 msgr2=0x7f4710198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:03.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.914+0000 7f47151d6640 1 --2- 172.21.15.98:0/1698374312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4710102dd0 0x7f4710198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:03.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.914+0000 7f47151d6640 1 -- 172.21.15.98:0/1698374312 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f471019d530 con 0x7f4710103780 2024-09-19T08:03:03.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.914+0000 7f47159d7640 1 --2- 172.21.15.98:0/1698374312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4710102dd0 0x7f4710198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:03.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.914+0000 7f47151d6640 1 --2- 172.21.15.98:0/1698374312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4710103780 0x7f4710198ac0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f47100697d0 tx=0x7f470000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:03.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.914+0000 7f4706ffd640 1 -- 172.21.15.98:0/1698374312 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470000cc40 con 0x7f4710103780 2024-09-19T08:03:03.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.914+0000 7f4717c3b640 1 -- 172.21.15.98:0/1698374312 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f471019d820 con 0x7f4710103780 2024-09-19T08:03:03.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.914+0000 7f4706ffd640 1 -- 172.21.15.98:0/1698374312 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f470000cde0 con 0x7f4710103780 2024-09-19T08:03:03.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.914+0000 7f4706ffd640 1 -- 172.21.15.98:0/1698374312 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47000188e0 con 0x7f4710103780 2024-09-19T08:03:03.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.915+0000 7f4717c3b640 1 -- 172.21.15.98:0/1698374312 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47101a5100 con 0x7f4710103780 2024-09-19T08:03:03.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.915+0000 7f4717c3b640 1 -- 172.21.15.98:0/1698374312 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47101031b0 con 0x7f4710103780 2024-09-19T08:03:03.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.917+0000 7f4706ffd640 1 -- 172.21.15.98:0/1698374312 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 21) ==== 138526+0+0 (secure 0 0 0) 0x7f4700018a80 con 0x7f4710103780 2024-09-19T08:03:03.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.918+0000 7f4706ffd640 1 --2- 172.21.15.98:0/1698374312 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f46fc0a3440 0x7f46fc0a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:03.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.919+0000 7f47159d7640 1 --2- 172.21.15.98:0/1698374312 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f46fc0a3440 0x7f46fc0a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:03.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.919+0000 7f4706ffd640 1 -- 172.21.15.98:0/1698374312 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f47000d0150 con 0x7f4710103780 2024-09-19T08:03:03.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.919+0000 7f47159d7640 1 --2- 172.21.15.98:0/1698374312 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f46fc0a3440 0x7f46fc0a5900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f470c011550 tx=0x7f470c005f90 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:03.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:03.921+0000 7f4706ffd640 1 -- 172.21.15.98:0/1698374312 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f470009c3a0 con 0x7f4710103780 2024-09-19T08:03:04.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: [19/Sep/2024:08:03:01] ENGINE Bus STARTING 2024-09-19T08:03:04.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: [19/Sep/2024:08:03:02] ENGINE Serving on http://172.21.15.98:8765 2024-09-19T08:03:04.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: [19/Sep/2024:08:03:02] ENGINE Serving on https://172.21.15.98:7150 2024-09-19T08:03:04.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: [19/Sep/2024:08:03:02] ENGINE Bus STARTED 2024-09-19T08:03:04.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: [19/Sep/2024:08:03:02] ENGINE Error in HTTPServer.serve 2024-09-19T08:03:04.121 INFO:journalctl@ceph.mon.c.smithi162.stdout: Traceback (most recent call last): 2024-09-19T08:03:04.121 INFO:journalctl@ceph.mon.c.smithi162.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-19T08:03:04.121 INFO:journalctl@ceph.mon.c.smithi162.stdout: self._connections.run(self.expiration_interval) 2024-09-19T08:03:04.121 INFO:journalctl@ceph.mon.c.smithi162.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-19T08:03:04.121 INFO:journalctl@ceph.mon.c.smithi162.stdout: self._run(expiration_interval) 2024-09-19T08:03:04.121 INFO:journalctl@ceph.mon.c.smithi162.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-19T08:03:04.121 INFO:journalctl@ceph.mon.c.smithi162.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-19T08:03:04.121 INFO:journalctl@ceph.mon.c.smithi162.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-19T08:03:04.121 INFO:journalctl@ceph.mon.c.smithi162.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-19T08:03:04.121 INFO:journalctl@ceph.mon.c.smithi162.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-19T08:03:04.121 INFO:journalctl@ceph.mon.c.smithi162.stdout: s = self.context.wrap_socket( 2024-09-19T08:03:04.121 INFO:journalctl@ceph.mon.c.smithi162.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-19T08:03:04.121 INFO:journalctl@ceph.mon.c.smithi162.stdout: return self.sslsocket_class._create( 2024-09-19T08:03:04.122 INFO:journalctl@ceph.mon.c.smithi162.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-19T08:03:04.122 INFO:journalctl@ceph.mon.c.smithi162.stdout: self.do_handshake() 2024-09-19T08:03:04.122 INFO:journalctl@ceph.mon.c.smithi162.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-19T08:03:04.122 INFO:journalctl@ceph.mon.c.smithi162.stdout: self._sslobj.do_handshake() 2024-09-19T08:03:04.122 INFO:journalctl@ceph.mon.c.smithi162.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-19T08:03:04.122 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.122 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.122 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:03:04.122 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2195507374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-19T08:03:04.122 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1598525919' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-19T08:03:04.122 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.122 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.122 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.122 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.123 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:04.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:04.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:03 smithi162 ceph-mon[35260]: mgrmap e21: x(active, since 2s), standbys: z, y 2024-09-19T08:03:04.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:04.168+0000 7f4717c3b640 1 -- 172.21.15.98:0/1698374312 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f471010a290 con 0x7f4710103780 2024-09-19T08:03:04.169 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: [19/Sep/2024:08:03:01] ENGINE Bus STARTING 2024-09-19T08:03:04.169 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: [19/Sep/2024:08:03:02] ENGINE Serving on http://172.21.15.98:8765 2024-09-19T08:03:04.169 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: [19/Sep/2024:08:03:02] ENGINE Serving on https://172.21.15.98:7150 2024-09-19T08:03:04.170 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: [19/Sep/2024:08:03:02] ENGINE Bus STARTED 2024-09-19T08:03:04.170 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: [19/Sep/2024:08:03:02] ENGINE Error in HTTPServer.serve 2024-09-19T08:03:04.170 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-09-19T08:03:04.170 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-19T08:03:04.170 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._connections.run(self.expiration_interval) 2024-09-19T08:03:04.170 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-19T08:03:04.170 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._run(expiration_interval) 2024-09-19T08:03:04.170 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-19T08:03:04.170 INFO:journalctl@ceph.mon.a.smithi098.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-19T08:03:04.170 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-19T08:03:04.170 INFO:journalctl@ceph.mon.a.smithi098.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-19T08:03:04.170 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-19T08:03:04.171 INFO:journalctl@ceph.mon.a.smithi098.stdout: s = self.context.wrap_socket( 2024-09-19T08:03:04.171 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-19T08:03:04.171 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.sslsocket_class._create( 2024-09-19T08:03:04.171 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-19T08:03:04.171 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.do_handshake() 2024-09-19T08:03:04.171 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-19T08:03:04.171 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._sslobj.do_handshake() 2024-09-19T08:03:04.171 INFO:journalctl@ceph.mon.a.smithi098.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-19T08:03:04.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:03:04.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2195507374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-19T08:03:04.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1598525919' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-19T08:03:04.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.173 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.173 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.173 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.173 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.173 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.173 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:04.173 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:04.173 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:03 smithi098 ceph-mon[31333]: mgrmap e21: x(active, since 2s), standbys: z, y 2024-09-19T08:03:04.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:04.169+0000 7f4706ffd640 1 -- 172.21.15.98:0/1698374312 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11983 (secure 0 0 0) 0x7f47000a06a0 con 0x7f4710103780 2024-09-19T08:03:04.174 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:04.174 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"btime":"2024-09-19T08:02:59:855935+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:02:59.805764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:03:04.174 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-09-19T08:03:04.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:04.171+0000 7f4717c3b640 1 -- 172.21.15.98:0/1698374312 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f46fc0a3440 msgr2=0x7f46fc0a5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:04.171+0000 7f4717c3b640 1 --2- 172.21.15.98:0/1698374312 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f46fc0a3440 0x7f46fc0a5900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f470c011550 tx=0x7f470c005f90 comp rx=0 tx=0).stop 2024-09-19T08:03:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:04.171+0000 7f4717c3b640 1 -- 172.21.15.98:0/1698374312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4710103780 msgr2=0x7f4710198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:04.171+0000 7f4717c3b640 1 --2- 172.21.15.98:0/1698374312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4710103780 0x7f4710198ac0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f47100697d0 tx=0x7f470000ee50 comp rx=0 tx=0).stop 2024-09-19T08:03:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:04.171+0000 7f4717c3b640 1 -- 172.21.15.98:0/1698374312 shutdown_connections 2024-09-19T08:03:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:04.171+0000 7f4717c3b640 1 --2- 172.21.15.98:0/1698374312 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f46fc0a3440 0x7f46fc0a5900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:04.171+0000 7f4717c3b640 1 --2- 172.21.15.98:0/1698374312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4710108300 0x7f471019ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:04.171+0000 7f4717c3b640 1 --2- 172.21.15.98:0/1698374312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4710103780 0x7f4710198ac0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:04.171+0000 7f4717c3b640 1 --2- 172.21.15.98:0/1698374312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4710102dd0 0x7f4710198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:04.171+0000 7f4717c3b640 1 -- 172.21.15.98:0/1698374312 >> 172.21.15.98:0/1698374312 conn(0x7f47100fc7d0 msgr2=0x7f4710105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:04.171+0000 7f4717c3b640 1 -- 172.21.15.98:0/1698374312 shutdown_connections 2024-09-19T08:03:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:04.171+0000 7f4717c3b640 1 -- 172.21.15.98:0/1698374312 wait complete. 2024-09-19T08:03:04.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: [19/Sep/2024:08:03:01] ENGINE Bus STARTING 2024-09-19T08:03:04.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: [19/Sep/2024:08:03:02] ENGINE Serving on http://172.21.15.98:8765 2024-09-19T08:03:04.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: [19/Sep/2024:08:03:02] ENGINE Serving on https://172.21.15.98:7150 2024-09-19T08:03:04.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: [19/Sep/2024:08:03:02] ENGINE Bus STARTED 2024-09-19T08:03:04.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: [19/Sep/2024:08:03:02] ENGINE Error in HTTPServer.serve 2024-09-19T08:03:04.178 INFO:journalctl@ceph.mon.b.smithi139.stdout: Traceback (most recent call last): 2024-09-19T08:03:04.178 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-19T08:03:04.178 INFO:journalctl@ceph.mon.b.smithi139.stdout: self._connections.run(self.expiration_interval) 2024-09-19T08:03:04.178 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-19T08:03:04.178 INFO:journalctl@ceph.mon.b.smithi139.stdout: self._run(expiration_interval) 2024-09-19T08:03:04.178 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-19T08:03:04.178 INFO:journalctl@ceph.mon.b.smithi139.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-19T08:03:04.178 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-19T08:03:04.178 INFO:journalctl@ceph.mon.b.smithi139.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-19T08:03:04.178 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-19T08:03:04.178 INFO:journalctl@ceph.mon.b.smithi139.stdout: s = self.context.wrap_socket( 2024-09-19T08:03:04.179 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-19T08:03:04.179 INFO:journalctl@ceph.mon.b.smithi139.stdout: return self.sslsocket_class._create( 2024-09-19T08:03:04.179 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-19T08:03:04.179 INFO:journalctl@ceph.mon.b.smithi139.stdout: self.do_handshake() 2024-09-19T08:03:04.179 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-19T08:03:04.179 INFO:journalctl@ceph.mon.b.smithi139.stdout: self._sslobj.do_handshake() 2024-09-19T08:03:04.179 INFO:journalctl@ceph.mon.b.smithi139.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-19T08:03:04.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:03:04.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2195507374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-19T08:03:04.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1598525919' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-19T08:03:04.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.180 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.180 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.180 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.180 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.180 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.180 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.180 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.180 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.180 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.180 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.180 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.180 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.181 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.181 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.181 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.181 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.181 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:03:04.181 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:04.181 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:04.181 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:03 smithi139 ceph-mon[34874]: mgrmap e21: x(active, since 2s), standbys: z, y 2024-09-19T08:03:04.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi162 to 176.0M 2024-09-19T08:03:04.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: Unable to set osd_memory_target on smithi162 to 184591052: error parsing value: Value '184591052' is below minimum 939524096 2024-09-19T08:03:04.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi139 to 176.0M 2024-09-19T08:03:04.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: Unable to set osd_memory_target on smithi139 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-19T08:03:04.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi098 to 176.0M 2024-09-19T08:03:04.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: Unable to set osd_memory_target on smithi098 to 184576716: error parsing value: Value '184576716' is below minimum 939524096 2024-09-19T08:03:04.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-19T08:03:04.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-19T08:03:04.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-19T08:03:04.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: Updating smithi139:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T08:03:04.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: Updating smithi162:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T08:03:04.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: Updating smithi098:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T08:03:04.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1698374312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:04.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:04.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:04.878 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:04.878 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:04 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:04.902 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-09-19T08:03:04.902 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-09-19T08:03:04.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T08:03:05.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.002+0000 7fc2bc752640 1 -- 172.21.15.98:0/3076643515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b4103610 msgr2=0x7fc2b410baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.002+0000 7fc2bc752640 1 --2- 172.21.15.98:0/3076643515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b4103610 0x7fc2b410baf0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4009900 tx=0x7fc2a402a480 comp rx=0 tx=0).stop 2024-09-19T08:03:05.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 -- 172.21.15.98:0/3076643515 shutdown_connections 2024-09-19T08:03:05.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 --2- 172.21.15.98:0/3076643515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc2b410c0c0 0x7fc2b410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 --2- 172.21.15.98:0/3076643515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b4103610 0x7fc2b410baf0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 --2- 172.21.15.98:0/3076643515 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2b4102cf0 0x7fc2b41030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 -- 172.21.15.98:0/3076643515 >> 172.21.15.98:0/3076643515 conn(0x7fc2b4077a10 msgr2=0x7fc2b40fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:05.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 -- 172.21.15.98:0/3076643515 shutdown_connections 2024-09-19T08:03:05.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 -- 172.21.15.98:0/3076643515 wait complete. 2024-09-19T08:03:05.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 Processor -- start 2024-09-19T08:03:05.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 -- start start 2024-09-19T08:03:05.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc2b4102cf0 0x7fc2b4198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:05.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b4103610 0x7fc2b41988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:05.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2b410c0c0 0x7fc2b419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:05.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b4111e90 con 0x7fc2b4103610 2024-09-19T08:03:05.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b4111d10 con 0x7fc2b410c0c0 2024-09-19T08:03:05.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2bc752640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b4112010 con 0x7fc2b4102cf0 2024-09-19T08:03:05.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2b9ced640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b4103610 0x7fc2b41988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:05.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2b9ced640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b4103610 0x7fc2b41988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55138/0 (socket says 172.21.15.98:55138) 2024-09-19T08:03:05.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2b9ced640 1 -- 172.21.15.98:0/3024544436 learned_addr learned my addr 172.21.15.98:0/3024544436 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:05.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.003+0000 7fc2ba4ee640 1 --2- 172.21.15.98:0/3024544436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc2b4102cf0 0x7fc2b4198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:05.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.004+0000 7fc2bacef640 1 --2- 172.21.15.98:0/3024544436 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2b410c0c0 0x7fc2b419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:05.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.004+0000 7fc2b9ced640 1 -- 172.21.15.98:0/3024544436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc2b4102cf0 msgr2=0x7fc2b4198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.004+0000 7fc2b9ced640 1 --2- 172.21.15.98:0/3024544436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc2b4102cf0 0x7fc2b4198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.004+0000 7fc2b9ced640 1 -- 172.21.15.98:0/3024544436 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2b410c0c0 msgr2=0x7fc2b419cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.004+0000 7fc2b9ced640 1 --2- 172.21.15.98:0/3024544436 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2b410c0c0 0x7fc2b419cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.004+0000 7fc2b9ced640 1 -- 172.21.15.98:0/3024544436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b419d350 con 0x7fc2b4103610 2024-09-19T08:03:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.004+0000 7fc2b9ced640 1 --2- 172.21.15.98:0/3024544436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b4103610 0x7fc2b41988a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4005ab0 tx=0x7fc2a4004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.004+0000 7fc2ab7fe640 1 -- 172.21.15.98:0/3024544436 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a403f030 con 0x7fc2b4103610 2024-09-19T08:03:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.004+0000 7fc2ab7fe640 1 -- 172.21.15.98:0/3024544436 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2a4003ce0 con 0x7fc2b4103610 2024-09-19T08:03:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.004+0000 7fc2ab7fe640 1 -- 172.21.15.98:0/3024544436 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4035070 con 0x7fc2b4103610 2024-09-19T08:03:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.004+0000 7fc2bc752640 1 -- 172.21.15.98:0/3024544436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b419d5e0 con 0x7fc2b4103610 2024-09-19T08:03:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.004+0000 7fc2bc752640 1 -- 172.21.15.98:0/3024544436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b41a4ed0 con 0x7fc2b4103610 2024-09-19T08:03:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.005+0000 7fc2bc752640 1 -- 172.21.15.98:0/3024544436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b41030d0 con 0x7fc2b4103610 2024-09-19T08:03:05.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.006+0000 7fc2ab7fe640 1 -- 172.21.15.98:0/3024544436 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 21) ==== 138526+0+0 (secure 0 0 0) 0x7fc2a4042030 con 0x7fc2b4103610 2024-09-19T08:03:05.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.007+0000 7fc2ab7fe640 1 --2- 172.21.15.98:0/3024544436 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc2a00a3450 0x7fc2a00a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:05.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.007+0000 7fc2ba4ee640 1 --2- 172.21.15.98:0/3024544436 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc2a00a3450 0x7fc2a00a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:05.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.007+0000 7fc2ab7fe640 1 -- 172.21.15.98:0/3024544436 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fc2a40f0790 con 0x7fc2b4103610 2024-09-19T08:03:05.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.007+0000 7fc2ba4ee640 1 --2- 172.21.15.98:0/3024544436 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc2a00a3450 0x7fc2a00a5910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac007ec0 tx=0x7fc2ac0075b0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:05.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.011+0000 7fc2ab7fe640 1 -- 172.21.15.98:0/3024544436 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fc2a40bd030 con 0x7fc2b4103610 2024-09-19T08:03:05.078 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi162 to 176.0M 2024-09-19T08:03:05.078 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: Unable to set osd_memory_target on smithi162 to 184591052: error parsing value: Value '184591052' is below minimum 939524096 2024-09-19T08:03:05.078 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi139 to 176.0M 2024-09-19T08:03:05.078 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: Unable to set osd_memory_target on smithi139 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-19T08:03:05.078 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi098 to 176.0M 2024-09-19T08:03:05.078 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: Unable to set osd_memory_target on smithi098 to 184576716: error parsing value: Value '184576716' is below minimum 939524096 2024-09-19T08:03:05.078 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-19T08:03:05.078 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-19T08:03:05.078 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-19T08:03:05.078 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: Updating smithi139:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T08:03:05.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: Updating smithi162:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T08:03:05.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: Updating smithi098:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T08:03:05.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1698374312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:05.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:05.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:05.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:05.079 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:04 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi162 to 176.0M 2024-09-19T08:03:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: Unable to set osd_memory_target on smithi162 to 184591052: error parsing value: Value '184591052' is below minimum 939524096 2024-09-19T08:03:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi139 to 176.0M 2024-09-19T08:03:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: Unable to set osd_memory_target on smithi139 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-19T08:03:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi098 to 176.0M 2024-09-19T08:03:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: Unable to set osd_memory_target on smithi098 to 184576716: error parsing value: Value '184576716' is below minimum 939524096 2024-09-19T08:03:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-19T08:03:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-19T08:03:05.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-19T08:03:05.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: Updating smithi139:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T08:03:05.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: Updating smithi162:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T08:03:05.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: Updating smithi098:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T08:03:05.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1698374312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:05.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:05.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:05.122 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:05.122 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:04 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:05.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.173+0000 7fc2bc752640 1 -- 172.21.15.98:0/3024544436 --> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc2b4107140 con 0x7fc2a00a3450 2024-09-19T08:03:05.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.180+0000 7fc2ab7fe640 1 -- 172.21.15.98:0/3024544436 <== mgr.15252 v2:172.21.15.98:6800/1961446727 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257946 (secure 0 0 0) 0x7fc2b4107140 con 0x7fc2a00a3450 2024-09-19T08:03:05.184 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:05.185 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-19T08:03:05.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.183+0000 7fc2bc752640 1 -- 172.21.15.98:0/3024544436 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc2a00a3450 msgr2=0x7fc2a00a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.183+0000 7fc2bc752640 1 --2- 172.21.15.98:0/3024544436 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc2a00a3450 0x7fc2a00a5910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac007ec0 tx=0x7fc2ac0075b0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.183+0000 7fc2bc752640 1 -- 172.21.15.98:0/3024544436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b4103610 msgr2=0x7fc2b41988a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.183+0000 7fc2bc752640 1 --2- 172.21.15.98:0/3024544436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b4103610 0x7fc2b41988a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4005ab0 tx=0x7fc2a4004360 comp rx=0 tx=0).stop 2024-09-19T08:03:05.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.183+0000 7fc2bc752640 1 -- 172.21.15.98:0/3024544436 shutdown_connections 2024-09-19T08:03:05.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.183+0000 7fc2bc752640 1 --2- 172.21.15.98:0/3024544436 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc2a00a3450 0x7fc2a00a5910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.183+0000 7fc2bc752640 1 --2- 172.21.15.98:0/3024544436 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2b410c0c0 0x7fc2b419cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.183+0000 7fc2bc752640 1 --2- 172.21.15.98:0/3024544436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b4103610 0x7fc2b41988a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.183+0000 7fc2bc752640 1 --2- 172.21.15.98:0/3024544436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc2b4102cf0 0x7fc2b4198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.183+0000 7fc2bc752640 1 -- 172.21.15.98:0/3024544436 >> 172.21.15.98:0/3024544436 conn(0x7fc2b4077a10 msgr2=0x7fc2b40fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:05.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.183+0000 7fc2bc752640 1 -- 172.21.15.98:0/3024544436 shutdown_connections 2024-09-19T08:03:05.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.183+0000 7fc2bc752640 1 -- 172.21.15.98:0/3024544436 wait complete. 2024-09-19T08:03:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-09-19T08:03:04.771716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191268,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1275,"num_read_kb":802,"num_write":555,"num_write_kb":4403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181990,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":939,"ondisk_log_size":939,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376028,"kb_used_data":14144,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565796,"statfs":{"total":1151940427776,"available":1151555375104,"internally_reserved":0,"allocated":14483456,"data_stored":8708426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":370193724},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.989911"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780911+0000","last_change":"2024-09-19T08:01:10.768950+0000","last_active":"2024-09-19T08:03:00.780911+0000","last_peered":"2024-09-19T08:03:00.780911+0000","last_clean":"2024-09-19T08:03:00.780911+0000","last_became_active":"2024-09-19T08:01:10.768047+0000","last_became_peered":"2024-09-19T08:01:10.768047+0000","last_unstale":"2024-09-19T08:03:00.780911+0000","last_undegraded":"2024-09-19T08:03:00.780911+0000","last_fullsized":"2024-09-19T08:03:00.780911+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:26.219079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994728+0000","last_change":"2024-09-19T08:01:10.783208+0000","last_active":"2024-09-19T08:03:00.994728+0000","last_peered":"2024-09-19T08:03:00.994728+0000","last_clean":"2024-09-19T08:03:00.994728+0000","last_became_active":"2024-09-19T08:01:10.782888+0000","last_became_peered":"2024-09-19T08:01:10.782888+0000","last_unstale":"2024-09-19T08:03:00.994728+0000","last_undegraded":"2024-09-19T08:03:00.994728+0000","last_fullsized":"2024-09-19T08:03:00.994728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:51:27.603166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994427+0000","last_change":"2024-09-19T08:01:10.770395+0000","last_active":"2024-09-19T08:03:00.994427+0000","last_peered":"2024-09-19T08:03:00.994427+0000","last_clean":"2024-09-19T08:03:00.994427+0000","last_became_active":"2024-09-19T08:01:10.770073+0000","last_became_peered":"2024-09-19T08:01:10.770073+0000","last_unstale":"2024-09-19T08:03:00.994427+0000","last_undegraded":"2024-09-19T08:03:00.994427+0000","last_fullsized":"2024-09-19T08:03:00.994427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:20.047023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761373+0000","last_change":"2024-09-19T08:01:10.777138+0000","last_active":"2024-09-19T08:03:00.761373+0000","last_peered":"2024-09-19T08:03:00.761373+0000","last_clean":"2024-09-19T08:03:00.761373+0000","last_became_active":"2024-09-19T08:01:10.776810+0000","last_became_peered":"2024-09-19T08:01:10.776810+0000","last_unstale":"2024-09-19T08:03:00.761373+0000","last_undegraded":"2024-09-19T08:03:00.761373+0000","last_fullsized":"2024-09-19T08:03:00.761373+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:39.983321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847497+0000","last_change":"2024-09-19T08:01:10.770435+0000","last_active":"2024-09-19T08:03:00.847497+0000","last_peered":"2024-09-19T08:03:00.847497+0000","last_clean":"2024-09-19T08:03:00.847497+0000","last_became_active":"2024-09-19T08:01:10.769686+0000","last_became_peered":"2024-09-19T08:01:10.769686+0000","last_unstale":"2024-09-19T08:03:00.847497+0000","last_undegraded":"2024-09-19T08:03:00.847497+0000","last_fullsized":"2024-09-19T08:03:00.847497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:10:19.610925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839257+0000","last_change":"2024-09-19T08:01:10.782382+0000","last_active":"2024-09-19T08:03:00.839257+0000","last_peered":"2024-09-19T08:03:00.839257+0000","last_clean":"2024-09-19T08:03:00.839257+0000","last_became_active":"2024-09-19T08:01:10.781996+0000","last_became_peered":"2024-09-19T08:01:10.781996+0000","last_unstale":"2024-09-19T08:03:00.839257+0000","last_undegraded":"2024-09-19T08:03:00.839257+0000","last_fullsized":"2024-09-19T08:03:00.839257+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:43:13.574336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837718+0000","last_change":"2024-09-19T08:01:10.773776+0000","last_active":"2024-09-19T08:03:00.837718+0000","last_peered":"2024-09-19T08:03:00.837718+0000","last_clean":"2024-09-19T08:03:00.837718+0000","last_became_active":"2024-09-19T08:01:10.772442+0000","last_became_peered":"2024-09-19T08:01:10.772442+0000","last_unstale":"2024-09-19T08:03:00.837718+0000","last_undegraded":"2024-09-19T08:03:00.837718+0000","last_fullsized":"2024-09-19T08:03:00.837718+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:08:18.155806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848134+0000","last_change":"2024-09-19T08:01:10.781443+0000","last_active":"2024-09-19T08:03:00.848134+0000","last_peered":"2024-09-19T08:03:00.848134+0000","last_clean":"2024-09-19T08:03:00.848134+0000","last_became_active":"2024-09-19T08:01:10.781083+0000","last_became_peered":"2024-09-19T08:01:10.781083+0000","last_unstale":"2024-09-19T08:03:00.848134+0000","last_undegraded":"2024-09-19T08:03:00.848134+0000","last_fullsized":"2024-09-19T08:03:00.848134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:23:22.631865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.998365+0000","last_change":"2024-09-19T08:01:10.767554+0000","last_active":"2024-09-19T08:03:00.998365+0000","last_peered":"2024-09-19T08:03:00.998365+0000","last_clean":"2024-09-19T08:03:00.998365+0000","last_became_active":"2024-09-19T08:01:10.767227+0000","last_became_peered":"2024-09-19T08:01:10.767227+0000","last_unstale":"2024-09-19T08:03:00.998365+0000","last_undegraded":"2024-09-19T08:03:00.998365+0000","last_fullsized":"2024-09-19T08:03:00.998365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:30:13.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839361+0000","last_change":"2024-09-19T08:01:10.783174+0000","last_active":"2024-09-19T08:03:00.839361+0000","last_peered":"2024-09-19T08:03:00.839361+0000","last_clean":"2024-09-19T08:03:00.839361+0000","last_became_active":"2024-09-19T08:01:10.782794+0000","last_became_peered":"2024-09-19T08:01:10.782794+0000","last_unstale":"2024-09-19T08:03:00.839361+0000","last_undegraded":"2024-09-19T08:03:00.839361+0000","last_fullsized":"2024-09-19T08:03:00.839361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:49:19.451107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838191+0000","last_change":"2024-09-19T08:01:10.779638+0000","last_active":"2024-09-19T08:03:00.838191+0000","last_peered":"2024-09-19T08:03:00.838191+0000","last_clean":"2024-09-19T08:03:00.838191+0000","last_became_active":"2024-09-19T08:01:10.779295+0000","last_became_peered":"2024-09-19T08:01:10.779295+0000","last_unstale":"2024-09-19T08:03:00.838191+0000","last_undegraded":"2024-09-19T08:03:00.838191+0000","last_fullsized":"2024-09-19T08:03:00.838191+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:35:10.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761539+0000","last_change":"2024-09-19T08:01:10.784487+0000","last_active":"2024-09-19T08:03:00.761539+0000","last_peered":"2024-09-19T08:03:00.761539+0000","last_clean":"2024-09-19T08:03:00.761539+0000","last_became_active":"2024-09-19T08:01:10.784114+0000","last_became_peered":"2024-09-19T08:01:10.784114+0000","last_unstale":"2024-09-19T08:03:00.761539+0000","last_undegraded":"2024-09-19T08:03:00.761539+0000","last_fullsized":"2024-09-19T08:03:00.761539+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:09:54.819802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758888+0000","last_change":"2024-09-19T08:01:10.783186+0000","last_active":"2024-09-19T08:03:00.758888+0000","last_peered":"2024-09-19T08:03:00.758888+0000","last_clean":"2024-09-19T08:03:00.758888+0000","last_became_active":"2024-09-19T08:01:10.782876+0000","last_became_peered":"2024-09-19T08:01:10.782876+0000","last_unstale":"2024-09-19T08:03:00.758888+0000","last_undegraded":"2024-09-19T08:03:00.758888+0000","last_fullsized":"2024-09-19T08:03:00.758888+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:26:15.252165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994474+0000","last_change":"2024-09-19T08:01:10.783237+0000","last_active":"2024-09-19T08:03:00.994474+0000","last_peered":"2024-09-19T08:03:00.994474+0000","last_clean":"2024-09-19T08:03:00.994474+0000","last_became_active":"2024-09-19T08:01:10.782966+0000","last_became_peered":"2024-09-19T08:01:10.782966+0000","last_unstale":"2024-09-19T08:03:00.994474+0000","last_undegraded":"2024-09-19T08:03:00.994474+0000","last_fullsized":"2024-09-19T08:03:00.994474+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:57:18.771737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994242+0000","last_change":"2024-09-19T08:01:10.771207+0000","last_active":"2024-09-19T08:03:00.994242+0000","last_peered":"2024-09-19T08:03:00.994242+0000","last_clean":"2024-09-19T08:03:00.994242+0000","last_became_active":"2024-09-19T08:01:10.770862+0000","last_became_peered":"2024-09-19T08:01:10.770862+0000","last_unstale":"2024-09-19T08:03:00.994242+0000","last_undegraded":"2024-09-19T08:03:00.994242+0000","last_fullsized":"2024-09-19T08:03:00.994242+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:08:09.313329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997263+0000","last_change":"2024-09-19T08:01:10.775191+0000","last_active":"2024-09-19T08:03:00.997263+0000","last_peered":"2024-09-19T08:03:00.997263+0000","last_clean":"2024-09-19T08:03:00.997263+0000","last_became_active":"2024-09-19T08:01:10.774832+0000","last_became_peered":"2024-09-19T08:01:10.774832+0000","last_unstale":"2024-09-19T08:03:00.997263+0000","last_undegraded":"2024-09-19T08:03:00.997263+0000","last_fullsized":"2024-09-19T08:03:00.997263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:57:29.800787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780968+0000","last_change":"2024-09-19T08:01:10.769115+0000","last_active":"2024-09-19T08:03:00.780968+0000","last_peered":"2024-09-19T08:03:00.780968+0000","last_clean":"2024-09-19T08:03:00.780968+0000","last_became_active":"2024-09-19T08:01:10.768088+0000","last_became_peered":"2024-09-19T08:01:10.768088+0000","last_unstale":"2024-09-19T08:03:00.780968+0000","last_undegraded":"2024-09-19T08:03:00.780968+0000","last_fullsized":"2024-09-19T08:03:00.780968+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:27:16.042722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844124+0000","last_change":"2024-09-19T08:01:10.781426+0000","last_active":"2024-09-19T08:03:00.844124+0000","last_peered":"2024-09-19T08:03:00.844124+0000","last_clean":"2024-09-19T08:03:00.844124+0000","last_became_active":"2024-09-19T08:01:10.781094+0000","last_became_peered":"2024-09-19T08:01:10.781094+0000","last_unstale":"2024-09-19T08:03:00.844124+0000","last_undegraded":"2024-09-19T08:03:00.844124+0000","last_fullsized":"2024-09-19T08:03:00.844124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:43:31.336610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760020+0000","last_change":"2024-09-19T08:01:10.766466+0000","last_active":"2024-09-19T08:03:00.760020+0000","last_peered":"2024-09-19T08:03:00.760020+0000","last_clean":"2024-09-19T08:03:00.760020+0000","last_became_active":"2024-09-19T08:01:10.766123+0000","last_became_peered":"2024-09-19T08:01:10.766123+0000","last_unstale":"2024-09-19T08:03:00.760020+0000","last_undegraded":"2024-09-19T08:03:00.760020+0000","last_fullsized":"2024-09-19T08:03:00.760020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:25.848603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780633+0000","last_change":"2024-09-19T08:01:10.772782+0000","last_active":"2024-09-19T08:03:00.780633+0000","last_peered":"2024-09-19T08:03:00.780633+0000","last_clean":"2024-09-19T08:03:00.780633+0000","last_became_active":"2024-09-19T08:01:10.772445+0000","last_became_peered":"2024-09-19T08:01:10.772445+0000","last_unstale":"2024-09-19T08:03:00.780633+0000","last_undegraded":"2024-09-19T08:03:00.780633+0000","last_fullsized":"2024-09-19T08:03:00.780633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:09.174005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759047+0000","last_change":"2024-09-19T08:01:10.771944+0000","last_active":"2024-09-19T08:03:00.759047+0000","last_peered":"2024-09-19T08:03:00.759047+0000","last_clean":"2024-09-19T08:03:00.759047+0000","last_became_active":"2024-09-19T08:01:10.771643+0000","last_became_peered":"2024-09-19T08:01:10.771643+0000","last_unstale":"2024-09-19T08:03:00.759047+0000","last_undegraded":"2024-09-19T08:03:00.759047+0000","last_fullsized":"2024-09-19T08:03:00.759047+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:46:27.121722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997451+0000","last_change":"2024-09-19T08:01:10.775144+0000","last_active":"2024-09-19T08:03:00.997451+0000","last_peered":"2024-09-19T08:03:00.997451+0000","last_clean":"2024-09-19T08:03:00.997451+0000","last_became_active":"2024-09-19T08:01:10.774769+0000","last_became_peered":"2024-09-19T08:01:10.774769+0000","last_unstale":"2024-09-19T08:03:00.997451+0000","last_undegraded":"2024-09-19T08:03:00.997451+0000","last_fullsized":"2024-09-19T08:03:00.997451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:59:47.053993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848212+0000","last_change":"2024-09-19T08:01:10.766750+0000","last_active":"2024-09-19T08:03:00.848212+0000","last_peered":"2024-09-19T08:03:00.848212+0000","last_clean":"2024-09-19T08:03:00.848212+0000","last_became_active":"2024-09-19T08:01:10.766450+0000","last_became_peered":"2024-09-19T08:01:10.766450+0000","last_unstale":"2024-09-19T08:03:00.848212+0000","last_undegraded":"2024-09-19T08:03:00.848212+0000","last_fullsized":"2024-09-19T08:03:00.848212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:47:58.133815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841186+0000","last_change":"2024-09-19T08:01:10.768430+0000","last_active":"2024-09-19T08:03:00.841186+0000","last_peered":"2024-09-19T08:03:00.841186+0000","last_clean":"2024-09-19T08:03:00.841186+0000","last_became_active":"2024-09-19T08:01:10.768092+0000","last_became_peered":"2024-09-19T08:01:10.768092+0000","last_unstale":"2024-09-19T08:03:00.841186+0000","last_undegraded":"2024-09-19T08:03:00.841186+0000","last_fullsized":"2024-09-19T08:03:00.841186+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:16:42.343138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849206+0000","last_change":"2024-09-19T08:01:10.771131+0000","last_active":"2024-09-19T08:03:00.849206+0000","last_peered":"2024-09-19T08:03:00.849206+0000","last_clean":"2024-09-19T08:03:00.849206+0000","last_became_active":"2024-09-19T08:01:10.770851+0000","last_became_peered":"2024-09-19T08:01:10.770851+0000","last_unstale":"2024-09-19T08:03:00.849206+0000","last_undegraded":"2024-09-19T08:03:00.849206+0000","last_fullsized":"2024-09-19T08:03:00.849206+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:07:15.830476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760273+0000","last_change":"2024-09-19T08:01:10.776220+0000","last_active":"2024-09-19T08:03:00.760273+0000","last_peered":"2024-09-19T08:03:00.760273+0000","last_clean":"2024-09-19T08:03:00.760273+0000","last_became_active":"2024-09-19T08:01:10.775889+0000","last_became_peered":"2024-09-19T08:01:10.775889+0000","last_unstale":"2024-09-19T08:03:00.760273+0000","last_undegraded":"2024-09-19T08:03:00.760273+0000","last_fullsized":"2024-09-19T08:03:00.760273+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:23.714675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780731+0000","last_change":"2024-09-19T08:01:10.776370+0000","last_active":"2024-09-19T08:03:00.780731+0000","last_peered":"2024-09-19T08:03:00.780731+0000","last_clean":"2024-09-19T08:03:00.780731+0000","last_became_active":"2024-09-19T08:01:10.776027+0000","last_became_peered":"2024-09-19T08:01:10.776027+0000","last_unstale":"2024-09-19T08:03:00.780731+0000","last_undegraded":"2024-09-19T08:03:00.780731+0000","last_fullsized":"2024-09-19T08:03:00.780731+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:48:46.293404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758951+0000","last_change":"2024-09-19T08:01:26.758971+0000","last_active":"2024-09-19T08:03:00.758951+0000","last_peered":"2024-09-19T08:03:00.758951+0000","last_clean":"2024-09-19T08:03:00.758951+0000","last_became_active":"2024-09-19T08:01:26.758524+0000","last_became_peered":"2024-09-19T08:01:26.758524+0000","last_unstale":"2024-09-19T08:03:00.758951+0000","last_undegraded":"2024-09-19T08:03:00.758951+0000","last_fullsized":"2024-09-19T08:03:00.758951+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:34:22.439327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994656+0000","last_change":"2024-09-19T08:01:10.767445+0000","last_active":"2024-09-19T08:03:00.994656+0000","last_peered":"2024-09-19T08:03:00.994656+0000","last_clean":"2024-09-19T08:03:00.994656+0000","last_became_active":"2024-09-19T08:01:10.767128+0000","last_became_peered":"2024-09-19T08:01:10.767128+0000","last_unstale":"2024-09-19T08:03:00.994656+0000","last_undegraded":"2024-09-19T08:03:00.994656+0000","last_fullsized":"2024-09-19T08:03:00.994656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:10.273948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759852+0000","last_change":"2024-09-19T08:01:10.768954+0000","last_active":"2024-09-19T08:03:00.759852+0000","last_peered":"2024-09-19T08:03:00.759852+0000","last_clean":"2024-09-19T08:03:00.759852+0000","last_became_active":"2024-09-19T08:01:10.768577+0000","last_became_peered":"2024-09-19T08:01:10.768577+0000","last_unstale":"2024-09-19T08:03:00.759852+0000","last_undegraded":"2024-09-19T08:03:00.759852+0000","last_fullsized":"2024-09-19T08:03:00.759852+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:20:21.812531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759843+0000","last_change":"2024-09-19T08:01:10.774104+0000","last_active":"2024-09-19T08:03:00.759843+0000","last_peered":"2024-09-19T08:03:00.759843+0000","last_clean":"2024-09-19T08:03:00.759843+0000","last_became_active":"2024-09-19T08:01:10.773721+0000","last_became_peered":"2024-09-19T08:01:10.773721+0000","last_unstale":"2024-09-19T08:03:00.759843+0000","last_undegraded":"2024-09-19T08:03:00.759843+0000","last_fullsized":"2024-09-19T08:03:00.759843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:45:34.945994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780513+0000","last_change":"2024-09-19T08:01:26.727296+0000","last_active":"2024-09-19T08:03:00.780513+0000","last_peered":"2024-09-19T08:03:00.780513+0000","last_clean":"2024-09-19T08:03:00.780513+0000","last_became_active":"2024-09-19T08:01:26.726631+0000","last_became_peered":"2024-09-19T08:01:26.726631+0000","last_unstale":"2024-09-19T08:03:00.780513+0000","last_undegraded":"2024-09-19T08:03:00.780513+0000","last_fullsized":"2024-09-19T08:03:00.780513+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:19:24.807050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847424+0000","last_change":"2024-09-19T08:01:10.765282+0000","last_active":"2024-09-19T08:03:00.847424+0000","last_peered":"2024-09-19T08:03:00.847424+0000","last_clean":"2024-09-19T08:03:00.847424+0000","last_became_active":"2024-09-19T08:01:10.764928+0000","last_became_peered":"2024-09-19T08:01:10.764928+0000","last_unstale":"2024-09-19T08:03:00.847424+0000","last_undegraded":"2024-09-19T08:03:00.847424+0000","last_fullsized":"2024-09-19T08:03:00.847424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:00:37.395110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"81'10","reported_seq":140,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.999561+0000","last_change":"2024-09-19T08:01:08.760686+0000","last_active":"2024-09-19T08:03:00.999561+0000","last_peered":"2024-09-19T08:03:00.999561+0000","last_clean":"2024-09-19T08:03:00.999561+0000","last_became_active":"2024-09-19T08:01:08.759912+0000","last_became_peered":"2024-09-19T08:01:08.759912+0000","last_unstale":"2024-09-19T08:03:00.999561+0000","last_undegraded":"2024-09-19T08:03:00.999561+0000","last_fullsized":"2024-09-19T08:03:00.999561+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:46:01.879611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2250,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":5,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841395+0000","last_change":"2024-09-19T08:01:10.779668+0000","last_active":"2024-09-19T08:03:00.841395+0000","last_peered":"2024-09-19T08:03:00.841395+0000","last_clean":"2024-09-19T08:03:00.841395+0000","last_became_active":"2024-09-19T08:01:10.779265+0000","last_became_peered":"2024-09-19T08:01:10.779265+0000","last_unstale":"2024-09-19T08:03:00.841395+0000","last_undegraded":"2024-09-19T08:03:00.841395+0000","last_fullsized":"2024-09-19T08:03:00.841395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:06.177700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838967+0000","last_change":"2024-09-19T08:01:08.762668+0000","last_active":"2024-09-19T08:03:00.838967+0000","last_peered":"2024-09-19T08:03:00.838967+0000","last_clean":"2024-09-19T08:03:00.838967+0000","last_became_active":"2024-09-19T08:01:08.762277+0000","last_became_peered":"2024-09-19T08:01:08.762277+0000","last_unstale":"2024-09-19T08:03:00.838967+0000","last_undegraded":"2024-09-19T08:03:00.838967+0000","last_fullsized":"2024-09-19T08:03:00.838967+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:36:59.316655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758585+0000","last_change":"2024-09-19T08:01:08.757401+0000","last_active":"2024-09-19T08:03:00.758585+0000","last_peered":"2024-09-19T08:03:00.758585+0000","last_clean":"2024-09-19T08:03:00.758585+0000","last_became_active":"2024-09-19T08:01:08.757067+0000","last_became_peered":"2024-09-19T08:01:08.757067+0000","last_unstale":"2024-09-19T08:03:00.758585+0000","last_undegraded":"2024-09-19T08:03:00.758585+0000","last_fullsized":"2024-09-19T08:03:00.758585+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:19.617830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848289+0000","last_change":"2024-09-19T08:01:10.765149+0000","last_active":"2024-09-19T08:03:00.848289+0000","last_peered":"2024-09-19T08:03:00.848289+0000","last_clean":"2024-09-19T08:03:00.848289+0000","last_became_active":"2024-09-19T08:01:10.764840+0000","last_became_peered":"2024-09-19T08:01:10.764840+0000","last_unstale":"2024-09-19T08:03:00.848289+0000","last_undegraded":"2024-09-19T08:03:00.848289+0000","last_fullsized":"2024-09-19T08:03:00.848289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:15.539307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841355+0000","last_change":"2024-09-19T08:01:08.759435+0000","last_active":"2024-09-19T08:03:00.841355+0000","last_peered":"2024-09-19T08:03:00.841355+0000","last_clean":"2024-09-19T08:03:00.841355+0000","last_became_active":"2024-09-19T08:01:08.759046+0000","last_became_peered":"2024-09-19T08:01:08.759046+0000","last_unstale":"2024-09-19T08:03:00.841355+0000","last_undegraded":"2024-09-19T08:03:00.841355+0000","last_fullsized":"2024-09-19T08:03:00.841355+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:21:24.077626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.843896+0000","last_change":"2024-09-19T08:01:10.778432+0000","last_active":"2024-09-19T08:03:00.843896+0000","last_peered":"2024-09-19T08:03:00.843896+0000","last_clean":"2024-09-19T08:03:00.843896+0000","last_became_active":"2024-09-19T08:01:10.778098+0000","last_became_peered":"2024-09-19T08:01:10.778098+0000","last_unstale":"2024-09-19T08:03:00.843896+0000","last_undegraded":"2024-09-19T08:03:00.843896+0000","last_fullsized":"2024-09-19T08:03:00.843896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:40:12.921165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":230,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839698+0000","last_change":"2024-09-19T08:01:08.764117+0000","last_active":"2024-09-19T08:03:00.839698+0000","last_peered":"2024-09-19T08:03:00.839698+0000","last_clean":"2024-09-19T08:03:00.839698+0000","last_became_active":"2024-09-19T08:01:08.763308+0000","last_became_peered":"2024-09-19T08:01:08.763308+0000","last_unstale":"2024-09-19T08:03:00.839698+0000","last_undegraded":"2024-09-19T08:03:00.839698+0000","last_fullsized":"2024-09-19T08:03:00.839698+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:19:57.014628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":200,"num_read_kb":200,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997921+0000","last_change":"2024-09-19T08:01:10.781771+0000","last_active":"2024-09-19T08:03:00.997921+0000","last_peered":"2024-09-19T08:03:00.997921+0000","last_clean":"2024-09-19T08:03:00.997921+0000","last_became_active":"2024-09-19T08:01:10.781432+0000","last_became_peered":"2024-09-19T08:01:10.781432+0000","last_unstale":"2024-09-19T08:03:00.997921+0000","last_undegraded":"2024-09-19T08:03:00.997921+0000","last_fullsized":"2024-09-19T08:03:00.997921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:25:15.067850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838023+0000","last_change":"2024-09-19T08:01:26.771171+0000","last_active":"2024-09-19T08:03:00.838023+0000","last_peered":"2024-09-19T08:03:00.838023+0000","last_clean":"2024-09-19T08:03:00.838023+0000","last_became_active":"2024-09-19T08:01:26.759274+0000","last_became_peered":"2024-09-19T08:01:26.759274+0000","last_unstale":"2024-09-19T08:03:00.838023+0000","last_undegraded":"2024-09-19T08:03:00.838023+0000","last_fullsized":"2024-09-19T08:03:00.838023+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:20:54.603420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994926+0000","last_change":"2024-09-19T08:01:10.768318+0000","last_active":"2024-09-19T08:03:00.994926+0000","last_peered":"2024-09-19T08:03:00.994926+0000","last_clean":"2024-09-19T08:03:00.994926+0000","last_became_active":"2024-09-19T08:01:10.768022+0000","last_became_peered":"2024-09-19T08:01:10.768022+0000","last_unstale":"2024-09-19T08:03:00.994926+0000","last_undegraded":"2024-09-19T08:03:00.994926+0000","last_fullsized":"2024-09-19T08:03:00.994926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:00:45.328638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841594+0000","last_change":"2024-09-19T08:01:09.750530+0000","last_active":"2024-09-19T08:03:00.841594+0000","last_peered":"2024-09-19T08:03:00.841594+0000","last_clean":"2024-09-19T08:03:00.841594+0000","last_became_active":"2024-09-19T08:01:09.750029+0000","last_became_peered":"2024-09-19T08:01:09.750029+0000","last_unstale":"2024-09-19T08:03:00.841594+0000","last_undegraded":"2024-09-19T08:03:00.841594+0000","last_fullsized":"2024-09-19T08:03:00.841594+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:01:16.062992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997659+0000","last_change":"2024-09-19T08:01:26.724636+0000","last_active":"2024-09-19T08:03:00.997659+0000","last_peered":"2024-09-19T08:03:00.997659+0000","last_clean":"2024-09-19T08:03:00.997659+0000","last_became_active":"2024-09-19T08:01:26.724171+0000","last_became_peered":"2024-09-19T08:01:26.724171+0000","last_unstale":"2024-09-19T08:03:00.997659+0000","last_undegraded":"2024-09-19T08:03:00.997659+0000","last_fullsized":"2024-09-19T08:03:00.997659+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:54:05.998374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'37","reported_seq":275,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997077+0000","last_change":"2024-09-19T08:01:08.761950+0000","last_active":"2024-09-19T08:03:00.997077+0000","last_peered":"2024-09-19T08:03:00.997077+0000","last_clean":"2024-09-19T08:03:00.997077+0000","last_became_active":"2024-09-19T08:01:08.761533+0000","last_became_peered":"2024-09-19T08:01:08.761533+0000","last_unstale":"2024-09-19T08:03:00.997077+0000","last_undegraded":"2024-09-19T08:03:00.997077+0000","last_fullsized":"2024-09-19T08:03:00.997077+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:07:26.668141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66480,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":214,"num_read_kb":171,"num_write":39,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847631+0000","last_change":"2024-09-19T08:01:10.774280+0000","last_active":"2024-09-19T08:03:00.847631+0000","last_peered":"2024-09-19T08:03:00.847631+0000","last_clean":"2024-09-19T08:03:00.847631+0000","last_became_active":"2024-09-19T08:01:10.773982+0000","last_became_peered":"2024-09-19T08:01:10.773982+0000","last_unstale":"2024-09-19T08:03:00.847631+0000","last_undegraded":"2024-09-19T08:03:00.847631+0000","last_fullsized":"2024-09-19T08:03:00.847631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:21:43.180057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760058+0000","last_change":"2024-09-19T08:01:08.761067+0000","last_active":"2024-09-19T08:03:00.760058+0000","last_peered":"2024-09-19T08:03:00.760058+0000","last_clean":"2024-09-19T08:03:00.760058+0000","last_became_active":"2024-09-19T08:01:08.760664+0000","last_became_peered":"2024-09-19T08:01:08.760664+0000","last_unstale":"2024-09-19T08:03:00.760058+0000","last_undegraded":"2024-09-19T08:03:00.760058+0000","last_fullsized":"2024-09-19T08:03:00.760058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:22:38.086891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758630+0000","last_change":"2024-09-19T08:01:10.771845+0000","last_active":"2024-09-19T08:03:00.758630+0000","last_peered":"2024-09-19T08:03:00.758630+0000","last_clean":"2024-09-19T08:03:00.758630+0000","last_became_active":"2024-09-19T08:01:10.771533+0000","last_became_peered":"2024-09-19T08:01:10.771533+0000","last_unstale":"2024-09-19T08:03:00.758630+0000","last_undegraded":"2024-09-19T08:03:00.758630+0000","last_fullsized":"2024-09-19T08:03:00.758630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:29:35.180763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841267+0000","last_change":"2024-09-19T08:01:08.763869+0000","last_active":"2024-09-19T08:03:00.841267+0000","last_peered":"2024-09-19T08:03:00.841267+0000","last_clean":"2024-09-19T08:03:00.841267+0000","last_became_active":"2024-09-19T08:01:08.763451+0000","last_became_peered":"2024-09-19T08:01:08.763451+0000","last_unstale":"2024-09-19T08:03:00.841267+0000","last_undegraded":"2024-09-19T08:03:00.841267+0000","last_fullsized":"2024-09-19T08:03:00.841267+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:32:40.901049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838028+0000","last_change":"2024-09-19T08:01:10.770823+0000","last_active":"2024-09-19T08:03:00.838028+0000","last_peered":"2024-09-19T08:03:00.838028+0000","last_clean":"2024-09-19T08:03:00.838028+0000","last_became_active":"2024-09-19T08:01:10.770462+0000","last_became_peered":"2024-09-19T08:01:10.770462+0000","last_unstale":"2024-09-19T08:03:00.838028+0000","last_undegraded":"2024-09-19T08:03:00.838028+0000","last_fullsized":"2024-09-19T08:03:00.838028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:04:32.832009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'7","reported_seq":134,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.993691+0000","last_change":"2024-09-19T08:01:08.759429+0000","last_active":"2024-09-19T08:03:00.993691+0000","last_peered":"2024-09-19T08:03:00.993691+0000","last_clean":"2024-09-19T08:03:00.993691+0000","last_became_active":"2024-09-19T08:01:08.759062+0000","last_became_peered":"2024-09-19T08:01:08.759062+0000","last_unstale":"2024-09-19T08:03:00.993691+0000","last_undegraded":"2024-09-19T08:03:00.993691+0000","last_fullsized":"2024-09-19T08:03:00.993691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:16:51.582435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1374,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":3,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837660+0000","last_change":"2024-09-19T08:01:10.784735+0000","last_active":"2024-09-19T08:03:00.837660+0000","last_peered":"2024-09-19T08:03:00.837660+0000","last_clean":"2024-09-19T08:03:00.837660+0000","last_became_active":"2024-09-19T08:01:10.784346+0000","last_became_peered":"2024-09-19T08:01:10.784346+0000","last_unstale":"2024-09-19T08:03:00.837660+0000","last_undegraded":"2024-09-19T08:03:00.837660+0000","last_fullsized":"2024-09-19T08:03:00.837660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:27:54.580391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847399+0000","last_change":"2024-09-19T08:01:09.749498+0000","last_active":"2024-09-19T08:03:00.847399+0000","last_peered":"2024-09-19T08:03:00.847399+0000","last_clean":"2024-09-19T08:03:00.847399+0000","last_became_active":"2024-09-19T08:01:09.749091+0000","last_became_peered":"2024-09-19T08:01:09.749091+0000","last_unstale":"2024-09-19T08:03:00.847399+0000","last_undegraded":"2024-09-19T08:03:00.847399+0000","last_fullsized":"2024-09-19T08:03:00.847399+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:56:13.644931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841217+0000","last_change":"2024-09-19T08:01:10.779599+0000","last_active":"2024-09-19T08:03:00.841217+0000","last_peered":"2024-09-19T08:03:00.841217+0000","last_clean":"2024-09-19T08:03:00.841217+0000","last_became_active":"2024-09-19T08:01:10.779219+0000","last_became_peered":"2024-09-19T08:01:10.779219+0000","last_unstale":"2024-09-19T08:03:00.841217+0000","last_undegraded":"2024-09-19T08:03:00.841217+0000","last_fullsized":"2024-09-19T08:03:00.841217+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:20.193753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847308+0000","last_change":"2024-09-19T08:01:08.761432+0000","last_active":"2024-09-19T08:03:00.847308+0000","last_peered":"2024-09-19T08:03:00.847308+0000","last_clean":"2024-09-19T08:03:00.847308+0000","last_became_active":"2024-09-19T08:01:08.761044+0000","last_became_peered":"2024-09-19T08:01:08.761044+0000","last_unstale":"2024-09-19T08:03:00.847308+0000","last_undegraded":"2024-09-19T08:03:00.847308+0000","last_fullsized":"2024-09-19T08:03:00.847308+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:12.237334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780765+0000","last_change":"2024-09-19T08:01:10.772559+0000","last_active":"2024-09-19T08:03:00.780765+0000","last_peered":"2024-09-19T08:03:00.780765+0000","last_clean":"2024-09-19T08:03:00.780765+0000","last_became_active":"2024-09-19T08:01:10.772176+0000","last_became_peered":"2024-09-19T08:01:10.772176+0000","last_unstale":"2024-09-19T08:03:00.780765+0000","last_undegraded":"2024-09-19T08:03:00.780765+0000","last_fullsized":"2024-09-19T08:03:00.780765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:27:16.296127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758996+0000","last_change":"2024-09-19T08:01:08.759989+0000","last_active":"2024-09-19T08:03:00.758996+0000","last_peered":"2024-09-19T08:03:00.758996+0000","last_clean":"2024-09-19T08:03:00.758996+0000","last_became_active":"2024-09-19T08:01:08.759705+0000","last_became_peered":"2024-09-19T08:01:08.759705+0000","last_unstale":"2024-09-19T08:03:00.758996+0000","last_undegraded":"2024-09-19T08:03:00.758996+0000","last_fullsized":"2024-09-19T08:03:00.758996+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:33:25.237567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838150+0000","last_change":"2024-09-19T08:01:10.776528+0000","last_active":"2024-09-19T08:03:00.838150+0000","last_peered":"2024-09-19T08:03:00.838150+0000","last_clean":"2024-09-19T08:03:00.838150+0000","last_became_active":"2024-09-19T08:01:10.776222+0000","last_became_peered":"2024-09-19T08:01:10.776222+0000","last_unstale":"2024-09-19T08:03:00.838150+0000","last_undegraded":"2024-09-19T08:03:00.838150+0000","last_fullsized":"2024-09-19T08:03:00.838150+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.787492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'41","reported_seq":73,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.957555+0000","last_change":"2024-09-19T08:01:08.759631+0000","last_active":"2024-09-19T08:03:00.957555+0000","last_peered":"2024-09-19T08:03:00.957555+0000","last_clean":"2024-09-19T08:03:00.957555+0000","last_became_active":"2024-09-19T08:01:08.759156+0000","last_became_peered":"2024-09-19T08:01:08.759156+0000","last_unstale":"2024-09-19T08:03:00.957555+0000","last_undegraded":"2024-09-19T08:03:00.957555+0000","last_fullsized":"2024-09-19T08:03:00.957555+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:48:50.910824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837969+0000","last_change":"2024-09-19T08:01:10.772660+0000","last_active":"2024-09-19T08:03:00.837969+0000","last_peered":"2024-09-19T08:03:00.837969+0000","last_clean":"2024-09-19T08:03:00.837969+0000","last_became_active":"2024-09-19T08:01:10.771816+0000","last_became_peered":"2024-09-19T08:01:10.771816+0000","last_unstale":"2024-09-19T08:03:00.837969+0000","last_undegraded":"2024-09-19T08:03:00.837969+0000","last_fullsized":"2024-09-19T08:03:00.837969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:04:32.667107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'7","reported_seq":134,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760174+0000","last_change":"2024-09-19T08:01:09.749844+0000","last_active":"2024-09-19T08:03:00.760174+0000","last_peered":"2024-09-19T08:03:00.760174+0000","last_clean":"2024-09-19T08:03:00.760174+0000","last_became_active":"2024-09-19T08:01:09.749410+0000","last_became_peered":"2024-09-19T08:01:09.749410+0000","last_unstale":"2024-09-19T08:03:00.760174+0000","last_undegraded":"2024-09-19T08:03:00.760174+0000","last_fullsized":"2024-09-19T08:03:00.760174+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:03:14.989814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1704,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838196+0000","last_change":"2024-09-19T08:01:10.773782+0000","last_active":"2024-09-19T08:03:00.838196+0000","last_peered":"2024-09-19T08:03:00.838196+0000","last_clean":"2024-09-19T08:03:00.838196+0000","last_became_active":"2024-09-19T08:01:10.773471+0000","last_became_peered":"2024-09-19T08:01:10.773471+0000","last_unstale":"2024-09-19T08:03:00.838196+0000","last_undegraded":"2024-09-19T08:03:00.838196+0000","last_fullsized":"2024-09-19T08:03:00.838196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:15.780259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":344,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837815+0000","last_change":"2024-09-19T08:00:01.677517+0000","last_active":"2024-09-19T08:03:00.837815+0000","last_peered":"2024-09-19T08:03:00.837815+0000","last_clean":"2024-09-19T08:03:00.837815+0000","last_became_active":"2024-09-19T08:00:01.677089+0000","last_became_peered":"2024-09-19T08:00:01.677089+0000","last_unstale":"2024-09-19T08:03:00.837815+0000","last_undegraded":"2024-09-19T08:03:00.837815+0000","last_fullsized":"2024-09-19T08:03:00.837815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_clean_scrub_stamp":"2024-09-19T07:56:43.962142+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:10:36.158811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761731+0000","last_change":"2024-09-19T08:01:08.759819+0000","last_active":"2024-09-19T08:03:00.761731+0000","last_peered":"2024-09-19T08:03:00.761731+0000","last_clean":"2024-09-19T08:03:00.761731+0000","last_became_active":"2024-09-19T08:01:08.759420+0000","last_became_peered":"2024-09-19T08:01:08.759420+0000","last_unstale":"2024-09-19T08:03:00.761731+0000","last_undegraded":"2024-09-19T08:03:00.761731+0000","last_fullsized":"2024-09-19T08:03:00.761731+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:06:16.852323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.843903+0000","last_change":"2024-09-19T08:01:26.724260+0000","last_active":"2024-09-19T08:03:00.843903+0000","last_peered":"2024-09-19T08:03:00.843903+0000","last_clean":"2024-09-19T08:03:00.843903+0000","last_became_active":"2024-09-19T08:01:26.723851+0000","last_became_peered":"2024-09-19T08:01:26.723851+0000","last_unstale":"2024-09-19T08:03:00.843903+0000","last_undegraded":"2024-09-19T08:03:00.843903+0000","last_fullsized":"2024-09-19T08:03:00.843903+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:18:46.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837812+0000","last_change":"2024-09-19T08:01:08.761291+0000","last_active":"2024-09-19T08:03:00.837812+0000","last_peered":"2024-09-19T08:03:00.837812+0000","last_clean":"2024-09-19T08:03:00.837812+0000","last_became_active":"2024-09-19T08:01:08.760238+0000","last_became_peered":"2024-09-19T08:01:08.760238+0000","last_unstale":"2024-09-19T08:03:00.837812+0000","last_undegraded":"2024-09-19T08:03:00.837812+0000","last_fullsized":"2024-09-19T08:03:00.837812+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:26:09.980573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758579+0000","last_change":"2024-09-19T08:01:10.773056+0000","last_active":"2024-09-19T08:03:00.758579+0000","last_peered":"2024-09-19T08:03:00.758579+0000","last_clean":"2024-09-19T08:03:00.758579+0000","last_became_active":"2024-09-19T08:01:10.772431+0000","last_became_peered":"2024-09-19T08:01:10.772431+0000","last_unstale":"2024-09-19T08:03:00.758579+0000","last_undegraded":"2024-09-19T08:03:00.758579+0000","last_fullsized":"2024-09-19T08:03:00.758579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:39:06.878086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":132,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.842523+0000","last_change":"2024-09-19T08:01:09.749707+0000","last_active":"2024-09-19T08:03:00.842523+0000","last_peered":"2024-09-19T08:03:00.842523+0000","last_clean":"2024-09-19T08:03:00.842523+0000","last_became_active":"2024-09-19T08:01:09.749013+0000","last_became_peered":"2024-09-19T08:01:09.749013+0000","last_unstale":"2024-09-19T08:03:00.842523+0000","last_undegraded":"2024-09-19T08:03:00.842523+0000","last_fullsized":"2024-09-19T08:03:00.842523+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:40:12.682158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841187+0000","last_change":"2024-09-19T08:01:10.774101+0000","last_active":"2024-09-19T08:03:00.841187+0000","last_peered":"2024-09-19T08:03:00.841187+0000","last_clean":"2024-09-19T08:03:00.841187+0000","last_became_active":"2024-09-19T08:01:10.773822+0000","last_became_peered":"2024-09-19T08:01:10.773822+0000","last_unstale":"2024-09-19T08:03:00.841187+0000","last_undegraded":"2024-09-19T08:03:00.841187+0000","last_fullsized":"2024-09-19T08:03:00.841187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:29.571118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"78'5","reported_seq":129,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837640+0000","last_change":"2024-09-19T08:01:08.761675+0000","last_active":"2024-09-19T08:03:00.837640+0000","last_peered":"2024-09-19T08:03:00.837640+0000","last_clean":"2024-09-19T08:03:00.837640+0000","last_became_active":"2024-09-19T08:01:08.761346+0000","last_became_peered":"2024-09-19T08:01:08.761346+0000","last_unstale":"2024-09-19T08:03:00.837640+0000","last_undegraded":"2024-09-19T08:03:00.837640+0000","last_fullsized":"2024-09-19T08:03:00.837640+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:51:19.237458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839233+0000","last_change":"2024-09-19T08:01:10.781565+0000","last_active":"2024-09-19T08:03:00.839233+0000","last_peered":"2024-09-19T08:03:00.839233+0000","last_clean":"2024-09-19T08:03:00.839233+0000","last_became_active":"2024-09-19T08:01:10.781301+0000","last_became_peered":"2024-09-19T08:01:10.781301+0000","last_unstale":"2024-09-19T08:03:00.839233+0000","last_undegraded":"2024-09-19T08:03:00.839233+0000","last_fullsized":"2024-09-19T08:03:00.839233+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:55:52.932009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844577+0000","last_change":"2024-09-19T08:01:09.748918+0000","last_active":"2024-09-19T08:03:00.844577+0000","last_peered":"2024-09-19T08:03:00.844577+0000","last_clean":"2024-09-19T08:03:00.844577+0000","last_became_active":"2024-09-19T08:01:09.748497+0000","last_became_peered":"2024-09-19T08:01:09.748497+0000","last_unstale":"2024-09-19T08:03:00.844577+0000","last_undegraded":"2024-09-19T08:03:00.844577+0000","last_fullsized":"2024-09-19T08:03:00.844577+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:20:31.030133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839231+0000","last_change":"2024-09-19T08:01:10.781388+0000","last_active":"2024-09-19T08:03:00.839231+0000","last_peered":"2024-09-19T08:03:00.839231+0000","last_clean":"2024-09-19T08:03:00.839231+0000","last_became_active":"2024-09-19T08:01:10.781063+0000","last_became_peered":"2024-09-19T08:01:10.781063+0000","last_unstale":"2024-09-19T08:03:00.839231+0000","last_undegraded":"2024-09-19T08:03:00.839231+0000","last_fullsized":"2024-09-19T08:03:00.839231+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:12:08.852717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847755+0000","last_change":"2024-09-19T08:01:09.750080+0000","last_active":"2024-09-19T08:03:00.847755+0000","last_peered":"2024-09-19T08:03:00.847755+0000","last_clean":"2024-09-19T08:03:00.847755+0000","last_became_active":"2024-09-19T08:01:09.749053+0000","last_became_peered":"2024-09-19T08:01:09.749053+0000","last_unstale":"2024-09-19T08:03:00.847755+0000","last_undegraded":"2024-09-19T08:03:00.847755+0000","last_fullsized":"2024-09-19T08:03:00.847755+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:15:45.292714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759144+0000","last_change":"2024-09-19T08:01:10.767070+0000","last_active":"2024-09-19T08:03:00.759144+0000","last_peered":"2024-09-19T08:03:00.759144+0000","last_clean":"2024-09-19T08:03:00.759144+0000","last_became_active":"2024-09-19T08:01:10.766760+0000","last_became_peered":"2024-09-19T08:01:10.766760+0000","last_unstale":"2024-09-19T08:03:00.759144+0000","last_undegraded":"2024-09-19T08:03:00.759144+0000","last_fullsized":"2024-09-19T08:03:00.759144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:17.841135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'51","reported_seq":98,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.960465+0000","last_change":"2024-09-19T08:01:08.756143+0000","last_active":"2024-09-19T08:03:00.960465+0000","last_peered":"2024-09-19T08:03:00.960465+0000","last_clean":"2024-09-19T08:03:00.960465+0000","last_became_active":"2024-09-19T08:01:08.755777+0000","last_became_peered":"2024-09-19T08:01:08.755777+0000","last_unstale":"2024-09-19T08:03:00.960465+0000","last_undegraded":"2024-09-19T08:03:00.960465+0000","last_fullsized":"2024-09-19T08:03:00.960465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:21.430673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":90,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849237+0000","last_change":"2024-09-19T08:01:10.774140+0000","last_active":"2024-09-19T08:03:00.849237+0000","last_peered":"2024-09-19T08:03:00.849237+0000","last_clean":"2024-09-19T08:03:00.849237+0000","last_became_active":"2024-09-19T08:01:10.773819+0000","last_became_peered":"2024-09-19T08:01:10.773819+0000","last_unstale":"2024-09-19T08:03:00.849237+0000","last_undegraded":"2024-09-19T08:03:00.849237+0000","last_fullsized":"2024-09-19T08:03:00.849237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:15:13.174013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761607+0000","last_change":"2024-09-19T08:01:09.169129+0000","last_active":"2024-09-19T08:03:00.761607+0000","last_peered":"2024-09-19T08:03:00.761607+0000","last_clean":"2024-09-19T08:03:00.761607+0000","last_became_active":"2024-09-19T08:01:09.168734+0000","last_became_peered":"2024-09-19T08:01:09.168734+0000","last_unstale":"2024-09-19T08:03:00.761607+0000","last_undegraded":"2024-09-19T08:03:00.761607+0000","last_fullsized":"2024-09-19T08:03:00.761607+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:01:52.232391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994759+0000","last_change":"2024-09-19T08:01:10.768592+0000","last_active":"2024-09-19T08:03:00.994759+0000","last_peered":"2024-09-19T08:03:00.994759+0000","last_clean":"2024-09-19T08:03:00.994759+0000","last_became_active":"2024-09-19T08:01:10.767645+0000","last_became_peered":"2024-09-19T08:01:10.767645+0000","last_unstale":"2024-09-19T08:03:00.994759+0000","last_undegraded":"2024-09-19T08:03:00.994759+0000","last_fullsized":"2024-09-19T08:03:00.994759+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:08:46.227461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997364+0000","last_change":"2024-09-19T08:01:09.749240+0000","last_active":"2024-09-19T08:03:00.997364+0000","last_peered":"2024-09-19T08:03:00.997364+0000","last_clean":"2024-09-19T08:03:00.997364+0000","last_became_active":"2024-09-19T08:01:09.748945+0000","last_became_peered":"2024-09-19T08:01:09.748945+0000","last_unstale":"2024-09-19T08:03:00.997364+0000","last_undegraded":"2024-09-19T08:03:00.997364+0000","last_fullsized":"2024-09-19T08:03:00.997364+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:25:13.791290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847326+0000","last_change":"2024-09-19T08:01:10.783133+0000","last_active":"2024-09-19T08:03:00.847326+0000","last_peered":"2024-09-19T08:03:00.847326+0000","last_clean":"2024-09-19T08:03:00.847326+0000","last_became_active":"2024-09-19T08:01:10.782797+0000","last_became_peered":"2024-09-19T08:01:10.782797+0000","last_unstale":"2024-09-19T08:03:00.847326+0000","last_undegraded":"2024-09-19T08:03:00.847326+0000","last_fullsized":"2024-09-19T08:03:00.847326+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:30:43.551877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848489+0000","last_change":"2024-09-19T08:01:08.757964+0000","last_active":"2024-09-19T08:03:00.848489+0000","last_peered":"2024-09-19T08:03:00.848489+0000","last_clean":"2024-09-19T08:03:00.848489+0000","last_became_active":"2024-09-19T08:01:08.757526+0000","last_became_peered":"2024-09-19T08:01:08.757526+0000","last_unstale":"2024-09-19T08:03:00.848489+0000","last_undegraded":"2024-09-19T08:03:00.848489+0000","last_fullsized":"2024-09-19T08:03:00.848489+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:03.565702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780557+0000","last_change":"2024-09-19T08:01:10.769974+0000","last_active":"2024-09-19T08:03:00.780557+0000","last_peered":"2024-09-19T08:03:00.780557+0000","last_clean":"2024-09-19T08:03:00.780557+0000","last_became_active":"2024-09-19T08:01:10.769650+0000","last_became_peered":"2024-09-19T08:01:10.769650+0000","last_unstale":"2024-09-19T08:03:00.780557+0000","last_undegraded":"2024-09-19T08:03:00.780557+0000","last_fullsized":"2024-09-19T08:03:00.780557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:01.755198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"78'3","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759999+0000","last_change":"2024-09-19T08:01:08.763476+0000","last_active":"2024-09-19T08:03:00.759999+0000","last_peered":"2024-09-19T08:03:00.759999+0000","last_clean":"2024-09-19T08:03:00.759999+0000","last_became_active":"2024-09-19T08:01:08.763154+0000","last_became_peered":"2024-09-19T08:01:08.763154+0000","last_unstale":"2024-09-19T08:03:00.759999+0000","last_undegraded":"2024-09-19T08:03:00.759999+0000","last_fullsized":"2024-09-19T08:03:00.759999+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:06.202452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848751+0000","last_change":"2024-09-19T08:01:10.784201+0000","last_active":"2024-09-19T08:03:00.848751+0000","last_peered":"2024-09-19T08:03:00.848751+0000","last_clean":"2024-09-19T08:03:00.848751+0000","last_became_active":"2024-09-19T08:01:10.783867+0000","last_became_peered":"2024-09-19T08:01:10.783867+0000","last_unstale":"2024-09-19T08:03:00.848751+0000","last_undegraded":"2024-09-19T08:03:00.848751+0000","last_fullsized":"2024-09-19T08:03:00.848751+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:43:50.082183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847290+0000","last_change":"2024-09-19T08:01:08.760479+0000","last_active":"2024-09-19T08:03:00.847290+0000","last_peered":"2024-09-19T08:03:00.847290+0000","last_clean":"2024-09-19T08:03:00.847290+0000","last_became_active":"2024-09-19T08:01:08.760004+0000","last_became_peered":"2024-09-19T08:01:08.760004+0000","last_unstale":"2024-09-19T08:03:00.847290+0000","last_undegraded":"2024-09-19T08:03:00.847290+0000","last_fullsized":"2024-09-19T08:03:00.847290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:41:19.956623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837851+0000","last_change":"2024-09-19T08:01:10.772184+0000","last_active":"2024-09-19T08:03:00.837851+0000","last_peered":"2024-09-19T08:03:00.837851+0000","last_clean":"2024-09-19T08:03:00.837851+0000","last_became_active":"2024-09-19T08:01:10.771825+0000","last_became_peered":"2024-09-19T08:01:10.771825+0000","last_unstale":"2024-09-19T08:03:00.837851+0000","last_undegraded":"2024-09-19T08:03:00.837851+0000","last_fullsized":"2024-09-19T08:03:00.837851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:36.764575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841567+0000","last_change":"2024-09-19T08:01:10.779368+0000","last_active":"2024-09-19T08:03:00.841567+0000","last_peered":"2024-09-19T08:03:00.841567+0000","last_clean":"2024-09-19T08:03:00.841567+0000","last_became_active":"2024-09-19T08:01:10.778884+0000","last_became_peered":"2024-09-19T08:01:10.778884+0000","last_unstale":"2024-09-19T08:03:00.841567+0000","last_undegraded":"2024-09-19T08:03:00.841567+0000","last_fullsized":"2024-09-19T08:03:00.841567+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:28:29.630300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780536+0000","last_change":"2024-09-19T08:01:26.741053+0000","last_active":"2024-09-19T08:03:00.780536+0000","last_peered":"2024-09-19T08:03:00.780536+0000","last_clean":"2024-09-19T08:03:00.780536+0000","last_became_active":"2024-09-19T08:01:26.724714+0000","last_became_peered":"2024-09-19T08:01:26.724714+0000","last_unstale":"2024-09-19T08:03:00.780536+0000","last_undegraded":"2024-09-19T08:03:00.780536+0000","last_fullsized":"2024-09-19T08:03:00.780536+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:59.583797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997323+0000","last_change":"2024-09-19T08:01:10.781389+0000","last_active":"2024-09-19T08:03:00.997323+0000","last_peered":"2024-09-19T08:03:00.997323+0000","last_clean":"2024-09-19T08:03:00.997323+0000","last_became_active":"2024-09-19T08:01:10.781080+0000","last_became_peered":"2024-09-19T08:01:10.781080+0000","last_unstale":"2024-09-19T08:03:00.997323+0000","last_undegraded":"2024-09-19T08:03:00.997323+0000","last_fullsized":"2024-09-19T08:03:00.997323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:40:45.225957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847701+0000","last_change":"2024-09-19T08:01:26.768102+0000","last_active":"2024-09-19T08:03:00.847701+0000","last_peered":"2024-09-19T08:03:00.847701+0000","last_clean":"2024-09-19T08:03:00.847701+0000","last_became_active":"2024-09-19T08:01:26.759462+0000","last_became_peered":"2024-09-19T08:01:26.759462+0000","last_unstale":"2024-09-19T08:03:00.847701+0000","last_undegraded":"2024-09-19T08:03:00.847701+0000","last_fullsized":"2024-09-19T08:03:00.847701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:02:06.815238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994921+0000","last_change":"2024-09-19T08:01:10.771031+0000","last_active":"2024-09-19T08:03:00.994921+0000","last_peered":"2024-09-19T08:03:00.994921+0000","last_clean":"2024-09-19T08:03:00.994921+0000","last_became_active":"2024-09-19T08:01:10.770675+0000","last_became_peered":"2024-09-19T08:01:10.770675+0000","last_unstale":"2024-09-19T08:03:00.994921+0000","last_undegraded":"2024-09-19T08:03:00.994921+0000","last_fullsized":"2024-09-19T08:03:00.994921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:42.203848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849208+0000","last_change":"2024-09-19T08:01:09.748622+0000","last_active":"2024-09-19T08:03:00.849208+0000","last_peered":"2024-09-19T08:03:00.849208+0000","last_clean":"2024-09-19T08:03:00.849208+0000","last_became_active":"2024-09-19T08:01:09.748202+0000","last_became_peered":"2024-09-19T08:01:09.748202+0000","last_unstale":"2024-09-19T08:03:00.849208+0000","last_undegraded":"2024-09-19T08:03:00.849208+0000","last_fullsized":"2024-09-19T08:03:00.849208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:17:39.861820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849073+0000","last_change":"2024-09-19T08:01:10.768949+0000","last_active":"2024-09-19T08:03:00.849073+0000","last_peered":"2024-09-19T08:03:00.849073+0000","last_clean":"2024-09-19T08:03:00.849073+0000","last_became_active":"2024-09-19T08:01:10.768159+0000","last_became_peered":"2024-09-19T08:01:10.768159+0000","last_unstale":"2024-09-19T08:03:00.849073+0000","last_undegraded":"2024-09-19T08:03:00.849073+0000","last_fullsized":"2024-09-19T08:03:00.849073+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:43.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"78'13","reported_seq":163,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844272+0000","last_change":"2024-09-19T08:01:26.735877+0000","last_active":"2024-09-19T08:03:00.844272+0000","last_peered":"2024-09-19T08:03:00.844272+0000","last_clean":"2024-09-19T08:03:00.844272+0000","last_became_active":"2024-09-19T08:01:26.723597+0000","last_became_peered":"2024-09-19T08:01:26.723597+0000","last_unstale":"2024-09-19T08:03:00.844272+0000","last_undegraded":"2024-09-19T08:03:00.844272+0000","last_fullsized":"2024-09-19T08:03:00.844272+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:43:42.340832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":99,"num_read_kb":99,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,1],"acting":[9,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600666,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1139,"num_read_kb":683,"num_write":372,"num_write_kb":1463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1254,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1801998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645347,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":905216,"data_stored":436392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874280,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32940,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712212,"statfs":{"total":95995035648,"available":95961305088,"internally_reserved":0,"allocated":1736704,"data_stored":1224499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005101,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32404,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712748,"statfs":{"total":95995035648,"available":95961853952,"internally_reserved":0,"allocated":1245184,"data_stored":762214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234033,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1294336,"data_stored":826874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462966,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1503232,"data_stored":1028781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593788,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36228,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708924,"statfs":{"total":95995035648,"available":95957938176,"internally_reserved":0,"allocated":901120,"data_stored":434251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822720,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1658880,"data_stored":1159945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986245,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":901120,"data_stored":434112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215177,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32656,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712496,"statfs":{"total":95995035648,"available":95961595904,"internally_reserved":0,"allocated":1445888,"data_stored":958892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476815,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":504026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607636,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":978944,"data_stored":502316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":921600,"data_stored":436124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":136048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":200602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T08:03:05.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T08:03:05.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.335+0000 7f3fc6368640 1 -- 172.21.15.98:0/433233171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc01069b0 msgr2=0x7f3fc0106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.335+0000 7f3fc6368640 1 --2- 172.21.15.98:0/433233171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc01069b0 0x7f3fc0106d90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0009930 tx=0x7f3fb002a420 comp rx=0 tx=0).stop 2024-09-19T08:03:05.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.335+0000 7f3fc6368640 1 -- 172.21.15.98:0/433233171 shutdown_connections 2024-09-19T08:03:05.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.335+0000 7f3fc6368640 1 --2- 172.21.15.98:0/433233171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fc0101340 0x7f3fc010d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.335+0000 7f3fc6368640 1 --2- 172.21.15.98:0/433233171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3fc01009a0 0x7f3fc0100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.335+0000 7f3fc6368640 1 --2- 172.21.15.98:0/433233171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc01069b0 0x7f3fc0106d90 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.335+0000 7f3fc6368640 1 -- 172.21.15.98:0/433233171 >> 172.21.15.98:0/433233171 conn(0x7f3fc00fc6c0 msgr2=0x7f3fc00feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:05.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.335+0000 7f3fc6368640 1 -- 172.21.15.98:0/433233171 shutdown_connections 2024-09-19T08:03:05.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.335+0000 7f3fc6368640 1 -- 172.21.15.98:0/433233171 wait complete. 2024-09-19T08:03:05.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.335+0000 7f3fc6368640 1 Processor -- start 2024-09-19T08:03:05.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.335+0000 7f3fc6368640 1 -- start start 2024-09-19T08:03:05.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc6368640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fc01009a0 0x7f3fc0198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:05.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc6368640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3fc0101340 0x7f3fc0198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:05.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc6368640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc01069b0 0x7f3fc019cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:05.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc6368640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc0110df0 con 0x7f3fc01069b0 2024-09-19T08:03:05.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc6368640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc0110c70 con 0x7f3fc0101340 2024-09-19T08:03:05.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc6368640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc0110f70 con 0x7f3fc01009a0 2024-09-19T08:03:05.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc4905640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc01069b0 0x7f3fc019cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:05.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc4905640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc01069b0 0x7f3fc019cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55158/0 (socket says 172.21.15.98:55158) 2024-09-19T08:03:05.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc4905640 1 -- 172.21.15.98:0/484040664 learned_addr learned my addr 172.21.15.98:0/484040664 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:05.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fbffff640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fc01009a0 0x7f3fc0198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:05.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fbf7fe640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3fc0101340 0x7f3fc0198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:05.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc4905640 1 -- 172.21.15.98:0/484040664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fc01009a0 msgr2=0x7f3fc0198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc4905640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fc01009a0 0x7f3fc0198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc4905640 1 -- 172.21.15.98:0/484040664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3fc0101340 msgr2=0x7f3fc0198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc4905640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3fc0101340 0x7f3fc0198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc4905640 1 -- 172.21.15.98:0/484040664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fc019d330 con 0x7f3fc01069b0 2024-09-19T08:03:05.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fbffff640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fc01009a0 0x7f3fc0198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:05.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fbf7fe640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3fc0101340 0x7f3fc0198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:05.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.336+0000 7f3fc4905640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc01069b0 0x7f3fc019cc00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800eaa0 tx=0x7f3fb800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:05.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.337+0000 7f3fbd7fa640 1 -- 172.21.15.98:0/484040664 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb800cdb0 con 0x7f3fc01069b0 2024-09-19T08:03:05.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.337+0000 7f3fbd7fa640 1 -- 172.21.15.98:0/484040664 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fb8005020 con 0x7f3fc01069b0 2024-09-19T08:03:05.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.337+0000 7f3fc6368640 1 -- 172.21.15.98:0/484040664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fc019d620 con 0x7f3fc01069b0 2024-09-19T08:03:05.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.337+0000 7f3fbd7fa640 1 -- 172.21.15.98:0/484040664 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb80116b0 con 0x7f3fc01069b0 2024-09-19T08:03:05.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.337+0000 7f3fc6368640 1 -- 172.21.15.98:0/484040664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fc01a4eb0 con 0x7f3fc01069b0 2024-09-19T08:03:05.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.338+0000 7f3fc6368640 1 -- 172.21.15.98:0/484040664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fc0100e00 con 0x7f3fc01069b0 2024-09-19T08:03:05.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.339+0000 7f3fbd7fa640 1 -- 172.21.15.98:0/484040664 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 21) ==== 138526+0+0 (secure 0 0 0) 0x7f3fb8005560 con 0x7f3fc01069b0 2024-09-19T08:03:05.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.339+0000 7f3fbd7fa640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3fa80a3450 0x7f3fa80a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:05.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.339+0000 7f3fbffff640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3fa80a3450 0x7f3fa80a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:05.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.339+0000 7f3fbd7fa640 1 -- 172.21.15.98:0/484040664 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f3fb80d0110 con 0x7f3fc01069b0 2024-09-19T08:03:05.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.340+0000 7f3fbffff640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3fa80a3450 0x7f3fa80a5910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0000c00 tx=0x7f3fb0005a20 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:05.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.344+0000 7f3fbd7fa640 1 -- 172.21.15.98:0/484040664 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f3fb8016030 con 0x7f3fc01069b0 2024-09-19T08:03:05.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.509+0000 7f3fc6368640 1 -- 172.21.15.98:0/484040664 --> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3fc006a3c0 con 0x7f3fa80a3450 2024-09-19T08:03:05.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.515+0000 7f3fbd7fa640 1 -- 172.21.15.98:0/484040664 <== mgr.15252 v2:172.21.15.98:6800/1961446727 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257946 (secure 0 0 0) 0x7f3fc006a3c0 con 0x7f3fa80a3450 2024-09-19T08:03:05.519 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:05.520 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-19T08:03:05.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.519+0000 7f3fc6368640 1 -- 172.21.15.98:0/484040664 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3fa80a3450 msgr2=0x7f3fa80a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.519+0000 7f3fc6368640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3fa80a3450 0x7f3fa80a5910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0000c00 tx=0x7f3fb0005a20 comp rx=0 tx=0).stop 2024-09-19T08:03:05.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.519+0000 7f3fc6368640 1 -- 172.21.15.98:0/484040664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc01069b0 msgr2=0x7f3fc019cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.519+0000 7f3fc6368640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc01069b0 0x7f3fc019cc00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800eaa0 tx=0x7f3fb800ef60 comp rx=0 tx=0).stop 2024-09-19T08:03:05.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.519+0000 7f3fc6368640 1 -- 172.21.15.98:0/484040664 shutdown_connections 2024-09-19T08:03:05.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.519+0000 7f3fc6368640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3fa80a3450 0x7f3fa80a5910 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.519+0000 7f3fc6368640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc01069b0 0x7f3fc019cc00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.519+0000 7f3fc6368640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3fc0101340 0x7f3fc0198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.519+0000 7f3fc6368640 1 --2- 172.21.15.98:0/484040664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fc01009a0 0x7f3fc0198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.519+0000 7f3fc6368640 1 -- 172.21.15.98:0/484040664 >> 172.21.15.98:0/484040664 conn(0x7f3fc00fc6c0 msgr2=0x7f3fc0104710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:05.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.519+0000 7f3fc6368640 1 -- 172.21.15.98:0/484040664 shutdown_connections 2024-09-19T08:03:05.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.519+0000 7f3fc6368640 1 -- 172.21.15.98:0/484040664 wait complete. 2024-09-19T08:03:05.532 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-09-19T08:03:04.771716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191268,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1275,"num_read_kb":802,"num_write":555,"num_write_kb":4403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181990,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":939,"ondisk_log_size":939,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376028,"kb_used_data":14144,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565796,"statfs":{"total":1151940427776,"available":1151555375104,"internally_reserved":0,"allocated":14483456,"data_stored":8708426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":370193724},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.989911"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780911+0000","last_change":"2024-09-19T08:01:10.768950+0000","last_active":"2024-09-19T08:03:00.780911+0000","last_peered":"2024-09-19T08:03:00.780911+0000","last_clean":"2024-09-19T08:03:00.780911+0000","last_became_active":"2024-09-19T08:01:10.768047+0000","last_became_peered":"2024-09-19T08:01:10.768047+0000","last_unstale":"2024-09-19T08:03:00.780911+0000","last_undegraded":"2024-09-19T08:03:00.780911+0000","last_fullsized":"2024-09-19T08:03:00.780911+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:26.219079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994728+0000","last_change":"2024-09-19T08:01:10.783208+0000","last_active":"2024-09-19T08:03:00.994728+0000","last_peered":"2024-09-19T08:03:00.994728+0000","last_clean":"2024-09-19T08:03:00.994728+0000","last_became_active":"2024-09-19T08:01:10.782888+0000","last_became_peered":"2024-09-19T08:01:10.782888+0000","last_unstale":"2024-09-19T08:03:00.994728+0000","last_undegraded":"2024-09-19T08:03:00.994728+0000","last_fullsized":"2024-09-19T08:03:00.994728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:51:27.603166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994427+0000","last_change":"2024-09-19T08:01:10.770395+0000","last_active":"2024-09-19T08:03:00.994427+0000","last_peered":"2024-09-19T08:03:00.994427+0000","last_clean":"2024-09-19T08:03:00.994427+0000","last_became_active":"2024-09-19T08:01:10.770073+0000","last_became_peered":"2024-09-19T08:01:10.770073+0000","last_unstale":"2024-09-19T08:03:00.994427+0000","last_undegraded":"2024-09-19T08:03:00.994427+0000","last_fullsized":"2024-09-19T08:03:00.994427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:20.047023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761373+0000","last_change":"2024-09-19T08:01:10.777138+0000","last_active":"2024-09-19T08:03:00.761373+0000","last_peered":"2024-09-19T08:03:00.761373+0000","last_clean":"2024-09-19T08:03:00.761373+0000","last_became_active":"2024-09-19T08:01:10.776810+0000","last_became_peered":"2024-09-19T08:01:10.776810+0000","last_unstale":"2024-09-19T08:03:00.761373+0000","last_undegraded":"2024-09-19T08:03:00.761373+0000","last_fullsized":"2024-09-19T08:03:00.761373+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:39.983321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847497+0000","last_change":"2024-09-19T08:01:10.770435+0000","last_active":"2024-09-19T08:03:00.847497+0000","last_peered":"2024-09-19T08:03:00.847497+0000","last_clean":"2024-09-19T08:03:00.847497+0000","last_became_active":"2024-09-19T08:01:10.769686+0000","last_became_peered":"2024-09-19T08:01:10.769686+0000","last_unstale":"2024-09-19T08:03:00.847497+0000","last_undegraded":"2024-09-19T08:03:00.847497+0000","last_fullsized":"2024-09-19T08:03:00.847497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:10:19.610925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839257+0000","last_change":"2024-09-19T08:01:10.782382+0000","last_active":"2024-09-19T08:03:00.839257+0000","last_peered":"2024-09-19T08:03:00.839257+0000","last_clean":"2024-09-19T08:03:00.839257+0000","last_became_active":"2024-09-19T08:01:10.781996+0000","last_became_peered":"2024-09-19T08:01:10.781996+0000","last_unstale":"2024-09-19T08:03:00.839257+0000","last_undegraded":"2024-09-19T08:03:00.839257+0000","last_fullsized":"2024-09-19T08:03:00.839257+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:43:13.574336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837718+0000","last_change":"2024-09-19T08:01:10.773776+0000","last_active":"2024-09-19T08:03:00.837718+0000","last_peered":"2024-09-19T08:03:00.837718+0000","last_clean":"2024-09-19T08:03:00.837718+0000","last_became_active":"2024-09-19T08:01:10.772442+0000","last_became_peered":"2024-09-19T08:01:10.772442+0000","last_unstale":"2024-09-19T08:03:00.837718+0000","last_undegraded":"2024-09-19T08:03:00.837718+0000","last_fullsized":"2024-09-19T08:03:00.837718+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:08:18.155806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848134+0000","last_change":"2024-09-19T08:01:10.781443+0000","last_active":"2024-09-19T08:03:00.848134+0000","last_peered":"2024-09-19T08:03:00.848134+0000","last_clean":"2024-09-19T08:03:00.848134+0000","last_became_active":"2024-09-19T08:01:10.781083+0000","last_became_peered":"2024-09-19T08:01:10.781083+0000","last_unstale":"2024-09-19T08:03:00.848134+0000","last_undegraded":"2024-09-19T08:03:00.848134+0000","last_fullsized":"2024-09-19T08:03:00.848134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:23:22.631865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.998365+0000","last_change":"2024-09-19T08:01:10.767554+0000","last_active":"2024-09-19T08:03:00.998365+0000","last_peered":"2024-09-19T08:03:00.998365+0000","last_clean":"2024-09-19T08:03:00.998365+0000","last_became_active":"2024-09-19T08:01:10.767227+0000","last_became_peered":"2024-09-19T08:01:10.767227+0000","last_unstale":"2024-09-19T08:03:00.998365+0000","last_undegraded":"2024-09-19T08:03:00.998365+0000","last_fullsized":"2024-09-19T08:03:00.998365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:30:13.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839361+0000","last_change":"2024-09-19T08:01:10.783174+0000","last_active":"2024-09-19T08:03:00.839361+0000","last_peered":"2024-09-19T08:03:00.839361+0000","last_clean":"2024-09-19T08:03:00.839361+0000","last_became_active":"2024-09-19T08:01:10.782794+0000","last_became_peered":"2024-09-19T08:01:10.782794+0000","last_unstale":"2024-09-19T08:03:00.839361+0000","last_undegraded":"2024-09-19T08:03:00.839361+0000","last_fullsized":"2024-09-19T08:03:00.839361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:49:19.451107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838191+0000","last_change":"2024-09-19T08:01:10.779638+0000","last_active":"2024-09-19T08:03:00.838191+0000","last_peered":"2024-09-19T08:03:00.838191+0000","last_clean":"2024-09-19T08:03:00.838191+0000","last_became_active":"2024-09-19T08:01:10.779295+0000","last_became_peered":"2024-09-19T08:01:10.779295+0000","last_unstale":"2024-09-19T08:03:00.838191+0000","last_undegraded":"2024-09-19T08:03:00.838191+0000","last_fullsized":"2024-09-19T08:03:00.838191+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:35:10.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761539+0000","last_change":"2024-09-19T08:01:10.784487+0000","last_active":"2024-09-19T08:03:00.761539+0000","last_peered":"2024-09-19T08:03:00.761539+0000","last_clean":"2024-09-19T08:03:00.761539+0000","last_became_active":"2024-09-19T08:01:10.784114+0000","last_became_peered":"2024-09-19T08:01:10.784114+0000","last_unstale":"2024-09-19T08:03:00.761539+0000","last_undegraded":"2024-09-19T08:03:00.761539+0000","last_fullsized":"2024-09-19T08:03:00.761539+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:09:54.819802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758888+0000","last_change":"2024-09-19T08:01:10.783186+0000","last_active":"2024-09-19T08:03:00.758888+0000","last_peered":"2024-09-19T08:03:00.758888+0000","last_clean":"2024-09-19T08:03:00.758888+0000","last_became_active":"2024-09-19T08:01:10.782876+0000","last_became_peered":"2024-09-19T08:01:10.782876+0000","last_unstale":"2024-09-19T08:03:00.758888+0000","last_undegraded":"2024-09-19T08:03:00.758888+0000","last_fullsized":"2024-09-19T08:03:00.758888+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:26:15.252165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994474+0000","last_change":"2024-09-19T08:01:10.783237+0000","last_active":"2024-09-19T08:03:00.994474+0000","last_peered":"2024-09-19T08:03:00.994474+0000","last_clean":"2024-09-19T08:03:00.994474+0000","last_became_active":"2024-09-19T08:01:10.782966+0000","last_became_peered":"2024-09-19T08:01:10.782966+0000","last_unstale":"2024-09-19T08:03:00.994474+0000","last_undegraded":"2024-09-19T08:03:00.994474+0000","last_fullsized":"2024-09-19T08:03:00.994474+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:57:18.771737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994242+0000","last_change":"2024-09-19T08:01:10.771207+0000","last_active":"2024-09-19T08:03:00.994242+0000","last_peered":"2024-09-19T08:03:00.994242+0000","last_clean":"2024-09-19T08:03:00.994242+0000","last_became_active":"2024-09-19T08:01:10.770862+0000","last_became_peered":"2024-09-19T08:01:10.770862+0000","last_unstale":"2024-09-19T08:03:00.994242+0000","last_undegraded":"2024-09-19T08:03:00.994242+0000","last_fullsized":"2024-09-19T08:03:00.994242+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:08:09.313329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997263+0000","last_change":"2024-09-19T08:01:10.775191+0000","last_active":"2024-09-19T08:03:00.997263+0000","last_peered":"2024-09-19T08:03:00.997263+0000","last_clean":"2024-09-19T08:03:00.997263+0000","last_became_active":"2024-09-19T08:01:10.774832+0000","last_became_peered":"2024-09-19T08:01:10.774832+0000","last_unstale":"2024-09-19T08:03:00.997263+0000","last_undegraded":"2024-09-19T08:03:00.997263+0000","last_fullsized":"2024-09-19T08:03:00.997263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:57:29.800787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780968+0000","last_change":"2024-09-19T08:01:10.769115+0000","last_active":"2024-09-19T08:03:00.780968+0000","last_peered":"2024-09-19T08:03:00.780968+0000","last_clean":"2024-09-19T08:03:00.780968+0000","last_became_active":"2024-09-19T08:01:10.768088+0000","last_became_peered":"2024-09-19T08:01:10.768088+0000","last_unstale":"2024-09-19T08:03:00.780968+0000","last_undegraded":"2024-09-19T08:03:00.780968+0000","last_fullsized":"2024-09-19T08:03:00.780968+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:27:16.042722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844124+0000","last_change":"2024-09-19T08:01:10.781426+0000","last_active":"2024-09-19T08:03:00.844124+0000","last_peered":"2024-09-19T08:03:00.844124+0000","last_clean":"2024-09-19T08:03:00.844124+0000","last_became_active":"2024-09-19T08:01:10.781094+0000","last_became_peered":"2024-09-19T08:01:10.781094+0000","last_unstale":"2024-09-19T08:03:00.844124+0000","last_undegraded":"2024-09-19T08:03:00.844124+0000","last_fullsized":"2024-09-19T08:03:00.844124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:43:31.336610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760020+0000","last_change":"2024-09-19T08:01:10.766466+0000","last_active":"2024-09-19T08:03:00.760020+0000","last_peered":"2024-09-19T08:03:00.760020+0000","last_clean":"2024-09-19T08:03:00.760020+0000","last_became_active":"2024-09-19T08:01:10.766123+0000","last_became_peered":"2024-09-19T08:01:10.766123+0000","last_unstale":"2024-09-19T08:03:00.760020+0000","last_undegraded":"2024-09-19T08:03:00.760020+0000","last_fullsized":"2024-09-19T08:03:00.760020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:25.848603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780633+0000","last_change":"2024-09-19T08:01:10.772782+0000","last_active":"2024-09-19T08:03:00.780633+0000","last_peered":"2024-09-19T08:03:00.780633+0000","last_clean":"2024-09-19T08:03:00.780633+0000","last_became_active":"2024-09-19T08:01:10.772445+0000","last_became_peered":"2024-09-19T08:01:10.772445+0000","last_unstale":"2024-09-19T08:03:00.780633+0000","last_undegraded":"2024-09-19T08:03:00.780633+0000","last_fullsized":"2024-09-19T08:03:00.780633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:09.174005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759047+0000","last_change":"2024-09-19T08:01:10.771944+0000","last_active":"2024-09-19T08:03:00.759047+0000","last_peered":"2024-09-19T08:03:00.759047+0000","last_clean":"2024-09-19T08:03:00.759047+0000","last_became_active":"2024-09-19T08:01:10.771643+0000","last_became_peered":"2024-09-19T08:01:10.771643+0000","last_unstale":"2024-09-19T08:03:00.759047+0000","last_undegraded":"2024-09-19T08:03:00.759047+0000","last_fullsized":"2024-09-19T08:03:00.759047+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:46:27.121722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997451+0000","last_change":"2024-09-19T08:01:10.775144+0000","last_active":"2024-09-19T08:03:00.997451+0000","last_peered":"2024-09-19T08:03:00.997451+0000","last_clean":"2024-09-19T08:03:00.997451+0000","last_became_active":"2024-09-19T08:01:10.774769+0000","last_became_peered":"2024-09-19T08:01:10.774769+0000","last_unstale":"2024-09-19T08:03:00.997451+0000","last_undegraded":"2024-09-19T08:03:00.997451+0000","last_fullsized":"2024-09-19T08:03:00.997451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:59:47.053993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848212+0000","last_change":"2024-09-19T08:01:10.766750+0000","last_active":"2024-09-19T08:03:00.848212+0000","last_peered":"2024-09-19T08:03:00.848212+0000","last_clean":"2024-09-19T08:03:00.848212+0000","last_became_active":"2024-09-19T08:01:10.766450+0000","last_became_peered":"2024-09-19T08:01:10.766450+0000","last_unstale":"2024-09-19T08:03:00.848212+0000","last_undegraded":"2024-09-19T08:03:00.848212+0000","last_fullsized":"2024-09-19T08:03:00.848212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:47:58.133815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841186+0000","last_change":"2024-09-19T08:01:10.768430+0000","last_active":"2024-09-19T08:03:00.841186+0000","last_peered":"2024-09-19T08:03:00.841186+0000","last_clean":"2024-09-19T08:03:00.841186+0000","last_became_active":"2024-09-19T08:01:10.768092+0000","last_became_peered":"2024-09-19T08:01:10.768092+0000","last_unstale":"2024-09-19T08:03:00.841186+0000","last_undegraded":"2024-09-19T08:03:00.841186+0000","last_fullsized":"2024-09-19T08:03:00.841186+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:16:42.343138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849206+0000","last_change":"2024-09-19T08:01:10.771131+0000","last_active":"2024-09-19T08:03:00.849206+0000","last_peered":"2024-09-19T08:03:00.849206+0000","last_clean":"2024-09-19T08:03:00.849206+0000","last_became_active":"2024-09-19T08:01:10.770851+0000","last_became_peered":"2024-09-19T08:01:10.770851+0000","last_unstale":"2024-09-19T08:03:00.849206+0000","last_undegraded":"2024-09-19T08:03:00.849206+0000","last_fullsized":"2024-09-19T08:03:00.849206+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:07:15.830476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760273+0000","last_change":"2024-09-19T08:01:10.776220+0000","last_active":"2024-09-19T08:03:00.760273+0000","last_peered":"2024-09-19T08:03:00.760273+0000","last_clean":"2024-09-19T08:03:00.760273+0000","last_became_active":"2024-09-19T08:01:10.775889+0000","last_became_peered":"2024-09-19T08:01:10.775889+0000","last_unstale":"2024-09-19T08:03:00.760273+0000","last_undegraded":"2024-09-19T08:03:00.760273+0000","last_fullsized":"2024-09-19T08:03:00.760273+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:23.714675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780731+0000","last_change":"2024-09-19T08:01:10.776370+0000","last_active":"2024-09-19T08:03:00.780731+0000","last_peered":"2024-09-19T08:03:00.780731+0000","last_clean":"2024-09-19T08:03:00.780731+0000","last_became_active":"2024-09-19T08:01:10.776027+0000","last_became_peered":"2024-09-19T08:01:10.776027+0000","last_unstale":"2024-09-19T08:03:00.780731+0000","last_undegraded":"2024-09-19T08:03:00.780731+0000","last_fullsized":"2024-09-19T08:03:00.780731+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:48:46.293404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758951+0000","last_change":"2024-09-19T08:01:26.758971+0000","last_active":"2024-09-19T08:03:00.758951+0000","last_peered":"2024-09-19T08:03:00.758951+0000","last_clean":"2024-09-19T08:03:00.758951+0000","last_became_active":"2024-09-19T08:01:26.758524+0000","last_became_peered":"2024-09-19T08:01:26.758524+0000","last_unstale":"2024-09-19T08:03:00.758951+0000","last_undegraded":"2024-09-19T08:03:00.758951+0000","last_fullsized":"2024-09-19T08:03:00.758951+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:34:22.439327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994656+0000","last_change":"2024-09-19T08:01:10.767445+0000","last_active":"2024-09-19T08:03:00.994656+0000","last_peered":"2024-09-19T08:03:00.994656+0000","last_clean":"2024-09-19T08:03:00.994656+0000","last_became_active":"2024-09-19T08:01:10.767128+0000","last_became_peered":"2024-09-19T08:01:10.767128+0000","last_unstale":"2024-09-19T08:03:00.994656+0000","last_undegraded":"2024-09-19T08:03:00.994656+0000","last_fullsized":"2024-09-19T08:03:00.994656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:10.273948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759852+0000","last_change":"2024-09-19T08:01:10.768954+0000","last_active":"2024-09-19T08:03:00.759852+0000","last_peered":"2024-09-19T08:03:00.759852+0000","last_clean":"2024-09-19T08:03:00.759852+0000","last_became_active":"2024-09-19T08:01:10.768577+0000","last_became_peered":"2024-09-19T08:01:10.768577+0000","last_unstale":"2024-09-19T08:03:00.759852+0000","last_undegraded":"2024-09-19T08:03:00.759852+0000","last_fullsized":"2024-09-19T08:03:00.759852+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:20:21.812531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759843+0000","last_change":"2024-09-19T08:01:10.774104+0000","last_active":"2024-09-19T08:03:00.759843+0000","last_peered":"2024-09-19T08:03:00.759843+0000","last_clean":"2024-09-19T08:03:00.759843+0000","last_became_active":"2024-09-19T08:01:10.773721+0000","last_became_peered":"2024-09-19T08:01:10.773721+0000","last_unstale":"2024-09-19T08:03:00.759843+0000","last_undegraded":"2024-09-19T08:03:00.759843+0000","last_fullsized":"2024-09-19T08:03:00.759843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:45:34.945994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780513+0000","last_change":"2024-09-19T08:01:26.727296+0000","last_active":"2024-09-19T08:03:00.780513+0000","last_peered":"2024-09-19T08:03:00.780513+0000","last_clean":"2024-09-19T08:03:00.780513+0000","last_became_active":"2024-09-19T08:01:26.726631+0000","last_became_peered":"2024-09-19T08:01:26.726631+0000","last_unstale":"2024-09-19T08:03:00.780513+0000","last_undegraded":"2024-09-19T08:03:00.780513+0000","last_fullsized":"2024-09-19T08:03:00.780513+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:19:24.807050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847424+0000","last_change":"2024-09-19T08:01:10.765282+0000","last_active":"2024-09-19T08:03:00.847424+0000","last_peered":"2024-09-19T08:03:00.847424+0000","last_clean":"2024-09-19T08:03:00.847424+0000","last_became_active":"2024-09-19T08:01:10.764928+0000","last_became_peered":"2024-09-19T08:01:10.764928+0000","last_unstale":"2024-09-19T08:03:00.847424+0000","last_undegraded":"2024-09-19T08:03:00.847424+0000","last_fullsized":"2024-09-19T08:03:00.847424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:00:37.395110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"81'10","reported_seq":140,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.999561+0000","last_change":"2024-09-19T08:01:08.760686+0000","last_active":"2024-09-19T08:03:00.999561+0000","last_peered":"2024-09-19T08:03:00.999561+0000","last_clean":"2024-09-19T08:03:00.999561+0000","last_became_active":"2024-09-19T08:01:08.759912+0000","last_became_peered":"2024-09-19T08:01:08.759912+0000","last_unstale":"2024-09-19T08:03:00.999561+0000","last_undegraded":"2024-09-19T08:03:00.999561+0000","last_fullsized":"2024-09-19T08:03:00.999561+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:46:01.879611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2250,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":5,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841395+0000","last_change":"2024-09-19T08:01:10.779668+0000","last_active":"2024-09-19T08:03:00.841395+0000","last_peered":"2024-09-19T08:03:00.841395+0000","last_clean":"2024-09-19T08:03:00.841395+0000","last_became_active":"2024-09-19T08:01:10.779265+0000","last_became_peered":"2024-09-19T08:01:10.779265+0000","last_unstale":"2024-09-19T08:03:00.841395+0000","last_undegraded":"2024-09-19T08:03:00.841395+0000","last_fullsized":"2024-09-19T08:03:00.841395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:06.177700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838967+0000","last_change":"2024-09-19T08:01:08.762668+0000","last_active":"2024-09-19T08:03:00.838967+0000","last_peered":"2024-09-19T08:03:00.838967+0000","last_clean":"2024-09-19T08:03:00.838967+0000","last_became_active":"2024-09-19T08:01:08.762277+0000","last_became_peered":"2024-09-19T08:01:08.762277+0000","last_unstale":"2024-09-19T08:03:00.838967+0000","last_undegraded":"2024-09-19T08:03:00.838967+0000","last_fullsized":"2024-09-19T08:03:00.838967+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:36:59.316655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758585+0000","last_change":"2024-09-19T08:01:08.757401+0000","last_active":"2024-09-19T08:03:00.758585+0000","last_peered":"2024-09-19T08:03:00.758585+0000","last_clean":"2024-09-19T08:03:00.758585+0000","last_became_active":"2024-09-19T08:01:08.757067+0000","last_became_peered":"2024-09-19T08:01:08.757067+0000","last_unstale":"2024-09-19T08:03:00.758585+0000","last_undegraded":"2024-09-19T08:03:00.758585+0000","last_fullsized":"2024-09-19T08:03:00.758585+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:19.617830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848289+0000","last_change":"2024-09-19T08:01:10.765149+0000","last_active":"2024-09-19T08:03:00.848289+0000","last_peered":"2024-09-19T08:03:00.848289+0000","last_clean":"2024-09-19T08:03:00.848289+0000","last_became_active":"2024-09-19T08:01:10.764840+0000","last_became_peered":"2024-09-19T08:01:10.764840+0000","last_unstale":"2024-09-19T08:03:00.848289+0000","last_undegraded":"2024-09-19T08:03:00.848289+0000","last_fullsized":"2024-09-19T08:03:00.848289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:15.539307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841355+0000","last_change":"2024-09-19T08:01:08.759435+0000","last_active":"2024-09-19T08:03:00.841355+0000","last_peered":"2024-09-19T08:03:00.841355+0000","last_clean":"2024-09-19T08:03:00.841355+0000","last_became_active":"2024-09-19T08:01:08.759046+0000","last_became_peered":"2024-09-19T08:01:08.759046+0000","last_unstale":"2024-09-19T08:03:00.841355+0000","last_undegraded":"2024-09-19T08:03:00.841355+0000","last_fullsized":"2024-09-19T08:03:00.841355+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:21:24.077626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.843896+0000","last_change":"2024-09-19T08:01:10.778432+0000","last_active":"2024-09-19T08:03:00.843896+0000","last_peered":"2024-09-19T08:03:00.843896+0000","last_clean":"2024-09-19T08:03:00.843896+0000","last_became_active":"2024-09-19T08:01:10.778098+0000","last_became_peered":"2024-09-19T08:01:10.778098+0000","last_unstale":"2024-09-19T08:03:00.843896+0000","last_undegraded":"2024-09-19T08:03:00.843896+0000","last_fullsized":"2024-09-19T08:03:00.843896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:40:12.921165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":230,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839698+0000","last_change":"2024-09-19T08:01:08.764117+0000","last_active":"2024-09-19T08:03:00.839698+0000","last_peered":"2024-09-19T08:03:00.839698+0000","last_clean":"2024-09-19T08:03:00.839698+0000","last_became_active":"2024-09-19T08:01:08.763308+0000","last_became_peered":"2024-09-19T08:01:08.763308+0000","last_unstale":"2024-09-19T08:03:00.839698+0000","last_undegraded":"2024-09-19T08:03:00.839698+0000","last_fullsized":"2024-09-19T08:03:00.839698+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:19:57.014628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":200,"num_read_kb":200,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997921+0000","last_change":"2024-09-19T08:01:10.781771+0000","last_active":"2024-09-19T08:03:00.997921+0000","last_peered":"2024-09-19T08:03:00.997921+0000","last_clean":"2024-09-19T08:03:00.997921+0000","last_became_active":"2024-09-19T08:01:10.781432+0000","last_became_peered":"2024-09-19T08:01:10.781432+0000","last_unstale":"2024-09-19T08:03:00.997921+0000","last_undegraded":"2024-09-19T08:03:00.997921+0000","last_fullsized":"2024-09-19T08:03:00.997921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:25:15.067850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838023+0000","last_change":"2024-09-19T08:01:26.771171+0000","last_active":"2024-09-19T08:03:00.838023+0000","last_peered":"2024-09-19T08:03:00.838023+0000","last_clean":"2024-09-19T08:03:00.838023+0000","last_became_active":"2024-09-19T08:01:26.759274+0000","last_became_peered":"2024-09-19T08:01:26.759274+0000","last_unstale":"2024-09-19T08:03:00.838023+0000","last_undegraded":"2024-09-19T08:03:00.838023+0000","last_fullsized":"2024-09-19T08:03:00.838023+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:20:54.603420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994926+0000","last_change":"2024-09-19T08:01:10.768318+0000","last_active":"2024-09-19T08:03:00.994926+0000","last_peered":"2024-09-19T08:03:00.994926+0000","last_clean":"2024-09-19T08:03:00.994926+0000","last_became_active":"2024-09-19T08:01:10.768022+0000","last_became_peered":"2024-09-19T08:01:10.768022+0000","last_unstale":"2024-09-19T08:03:00.994926+0000","last_undegraded":"2024-09-19T08:03:00.994926+0000","last_fullsized":"2024-09-19T08:03:00.994926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:00:45.328638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841594+0000","last_change":"2024-09-19T08:01:09.750530+0000","last_active":"2024-09-19T08:03:00.841594+0000","last_peered":"2024-09-19T08:03:00.841594+0000","last_clean":"2024-09-19T08:03:00.841594+0000","last_became_active":"2024-09-19T08:01:09.750029+0000","last_became_peered":"2024-09-19T08:01:09.750029+0000","last_unstale":"2024-09-19T08:03:00.841594+0000","last_undegraded":"2024-09-19T08:03:00.841594+0000","last_fullsized":"2024-09-19T08:03:00.841594+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:01:16.062992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997659+0000","last_change":"2024-09-19T08:01:26.724636+0000","last_active":"2024-09-19T08:03:00.997659+0000","last_peered":"2024-09-19T08:03:00.997659+0000","last_clean":"2024-09-19T08:03:00.997659+0000","last_became_active":"2024-09-19T08:01:26.724171+0000","last_became_peered":"2024-09-19T08:01:26.724171+0000","last_unstale":"2024-09-19T08:03:00.997659+0000","last_undegraded":"2024-09-19T08:03:00.997659+0000","last_fullsized":"2024-09-19T08:03:00.997659+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:54:05.998374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'37","reported_seq":275,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997077+0000","last_change":"2024-09-19T08:01:08.761950+0000","last_active":"2024-09-19T08:03:00.997077+0000","last_peered":"2024-09-19T08:03:00.997077+0000","last_clean":"2024-09-19T08:03:00.997077+0000","last_became_active":"2024-09-19T08:01:08.761533+0000","last_became_peered":"2024-09-19T08:01:08.761533+0000","last_unstale":"2024-09-19T08:03:00.997077+0000","last_undegraded":"2024-09-19T08:03:00.997077+0000","last_fullsized":"2024-09-19T08:03:00.997077+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:07:26.668141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66480,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":214,"num_read_kb":171,"num_write":39,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847631+0000","last_change":"2024-09-19T08:01:10.774280+0000","last_active":"2024-09-19T08:03:00.847631+0000","last_peered":"2024-09-19T08:03:00.847631+0000","last_clean":"2024-09-19T08:03:00.847631+0000","last_became_active":"2024-09-19T08:01:10.773982+0000","last_became_peered":"2024-09-19T08:01:10.773982+0000","last_unstale":"2024-09-19T08:03:00.847631+0000","last_undegraded":"2024-09-19T08:03:00.847631+0000","last_fullsized":"2024-09-19T08:03:00.847631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:21:43.180057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760058+0000","last_change":"2024-09-19T08:01:08.761067+0000","last_active":"2024-09-19T08:03:00.760058+0000","last_peered":"2024-09-19T08:03:00.760058+0000","last_clean":"2024-09-19T08:03:00.760058+0000","last_became_active":"2024-09-19T08:01:08.760664+0000","last_became_peered":"2024-09-19T08:01:08.760664+0000","last_unstale":"2024-09-19T08:03:00.760058+0000","last_undegraded":"2024-09-19T08:03:00.760058+0000","last_fullsized":"2024-09-19T08:03:00.760058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:22:38.086891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758630+0000","last_change":"2024-09-19T08:01:10.771845+0000","last_active":"2024-09-19T08:03:00.758630+0000","last_peered":"2024-09-19T08:03:00.758630+0000","last_clean":"2024-09-19T08:03:00.758630+0000","last_became_active":"2024-09-19T08:01:10.771533+0000","last_became_peered":"2024-09-19T08:01:10.771533+0000","last_unstale":"2024-09-19T08:03:00.758630+0000","last_undegraded":"2024-09-19T08:03:00.758630+0000","last_fullsized":"2024-09-19T08:03:00.758630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:29:35.180763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841267+0000","last_change":"2024-09-19T08:01:08.763869+0000","last_active":"2024-09-19T08:03:00.841267+0000","last_peered":"2024-09-19T08:03:00.841267+0000","last_clean":"2024-09-19T08:03:00.841267+0000","last_became_active":"2024-09-19T08:01:08.763451+0000","last_became_peered":"2024-09-19T08:01:08.763451+0000","last_unstale":"2024-09-19T08:03:00.841267+0000","last_undegraded":"2024-09-19T08:03:00.841267+0000","last_fullsized":"2024-09-19T08:03:00.841267+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:32:40.901049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838028+0000","last_change":"2024-09-19T08:01:10.770823+0000","last_active":"2024-09-19T08:03:00.838028+0000","last_peered":"2024-09-19T08:03:00.838028+0000","last_clean":"2024-09-19T08:03:00.838028+0000","last_became_active":"2024-09-19T08:01:10.770462+0000","last_became_peered":"2024-09-19T08:01:10.770462+0000","last_unstale":"2024-09-19T08:03:00.838028+0000","last_undegraded":"2024-09-19T08:03:00.838028+0000","last_fullsized":"2024-09-19T08:03:00.838028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:04:32.832009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'7","reported_seq":134,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.993691+0000","last_change":"2024-09-19T08:01:08.759429+0000","last_active":"2024-09-19T08:03:00.993691+0000","last_peered":"2024-09-19T08:03:00.993691+0000","last_clean":"2024-09-19T08:03:00.993691+0000","last_became_active":"2024-09-19T08:01:08.759062+0000","last_became_peered":"2024-09-19T08:01:08.759062+0000","last_unstale":"2024-09-19T08:03:00.993691+0000","last_undegraded":"2024-09-19T08:03:00.993691+0000","last_fullsized":"2024-09-19T08:03:00.993691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:16:51.582435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1374,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":3,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837660+0000","last_change":"2024-09-19T08:01:10.784735+0000","last_active":"2024-09-19T08:03:00.837660+0000","last_peered":"2024-09-19T08:03:00.837660+0000","last_clean":"2024-09-19T08:03:00.837660+0000","last_became_active":"2024-09-19T08:01:10.784346+0000","last_became_peered":"2024-09-19T08:01:10.784346+0000","last_unstale":"2024-09-19T08:03:00.837660+0000","last_undegraded":"2024-09-19T08:03:00.837660+0000","last_fullsized":"2024-09-19T08:03:00.837660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:27:54.580391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847399+0000","last_change":"2024-09-19T08:01:09.749498+0000","last_active":"2024-09-19T08:03:00.847399+0000","last_peered":"2024-09-19T08:03:00.847399+0000","last_clean":"2024-09-19T08:03:00.847399+0000","last_became_active":"2024-09-19T08:01:09.749091+0000","last_became_peered":"2024-09-19T08:01:09.749091+0000","last_unstale":"2024-09-19T08:03:00.847399+0000","last_undegraded":"2024-09-19T08:03:00.847399+0000","last_fullsized":"2024-09-19T08:03:00.847399+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:56:13.644931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841217+0000","last_change":"2024-09-19T08:01:10.779599+0000","last_active":"2024-09-19T08:03:00.841217+0000","last_peered":"2024-09-19T08:03:00.841217+0000","last_clean":"2024-09-19T08:03:00.841217+0000","last_became_active":"2024-09-19T08:01:10.779219+0000","last_became_peered":"2024-09-19T08:01:10.779219+0000","last_unstale":"2024-09-19T08:03:00.841217+0000","last_undegraded":"2024-09-19T08:03:00.841217+0000","last_fullsized":"2024-09-19T08:03:00.841217+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:20.193753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847308+0000","last_change":"2024-09-19T08:01:08.761432+0000","last_active":"2024-09-19T08:03:00.847308+0000","last_peered":"2024-09-19T08:03:00.847308+0000","last_clean":"2024-09-19T08:03:00.847308+0000","last_became_active":"2024-09-19T08:01:08.761044+0000","last_became_peered":"2024-09-19T08:01:08.761044+0000","last_unstale":"2024-09-19T08:03:00.847308+0000","last_undegraded":"2024-09-19T08:03:00.847308+0000","last_fullsized":"2024-09-19T08:03:00.847308+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:12.237334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780765+0000","last_change":"2024-09-19T08:01:10.772559+0000","last_active":"2024-09-19T08:03:00.780765+0000","last_peered":"2024-09-19T08:03:00.780765+0000","last_clean":"2024-09-19T08:03:00.780765+0000","last_became_active":"2024-09-19T08:01:10.772176+0000","last_became_peered":"2024-09-19T08:01:10.772176+0000","last_unstale":"2024-09-19T08:03:00.780765+0000","last_undegraded":"2024-09-19T08:03:00.780765+0000","last_fullsized":"2024-09-19T08:03:00.780765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:27:16.296127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758996+0000","last_change":"2024-09-19T08:01:08.759989+0000","last_active":"2024-09-19T08:03:00.758996+0000","last_peered":"2024-09-19T08:03:00.758996+0000","last_clean":"2024-09-19T08:03:00.758996+0000","last_became_active":"2024-09-19T08:01:08.759705+0000","last_became_peered":"2024-09-19T08:01:08.759705+0000","last_unstale":"2024-09-19T08:03:00.758996+0000","last_undegraded":"2024-09-19T08:03:00.758996+0000","last_fullsized":"2024-09-19T08:03:00.758996+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:33:25.237567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838150+0000","last_change":"2024-09-19T08:01:10.776528+0000","last_active":"2024-09-19T08:03:00.838150+0000","last_peered":"2024-09-19T08:03:00.838150+0000","last_clean":"2024-09-19T08:03:00.838150+0000","last_became_active":"2024-09-19T08:01:10.776222+0000","last_became_peered":"2024-09-19T08:01:10.776222+0000","last_unstale":"2024-09-19T08:03:00.838150+0000","last_undegraded":"2024-09-19T08:03:00.838150+0000","last_fullsized":"2024-09-19T08:03:00.838150+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.787492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'41","reported_seq":73,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.957555+0000","last_change":"2024-09-19T08:01:08.759631+0000","last_active":"2024-09-19T08:03:00.957555+0000","last_peered":"2024-09-19T08:03:00.957555+0000","last_clean":"2024-09-19T08:03:00.957555+0000","last_became_active":"2024-09-19T08:01:08.759156+0000","last_became_peered":"2024-09-19T08:01:08.759156+0000","last_unstale":"2024-09-19T08:03:00.957555+0000","last_undegraded":"2024-09-19T08:03:00.957555+0000","last_fullsized":"2024-09-19T08:03:00.957555+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:48:50.910824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837969+0000","last_change":"2024-09-19T08:01:10.772660+0000","last_active":"2024-09-19T08:03:00.837969+0000","last_peered":"2024-09-19T08:03:00.837969+0000","last_clean":"2024-09-19T08:03:00.837969+0000","last_became_active":"2024-09-19T08:01:10.771816+0000","last_became_peered":"2024-09-19T08:01:10.771816+0000","last_unstale":"2024-09-19T08:03:00.837969+0000","last_undegraded":"2024-09-19T08:03:00.837969+0000","last_fullsized":"2024-09-19T08:03:00.837969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:04:32.667107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'7","reported_seq":134,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760174+0000","last_change":"2024-09-19T08:01:09.749844+0000","last_active":"2024-09-19T08:03:00.760174+0000","last_peered":"2024-09-19T08:03:00.760174+0000","last_clean":"2024-09-19T08:03:00.760174+0000","last_became_active":"2024-09-19T08:01:09.749410+0000","last_became_peered":"2024-09-19T08:01:09.749410+0000","last_unstale":"2024-09-19T08:03:00.760174+0000","last_undegraded":"2024-09-19T08:03:00.760174+0000","last_fullsized":"2024-09-19T08:03:00.760174+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:03:14.989814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1704,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838196+0000","last_change":"2024-09-19T08:01:10.773782+0000","last_active":"2024-09-19T08:03:00.838196+0000","last_peered":"2024-09-19T08:03:00.838196+0000","last_clean":"2024-09-19T08:03:00.838196+0000","last_became_active":"2024-09-19T08:01:10.773471+0000","last_became_peered":"2024-09-19T08:01:10.773471+0000","last_unstale":"2024-09-19T08:03:00.838196+0000","last_undegraded":"2024-09-19T08:03:00.838196+0000","last_fullsized":"2024-09-19T08:03:00.838196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:15.780259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":344,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837815+0000","last_change":"2024-09-19T08:00:01.677517+0000","last_active":"2024-09-19T08:03:00.837815+0000","last_peered":"2024-09-19T08:03:00.837815+0000","last_clean":"2024-09-19T08:03:00.837815+0000","last_became_active":"2024-09-19T08:00:01.677089+0000","last_became_peered":"2024-09-19T08:00:01.677089+0000","last_unstale":"2024-09-19T08:03:00.837815+0000","last_undegraded":"2024-09-19T08:03:00.837815+0000","last_fullsized":"2024-09-19T08:03:00.837815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_clean_scrub_stamp":"2024-09-19T07:56:43.962142+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:10:36.158811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761731+0000","last_change":"2024-09-19T08:01:08.759819+0000","last_active":"2024-09-19T08:03:00.761731+0000","last_peered":"2024-09-19T08:03:00.761731+0000","last_clean":"2024-09-19T08:03:00.761731+0000","last_became_active":"2024-09-19T08:01:08.759420+0000","last_became_peered":"2024-09-19T08:01:08.759420+0000","last_unstale":"2024-09-19T08:03:00.761731+0000","last_undegraded":"2024-09-19T08:03:00.761731+0000","last_fullsized":"2024-09-19T08:03:00.761731+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:06:16.852323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.843903+0000","last_change":"2024-09-19T08:01:26.724260+0000","last_active":"2024-09-19T08:03:00.843903+0000","last_peered":"2024-09-19T08:03:00.843903+0000","last_clean":"2024-09-19T08:03:00.843903+0000","last_became_active":"2024-09-19T08:01:26.723851+0000","last_became_peered":"2024-09-19T08:01:26.723851+0000","last_unstale":"2024-09-19T08:03:00.843903+0000","last_undegraded":"2024-09-19T08:03:00.843903+0000","last_fullsized":"2024-09-19T08:03:00.843903+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:18:46.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837812+0000","last_change":"2024-09-19T08:01:08.761291+0000","last_active":"2024-09-19T08:03:00.837812+0000","last_peered":"2024-09-19T08:03:00.837812+0000","last_clean":"2024-09-19T08:03:00.837812+0000","last_became_active":"2024-09-19T08:01:08.760238+0000","last_became_peered":"2024-09-19T08:01:08.760238+0000","last_unstale":"2024-09-19T08:03:00.837812+0000","last_undegraded":"2024-09-19T08:03:00.837812+0000","last_fullsized":"2024-09-19T08:03:00.837812+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:26:09.980573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758579+0000","last_change":"2024-09-19T08:01:10.773056+0000","last_active":"2024-09-19T08:03:00.758579+0000","last_peered":"2024-09-19T08:03:00.758579+0000","last_clean":"2024-09-19T08:03:00.758579+0000","last_became_active":"2024-09-19T08:01:10.772431+0000","last_became_peered":"2024-09-19T08:01:10.772431+0000","last_unstale":"2024-09-19T08:03:00.758579+0000","last_undegraded":"2024-09-19T08:03:00.758579+0000","last_fullsized":"2024-09-19T08:03:00.758579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:39:06.878086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":132,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.842523+0000","last_change":"2024-09-19T08:01:09.749707+0000","last_active":"2024-09-19T08:03:00.842523+0000","last_peered":"2024-09-19T08:03:00.842523+0000","last_clean":"2024-09-19T08:03:00.842523+0000","last_became_active":"2024-09-19T08:01:09.749013+0000","last_became_peered":"2024-09-19T08:01:09.749013+0000","last_unstale":"2024-09-19T08:03:00.842523+0000","last_undegraded":"2024-09-19T08:03:00.842523+0000","last_fullsized":"2024-09-19T08:03:00.842523+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:40:12.682158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841187+0000","last_change":"2024-09-19T08:01:10.774101+0000","last_active":"2024-09-19T08:03:00.841187+0000","last_peered":"2024-09-19T08:03:00.841187+0000","last_clean":"2024-09-19T08:03:00.841187+0000","last_became_active":"2024-09-19T08:01:10.773822+0000","last_became_peered":"2024-09-19T08:01:10.773822+0000","last_unstale":"2024-09-19T08:03:00.841187+0000","last_undegraded":"2024-09-19T08:03:00.841187+0000","last_fullsized":"2024-09-19T08:03:00.841187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:29.571118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"78'5","reported_seq":129,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837640+0000","last_change":"2024-09-19T08:01:08.761675+0000","last_active":"2024-09-19T08:03:00.837640+0000","last_peered":"2024-09-19T08:03:00.837640+0000","last_clean":"2024-09-19T08:03:00.837640+0000","last_became_active":"2024-09-19T08:01:08.761346+0000","last_became_peered":"2024-09-19T08:01:08.761346+0000","last_unstale":"2024-09-19T08:03:00.837640+0000","last_undegraded":"2024-09-19T08:03:00.837640+0000","last_fullsized":"2024-09-19T08:03:00.837640+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:51:19.237458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839233+0000","last_change":"2024-09-19T08:01:10.781565+0000","last_active":"2024-09-19T08:03:00.839233+0000","last_peered":"2024-09-19T08:03:00.839233+0000","last_clean":"2024-09-19T08:03:00.839233+0000","last_became_active":"2024-09-19T08:01:10.781301+0000","last_became_peered":"2024-09-19T08:01:10.781301+0000","last_unstale":"2024-09-19T08:03:00.839233+0000","last_undegraded":"2024-09-19T08:03:00.839233+0000","last_fullsized":"2024-09-19T08:03:00.839233+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:55:52.932009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844577+0000","last_change":"2024-09-19T08:01:09.748918+0000","last_active":"2024-09-19T08:03:00.844577+0000","last_peered":"2024-09-19T08:03:00.844577+0000","last_clean":"2024-09-19T08:03:00.844577+0000","last_became_active":"2024-09-19T08:01:09.748497+0000","last_became_peered":"2024-09-19T08:01:09.748497+0000","last_unstale":"2024-09-19T08:03:00.844577+0000","last_undegraded":"2024-09-19T08:03:00.844577+0000","last_fullsized":"2024-09-19T08:03:00.844577+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:20:31.030133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839231+0000","last_change":"2024-09-19T08:01:10.781388+0000","last_active":"2024-09-19T08:03:00.839231+0000","last_peered":"2024-09-19T08:03:00.839231+0000","last_clean":"2024-09-19T08:03:00.839231+0000","last_became_active":"2024-09-19T08:01:10.781063+0000","last_became_peered":"2024-09-19T08:01:10.781063+0000","last_unstale":"2024-09-19T08:03:00.839231+0000","last_undegraded":"2024-09-19T08:03:00.839231+0000","last_fullsized":"2024-09-19T08:03:00.839231+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:12:08.852717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847755+0000","last_change":"2024-09-19T08:01:09.750080+0000","last_active":"2024-09-19T08:03:00.847755+0000","last_peered":"2024-09-19T08:03:00.847755+0000","last_clean":"2024-09-19T08:03:00.847755+0000","last_became_active":"2024-09-19T08:01:09.749053+0000","last_became_peered":"2024-09-19T08:01:09.749053+0000","last_unstale":"2024-09-19T08:03:00.847755+0000","last_undegraded":"2024-09-19T08:03:00.847755+0000","last_fullsized":"2024-09-19T08:03:00.847755+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:15:45.292714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759144+0000","last_change":"2024-09-19T08:01:10.767070+0000","last_active":"2024-09-19T08:03:00.759144+0000","last_peered":"2024-09-19T08:03:00.759144+0000","last_clean":"2024-09-19T08:03:00.759144+0000","last_became_active":"2024-09-19T08:01:10.766760+0000","last_became_peered":"2024-09-19T08:01:10.766760+0000","last_unstale":"2024-09-19T08:03:00.759144+0000","last_undegraded":"2024-09-19T08:03:00.759144+0000","last_fullsized":"2024-09-19T08:03:00.759144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:17.841135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'51","reported_seq":98,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.960465+0000","last_change":"2024-09-19T08:01:08.756143+0000","last_active":"2024-09-19T08:03:00.960465+0000","last_peered":"2024-09-19T08:03:00.960465+0000","last_clean":"2024-09-19T08:03:00.960465+0000","last_became_active":"2024-09-19T08:01:08.755777+0000","last_became_peered":"2024-09-19T08:01:08.755777+0000","last_unstale":"2024-09-19T08:03:00.960465+0000","last_undegraded":"2024-09-19T08:03:00.960465+0000","last_fullsized":"2024-09-19T08:03:00.960465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:21.430673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":90,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849237+0000","last_change":"2024-09-19T08:01:10.774140+0000","last_active":"2024-09-19T08:03:00.849237+0000","last_peered":"2024-09-19T08:03:00.849237+0000","last_clean":"2024-09-19T08:03:00.849237+0000","last_became_active":"2024-09-19T08:01:10.773819+0000","last_became_peered":"2024-09-19T08:01:10.773819+0000","last_unstale":"2024-09-19T08:03:00.849237+0000","last_undegraded":"2024-09-19T08:03:00.849237+0000","last_fullsized":"2024-09-19T08:03:00.849237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:15:13.174013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761607+0000","last_change":"2024-09-19T08:01:09.169129+0000","last_active":"2024-09-19T08:03:00.761607+0000","last_peered":"2024-09-19T08:03:00.761607+0000","last_clean":"2024-09-19T08:03:00.761607+0000","last_became_active":"2024-09-19T08:01:09.168734+0000","last_became_peered":"2024-09-19T08:01:09.168734+0000","last_unstale":"2024-09-19T08:03:00.761607+0000","last_undegraded":"2024-09-19T08:03:00.761607+0000","last_fullsized":"2024-09-19T08:03:00.761607+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:01:52.232391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994759+0000","last_change":"2024-09-19T08:01:10.768592+0000","last_active":"2024-09-19T08:03:00.994759+0000","last_peered":"2024-09-19T08:03:00.994759+0000","last_clean":"2024-09-19T08:03:00.994759+0000","last_became_active":"2024-09-19T08:01:10.767645+0000","last_became_peered":"2024-09-19T08:01:10.767645+0000","last_unstale":"2024-09-19T08:03:00.994759+0000","last_undegraded":"2024-09-19T08:03:00.994759+0000","last_fullsized":"2024-09-19T08:03:00.994759+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:08:46.227461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997364+0000","last_change":"2024-09-19T08:01:09.749240+0000","last_active":"2024-09-19T08:03:00.997364+0000","last_peered":"2024-09-19T08:03:00.997364+0000","last_clean":"2024-09-19T08:03:00.997364+0000","last_became_active":"2024-09-19T08:01:09.748945+0000","last_became_peered":"2024-09-19T08:01:09.748945+0000","last_unstale":"2024-09-19T08:03:00.997364+0000","last_undegraded":"2024-09-19T08:03:00.997364+0000","last_fullsized":"2024-09-19T08:03:00.997364+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:25:13.791290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847326+0000","last_change":"2024-09-19T08:01:10.783133+0000","last_active":"2024-09-19T08:03:00.847326+0000","last_peered":"2024-09-19T08:03:00.847326+0000","last_clean":"2024-09-19T08:03:00.847326+0000","last_became_active":"2024-09-19T08:01:10.782797+0000","last_became_peered":"2024-09-19T08:01:10.782797+0000","last_unstale":"2024-09-19T08:03:00.847326+0000","last_undegraded":"2024-09-19T08:03:00.847326+0000","last_fullsized":"2024-09-19T08:03:00.847326+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:30:43.551877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848489+0000","last_change":"2024-09-19T08:01:08.757964+0000","last_active":"2024-09-19T08:03:00.848489+0000","last_peered":"2024-09-19T08:03:00.848489+0000","last_clean":"2024-09-19T08:03:00.848489+0000","last_became_active":"2024-09-19T08:01:08.757526+0000","last_became_peered":"2024-09-19T08:01:08.757526+0000","last_unstale":"2024-09-19T08:03:00.848489+0000","last_undegraded":"2024-09-19T08:03:00.848489+0000","last_fullsized":"2024-09-19T08:03:00.848489+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:03.565702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780557+0000","last_change":"2024-09-19T08:01:10.769974+0000","last_active":"2024-09-19T08:03:00.780557+0000","last_peered":"2024-09-19T08:03:00.780557+0000","last_clean":"2024-09-19T08:03:00.780557+0000","last_became_active":"2024-09-19T08:01:10.769650+0000","last_became_peered":"2024-09-19T08:01:10.769650+0000","last_unstale":"2024-09-19T08:03:00.780557+0000","last_undegraded":"2024-09-19T08:03:00.780557+0000","last_fullsized":"2024-09-19T08:03:00.780557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:01.755198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"78'3","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759999+0000","last_change":"2024-09-19T08:01:08.763476+0000","last_active":"2024-09-19T08:03:00.759999+0000","last_peered":"2024-09-19T08:03:00.759999+0000","last_clean":"2024-09-19T08:03:00.759999+0000","last_became_active":"2024-09-19T08:01:08.763154+0000","last_became_peered":"2024-09-19T08:01:08.763154+0000","last_unstale":"2024-09-19T08:03:00.759999+0000","last_undegraded":"2024-09-19T08:03:00.759999+0000","last_fullsized":"2024-09-19T08:03:00.759999+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:06.202452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848751+0000","last_change":"2024-09-19T08:01:10.784201+0000","last_active":"2024-09-19T08:03:00.848751+0000","last_peered":"2024-09-19T08:03:00.848751+0000","last_clean":"2024-09-19T08:03:00.848751+0000","last_became_active":"2024-09-19T08:01:10.783867+0000","last_became_peered":"2024-09-19T08:01:10.783867+0000","last_unstale":"2024-09-19T08:03:00.848751+0000","last_undegraded":"2024-09-19T08:03:00.848751+0000","last_fullsized":"2024-09-19T08:03:00.848751+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:43:50.082183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847290+0000","last_change":"2024-09-19T08:01:08.760479+0000","last_active":"2024-09-19T08:03:00.847290+0000","last_peered":"2024-09-19T08:03:00.847290+0000","last_clean":"2024-09-19T08:03:00.847290+0000","last_became_active":"2024-09-19T08:01:08.760004+0000","last_became_peered":"2024-09-19T08:01:08.760004+0000","last_unstale":"2024-09-19T08:03:00.847290+0000","last_undegraded":"2024-09-19T08:03:00.847290+0000","last_fullsized":"2024-09-19T08:03:00.847290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:41:19.956623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837851+0000","last_change":"2024-09-19T08:01:10.772184+0000","last_active":"2024-09-19T08:03:00.837851+0000","last_peered":"2024-09-19T08:03:00.837851+0000","last_clean":"2024-09-19T08:03:00.837851+0000","last_became_active":"2024-09-19T08:01:10.771825+0000","last_became_peered":"2024-09-19T08:01:10.771825+0000","last_unstale":"2024-09-19T08:03:00.837851+0000","last_undegraded":"2024-09-19T08:03:00.837851+0000","last_fullsized":"2024-09-19T08:03:00.837851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:36.764575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841567+0000","last_change":"2024-09-19T08:01:10.779368+0000","last_active":"2024-09-19T08:03:00.841567+0000","last_peered":"2024-09-19T08:03:00.841567+0000","last_clean":"2024-09-19T08:03:00.841567+0000","last_became_active":"2024-09-19T08:01:10.778884+0000","last_became_peered":"2024-09-19T08:01:10.778884+0000","last_unstale":"2024-09-19T08:03:00.841567+0000","last_undegraded":"2024-09-19T08:03:00.841567+0000","last_fullsized":"2024-09-19T08:03:00.841567+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:28:29.630300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780536+0000","last_change":"2024-09-19T08:01:26.741053+0000","last_active":"2024-09-19T08:03:00.780536+0000","last_peered":"2024-09-19T08:03:00.780536+0000","last_clean":"2024-09-19T08:03:00.780536+0000","last_became_active":"2024-09-19T08:01:26.724714+0000","last_became_peered":"2024-09-19T08:01:26.724714+0000","last_unstale":"2024-09-19T08:03:00.780536+0000","last_undegraded":"2024-09-19T08:03:00.780536+0000","last_fullsized":"2024-09-19T08:03:00.780536+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:59.583797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997323+0000","last_change":"2024-09-19T08:01:10.781389+0000","last_active":"2024-09-19T08:03:00.997323+0000","last_peered":"2024-09-19T08:03:00.997323+0000","last_clean":"2024-09-19T08:03:00.997323+0000","last_became_active":"2024-09-19T08:01:10.781080+0000","last_became_peered":"2024-09-19T08:01:10.781080+0000","last_unstale":"2024-09-19T08:03:00.997323+0000","last_undegraded":"2024-09-19T08:03:00.997323+0000","last_fullsized":"2024-09-19T08:03:00.997323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:40:45.225957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847701+0000","last_change":"2024-09-19T08:01:26.768102+0000","last_active":"2024-09-19T08:03:00.847701+0000","last_peered":"2024-09-19T08:03:00.847701+0000","last_clean":"2024-09-19T08:03:00.847701+0000","last_became_active":"2024-09-19T08:01:26.759462+0000","last_became_peered":"2024-09-19T08:01:26.759462+0000","last_unstale":"2024-09-19T08:03:00.847701+0000","last_undegraded":"2024-09-19T08:03:00.847701+0000","last_fullsized":"2024-09-19T08:03:00.847701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:02:06.815238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994921+0000","last_change":"2024-09-19T08:01:10.771031+0000","last_active":"2024-09-19T08:03:00.994921+0000","last_peered":"2024-09-19T08:03:00.994921+0000","last_clean":"2024-09-19T08:03:00.994921+0000","last_became_active":"2024-09-19T08:01:10.770675+0000","last_became_peered":"2024-09-19T08:01:10.770675+0000","last_unstale":"2024-09-19T08:03:00.994921+0000","last_undegraded":"2024-09-19T08:03:00.994921+0000","last_fullsized":"2024-09-19T08:03:00.994921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:42.203848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849208+0000","last_change":"2024-09-19T08:01:09.748622+0000","last_active":"2024-09-19T08:03:00.849208+0000","last_peered":"2024-09-19T08:03:00.849208+0000","last_clean":"2024-09-19T08:03:00.849208+0000","last_became_active":"2024-09-19T08:01:09.748202+0000","last_became_peered":"2024-09-19T08:01:09.748202+0000","last_unstale":"2024-09-19T08:03:00.849208+0000","last_undegraded":"2024-09-19T08:03:00.849208+0000","last_fullsized":"2024-09-19T08:03:00.849208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:17:39.861820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849073+0000","last_change":"2024-09-19T08:01:10.768949+0000","last_active":"2024-09-19T08:03:00.849073+0000","last_peered":"2024-09-19T08:03:00.849073+0000","last_clean":"2024-09-19T08:03:00.849073+0000","last_became_active":"2024-09-19T08:01:10.768159+0000","last_became_peered":"2024-09-19T08:01:10.768159+0000","last_unstale":"2024-09-19T08:03:00.849073+0000","last_undegraded":"2024-09-19T08:03:00.849073+0000","last_fullsized":"2024-09-19T08:03:00.849073+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:43.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"78'13","reported_seq":163,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844272+0000","last_change":"2024-09-19T08:01:26.735877+0000","last_active":"2024-09-19T08:03:00.844272+0000","last_peered":"2024-09-19T08:03:00.844272+0000","last_clean":"2024-09-19T08:03:00.844272+0000","last_became_active":"2024-09-19T08:01:26.723597+0000","last_became_peered":"2024-09-19T08:01:26.723597+0000","last_unstale":"2024-09-19T08:03:00.844272+0000","last_undegraded":"2024-09-19T08:03:00.844272+0000","last_fullsized":"2024-09-19T08:03:00.844272+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:43:42.340832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":99,"num_read_kb":99,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,1],"acting":[9,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600666,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1139,"num_read_kb":683,"num_write":372,"num_write_kb":1463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1254,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1801998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645347,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":905216,"data_stored":436392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874280,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32940,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712212,"statfs":{"total":95995035648,"available":95961305088,"internally_reserved":0,"allocated":1736704,"data_stored":1224499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005101,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32404,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712748,"statfs":{"total":95995035648,"available":95961853952,"internally_reserved":0,"allocated":1245184,"data_stored":762214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234033,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1294336,"data_stored":826874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462966,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1503232,"data_stored":1028781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593788,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36228,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708924,"statfs":{"total":95995035648,"available":95957938176,"internally_reserved":0,"allocated":901120,"data_stored":434251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822720,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1658880,"data_stored":1159945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986245,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":901120,"data_stored":434112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215177,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32656,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712496,"statfs":{"total":95995035648,"available":95961595904,"internally_reserved":0,"allocated":1445888,"data_stored":958892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476815,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":504026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607636,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":978944,"data_stored":502316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":921600,"data_stored":436124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":136048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":200602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T08:03:05.537 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-09-19T08:03:05.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T08:03:05.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.675+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/1328730381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cb8102cf0 msgr2=0x7f6cb81030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.675+0000 7f6cbdf9f640 1 --2- 172.21.15.98:0/1328730381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cb8102cf0 0x7f6cb81030d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009900 tx=0x7f6ca802a480 comp rx=0 tx=0).stop 2024-09-19T08:03:05.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.676+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/1328730381 shutdown_connections 2024-09-19T08:03:05.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.676+0000 7f6cbdf9f640 1 --2- 172.21.15.98:0/1328730381 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cb810c0c0 0x7f6cb810e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.676+0000 7f6cbdf9f640 1 --2- 172.21.15.98:0/1328730381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb8103610 0x7f6cb810baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.676+0000 7f6cbdf9f640 1 --2- 172.21.15.98:0/1328730381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cb8102cf0 0x7f6cb81030d0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.676+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/1328730381 >> 172.21.15.98:0/1328730381 conn(0x7f6cb8077a10 msgr2=0x7f6cb80fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:05.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.676+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/1328730381 shutdown_connections 2024-09-19T08:03:05.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.676+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/1328730381 wait complete. 2024-09-19T08:03:05.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.676+0000 7f6cbdf9f640 1 Processor -- start 2024-09-19T08:03:05.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.676+0000 7f6cbdf9f640 1 -- start start 2024-09-19T08:03:05.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cbdf9f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb8102cf0 0x7f6cb8198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:05.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cbdf9f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cb8103610 0x7f6cb8198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:05.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cbdf9f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cb810c0c0 0x7f6cb819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:05.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cbdf9f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb8111ef0 con 0x7f6cb8103610 2024-09-19T08:03:05.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cbdf9f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb8111d70 con 0x7f6cb810c0c0 2024-09-19T08:03:05.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cbdf9f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb8112070 con 0x7f6cb8102cf0 2024-09-19T08:03:05.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cb8103610 0x7f6cb8198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:05.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cb8103610 0x7f6cb8198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55186/0 (socket says 172.21.15.98:55186) 2024-09-19T08:03:05.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb6ffd640 1 -- 172.21.15.98:0/2340286957 learned_addr learned my addr 172.21.15.98:0/2340286957 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:05.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb77fe640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb8102cf0 0x7f6cb8198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:05.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb7fff640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cb810c0c0 0x7f6cb819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:05.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb6ffd640 1 -- 172.21.15.98:0/2340286957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb8102cf0 msgr2=0x7f6cb8198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb6ffd640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb8102cf0 0x7f6cb8198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb6ffd640 1 -- 172.21.15.98:0/2340286957 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cb810c0c0 msgr2=0x7f6cb819ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb6ffd640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cb810c0c0 0x7f6cb819ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb6ffd640 1 -- 172.21.15.98:0/2340286957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb819d3f0 con 0x7f6cb8103610 2024-09-19T08:03:05.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb7fff640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cb810c0c0 0x7f6cb819ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:05.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb6ffd640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cb8103610 0x7f6cb8198940 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6cac00d770 tx=0x7f6cac00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:05.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb4ff9640 1 -- 172.21.15.98:0/2340286957 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cac017070 con 0x7f6cb8103610 2024-09-19T08:03:05.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/2340286957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cb819d6e0 con 0x7f6cb8103610 2024-09-19T08:03:05.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb4ff9640 1 -- 172.21.15.98:0/2340286957 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cac005500 con 0x7f6cb8103610 2024-09-19T08:03:05.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cb4ff9640 1 -- 172.21.15.98:0/2340286957 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cac004e00 con 0x7f6cb8103610 2024-09-19T08:03:05.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.677+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/2340286957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cb81a4f70 con 0x7f6cb8103610 2024-09-19T08:03:05.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.678+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/2340286957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cb81030d0 con 0x7f6cb8103610 2024-09-19T08:03:05.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.680+0000 7f6cb4ff9640 1 -- 172.21.15.98:0/2340286957 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f6cac004fa0 con 0x7f6cb8103610 2024-09-19T08:03:05.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.680+0000 7f6cb4ff9640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6ca00a34e0 0x7f6ca00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:05.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.680+0000 7f6cb77fe640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6ca00a34e0 0x7f6ca00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:05.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.680+0000 7f6cb4ff9640 1 -- 172.21.15.98:0/2340286957 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f6cac0cfe80 con 0x7f6cb8103610 2024-09-19T08:03:05.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.681+0000 7f6cb77fe640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6ca00a34e0 0x7f6ca00a59a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8005ab0 tx=0x7f6ca8005a20 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:05.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.683+0000 7f6cb4ff9640 1 -- 172.21.15.98:0/2340286957 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f6cac09c640 con 0x7f6cb8103610 2024-09-19T08:03:05.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.849+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/2340286957 --> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6cb8062630 con 0x7f6ca00a34e0 2024-09-19T08:03:05.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.855+0000 7f6cb4ff9640 1 -- 172.21.15.98:0/2340286957 <== mgr.15252 v2:172.21.15.98:6800/1961446727 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257946 (secure 0 0 0) 0x7f6cb8062630 con 0x7f6ca00a34e0 2024-09-19T08:03:05.859 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-19T08:03:05.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.858+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/2340286957 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6ca00a34e0 msgr2=0x7f6ca00a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.858+0000 7f6cbdf9f640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6ca00a34e0 0x7f6ca00a59a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8005ab0 tx=0x7f6ca8005a20 comp rx=0 tx=0).stop 2024-09-19T08:03:05.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.858+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/2340286957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cb8103610 msgr2=0x7f6cb8198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:05.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.858+0000 7f6cbdf9f640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cb8103610 0x7f6cb8198940 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6cac00d770 tx=0x7f6cac00dc30 comp rx=0 tx=0).stop 2024-09-19T08:03:05.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.858+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/2340286957 shutdown_connections 2024-09-19T08:03:05.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.858+0000 7f6cbdf9f640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6ca00a34e0 0x7f6ca00a59a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.858+0000 7f6cbdf9f640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cb810c0c0 0x7f6cb819ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.858+0000 7f6cbdf9f640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cb8103610 0x7f6cb8198940 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.858+0000 7f6cbdf9f640 1 --2- 172.21.15.98:0/2340286957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb8102cf0 0x7f6cb8198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:05.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.858+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/2340286957 >> 172.21.15.98:0/2340286957 conn(0x7f6cb8077a10 msgr2=0x7f6cb80fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:05.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.858+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/2340286957 shutdown_connections 2024-09-19T08:03:05.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:05.858+0000 7f6cbdf9f640 1 -- 172.21.15.98:0/2340286957 wait complete. 2024-09-19T08:03:05.869 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:05 smithi098 ceph-mon[31333]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-09-19T08:03:05.869 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:05 smithi098 ceph-mon[31333]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-19T08:03:05.869 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:05 smithi098 ceph-mon[31333]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-19T08:03:05.869 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:05 smithi098 ceph-mon[31333]: Updating smithi139:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T08:03:05.869 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:05 smithi098 ceph-mon[31333]: Updating smithi162:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T08:03:05.869 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:05 smithi098 ceph-mon[31333]: Updating smithi098:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T08:03:05.870 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:05 smithi098 ceph-mon[31333]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:03:05.870 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:05 smithi098 ceph-mon[31333]: mgrmap e22: x(active, since 4s), standbys: z, y 2024-09-19T08:03:05.870 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-09-19T08:03:04.771716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191268,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1275,"num_read_kb":802,"num_write":555,"num_write_kb":4403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181990,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":939,"ondisk_log_size":939,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376028,"kb_used_data":14144,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565796,"statfs":{"total":1151940427776,"available":1151555375104,"internally_reserved":0,"allocated":14483456,"data_stored":8708426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":370193724},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.989911"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780911+0000","last_change":"2024-09-19T08:01:10.768950+0000","last_active":"2024-09-19T08:03:00.780911+0000","last_peered":"2024-09-19T08:03:00.780911+0000","last_clean":"2024-09-19T08:03:00.780911+0000","last_became_active":"2024-09-19T08:01:10.768047+0000","last_became_peered":"2024-09-19T08:01:10.768047+0000","last_unstale":"2024-09-19T08:03:00.780911+0000","last_undegraded":"2024-09-19T08:03:00.780911+0000","last_fullsized":"2024-09-19T08:03:00.780911+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:26.219079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994728+0000","last_change":"2024-09-19T08:01:10.783208+0000","last_active":"2024-09-19T08:03:00.994728+0000","last_peered":"2024-09-19T08:03:00.994728+0000","last_clean":"2024-09-19T08:03:00.994728+0000","last_became_active":"2024-09-19T08:01:10.782888+0000","last_became_peered":"2024-09-19T08:01:10.782888+0000","last_unstale":"2024-09-19T08:03:00.994728+0000","last_undegraded":"2024-09-19T08:03:00.994728+0000","last_fullsized":"2024-09-19T08:03:00.994728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:51:27.603166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994427+0000","last_change":"2024-09-19T08:01:10.770395+0000","last_active":"2024-09-19T08:03:00.994427+0000","last_peered":"2024-09-19T08:03:00.994427+0000","last_clean":"2024-09-19T08:03:00.994427+0000","last_became_active":"2024-09-19T08:01:10.770073+0000","last_became_peered":"2024-09-19T08:01:10.770073+0000","last_unstale":"2024-09-19T08:03:00.994427+0000","last_undegraded":"2024-09-19T08:03:00.994427+0000","last_fullsized":"2024-09-19T08:03:00.994427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:20.047023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761373+0000","last_change":"2024-09-19T08:01:10.777138+0000","last_active":"2024-09-19T08:03:00.761373+0000","last_peered":"2024-09-19T08:03:00.761373+0000","last_clean":"2024-09-19T08:03:00.761373+0000","last_became_active":"2024-09-19T08:01:10.776810+0000","last_became_peered":"2024-09-19T08:01:10.776810+0000","last_unstale":"2024-09-19T08:03:00.761373+0000","last_undegraded":"2024-09-19T08:03:00.761373+0000","last_fullsized":"2024-09-19T08:03:00.761373+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:39.983321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847497+0000","last_change":"2024-09-19T08:01:10.770435+0000","last_active":"2024-09-19T08:03:00.847497+0000","last_peered":"2024-09-19T08:03:00.847497+0000","last_clean":"2024-09-19T08:03:00.847497+0000","last_became_active":"2024-09-19T08:01:10.769686+0000","last_became_peered":"2024-09-19T08:01:10.769686+0000","last_unstale":"2024-09-19T08:03:00.847497+0000","last_undegraded":"2024-09-19T08:03:00.847497+0000","last_fullsized":"2024-09-19T08:03:00.847497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:10:19.610925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839257+0000","last_change":"2024-09-19T08:01:10.782382+0000","last_active":"2024-09-19T08:03:00.839257+0000","last_peered":"2024-09-19T08:03:00.839257+0000","last_clean":"2024-09-19T08:03:00.839257+0000","last_became_active":"2024-09-19T08:01:10.781996+0000","last_became_peered":"2024-09-19T08:01:10.781996+0000","last_unstale":"2024-09-19T08:03:00.839257+0000","last_undegraded":"2024-09-19T08:03:00.839257+0000","last_fullsized":"2024-09-19T08:03:00.839257+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:43:13.574336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837718+0000","last_change":"2024-09-19T08:01:10.773776+0000","last_active":"2024-09-19T08:03:00.837718+0000","last_peered":"2024-09-19T08:03:00.837718+0000","last_clean":"2024-09-19T08:03:00.837718+0000","last_became_active":"2024-09-19T08:01:10.772442+0000","last_became_peered":"2024-09-19T08:01:10.772442+0000","last_unstale":"2024-09-19T08:03:00.837718+0000","last_undegraded":"2024-09-19T08:03:00.837718+0000","last_fullsized":"2024-09-19T08:03:00.837718+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:08:18.155806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848134+0000","last_change":"2024-09-19T08:01:10.781443+0000","last_active":"2024-09-19T08:03:00.848134+0000","last_peered":"2024-09-19T08:03:00.848134+0000","last_clean":"2024-09-19T08:03:00.848134+0000","last_became_active":"2024-09-19T08:01:10.781083+0000","last_became_peered":"2024-09-19T08:01:10.781083+0000","last_unstale":"2024-09-19T08:03:00.848134+0000","last_undegraded":"2024-09-19T08:03:00.848134+0000","last_fullsized":"2024-09-19T08:03:00.848134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:23:22.631865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.998365+0000","last_change":"2024-09-19T08:01:10.767554+0000","last_active":"2024-09-19T08:03:00.998365+0000","last_peered":"2024-09-19T08:03:00.998365+0000","last_clean":"2024-09-19T08:03:00.998365+0000","last_became_active":"2024-09-19T08:01:10.767227+0000","last_became_peered":"2024-09-19T08:01:10.767227+0000","last_unstale":"2024-09-19T08:03:00.998365+0000","last_undegraded":"2024-09-19T08:03:00.998365+0000","last_fullsized":"2024-09-19T08:03:00.998365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:30:13.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839361+0000","last_change":"2024-09-19T08:01:10.783174+0000","last_active":"2024-09-19T08:03:00.839361+0000","last_peered":"2024-09-19T08:03:00.839361+0000","last_clean":"2024-09-19T08:03:00.839361+0000","last_became_active":"2024-09-19T08:01:10.782794+0000","last_became_peered":"2024-09-19T08:01:10.782794+0000","last_unstale":"2024-09-19T08:03:00.839361+0000","last_undegraded":"2024-09-19T08:03:00.839361+0000","last_fullsized":"2024-09-19T08:03:00.839361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:49:19.451107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838191+0000","last_change":"2024-09-19T08:01:10.779638+0000","last_active":"2024-09-19T08:03:00.838191+0000","last_peered":"2024-09-19T08:03:00.838191+0000","last_clean":"2024-09-19T08:03:00.838191+0000","last_became_active":"2024-09-19T08:01:10.779295+0000","last_became_peered":"2024-09-19T08:01:10.779295+0000","last_unstale":"2024-09-19T08:03:00.838191+0000","last_undegraded":"2024-09-19T08:03:00.838191+0000","last_fullsized":"2024-09-19T08:03:00.838191+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:35:10.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761539+0000","last_change":"2024-09-19T08:01:10.784487+0000","last_active":"2024-09-19T08:03:00.761539+0000","last_peered":"2024-09-19T08:03:00.761539+0000","last_clean":"2024-09-19T08:03:00.761539+0000","last_became_active":"2024-09-19T08:01:10.784114+0000","last_became_peered":"2024-09-19T08:01:10.784114+0000","last_unstale":"2024-09-19T08:03:00.761539+0000","last_undegraded":"2024-09-19T08:03:00.761539+0000","last_fullsized":"2024-09-19T08:03:00.761539+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:09:54.819802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758888+0000","last_change":"2024-09-19T08:01:10.783186+0000","last_active":"2024-09-19T08:03:00.758888+0000","last_peered":"2024-09-19T08:03:00.758888+0000","last_clean":"2024-09-19T08:03:00.758888+0000","last_became_active":"2024-09-19T08:01:10.782876+0000","last_became_peered":"2024-09-19T08:01:10.782876+0000","last_unstale":"2024-09-19T08:03:00.758888+0000","last_undegraded":"2024-09-19T08:03:00.758888+0000","last_fullsized":"2024-09-19T08:03:00.758888+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:26:15.252165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994474+0000","last_change":"2024-09-19T08:01:10.783237+0000","last_active":"2024-09-19T08:03:00.994474+0000","last_peered":"2024-09-19T08:03:00.994474+0000","last_clean":"2024-09-19T08:03:00.994474+0000","last_became_active":"2024-09-19T08:01:10.782966+0000","last_became_peered":"2024-09-19T08:01:10.782966+0000","last_unstale":"2024-09-19T08:03:00.994474+0000","last_undegraded":"2024-09-19T08:03:00.994474+0000","last_fullsized":"2024-09-19T08:03:00.994474+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:57:18.771737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994242+0000","last_change":"2024-09-19T08:01:10.771207+0000","last_active":"2024-09-19T08:03:00.994242+0000","last_peered":"2024-09-19T08:03:00.994242+0000","last_clean":"2024-09-19T08:03:00.994242+0000","last_became_active":"2024-09-19T08:01:10.770862+0000","last_became_peered":"2024-09-19T08:01:10.770862+0000","last_unstale":"2024-09-19T08:03:00.994242+0000","last_undegraded":"2024-09-19T08:03:00.994242+0000","last_fullsized":"2024-09-19T08:03:00.994242+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:08:09.313329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997263+0000","last_change":"2024-09-19T08:01:10.775191+0000","last_active":"2024-09-19T08:03:00.997263+0000","last_peered":"2024-09-19T08:03:00.997263+0000","last_clean":"2024-09-19T08:03:00.997263+0000","last_became_active":"2024-09-19T08:01:10.774832+0000","last_became_peered":"2024-09-19T08:01:10.774832+0000","last_unstale":"2024-09-19T08:03:00.997263+0000","last_undegraded":"2024-09-19T08:03:00.997263+0000","last_fullsized":"2024-09-19T08:03:00.997263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:57:29.800787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780968+0000","last_change":"2024-09-19T08:01:10.769115+0000","last_active":"2024-09-19T08:03:00.780968+0000","last_peered":"2024-09-19T08:03:00.780968+0000","last_clean":"2024-09-19T08:03:00.780968+0000","last_became_active":"2024-09-19T08:01:10.768088+0000","last_became_peered":"2024-09-19T08:01:10.768088+0000","last_unstale":"2024-09-19T08:03:00.780968+0000","last_undegraded":"2024-09-19T08:03:00.780968+0000","last_fullsized":"2024-09-19T08:03:00.780968+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:27:16.042722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844124+0000","last_change":"2024-09-19T08:01:10.781426+0000","last_active":"2024-09-19T08:03:00.844124+0000","last_peered":"2024-09-19T08:03:00.844124+0000","last_clean":"2024-09-19T08:03:00.844124+0000","last_became_active":"2024-09-19T08:01:10.781094+0000","last_became_peered":"2024-09-19T08:01:10.781094+0000","last_unstale":"2024-09-19T08:03:00.844124+0000","last_undegraded":"2024-09-19T08:03:00.844124+0000","last_fullsized":"2024-09-19T08:03:00.844124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:43:31.336610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760020+0000","last_change":"2024-09-19T08:01:10.766466+0000","last_active":"2024-09-19T08:03:00.760020+0000","last_peered":"2024-09-19T08:03:00.760020+0000","last_clean":"2024-09-19T08:03:00.760020+0000","last_became_active":"2024-09-19T08:01:10.766123+0000","last_became_peered":"2024-09-19T08:01:10.766123+0000","last_unstale":"2024-09-19T08:03:00.760020+0000","last_undegraded":"2024-09-19T08:03:00.760020+0000","last_fullsized":"2024-09-19T08:03:00.760020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:25.848603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780633+0000","last_change":"2024-09-19T08:01:10.772782+0000","last_active":"2024-09-19T08:03:00.780633+0000","last_peered":"2024-09-19T08:03:00.780633+0000","last_clean":"2024-09-19T08:03:00.780633+0000","last_became_active":"2024-09-19T08:01:10.772445+0000","last_became_peered":"2024-09-19T08:01:10.772445+0000","last_unstale":"2024-09-19T08:03:00.780633+0000","last_undegraded":"2024-09-19T08:03:00.780633+0000","last_fullsized":"2024-09-19T08:03:00.780633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:09.174005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759047+0000","last_change":"2024-09-19T08:01:10.771944+0000","last_active":"2024-09-19T08:03:00.759047+0000","last_peered":"2024-09-19T08:03:00.759047+0000","last_clean":"2024-09-19T08:03:00.759047+0000","last_became_active":"2024-09-19T08:01:10.771643+0000","last_became_peered":"2024-09-19T08:01:10.771643+0000","last_unstale":"2024-09-19T08:03:00.759047+0000","last_undegraded":"2024-09-19T08:03:00.759047+0000","last_fullsized":"2024-09-19T08:03:00.759047+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:46:27.121722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997451+0000","last_change":"2024-09-19T08:01:10.775144+0000","last_active":"2024-09-19T08:03:00.997451+0000","last_peered":"2024-09-19T08:03:00.997451+0000","last_clean":"2024-09-19T08:03:00.997451+0000","last_became_active":"2024-09-19T08:01:10.774769+0000","last_became_peered":"2024-09-19T08:01:10.774769+0000","last_unstale":"2024-09-19T08:03:00.997451+0000","last_undegraded":"2024-09-19T08:03:00.997451+0000","last_fullsized":"2024-09-19T08:03:00.997451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:59:47.053993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848212+0000","last_change":"2024-09-19T08:01:10.766750+0000","last_active":"2024-09-19T08:03:00.848212+0000","last_peered":"2024-09-19T08:03:00.848212+0000","last_clean":"2024-09-19T08:03:00.848212+0000","last_became_active":"2024-09-19T08:01:10.766450+0000","last_became_peered":"2024-09-19T08:01:10.766450+0000","last_unstale":"2024-09-19T08:03:00.848212+0000","last_undegraded":"2024-09-19T08:03:00.848212+0000","last_fullsized":"2024-09-19T08:03:00.848212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:47:58.133815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841186+0000","last_change":"2024-09-19T08:01:10.768430+0000","last_active":"2024-09-19T08:03:00.841186+0000","last_peered":"2024-09-19T08:03:00.841186+0000","last_clean":"2024-09-19T08:03:00.841186+0000","last_became_active":"2024-09-19T08:01:10.768092+0000","last_became_peered":"2024-09-19T08:01:10.768092+0000","last_unstale":"2024-09-19T08:03:00.841186+0000","last_undegraded":"2024-09-19T08:03:00.841186+0000","last_fullsized":"2024-09-19T08:03:00.841186+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:16:42.343138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849206+0000","last_change":"2024-09-19T08:01:10.771131+0000","last_active":"2024-09-19T08:03:00.849206+0000","last_peered":"2024-09-19T08:03:00.849206+0000","last_clean":"2024-09-19T08:03:00.849206+0000","last_became_active":"2024-09-19T08:01:10.770851+0000","last_became_peered":"2024-09-19T08:01:10.770851+0000","last_unstale":"2024-09-19T08:03:00.849206+0000","last_undegraded":"2024-09-19T08:03:00.849206+0000","last_fullsized":"2024-09-19T08:03:00.849206+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:07:15.830476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760273+0000","last_change":"2024-09-19T08:01:10.776220+0000","last_active":"2024-09-19T08:03:00.760273+0000","last_peered":"2024-09-19T08:03:00.760273+0000","last_clean":"2024-09-19T08:03:00.760273+0000","last_became_active":"2024-09-19T08:01:10.775889+0000","last_became_peered":"2024-09-19T08:01:10.775889+0000","last_unstale":"2024-09-19T08:03:00.760273+0000","last_undegraded":"2024-09-19T08:03:00.760273+0000","last_fullsized":"2024-09-19T08:03:00.760273+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:23.714675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780731+0000","last_change":"2024-09-19T08:01:10.776370+0000","last_active":"2024-09-19T08:03:00.780731+0000","last_peered":"2024-09-19T08:03:00.780731+0000","last_clean":"2024-09-19T08:03:00.780731+0000","last_became_active":"2024-09-19T08:01:10.776027+0000","last_became_peered":"2024-09-19T08:01:10.776027+0000","last_unstale":"2024-09-19T08:03:00.780731+0000","last_undegraded":"2024-09-19T08:03:00.780731+0000","last_fullsized":"2024-09-19T08:03:00.780731+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:48:46.293404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758951+0000","last_change":"2024-09-19T08:01:26.758971+0000","last_active":"2024-09-19T08:03:00.758951+0000","last_peered":"2024-09-19T08:03:00.758951+0000","last_clean":"2024-09-19T08:03:00.758951+0000","last_became_active":"2024-09-19T08:01:26.758524+0000","last_became_peered":"2024-09-19T08:01:26.758524+0000","last_unstale":"2024-09-19T08:03:00.758951+0000","last_undegraded":"2024-09-19T08:03:00.758951+0000","last_fullsized":"2024-09-19T08:03:00.758951+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:34:22.439327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994656+0000","last_change":"2024-09-19T08:01:10.767445+0000","last_active":"2024-09-19T08:03:00.994656+0000","last_peered":"2024-09-19T08:03:00.994656+0000","last_clean":"2024-09-19T08:03:00.994656+0000","last_became_active":"2024-09-19T08:01:10.767128+0000","last_became_peered":"2024-09-19T08:01:10.767128+0000","last_unstale":"2024-09-19T08:03:00.994656+0000","last_undegraded":"2024-09-19T08:03:00.994656+0000","last_fullsized":"2024-09-19T08:03:00.994656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:10.273948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759852+0000","last_change":"2024-09-19T08:01:10.768954+0000","last_active":"2024-09-19T08:03:00.759852+0000","last_peered":"2024-09-19T08:03:00.759852+0000","last_clean":"2024-09-19T08:03:00.759852+0000","last_became_active":"2024-09-19T08:01:10.768577+0000","last_became_peered":"2024-09-19T08:01:10.768577+0000","last_unstale":"2024-09-19T08:03:00.759852+0000","last_undegraded":"2024-09-19T08:03:00.759852+0000","last_fullsized":"2024-09-19T08:03:00.759852+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:20:21.812531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759843+0000","last_change":"2024-09-19T08:01:10.774104+0000","last_active":"2024-09-19T08:03:00.759843+0000","last_peered":"2024-09-19T08:03:00.759843+0000","last_clean":"2024-09-19T08:03:00.759843+0000","last_became_active":"2024-09-19T08:01:10.773721+0000","last_became_peered":"2024-09-19T08:01:10.773721+0000","last_unstale":"2024-09-19T08:03:00.759843+0000","last_undegraded":"2024-09-19T08:03:00.759843+0000","last_fullsized":"2024-09-19T08:03:00.759843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:45:34.945994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780513+0000","last_change":"2024-09-19T08:01:26.727296+0000","last_active":"2024-09-19T08:03:00.780513+0000","last_peered":"2024-09-19T08:03:00.780513+0000","last_clean":"2024-09-19T08:03:00.780513+0000","last_became_active":"2024-09-19T08:01:26.726631+0000","last_became_peered":"2024-09-19T08:01:26.726631+0000","last_unstale":"2024-09-19T08:03:00.780513+0000","last_undegraded":"2024-09-19T08:03:00.780513+0000","last_fullsized":"2024-09-19T08:03:00.780513+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:19:24.807050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847424+0000","last_change":"2024-09-19T08:01:10.765282+0000","last_active":"2024-09-19T08:03:00.847424+0000","last_peered":"2024-09-19T08:03:00.847424+0000","last_clean":"2024-09-19T08:03:00.847424+0000","last_became_active":"2024-09-19T08:01:10.764928+0000","last_became_peered":"2024-09-19T08:01:10.764928+0000","last_unstale":"2024-09-19T08:03:00.847424+0000","last_undegraded":"2024-09-19T08:03:00.847424+0000","last_fullsized":"2024-09-19T08:03:00.847424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:00:37.395110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"81'10","reported_seq":140,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.999561+0000","last_change":"2024-09-19T08:01:08.760686+0000","last_active":"2024-09-19T08:03:00.999561+0000","last_peered":"2024-09-19T08:03:00.999561+0000","last_clean":"2024-09-19T08:03:00.999561+0000","last_became_active":"2024-09-19T08:01:08.759912+0000","last_became_peered":"2024-09-19T08:01:08.759912+0000","last_unstale":"2024-09-19T08:03:00.999561+0000","last_undegraded":"2024-09-19T08:03:00.999561+0000","last_fullsized":"2024-09-19T08:03:00.999561+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:46:01.879611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2250,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":5,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841395+0000","last_change":"2024-09-19T08:01:10.779668+0000","last_active":"2024-09-19T08:03:00.841395+0000","last_peered":"2024-09-19T08:03:00.841395+0000","last_clean":"2024-09-19T08:03:00.841395+0000","last_became_active":"2024-09-19T08:01:10.779265+0000","last_became_peered":"2024-09-19T08:01:10.779265+0000","last_unstale":"2024-09-19T08:03:00.841395+0000","last_undegraded":"2024-09-19T08:03:00.841395+0000","last_fullsized":"2024-09-19T08:03:00.841395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:06.177700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838967+0000","last_change":"2024-09-19T08:01:08.762668+0000","last_active":"2024-09-19T08:03:00.838967+0000","last_peered":"2024-09-19T08:03:00.838967+0000","last_clean":"2024-09-19T08:03:00.838967+0000","last_became_active":"2024-09-19T08:01:08.762277+0000","last_became_peered":"2024-09-19T08:01:08.762277+0000","last_unstale":"2024-09-19T08:03:00.838967+0000","last_undegraded":"2024-09-19T08:03:00.838967+0000","last_fullsized":"2024-09-19T08:03:00.838967+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:36:59.316655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758585+0000","last_change":"2024-09-19T08:01:08.757401+0000","last_active":"2024-09-19T08:03:00.758585+0000","last_peered":"2024-09-19T08:03:00.758585+0000","last_clean":"2024-09-19T08:03:00.758585+0000","last_became_active":"2024-09-19T08:01:08.757067+0000","last_became_peered":"2024-09-19T08:01:08.757067+0000","last_unstale":"2024-09-19T08:03:00.758585+0000","last_undegraded":"2024-09-19T08:03:00.758585+0000","last_fullsized":"2024-09-19T08:03:00.758585+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:19.617830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848289+0000","last_change":"2024-09-19T08:01:10.765149+0000","last_active":"2024-09-19T08:03:00.848289+0000","last_peered":"2024-09-19T08:03:00.848289+0000","last_clean":"2024-09-19T08:03:00.848289+0000","last_became_active":"2024-09-19T08:01:10.764840+0000","last_became_peered":"2024-09-19T08:01:10.764840+0000","last_unstale":"2024-09-19T08:03:00.848289+0000","last_undegraded":"2024-09-19T08:03:00.848289+0000","last_fullsized":"2024-09-19T08:03:00.848289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:15.539307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841355+0000","last_change":"2024-09-19T08:01:08.759435+0000","last_active":"2024-09-19T08:03:00.841355+0000","last_peered":"2024-09-19T08:03:00.841355+0000","last_clean":"2024-09-19T08:03:00.841355+0000","last_became_active":"2024-09-19T08:01:08.759046+0000","last_became_peered":"2024-09-19T08:01:08.759046+0000","last_unstale":"2024-09-19T08:03:00.841355+0000","last_undegraded":"2024-09-19T08:03:00.841355+0000","last_fullsized":"2024-09-19T08:03:00.841355+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:21:24.077626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.843896+0000","last_change":"2024-09-19T08:01:10.778432+0000","last_active":"2024-09-19T08:03:00.843896+0000","last_peered":"2024-09-19T08:03:00.843896+0000","last_clean":"2024-09-19T08:03:00.843896+0000","last_became_active":"2024-09-19T08:01:10.778098+0000","last_became_peered":"2024-09-19T08:01:10.778098+0000","last_unstale":"2024-09-19T08:03:00.843896+0000","last_undegraded":"2024-09-19T08:03:00.843896+0000","last_fullsized":"2024-09-19T08:03:00.843896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:40:12.921165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":230,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839698+0000","last_change":"2024-09-19T08:01:08.764117+0000","last_active":"2024-09-19T08:03:00.839698+0000","last_peered":"2024-09-19T08:03:00.839698+0000","last_clean":"2024-09-19T08:03:00.839698+0000","last_became_active":"2024-09-19T08:01:08.763308+0000","last_became_peered":"2024-09-19T08:01:08.763308+0000","last_unstale":"2024-09-19T08:03:00.839698+0000","last_undegraded":"2024-09-19T08:03:00.839698+0000","last_fullsized":"2024-09-19T08:03:00.839698+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:19:57.014628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":200,"num_read_kb":200,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997921+0000","last_change":"2024-09-19T08:01:10.781771+0000","last_active":"2024-09-19T08:03:00.997921+0000","last_peered":"2024-09-19T08:03:00.997921+0000","last_clean":"2024-09-19T08:03:00.997921+0000","last_became_active":"2024-09-19T08:01:10.781432+0000","last_became_peered":"2024-09-19T08:01:10.781432+0000","last_unstale":"2024-09-19T08:03:00.997921+0000","last_undegraded":"2024-09-19T08:03:00.997921+0000","last_fullsized":"2024-09-19T08:03:00.997921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:25:15.067850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838023+0000","last_change":"2024-09-19T08:01:26.771171+0000","last_active":"2024-09-19T08:03:00.838023+0000","last_peered":"2024-09-19T08:03:00.838023+0000","last_clean":"2024-09-19T08:03:00.838023+0000","last_became_active":"2024-09-19T08:01:26.759274+0000","last_became_peered":"2024-09-19T08:01:26.759274+0000","last_unstale":"2024-09-19T08:03:00.838023+0000","last_undegraded":"2024-09-19T08:03:00.838023+0000","last_fullsized":"2024-09-19T08:03:00.838023+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:20:54.603420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994926+0000","last_change":"2024-09-19T08:01:10.768318+0000","last_active":"2024-09-19T08:03:00.994926+0000","last_peered":"2024-09-19T08:03:00.994926+0000","last_clean":"2024-09-19T08:03:00.994926+0000","last_became_active":"2024-09-19T08:01:10.768022+0000","last_became_peered":"2024-09-19T08:01:10.768022+0000","last_unstale":"2024-09-19T08:03:00.994926+0000","last_undegraded":"2024-09-19T08:03:00.994926+0000","last_fullsized":"2024-09-19T08:03:00.994926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:00:45.328638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841594+0000","last_change":"2024-09-19T08:01:09.750530+0000","last_active":"2024-09-19T08:03:00.841594+0000","last_peered":"2024-09-19T08:03:00.841594+0000","last_clean":"2024-09-19T08:03:00.841594+0000","last_became_active":"2024-09-19T08:01:09.750029+0000","last_became_peered":"2024-09-19T08:01:09.750029+0000","last_unstale":"2024-09-19T08:03:00.841594+0000","last_undegraded":"2024-09-19T08:03:00.841594+0000","last_fullsized":"2024-09-19T08:03:00.841594+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:01:16.062992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997659+0000","last_change":"2024-09-19T08:01:26.724636+0000","last_active":"2024-09-19T08:03:00.997659+0000","last_peered":"2024-09-19T08:03:00.997659+0000","last_clean":"2024-09-19T08:03:00.997659+0000","last_became_active":"2024-09-19T08:01:26.724171+0000","last_became_peered":"2024-09-19T08:01:26.724171+0000","last_unstale":"2024-09-19T08:03:00.997659+0000","last_undegraded":"2024-09-19T08:03:00.997659+0000","last_fullsized":"2024-09-19T08:03:00.997659+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:54:05.998374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'37","reported_seq":275,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997077+0000","last_change":"2024-09-19T08:01:08.761950+0000","last_active":"2024-09-19T08:03:00.997077+0000","last_peered":"2024-09-19T08:03:00.997077+0000","last_clean":"2024-09-19T08:03:00.997077+0000","last_became_active":"2024-09-19T08:01:08.761533+0000","last_became_peered":"2024-09-19T08:01:08.761533+0000","last_unstale":"2024-09-19T08:03:00.997077+0000","last_undegraded":"2024-09-19T08:03:00.997077+0000","last_fullsized":"2024-09-19T08:03:00.997077+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:07:26.668141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66480,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":214,"num_read_kb":171,"num_write":39,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847631+0000","last_change":"2024-09-19T08:01:10.774280+0000","last_active":"2024-09-19T08:03:00.847631+0000","last_peered":"2024-09-19T08:03:00.847631+0000","last_clean":"2024-09-19T08:03:00.847631+0000","last_became_active":"2024-09-19T08:01:10.773982+0000","last_became_peered":"2024-09-19T08:01:10.773982+0000","last_unstale":"2024-09-19T08:03:00.847631+0000","last_undegraded":"2024-09-19T08:03:00.847631+0000","last_fullsized":"2024-09-19T08:03:00.847631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:21:43.180057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760058+0000","last_change":"2024-09-19T08:01:08.761067+0000","last_active":"2024-09-19T08:03:00.760058+0000","last_peered":"2024-09-19T08:03:00.760058+0000","last_clean":"2024-09-19T08:03:00.760058+0000","last_became_active":"2024-09-19T08:01:08.760664+0000","last_became_peered":"2024-09-19T08:01:08.760664+0000","last_unstale":"2024-09-19T08:03:00.760058+0000","last_undegraded":"2024-09-19T08:03:00.760058+0000","last_fullsized":"2024-09-19T08:03:00.760058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:22:38.086891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758630+0000","last_change":"2024-09-19T08:01:10.771845+0000","last_active":"2024-09-19T08:03:00.758630+0000","last_peered":"2024-09-19T08:03:00.758630+0000","last_clean":"2024-09-19T08:03:00.758630+0000","last_became_active":"2024-09-19T08:01:10.771533+0000","last_became_peered":"2024-09-19T08:01:10.771533+0000","last_unstale":"2024-09-19T08:03:00.758630+0000","last_undegraded":"2024-09-19T08:03:00.758630+0000","last_fullsized":"2024-09-19T08:03:00.758630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:29:35.180763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841267+0000","last_change":"2024-09-19T08:01:08.763869+0000","last_active":"2024-09-19T08:03:00.841267+0000","last_peered":"2024-09-19T08:03:00.841267+0000","last_clean":"2024-09-19T08:03:00.841267+0000","last_became_active":"2024-09-19T08:01:08.763451+0000","last_became_peered":"2024-09-19T08:01:08.763451+0000","last_unstale":"2024-09-19T08:03:00.841267+0000","last_undegraded":"2024-09-19T08:03:00.841267+0000","last_fullsized":"2024-09-19T08:03:00.841267+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:32:40.901049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838028+0000","last_change":"2024-09-19T08:01:10.770823+0000","last_active":"2024-09-19T08:03:00.838028+0000","last_peered":"2024-09-19T08:03:00.838028+0000","last_clean":"2024-09-19T08:03:00.838028+0000","last_became_active":"2024-09-19T08:01:10.770462+0000","last_became_peered":"2024-09-19T08:01:10.770462+0000","last_unstale":"2024-09-19T08:03:00.838028+0000","last_undegraded":"2024-09-19T08:03:00.838028+0000","last_fullsized":"2024-09-19T08:03:00.838028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:04:32.832009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'7","reported_seq":134,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.993691+0000","last_change":"2024-09-19T08:01:08.759429+0000","last_active":"2024-09-19T08:03:00.993691+0000","last_peered":"2024-09-19T08:03:00.993691+0000","last_clean":"2024-09-19T08:03:00.993691+0000","last_became_active":"2024-09-19T08:01:08.759062+0000","last_became_peered":"2024-09-19T08:01:08.759062+0000","last_unstale":"2024-09-19T08:03:00.993691+0000","last_undegraded":"2024-09-19T08:03:00.993691+0000","last_fullsized":"2024-09-19T08:03:00.993691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:16:51.582435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1374,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":3,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837660+0000","last_change":"2024-09-19T08:01:10.784735+0000","last_active":"2024-09-19T08:03:00.837660+0000","last_peered":"2024-09-19T08:03:00.837660+0000","last_clean":"2024-09-19T08:03:00.837660+0000","last_became_active":"2024-09-19T08:01:10.784346+0000","last_became_peered":"2024-09-19T08:01:10.784346+0000","last_unstale":"2024-09-19T08:03:00.837660+0000","last_undegraded":"2024-09-19T08:03:00.837660+0000","last_fullsized":"2024-09-19T08:03:00.837660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:27:54.580391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847399+0000","last_change":"2024-09-19T08:01:09.749498+0000","last_active":"2024-09-19T08:03:00.847399+0000","last_peered":"2024-09-19T08:03:00.847399+0000","last_clean":"2024-09-19T08:03:00.847399+0000","last_became_active":"2024-09-19T08:01:09.749091+0000","last_became_peered":"2024-09-19T08:01:09.749091+0000","last_unstale":"2024-09-19T08:03:00.847399+0000","last_undegraded":"2024-09-19T08:03:00.847399+0000","last_fullsized":"2024-09-19T08:03:00.847399+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:56:13.644931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841217+0000","last_change":"2024-09-19T08:01:10.779599+0000","last_active":"2024-09-19T08:03:00.841217+0000","last_peered":"2024-09-19T08:03:00.841217+0000","last_clean":"2024-09-19T08:03:00.841217+0000","last_became_active":"2024-09-19T08:01:10.779219+0000","last_became_peered":"2024-09-19T08:01:10.779219+0000","last_unstale":"2024-09-19T08:03:00.841217+0000","last_undegraded":"2024-09-19T08:03:00.841217+0000","last_fullsized":"2024-09-19T08:03:00.841217+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:20.193753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847308+0000","last_change":"2024-09-19T08:01:08.761432+0000","last_active":"2024-09-19T08:03:00.847308+0000","last_peered":"2024-09-19T08:03:00.847308+0000","last_clean":"2024-09-19T08:03:00.847308+0000","last_became_active":"2024-09-19T08:01:08.761044+0000","last_became_peered":"2024-09-19T08:01:08.761044+0000","last_unstale":"2024-09-19T08:03:00.847308+0000","last_undegraded":"2024-09-19T08:03:00.847308+0000","last_fullsized":"2024-09-19T08:03:00.847308+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:12.237334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780765+0000","last_change":"2024-09-19T08:01:10.772559+0000","last_active":"2024-09-19T08:03:00.780765+0000","last_peered":"2024-09-19T08:03:00.780765+0000","last_clean":"2024-09-19T08:03:00.780765+0000","last_became_active":"2024-09-19T08:01:10.772176+0000","last_became_peered":"2024-09-19T08:01:10.772176+0000","last_unstale":"2024-09-19T08:03:00.780765+0000","last_undegraded":"2024-09-19T08:03:00.780765+0000","last_fullsized":"2024-09-19T08:03:00.780765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:27:16.296127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758996+0000","last_change":"2024-09-19T08:01:08.759989+0000","last_active":"2024-09-19T08:03:00.758996+0000","last_peered":"2024-09-19T08:03:00.758996+0000","last_clean":"2024-09-19T08:03:00.758996+0000","last_became_active":"2024-09-19T08:01:08.759705+0000","last_became_peered":"2024-09-19T08:01:08.759705+0000","last_unstale":"2024-09-19T08:03:00.758996+0000","last_undegraded":"2024-09-19T08:03:00.758996+0000","last_fullsized":"2024-09-19T08:03:00.758996+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:33:25.237567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838150+0000","last_change":"2024-09-19T08:01:10.776528+0000","last_active":"2024-09-19T08:03:00.838150+0000","last_peered":"2024-09-19T08:03:00.838150+0000","last_clean":"2024-09-19T08:03:00.838150+0000","last_became_active":"2024-09-19T08:01:10.776222+0000","last_became_peered":"2024-09-19T08:01:10.776222+0000","last_unstale":"2024-09-19T08:03:00.838150+0000","last_undegraded":"2024-09-19T08:03:00.838150+0000","last_fullsized":"2024-09-19T08:03:00.838150+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.787492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'41","reported_seq":73,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.957555+0000","last_change":"2024-09-19T08:01:08.759631+0000","last_active":"2024-09-19T08:03:00.957555+0000","last_peered":"2024-09-19T08:03:00.957555+0000","last_clean":"2024-09-19T08:03:00.957555+0000","last_became_active":"2024-09-19T08:01:08.759156+0000","last_became_peered":"2024-09-19T08:01:08.759156+0000","last_unstale":"2024-09-19T08:03:00.957555+0000","last_undegraded":"2024-09-19T08:03:00.957555+0000","last_fullsized":"2024-09-19T08:03:00.957555+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:48:50.910824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837969+0000","last_change":"2024-09-19T08:01:10.772660+0000","last_active":"2024-09-19T08:03:00.837969+0000","last_peered":"2024-09-19T08:03:00.837969+0000","last_clean":"2024-09-19T08:03:00.837969+0000","last_became_active":"2024-09-19T08:01:10.771816+0000","last_became_peered":"2024-09-19T08:01:10.771816+0000","last_unstale":"2024-09-19T08:03:00.837969+0000","last_undegraded":"2024-09-19T08:03:00.837969+0000","last_fullsized":"2024-09-19T08:03:00.837969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:04:32.667107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'7","reported_seq":134,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760174+0000","last_change":"2024-09-19T08:01:09.749844+0000","last_active":"2024-09-19T08:03:00.760174+0000","last_peered":"2024-09-19T08:03:00.760174+0000","last_clean":"2024-09-19T08:03:00.760174+0000","last_became_active":"2024-09-19T08:01:09.749410+0000","last_became_peered":"2024-09-19T08:01:09.749410+0000","last_unstale":"2024-09-19T08:03:00.760174+0000","last_undegraded":"2024-09-19T08:03:00.760174+0000","last_fullsized":"2024-09-19T08:03:00.760174+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:03:14.989814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1704,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838196+0000","last_change":"2024-09-19T08:01:10.773782+0000","last_active":"2024-09-19T08:03:00.838196+0000","last_peered":"2024-09-19T08:03:00.838196+0000","last_clean":"2024-09-19T08:03:00.838196+0000","last_became_active":"2024-09-19T08:01:10.773471+0000","last_became_peered":"2024-09-19T08:01:10.773471+0000","last_unstale":"2024-09-19T08:03:00.838196+0000","last_undegraded":"2024-09-19T08:03:00.838196+0000","last_fullsized":"2024-09-19T08:03:00.838196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:15.780259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":344,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837815+0000","last_change":"2024-09-19T08:00:01.677517+0000","last_active":"2024-09-19T08:03:00.837815+0000","last_peered":"2024-09-19T08:03:00.837815+0000","last_clean":"2024-09-19T08:03:00.837815+0000","last_became_active":"2024-09-19T08:00:01.677089+0000","last_became_peered":"2024-09-19T08:00:01.677089+0000","last_unstale":"2024-09-19T08:03:00.837815+0000","last_undegraded":"2024-09-19T08:03:00.837815+0000","last_fullsized":"2024-09-19T08:03:00.837815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_clean_scrub_stamp":"2024-09-19T07:56:43.962142+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:10:36.158811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761731+0000","last_change":"2024-09-19T08:01:08.759819+0000","last_active":"2024-09-19T08:03:00.761731+0000","last_peered":"2024-09-19T08:03:00.761731+0000","last_clean":"2024-09-19T08:03:00.761731+0000","last_became_active":"2024-09-19T08:01:08.759420+0000","last_became_peered":"2024-09-19T08:01:08.759420+0000","last_unstale":"2024-09-19T08:03:00.761731+0000","last_undegraded":"2024-09-19T08:03:00.761731+0000","last_fullsized":"2024-09-19T08:03:00.761731+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:06:16.852323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.843903+0000","last_change":"2024-09-19T08:01:26.724260+0000","last_active":"2024-09-19T08:03:00.843903+0000","last_peered":"2024-09-19T08:03:00.843903+0000","last_clean":"2024-09-19T08:03:00.843903+0000","last_became_active":"2024-09-19T08:01:26.723851+0000","last_became_peered":"2024-09-19T08:01:26.723851+0000","last_unstale":"2024-09-19T08:03:00.843903+0000","last_undegraded":"2024-09-19T08:03:00.843903+0000","last_fullsized":"2024-09-19T08:03:00.843903+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:18:46.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837812+0000","last_change":"2024-09-19T08:01:08.761291+0000","last_active":"2024-09-19T08:03:00.837812+0000","last_peered":"2024-09-19T08:03:00.837812+0000","last_clean":"2024-09-19T08:03:00.837812+0000","last_became_active":"2024-09-19T08:01:08.760238+0000","last_became_peered":"2024-09-19T08:01:08.760238+0000","last_unstale":"2024-09-19T08:03:00.837812+0000","last_undegraded":"2024-09-19T08:03:00.837812+0000","last_fullsized":"2024-09-19T08:03:00.837812+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:26:09.980573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758579+0000","last_change":"2024-09-19T08:01:10.773056+0000","last_active":"2024-09-19T08:03:00.758579+0000","last_peered":"2024-09-19T08:03:00.758579+0000","last_clean":"2024-09-19T08:03:00.758579+0000","last_became_active":"2024-09-19T08:01:10.772431+0000","last_became_peered":"2024-09-19T08:01:10.772431+0000","last_unstale":"2024-09-19T08:03:00.758579+0000","last_undegraded":"2024-09-19T08:03:00.758579+0000","last_fullsized":"2024-09-19T08:03:00.758579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:39:06.878086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":132,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.842523+0000","last_change":"2024-09-19T08:01:09.749707+0000","last_active":"2024-09-19T08:03:00.842523+0000","last_peered":"2024-09-19T08:03:00.842523+0000","last_clean":"2024-09-19T08:03:00.842523+0000","last_became_active":"2024-09-19T08:01:09.749013+0000","last_became_peered":"2024-09-19T08:01:09.749013+0000","last_unstale":"2024-09-19T08:03:00.842523+0000","last_undegraded":"2024-09-19T08:03:00.842523+0000","last_fullsized":"2024-09-19T08:03:00.842523+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:40:12.682158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841187+0000","last_change":"2024-09-19T08:01:10.774101+0000","last_active":"2024-09-19T08:03:00.841187+0000","last_peered":"2024-09-19T08:03:00.841187+0000","last_clean":"2024-09-19T08:03:00.841187+0000","last_became_active":"2024-09-19T08:01:10.773822+0000","last_became_peered":"2024-09-19T08:01:10.773822+0000","last_unstale":"2024-09-19T08:03:00.841187+0000","last_undegraded":"2024-09-19T08:03:00.841187+0000","last_fullsized":"2024-09-19T08:03:00.841187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:29.571118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"78'5","reported_seq":129,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837640+0000","last_change":"2024-09-19T08:01:08.761675+0000","last_active":"2024-09-19T08:03:00.837640+0000","last_peered":"2024-09-19T08:03:00.837640+0000","last_clean":"2024-09-19T08:03:00.837640+0000","last_became_active":"2024-09-19T08:01:08.761346+0000","last_became_peered":"2024-09-19T08:01:08.761346+0000","last_unstale":"2024-09-19T08:03:00.837640+0000","last_undegraded":"2024-09-19T08:03:00.837640+0000","last_fullsized":"2024-09-19T08:03:00.837640+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:51:19.237458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839233+0000","last_change":"2024-09-19T08:01:10.781565+0000","last_active":"2024-09-19T08:03:00.839233+0000","last_peered":"2024-09-19T08:03:00.839233+0000","last_clean":"2024-09-19T08:03:00.839233+0000","last_became_active":"2024-09-19T08:01:10.781301+0000","last_became_peered":"2024-09-19T08:01:10.781301+0000","last_unstale":"2024-09-19T08:03:00.839233+0000","last_undegraded":"2024-09-19T08:03:00.839233+0000","last_fullsized":"2024-09-19T08:03:00.839233+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:55:52.932009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844577+0000","last_change":"2024-09-19T08:01:09.748918+0000","last_active":"2024-09-19T08:03:00.844577+0000","last_peered":"2024-09-19T08:03:00.844577+0000","last_clean":"2024-09-19T08:03:00.844577+0000","last_became_active":"2024-09-19T08:01:09.748497+0000","last_became_peered":"2024-09-19T08:01:09.748497+0000","last_unstale":"2024-09-19T08:03:00.844577+0000","last_undegraded":"2024-09-19T08:03:00.844577+0000","last_fullsized":"2024-09-19T08:03:00.844577+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:20:31.030133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839231+0000","last_change":"2024-09-19T08:01:10.781388+0000","last_active":"2024-09-19T08:03:00.839231+0000","last_peered":"2024-09-19T08:03:00.839231+0000","last_clean":"2024-09-19T08:03:00.839231+0000","last_became_active":"2024-09-19T08:01:10.781063+0000","last_became_peered":"2024-09-19T08:01:10.781063+0000","last_unstale":"2024-09-19T08:03:00.839231+0000","last_undegraded":"2024-09-19T08:03:00.839231+0000","last_fullsized":"2024-09-19T08:03:00.839231+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:12:08.852717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847755+0000","last_change":"2024-09-19T08:01:09.750080+0000","last_active":"2024-09-19T08:03:00.847755+0000","last_peered":"2024-09-19T08:03:00.847755+0000","last_clean":"2024-09-19T08:03:00.847755+0000","last_became_active":"2024-09-19T08:01:09.749053+0000","last_became_peered":"2024-09-19T08:01:09.749053+0000","last_unstale":"2024-09-19T08:03:00.847755+0000","last_undegraded":"2024-09-19T08:03:00.847755+0000","last_fullsized":"2024-09-19T08:03:00.847755+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:15:45.292714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759144+0000","last_change":"2024-09-19T08:01:10.767070+0000","last_active":"2024-09-19T08:03:00.759144+0000","last_peered":"2024-09-19T08:03:00.759144+0000","last_clean":"2024-09-19T08:03:00.759144+0000","last_became_active":"2024-09-19T08:01:10.766760+0000","last_became_peered":"2024-09-19T08:01:10.766760+0000","last_unstale":"2024-09-19T08:03:00.759144+0000","last_undegraded":"2024-09-19T08:03:00.759144+0000","last_fullsized":"2024-09-19T08:03:00.759144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:17.841135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'51","reported_seq":98,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.960465+0000","last_change":"2024-09-19T08:01:08.756143+0000","last_active":"2024-09-19T08:03:00.960465+0000","last_peered":"2024-09-19T08:03:00.960465+0000","last_clean":"2024-09-19T08:03:00.960465+0000","last_became_active":"2024-09-19T08:01:08.755777+0000","last_became_peered":"2024-09-19T08:01:08.755777+0000","last_unstale":"2024-09-19T08:03:00.960465+0000","last_undegraded":"2024-09-19T08:03:00.960465+0000","last_fullsized":"2024-09-19T08:03:00.960465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:21.430673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":90,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849237+0000","last_change":"2024-09-19T08:01:10.774140+0000","last_active":"2024-09-19T08:03:00.849237+0000","last_peered":"2024-09-19T08:03:00.849237+0000","last_clean":"2024-09-19T08:03:00.849237+0000","last_became_active":"2024-09-19T08:01:10.773819+0000","last_became_peered":"2024-09-19T08:01:10.773819+0000","last_unstale":"2024-09-19T08:03:00.849237+0000","last_undegraded":"2024-09-19T08:03:00.849237+0000","last_fullsized":"2024-09-19T08:03:00.849237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:15:13.174013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761607+0000","last_change":"2024-09-19T08:01:09.169129+0000","last_active":"2024-09-19T08:03:00.761607+0000","last_peered":"2024-09-19T08:03:00.761607+0000","last_clean":"2024-09-19T08:03:00.761607+0000","last_became_active":"2024-09-19T08:01:09.168734+0000","last_became_peered":"2024-09-19T08:01:09.168734+0000","last_unstale":"2024-09-19T08:03:00.761607+0000","last_undegraded":"2024-09-19T08:03:00.761607+0000","last_fullsized":"2024-09-19T08:03:00.761607+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:01:52.232391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994759+0000","last_change":"2024-09-19T08:01:10.768592+0000","last_active":"2024-09-19T08:03:00.994759+0000","last_peered":"2024-09-19T08:03:00.994759+0000","last_clean":"2024-09-19T08:03:00.994759+0000","last_became_active":"2024-09-19T08:01:10.767645+0000","last_became_peered":"2024-09-19T08:01:10.767645+0000","last_unstale":"2024-09-19T08:03:00.994759+0000","last_undegraded":"2024-09-19T08:03:00.994759+0000","last_fullsized":"2024-09-19T08:03:00.994759+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:08:46.227461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997364+0000","last_change":"2024-09-19T08:01:09.749240+0000","last_active":"2024-09-19T08:03:00.997364+0000","last_peered":"2024-09-19T08:03:00.997364+0000","last_clean":"2024-09-19T08:03:00.997364+0000","last_became_active":"2024-09-19T08:01:09.748945+0000","last_became_peered":"2024-09-19T08:01:09.748945+0000","last_unstale":"2024-09-19T08:03:00.997364+0000","last_undegraded":"2024-09-19T08:03:00.997364+0000","last_fullsized":"2024-09-19T08:03:00.997364+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:25:13.791290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847326+0000","last_change":"2024-09-19T08:01:10.783133+0000","last_active":"2024-09-19T08:03:00.847326+0000","last_peered":"2024-09-19T08:03:00.847326+0000","last_clean":"2024-09-19T08:03:00.847326+0000","last_became_active":"2024-09-19T08:01:10.782797+0000","last_became_peered":"2024-09-19T08:01:10.782797+0000","last_unstale":"2024-09-19T08:03:00.847326+0000","last_undegraded":"2024-09-19T08:03:00.847326+0000","last_fullsized":"2024-09-19T08:03:00.847326+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:30:43.551877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848489+0000","last_change":"2024-09-19T08:01:08.757964+0000","last_active":"2024-09-19T08:03:00.848489+0000","last_peered":"2024-09-19T08:03:00.848489+0000","last_clean":"2024-09-19T08:03:00.848489+0000","last_became_active":"2024-09-19T08:01:08.757526+0000","last_became_peered":"2024-09-19T08:01:08.757526+0000","last_unstale":"2024-09-19T08:03:00.848489+0000","last_undegraded":"2024-09-19T08:03:00.848489+0000","last_fullsized":"2024-09-19T08:03:00.848489+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:03.565702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780557+0000","last_change":"2024-09-19T08:01:10.769974+0000","last_active":"2024-09-19T08:03:00.780557+0000","last_peered":"2024-09-19T08:03:00.780557+0000","last_clean":"2024-09-19T08:03:00.780557+0000","last_became_active":"2024-09-19T08:01:10.769650+0000","last_became_peered":"2024-09-19T08:01:10.769650+0000","last_unstale":"2024-09-19T08:03:00.780557+0000","last_undegraded":"2024-09-19T08:03:00.780557+0000","last_fullsized":"2024-09-19T08:03:00.780557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:01.755198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"78'3","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759999+0000","last_change":"2024-09-19T08:01:08.763476+0000","last_active":"2024-09-19T08:03:00.759999+0000","last_peered":"2024-09-19T08:03:00.759999+0000","last_clean":"2024-09-19T08:03:00.759999+0000","last_became_active":"2024-09-19T08:01:08.763154+0000","last_became_peered":"2024-09-19T08:01:08.763154+0000","last_unstale":"2024-09-19T08:03:00.759999+0000","last_undegraded":"2024-09-19T08:03:00.759999+0000","last_fullsized":"2024-09-19T08:03:00.759999+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:06.202452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848751+0000","last_change":"2024-09-19T08:01:10.784201+0000","last_active":"2024-09-19T08:03:00.848751+0000","last_peered":"2024-09-19T08:03:00.848751+0000","last_clean":"2024-09-19T08:03:00.848751+0000","last_became_active":"2024-09-19T08:01:10.783867+0000","last_became_peered":"2024-09-19T08:01:10.783867+0000","last_unstale":"2024-09-19T08:03:00.848751+0000","last_undegraded":"2024-09-19T08:03:00.848751+0000","last_fullsized":"2024-09-19T08:03:00.848751+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:43:50.082183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847290+0000","last_change":"2024-09-19T08:01:08.760479+0000","last_active":"2024-09-19T08:03:00.847290+0000","last_peered":"2024-09-19T08:03:00.847290+0000","last_clean":"2024-09-19T08:03:00.847290+0000","last_became_active":"2024-09-19T08:01:08.760004+0000","last_became_peered":"2024-09-19T08:01:08.760004+0000","last_unstale":"2024-09-19T08:03:00.847290+0000","last_undegraded":"2024-09-19T08:03:00.847290+0000","last_fullsized":"2024-09-19T08:03:00.847290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:41:19.956623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837851+0000","last_change":"2024-09-19T08:01:10.772184+0000","last_active":"2024-09-19T08:03:00.837851+0000","last_peered":"2024-09-19T08:03:00.837851+0000","last_clean":"2024-09-19T08:03:00.837851+0000","last_became_active":"2024-09-19T08:01:10.771825+0000","last_became_peered":"2024-09-19T08:01:10.771825+0000","last_unstale":"2024-09-19T08:03:00.837851+0000","last_undegraded":"2024-09-19T08:03:00.837851+0000","last_fullsized":"2024-09-19T08:03:00.837851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:36.764575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841567+0000","last_change":"2024-09-19T08:01:10.779368+0000","last_active":"2024-09-19T08:03:00.841567+0000","last_peered":"2024-09-19T08:03:00.841567+0000","last_clean":"2024-09-19T08:03:00.841567+0000","last_became_active":"2024-09-19T08:01:10.778884+0000","last_became_peered":"2024-09-19T08:01:10.778884+0000","last_unstale":"2024-09-19T08:03:00.841567+0000","last_undegraded":"2024-09-19T08:03:00.841567+0000","last_fullsized":"2024-09-19T08:03:00.841567+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:28:29.630300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780536+0000","last_change":"2024-09-19T08:01:26.741053+0000","last_active":"2024-09-19T08:03:00.780536+0000","last_peered":"2024-09-19T08:03:00.780536+0000","last_clean":"2024-09-19T08:03:00.780536+0000","last_became_active":"2024-09-19T08:01:26.724714+0000","last_became_peered":"2024-09-19T08:01:26.724714+0000","last_unstale":"2024-09-19T08:03:00.780536+0000","last_undegraded":"2024-09-19T08:03:00.780536+0000","last_fullsized":"2024-09-19T08:03:00.780536+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:59.583797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997323+0000","last_change":"2024-09-19T08:01:10.781389+0000","last_active":"2024-09-19T08:03:00.997323+0000","last_peered":"2024-09-19T08:03:00.997323+0000","last_clean":"2024-09-19T08:03:00.997323+0000","last_became_active":"2024-09-19T08:01:10.781080+0000","last_became_peered":"2024-09-19T08:01:10.781080+0000","last_unstale":"2024-09-19T08:03:00.997323+0000","last_undegraded":"2024-09-19T08:03:00.997323+0000","last_fullsized":"2024-09-19T08:03:00.997323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:40:45.225957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847701+0000","last_change":"2024-09-19T08:01:26.768102+0000","last_active":"2024-09-19T08:03:00.847701+0000","last_peered":"2024-09-19T08:03:00.847701+0000","last_clean":"2024-09-19T08:03:00.847701+0000","last_became_active":"2024-09-19T08:01:26.759462+0000","last_became_peered":"2024-09-19T08:01:26.759462+0000","last_unstale":"2024-09-19T08:03:00.847701+0000","last_undegraded":"2024-09-19T08:03:00.847701+0000","last_fullsized":"2024-09-19T08:03:00.847701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:02:06.815238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994921+0000","last_change":"2024-09-19T08:01:10.771031+0000","last_active":"2024-09-19T08:03:00.994921+0000","last_peered":"2024-09-19T08:03:00.994921+0000","last_clean":"2024-09-19T08:03:00.994921+0000","last_became_active":"2024-09-19T08:01:10.770675+0000","last_became_peered":"2024-09-19T08:01:10.770675+0000","last_unstale":"2024-09-19T08:03:00.994921+0000","last_undegraded":"2024-09-19T08:03:00.994921+0000","last_fullsized":"2024-09-19T08:03:00.994921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:42.203848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849208+0000","last_change":"2024-09-19T08:01:09.748622+0000","last_active":"2024-09-19T08:03:00.849208+0000","last_peered":"2024-09-19T08:03:00.849208+0000","last_clean":"2024-09-19T08:03:00.849208+0000","last_became_active":"2024-09-19T08:01:09.748202+0000","last_became_peered":"2024-09-19T08:01:09.748202+0000","last_unstale":"2024-09-19T08:03:00.849208+0000","last_undegraded":"2024-09-19T08:03:00.849208+0000","last_fullsized":"2024-09-19T08:03:00.849208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:17:39.861820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849073+0000","last_change":"2024-09-19T08:01:10.768949+0000","last_active":"2024-09-19T08:03:00.849073+0000","last_peered":"2024-09-19T08:03:00.849073+0000","last_clean":"2024-09-19T08:03:00.849073+0000","last_became_active":"2024-09-19T08:01:10.768159+0000","last_became_peered":"2024-09-19T08:01:10.768159+0000","last_unstale":"2024-09-19T08:03:00.849073+0000","last_undegraded":"2024-09-19T08:03:00.849073+0000","last_fullsized":"2024-09-19T08:03:00.849073+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:43.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"78'13","reported_seq":163,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844272+0000","last_change":"2024-09-19T08:01:26.735877+0000","last_active":"2024-09-19T08:03:00.844272+0000","last_peered":"2024-09-19T08:03:00.844272+0000","last_clean":"2024-09-19T08:03:00.844272+0000","last_became_active":"2024-09-19T08:01:26.723597+0000","last_became_peered":"2024-09-19T08:01:26.723597+0000","last_unstale":"2024-09-19T08:03:00.844272+0000","last_undegraded":"2024-09-19T08:03:00.844272+0000","last_fullsized":"2024-09-19T08:03:00.844272+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:43:42.340832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":99,"num_read_kb":99,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,1],"acting":[9,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600666,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1139,"num_read_kb":683,"num_write":372,"num_write_kb":1463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1254,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1801998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645347,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":905216,"data_stored":436392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874280,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32940,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712212,"statfs":{"total":95995035648,"available":95961305088,"internally_reserved":0,"allocated":1736704,"data_stored":1224499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005101,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32404,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712748,"statfs":{"total":95995035648,"available":95961853952,"internally_reserved":0,"allocated":1245184,"data_stored":762214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234033,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1294336,"data_stored":826874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462966,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1503232,"data_stored":1028781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593788,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36228,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708924,"statfs":{"total":95995035648,"available":95957938176,"internally_reserved":0,"allocated":901120,"data_stored":434251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822720,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1658880,"data_stored":1159945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986245,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":901120,"data_stored":434112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215177,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32656,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712496,"statfs":{"total":95995035648,"available":95961595904,"internally_reserved":0,"allocated":1445888,"data_stored":958892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476815,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":504026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607636,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":978944,"data_stored":502316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":921600,"data_stored":436124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":136048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":200602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T08:03:05.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:03:06.041 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:05 smithi162 ceph-mon[35260]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-09-19T08:03:06.041 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:05 smithi162 ceph-mon[35260]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-19T08:03:06.041 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:05 smithi162 ceph-mon[35260]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-19T08:03:06.042 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:05 smithi162 ceph-mon[35260]: Updating smithi139:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T08:03:06.042 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:05 smithi162 ceph-mon[35260]: Updating smithi162:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T08:03:06.042 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:05 smithi162 ceph-mon[35260]: Updating smithi098:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T08:03:06.042 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:05 smithi162 ceph-mon[35260]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:03:06.042 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:05 smithi162 ceph-mon[35260]: mgrmap e22: x(active, since 4s), standbys: z, y 2024-09-19T08:03:06.153 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:06.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:05 smithi139 ceph-mon[34874]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-09-19T08:03:06.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:05 smithi139 ceph-mon[34874]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-19T08:03:06.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:05 smithi139 ceph-mon[34874]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-19T08:03:06.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:05 smithi139 ceph-mon[34874]: Updating smithi139:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T08:03:06.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:05 smithi139 ceph-mon[34874]: Updating smithi162:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T08:03:06.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:05 smithi139 ceph-mon[34874]: Updating smithi098:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T08:03:06.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:05 smithi139 ceph-mon[34874]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:03:06.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:05 smithi139 ceph-mon[34874]: mgrmap e22: x(active, since 4s), standbys: z, y 2024-09-19T08:03:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.625+0000 7f2c8000f640 1 -- 172.21.15.98:0/2954882678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c78102d40 msgr2=0x7f2c78103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.625+0000 7f2c8000f640 1 --2- 172.21.15.98:0/2954882678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c78102d40 0x7f2c78103120 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2c74009900 tx=0x7f2c7402a480 comp rx=0 tx=0).stop 2024-09-19T08:03:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.625+0000 7f2c8000f640 1 -- 172.21.15.98:0/2954882678 shutdown_connections 2024-09-19T08:03:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.625+0000 7f2c8000f640 1 --2- 172.21.15.98:0/2954882678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c7810c170 0x7f2c7810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.625+0000 7f2c8000f640 1 --2- 172.21.15.98:0/2954882678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c78103660 0x7f2c7810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.625+0000 7f2c8000f640 1 --2- 172.21.15.98:0/2954882678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c78102d40 0x7f2c78103120 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.625+0000 7f2c8000f640 1 -- 172.21.15.98:0/2954882678 >> 172.21.15.98:0/2954882678 conn(0x7f2c780fc7a0 msgr2=0x7f2c780febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.625+0000 7f2c8000f640 1 -- 172.21.15.98:0/2954882678 shutdown_connections 2024-09-19T08:03:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.625+0000 7f2c8000f640 1 -- 172.21.15.98:0/2954882678 wait complete. 2024-09-19T08:03:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.626+0000 7f2c8000f640 1 Processor -- start 2024-09-19T08:03:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.626+0000 7f2c8000f640 1 -- start start 2024-09-19T08:03:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.626+0000 7f2c8000f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c78102d40 0x7f2c781984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.626+0000 7f2c8000f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c78103660 0x7f2c78198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.626+0000 7f2c8000f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c7810c170 0x7f2c7819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.626+0000 7f2c8000f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2c78111fd0 con 0x7f2c78103660 2024-09-19T08:03:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.626+0000 7f2c8000f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2c78111e50 con 0x7f2c7810c170 2024-09-19T08:03:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.626+0000 7f2c8000f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2c78112150 con 0x7f2c78102d40 2024-09-19T08:03:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c7ddab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c78102d40 0x7f2c781984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c7ddab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c78102d40 0x7f2c781984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:55358/0 (socket says 172.21.15.98:55358) 2024-09-19T08:03:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c7d5aa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c78103660 0x7f2c78198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c7ddab640 1 -- 172.21.15.98:0/3633320405 learned_addr learned my addr 172.21.15.98:0/3633320405 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c7e5ac640 1 --2- 172.21.15.98:0/3633320405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c7810c170 0x7f2c7819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c7ddab640 1 -- 172.21.15.98:0/3633320405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c7810c170 msgr2=0x7f2c7819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c7ddab640 1 --2- 172.21.15.98:0/3633320405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c7810c170 0x7f2c7819cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c7ddab640 1 -- 172.21.15.98:0/3633320405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c78103660 msgr2=0x7f2c78198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c7ddab640 1 --2- 172.21.15.98:0/3633320405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c78103660 0x7f2c78198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c7ddab640 1 -- 172.21.15.98:0/3633320405 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c7819d480 con 0x7f2c78102d40 2024-09-19T08:03:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c7d5aa640 1 --2- 172.21.15.98:0/3633320405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c78103660 0x7f2c78198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c7ddab640 1 --2- 172.21.15.98:0/3633320405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c78102d40 0x7f2c781984d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2c74006010 tx=0x7f2c740042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c6effd640 1 -- 172.21.15.98:0/3633320405 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7403f030 con 0x7f2c78102d40 2024-09-19T08:03:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c8000f640 1 -- 172.21.15.98:0/3633320405 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c7819d710 con 0x7f2c78102d40 2024-09-19T08:03:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c6effd640 1 -- 172.21.15.98:0/3633320405 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c74003ce0 con 0x7f2c78102d40 2024-09-19T08:03:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.627+0000 7f2c6effd640 1 -- 172.21.15.98:0/3633320405 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c74035070 con 0x7f2c78102d40 2024-09-19T08:03:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.628+0000 7f2c8000f640 1 -- 172.21.15.98:0/3633320405 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c781a5050 con 0x7f2c78102d40 2024-09-19T08:03:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.628+0000 7f2c8000f640 1 -- 172.21.15.98:0/3633320405 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c78103120 con 0x7f2c78102d40 2024-09-19T08:03:06.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.630+0000 7f2c6effd640 1 -- 172.21.15.98:0/3633320405 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f2c74042030 con 0x7f2c78102d40 2024-09-19T08:03:06.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.630+0000 7f2c6effd640 1 --2- 172.21.15.98:0/3633320405 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2c640a34d0 0x7f2c640a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:06.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.630+0000 7f2c7d5aa640 1 --2- 172.21.15.98:0/3633320405 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2c640a34d0 0x7f2c640a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:06.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.631+0000 7f2c6effd640 1 -- 172.21.15.98:0/3633320405 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f2c740f0630 con 0x7f2c78102d40 2024-09-19T08:03:06.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.631+0000 7f2c7d5aa640 1 --2- 172.21.15.98:0/3633320405 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2c640a34d0 0x7f2c640a5990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2c7810d5b0 tx=0x7f2c68009060 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:06.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.638+0000 7f2c6effd640 1 -- 172.21.15.98:0/3633320405 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2c740bc8d0 con 0x7f2c78102d40 2024-09-19T08:03:06.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.842+0000 7f2c8000f640 1 -- 172.21.15.98:0/3633320405 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c78107210 con 0x7f2c78102d40 2024-09-19T08:03:06.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:06 smithi098 ceph-mon[31333]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:06.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:06 smithi098 ceph-mon[31333]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.843+0000 7f2c6effd640 1 -- 172.21.15.98:0/3633320405 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11983 (secure 0 0 0) 0x7f2c740c0bd0 con 0x7f2c78102d40 2024-09-19T08:03:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:06.845 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"btime":"2024-09-19T08:02:59:855935+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:02:59.805764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:03:06.846 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-09-19T08:03:06.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.845+0000 7f2c8000f640 1 -- 172.21.15.98:0/3633320405 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2c640a34d0 msgr2=0x7f2c640a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.846+0000 7f2c8000f640 1 --2- 172.21.15.98:0/3633320405 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2c640a34d0 0x7f2c640a5990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2c7810d5b0 tx=0x7f2c68009060 comp rx=0 tx=0).stop 2024-09-19T08:03:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.846+0000 7f2c8000f640 1 -- 172.21.15.98:0/3633320405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c78102d40 msgr2=0x7f2c781984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.846+0000 7f2c8000f640 1 --2- 172.21.15.98:0/3633320405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c78102d40 0x7f2c781984d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2c74006010 tx=0x7f2c740042f0 comp rx=0 tx=0).stop 2024-09-19T08:03:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.846+0000 7f2c8000f640 1 -- 172.21.15.98:0/3633320405 shutdown_connections 2024-09-19T08:03:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.846+0000 7f2c8000f640 1 --2- 172.21.15.98:0/3633320405 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2c640a34d0 0x7f2c640a5990 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.846+0000 7f2c8000f640 1 --2- 172.21.15.98:0/3633320405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c7810c170 0x7f2c7819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.846+0000 7f2c8000f640 1 --2- 172.21.15.98:0/3633320405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c78103660 0x7f2c78198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.846+0000 7f2c8000f640 1 --2- 172.21.15.98:0/3633320405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c78102d40 0x7f2c781984d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.846+0000 7f2c8000f640 1 -- 172.21.15.98:0/3633320405 >> 172.21.15.98:0/3633320405 conn(0x7f2c780fc7a0 msgr2=0x7f2c780fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.846+0000 7f2c8000f640 1 -- 172.21.15.98:0/3633320405 shutdown_connections 2024-09-19T08:03:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:06.846+0000 7f2c8000f640 1 -- 172.21.15.98:0/3633320405 wait complete. 2024-09-19T08:03:07.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:06 smithi162 ceph-mon[35260]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:07.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:06 smithi162 ceph-mon[35260]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:07.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:06 smithi139 ceph-mon[34874]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:07.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:06 smithi139 ceph-mon[34874]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:07.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T08:03:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:07.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:07 smithi098 ceph-mon[31333]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:07.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:07 smithi098 ceph-mon[31333]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-19T08:03:07.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:07 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3633320405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:07 smithi162 ceph-mon[35260]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:07 smithi162 ceph-mon[35260]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-19T08:03:08.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:07 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3633320405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:08.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:07 smithi139 ceph-mon[34874]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:08.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:07 smithi139 ceph-mon[34874]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-19T08:03:08.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:07 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3633320405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.333+0000 7fe9341bc640 1 -- 172.21.15.98:0/2303523621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe92c101420 msgr2=0x7fe92c10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.333+0000 7fe9341bc640 1 --2- 172.21.15.98:0/2303523621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe92c101420 0x7fe92c10d950 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe924009cd0 tx=0x7fe92402a8e0 comp rx=0 tx=0).stop 2024-09-19T08:03:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.333+0000 7fe9341bc640 1 -- 172.21.15.98:0/2303523621 shutdown_connections 2024-09-19T08:03:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.333+0000 7fe9341bc640 1 --2- 172.21.15.98:0/2303523621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe92c101420 0x7fe92c10d950 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.333+0000 7fe9341bc640 1 --2- 172.21.15.98:0/2303523621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe92c100a80 0x7fe92c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.333+0000 7fe9341bc640 1 --2- 172.21.15.98:0/2303523621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe92c106a80 0x7fe92c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.333+0000 7fe9341bc640 1 -- 172.21.15.98:0/2303523621 >> 172.21.15.98:0/2303523621 conn(0x7fe92c0fc7a0 msgr2=0x7fe92c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.333+0000 7fe9341bc640 1 -- 172.21.15.98:0/2303523621 shutdown_connections 2024-09-19T08:03:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.333+0000 7fe9341bc640 1 -- 172.21.15.98:0/2303523621 wait complete. 2024-09-19T08:03:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe9341bc640 1 Processor -- start 2024-09-19T08:03:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe9341bc640 1 -- start start 2024-09-19T08:03:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe9341bc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe92c100a80 0x7fe92c1974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe9341bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe92c101420 0x7fe92c1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe9341bc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe92c106a80 0x7fe92c19bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe9341bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe92c110f20 con 0x7fe92c101420 2024-09-19T08:03:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe9341bc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe92c110da0 con 0x7fe92c100a80 2024-09-19T08:03:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe9341bc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe92c1110a0 con 0x7fe92c106a80 2024-09-19T08:03:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe931757640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe92c101420 0x7fe92c1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe931757640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe92c101420 0x7fe92c1979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47240/0 (socket says 172.21.15.98:47240) 2024-09-19T08:03:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe931757640 1 -- 172.21.15.98:0/587382706 learned_addr learned my addr 172.21.15.98:0/587382706 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe931f58640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe92c100a80 0x7fe92c1974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe932759640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe92c106a80 0x7fe92c19bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe931757640 1 -- 172.21.15.98:0/587382706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe92c106a80 msgr2=0x7fe92c19bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe931757640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe92c106a80 0x7fe92c19bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe931757640 1 -- 172.21.15.98:0/587382706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe92c100a80 msgr2=0x7fe92c1974a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe931757640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe92c100a80 0x7fe92c1974a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe931757640 1 -- 172.21.15.98:0/587382706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe92c19c450 con 0x7fe92c101420 2024-09-19T08:03:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.334+0000 7fe931f58640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe92c100a80 0x7fe92c1974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.335+0000 7fe931757640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe92c101420 0x7fe92c1979e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe92800b4a0 tx=0x7fe92800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.335+0000 7fe91effd640 1 -- 172.21.15.98:0/587382706 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe928014070 con 0x7fe92c101420 2024-09-19T08:03:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.335+0000 7fe91effd640 1 -- 172.21.15.98:0/587382706 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe928010af0 con 0x7fe92c101420 2024-09-19T08:03:08.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.335+0000 7fe9341bc640 1 -- 172.21.15.98:0/587382706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe92c19c740 con 0x7fe92c101420 2024-09-19T08:03:08.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.335+0000 7fe91effd640 1 -- 172.21.15.98:0/587382706 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe928017d10 con 0x7fe92c101420 2024-09-19T08:03:08.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.335+0000 7fe9341bc640 1 -- 172.21.15.98:0/587382706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe92c1a4020 con 0x7fe92c101420 2024-09-19T08:03:08.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.336+0000 7fe9341bc640 1 -- 172.21.15.98:0/587382706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe92c100f60 con 0x7fe92c101420 2024-09-19T08:03:08.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.337+0000 7fe91effd640 1 -- 172.21.15.98:0/587382706 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7fe928017070 con 0x7fe92c101420 2024-09-19T08:03:08.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.338+0000 7fe91effd640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe9180a3550 0x7fe9180a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:08.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.338+0000 7fe931f58640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe9180a3550 0x7fe9180a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:08.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.338+0000 7fe91effd640 1 -- 172.21.15.98:0/587382706 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fe9280cf230 con 0x7fe92c101420 2024-09-19T08:03:08.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.338+0000 7fe931f58640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe9180a3550 0x7fe9180a5a10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe920009930 tx=0x7fe920008040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:08.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.342+0000 7fe91effd640 1 -- 172.21.15.98:0/587382706 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fe92809bb00 con 0x7fe92c101420 2024-09-19T08:03:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.513+0000 7fe9341bc640 1 -- 172.21.15.98:0/587382706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe92c02d7c0 con 0x7fe92c101420 2024-09-19T08:03:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.516+0000 7fe91effd640 1 -- 172.21.15.98:0/587382706 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+23128 (secure 0 0 0) 0x7fe92809b400 con 0x7fe92c101420 2024-09-19T08:03:08.517 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:08.517 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","created":"2024-09-19T07:53:39.127398+0000","modified":"2024-09-19T08:03:00.753313+0000","last_up_change":"2024-09-19T08:00:22.467777+0000","last_in_change":"2024-09-19T07:59:58.538492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:56:43.773892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T08:01:07.693796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T08:01:09.518028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d01ce6-165d-493d-ae98-d8ad9e5d4134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6803","nonce":1719360533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6805","nonce":1719360533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6809","nonce":1719360533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6807","nonce":1719360533}]},"public_addr":"172.21.15.98:6803/1719360533","cluster_addr":"172.21.15.98:6805/1719360533","heartbeat_back_addr":"172.21.15.98:6809/1719360533","heartbeat_front_addr":"172.21.15.98:6807/1719360533","state":["exists","up"]},{"osd":1,"uuid":"34724bd5-0738-4473-b087-c09031c1ee2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036962626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036962626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036962626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036962626}]},"public_addr":"172.21.15.139:6801/4036962626","cluster_addr":"172.21.15.139:6803/4036962626","heartbeat_back_addr":"172.21.15.139:6807/4036962626","heartbeat_front_addr":"172.21.15.139:6805/4036962626","state":["exists","up"]},{"osd":2,"uuid":"16985ebe-0e32-42ee-bded-b7838e290afc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6801","nonce":2811412891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6803","nonce":2811412891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6807","nonce":2811412891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6805","nonce":2811412891}]},"public_addr":"172.21.15.162:6801/2811412891","cluster_addr":"172.21.15.162:6803/2811412891","heartbeat_back_addr":"172.21.15.162:6807/2811412891","heartbeat_front_addr":"172.21.15.162:6805/2811412891","state":["exists","up"]},{"osd":3,"uuid":"b140e466-3119-49cb-aec2-ba6ca4d10d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6811","nonce":1383392014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6813","nonce":1383392014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6817","nonce":1383392014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6815","nonce":1383392014}]},"public_addr":"172.21.15.98:6811/1383392014","cluster_addr":"172.21.15.98:6813/1383392014","heartbeat_back_addr":"172.21.15.98:6817/1383392014","heartbeat_front_addr":"172.21.15.98:6815/1383392014","state":["exists","up"]},{"osd":4,"uuid":"b4412813-df57-4ac6-8fb7-f57b69c74768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6809","nonce":116453520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6811","nonce":116453520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6815","nonce":116453520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6813","nonce":116453520}]},"public_addr":"172.21.15.139:6809/116453520","cluster_addr":"172.21.15.139:6811/116453520","heartbeat_back_addr":"172.21.15.139:6815/116453520","heartbeat_front_addr":"172.21.15.139:6813/116453520","state":["exists","up"]},{"osd":5,"uuid":"d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6809","nonce":3769530297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6811","nonce":3769530297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6815","nonce":3769530297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6813","nonce":3769530297}]},"public_addr":"172.21.15.162:6809/3769530297","cluster_addr":"172.21.15.162:6811/3769530297","heartbeat_back_addr":"172.21.15.162:6815/3769530297","heartbeat_front_addr":"172.21.15.162:6813/3769530297","state":["exists","up"]},{"osd":6,"uuid":"ba1c1590-6aea-40dc-90c8-e21b88896909","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6819","nonce":692373267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6821","nonce":692373267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6825","nonce":692373267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6823","nonce":692373267}]},"public_addr":"172.21.15.98:6819/692373267","cluster_addr":"172.21.15.98:6821/692373267","heartbeat_back_addr":"172.21.15.98:6825/692373267","heartbeat_front_addr":"172.21.15.98:6823/692373267","state":["exists","up"]},{"osd":7,"uuid":"eb9093ad-a068-4a63-ab87-0b50522cbc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6817","nonce":1963811650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6819","nonce":1963811650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6823","nonce":1963811650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6821","nonce":1963811650}]},"public_addr":"172.21.15.139:6817/1963811650","cluster_addr":"172.21.15.139:6819/1963811650","heartbeat_back_addr":"172.21.15.139:6823/1963811650","heartbeat_front_addr":"172.21.15.139:6821/1963811650","state":["exists","up"]},{"osd":8,"uuid":"31ca330a-fc45-466d-81c2-ef4386b3cc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6817","nonce":3607966458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6819","nonce":3607966458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6823","nonce":3607966458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6821","nonce":3607966458}]},"public_addr":"172.21.15.162:6817/3607966458","cluster_addr":"172.21.15.162:6819/3607966458","heartbeat_back_addr":"172.21.15.162:6823/3607966458","heartbeat_front_addr":"172.21.15.162:6821/3607966458","state":["exists","up"]},{"osd":9,"uuid":"9c4f216f-aef1-4367-9c58-b913b0c4f7db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6827","nonce":2050391244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6829","nonce":2050391244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6833","nonce":2050391244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6831","nonce":2050391244}]},"public_addr":"172.21.15.98:6827/2050391244","cluster_addr":"172.21.15.98:6829/2050391244","heartbeat_back_addr":"172.21.15.98:6833/2050391244","heartbeat_front_addr":"172.21.15.98:6831/2050391244","state":["exists","up"]},{"osd":10,"uuid":"03cc7df6-84ae-4d9a-a3d7-74c91e1744e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6825","nonce":2336761639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6827","nonce":2336761639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6831","nonce":2336761639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6829","nonce":2336761639}]},"public_addr":"172.21.15.139:6825/2336761639","cluster_addr":"172.21.15.139:6827/2336761639","heartbeat_back_addr":"172.21.15.139:6831/2336761639","heartbeat_front_addr":"172.21.15.139:6829/2336761639","state":["exists","up"]},{"osd":11,"uuid":"29e95498-4952-46d4-8fa6-6d8e2b862111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6825","nonce":1231011554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6827","nonce":1231011554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6831","nonce":1231011554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6829","nonce":1231011554}]},"public_addr":"172.21.15.162:6825/1231011554","cluster_addr":"172.21.15.162:6827/1231011554","heartbeat_back_addr":"172.21.15.162:6831/1231011554","heartbeat_front_addr":"172.21.15.162:6829/1231011554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:55:49.995244+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:15.655543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:40.331165+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:04.688881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:29.174966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:53.663161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:17.407551+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:42.584296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:06.317219+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:31.078625+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:55.689269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T08:00:19.659748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1},{"from":7,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1252058007":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/921776066":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/336479555":"2024-09-20T08:03:00.753287+0000","172.21.15.98:6800/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6800/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/1497156658":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6801/470997812":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/1238193420":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6800/470997812":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/3838590070":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3842627255":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3866031018":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3194864054":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3067331617":"2024-09-20T08:03:00.753287+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T08:03:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.518+0000 7fe9341bc640 1 -- 172.21.15.98:0/587382706 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe9180a3550 msgr2=0x7fe9180a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.518+0000 7fe9341bc640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe9180a3550 0x7fe9180a5a10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe920009930 tx=0x7fe920008040 comp rx=0 tx=0).stop 2024-09-19T08:03:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.518+0000 7fe9341bc640 1 -- 172.21.15.98:0/587382706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe92c101420 msgr2=0x7fe92c1979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.518+0000 7fe9341bc640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe92c101420 0x7fe92c1979e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe92800b4a0 tx=0x7fe92800b960 comp rx=0 tx=0).stop 2024-09-19T08:03:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.518+0000 7fe9341bc640 1 -- 172.21.15.98:0/587382706 shutdown_connections 2024-09-19T08:03:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.518+0000 7fe9341bc640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe9180a3550 0x7fe9180a5a10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.518+0000 7fe9341bc640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe92c106a80 0x7fe92c19bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.518+0000 7fe9341bc640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe92c101420 0x7fe92c1979e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.518+0000 7fe9341bc640 1 --2- 172.21.15.98:0/587382706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe92c100a80 0x7fe92c1974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.518+0000 7fe9341bc640 1 -- 172.21.15.98:0/587382706 >> 172.21.15.98:0/587382706 conn(0x7fe92c0fc7a0 msgr2=0x7fe92c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.518+0000 7fe9341bc640 1 -- 172.21.15.98:0/587382706 shutdown_connections 2024-09-19T08:03:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:08.518+0000 7fe9341bc640 1 -- 172.21.15.98:0/587382706 wait complete. 2024-09-19T08:03:09.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:08 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/587382706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:03:09.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:08 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/587382706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:03:09.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:08 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/587382706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:03:09.197 DEBUG:tasks.fwd_scrub:Yielding 2024-09-19T08:03:09.197 INFO:teuthology.run_tasks:Running task exec... 2024-09-19T08:03:09.209 INFO:teuthology.task.exec:Executing custom commands... 2024-09-19T08:03:09.209 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-09-19T08:03:09.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-09-19T08:03:09.210 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-09-19T08:03:09.210 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:03:09.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:03:09.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.314+0000 7fce1fee4640 1 -- 172.21.15.98:0/2967564355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce181009b0 msgr2=0x7fce18100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.314+0000 7fce1fee4640 1 --2- 172.21.15.98:0/2967564355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce181009b0 0x7fce18100e10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fce14009930 tx=0x7fce1402a420 comp rx=0 tx=0).stop 2024-09-19T08:03:09.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.314+0000 7fce1fee4640 1 -- 172.21.15.98:0/2967564355 shutdown_connections 2024-09-19T08:03:09.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.314+0000 7fce1fee4640 1 --2- 172.21.15.98:0/2967564355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce18101350 0x7fce1810d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.314+0000 7fce1fee4640 1 --2- 172.21.15.98:0/2967564355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce181009b0 0x7fce18100e10 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.314+0000 7fce1fee4640 1 --2- 172.21.15.98:0/2967564355 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce181069c0 0x7fce18106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.314+0000 7fce1fee4640 1 -- 172.21.15.98:0/2967564355 >> 172.21.15.98:0/2967564355 conn(0x7fce18077990 msgr2=0x7fce180fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:09.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.314+0000 7fce1fee4640 1 -- 172.21.15.98:0/2967564355 shutdown_connections 2024-09-19T08:03:09.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.314+0000 7fce1fee4640 1 -- 172.21.15.98:0/2967564355 wait complete. 2024-09-19T08:03:09.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.315+0000 7fce1fee4640 1 Processor -- start 2024-09-19T08:03:09.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.315+0000 7fce1fee4640 1 -- start start 2024-09-19T08:03:09.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.315+0000 7fce1fee4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce181009b0 0x7fce181991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.315+0000 7fce1fee4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce18101350 0x7fce18199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.315+0000 7fce1fee4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce181069c0 0x7fce181a4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.315+0000 7fce1fee4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fce18110e50 con 0x7fce181069c0 2024-09-19T08:03:09.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.315+0000 7fce1fee4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fce18110cd0 con 0x7fce181009b0 2024-09-19T08:03:09.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.315+0000 7fce1fee4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce18110fd0 con 0x7fce18101350 2024-09-19T08:03:09.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.315+0000 7fce1e481640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce181069c0 0x7fce181a4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce1e481640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce181069c0 0x7fce181a4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47258/0 (socket says 172.21.15.98:47258) 2024-09-19T08:03:09.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce1e481640 1 -- 172.21.15.98:0/2325854338 learned_addr learned my addr 172.21.15.98:0/2325854338 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:09.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce1d47f640 1 --2- 172.21.15.98:0/2325854338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce18101350 0x7fce18199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce1dc80640 1 --2- 172.21.15.98:0/2325854338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce181009b0 0x7fce181991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce1e481640 1 -- 172.21.15.98:0/2325854338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce18101350 msgr2=0x7fce18199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce1e481640 1 --2- 172.21.15.98:0/2325854338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce18101350 0x7fce18199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce1e481640 1 -- 172.21.15.98:0/2325854338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce181009b0 msgr2=0x7fce181991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce1e481640 1 --2- 172.21.15.98:0/2325854338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce181009b0 0x7fce181991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce1e481640 1 -- 172.21.15.98:0/2325854338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce181a47a0 con 0x7fce181069c0 2024-09-19T08:03:09.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce1e481640 1 --2- 172.21.15.98:0/2325854338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce181069c0 0x7fce181a4260 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fce1000cad0 tx=0x7fce1000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:09.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce0effd640 1 -- 172.21.15.98:0/2325854338 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1000ae20 con 0x7fce181069c0 2024-09-19T08:03:09.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce0effd640 1 -- 172.21.15.98:0/2325854338 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce10005020 con 0x7fce181069c0 2024-09-19T08:03:09.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce0effd640 1 -- 172.21.15.98:0/2325854338 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1000f6f0 con 0x7fce181069c0 2024-09-19T08:03:09.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce1fee4640 1 -- 172.21.15.98:0/2325854338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce181a4970 con 0x7fce181069c0 2024-09-19T08:03:09.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.316+0000 7fce1fee4640 1 -- 172.21.15.98:0/2325854338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce181a4ee0 con 0x7fce181069c0 2024-09-19T08:03:09.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.318+0000 7fce1fee4640 1 -- 172.21.15.98:0/2325854338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce18100e10 con 0x7fce181069c0 2024-09-19T08:03:09.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.318+0000 7fce0effd640 1 -- 172.21.15.98:0/2325854338 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7fce100051d0 con 0x7fce181069c0 2024-09-19T08:03:09.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.319+0000 7fce0effd640 1 --2- 172.21.15.98:0/2325854338 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fce040a3600 0x7fce040a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.319+0000 7fce1dc80640 1 --2- 172.21.15.98:0/2325854338 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fce040a3600 0x7fce040a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.319+0000 7fce0effd640 1 -- 172.21.15.98:0/2325854338 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fce100d01c0 con 0x7fce181069c0 2024-09-19T08:03:09.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.320+0000 7fce1dc80640 1 --2- 172.21.15.98:0/2325854338 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fce040a3600 0x7fce040a5ac0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fce080058c0 tx=0x7fce0800a2d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:09.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.324+0000 7fce0effd640 1 -- 172.21.15.98:0/2325854338 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fce10014030 con 0x7fce181069c0 2024-09-19T08:03:09.464 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.503+0000 7fce1fee4640 1 -- 172.21.15.98:0/2325854338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7fce181a5170 con 0x7fce181069c0 2024-09-19T08:03:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.507+0000 7fce0effd640 1 -- 172.21.15.98:0/2325854338 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v31)=0 v31) ==== 129+0+0 (secure 0 0 0) 0x7fce1009c330 con 0x7fce181069c0 2024-09-19T08:03:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.509+0000 7fce1fee4640 1 -- 172.21.15.98:0/2325854338 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fce040a3600 msgr2=0x7fce040a5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.509+0000 7fce1fee4640 1 --2- 172.21.15.98:0/2325854338 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fce040a3600 0x7fce040a5ac0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fce080058c0 tx=0x7fce0800a2d0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.509+0000 7fce1fee4640 1 -- 172.21.15.98:0/2325854338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce181069c0 msgr2=0x7fce181a4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.509+0000 7fce1fee4640 1 --2- 172.21.15.98:0/2325854338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce181069c0 0x7fce181a4260 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fce1000cad0 tx=0x7fce1000cf90 comp rx=0 tx=0).stop 2024-09-19T08:03:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.509+0000 7fce1fee4640 1 -- 172.21.15.98:0/2325854338 shutdown_connections 2024-09-19T08:03:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.509+0000 7fce1fee4640 1 --2- 172.21.15.98:0/2325854338 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fce040a3600 0x7fce040a5ac0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.509+0000 7fce1fee4640 1 --2- 172.21.15.98:0/2325854338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce181069c0 0x7fce181a4260 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.509+0000 7fce1fee4640 1 --2- 172.21.15.98:0/2325854338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce18101350 0x7fce18199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.509+0000 7fce1fee4640 1 --2- 172.21.15.98:0/2325854338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce181009b0 0x7fce181991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.509+0000 7fce1fee4640 1 -- 172.21.15.98:0/2325854338 >> 172.21.15.98:0/2325854338 conn(0x7fce18077990 msgr2=0x7fce181036f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.509+0000 7fce1fee4640 1 -- 172.21.15.98:0/2325854338 shutdown_connections 2024-09-19T08:03:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.509+0000 7fce1fee4640 1 -- 172.21.15.98:0/2325854338 wait complete. 2024-09-19T08:03:09.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-09-19T08:03:09.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.626+0000 7effcaf0a640 1 -- 172.21.15.98:0/3497818234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc4101ad0 msgr2=0x7effc410d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.626+0000 7effcaf0a640 1 --2- 172.21.15.98:0/3497818234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc4101ad0 0x7effc410d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7effbc009ca0 tx=0x7effbc02a8c0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.626+0000 7effcaf0a640 1 -- 172.21.15.98:0/3497818234 shutdown_connections 2024-09-19T08:03:09.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.626+0000 7effcaf0a640 1 --2- 172.21.15.98:0/3497818234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc4101ad0 0x7effc410d990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.626+0000 7effcaf0a640 1 --2- 172.21.15.98:0/3497818234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7effc4101150 0x7effc4101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.626+0000 7effcaf0a640 1 --2- 172.21.15.98:0/3497818234 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7effc41028d0 0x7effc4102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.626+0000 7effcaf0a640 1 -- 172.21.15.98:0/3497818234 >> 172.21.15.98:0/3497818234 conn(0x7effc40770f0 msgr2=0x7effc40fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:09.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.626+0000 7effcaf0a640 1 -- 172.21.15.98:0/3497818234 shutdown_connections 2024-09-19T08:03:09.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.626+0000 7effcaf0a640 1 -- 172.21.15.98:0/3497818234 wait complete. 2024-09-19T08:03:09.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.626+0000 7effcaf0a640 1 Processor -- start 2024-09-19T08:03:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.626+0000 7effcaf0a640 1 -- start start 2024-09-19T08:03:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effcaf0a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7effc4101150 0x7effc41982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effcaf0a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc4101ad0 0x7effc4198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effcaf0a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7effc41028d0 0x7effc419cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effcaf0a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7effc4110eb0 con 0x7effc4101ad0 2024-09-19T08:03:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effcaf0a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7effc4110d30 con 0x7effc4101150 2024-09-19T08:03:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effcaf0a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7effc4111030 con 0x7effc41028d0 2024-09-19T08:03:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effc3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc4101ad0 0x7effc4198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effc3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc4101ad0 0x7effc4198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47272/0 (socket says 172.21.15.98:47272) 2024-09-19T08:03:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effc3fff640 1 -- 172.21.15.98:0/2892987531 learned_addr learned my addr 172.21.15.98:0/2892987531 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effc94a7640 1 --2- 172.21.15.98:0/2892987531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7effc41028d0 0x7effc419cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effc8ca6640 1 --2- 172.21.15.98:0/2892987531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7effc4101150 0x7effc41982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effc3fff640 1 -- 172.21.15.98:0/2892987531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7effc41028d0 msgr2=0x7effc419cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effc3fff640 1 --2- 172.21.15.98:0/2892987531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7effc41028d0 0x7effc419cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effc3fff640 1 -- 172.21.15.98:0/2892987531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7effc4101150 msgr2=0x7effc41982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effc3fff640 1 --2- 172.21.15.98:0/2892987531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7effc4101150 0x7effc41982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effc3fff640 1 -- 172.21.15.98:0/2892987531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc419d2e0 con 0x7effc4101ad0 2024-09-19T08:03:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effc3fff640 1 --2- 172.21.15.98:0/2892987531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc4101ad0 0x7effc4198830 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7effb000d950 tx=0x7effb000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effc1ffb640 1 -- 172.21.15.98:0/2892987531 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0017030 con 0x7effc4101ad0 2024-09-19T08:03:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effc1ffb640 1 -- 172.21.15.98:0/2892987531 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effb0004020 con 0x7effc4101ad0 2024-09-19T08:03:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effcaf0a640 1 -- 172.21.15.98:0/2892987531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc419d5d0 con 0x7effc4101ad0 2024-09-19T08:03:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.627+0000 7effc1ffb640 1 -- 172.21.15.98:0/2892987531 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb00047a0 con 0x7effc4101ad0 2024-09-19T08:03:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.628+0000 7effcaf0a640 1 -- 172.21.15.98:0/2892987531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc41a4e60 con 0x7effc4101ad0 2024-09-19T08:03:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.628+0000 7effcaf0a640 1 -- 172.21.15.98:0/2892987531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effc4101590 con 0x7effc4101ad0 2024-09-19T08:03:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.629+0000 7effc1ffb640 1 -- 172.21.15.98:0/2892987531 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7effb00041d0 con 0x7effc4101ad0 2024-09-19T08:03:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.630+0000 7effc1ffb640 1 --2- 172.21.15.98:0/2892987531 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7effb40a3580 0x7effb40a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.630+0000 7effc8ca6640 1 --2- 172.21.15.98:0/2892987531 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7effb40a3580 0x7effb40a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.631+0000 7effc1ffb640 1 -- 172.21.15.98:0/2892987531 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7effb00cf960 con 0x7effc4101ad0 2024-09-19T08:03:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.631+0000 7effc8ca6640 1 --2- 172.21.15.98:0/2892987531 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7effb40a3580 0x7effb40a5a40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7effb800bb60 tx=0x7effb800b390 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:09.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.635+0000 7effc1ffb640 1 -- 172.21.15.98:0/2892987531 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7effb009c120 con 0x7effc4101ad0 2024-09-19T08:03:09.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.825+0000 7effcaf0a640 1 -- 172.21.15.98:0/2892987531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7effc41a5030 con 0x7effc4101ad0 2024-09-19T08:03:09.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.831+0000 7effc1ffb640 1 -- 172.21.15.98:0/2892987531 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v32)=0 v32) ==== 138+0+0 (secure 0 0 0) 0x7effb009ba20 con 0x7effc4101ad0 2024-09-19T08:03:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.834+0000 7effcaf0a640 1 -- 172.21.15.98:0/2892987531 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7effb40a3580 msgr2=0x7effb40a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.834+0000 7effcaf0a640 1 --2- 172.21.15.98:0/2892987531 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7effb40a3580 0x7effb40a5a40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7effb800bb60 tx=0x7effb800b390 comp rx=0 tx=0).stop 2024-09-19T08:03:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.834+0000 7effcaf0a640 1 -- 172.21.15.98:0/2892987531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc4101ad0 msgr2=0x7effc4198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.834+0000 7effcaf0a640 1 --2- 172.21.15.98:0/2892987531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc4101ad0 0x7effc4198830 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7effb000d950 tx=0x7effb000de10 comp rx=0 tx=0).stop 2024-09-19T08:03:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.834+0000 7effcaf0a640 1 -- 172.21.15.98:0/2892987531 shutdown_connections 2024-09-19T08:03:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.834+0000 7effcaf0a640 1 --2- 172.21.15.98:0/2892987531 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7effb40a3580 0x7effb40a5a40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.834+0000 7effcaf0a640 1 --2- 172.21.15.98:0/2892987531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7effc41028d0 0x7effc419cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.834+0000 7effcaf0a640 1 --2- 172.21.15.98:0/2892987531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc4101ad0 0x7effc4198830 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.834+0000 7effcaf0a640 1 --2- 172.21.15.98:0/2892987531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7effc4101150 0x7effc41982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.834+0000 7effcaf0a640 1 -- 172.21.15.98:0/2892987531 >> 172.21.15.98:0/2892987531 conn(0x7effc40770f0 msgr2=0x7effc4100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:09.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.834+0000 7effcaf0a640 1 -- 172.21.15.98:0/2892987531 shutdown_connections 2024-09-19T08:03:09.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.834+0000 7effcaf0a640 1 -- 172.21.15.98:0/2892987531 wait complete. 2024-09-19T08:03:09.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-09-19T08:03:09.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.951+0000 7f09083b9640 1 -- 172.21.15.98:0/852708379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09001009b0 msgr2=0x7f0900100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.951+0000 7f09083b9640 1 --2- 172.21.15.98:0/852708379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09001009b0 0x7f0900100e10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f08f0009930 tx=0x7f08f002a420 comp rx=0 tx=0).stop 2024-09-19T08:03:09.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.951+0000 7f09083b9640 1 -- 172.21.15.98:0/852708379 shutdown_connections 2024-09-19T08:03:09.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.951+0000 7f09083b9640 1 --2- 172.21.15.98:0/852708379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0900101350 0x7f090010d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.951+0000 7f09083b9640 1 --2- 172.21.15.98:0/852708379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09001009b0 0x7f0900100e10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.951+0000 7f09083b9640 1 --2- 172.21.15.98:0/852708379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09001069c0 0x7f0900106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.951+0000 7f09083b9640 1 -- 172.21.15.98:0/852708379 >> 172.21.15.98:0/852708379 conn(0x7f0900077990 msgr2=0x7f09000fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:09.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.951+0000 7f09083b9640 1 -- 172.21.15.98:0/852708379 shutdown_connections 2024-09-19T08:03:09.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.951+0000 7f09083b9640 1 -- 172.21.15.98:0/852708379 wait complete. 2024-09-19T08:03:09.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.952+0000 7f09083b9640 1 Processor -- start 2024-09-19T08:03:09.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.952+0000 7f09083b9640 1 -- start start 2024-09-19T08:03:09.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.952+0000 7f09083b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09001009b0 0x7f0900198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.952+0000 7f09083b9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0900101350 0x7f0900198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.952+0000 7f09083b9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09001069c0 0x7f090019cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.952+0000 7f09083b9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0900110e00 con 0x7f0900101350 2024-09-19T08:03:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.952+0000 7f09083b9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0900110c80 con 0x7f09001069c0 2024-09-19T08:03:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.952+0000 7f09083b9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0900110f80 con 0x7f09001009b0 2024-09-19T08:03:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.952+0000 7f0905954640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0900101350 0x7f0900198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.952+0000 7f0905954640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0900101350 0x7f0900198890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47296/0 (socket says 172.21.15.98:47296) 2024-09-19T08:03:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.952+0000 7f0905954640 1 -- 172.21.15.98:0/1973682586 learned_addr learned my addr 172.21.15.98:0/1973682586 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.952+0000 7f0906155640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09001009b0 0x7f0900198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.953+0000 7f0906956640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09001069c0 0x7f090019cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.953+0000 7f0905954640 1 -- 172.21.15.98:0/1973682586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09001009b0 msgr2=0x7f0900198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.953+0000 7f0905954640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09001009b0 0x7f0900198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.953+0000 7f0905954640 1 -- 172.21.15.98:0/1973682586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09001069c0 msgr2=0x7f090019cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.953+0000 7f0905954640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09001069c0 0x7f090019cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.953+0000 7f0905954640 1 -- 172.21.15.98:0/1973682586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090019d340 con 0x7f0900101350 2024-09-19T08:03:09.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.953+0000 7f0906155640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09001009b0 0x7f0900198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:09.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.953+0000 7f0905954640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0900101350 0x7f0900198890 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f08f0009230 tx=0x7f08f0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:09.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.953+0000 7f08f77fe640 1 -- 172.21.15.98:0/1973682586 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f003f030 con 0x7f0900101350 2024-09-19T08:03:09.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.953+0000 7f08f77fe640 1 -- 172.21.15.98:0/1973682586 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08f0003ad0 con 0x7f0900101350 2024-09-19T08:03:09.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.953+0000 7f08f77fe640 1 -- 172.21.15.98:0/1973682586 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f0035070 con 0x7f0900101350 2024-09-19T08:03:09.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.953+0000 7f09083b9640 1 -- 172.21.15.98:0/1973682586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f090019d5d0 con 0x7f0900101350 2024-09-19T08:03:09.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.953+0000 7f09083b9640 1 -- 172.21.15.98:0/1973682586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09001a4ec0 con 0x7f0900101350 2024-09-19T08:03:09.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.954+0000 7f09083b9640 1 -- 172.21.15.98:0/1973682586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09001020f0 con 0x7f0900101350 2024-09-19T08:03:09.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.956+0000 7f08f77fe640 1 -- 172.21.15.98:0/1973682586 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f08f0042030 con 0x7f0900101350 2024-09-19T08:03:09.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.957+0000 7f08f77fe640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08ec0a3580 0x7f08ec0a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.958+0000 7f0906155640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08ec0a3580 0x7f08ec0a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.958+0000 7f08f77fe640 1 -- 172.21.15.98:0/1973682586 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f08f00f08e0 con 0x7f0900101350 2024-09-19T08:03:09.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.958+0000 7f0906155640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08ec0a3580 0x7f08ec0a5a40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009ba0 tx=0x7f08fc005f90 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:09.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.960+0000 7f08f77fe640 1 -- 172.21.15.98:0/1973682586 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f08f00bd170 con 0x7f0900101350 2024-09-19T08:03:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.971+0000 7f36c7c32640 1 -- 172.21.15.98:0/2942137277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36c0103740 msgr2=0x7f36c0107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.971+0000 7f36c7c32640 1 --2- 172.21.15.98:0/2942137277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36c0103740 0x7f36c0107b80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009930 tx=0x7f36b002a420 comp rx=0 tx=0).stop 2024-09-19T08:03:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.972+0000 7f36c7c32640 1 -- 172.21.15.98:0/2942137277 shutdown_connections 2024-09-19T08:03:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.972+0000 7f36c7c32640 1 --2- 172.21.15.98:0/2942137277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c0108290 0x7f36c010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.972+0000 7f36c7c32640 1 --2- 172.21.15.98:0/2942137277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36c0103740 0x7f36c0107b80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.972+0000 7f36c7c32640 1 --2- 172.21.15.98:0/2942137277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36c0102d90 0x7f36c0103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.972+0000 7f36c7c32640 1 -- 172.21.15.98:0/2942137277 >> 172.21.15.98:0/2942137277 conn(0x7f36c00fc7d0 msgr2=0x7f36c00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.972+0000 7f36c7c32640 1 -- 172.21.15.98:0/2942137277 shutdown_connections 2024-09-19T08:03:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.972+0000 7f36c7c32640 1 -- 172.21.15.98:0/2942137277 wait complete. 2024-09-19T08:03:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.973+0000 7f36c7c32640 1 Processor -- start 2024-09-19T08:03:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.973+0000 7f36c7c32640 1 -- start start 2024-09-19T08:03:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.973+0000 7f36c7c32640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c0102d90 0x7f36c01009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.973+0000 7f36c7c32640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36c0108290 0x7f36c0100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.973+0000 7f36c7c32640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36c0101470 0x7f36c01a6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.973+0000 7f36c7c32640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f36c0112050 con 0x7f36c0101470 2024-09-19T08:03:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.973+0000 7f36c7c32640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f36c0111ed0 con 0x7f36c0108290 2024-09-19T08:03:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.973+0000 7f36c7c32640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36c01121d0 con 0x7f36c0102d90 2024-09-19T08:03:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36c61cf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36c0101470 0x7f36c01a6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36c61cf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36c0101470 0x7f36c01a6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47316/0 (socket says 172.21.15.98:47316) 2024-09-19T08:03:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36c61cf640 1 -- 172.21.15.98:0/1353722751 learned_addr learned my addr 172.21.15.98:0/1353722751 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36c51cd640 1 --2- 172.21.15.98:0/1353722751 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36c0108290 0x7f36c0100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36c59ce640 1 --2- 172.21.15.98:0/1353722751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c0102d90 0x7f36c01009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36c61cf640 1 -- 172.21.15.98:0/1353722751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c0102d90 msgr2=0x7f36c01009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36c61cf640 1 --2- 172.21.15.98:0/1353722751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c0102d90 0x7f36c01009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36c61cf640 1 -- 172.21.15.98:0/1353722751 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36c0108290 msgr2=0x7f36c0100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36c61cf640 1 --2- 172.21.15.98:0/1353722751 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36c0108290 0x7f36c0100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36c61cf640 1 -- 172.21.15.98:0/1353722751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36c01a7030 con 0x7f36c0101470 2024-09-19T08:03:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36c61cf640 1 --2- 172.21.15.98:0/1353722751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36c0101470 0x7f36c01a6960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f36b800eaa0 tx=0x7f36b800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36b6ffd640 1 -- 172.21.15.98:0/1353722751 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b800cdb0 con 0x7f36c0101470 2024-09-19T08:03:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36b6ffd640 1 -- 172.21.15.98:0/1353722751 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36b8005020 con 0x7f36c0101470 2024-09-19T08:03:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36b6ffd640 1 -- 172.21.15.98:0/1353722751 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b80116d0 con 0x7f36c0101470 2024-09-19T08:03:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36c7c32640 1 -- 172.21.15.98:0/1353722751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36c01a7320 con 0x7f36c0101470 2024-09-19T08:03:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.974+0000 7f36c7c32640 1 -- 172.21.15.98:0/1353722751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36c01a77e0 con 0x7f36c0101470 2024-09-19T08:03:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.977+0000 7f36b6ffd640 1 -- 172.21.15.98:0/1353722751 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f36b8005560 con 0x7f36c0101470 2024-09-19T08:03:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.977+0000 7f36ca00b740 1 Processor -- start 2024-09-19T08:03:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.977+0000 7f36ca00b740 1 -- start start 2024-09-19T08:03:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.977+0000 7f36ca00b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b0b60 0x5584a38b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.977+0000 7f36ca00b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38b1480 0x5584a38bb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.977+0000 7f36ca00b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38bbd40 0x5584a38be130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.977+0000 7f36ca00b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5584a3816280 con 0x5584a38bbd40 2024-09-19T08:03:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.977+0000 7f36ca00b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5584a37bdb10 con 0x5584a38b1480 2024-09-19T08:03:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.977+0000 7f36ca00b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5584a3812240 con 0x5584a38b0b60 2024-09-19T08:03:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.977+0000 7f36c51cd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38bbd40 0x5584a38be130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.977+0000 7f36c51cd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38bbd40 0x5584a38be130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47324/0 (socket says 172.21.15.98:47324) 2024-09-19T08:03:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.978+0000 7f36c51cd640 1 -- 172.21.15.98:0/2490269349 learned_addr learned my addr 172.21.15.98:0/2490269349 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.978+0000 7f36c59ce640 1 --2- 172.21.15.98:0/2490269349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38b1480 0x5584a38bb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.978+0000 7f36c51cd640 1 --2- 172.21.15.98:0/2490269349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b0b60 0x5584a38b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.978+0000 7f36b6ffd640 1 --2- 172.21.15.98:0/1353722751 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f36ac0a34d0 0x7f36ac0a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.978+0000 7f36c51cd640 1 -- 172.21.15.98:0/2490269349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b0b60 msgr2=0x5584a38b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.978+0000 7f36c51cd640 1 --2- 172.21.15.98:0/2490269349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b0b60 0x5584a38b0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.978+0000 7f36c51cd640 1 -- 172.21.15.98:0/2490269349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38b1480 msgr2=0x5584a38bb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.978+0000 7f36c51cd640 1 --2- 172.21.15.98:0/2490269349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38b1480 0x5584a38bb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.978+0000 7f36c51cd640 1 -- 172.21.15.98:0/2490269349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584a37ca1f0 con 0x5584a38bbd40 2024-09-19T08:03:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.978+0000 7f36c59ce640 1 --2- 172.21.15.98:0/2490269349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38b1480 0x5584a38bb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.978+0000 7f36c61cf640 1 --2- 172.21.15.98:0/1353722751 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f36ac0a34d0 0x7f36ac0a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.978+0000 7f36b6ffd640 1 -- 172.21.15.98:0/1353722751 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f36b80d00a0 con 0x7f36c0101470 2024-09-19T08:03:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.978+0000 7f36c51cd640 1 --2- 172.21.15.98:0/2490269349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38bbd40 0x5584a38be130 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009900 tx=0x7f36b0005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52f9831daa6ad8f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.978+0000 7f36c61cf640 1 --2- 172.21.15.98:0/1353722751 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f36ac0a34d0 0x7f36ac0a5990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f36b80cd700 tx=0x7f36b801eb40 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.979+0000 7f36abfff640 1 -- 172.21.15.98:0/2490269349 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b0035070 con 0x5584a38bbd40 2024-09-19T08:03:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.979+0000 7f36abfff640 1 -- 172.21.15.98:0/2490269349 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36b0004030 con 0x5584a38bbd40 2024-09-19T08:03:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.979+0000 7f36abfff640 1 -- 172.21.15.98:0/2490269349 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b0003be0 con 0x5584a38bbd40 2024-09-19T08:03:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.979+0000 7f36ca00b740 1 -- 172.21.15.98:0/2490269349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38bbd40 msgr2=0x5584a38be130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.979+0000 7f36ca00b740 1 --2- 172.21.15.98:0/2490269349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38bbd40 0x5584a38be130 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009900 tx=0x7f36b0005780 comp rx=0 tx=0).stop 2024-09-19T08:03:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.979+0000 7f36ca00b740 1 -- 172.21.15.98:0/2490269349 shutdown_connections 2024-09-19T08:03:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.979+0000 7f36ca00b740 1 --2- 172.21.15.98:0/2490269349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38bbd40 0x5584a38be130 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.979+0000 7f36ca00b740 1 --2- 172.21.15.98:0/2490269349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38b1480 0x5584a38bb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.979+0000 7f36ca00b740 1 --2- 172.21.15.98:0/2490269349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b0b60 0x5584a38b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.979+0000 7f36ca00b740 1 -- 172.21.15.98:0/2490269349 >> 172.21.15.98:0/2490269349 conn(0x5584a38ad0f0 msgr2=0x5584a38ad4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.979+0000 7f36ca00b740 1 -- 172.21.15.98:0/2490269349 shutdown_connections 2024-09-19T08:03:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.979+0000 7f36ca00b740 1 -- 172.21.15.98:0/2490269349 wait complete. 2024-09-19T08:03:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.980+0000 7f36ca00b740 1 -- start start 2024-09-19T08:03:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.980+0000 7f36ca00b740 1 Processor -- start 2024-09-19T08:03:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.980+0000 7f36ca00b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b0b60 0x5584a38bb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.980+0000 7f36ca00b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b1480 0x5584a38b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.980+0000 7f36ca00b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38bbd40 0x5584a38b9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.980+0000 7f36ca00b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5584a38be910 con 0x5584a38b1480 2024-09-19T08:03:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.980+0000 7f36ca00b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5584a37d0f10 con 0x5584a38bbd40 2024-09-19T08:03:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.980+0000 7f36ca00b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5584a38bea90 con 0x5584a38b0b60 2024-09-19T08:03:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.980+0000 7f36ca00b740 10 client.0 _ll_register_callbacks cb 0x5584a3799a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36c59ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b1480 0x5584a38b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36c51cd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38bbd40 0x5584a38b9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36c51cd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38bbd40 0x5584a38b9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:35060/0 (socket says 172.21.15.98:35060) 2024-09-19T08:03:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36c51cd640 1 -- 172.21.15.98:0/3194215563 learned_addr learned my addr 172.21.15.98:0/3194215563 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36c59ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b1480 0x5584a38b8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47334/0 (socket says 172.21.15.98:47334) 2024-09-19T08:03:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36c51cd640 1 --2- 172.21.15.98:0/3194215563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b0b60 0x5584a38bb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36c51cd640 1 -- 172.21.15.98:0/3194215563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b0b60 msgr2=0x5584a38bb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36c51cd640 1 --2- 172.21.15.98:0/3194215563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b0b60 0x5584a38bb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36c51cd640 1 -- 172.21.15.98:0/3194215563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b1480 msgr2=0x5584a38b8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36c51cd640 1 --2- 172.21.15.98:0/3194215563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b1480 0x5584a38b8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36c51cd640 1 -- 172.21.15.98:0/3194215563 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584a35876a0 con 0x5584a38bbd40 2024-09-19T08:03:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36c59ce640 1 --2- 172.21.15.98:0/3194215563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b1480 0x5584a38b8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36c51cd640 1 --2- 172.21.15.98:0/3194215563 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38bbd40 0x5584a38b9280 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009850 tx=0x7f36b0003720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6914f32f98814e43 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-09-19T08:03:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36a97fa640 1 -- 172.21.15.98:0/3194215563 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b0035070 con 0x5584a38bbd40 2024-09-19T08:03:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36a97fa640 1 -- 172.21.15.98:0/3194215563 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f36b0052040 con 0x5584a38bbd40 2024-09-19T08:03:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36ca00b740 1 -- 172.21.15.98:0/3194215563 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36b0004030 con 0x5584a38bbd40 2024-09-19T08:03:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36a97fa640 1 -- 172.21.15.98:0/3194215563 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f36b005c050 con 0x5584a38bbd40 2024-09-19T08:03:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36ca00b740 10 client.24899 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.981+0000 7f36ca00b740 1 -- 172.21.15.98:0/3194215563 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5584a38be910 con 0x5584a38bbd40 2024-09-19T08:03:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.982+0000 7f36a97fa640 1 -- 172.21.15.98:0/3194215563 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36b0004030 con 0x5584a38bbd40 2024-09-19T08:03:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.982+0000 7f36a97fa640 1 -- 172.21.15.98:0/3194215563 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f36b003d6f0 con 0x5584a38bbd40 2024-09-19T08:03:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.982+0000 7f36ca00b740 10 client.24899 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.982+0000 7f36ca00b740 10 client.24899 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:03:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.982+0000 7f36ca00b740 20 client.24899 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.982+0000 7f36ca00b740 1 --2- 172.21.15.98:0/3194215563 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5584a38cb5a0 0x5584a38cd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.982+0000 7f36ca00b740 4 client.24899 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.982+0000 7f36ca00b740 1 -- 172.21.15.98:0/3194215563 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584a38237f0 con 0x5584a38cb5a0 2024-09-19T08:03:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.982+0000 7f36c59ce640 1 --2- 172.21.15.98:0/3194215563 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5584a38cb5a0 0x5584a38cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.983+0000 7f36c59ce640 1 --2- 172.21.15.98:0/3194215563 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5584a38cb5a0 0x5584a38cd990 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0b85dfb2d660223 server_cookie=8df1960a279eda08 in_seq=0 out_seq=0 2024-09-19T08:03:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.983+0000 7f36a97fa640 10 client.24899 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.983+0000 7f36a97fa640 1 -- 172.21.15.98:0/3194215563 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5584a38237f0 con 0x5584a38cb5a0 2024-09-19T08:03:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.983+0000 7f36a97fa640 10 client.24899 handle_command_reply: tid=0 2024-09-19T08:03:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.983+0000 7f36ca00b740 1 client.24899 shutdown 2024-09-19T08:03:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.983+0000 7f36ca00b740 1 -- 172.21.15.98:0/3194215563 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38bbd40 msgr2=0x5584a38b9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.983+0000 7f36ca00b740 1 --2- 172.21.15.98:0/3194215563 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38bbd40 0x5584a38b9280 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009850 tx=0x7f36b0003720 comp rx=0 tx=0).stop 2024-09-19T08:03:09.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.983+0000 7f36ca00b740 1 -- 172.21.15.98:0/3194215563 shutdown_connections 2024-09-19T08:03:09.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.983+0000 7f36ca00b740 1 --2- 172.21.15.98:0/3194215563 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5584a38cb5a0 0x5584a38cd990 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.983+0000 7f36ca00b740 1 --2- 172.21.15.98:0/3194215563 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38bbd40 0x5584a38b9280 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.983+0000 7f36ca00b740 1 --2- 172.21.15.98:0/3194215563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b1480 0x5584a38b8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.984+0000 7f36ca00b740 1 --2- 172.21.15.98:0/3194215563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b0b60 0x5584a38bb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:09.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.984+0000 7f36a97fa640 0 client.24899 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:09.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.984+0000 7f36ca00b740 1 -- 172.21.15.98:0/3194215563 >> 172.21.15.98:0/3194215563 conn(0x5584a38ad0f0 msgr2=0x5584a38b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:09.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.984+0000 7f36ca00b740 1 -- 172.21.15.98:0/3194215563 shutdown_connections 2024-09-19T08:03:09.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.984+0000 7f36ca00b740 1 -- 172.21.15.98:0/3194215563 wait complete. 2024-09-19T08:03:09.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:09.984+0000 7f36ca00b740 20 client.24899 trim_cache size 0 max 16384 2024-09-19T08:03:10.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36ca00b740 1 Processor -- start 2024-09-19T08:03:10.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36ca00b740 1 -- start start 2024-09-19T08:03:10.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36ca00b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b0b60 0x5584a38b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36ca00b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b1480 0x5584a38b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36ca00b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38bbd40 0x5584a38d30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36ca00b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f36c01affe0 con 0x5584a38b0b60 2024-09-19T08:03:10.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36ca00b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f36c01b1b20 con 0x5584a38bbd40 2024-09-19T08:03:10.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36ca00b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36c01b06b0 con 0x5584a38b1480 2024-09-19T08:03:10.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36c51cd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b0b60 0x5584a38b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:10.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36c51cd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b0b60 0x5584a38b7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47348/0 (socket says 172.21.15.98:47348) 2024-09-19T08:03:10.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36c51cd640 1 -- 172.21.15.98:0/2902897967 learned_addr learned my addr 172.21.15.98:0/2902897967 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36c59ce640 1 --2- 172.21.15.98:0/2902897967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b1480 0x5584a38b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36c51cd640 1 --2- 172.21.15.98:0/2902897967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38bbd40 0x5584a38d30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36c51cd640 1 -- 172.21.15.98:0/2902897967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b1480 msgr2=0x5584a38b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36c51cd640 1 --2- 172.21.15.98:0/2902897967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b1480 0x5584a38b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36c51cd640 1 -- 172.21.15.98:0/2902897967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38bbd40 msgr2=0x5584a38d30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36c51cd640 1 --2- 172.21.15.98:0/2902897967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38bbd40 0x5584a38d30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.001+0000 7f36c51cd640 1 -- 172.21.15.98:0/2902897967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584a38acf20 con 0x5584a38b0b60 2024-09-19T08:03:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.002+0000 7f36c51cd640 1 --2- 172.21.15.98:0/2902897967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b0b60 0x5584a38b7a30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f36b0033000 tx=0x7f36b002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a4c8d52c69cffa server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.002+0000 7f36ab7fe640 1 -- 172.21.15.98:0/2902897967 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b0035070 con 0x5584a38b0b60 2024-09-19T08:03:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.002+0000 7f36ab7fe640 1 -- 172.21.15.98:0/2902897967 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36b0004770 con 0x5584a38b0b60 2024-09-19T08:03:10.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.002+0000 7f36ab7fe640 1 -- 172.21.15.98:0/2902897967 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b0005290 con 0x5584a38b0b60 2024-09-19T08:03:10.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.003+0000 7f36ca00b740 1 -- 172.21.15.98:0/2902897967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b0b60 msgr2=0x5584a38b7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.003+0000 7f36ca00b740 1 --2- 172.21.15.98:0/2902897967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b0b60 0x5584a38b7a30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f36b0033000 tx=0x7f36b002a420 comp rx=0 tx=0).stop 2024-09-19T08:03:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.003+0000 7f36ca00b740 1 -- 172.21.15.98:0/2902897967 shutdown_connections 2024-09-19T08:03:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.003+0000 7f36ca00b740 1 --2- 172.21.15.98:0/2902897967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38bbd40 0x5584a38d30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.003+0000 7f36ca00b740 1 --2- 172.21.15.98:0/2902897967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38b1480 0x5584a38b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.003+0000 7f36ca00b740 1 --2- 172.21.15.98:0/2902897967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b0b60 0x5584a38b7a30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.003+0000 7f36ca00b740 1 -- 172.21.15.98:0/2902897967 >> 172.21.15.98:0/2902897967 conn(0x5584a38ad0f0 msgr2=0x5584a38b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.003+0000 7f36ca00b740 1 -- 172.21.15.98:0/2902897967 shutdown_connections 2024-09-19T08:03:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.003+0000 7f36ca00b740 1 -- 172.21.15.98:0/2902897967 wait complete. 2024-09-19T08:03:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.003+0000 7f36ca00b740 1 -- start start 2024-09-19T08:03:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.003+0000 7f36ca00b740 1 Processor -- start 2024-09-19T08:03:10.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.004+0000 7f36ca00b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38b0b60 0x5584a38c4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.004+0000 7f36ca00b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b1480 0x5584a38becd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.004+0000 7f36ca00b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38bbd40 0x5584a38bf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.004+0000 7f36ca00b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f36c01b1450 con 0x5584a38b1480 2024-09-19T08:03:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.004+0000 7f36ca00b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f36c01b0d80 con 0x5584a38b0b60 2024-09-19T08:03:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.004+0000 7f36ca00b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36c01adfc0 con 0x5584a38bbd40 2024-09-19T08:03:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.004+0000 7f36c59ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b1480 0x5584a38becd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.004+0000 7f36c59ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b1480 0x5584a38becd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47354/0 (socket says 172.21.15.98:47354) 2024-09-19T08:03:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.004+0000 7f36c59ce640 1 -- 172.21.15.98:0/2358557053 learned_addr learned my addr 172.21.15.98:0/2358557053 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.004+0000 7f36c51cd640 1 --2- 172.21.15.98:0/2358557053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38b0b60 0x5584a38c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.004+0000 7f36c51cd640 1 --2- 172.21.15.98:0/2358557053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38bbd40 0x5584a38bf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.005+0000 7f36c59ce640 1 -- 172.21.15.98:0/2358557053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38bbd40 msgr2=0x5584a38bf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.005+0000 7f36c59ce640 1 --2- 172.21.15.98:0/2358557053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38bbd40 0x5584a38bf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.005+0000 7f36c59ce640 1 -- 172.21.15.98:0/2358557053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38b0b60 msgr2=0x5584a38c4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:10.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.005+0000 7f36c59ce640 1 --2- 172.21.15.98:0/2358557053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38b0b60 0x5584a38c4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.005+0000 7f36c59ce640 1 -- 172.21.15.98:0/2358557053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584a38bfa10 con 0x5584a38b1480 2024-09-19T08:03:10.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.005+0000 7f36ca00b740 10 client.0 _ll_register_callbacks cb 0x7f36b0004200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:10.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.005+0000 7f36c59ce640 1 --2- 172.21.15.98:0/2358557053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b1480 0x5584a38becd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x5584a38adf50 tx=0x7f36bc013980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbea4ee4781d2d9e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:10.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.005+0000 7f36a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:10.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.005+0000 7f36ca00b740 1 -- 172.21.15.98:0/2358557053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36b0004770 con 0x5584a38b1480 2024-09-19T08:03:10.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.005+0000 7f36a97fa640 1 -- 172.21.15.98:0/2358557053 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc004220 con 0x5584a38b1480 2024-09-19T08:03:10.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.005+0000 7f36a97fa640 1 -- 172.21.15.98:0/2358557053 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f36bc0043c0 con 0x5584a38b1480 2024-09-19T08:03:10.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.005+0000 7f36a97fa640 1 -- 172.21.15.98:0/2358557053 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f36bc005020 con 0x5584a38b1480 2024-09-19T08:03:10.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.005+0000 7f36ca00b740 10 client.15414 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:10.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.005+0000 7f36ca00b740 1 -- 172.21.15.98:0/2358557053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5584a3936ff0 con 0x5584a38b1480 2024-09-19T08:03:10.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.006+0000 7f36a97fa640 1 -- 172.21.15.98:0/2358557053 <== mon.0 v2:172.21.15.98:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f36bc0051c0 con 0x5584a38b1480 2024-09-19T08:03:10.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.006+0000 7f36ca00b740 10 client.15414 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:10.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.006+0000 7f36ca00b740 10 client.15414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:03:10.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.006+0000 7f36ca00b740 20 client.15414 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:10.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.006+0000 7f36ca00b740 1 --2- 172.21.15.98:0/2358557053 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5584a39375c0 0x5584a3939a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.006+0000 7f36ca00b740 4 client.15414 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:03:10.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.006+0000 7f36ca00b740 1 -- 172.21.15.98:0/2358557053 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5584a381dd00 con 0x5584a39375c0 2024-09-19T08:03:10.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.006+0000 7f36c51cd640 1 --2- 172.21.15.98:0/2358557053 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5584a39375c0 0x5584a3939a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:10.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.006+0000 7f36c51cd640 1 --2- 172.21.15.98:0/2358557053 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5584a39375c0 0x5584a3939a00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0de89cde0dbb6b2 server_cookie=2126d30cf3390c6a in_seq=0 out_seq=0 2024-09-19T08:03:10.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.006+0000 7f36a97fa640 10 client.15414 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:10.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.007+0000 7f36a97fa640 1 -- 172.21.15.98:0/2358557053 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5584a381dd00 con 0x5584a39375c0 2024-09-19T08:03:10.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.007+0000 7f36a97fa640 10 client.15414 handle_command_reply: tid=0 2024-09-19T08:03:10.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.007+0000 7f36ca00b740 1 client.15414 shutdown 2024-09-19T08:03:10.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36ca00b740 1 -- 172.21.15.98:0/2358557053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b1480 msgr2=0x5584a38becd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36ca00b740 1 --2- 172.21.15.98:0/2358557053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b1480 0x5584a38becd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x5584a38adf50 tx=0x7f36bc013980 comp rx=0 tx=0).stop 2024-09-19T08:03:10.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36ca00b740 1 -- 172.21.15.98:0/2358557053 shutdown_connections 2024-09-19T08:03:10.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36ca00b740 1 --2- 172.21.15.98:0/2358557053 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5584a39375c0 0x5584a3939a00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36ca00b740 1 --2- 172.21.15.98:0/2358557053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5584a38bbd40 0x5584a38bf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36ca00b740 1 --2- 172.21.15.98:0/2358557053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584a38b1480 0x5584a38becd0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36ca00b740 1 --2- 172.21.15.98:0/2358557053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584a38b0b60 0x5584a38c4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36a97fa640 0 client.15414 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36ca00b740 1 -- 172.21.15.98:0/2358557053 >> 172.21.15.98:0/2358557053 conn(0x5584a38ad0f0 msgr2=0x5584a38ba980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36ca00b740 1 -- 172.21.15.98:0/2358557053 shutdown_connections 2024-09-19T08:03:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36ca00b740 1 -- 172.21.15.98:0/2358557053 wait complete. 2024-09-19T08:03:10.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36ca00b740 20 client.15414 trim_cache size 0 max 16384 2024-09-19T08:03:10.013 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:03:10.013 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:03:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "7089789c-af16-4525-b78a-9a98a14763c4", 2024-09-19T08:03:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:03:10.014 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:03:10.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36c7c32640 1 -- 172.21.15.98:0/1353722751 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f36ac0a34d0 msgr2=0x7f36ac0a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36c7c32640 1 --2- 172.21.15.98:0/1353722751 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f36ac0a34d0 0x7f36ac0a5990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f36b80cd700 tx=0x7f36b801eb40 comp rx=0 tx=0).stop 2024-09-19T08:03:10.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36c7c32640 1 -- 172.21.15.98:0/1353722751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36c0101470 msgr2=0x7f36c01a6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.008+0000 7f36c7c32640 1 --2- 172.21.15.98:0/1353722751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36c0101470 0x7f36c01a6960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f36b800eaa0 tx=0x7f36b800ef60 comp rx=0 tx=0).stop 2024-09-19T08:03:10.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.009+0000 7f36c7c32640 1 -- 172.21.15.98:0/1353722751 shutdown_connections 2024-09-19T08:03:10.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.009+0000 7f36c7c32640 1 --2- 172.21.15.98:0/1353722751 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f36ac0a34d0 0x7f36ac0a5990 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.009+0000 7f36c7c32640 1 --2- 172.21.15.98:0/1353722751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36c0101470 0x7f36c01a6960 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.009+0000 7f36c7c32640 1 --2- 172.21.15.98:0/1353722751 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36c0108290 0x7f36c0100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.009+0000 7f36c7c32640 1 --2- 172.21.15.98:0/1353722751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c0102d90 0x7f36c01009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.009+0000 7f36c7c32640 1 -- 172.21.15.98:0/1353722751 >> 172.21.15.98:0/1353722751 conn(0x7f36c00fc7d0 msgr2=0x7f36c00fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:10.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.009+0000 7f36c7c32640 1 -- 172.21.15.98:0/1353722751 shutdown_connections 2024-09-19T08:03:10.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.009+0000 7f36c7c32640 1 -- 172.21.15.98:0/1353722751 wait complete. 2024-09-19T08:03:10.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:09 smithi162 ceph-mon[35260]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.3 KiB/s wr, 26 op/s 2024-09-19T08:03:10.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:09 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2325854338' entity='client.admin' 2024-09-19T08:03:10.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:09 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:10.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:09 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:10.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:09 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:10.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:09 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:10.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.139+0000 7f09083b9640 1 -- 172.21.15.98:0/1973682586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f0900100e10 con 0x7f0900101350 2024-09-19T08:03:10.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.145+0000 7f08f77fe640 1 -- 172.21.15.98:0/1973682586 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v33)=0 v33) ==== 133+0+0 (secure 0 0 0) 0x7f0900100e10 con 0x7f0900101350 2024-09-19T08:03:10.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.148+0000 7f09083b9640 1 -- 172.21.15.98:0/1973682586 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08ec0a3580 msgr2=0x7f08ec0a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.148+0000 7f09083b9640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08ec0a3580 0x7f08ec0a5a40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009ba0 tx=0x7f08fc005f90 comp rx=0 tx=0).stop 2024-09-19T08:03:10.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.148+0000 7f09083b9640 1 -- 172.21.15.98:0/1973682586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0900101350 msgr2=0x7f0900198890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.148+0000 7f09083b9640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0900101350 0x7f0900198890 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f08f0009230 tx=0x7f08f0004270 comp rx=0 tx=0).stop 2024-09-19T08:03:10.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.148+0000 7f09083b9640 1 -- 172.21.15.98:0/1973682586 shutdown_connections 2024-09-19T08:03:10.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.148+0000 7f09083b9640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08ec0a3580 0x7f08ec0a5a40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.148+0000 7f09083b9640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09001069c0 0x7f090019cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.148+0000 7f09083b9640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0900101350 0x7f0900198890 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.148+0000 7f09083b9640 1 --2- 172.21.15.98:0/1973682586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09001009b0 0x7f0900198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.148+0000 7f09083b9640 1 -- 172.21.15.98:0/1973682586 >> 172.21.15.98:0/1973682586 conn(0x7f0900077990 msgr2=0x7f09000699e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:10.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.148+0000 7f09083b9640 1 -- 172.21.15.98:0/1973682586 shutdown_connections 2024-09-19T08:03:10.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.148+0000 7f09083b9640 1 -- 172.21.15.98:0/1973682586 wait complete. 2024-09-19T08:03:10.160 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:09 smithi098 ceph-mon[31333]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.3 KiB/s wr, 26 op/s 2024-09-19T08:03:10.160 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:09 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2325854338' entity='client.admin' 2024-09-19T08:03:10.160 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:09 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:10.160 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:09 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:10.160 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:09 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:10.160 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:09 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:10.161 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-09-19T08:03:10.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:09 smithi139 ceph-mon[34874]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.3 KiB/s wr, 26 op/s 2024-09-19T08:03:10.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:09 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2325854338' entity='client.admin' 2024-09-19T08:03:10.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:09 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:10.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:09 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:10.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:09 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:10.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:09 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.266+0000 7f860b077640 1 -- 172.21.15.98:0/1864123890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f860410c0d0 msgr2=0x7f860410e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.266+0000 7f860b077640 1 --2- 172.21.15.98:0/1864123890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f860410c0d0 0x7f860410e590 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009ca0 tx=0x7f85fc02a950 comp rx=0 tx=0).stop 2024-09-19T08:03:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.266+0000 7f860b077640 1 -- 172.21.15.98:0/1864123890 shutdown_connections 2024-09-19T08:03:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.266+0000 7f860b077640 1 --2- 172.21.15.98:0/1864123890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f860410c0d0 0x7f860410e590 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.266+0000 7f860b077640 1 --2- 172.21.15.98:0/1864123890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86041035d0 0x7f860410bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.266+0000 7f860b077640 1 --2- 172.21.15.98:0/1864123890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8604102cb0 0x7f8604103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.266+0000 7f860b077640 1 -- 172.21.15.98:0/1864123890 >> 172.21.15.98:0/1864123890 conn(0x7f86040fc6c0 msgr2=0x7f86040feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.266+0000 7f860b077640 1 -- 172.21.15.98:0/1864123890 shutdown_connections 2024-09-19T08:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.266+0000 7f860b077640 1 -- 172.21.15.98:0/1864123890 wait complete. 2024-09-19T08:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f860b077640 1 Processor -- start 2024-09-19T08:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f860b077640 1 -- start start 2024-09-19T08:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f860b077640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8604102cb0 0x7f86041982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f860b077640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86041035d0 0x7f8604198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f860b077640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f860410c0d0 0x7f860419cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f860b077640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8604111ef0 con 0x7f86041035d0 2024-09-19T08:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f860b077640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8604111d70 con 0x7f8604102cb0 2024-09-19T08:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f860b077640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8604112070 con 0x7f860410c0d0 2024-09-19T08:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f8603fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86041035d0 0x7f8604198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f8603fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86041035d0 0x7f8604198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47364/0 (socket says 172.21.15.98:47364) 2024-09-19T08:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f8603fff640 1 -- 172.21.15.98:0/3778075195 learned_addr learned my addr 172.21.15.98:0/3778075195 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f8608e13640 1 --2- 172.21.15.98:0/3778075195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8604102cb0 0x7f86041982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f8609614640 1 --2- 172.21.15.98:0/3778075195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f860410c0d0 0x7f860419cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f8603fff640 1 -- 172.21.15.98:0/3778075195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f860410c0d0 msgr2=0x7f860419cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.267+0000 7f8603fff640 1 --2- 172.21.15.98:0/3778075195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f860410c0d0 0x7f860419cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.268+0000 7f8603fff640 1 -- 172.21.15.98:0/3778075195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8604102cb0 msgr2=0x7f86041982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.268+0000 7f8603fff640 1 --2- 172.21.15.98:0/3778075195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8604102cb0 0x7f86041982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.268+0000 7f8603fff640 1 -- 172.21.15.98:0/3778075195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860419d2b0 con 0x7f86041035d0 2024-09-19T08:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.268+0000 7f8603fff640 1 --2- 172.21.15.98:0/3778075195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86041035d0 0x7f8604198800 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f85f000d950 tx=0x7f85f000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.268+0000 7f8601ffb640 1 -- 172.21.15.98:0/3778075195 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f0017030 con 0x7f86041035d0 2024-09-19T08:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.268+0000 7f8601ffb640 1 -- 172.21.15.98:0/3778075195 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85f0004020 con 0x7f86041035d0 2024-09-19T08:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.268+0000 7f860b077640 1 -- 172.21.15.98:0/3778075195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860419d5a0 con 0x7f86041035d0 2024-09-19T08:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.268+0000 7f8601ffb640 1 -- 172.21.15.98:0/3778075195 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f00047a0 con 0x7f86041035d0 2024-09-19T08:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.268+0000 7f860b077640 1 -- 172.21.15.98:0/3778075195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86041a4e30 con 0x7f86041035d0 2024-09-19T08:03:10.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.269+0000 7f860b077640 1 -- 172.21.15.98:0/3778075195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8604103090 con 0x7f86041035d0 2024-09-19T08:03:10.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.271+0000 7f8601ffb640 1 -- 172.21.15.98:0/3778075195 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f85f00041d0 con 0x7f86041035d0 2024-09-19T08:03:10.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.271+0000 7f8601ffb640 1 --2- 172.21.15.98:0/3778075195 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f85f40a3580 0x7f85f40a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.271+0000 7f8601ffb640 1 -- 172.21.15.98:0/3778075195 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f85f00cf840 con 0x7f86041035d0 2024-09-19T08:03:10.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.271+0000 7f8608e13640 1 --2- 172.21.15.98:0/3778075195 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f85f40a3580 0x7f85f40a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:10.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.272+0000 7f8608e13640 1 --2- 172.21.15.98:0/3778075195 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f85f40a3580 0x7f85f40a5a40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f85f80058c0 tx=0x7f85f800a2d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:10.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.276+0000 7f8601ffb640 1 -- 172.21.15.98:0/3778075195 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f85f009c000 con 0x7f86041035d0 2024-09-19T08:03:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.465+0000 7f860b077640 1 -- 172.21.15.98:0/3778075195 --> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f86040fe9e0 con 0x7f85f40a3580 2024-09-19T08:03:10.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.534+0000 7f8601ffb640 1 -- 172.21.15.98:0/3778075195 <== mgr.15252 v2:172.21.15.98:6800/1961446727 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f86040fe9e0 con 0x7f85f40a3580 2024-09-19T08:03:10.535 INFO:teuthology.orchestra.run.smithi098.stdout:Schedule set for path / 2024-09-19T08:03:10.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.536+0000 7f860b077640 1 -- 172.21.15.98:0/3778075195 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f85f40a3580 msgr2=0x7f85f40a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.536+0000 7f860b077640 1 --2- 172.21.15.98:0/3778075195 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f85f40a3580 0x7f85f40a5a40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f85f80058c0 tx=0x7f85f800a2d0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.536+0000 7f860b077640 1 -- 172.21.15.98:0/3778075195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86041035d0 msgr2=0x7f8604198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.536+0000 7f860b077640 1 --2- 172.21.15.98:0/3778075195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86041035d0 0x7f8604198800 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f85f000d950 tx=0x7f85f000de10 comp rx=0 tx=0).stop 2024-09-19T08:03:10.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.536+0000 7f860b077640 1 -- 172.21.15.98:0/3778075195 shutdown_connections 2024-09-19T08:03:10.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.536+0000 7f860b077640 1 --2- 172.21.15.98:0/3778075195 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f85f40a3580 0x7f85f40a5a40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.536+0000 7f860b077640 1 --2- 172.21.15.98:0/3778075195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f860410c0d0 0x7f860419cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.536+0000 7f860b077640 1 --2- 172.21.15.98:0/3778075195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86041035d0 0x7f8604198800 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.536+0000 7f860b077640 1 --2- 172.21.15.98:0/3778075195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8604102cb0 0x7f86041982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.536+0000 7f860b077640 1 -- 172.21.15.98:0/3778075195 >> 172.21.15.98:0/3778075195 conn(0x7f86040fc6c0 msgr2=0x7f86040fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:10.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.536+0000 7f860b077640 1 -- 172.21.15.98:0/3778075195 shutdown_connections 2024-09-19T08:03:10.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.536+0000 7f860b077640 1 -- 172.21.15.98:0/3778075195 wait complete. 2024-09-19T08:03:10.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-09-19T08:03:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.651+0000 7f310417f640 1 -- 172.21.15.98:0/3952698811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30fc10c110 msgr2=0x7f30fc10e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.651+0000 7f310417f640 1 --2- 172.21.15.98:0/3952698811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30fc10c110 0x7f30fc10e590 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009900 tx=0x7f30f402a480 comp rx=0 tx=0).stop 2024-09-19T08:03:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.651+0000 7f310417f640 1 -- 172.21.15.98:0/3952698811 shutdown_connections 2024-09-19T08:03:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.651+0000 7f310417f640 1 --2- 172.21.15.98:0/3952698811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30fc10c110 0x7f30fc10e590 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.651+0000 7f310417f640 1 --2- 172.21.15.98:0/3952698811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30fc1035e0 0x7f30fc10bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.651+0000 7f310417f640 1 --2- 172.21.15.98:0/3952698811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30fc102cc0 0x7f30fc1030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.651+0000 7f310417f640 1 -- 172.21.15.98:0/3952698811 >> 172.21.15.98:0/3952698811 conn(0x7f30fc0fc6b0 msgr2=0x7f30fc0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.651+0000 7f310417f640 1 -- 172.21.15.98:0/3952698811 shutdown_connections 2024-09-19T08:03:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.651+0000 7f310417f640 1 -- 172.21.15.98:0/3952698811 wait complete. 2024-09-19T08:03:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.651+0000 7f310417f640 1 Processor -- start 2024-09-19T08:03:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.651+0000 7f310417f640 1 -- start start 2024-09-19T08:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.651+0000 7f310417f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30fc102cc0 0x7f30fc198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.651+0000 7f310417f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30fc1035e0 0x7f30fc1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.651+0000 7f310417f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30fc10c110 0x7f30fc19cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f310417f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc111eb0 con 0x7f30fc102cc0 2024-09-19T08:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f310417f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc111d30 con 0x7f30fc10c110 2024-09-19T08:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f310417f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc112030 con 0x7f30fc1035e0 2024-09-19T08:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f3101f1b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30fc102cc0 0x7f30fc198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f3101f1b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30fc102cc0 0x7f30fc198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47384/0 (socket says 172.21.15.98:47384) 2024-09-19T08:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f3101f1b640 1 -- 172.21.15.98:0/2917923014 learned_addr learned my addr 172.21.15.98:0/2917923014 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f310171a640 1 --2- 172.21.15.98:0/2917923014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30fc1035e0 0x7f30fc1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f3101f1b640 1 -- 172.21.15.98:0/2917923014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30fc1035e0 msgr2=0x7f30fc1987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f3101f1b640 1 --2- 172.21.15.98:0/2917923014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30fc1035e0 0x7f30fc1987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f3101f1b640 1 -- 172.21.15.98:0/2917923014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30fc10c110 msgr2=0x7f30fc19cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f3101f1b640 1 --2- 172.21.15.98:0/2917923014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30fc10c110 0x7f30fc19cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f3101f1b640 1 -- 172.21.15.98:0/2917923014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30fc19d270 con 0x7f30fc102cc0 2024-09-19T08:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f3101f1b640 1 --2- 172.21.15.98:0/2917923014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30fc102cc0 0x7f30fc198280 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f30ec00c920 tx=0x7f30ec00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f30f2ffd640 1 -- 172.21.15.98:0/2917923014 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec007c70 con 0x7f30fc102cc0 2024-09-19T08:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f310417f640 1 -- 172.21.15.98:0/2917923014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30fc19d560 con 0x7f30fc102cc0 2024-09-19T08:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f30f2ffd640 1 -- 172.21.15.98:0/2917923014 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30ec007e10 con 0x7f30fc102cc0 2024-09-19T08:03:10.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f30f2ffd640 1 -- 172.21.15.98:0/2917923014 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec018890 con 0x7f30fc102cc0 2024-09-19T08:03:10.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.652+0000 7f310417f640 1 -- 172.21.15.98:0/2917923014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30fc1a4df0 con 0x7f30fc102cc0 2024-09-19T08:03:10.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.653+0000 7f310417f640 1 -- 172.21.15.98:0/2917923014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30fc1030a0 con 0x7f30fc102cc0 2024-09-19T08:03:10.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.654+0000 7f30f2ffd640 1 -- 172.21.15.98:0/2917923014 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f30ec018a70 con 0x7f30fc102cc0 2024-09-19T08:03:10.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.655+0000 7f30f2ffd640 1 --2- 172.21.15.98:0/2917923014 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f30e80a3580 0x7f30e80a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:10.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.655+0000 7f310171a640 1 --2- 172.21.15.98:0/2917923014 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f30e80a3580 0x7f30e80a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:10.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.655+0000 7f30f2ffd640 1 -- 172.21.15.98:0/2917923014 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f30ec0d01a0 con 0x7f30fc102cc0 2024-09-19T08:03:10.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.655+0000 7f310171a640 1 --2- 172.21.15.98:0/2917923014 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f30e80a3580 0x7f30e80a5a40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f30f8005c40 tx=0x7f30f8005bd0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:10.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.659+0000 7f30f2ffd640 1 -- 172.21.15.98:0/2917923014 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f30ec014030 con 0x7f30fc102cc0 2024-09-19T08:03:10.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:03:10.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.860+0000 7f310417f640 1 -- 172.21.15.98:0/2917923014 --> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f30fc0fe9d0 con 0x7f30e80a3580 2024-09-19T08:03:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.921+0000 7f30f2ffd640 1 -- 172.21.15.98:0/2917923014 <== mgr.15252 v2:172.21.15.98:6800/1961446727 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f30fc0fe9d0 con 0x7f30e80a3580 2024-09-19T08:03:10.922 INFO:teuthology.orchestra.run.smithi098.stdout:Retention added to path / 2024-09-19T08:03:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.923+0000 7f310417f640 1 -- 172.21.15.98:0/2917923014 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f30e80a3580 msgr2=0x7f30e80a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.923+0000 7f310417f640 1 --2- 172.21.15.98:0/2917923014 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f30e80a3580 0x7f30e80a5a40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f30f8005c40 tx=0x7f30f8005bd0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.923+0000 7f310417f640 1 -- 172.21.15.98:0/2917923014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30fc102cc0 msgr2=0x7f30fc198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.923+0000 7f310417f640 1 --2- 172.21.15.98:0/2917923014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30fc102cc0 0x7f30fc198280 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f30ec00c920 tx=0x7f30ec00cde0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.923+0000 7f310417f640 1 -- 172.21.15.98:0/2917923014 shutdown_connections 2024-09-19T08:03:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.923+0000 7f310417f640 1 --2- 172.21.15.98:0/2917923014 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f30e80a3580 0x7f30e80a5a40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.923+0000 7f310417f640 1 --2- 172.21.15.98:0/2917923014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30fc10c110 0x7f30fc19cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.923+0000 7f310417f640 1 --2- 172.21.15.98:0/2917923014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30fc1035e0 0x7f30fc1987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.923+0000 7f310417f640 1 --2- 172.21.15.98:0/2917923014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30fc102cc0 0x7f30fc198280 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.923+0000 7f310417f640 1 -- 172.21.15.98:0/2917923014 >> 172.21.15.98:0/2917923014 conn(0x7f30fc0fc6b0 msgr2=0x7f30fc0fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.923+0000 7f310417f640 1 -- 172.21.15.98:0/2917923014 shutdown_connections 2024-09-19T08:03:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:10.923+0000 7f310417f640 1 -- 172.21.15.98:0/2917923014 wait complete. 2024-09-19T08:03:10.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-09-19T08:03:11.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.081+0000 7fa96d986640 1 -- 172.21.15.98:0/996600711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9681009b0 msgr2=0x7fa968100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.081+0000 7fa96d986640 1 --2- 172.21.15.98:0/996600711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9681009b0 0x7fa968100e10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009930 tx=0x7fa95c02a420 comp rx=0 tx=0).stop 2024-09-19T08:03:11.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.081+0000 7fa96d986640 1 -- 172.21.15.98:0/996600711 shutdown_connections 2024-09-19T08:03:11.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.081+0000 7fa96d986640 1 --2- 172.21.15.98:0/996600711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa968101350 0x7fa96810d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.081+0000 7fa96d986640 1 --2- 172.21.15.98:0/996600711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9681009b0 0x7fa968100e10 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.081+0000 7fa96d986640 1 --2- 172.21.15.98:0/996600711 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9681069c0 0x7fa968106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.081+0000 7fa96d986640 1 -- 172.21.15.98:0/996600711 >> 172.21.15.98:0/996600711 conn(0x7fa968077990 msgr2=0x7fa9680fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:11.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.081+0000 7fa96d986640 1 -- 172.21.15.98:0/996600711 shutdown_connections 2024-09-19T08:03:11.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.081+0000 7fa96d986640 1 -- 172.21.15.98:0/996600711 wait complete. 2024-09-19T08:03:11.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.082+0000 7fa96d986640 1 Processor -- start 2024-09-19T08:03:11.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.082+0000 7fa96d986640 1 -- start start 2024-09-19T08:03:11.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.082+0000 7fa96d986640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9681009b0 0x7fa9680fcde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.082+0000 7fa96d986640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa968101350 0x7fa9680fd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.082+0000 7fa96d986640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9681069c0 0x7fa9680fd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.082+0000 7fa96d986640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa968110dc0 con 0x7fa968101350 2024-09-19T08:03:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.082+0000 7fa96d986640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa968110c40 con 0x7fa9681069c0 2024-09-19T08:03:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.082+0000 7fa96d986640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa968110f40 con 0x7fa9681009b0 2024-09-19T08:03:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.082+0000 7fa9667fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa968101350 0x7fa9680fd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa9667fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa968101350 0x7fa9680fd320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47412/0 (socket says 172.21.15.98:47412) 2024-09-19T08:03:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa9667fc640 1 -- 172.21.15.98:0/1672677113 learned_addr learned my addr 172.21.15.98:0/1672677113 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa966ffd640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9681009b0 0x7fa9680fcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa9677fe640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9681069c0 0x7fa9680fd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa9667fc640 1 -- 172.21.15.98:0/1672677113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9681009b0 msgr2=0x7fa9680fcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa9667fc640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9681009b0 0x7fa9680fcde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa9667fc640 1 -- 172.21.15.98:0/1672677113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9681069c0 msgr2=0x7fa9680fd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa9667fc640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9681069c0 0x7fa9680fd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa9667fc640 1 -- 172.21.15.98:0/1672677113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9681a6870 con 0x7fa968101350 2024-09-19T08:03:11.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa966ffd640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9681009b0 0x7fa9680fcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:11.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa9677fe640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9681069c0 0x7fa9680fd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:11.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa9667fc640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa968101350 0x7fa9680fd320 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009230 tx=0x7fa95c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:11.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa96c984640 1 -- 172.21.15.98:0/1672677113 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c03f030 con 0x7fa968101350 2024-09-19T08:03:11.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa96c984640 1 -- 172.21.15.98:0/1672677113 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa95c003ad0 con 0x7fa968101350 2024-09-19T08:03:11.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa96c984640 1 -- 172.21.15.98:0/1672677113 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c035070 con 0x7fa968101350 2024-09-19T08:03:11.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa96d986640 1 -- 172.21.15.98:0/1672677113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9681a6b00 con 0x7fa968101350 2024-09-19T08:03:11.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.083+0000 7fa96d986640 1 -- 172.21.15.98:0/1672677113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9681a6fa0 con 0x7fa968101350 2024-09-19T08:03:11.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.084+0000 7fa96d986640 1 -- 172.21.15.98:0/1672677113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa968100e10 con 0x7fa968101350 2024-09-19T08:03:11.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.086+0000 7fa96c984640 1 -- 172.21.15.98:0/1672677113 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7fa95c042030 con 0x7fa968101350 2024-09-19T08:03:11.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.087+0000 7fa96c984640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa9500a3580 0x7fa9500a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.087+0000 7fa966ffd640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa9500a3580 0x7fa9500a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.087+0000 7fa96c984640 1 -- 172.21.15.98:0/1672677113 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fa95c0f0600 con 0x7fa968101350 2024-09-19T08:03:11.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.087+0000 7fa966ffd640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa9500a3580 0x7fa9500a5a40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa958005fd0 tx=0x7fa958005850 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:11.093 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:11.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.093+0000 7fa96c984640 1 -- 172.21.15.98:0/1672677113 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fa95c0bce90 con 0x7fa968101350 2024-09-19T08:03:11.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2892987531' entity='client.admin' 2024-09-19T08:03:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:03:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:03:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:03:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:03:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1973682586' entity='client.admin' 2024-09-19T08:03:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:11.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:11.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:10 smithi162 ceph-mon[35260]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.0 KiB/s wr, 20 op/s 2024-09-19T08:03:11.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2892987531' entity='client.admin' 2024-09-19T08:03:11.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:11.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:11.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:11.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:11.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:03:11.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:03:11.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:03:11.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:03:11.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1973682586' entity='client.admin' 2024-09-19T08:03:11.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:11.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:11.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:11.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:11.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:11.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:10 smithi139 ceph-mon[34874]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.0 KiB/s wr, 20 op/s 2024-09-19T08:03:11.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2892987531' entity='client.admin' 2024-09-19T08:03:11.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:11.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:11.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:11.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:11.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:03:11.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:03:11.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:03:11.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:03:11.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1973682586' entity='client.admin' 2024-09-19T08:03:11.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:11.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:11.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:11.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:03:11.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:11.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:10 smithi098 ceph-mon[31333]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.0 KiB/s wr, 20 op/s 2024-09-19T08:03:11.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.294+0000 7fa96d986640 1 -- 172.21.15.98:0/1672677113 --> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fa96806a0b0 con 0x7fa9500a3580 2024-09-19T08:03:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.301+0000 7fa96c984640 1 -- 172.21.15.98:0/1672677113 <== mgr.15252 v2:172.21.15.98:6800/1961446727 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fa96806a0b0 con 0x7fa9500a3580 2024-09-19T08:03:11.302 INFO:teuthology.orchestra.run.smithi098.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-09-19T00:00:00", "created": "2024-09-19T08:03:10", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-09-19T08:03:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.303+0000 7fa96d986640 1 -- 172.21.15.98:0/1672677113 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa9500a3580 msgr2=0x7fa9500a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.303+0000 7fa96d986640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa9500a3580 0x7fa9500a5a40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa958005fd0 tx=0x7fa958005850 comp rx=0 tx=0).stop 2024-09-19T08:03:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.303+0000 7fa96d986640 1 -- 172.21.15.98:0/1672677113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa968101350 msgr2=0x7fa9680fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.303+0000 7fa96d986640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa968101350 0x7fa9680fd320 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009230 tx=0x7fa95c004270 comp rx=0 tx=0).stop 2024-09-19T08:03:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.303+0000 7fa96d986640 1 -- 172.21.15.98:0/1672677113 shutdown_connections 2024-09-19T08:03:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.303+0000 7fa96d986640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa9500a3580 0x7fa9500a5a40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.303+0000 7fa96d986640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9681069c0 0x7fa9680fd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.303+0000 7fa96d986640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa968101350 0x7fa9680fd320 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.303+0000 7fa96d986640 1 --2- 172.21.15.98:0/1672677113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9681009b0 0x7fa9680fcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.303+0000 7fa96d986640 1 -- 172.21.15.98:0/1672677113 >> 172.21.15.98:0/1672677113 conn(0x7fa968077990 msgr2=0x7fa9680699a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.303+0000 7fa96d986640 1 -- 172.21.15.98:0/1672677113 shutdown_connections 2024-09-19T08:03:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.303+0000 7fa96d986640 1 -- 172.21.15.98:0/1672677113 wait complete. 2024-09-19T08:03:11.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-09-19T08:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.419+0000 7f95016f1640 1 -- 172.21.15.98:0/3328902685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94fc107b70 msgr2=0x7f94fc107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.419+0000 7f95016f1640 1 --2- 172.21.15.98:0/3328902685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94fc107b70 0x7f94fc107f50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009930 tx=0x7f94ec02a440 comp rx=0 tx=0).stop 2024-09-19T08:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.419+0000 7f95016f1640 1 -- 172.21.15.98:0/3328902685 shutdown_connections 2024-09-19T08:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.419+0000 7f95016f1640 1 --2- 172.21.15.98:0/3328902685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94fc102470 0x7f94fc10e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.419+0000 7f95016f1640 1 --2- 172.21.15.98:0/3328902685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94fc101ad0 0x7f94fc101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.419+0000 7f95016f1640 1 --2- 172.21.15.98:0/3328902685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94fc107b70 0x7f94fc107f50 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.419+0000 7f95016f1640 1 -- 172.21.15.98:0/3328902685 >> 172.21.15.98:0/3328902685 conn(0x7f94fc0fd7d0 msgr2=0x7f94fc0ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:11.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.419+0000 7f95016f1640 1 -- 172.21.15.98:0/3328902685 shutdown_connections 2024-09-19T08:03:11.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.419+0000 7f95016f1640 1 -- 172.21.15.98:0/3328902685 wait complete. 2024-09-19T08:03:11.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f95016f1640 1 Processor -- start 2024-09-19T08:03:11.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f95016f1640 1 -- start start 2024-09-19T08:03:11.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f95016f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94fc101ad0 0x7f94fc19a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f95016f1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94fc102470 0x7f94fc19aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f95016f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94fc107b70 0x7f94fc19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f95016f1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94fc111f40 con 0x7f94fc101ad0 2024-09-19T08:03:11.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f95016f1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f94fc111dc0 con 0x7f94fc102470 2024-09-19T08:03:11.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f95016f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94fc1120c0 con 0x7f94fc107b70 2024-09-19T08:03:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f94faffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94fc101ad0 0x7f94fc19a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f94faffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94fc101ad0 0x7f94fc19a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47418/0 (socket says 172.21.15.98:47418) 2024-09-19T08:03:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f94faffd640 1 -- 172.21.15.98:0/517464028 learned_addr learned my addr 172.21.15.98:0/517464028 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f94fb7fe640 1 --2- 172.21.15.98:0/517464028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94fc107b70 0x7f94fc19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f94faffd640 1 -- 172.21.15.98:0/517464028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94fc107b70 msgr2=0x7f94fc19ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f94faffd640 1 --2- 172.21.15.98:0/517464028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94fc107b70 0x7f94fc19ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f94faffd640 1 -- 172.21.15.98:0/517464028 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94fc102470 msgr2=0x7f94fc19aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f94faffd640 1 --2- 172.21.15.98:0/517464028 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94fc102470 0x7f94fc19aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f94faffd640 1 -- 172.21.15.98:0/517464028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94fc19f5a0 con 0x7f94fc101ad0 2024-09-19T08:03:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.420+0000 7f94fb7fe640 1 --2- 172.21.15.98:0/517464028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94fc107b70 0x7f94fc19ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.421+0000 7f94faffd640 1 --2- 172.21.15.98:0/517464028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94fc101ad0 0x7f94fc19a5b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009230 tx=0x7f94ec004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.421+0000 7f94f8ff9640 1 -- 172.21.15.98:0/517464028 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec03f030 con 0x7f94fc101ad0 2024-09-19T08:03:11.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.421+0000 7f94f8ff9640 1 -- 172.21.15.98:0/517464028 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94ec003ad0 con 0x7f94fc101ad0 2024-09-19T08:03:11.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.421+0000 7f94f8ff9640 1 -- 172.21.15.98:0/517464028 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec035070 con 0x7f94fc101ad0 2024-09-19T08:03:11.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.421+0000 7f95016f1640 1 -- 172.21.15.98:0/517464028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94fc19f830 con 0x7f94fc101ad0 2024-09-19T08:03:11.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.421+0000 7f95016f1640 1 -- 172.21.15.98:0/517464028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94fc1a7120 con 0x7f94fc101ad0 2024-09-19T08:03:11.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.422+0000 7f95016f1640 1 -- 172.21.15.98:0/517464028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94fc101f30 con 0x7f94fc101ad0 2024-09-19T08:03:11.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.423+0000 7f94f8ff9640 1 -- 172.21.15.98:0/517464028 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f94ec042030 con 0x7f94fc101ad0 2024-09-19T08:03:11.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.424+0000 7f94f8ff9640 1 --2- 172.21.15.98:0/517464028 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94e80a3580 0x7f94e80a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.424+0000 7f94f3fff640 1 --2- 172.21.15.98:0/517464028 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94e80a3580 0x7f94e80a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.424+0000 7f94f8ff9640 1 -- 172.21.15.98:0/517464028 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f94ec0f09c0 con 0x7f94fc101ad0 2024-09-19T08:03:11.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.424+0000 7f94f3fff640 1 --2- 172.21.15.98:0/517464028 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94e80a3580 0x7f94e80a5a40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f94fc10ec10 tx=0x7f94e4006c40 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.429+0000 7f94f8ff9640 1 -- 172.21.15.98:0/517464028 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f94ec0bd250 con 0x7f94fc101ad0 2024-09-19T08:03:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.597+0000 7f66aa3fb640 1 -- 172.21.15.98:0/416336194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a4102570 msgr2=0x7f66a41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.597+0000 7f66aa3fb640 1 --2- 172.21.15.98:0/416336194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a4102570 0x7f66a41029b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f668c009900 tx=0x7f668c02a480 comp rx=0 tx=0).stop 2024-09-19T08:03:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.597+0000 7f66aa3fb640 1 -- 172.21.15.98:0/416336194 shutdown_connections 2024-09-19T08:03:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.597+0000 7f66aa3fb640 1 --2- 172.21.15.98:0/416336194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66a4102ef0 0x7f66a410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.597+0000 7f66aa3fb640 1 --2- 172.21.15.98:0/416336194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a4102570 0x7f66a41029b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.597+0000 7f66aa3fb640 1 --2- 172.21.15.98:0/416336194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a4100e20 0x7f66a4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.597+0000 7f66aa3fb640 1 -- 172.21.15.98:0/416336194 >> 172.21.15.98:0/416336194 conn(0x7f66a40fc7d0 msgr2=0x7f66a40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.597+0000 7f66aa3fb640 1 -- 172.21.15.98:0/416336194 shutdown_connections 2024-09-19T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.597+0000 7f66aa3fb640 1 -- 172.21.15.98:0/416336194 wait complete. 2024-09-19T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.598+0000 7f66aa3fb640 1 Processor -- start 2024-09-19T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.598+0000 7f66aa3fb640 1 -- start start 2024-09-19T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.598+0000 7f66aa3fb640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a4100e20 0x7f66a40fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.598+0000 7f66aa3fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a4102570 0x7f66a40fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.598+0000 7f66aa3fb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66a4102ef0 0x7f66a40fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.598+0000 7f66aa3fb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66a4110f80 con 0x7f66a4102570 2024-09-19T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.598+0000 7f66aa3fb640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f66a4110e00 con 0x7f66a4100e20 2024-09-19T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.598+0000 7f66aa3fb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f66a4111100 con 0x7f66a4102ef0 2024-09-19T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a4102570 0x7f66a40fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a4102570 0x7f66a40fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47428/0 (socket says 172.21.15.98:47428) 2024-09-19T08:03:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a37fe640 1 -- 172.21.15.98:0/2594967625 learned_addr learned my addr 172.21.15.98:0/2594967625 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a3fff640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a4100e20 0x7f66a40fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a8998640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66a4102ef0 0x7f66a40fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a37fe640 1 -- 172.21.15.98:0/2594967625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66a4102ef0 msgr2=0x7f66a40fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a37fe640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66a4102ef0 0x7f66a40fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a37fe640 1 -- 172.21.15.98:0/2594967625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a4100e20 msgr2=0x7f66a40fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a37fe640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a4100e20 0x7f66a40fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a37fe640 1 -- 172.21.15.98:0/2594967625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a41a6970 con 0x7f66a4102570 2024-09-19T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a8998640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66a4102ef0 0x7f66a40fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a37fe640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a4102570 0x7f66a40fd1c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f668c006010 tx=0x7f668c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a17fa640 1 -- 172.21.15.98:0/2594967625 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c03f070 con 0x7f66a4102570 2024-09-19T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66aa3fb640 1 -- 172.21.15.98:0/2594967625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a41a6c00 con 0x7f66a4102570 2024-09-19T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a17fa640 1 -- 172.21.15.98:0/2594967625 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f668c031c30 con 0x7f66a4102570 2024-09-19T08:03:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66a17fa640 1 -- 172.21.15.98:0/2594967625 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c035030 con 0x7f66a4102570 2024-09-19T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.599+0000 7f66aa3fb640 1 -- 172.21.15.98:0/2594967625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a41a7130 con 0x7f66a4102570 2024-09-19T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.602+0000 7f66a17fa640 1 -- 172.21.15.98:0/2594967625 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f668c042070 con 0x7f66a4102570 2024-09-19T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.602+0000 7f66ac7d4740 1 Processor -- start 2024-09-19T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.602+0000 7f66ac7d4740 1 -- start start 2024-09-19T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.602+0000 7f66ac7d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 0x561e1695b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.602+0000 7f66ac7d4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e1695bdc0 0x561e16966140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.602+0000 7f66ac7d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e16966680 0x561e16968a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.602+0000 7f66ac7d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561e168c12e0 con 0x561e16966680 2024-09-19T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.602+0000 7f66ac7d4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561e16868ce0 con 0x561e1695bdc0 2024-09-19T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.602+0000 7f66ac7d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561e168bd2a0 con 0x561e1695b4a0 2024-09-19T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.602+0000 7f66a17fa640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f66900a34d0 0x7f66900a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.602+0000 7f66a37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e16966680 0x561e16968a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.602+0000 7f66a37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e16966680 0x561e16968a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47436/0 (socket says 172.21.15.98:47436) 2024-09-19T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.602+0000 7f66a37fe640 1 -- 172.21.15.98:0/278723566 learned_addr learned my addr 172.21.15.98:0/278723566 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.602+0000 7f66a8998640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f66900a34d0 0x7f66900a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.603+0000 7f66a8998640 1 --2- 172.21.15.98:0/278723566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 0x561e1695b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.603+0000 7f66a3fff640 1 --2- 172.21.15.98:0/278723566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e1695bdc0 0x561e16966140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.603+0000 7f66a17fa640 1 -- 172.21.15.98:0/2594967625 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f668c0f05b0 con 0x7f66a4102570 2024-09-19T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.603+0000 7f66a37fe640 1 -- 172.21.15.98:0/278723566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 msgr2=0x561e1695b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.603+0000 7f66a37fe640 1 --2- 172.21.15.98:0/278723566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 0x561e1695b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.603+0000 7f66a37fe640 1 -- 172.21.15.98:0/278723566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e1695bdc0 msgr2=0x561e16966140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.603+0000 7f66a37fe640 1 --2- 172.21.15.98:0/278723566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e1695bdc0 0x561e16966140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.603+0000 7f66a37fe640 1 -- 172.21.15.98:0/278723566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e1684d440 con 0x561e16966680 2024-09-19T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.603+0000 7f66a3fff640 1 --2- 172.21.15.98:0/278723566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e1695bdc0 0x561e16966140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.603+0000 7f66a8998640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f66900a34d0 0x7f66900a5990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f669c009fd0 tx=0x7f669c0096f0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.603+0000 7f66a37fe640 1 --2- 172.21.15.98:0/278723566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e16966680 0x561e16968a70 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f668c0bc440 tx=0x7f668c03b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351cf7ad2d1e1a6f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.603+0000 7f669a7fc640 1 -- 172.21.15.98:0/278723566 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c049070 con 0x561e16966680 2024-09-19T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.603+0000 7f669a7fc640 1 -- 172.21.15.98:0/278723566 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f668c0bce40 con 0x561e16966680 2024-09-19T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.603+0000 7f669a7fc640 1 -- 172.21.15.98:0/278723566 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c0b8db0 con 0x561e16966680 2024-09-19T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.604+0000 7f66ac7d4740 1 -- 172.21.15.98:0/278723566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e16966680 msgr2=0x561e16968a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.604+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/278723566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e16966680 0x561e16968a70 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f668c0bc440 tx=0x7f668c03b790 comp rx=0 tx=0).stop 2024-09-19T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.604+0000 7f66ac7d4740 1 -- 172.21.15.98:0/278723566 shutdown_connections 2024-09-19T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.604+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/278723566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e16966680 0x561e16968a70 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.604+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/278723566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e1695bdc0 0x561e16966140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.604+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/278723566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 0x561e1695b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.604+0000 7f66ac7d4740 1 -- 172.21.15.98:0/278723566 >> 172.21.15.98:0/278723566 conn(0x561e16957a30 msgr2=0x561e16957e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.604+0000 7f66ac7d4740 1 -- 172.21.15.98:0/278723566 shutdown_connections 2024-09-19T08:03:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.604+0000 7f66ac7d4740 1 -- 172.21.15.98:0/278723566 wait complete. 2024-09-19T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.604+0000 7f66ac7d4740 1 -- start start 2024-09-19T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66ac7d4740 1 Processor -- start 2024-09-19T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66ac7d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e1695b4a0 0x561e16966030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66ac7d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695bdc0 0x561e169632e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66ac7d4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e16966680 0x561e16963880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66ac7d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561e16969010 con 0x561e1695b4a0 2024-09-19T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66ac7d4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561e1687c050 con 0x561e16966680 2024-09-19T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66ac7d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561e16969190 con 0x561e1695bdc0 2024-09-19T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66ac7d4740 10 client.0 _ll_register_callbacks cb 0x561e168bc160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66a3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e1695b4a0 0x561e16966030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66a3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e1695b4a0 0x561e16966030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47444/0 (socket says 172.21.15.98:47444) 2024-09-19T08:03:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66a3fff640 1 -- 172.21.15.98:0/2866300956 learned_addr learned my addr 172.21.15.98:0/2866300956 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66a37fe640 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695bdc0 0x561e169632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66a8998640 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e16966680 0x561e16963880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66a3fff640 1 -- 172.21.15.98:0/2866300956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695bdc0 msgr2=0x561e169632e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66a3fff640 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695bdc0 0x561e169632e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66a3fff640 1 -- 172.21.15.98:0/2866300956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e16966680 msgr2=0x561e16963880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66a3fff640 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e16966680 0x561e16963880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66a3fff640 1 -- 172.21.15.98:0/2866300956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e166323c0 con 0x561e1695b4a0 2024-09-19T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66a37fe640 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695bdc0 0x561e169632e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66a8998640 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e16966680 0x561e16963880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66a3fff640 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e1695b4a0 0x561e16966030 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6694012700 tx=0x7f6694012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bbb9207bff4157a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.605+0000 7f66837fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.606+0000 7f66837fe640 1 -- 172.21.15.98:0/2866300956 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6694004030 con 0x561e1695b4a0 2024-09-19T08:03:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.606+0000 7f66837fe640 1 -- 172.21.15.98:0/2866300956 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f66940041d0 con 0x561e1695b4a0 2024-09-19T08:03:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.606+0000 7f66ac7d4740 1 -- 172.21.15.98:0/2866300956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f668c0bce40 con 0x561e1695b4a0 2024-09-19T08:03:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.606+0000 7f66837fe640 1 -- 172.21.15.98:0/2866300956 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f66940043d0 con 0x561e1695b4a0 2024-09-19T08:03:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.606+0000 7f66ac7d4740 10 client.15450 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.606+0000 7f66ac7d4740 1 -- 172.21.15.98:0/2866300956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561e16974970 con 0x561e1695b4a0 2024-09-19T08:03:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.606+0000 7f66837fe640 1 -- 172.21.15.98:0/2866300956 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f668c0bce40 con 0x561e1695b4a0 2024-09-19T08:03:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.606+0000 7f66837fe640 1 -- 172.21.15.98:0/2866300956 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6694003ce0 con 0x561e1695b4a0 2024-09-19T08:03:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.606+0000 7f66ac7d4740 10 client.15450 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.606+0000 7f66ac7d4740 10 client.15450 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:03:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.606+0000 7f66ac7d4740 20 client.15450 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.606+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e16975d80 0x561e16978170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.606+0000 7f66ac7d4740 4 client.15450 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.606+0000 7f66ac7d4740 1 -- 172.21.15.98:0/2866300956 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e168ce850 con 0x561e16975d80 2024-09-19T08:03:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.607+0000 7f66a37fe640 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e16975d80 0x561e16978170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.607+0000 7f66a37fe640 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e16975d80 0x561e16978170 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21d6a1cf9cd89a7d server_cookie=376929005461d7c8 in_seq=0 out_seq=0 2024-09-19T08:03:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.607+0000 7f66837fe640 10 client.15450 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66837fe640 1 -- 172.21.15.98:0/2866300956 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561e168ce850 con 0x561e16975d80 2024-09-19T08:03:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66837fe640 10 client.15450 handle_command_reply: tid=0 2024-09-19T08:03:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66ac7d4740 1 client.15450 shutdown 2024-09-19T08:03:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66ac7d4740 1 -- 172.21.15.98:0/2866300956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e1695b4a0 msgr2=0x561e16966030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e1695b4a0 0x561e16966030 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6694012700 tx=0x7f6694012bc0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66ac7d4740 1 -- 172.21.15.98:0/2866300956 shutdown_connections 2024-09-19T08:03:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e16975d80 0x561e16978170 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e16966680 0x561e16963880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695bdc0 0x561e169632e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/2866300956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e1695b4a0 0x561e16966030 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66ac7d4740 1 -- 172.21.15.98:0/2866300956 >> 172.21.15.98:0/2866300956 conn(0x561e16957a30 msgr2=0x561e1695d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66837fe640 0 client.15450 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66ac7d4740 1 -- 172.21.15.98:0/2866300956 shutdown_connections 2024-09-19T08:03:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66ac7d4740 1 -- 172.21.15.98:0/2866300956 wait complete. 2024-09-19T08:03:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.608+0000 7f66ac7d4740 20 client.15450 trim_cache size 0 max 16384 2024-09-19T08:03:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66ac7d4740 1 Processor -- start 2024-09-19T08:03:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66ac7d4740 1 -- start start 2024-09-19T08:03:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f95016f1640 1 -- 172.21.15.98:0/517464028 --> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f94fc0fecb0 con 0x7f94e80a3580 2024-09-19T08:03:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66ac7d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 0x561e16972130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66ac7d4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e1695bdc0 0x561e16957e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66ac7d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e16966680 0x561e1697da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66ac7d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66a41afa10 con 0x561e16966680 2024-09-19T08:03:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66ac7d4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f66a41b1550 con 0x561e1695bdc0 2024-09-19T08:03:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66ac7d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f66a41b00e0 con 0x561e1695b4a0 2024-09-19T08:03:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66a8998640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e16966680 0x561e1697da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66a37fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e1695bdc0 0x561e16957e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66a37fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e1695bdc0 0x561e16957e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:35210/0 (socket says 172.21.15.98:35210) 2024-09-19T08:03:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66a37fe640 1 -- 172.21.15.98:0/69217533 learned_addr learned my addr 172.21.15.98:0/69217533 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66a3fff640 1 --2- 172.21.15.98:0/69217533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 0x561e16972130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66a37fe640 1 -- 172.21.15.98:0/69217533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 msgr2=0x561e16972130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66a37fe640 1 --2- 172.21.15.98:0/69217533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 0x561e16972130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66a37fe640 1 -- 172.21.15.98:0/69217533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e16966680 msgr2=0x561e1697da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66a37fe640 1 --2- 172.21.15.98:0/69217533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e16966680 0x561e1697da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.626+0000 7f66a37fe640 1 -- 172.21.15.98:0/69217533 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e16962290 con 0x561e1695bdc0 2024-09-19T08:03:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.627+0000 7f66a37fe640 1 --2- 172.21.15.98:0/69217533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e1695bdc0 0x561e16957e10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f668c0b9110 tx=0x7f668c0c7040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15c5bb2ef9f18092 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.627+0000 7f6699ffb640 1 -- 172.21.15.98:0/69217533 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c0d5070 con 0x561e1695bdc0 2024-09-19T08:03:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.627+0000 7f6699ffb640 1 -- 172.21.15.98:0/69217533 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f668c0c0af0 con 0x561e1695bdc0 2024-09-19T08:03:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.627+0000 7f6699ffb640 1 -- 172.21.15.98:0/69217533 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c049040 con 0x561e1695bdc0 2024-09-19T08:03:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.628+0000 7f66ac7d4740 1 -- 172.21.15.98:0/69217533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e1695bdc0 msgr2=0x561e16957e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.628+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/69217533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e1695bdc0 0x561e16957e10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f668c0b9110 tx=0x7f668c0c7040 comp rx=0 tx=0).stop 2024-09-19T08:03:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.628+0000 7f66ac7d4740 1 -- 172.21.15.98:0/69217533 shutdown_connections 2024-09-19T08:03:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.628+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/69217533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e16966680 0x561e1697da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.628+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/69217533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e1695bdc0 0x561e16957e10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.628+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/69217533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 0x561e16972130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.628+0000 7f66ac7d4740 1 -- 172.21.15.98:0/69217533 >> 172.21.15.98:0/69217533 conn(0x561e16957a30 msgr2=0x561e1695e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.628+0000 7f66ac7d4740 1 -- 172.21.15.98:0/69217533 shutdown_connections 2024-09-19T08:03:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.628+0000 7f66ac7d4740 1 -- 172.21.15.98:0/69217533 wait complete. 2024-09-19T08:03:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.628+0000 7f66ac7d4740 1 -- start start 2024-09-19T08:03:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.628+0000 7f66ac7d4740 1 Processor -- start 2024-09-19T08:03:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66ac7d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 0x561e1696eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66ac7d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e1695bdc0 0x561e16969e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66ac7d4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e16966680 0x561e1696a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66ac7d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66a41b0e80 con 0x561e1695bdc0 2024-09-19T08:03:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66ac7d4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f66a41b07b0 con 0x561e16966680 2024-09-19T08:03:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66ac7d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f66a41ad9f0 con 0x561e1695b4a0 2024-09-19T08:03:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66ac7d4740 10 client.0 _ll_register_callbacks cb 0x7f6694009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66a37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e1695bdc0 0x561e16969e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66a3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 0x561e1696eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66a8998640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e16966680 0x561e1696a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66a3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 0x561e1696eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:55608/0 (socket says 172.21.15.98:55608) 2024-09-19T08:03:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66a3fff640 1 -- 172.21.15.98:0/3892957727 learned_addr learned my addr 172.21.15.98:0/3892957727 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66a3fff640 1 -- 172.21.15.98:0/3892957727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e16966680 msgr2=0x561e1696a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66a3fff640 1 --2- 172.21.15.98:0/3892957727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e16966680 0x561e1696a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66a3fff640 1 -- 172.21.15.98:0/3892957727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e1695bdc0 msgr2=0x561e16969e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66a3fff640 1 --2- 172.21.15.98:0/3892957727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e1695bdc0 0x561e16969e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66a3fff640 1 -- 172.21.15.98:0/3892957727 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e169df9f0 con 0x561e1695b4a0 2024-09-19T08:03:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66a8998640 1 --2- 172.21.15.98:0/3892957727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e16966680 0x561e1696a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.629+0000 7f66a37fe640 1 --2- 172.21.15.98:0/3892957727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e1695bdc0 0x561e16969e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.630+0000 7f66a3fff640 1 --2- 172.21.15.98:0/3892957727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 0x561e1696eb40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6694000c00 tx=0x7f6694009dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cebc0215f790a47 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.630+0000 7f66837fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-09-19T08:03:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.630+0000 7f66837fe640 1 -- 172.21.15.98:0/3892957727 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669402b070 con 0x561e1695b4a0 2024-09-19T08:03:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.630+0000 7f66837fe640 1 -- 172.21.15.98:0/3892957727 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f669400fbb0 con 0x561e1695b4a0 2024-09-19T08:03:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.630+0000 7f66ac7d4740 1 -- 172.21.15.98:0/3892957727 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f668c0c0af0 con 0x561e1695b4a0 2024-09-19T08:03:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.630+0000 7f66837fe640 1 -- 172.21.15.98:0/3892957727 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f669400fdb0 con 0x561e1695b4a0 2024-09-19T08:03:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.630+0000 7f66ac7d4740 10 client.24934 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.630+0000 7f66ac7d4740 1 -- 172.21.15.98:0/3892957727 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561e169e0b50 con 0x561e1695b4a0 2024-09-19T08:03:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.630+0000 7f66837fe640 1 -- 172.21.15.98:0/3892957727 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f668c0c0af0 con 0x561e1695b4a0 2024-09-19T08:03:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.631+0000 7f66837fe640 1 -- 172.21.15.98:0/3892957727 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6694025c10 con 0x561e1695b4a0 2024-09-19T08:03:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.631+0000 7f66ac7d4740 10 client.24934 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.631+0000 7f66ac7d4740 10 client.24934 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:03:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.631+0000 7f66ac7d4740 20 client.24934 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.631+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/3892957727 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e169e1250 0x561e169e3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.631+0000 7f66ac7d4740 4 client.24934 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:03:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.631+0000 7f66ac7d4740 1 -- 172.21.15.98:0/3892957727 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561e168c8d60 con 0x561e169e1250 2024-09-19T08:03:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.631+0000 7f66a37fe640 1 --2- 172.21.15.98:0/3892957727 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e169e1250 0x561e169e3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.631+0000 7f66a37fe640 1 --2- 172.21.15.98:0/3892957727 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e169e1250 0x561e169e3670 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdc21b842e3391a3 server_cookie=a9a1a62dc0c60cd7 in_seq=0 out_seq=0 2024-09-19T08:03:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.631+0000 7f66837fe640 10 client.24934 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.631+0000 7f66837fe640 1 -- 172.21.15.98:0/3892957727 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561e168c8d60 con 0x561e169e1250 2024-09-19T08:03:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.631+0000 7f66837fe640 10 client.24934 handle_command_reply: tid=0 2024-09-19T08:03:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.631+0000 7f66ac7d4740 1 client.24934 shutdown 2024-09-19T08:03:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.632+0000 7f66ac7d4740 1 -- 172.21.15.98:0/3892957727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 msgr2=0x561e1696eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.632+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/3892957727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 0x561e1696eb40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6694000c00 tx=0x7f6694009dc0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.632+0000 7f66ac7d4740 1 -- 172.21.15.98:0/3892957727 shutdown_connections 2024-09-19T08:03:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.632+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/3892957727 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e169e1250 0x561e169e3670 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.632+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/3892957727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e16966680 0x561e1696a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.632+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/3892957727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e1695bdc0 0x561e16969e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.632+0000 7f66ac7d4740 1 --2- 172.21.15.98:0/3892957727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e1695b4a0 0x561e1696eb40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.632+0000 7f66ac7d4740 1 -- 172.21.15.98:0/3892957727 >> 172.21.15.98:0/3892957727 conn(0x561e16957a30 msgr2=0x561e16956270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.632+0000 7f66837fe640 0 client.24934 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.632+0000 7f66ac7d4740 1 -- 172.21.15.98:0/3892957727 shutdown_connections 2024-09-19T08:03:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.632+0000 7f66ac7d4740 1 -- 172.21.15.98:0/3892957727 wait complete. 2024-09-19T08:03:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.632+0000 7f66ac7d4740 20 client.24934 trim_cache size 0 max 16384 2024-09-19T08:03:11.648 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:03:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-19T08:03:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-09-19T08:03:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "7089789c-af16-4525-b78a-9a98a14763c4": { 2024-09-19T08:03:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-09-19T08:03:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "7089789c-af16-4525-b78a-9a98a14763c4", 2024-09-19T08:03:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-09-19T08:03:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:03:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:03:11.648 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:03:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.633+0000 7f66aa3fb640 1 -- 172.21.15.98:0/2594967625 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f66900a34d0 msgr2=0x7f66900a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.633+0000 7f66aa3fb640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f66900a34d0 0x7f66900a5990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f669c009fd0 tx=0x7f669c0096f0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.633+0000 7f66aa3fb640 1 -- 172.21.15.98:0/2594967625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a4102570 msgr2=0x7f66a40fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.633+0000 7f66aa3fb640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a4102570 0x7f66a40fd1c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f668c006010 tx=0x7f668c003c10 comp rx=0 tx=0).stop 2024-09-19T08:03:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.633+0000 7f66aa3fb640 1 -- 172.21.15.98:0/2594967625 shutdown_connections 2024-09-19T08:03:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.633+0000 7f66aa3fb640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f66900a34d0 0x7f66900a5990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.633+0000 7f66aa3fb640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66a4102ef0 0x7f66a40fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.633+0000 7f66aa3fb640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a4102570 0x7f66a40fd1c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.633+0000 7f66aa3fb640 1 --2- 172.21.15.98:0/2594967625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a4100e20 0x7f66a40fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.633+0000 7f66aa3fb640 1 -- 172.21.15.98:0/2594967625 >> 172.21.15.98:0/2594967625 conn(0x7f66a40fc7d0 msgr2=0x7f66a406a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.633+0000 7f94f8ff9640 1 -- 172.21.15.98:0/517464028 <== mgr.15252 v2:172.21.15.98:6800/1961446727 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f94fc0fecb0 con 0x7f94e80a3580 2024-09-19T08:03:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.633+0000 7f66aa3fb640 1 -- 172.21.15.98:0/2594967625 shutdown_connections 2024-09-19T08:03:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.633+0000 7f66aa3fb640 1 -- 172.21.15.98:0/2594967625 wait complete. 2024-09-19T08:03:11.650 INFO:teuthology.orchestra.run.smithi098.stdout:/ 1m 6m3h 2024-09-19T08:03:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.635+0000 7f95016f1640 1 -- 172.21.15.98:0/517464028 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94e80a3580 msgr2=0x7f94e80a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.635+0000 7f95016f1640 1 --2- 172.21.15.98:0/517464028 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94e80a3580 0x7f94e80a5a40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f94fc10ec10 tx=0x7f94e4006c40 comp rx=0 tx=0).stop 2024-09-19T08:03:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.635+0000 7f95016f1640 1 -- 172.21.15.98:0/517464028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94fc101ad0 msgr2=0x7f94fc19a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.635+0000 7f95016f1640 1 --2- 172.21.15.98:0/517464028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94fc101ad0 0x7f94fc19a5b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009230 tx=0x7f94ec004270 comp rx=0 tx=0).stop 2024-09-19T08:03:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.635+0000 7f95016f1640 1 -- 172.21.15.98:0/517464028 shutdown_connections 2024-09-19T08:03:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.635+0000 7f95016f1640 1 --2- 172.21.15.98:0/517464028 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94e80a3580 0x7f94e80a5a40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.635+0000 7f95016f1640 1 --2- 172.21.15.98:0/517464028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94fc107b70 0x7f94fc19ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.635+0000 7f95016f1640 1 --2- 172.21.15.98:0/517464028 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94fc102470 0x7f94fc19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.635+0000 7f95016f1640 1 --2- 172.21.15.98:0/517464028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94fc101ad0 0x7f94fc19a5b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.635+0000 7f95016f1640 1 -- 172.21.15.98:0/517464028 >> 172.21.15.98:0/517464028 conn(0x7f94fc0fd7d0 msgr2=0x7f94fc10d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.635+0000 7f95016f1640 1 -- 172.21.15.98:0/517464028 shutdown_connections 2024-09-19T08:03:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:11.635+0000 7f95016f1640 1 -- 172.21.15.98:0/517464028 wait complete. 2024-09-19T08:03:11.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-09-19T08:03:11.680 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-09-19T08:03:11.692 INFO:teuthology.run_tasks:Running task quiescer... 2024-09-19T08:03:11.739 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-19T08:03:11.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-19T08:03:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:12 smithi139 ceph-mon[34874]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:12 smithi139 ceph-mon[34874]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:12.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:12 smithi139 ceph-mon[34874]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:12.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:12 smithi098 ceph-mon[31333]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:12.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:12 smithi098 ceph-mon[31333]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:12.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:12 smithi098 ceph-mon[31333]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:12.490 INFO:tasks.cephfs.filesystem:scrub status for tag:7089789c-af16-4525-b78a-9a98a14763c4 - {'path': '/', 'tag': '7089789c-af16-4525-b78a-9a98a14763c4', 'options': 'recursive,force'} 2024-09-19T08:03:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:12.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:12 smithi162 ceph-mon[35260]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:12.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:12 smithi162 ceph-mon[35260]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:12.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:12 smithi162 ceph-mon[35260]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:03:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.020+0000 7f59047fa640 1 -- 172.21.15.98:0/780765455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc106f10 msgr2=0x7f58fc10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.020+0000 7f59047fa640 1 --2- 172.21.15.98:0/780765455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc106f10 0x7f58fc10ea90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f58ec009900 tx=0x7f58ec02a410 comp rx=0 tx=0).stop 2024-09-19T08:03:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.020+0000 7f59047fa640 1 -- 172.21.15.98:0/780765455 shutdown_connections 2024-09-19T08:03:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.020+0000 7f59047fa640 1 --2- 172.21.15.98:0/780765455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc106f10 0x7f58fc10ea90 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.020+0000 7f59047fa640 1 --2- 172.21.15.98:0/780765455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58fc102790 0x7f58fc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.020+0000 7f59047fa640 1 --2- 172.21.15.98:0/780765455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58fc101de0 0x7f58fc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.020+0000 7f59047fa640 1 -- 172.21.15.98:0/780765455 >> 172.21.15.98:0/780765455 conn(0x7f58fc0fd8c0 msgr2=0x7f58fc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.020+0000 7f59047fa640 1 -- 172.21.15.98:0/780765455 shutdown_connections 2024-09-19T08:03:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.020+0000 7f59047fa640 1 -- 172.21.15.98:0/780765455 wait complete. 2024-09-19T08:03:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.021+0000 7f59047fa640 1 Processor -- start 2024-09-19T08:03:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.021+0000 7f59047fa640 1 -- start start 2024-09-19T08:03:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.021+0000 7f59047fa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc101de0 0x7f58fc1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.021+0000 7f59047fa640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58fc102790 0x7f58fc198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.021+0000 7f59047fa640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58fc106f10 0x7f58fc19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.021+0000 7f59047fa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc112010 con 0x7f58fc101de0 2024-09-19T08:03:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.021+0000 7f59047fa640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc111e90 con 0x7f58fc106f10 2024-09-19T08:03:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.021+0000 7f59047fa640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc112190 con 0x7f58fc102790 2024-09-19T08:03:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.021+0000 7f5902596640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc101de0 0x7f58fc1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.021+0000 7f5902596640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc101de0 0x7f58fc1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47496/0 (socket says 172.21.15.98:47496) 2024-09-19T08:03:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.021+0000 7f5902596640 1 -- 172.21.15.98:0/4069029397 learned_addr learned my addr 172.21.15.98:0/4069029397 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.022+0000 7f5901d95640 1 --2- 172.21.15.98:0/4069029397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58fc102790 0x7f58fc198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.022+0000 7f5902596640 1 -- 172.21.15.98:0/4069029397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58fc102790 msgr2=0x7f58fc198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.022+0000 7f5902596640 1 --2- 172.21.15.98:0/4069029397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58fc102790 0x7f58fc198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.022+0000 7f5902596640 1 -- 172.21.15.98:0/4069029397 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58fc106f10 msgr2=0x7f58fc19cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.022+0000 7f5902596640 1 --2- 172.21.15.98:0/4069029397 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58fc106f10 0x7f58fc19cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.022+0000 7f5902596640 1 -- 172.21.15.98:0/4069029397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58fc19d5a0 con 0x7f58fc101de0 2024-09-19T08:03:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.022+0000 7f5901d95640 1 --2- 172.21.15.98:0/4069029397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58fc102790 0x7f58fc198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.022+0000 7f5902596640 1 --2- 172.21.15.98:0/4069029397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc101de0 0x7f58fc1985f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f58f400b4a0 tx=0x7f58f400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.022+0000 7f58fb7fe640 1 -- 172.21.15.98:0/4069029397 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f4014070 con 0x7f58fc101de0 2024-09-19T08:03:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.022+0000 7f59047fa640 1 -- 172.21.15.98:0/4069029397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58fc19d890 con 0x7f58fc101de0 2024-09-19T08:03:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.022+0000 7f58fb7fe640 1 -- 172.21.15.98:0/4069029397 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f4010af0 con 0x7f58fc101de0 2024-09-19T08:03:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.022+0000 7f58fb7fe640 1 -- 172.21.15.98:0/4069029397 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f4017d10 con 0x7f58fc101de0 2024-09-19T08:03:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.022+0000 7f59047fa640 1 -- 172.21.15.98:0/4069029397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58fc1a5170 con 0x7f58fc101de0 2024-09-19T08:03:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.023+0000 7f59047fa640 1 -- 172.21.15.98:0/4069029397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58fc102240 con 0x7f58fc101de0 2024-09-19T08:03:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.024+0000 7f58fb7fe640 1 -- 172.21.15.98:0/4069029397 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f58f4017070 con 0x7f58fc101de0 2024-09-19T08:03:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.025+0000 7f58fb7fe640 1 --2- 172.21.15.98:0/4069029397 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58e40a34d0 0x7f58e40a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.025+0000 7f58fb7fe640 1 -- 172.21.15.98:0/4069029397 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f58f40cf340 con 0x7f58fc101de0 2024-09-19T08:03:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.025+0000 7f5901d95640 1 --2- 172.21.15.98:0/4069029397 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58e40a34d0 0x7f58e40a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.025+0000 7f5901d95640 1 --2- 172.21.15.98:0/4069029397 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58e40a34d0 0x7f58e40a5990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0ffe70 tx=0x7f58f00073c0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.029+0000 7f58fb7fe640 1 -- 172.21.15.98:0/4069029397 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f58f409bc10 con 0x7f58fc101de0 2024-09-19T08:03:13.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.193+0000 7f59047fa640 1 -- 172.21.15.98:0/4069029397 --> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f58fc062630 con 0x7f58e40a34d0 2024-09-19T08:03:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.200+0000 7f58fb7fe640 1 -- 172.21.15.98:0/4069029397 <== mgr.15252 v2:172.21.15.98:6800/1961446727 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257816 (secure 0 0 0) 0x7f58fc062630 con 0x7f58e40a34d0 2024-09-19T08:03:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:13.204 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-19T08:03:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.203+0000 7f59047fa640 1 -- 172.21.15.98:0/4069029397 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58e40a34d0 msgr2=0x7f58e40a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.203+0000 7f59047fa640 1 --2- 172.21.15.98:0/4069029397 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58e40a34d0 0x7f58e40a5990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0ffe70 tx=0x7f58f00073c0 comp rx=0 tx=0).stop 2024-09-19T08:03:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.203+0000 7f59047fa640 1 -- 172.21.15.98:0/4069029397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc101de0 msgr2=0x7f58fc1985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.203+0000 7f59047fa640 1 --2- 172.21.15.98:0/4069029397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc101de0 0x7f58fc1985f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f58f400b4a0 tx=0x7f58f400b960 comp rx=0 tx=0).stop 2024-09-19T08:03:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.203+0000 7f59047fa640 1 -- 172.21.15.98:0/4069029397 shutdown_connections 2024-09-19T08:03:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.203+0000 7f59047fa640 1 --2- 172.21.15.98:0/4069029397 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58e40a34d0 0x7f58e40a5990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.203+0000 7f59047fa640 1 --2- 172.21.15.98:0/4069029397 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58fc106f10 0x7f58fc19cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.203+0000 7f59047fa640 1 --2- 172.21.15.98:0/4069029397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58fc102790 0x7f58fc198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.203+0000 7f59047fa640 1 --2- 172.21.15.98:0/4069029397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc101de0 0x7f58fc1985f0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.203+0000 7f59047fa640 1 -- 172.21.15.98:0/4069029397 >> 172.21.15.98:0/4069029397 conn(0x7f58fc0fd8c0 msgr2=0x7f58fc0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.203+0000 7f59047fa640 1 -- 172.21.15.98:0/4069029397 shutdown_connections 2024-09-19T08:03:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:13.203+0000 7f59047fa640 1 -- 172.21.15.98:0/4069029397 wait complete. 2024-09-19T08:03:13.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:13 smithi139 ceph-mon[34874]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 56 op/s 2024-09-19T08:03:13.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:13 smithi098 ceph-mon[31333]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 56 op/s 2024-09-19T08:03:13.619 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:13 smithi162 ceph-mon[35260]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 56 op/s 2024-09-19T08:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-09-19T08:03:12.773319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1391599,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1736,"num_read_kb":1304,"num_write":710,"num_write_kb":5326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181990,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1036,"ondisk_log_size":1036,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376776,"kb_used_data":14784,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565048,"statfs":{"total":1151940427776,"available":1151554609152,"internally_reserved":0,"allocated":15138816,"data_stored":9375741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":370193724},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":200331,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":461,"num_read_kb":502,"num_write":155,"num_write_kb":923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.991513"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780911+0000","last_change":"2024-09-19T08:01:10.768950+0000","last_active":"2024-09-19T08:03:00.780911+0000","last_peered":"2024-09-19T08:03:00.780911+0000","last_clean":"2024-09-19T08:03:00.780911+0000","last_became_active":"2024-09-19T08:01:10.768047+0000","last_became_peered":"2024-09-19T08:01:10.768047+0000","last_unstale":"2024-09-19T08:03:00.780911+0000","last_undegraded":"2024-09-19T08:03:00.780911+0000","last_fullsized":"2024-09-19T08:03:00.780911+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:26.219079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994728+0000","last_change":"2024-09-19T08:01:10.783208+0000","last_active":"2024-09-19T08:03:00.994728+0000","last_peered":"2024-09-19T08:03:00.994728+0000","last_clean":"2024-09-19T08:03:00.994728+0000","last_became_active":"2024-09-19T08:01:10.782888+0000","last_became_peered":"2024-09-19T08:01:10.782888+0000","last_unstale":"2024-09-19T08:03:00.994728+0000","last_undegraded":"2024-09-19T08:03:00.994728+0000","last_fullsized":"2024-09-19T08:03:00.994728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:51:27.603166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994427+0000","last_change":"2024-09-19T08:01:10.770395+0000","last_active":"2024-09-19T08:03:00.994427+0000","last_peered":"2024-09-19T08:03:00.994427+0000","last_clean":"2024-09-19T08:03:00.994427+0000","last_became_active":"2024-09-19T08:01:10.770073+0000","last_became_peered":"2024-09-19T08:01:10.770073+0000","last_unstale":"2024-09-19T08:03:00.994427+0000","last_undegraded":"2024-09-19T08:03:00.994427+0000","last_fullsized":"2024-09-19T08:03:00.994427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:20.047023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761433+0000","last_change":"2024-09-19T08:01:10.777138+0000","last_active":"2024-09-19T08:03:00.761433+0000","last_peered":"2024-09-19T08:03:00.761433+0000","last_clean":"2024-09-19T08:03:00.761433+0000","last_became_active":"2024-09-19T08:01:10.776810+0000","last_became_peered":"2024-09-19T08:01:10.776810+0000","last_unstale":"2024-09-19T08:03:00.761433+0000","last_undegraded":"2024-09-19T08:03:00.761433+0000","last_fullsized":"2024-09-19T08:03:00.761433+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:39.983321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847497+0000","last_change":"2024-09-19T08:01:10.770435+0000","last_active":"2024-09-19T08:03:00.847497+0000","last_peered":"2024-09-19T08:03:00.847497+0000","last_clean":"2024-09-19T08:03:00.847497+0000","last_became_active":"2024-09-19T08:01:10.769686+0000","last_became_peered":"2024-09-19T08:01:10.769686+0000","last_unstale":"2024-09-19T08:03:00.847497+0000","last_undegraded":"2024-09-19T08:03:00.847497+0000","last_fullsized":"2024-09-19T08:03:00.847497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:10:19.610925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839257+0000","last_change":"2024-09-19T08:01:10.782382+0000","last_active":"2024-09-19T08:03:00.839257+0000","last_peered":"2024-09-19T08:03:00.839257+0000","last_clean":"2024-09-19T08:03:00.839257+0000","last_became_active":"2024-09-19T08:01:10.781996+0000","last_became_peered":"2024-09-19T08:01:10.781996+0000","last_unstale":"2024-09-19T08:03:00.839257+0000","last_undegraded":"2024-09-19T08:03:00.839257+0000","last_fullsized":"2024-09-19T08:03:00.839257+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:43:13.574336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837718+0000","last_change":"2024-09-19T08:01:10.773776+0000","last_active":"2024-09-19T08:03:00.837718+0000","last_peered":"2024-09-19T08:03:00.837718+0000","last_clean":"2024-09-19T08:03:00.837718+0000","last_became_active":"2024-09-19T08:01:10.772442+0000","last_became_peered":"2024-09-19T08:01:10.772442+0000","last_unstale":"2024-09-19T08:03:00.837718+0000","last_undegraded":"2024-09-19T08:03:00.837718+0000","last_fullsized":"2024-09-19T08:03:00.837718+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:08:18.155806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848134+0000","last_change":"2024-09-19T08:01:10.781443+0000","last_active":"2024-09-19T08:03:00.848134+0000","last_peered":"2024-09-19T08:03:00.848134+0000","last_clean":"2024-09-19T08:03:00.848134+0000","last_became_active":"2024-09-19T08:01:10.781083+0000","last_became_peered":"2024-09-19T08:01:10.781083+0000","last_unstale":"2024-09-19T08:03:00.848134+0000","last_undegraded":"2024-09-19T08:03:00.848134+0000","last_fullsized":"2024-09-19T08:03:00.848134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:23:22.631865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.998365+0000","last_change":"2024-09-19T08:01:10.767554+0000","last_active":"2024-09-19T08:03:00.998365+0000","last_peered":"2024-09-19T08:03:00.998365+0000","last_clean":"2024-09-19T08:03:00.998365+0000","last_became_active":"2024-09-19T08:01:10.767227+0000","last_became_peered":"2024-09-19T08:01:10.767227+0000","last_unstale":"2024-09-19T08:03:00.998365+0000","last_undegraded":"2024-09-19T08:03:00.998365+0000","last_fullsized":"2024-09-19T08:03:00.998365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:30:13.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839361+0000","last_change":"2024-09-19T08:01:10.783174+0000","last_active":"2024-09-19T08:03:00.839361+0000","last_peered":"2024-09-19T08:03:00.839361+0000","last_clean":"2024-09-19T08:03:00.839361+0000","last_became_active":"2024-09-19T08:01:10.782794+0000","last_became_peered":"2024-09-19T08:01:10.782794+0000","last_unstale":"2024-09-19T08:03:00.839361+0000","last_undegraded":"2024-09-19T08:03:00.839361+0000","last_fullsized":"2024-09-19T08:03:00.839361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:49:19.451107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838191+0000","last_change":"2024-09-19T08:01:10.779638+0000","last_active":"2024-09-19T08:03:00.838191+0000","last_peered":"2024-09-19T08:03:00.838191+0000","last_clean":"2024-09-19T08:03:00.838191+0000","last_became_active":"2024-09-19T08:01:10.779295+0000","last_became_peered":"2024-09-19T08:01:10.779295+0000","last_unstale":"2024-09-19T08:03:00.838191+0000","last_undegraded":"2024-09-19T08:03:00.838191+0000","last_fullsized":"2024-09-19T08:03:00.838191+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:35:10.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761539+0000","last_change":"2024-09-19T08:01:10.784487+0000","last_active":"2024-09-19T08:03:00.761539+0000","last_peered":"2024-09-19T08:03:00.761539+0000","last_clean":"2024-09-19T08:03:00.761539+0000","last_became_active":"2024-09-19T08:01:10.784114+0000","last_became_peered":"2024-09-19T08:01:10.784114+0000","last_unstale":"2024-09-19T08:03:00.761539+0000","last_undegraded":"2024-09-19T08:03:00.761539+0000","last_fullsized":"2024-09-19T08:03:00.761539+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:09:54.819802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758888+0000","last_change":"2024-09-19T08:01:10.783186+0000","last_active":"2024-09-19T08:03:00.758888+0000","last_peered":"2024-09-19T08:03:00.758888+0000","last_clean":"2024-09-19T08:03:00.758888+0000","last_became_active":"2024-09-19T08:01:10.782876+0000","last_became_peered":"2024-09-19T08:01:10.782876+0000","last_unstale":"2024-09-19T08:03:00.758888+0000","last_undegraded":"2024-09-19T08:03:00.758888+0000","last_fullsized":"2024-09-19T08:03:00.758888+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:26:15.252165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994474+0000","last_change":"2024-09-19T08:01:10.783237+0000","last_active":"2024-09-19T08:03:00.994474+0000","last_peered":"2024-09-19T08:03:00.994474+0000","last_clean":"2024-09-19T08:03:00.994474+0000","last_became_active":"2024-09-19T08:01:10.782966+0000","last_became_peered":"2024-09-19T08:01:10.782966+0000","last_unstale":"2024-09-19T08:03:00.994474+0000","last_undegraded":"2024-09-19T08:03:00.994474+0000","last_fullsized":"2024-09-19T08:03:00.994474+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:57:18.771737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994242+0000","last_change":"2024-09-19T08:01:10.771207+0000","last_active":"2024-09-19T08:03:00.994242+0000","last_peered":"2024-09-19T08:03:00.994242+0000","last_clean":"2024-09-19T08:03:00.994242+0000","last_became_active":"2024-09-19T08:01:10.770862+0000","last_became_peered":"2024-09-19T08:01:10.770862+0000","last_unstale":"2024-09-19T08:03:00.994242+0000","last_undegraded":"2024-09-19T08:03:00.994242+0000","last_fullsized":"2024-09-19T08:03:00.994242+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:08:09.313329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997263+0000","last_change":"2024-09-19T08:01:10.775191+0000","last_active":"2024-09-19T08:03:00.997263+0000","last_peered":"2024-09-19T08:03:00.997263+0000","last_clean":"2024-09-19T08:03:00.997263+0000","last_became_active":"2024-09-19T08:01:10.774832+0000","last_became_peered":"2024-09-19T08:01:10.774832+0000","last_unstale":"2024-09-19T08:03:00.997263+0000","last_undegraded":"2024-09-19T08:03:00.997263+0000","last_fullsized":"2024-09-19T08:03:00.997263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:57:29.800787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780968+0000","last_change":"2024-09-19T08:01:10.769115+0000","last_active":"2024-09-19T08:03:00.780968+0000","last_peered":"2024-09-19T08:03:00.780968+0000","last_clean":"2024-09-19T08:03:00.780968+0000","last_became_active":"2024-09-19T08:01:10.768088+0000","last_became_peered":"2024-09-19T08:01:10.768088+0000","last_unstale":"2024-09-19T08:03:00.780968+0000","last_undegraded":"2024-09-19T08:03:00.780968+0000","last_fullsized":"2024-09-19T08:03:00.780968+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:27:16.042722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844124+0000","last_change":"2024-09-19T08:01:10.781426+0000","last_active":"2024-09-19T08:03:00.844124+0000","last_peered":"2024-09-19T08:03:00.844124+0000","last_clean":"2024-09-19T08:03:00.844124+0000","last_became_active":"2024-09-19T08:01:10.781094+0000","last_became_peered":"2024-09-19T08:01:10.781094+0000","last_unstale":"2024-09-19T08:03:00.844124+0000","last_undegraded":"2024-09-19T08:03:00.844124+0000","last_fullsized":"2024-09-19T08:03:00.844124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:43:31.336610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760020+0000","last_change":"2024-09-19T08:01:10.766466+0000","last_active":"2024-09-19T08:03:00.760020+0000","last_peered":"2024-09-19T08:03:00.760020+0000","last_clean":"2024-09-19T08:03:00.760020+0000","last_became_active":"2024-09-19T08:01:10.766123+0000","last_became_peered":"2024-09-19T08:01:10.766123+0000","last_unstale":"2024-09-19T08:03:00.760020+0000","last_undegraded":"2024-09-19T08:03:00.760020+0000","last_fullsized":"2024-09-19T08:03:00.760020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:25.848603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780633+0000","last_change":"2024-09-19T08:01:10.772782+0000","last_active":"2024-09-19T08:03:00.780633+0000","last_peered":"2024-09-19T08:03:00.780633+0000","last_clean":"2024-09-19T08:03:00.780633+0000","last_became_active":"2024-09-19T08:01:10.772445+0000","last_became_peered":"2024-09-19T08:01:10.772445+0000","last_unstale":"2024-09-19T08:03:00.780633+0000","last_undegraded":"2024-09-19T08:03:00.780633+0000","last_fullsized":"2024-09-19T08:03:00.780633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:09.174005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759047+0000","last_change":"2024-09-19T08:01:10.771944+0000","last_active":"2024-09-19T08:03:00.759047+0000","last_peered":"2024-09-19T08:03:00.759047+0000","last_clean":"2024-09-19T08:03:00.759047+0000","last_became_active":"2024-09-19T08:01:10.771643+0000","last_became_peered":"2024-09-19T08:01:10.771643+0000","last_unstale":"2024-09-19T08:03:00.759047+0000","last_undegraded":"2024-09-19T08:03:00.759047+0000","last_fullsized":"2024-09-19T08:03:00.759047+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:46:27.121722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997451+0000","last_change":"2024-09-19T08:01:10.775144+0000","last_active":"2024-09-19T08:03:00.997451+0000","last_peered":"2024-09-19T08:03:00.997451+0000","last_clean":"2024-09-19T08:03:00.997451+0000","last_became_active":"2024-09-19T08:01:10.774769+0000","last_became_peered":"2024-09-19T08:01:10.774769+0000","last_unstale":"2024-09-19T08:03:00.997451+0000","last_undegraded":"2024-09-19T08:03:00.997451+0000","last_fullsized":"2024-09-19T08:03:00.997451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:59:47.053993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848212+0000","last_change":"2024-09-19T08:01:10.766750+0000","last_active":"2024-09-19T08:03:00.848212+0000","last_peered":"2024-09-19T08:03:00.848212+0000","last_clean":"2024-09-19T08:03:00.848212+0000","last_became_active":"2024-09-19T08:01:10.766450+0000","last_became_peered":"2024-09-19T08:01:10.766450+0000","last_unstale":"2024-09-19T08:03:00.848212+0000","last_undegraded":"2024-09-19T08:03:00.848212+0000","last_fullsized":"2024-09-19T08:03:00.848212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:47:58.133815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841186+0000","last_change":"2024-09-19T08:01:10.768430+0000","last_active":"2024-09-19T08:03:00.841186+0000","last_peered":"2024-09-19T08:03:00.841186+0000","last_clean":"2024-09-19T08:03:00.841186+0000","last_became_active":"2024-09-19T08:01:10.768092+0000","last_became_peered":"2024-09-19T08:01:10.768092+0000","last_unstale":"2024-09-19T08:03:00.841186+0000","last_undegraded":"2024-09-19T08:03:00.841186+0000","last_fullsized":"2024-09-19T08:03:00.841186+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:16:42.343138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849206+0000","last_change":"2024-09-19T08:01:10.771131+0000","last_active":"2024-09-19T08:03:00.849206+0000","last_peered":"2024-09-19T08:03:00.849206+0000","last_clean":"2024-09-19T08:03:00.849206+0000","last_became_active":"2024-09-19T08:01:10.770851+0000","last_became_peered":"2024-09-19T08:01:10.770851+0000","last_unstale":"2024-09-19T08:03:00.849206+0000","last_undegraded":"2024-09-19T08:03:00.849206+0000","last_fullsized":"2024-09-19T08:03:00.849206+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:07:15.830476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760273+0000","last_change":"2024-09-19T08:01:10.776220+0000","last_active":"2024-09-19T08:03:00.760273+0000","last_peered":"2024-09-19T08:03:00.760273+0000","last_clean":"2024-09-19T08:03:00.760273+0000","last_became_active":"2024-09-19T08:01:10.775889+0000","last_became_peered":"2024-09-19T08:01:10.775889+0000","last_unstale":"2024-09-19T08:03:00.760273+0000","last_undegraded":"2024-09-19T08:03:00.760273+0000","last_fullsized":"2024-09-19T08:03:00.760273+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:23.714675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780731+0000","last_change":"2024-09-19T08:01:10.776370+0000","last_active":"2024-09-19T08:03:00.780731+0000","last_peered":"2024-09-19T08:03:00.780731+0000","last_clean":"2024-09-19T08:03:00.780731+0000","last_became_active":"2024-09-19T08:01:10.776027+0000","last_became_peered":"2024-09-19T08:01:10.776027+0000","last_unstale":"2024-09-19T08:03:00.780731+0000","last_undegraded":"2024-09-19T08:03:00.780731+0000","last_fullsized":"2024-09-19T08:03:00.780731+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:48:46.293404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758951+0000","last_change":"2024-09-19T08:01:26.758971+0000","last_active":"2024-09-19T08:03:00.758951+0000","last_peered":"2024-09-19T08:03:00.758951+0000","last_clean":"2024-09-19T08:03:00.758951+0000","last_became_active":"2024-09-19T08:01:26.758524+0000","last_became_peered":"2024-09-19T08:01:26.758524+0000","last_unstale":"2024-09-19T08:03:00.758951+0000","last_undegraded":"2024-09-19T08:03:00.758951+0000","last_fullsized":"2024-09-19T08:03:00.758951+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:34:22.439327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994656+0000","last_change":"2024-09-19T08:01:10.767445+0000","last_active":"2024-09-19T08:03:00.994656+0000","last_peered":"2024-09-19T08:03:00.994656+0000","last_clean":"2024-09-19T08:03:00.994656+0000","last_became_active":"2024-09-19T08:01:10.767128+0000","last_became_peered":"2024-09-19T08:01:10.767128+0000","last_unstale":"2024-09-19T08:03:00.994656+0000","last_undegraded":"2024-09-19T08:03:00.994656+0000","last_fullsized":"2024-09-19T08:03:00.994656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:10.273948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759852+0000","last_change":"2024-09-19T08:01:10.768954+0000","last_active":"2024-09-19T08:03:00.759852+0000","last_peered":"2024-09-19T08:03:00.759852+0000","last_clean":"2024-09-19T08:03:00.759852+0000","last_became_active":"2024-09-19T08:01:10.768577+0000","last_became_peered":"2024-09-19T08:01:10.768577+0000","last_unstale":"2024-09-19T08:03:00.759852+0000","last_undegraded":"2024-09-19T08:03:00.759852+0000","last_fullsized":"2024-09-19T08:03:00.759852+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:20:21.812531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759843+0000","last_change":"2024-09-19T08:01:10.774104+0000","last_active":"2024-09-19T08:03:00.759843+0000","last_peered":"2024-09-19T08:03:00.759843+0000","last_clean":"2024-09-19T08:03:00.759843+0000","last_became_active":"2024-09-19T08:01:10.773721+0000","last_became_peered":"2024-09-19T08:01:10.773721+0000","last_unstale":"2024-09-19T08:03:00.759843+0000","last_undegraded":"2024-09-19T08:03:00.759843+0000","last_fullsized":"2024-09-19T08:03:00.759843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:45:34.945994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780513+0000","last_change":"2024-09-19T08:01:26.727296+0000","last_active":"2024-09-19T08:03:00.780513+0000","last_peered":"2024-09-19T08:03:00.780513+0000","last_clean":"2024-09-19T08:03:00.780513+0000","last_became_active":"2024-09-19T08:01:26.726631+0000","last_became_peered":"2024-09-19T08:01:26.726631+0000","last_unstale":"2024-09-19T08:03:00.780513+0000","last_undegraded":"2024-09-19T08:03:00.780513+0000","last_fullsized":"2024-09-19T08:03:00.780513+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:19:24.807050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847424+0000","last_change":"2024-09-19T08:01:10.765282+0000","last_active":"2024-09-19T08:03:00.847424+0000","last_peered":"2024-09-19T08:03:00.847424+0000","last_clean":"2024-09-19T08:03:00.847424+0000","last_became_active":"2024-09-19T08:01:10.764928+0000","last_became_peered":"2024-09-19T08:01:10.764928+0000","last_unstale":"2024-09-19T08:03:00.847424+0000","last_undegraded":"2024-09-19T08:03:00.847424+0000","last_fullsized":"2024-09-19T08:03:00.847424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:00:37.395110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"81'11","reported_seq":152,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:09.854344+0000","last_change":"2024-09-19T08:01:08.760686+0000","last_active":"2024-09-19T08:03:09.854344+0000","last_peered":"2024-09-19T08:03:09.854344+0000","last_clean":"2024-09-19T08:03:09.854344+0000","last_became_active":"2024-09-19T08:01:08.759912+0000","last_became_peered":"2024-09-19T08:01:08.759912+0000","last_unstale":"2024-09-19T08:03:09.854344+0000","last_undegraded":"2024-09-19T08:03:09.854344+0000","last_fullsized":"2024-09-19T08:03:09.854344+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:46:01.879611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2643,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":7,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841395+0000","last_change":"2024-09-19T08:01:10.779668+0000","last_active":"2024-09-19T08:03:00.841395+0000","last_peered":"2024-09-19T08:03:00.841395+0000","last_clean":"2024-09-19T08:03:00.841395+0000","last_became_active":"2024-09-19T08:01:10.779265+0000","last_became_peered":"2024-09-19T08:01:10.779265+0000","last_unstale":"2024-09-19T08:03:00.841395+0000","last_undegraded":"2024-09-19T08:03:00.841395+0000","last_fullsized":"2024-09-19T08:03:00.841395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:06.177700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838967+0000","last_change":"2024-09-19T08:01:08.762668+0000","last_active":"2024-09-19T08:03:00.838967+0000","last_peered":"2024-09-19T08:03:00.838967+0000","last_clean":"2024-09-19T08:03:00.838967+0000","last_became_active":"2024-09-19T08:01:08.762277+0000","last_became_peered":"2024-09-19T08:01:08.762277+0000","last_unstale":"2024-09-19T08:03:00.838967+0000","last_undegraded":"2024-09-19T08:03:00.838967+0000","last_fullsized":"2024-09-19T08:03:00.838967+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:36:59.316655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758585+0000","last_change":"2024-09-19T08:01:08.757401+0000","last_active":"2024-09-19T08:03:00.758585+0000","last_peered":"2024-09-19T08:03:00.758585+0000","last_clean":"2024-09-19T08:03:00.758585+0000","last_became_active":"2024-09-19T08:01:08.757067+0000","last_became_peered":"2024-09-19T08:01:08.757067+0000","last_unstale":"2024-09-19T08:03:00.758585+0000","last_undegraded":"2024-09-19T08:03:00.758585+0000","last_fullsized":"2024-09-19T08:03:00.758585+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:19.617830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848289+0000","last_change":"2024-09-19T08:01:10.765149+0000","last_active":"2024-09-19T08:03:00.848289+0000","last_peered":"2024-09-19T08:03:00.848289+0000","last_clean":"2024-09-19T08:03:00.848289+0000","last_became_active":"2024-09-19T08:01:10.764840+0000","last_became_peered":"2024-09-19T08:01:10.764840+0000","last_unstale":"2024-09-19T08:03:00.848289+0000","last_undegraded":"2024-09-19T08:03:00.848289+0000","last_fullsized":"2024-09-19T08:03:00.848289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:15.539307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841355+0000","last_change":"2024-09-19T08:01:08.759435+0000","last_active":"2024-09-19T08:03:00.841355+0000","last_peered":"2024-09-19T08:03:00.841355+0000","last_clean":"2024-09-19T08:03:00.841355+0000","last_became_active":"2024-09-19T08:01:08.759046+0000","last_became_peered":"2024-09-19T08:01:08.759046+0000","last_unstale":"2024-09-19T08:03:00.841355+0000","last_undegraded":"2024-09-19T08:03:00.841355+0000","last_fullsized":"2024-09-19T08:03:00.841355+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:21:24.077626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.843896+0000","last_change":"2024-09-19T08:01:10.778432+0000","last_active":"2024-09-19T08:03:00.843896+0000","last_peered":"2024-09-19T08:03:00.843896+0000","last_clean":"2024-09-19T08:03:00.843896+0000","last_became_active":"2024-09-19T08:01:10.778098+0000","last_became_peered":"2024-09-19T08:01:10.778098+0000","last_unstale":"2024-09-19T08:03:00.843896+0000","last_undegraded":"2024-09-19T08:03:00.843896+0000","last_fullsized":"2024-09-19T08:03:00.843896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:40:12.921165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":250,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.855304+0000","last_change":"2024-09-19T08:01:08.764117+0000","last_active":"2024-09-19T08:03:10.855304+0000","last_peered":"2024-09-19T08:03:10.855304+0000","last_clean":"2024-09-19T08:03:10.855304+0000","last_became_active":"2024-09-19T08:01:08.763308+0000","last_became_peered":"2024-09-19T08:01:08.763308+0000","last_unstale":"2024-09-19T08:03:10.855304+0000","last_undegraded":"2024-09-19T08:03:10.855304+0000","last_fullsized":"2024-09-19T08:03:10.855304+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:19:57.014628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":220,"num_read_kb":220,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997921+0000","last_change":"2024-09-19T08:01:10.781771+0000","last_active":"2024-09-19T08:03:00.997921+0000","last_peered":"2024-09-19T08:03:00.997921+0000","last_clean":"2024-09-19T08:03:00.997921+0000","last_became_active":"2024-09-19T08:01:10.781432+0000","last_became_peered":"2024-09-19T08:01:10.781432+0000","last_unstale":"2024-09-19T08:03:00.997921+0000","last_undegraded":"2024-09-19T08:03:00.997921+0000","last_fullsized":"2024-09-19T08:03:00.997921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:25:15.067850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838023+0000","last_change":"2024-09-19T08:01:26.771171+0000","last_active":"2024-09-19T08:03:00.838023+0000","last_peered":"2024-09-19T08:03:00.838023+0000","last_clean":"2024-09-19T08:03:00.838023+0000","last_became_active":"2024-09-19T08:01:26.759274+0000","last_became_peered":"2024-09-19T08:01:26.759274+0000","last_unstale":"2024-09-19T08:03:00.838023+0000","last_undegraded":"2024-09-19T08:03:00.838023+0000","last_fullsized":"2024-09-19T08:03:00.838023+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:20:54.603420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994926+0000","last_change":"2024-09-19T08:01:10.768318+0000","last_active":"2024-09-19T08:03:00.994926+0000","last_peered":"2024-09-19T08:03:00.994926+0000","last_clean":"2024-09-19T08:03:00.994926+0000","last_became_active":"2024-09-19T08:01:10.768022+0000","last_became_peered":"2024-09-19T08:01:10.768022+0000","last_unstale":"2024-09-19T08:03:00.994926+0000","last_undegraded":"2024-09-19T08:03:00.994926+0000","last_fullsized":"2024-09-19T08:03:00.994926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:00:45.328638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841594+0000","last_change":"2024-09-19T08:01:09.750530+0000","last_active":"2024-09-19T08:03:00.841594+0000","last_peered":"2024-09-19T08:03:00.841594+0000","last_clean":"2024-09-19T08:03:00.841594+0000","last_became_active":"2024-09-19T08:01:09.750029+0000","last_became_peered":"2024-09-19T08:01:09.750029+0000","last_unstale":"2024-09-19T08:03:00.841594+0000","last_undegraded":"2024-09-19T08:03:00.841594+0000","last_fullsized":"2024-09-19T08:03:00.841594+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:01:16.062992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997659+0000","last_change":"2024-09-19T08:01:26.724636+0000","last_active":"2024-09-19T08:03:00.997659+0000","last_peered":"2024-09-19T08:03:00.997659+0000","last_clean":"2024-09-19T08:03:00.997659+0000","last_became_active":"2024-09-19T08:01:26.724171+0000","last_became_peered":"2024-09-19T08:01:26.724171+0000","last_unstale":"2024-09-19T08:03:00.997659+0000","last_undegraded":"2024-09-19T08:03:00.997659+0000","last_fullsized":"2024-09-19T08:03:00.997659+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:54:05.998374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'39","reported_seq":300,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.855298+0000","last_change":"2024-09-19T08:01:08.761950+0000","last_active":"2024-09-19T08:03:10.855298+0000","last_peered":"2024-09-19T08:03:10.855298+0000","last_clean":"2024-09-19T08:03:10.855298+0000","last_became_active":"2024-09-19T08:01:08.761533+0000","last_became_peered":"2024-09-19T08:01:08.761533+0000","last_unstale":"2024-09-19T08:03:10.855298+0000","last_undegraded":"2024-09-19T08:03:10.855298+0000","last_fullsized":"2024-09-19T08:03:10.855298+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:07:26.668141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69393,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":237,"num_read_kb":187,"num_write":41,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847631+0000","last_change":"2024-09-19T08:01:10.774280+0000","last_active":"2024-09-19T08:03:00.847631+0000","last_peered":"2024-09-19T08:03:00.847631+0000","last_clean":"2024-09-19T08:03:00.847631+0000","last_became_active":"2024-09-19T08:01:10.773982+0000","last_became_peered":"2024-09-19T08:01:10.773982+0000","last_unstale":"2024-09-19T08:03:00.847631+0000","last_undegraded":"2024-09-19T08:03:00.847631+0000","last_fullsized":"2024-09-19T08:03:00.847631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:21:43.180057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760058+0000","last_change":"2024-09-19T08:01:08.761067+0000","last_active":"2024-09-19T08:03:00.760058+0000","last_peered":"2024-09-19T08:03:00.760058+0000","last_clean":"2024-09-19T08:03:00.760058+0000","last_became_active":"2024-09-19T08:01:08.760664+0000","last_became_peered":"2024-09-19T08:01:08.760664+0000","last_unstale":"2024-09-19T08:03:00.760058+0000","last_undegraded":"2024-09-19T08:03:00.760058+0000","last_fullsized":"2024-09-19T08:03:00.760058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:22:38.086891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758630+0000","last_change":"2024-09-19T08:01:10.771845+0000","last_active":"2024-09-19T08:03:00.758630+0000","last_peered":"2024-09-19T08:03:00.758630+0000","last_clean":"2024-09-19T08:03:00.758630+0000","last_became_active":"2024-09-19T08:01:10.771533+0000","last_became_peered":"2024-09-19T08:01:10.771533+0000","last_unstale":"2024-09-19T08:03:00.758630+0000","last_undegraded":"2024-09-19T08:03:00.758630+0000","last_fullsized":"2024-09-19T08:03:00.758630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:29:35.180763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841267+0000","last_change":"2024-09-19T08:01:08.763869+0000","last_active":"2024-09-19T08:03:00.841267+0000","last_peered":"2024-09-19T08:03:00.841267+0000","last_clean":"2024-09-19T08:03:00.841267+0000","last_became_active":"2024-09-19T08:01:08.763451+0000","last_became_peered":"2024-09-19T08:01:08.763451+0000","last_unstale":"2024-09-19T08:03:00.841267+0000","last_undegraded":"2024-09-19T08:03:00.841267+0000","last_fullsized":"2024-09-19T08:03:00.841267+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:32:40.901049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838028+0000","last_change":"2024-09-19T08:01:10.770823+0000","last_active":"2024-09-19T08:03:00.838028+0000","last_peered":"2024-09-19T08:03:00.838028+0000","last_clean":"2024-09-19T08:03:00.838028+0000","last_became_active":"2024-09-19T08:01:10.770462+0000","last_became_peered":"2024-09-19T08:01:10.770462+0000","last_unstale":"2024-09-19T08:03:00.838028+0000","last_undegraded":"2024-09-19T08:03:00.838028+0000","last_fullsized":"2024-09-19T08:03:00.838028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:04:32.832009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'7","reported_seq":145,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:09.854029+0000","last_change":"2024-09-19T08:01:08.759429+0000","last_active":"2024-09-19T08:03:09.854029+0000","last_peered":"2024-09-19T08:03:09.854029+0000","last_clean":"2024-09-19T08:03:09.854029+0000","last_became_active":"2024-09-19T08:01:08.759062+0000","last_became_peered":"2024-09-19T08:01:08.759062+0000","last_unstale":"2024-09-19T08:03:09.854029+0000","last_undegraded":"2024-09-19T08:03:09.854029+0000","last_fullsized":"2024-09-19T08:03:09.854029+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:16:51.582435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1374,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837660+0000","last_change":"2024-09-19T08:01:10.784735+0000","last_active":"2024-09-19T08:03:00.837660+0000","last_peered":"2024-09-19T08:03:00.837660+0000","last_clean":"2024-09-19T08:03:00.837660+0000","last_became_active":"2024-09-19T08:01:10.784346+0000","last_became_peered":"2024-09-19T08:01:10.784346+0000","last_unstale":"2024-09-19T08:03:00.837660+0000","last_undegraded":"2024-09-19T08:03:00.837660+0000","last_fullsized":"2024-09-19T08:03:00.837660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:27:54.580391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847399+0000","last_change":"2024-09-19T08:01:09.749498+0000","last_active":"2024-09-19T08:03:00.847399+0000","last_peered":"2024-09-19T08:03:00.847399+0000","last_clean":"2024-09-19T08:03:00.847399+0000","last_became_active":"2024-09-19T08:01:09.749091+0000","last_became_peered":"2024-09-19T08:01:09.749091+0000","last_unstale":"2024-09-19T08:03:00.847399+0000","last_undegraded":"2024-09-19T08:03:00.847399+0000","last_fullsized":"2024-09-19T08:03:00.847399+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:56:13.644931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841217+0000","last_change":"2024-09-19T08:01:10.779599+0000","last_active":"2024-09-19T08:03:00.841217+0000","last_peered":"2024-09-19T08:03:00.841217+0000","last_clean":"2024-09-19T08:03:00.841217+0000","last_became_active":"2024-09-19T08:01:10.779219+0000","last_became_peered":"2024-09-19T08:01:10.779219+0000","last_unstale":"2024-09-19T08:03:00.841217+0000","last_undegraded":"2024-09-19T08:03:00.841217+0000","last_fullsized":"2024-09-19T08:03:00.841217+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:20.193753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847308+0000","last_change":"2024-09-19T08:01:08.761432+0000","last_active":"2024-09-19T08:03:00.847308+0000","last_peered":"2024-09-19T08:03:00.847308+0000","last_clean":"2024-09-19T08:03:00.847308+0000","last_became_active":"2024-09-19T08:01:08.761044+0000","last_became_peered":"2024-09-19T08:01:08.761044+0000","last_unstale":"2024-09-19T08:03:00.847308+0000","last_undegraded":"2024-09-19T08:03:00.847308+0000","last_fullsized":"2024-09-19T08:03:00.847308+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:12.237334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780765+0000","last_change":"2024-09-19T08:01:10.772559+0000","last_active":"2024-09-19T08:03:00.780765+0000","last_peered":"2024-09-19T08:03:00.780765+0000","last_clean":"2024-09-19T08:03:00.780765+0000","last_became_active":"2024-09-19T08:01:10.772176+0000","last_became_peered":"2024-09-19T08:01:10.772176+0000","last_unstale":"2024-09-19T08:03:00.780765+0000","last_undegraded":"2024-09-19T08:03:00.780765+0000","last_fullsized":"2024-09-19T08:03:00.780765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:27:16.296127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758996+0000","last_change":"2024-09-19T08:01:08.759989+0000","last_active":"2024-09-19T08:03:00.758996+0000","last_peered":"2024-09-19T08:03:00.758996+0000","last_clean":"2024-09-19T08:03:00.758996+0000","last_became_active":"2024-09-19T08:01:08.759705+0000","last_became_peered":"2024-09-19T08:01:08.759705+0000","last_unstale":"2024-09-19T08:03:00.758996+0000","last_undegraded":"2024-09-19T08:03:00.758996+0000","last_fullsized":"2024-09-19T08:03:00.758996+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:33:25.237567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838150+0000","last_change":"2024-09-19T08:01:10.776528+0000","last_active":"2024-09-19T08:03:00.838150+0000","last_peered":"2024-09-19T08:03:00.838150+0000","last_clean":"2024-09-19T08:03:00.838150+0000","last_became_active":"2024-09-19T08:01:10.776222+0000","last_became_peered":"2024-09-19T08:01:10.776222+0000","last_unstale":"2024-09-19T08:03:00.838150+0000","last_undegraded":"2024-09-19T08:03:00.838150+0000","last_fullsized":"2024-09-19T08:03:00.838150+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.787492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'67","reported_seq":122,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:11.300163+0000","last_change":"2024-09-19T08:01:08.759631+0000","last_active":"2024-09-19T08:03:11.300163+0000","last_peered":"2024-09-19T08:03:11.300163+0000","last_clean":"2024-09-19T08:03:11.300163+0000","last_became_active":"2024-09-19T08:01:08.759156+0000","last_became_peered":"2024-09-19T08:01:08.759156+0000","last_unstale":"2024-09-19T08:03:11.300163+0000","last_undegraded":"2024-09-19T08:03:11.300163+0000","last_fullsized":"2024-09-19T08:03:11.300163+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:48:50.910824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837969+0000","last_change":"2024-09-19T08:01:10.772660+0000","last_active":"2024-09-19T08:03:00.837969+0000","last_peered":"2024-09-19T08:03:00.837969+0000","last_clean":"2024-09-19T08:03:00.837969+0000","last_became_active":"2024-09-19T08:01:10.771816+0000","last_became_peered":"2024-09-19T08:01:10.771816+0000","last_unstale":"2024-09-19T08:03:00.837969+0000","last_undegraded":"2024-09-19T08:03:00.837969+0000","last_fullsized":"2024-09-19T08:03:00.837969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:04:32.667107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'7","reported_seq":144,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:09.883643+0000","last_change":"2024-09-19T08:01:09.749844+0000","last_active":"2024-09-19T08:03:09.883643+0000","last_peered":"2024-09-19T08:03:09.883643+0000","last_clean":"2024-09-19T08:03:09.883643+0000","last_became_active":"2024-09-19T08:01:09.749410+0000","last_became_peered":"2024-09-19T08:01:09.749410+0000","last_unstale":"2024-09-19T08:03:09.883643+0000","last_undegraded":"2024-09-19T08:03:09.883643+0000","last_fullsized":"2024-09-19T08:03:09.883643+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:03:14.989814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1704,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":113,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838196+0000","last_change":"2024-09-19T08:01:10.773782+0000","last_active":"2024-09-19T08:03:00.838196+0000","last_peered":"2024-09-19T08:03:00.838196+0000","last_clean":"2024-09-19T08:03:00.838196+0000","last_became_active":"2024-09-19T08:01:10.773471+0000","last_became_peered":"2024-09-19T08:01:10.773471+0000","last_unstale":"2024-09-19T08:03:00.838196+0000","last_undegraded":"2024-09-19T08:03:00.838196+0000","last_fullsized":"2024-09-19T08:03:00.838196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:15.780259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"81'148","reported_seq":388,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:02.852179+0000","last_change":"2024-09-19T08:00:01.677517+0000","last_active":"2024-09-19T08:03:02.852179+0000","last_peered":"2024-09-19T08:03:02.852179+0000","last_clean":"2024-09-19T08:03:02.852179+0000","last_became_active":"2024-09-19T08:00:01.677089+0000","last_became_peered":"2024-09-19T08:00:01.677089+0000","last_unstale":"2024-09-19T08:03:02.852179+0000","last_undegraded":"2024-09-19T08:03:02.852179+0000","last_fullsized":"2024-09-19T08:03:02.852179+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_clean_scrub_stamp":"2024-09-19T07:56:43.962142+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:10:36.158811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761731+0000","last_change":"2024-09-19T08:01:08.759819+0000","last_active":"2024-09-19T08:03:00.761731+0000","last_peered":"2024-09-19T08:03:00.761731+0000","last_clean":"2024-09-19T08:03:00.761731+0000","last_became_active":"2024-09-19T08:01:08.759420+0000","last_became_peered":"2024-09-19T08:01:08.759420+0000","last_unstale":"2024-09-19T08:03:00.761731+0000","last_undegraded":"2024-09-19T08:03:00.761731+0000","last_fullsized":"2024-09-19T08:03:00.761731+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:06:16.852323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.843903+0000","last_change":"2024-09-19T08:01:26.724260+0000","last_active":"2024-09-19T08:03:00.843903+0000","last_peered":"2024-09-19T08:03:00.843903+0000","last_clean":"2024-09-19T08:03:00.843903+0000","last_became_active":"2024-09-19T08:01:26.723851+0000","last_became_peered":"2024-09-19T08:01:26.723851+0000","last_unstale":"2024-09-19T08:03:00.843903+0000","last_undegraded":"2024-09-19T08:03:00.843903+0000","last_fullsized":"2024-09-19T08:03:00.843903+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:18:46.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837872+0000","last_change":"2024-09-19T08:01:08.761291+0000","last_active":"2024-09-19T08:03:00.837872+0000","last_peered":"2024-09-19T08:03:00.837872+0000","last_clean":"2024-09-19T08:03:00.837872+0000","last_became_active":"2024-09-19T08:01:08.760238+0000","last_became_peered":"2024-09-19T08:01:08.760238+0000","last_unstale":"2024-09-19T08:03:00.837872+0000","last_undegraded":"2024-09-19T08:03:00.837872+0000","last_fullsized":"2024-09-19T08:03:00.837872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:26:09.980573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758639+0000","last_change":"2024-09-19T08:01:10.773056+0000","last_active":"2024-09-19T08:03:00.758639+0000","last_peered":"2024-09-19T08:03:00.758639+0000","last_clean":"2024-09-19T08:03:00.758639+0000","last_became_active":"2024-09-19T08:01:10.772431+0000","last_became_peered":"2024-09-19T08:01:10.772431+0000","last_unstale":"2024-09-19T08:03:00.758639+0000","last_undegraded":"2024-09-19T08:03:00.758639+0000","last_fullsized":"2024-09-19T08:03:00.758639+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:39:06.878086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":143,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.854713+0000","last_change":"2024-09-19T08:01:09.749707+0000","last_active":"2024-09-19T08:03:10.854713+0000","last_peered":"2024-09-19T08:03:10.854713+0000","last_clean":"2024-09-19T08:03:10.854713+0000","last_became_active":"2024-09-19T08:01:09.749013+0000","last_became_peered":"2024-09-19T08:01:09.749013+0000","last_unstale":"2024-09-19T08:03:10.854713+0000","last_undegraded":"2024-09-19T08:03:10.854713+0000","last_fullsized":"2024-09-19T08:03:10.854713+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:40:12.682158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841250+0000","last_change":"2024-09-19T08:01:10.774101+0000","last_active":"2024-09-19T08:03:00.841250+0000","last_peered":"2024-09-19T08:03:00.841250+0000","last_clean":"2024-09-19T08:03:00.841250+0000","last_became_active":"2024-09-19T08:01:10.773822+0000","last_became_peered":"2024-09-19T08:01:10.773822+0000","last_unstale":"2024-09-19T08:03:00.841250+0000","last_undegraded":"2024-09-19T08:03:00.841250+0000","last_fullsized":"2024-09-19T08:03:00.841250+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:29.571118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"81'6","reported_seq":142,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.855374+0000","last_change":"2024-09-19T08:01:08.761675+0000","last_active":"2024-09-19T08:03:10.855374+0000","last_peered":"2024-09-19T08:03:10.855374+0000","last_clean":"2024-09-19T08:03:10.855374+0000","last_became_active":"2024-09-19T08:01:08.761346+0000","last_became_peered":"2024-09-19T08:01:08.761346+0000","last_unstale":"2024-09-19T08:03:10.855374+0000","last_undegraded":"2024-09-19T08:03:10.855374+0000","last_fullsized":"2024-09-19T08:03:10.855374+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:51:19.237458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839233+0000","last_change":"2024-09-19T08:01:10.781565+0000","last_active":"2024-09-19T08:03:00.839233+0000","last_peered":"2024-09-19T08:03:00.839233+0000","last_clean":"2024-09-19T08:03:00.839233+0000","last_became_active":"2024-09-19T08:01:10.781301+0000","last_became_peered":"2024-09-19T08:01:10.781301+0000","last_unstale":"2024-09-19T08:03:00.839233+0000","last_undegraded":"2024-09-19T08:03:00.839233+0000","last_fullsized":"2024-09-19T08:03:00.839233+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:55:52.932009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844577+0000","last_change":"2024-09-19T08:01:09.748918+0000","last_active":"2024-09-19T08:03:00.844577+0000","last_peered":"2024-09-19T08:03:00.844577+0000","last_clean":"2024-09-19T08:03:00.844577+0000","last_became_active":"2024-09-19T08:01:09.748497+0000","last_became_peered":"2024-09-19T08:01:09.748497+0000","last_unstale":"2024-09-19T08:03:00.844577+0000","last_undegraded":"2024-09-19T08:03:00.844577+0000","last_fullsized":"2024-09-19T08:03:00.844577+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:20:31.030133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839231+0000","last_change":"2024-09-19T08:01:10.781388+0000","last_active":"2024-09-19T08:03:00.839231+0000","last_peered":"2024-09-19T08:03:00.839231+0000","last_clean":"2024-09-19T08:03:00.839231+0000","last_became_active":"2024-09-19T08:01:10.781063+0000","last_became_peered":"2024-09-19T08:01:10.781063+0000","last_unstale":"2024-09-19T08:03:00.839231+0000","last_undegraded":"2024-09-19T08:03:00.839231+0000","last_fullsized":"2024-09-19T08:03:00.839231+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:12:08.852717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847755+0000","last_change":"2024-09-19T08:01:09.750080+0000","last_active":"2024-09-19T08:03:00.847755+0000","last_peered":"2024-09-19T08:03:00.847755+0000","last_clean":"2024-09-19T08:03:00.847755+0000","last_became_active":"2024-09-19T08:01:09.749053+0000","last_became_peered":"2024-09-19T08:01:09.749053+0000","last_unstale":"2024-09-19T08:03:00.847755+0000","last_undegraded":"2024-09-19T08:03:00.847755+0000","last_fullsized":"2024-09-19T08:03:00.847755+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:15:45.292714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759144+0000","last_change":"2024-09-19T08:01:10.767070+0000","last_active":"2024-09-19T08:03:00.759144+0000","last_peered":"2024-09-19T08:03:00.759144+0000","last_clean":"2024-09-19T08:03:00.759144+0000","last_became_active":"2024-09-19T08:01:10.766760+0000","last_became_peered":"2024-09-19T08:01:10.766760+0000","last_unstale":"2024-09-19T08:03:00.759144+0000","last_undegraded":"2024-09-19T08:03:00.759144+0000","last_fullsized":"2024-09-19T08:03:00.759144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:17.841135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'105","reported_seq":221,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:11.632933+0000","last_change":"2024-09-19T08:01:08.756143+0000","last_active":"2024-09-19T08:03:11.632933+0000","last_peered":"2024-09-19T08:03:11.632933+0000","last_clean":"2024-09-19T08:03:11.632933+0000","last_became_active":"2024-09-19T08:01:08.755777+0000","last_became_peered":"2024-09-19T08:01:08.755777+0000","last_unstale":"2024-09-19T08:03:11.632933+0000","last_undegraded":"2024-09-19T08:03:11.632933+0000","last_fullsized":"2024-09-19T08:03:11.632933+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:21.430673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":191,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849237+0000","last_change":"2024-09-19T08:01:10.774140+0000","last_active":"2024-09-19T08:03:00.849237+0000","last_peered":"2024-09-19T08:03:00.849237+0000","last_clean":"2024-09-19T08:03:00.849237+0000","last_became_active":"2024-09-19T08:01:10.773819+0000","last_became_peered":"2024-09-19T08:01:10.773819+0000","last_unstale":"2024-09-19T08:03:00.849237+0000","last_undegraded":"2024-09-19T08:03:00.849237+0000","last_fullsized":"2024-09-19T08:03:00.849237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:15:13.174013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761607+0000","last_change":"2024-09-19T08:01:09.169129+0000","last_active":"2024-09-19T08:03:00.761607+0000","last_peered":"2024-09-19T08:03:00.761607+0000","last_clean":"2024-09-19T08:03:00.761607+0000","last_became_active":"2024-09-19T08:01:09.168734+0000","last_became_peered":"2024-09-19T08:01:09.168734+0000","last_unstale":"2024-09-19T08:03:00.761607+0000","last_undegraded":"2024-09-19T08:03:00.761607+0000","last_fullsized":"2024-09-19T08:03:00.761607+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:01:52.232391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994759+0000","last_change":"2024-09-19T08:01:10.768592+0000","last_active":"2024-09-19T08:03:00.994759+0000","last_peered":"2024-09-19T08:03:00.994759+0000","last_clean":"2024-09-19T08:03:00.994759+0000","last_became_active":"2024-09-19T08:01:10.767645+0000","last_became_peered":"2024-09-19T08:01:10.767645+0000","last_unstale":"2024-09-19T08:03:00.994759+0000","last_undegraded":"2024-09-19T08:03:00.994759+0000","last_fullsized":"2024-09-19T08:03:00.994759+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:08:46.227461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997364+0000","last_change":"2024-09-19T08:01:09.749240+0000","last_active":"2024-09-19T08:03:00.997364+0000","last_peered":"2024-09-19T08:03:00.997364+0000","last_clean":"2024-09-19T08:03:00.997364+0000","last_became_active":"2024-09-19T08:01:09.748945+0000","last_became_peered":"2024-09-19T08:01:09.748945+0000","last_unstale":"2024-09-19T08:03:00.997364+0000","last_undegraded":"2024-09-19T08:03:00.997364+0000","last_fullsized":"2024-09-19T08:03:00.997364+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:25:13.791290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847326+0000","last_change":"2024-09-19T08:01:10.783133+0000","last_active":"2024-09-19T08:03:00.847326+0000","last_peered":"2024-09-19T08:03:00.847326+0000","last_clean":"2024-09-19T08:03:00.847326+0000","last_became_active":"2024-09-19T08:01:10.782797+0000","last_became_peered":"2024-09-19T08:01:10.782797+0000","last_unstale":"2024-09-19T08:03:00.847326+0000","last_undegraded":"2024-09-19T08:03:00.847326+0000","last_fullsized":"2024-09-19T08:03:00.847326+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:30:43.551877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848489+0000","last_change":"2024-09-19T08:01:08.757964+0000","last_active":"2024-09-19T08:03:00.848489+0000","last_peered":"2024-09-19T08:03:00.848489+0000","last_clean":"2024-09-19T08:03:00.848489+0000","last_became_active":"2024-09-19T08:01:08.757526+0000","last_became_peered":"2024-09-19T08:01:08.757526+0000","last_unstale":"2024-09-19T08:03:00.848489+0000","last_undegraded":"2024-09-19T08:03:00.848489+0000","last_fullsized":"2024-09-19T08:03:00.848489+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:03.565702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780557+0000","last_change":"2024-09-19T08:01:10.769974+0000","last_active":"2024-09-19T08:03:00.780557+0000","last_peered":"2024-09-19T08:03:00.780557+0000","last_clean":"2024-09-19T08:03:00.780557+0000","last_became_active":"2024-09-19T08:01:10.769650+0000","last_became_peered":"2024-09-19T08:01:10.769650+0000","last_unstale":"2024-09-19T08:03:00.780557+0000","last_undegraded":"2024-09-19T08:03:00.780557+0000","last_fullsized":"2024-09-19T08:03:00.780557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:01.755198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"81'4","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760043+0000","last_change":"2024-09-19T08:01:08.763476+0000","last_active":"2024-09-19T08:03:00.760043+0000","last_peered":"2024-09-19T08:03:00.760043+0000","last_clean":"2024-09-19T08:03:00.760043+0000","last_became_active":"2024-09-19T08:01:08.763154+0000","last_became_peered":"2024-09-19T08:01:08.763154+0000","last_unstale":"2024-09-19T08:03:00.760043+0000","last_undegraded":"2024-09-19T08:03:00.760043+0000","last_fullsized":"2024-09-19T08:03:00.760043+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:06.202452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848751+0000","last_change":"2024-09-19T08:01:10.784201+0000","last_active":"2024-09-19T08:03:00.848751+0000","last_peered":"2024-09-19T08:03:00.848751+0000","last_clean":"2024-09-19T08:03:00.848751+0000","last_became_active":"2024-09-19T08:01:10.783867+0000","last_became_peered":"2024-09-19T08:01:10.783867+0000","last_unstale":"2024-09-19T08:03:00.848751+0000","last_undegraded":"2024-09-19T08:03:00.848751+0000","last_fullsized":"2024-09-19T08:03:00.848751+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:43:50.082183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847290+0000","last_change":"2024-09-19T08:01:08.760479+0000","last_active":"2024-09-19T08:03:00.847290+0000","last_peered":"2024-09-19T08:03:00.847290+0000","last_clean":"2024-09-19T08:03:00.847290+0000","last_became_active":"2024-09-19T08:01:08.760004+0000","last_became_peered":"2024-09-19T08:01:08.760004+0000","last_unstale":"2024-09-19T08:03:00.847290+0000","last_undegraded":"2024-09-19T08:03:00.847290+0000","last_fullsized":"2024-09-19T08:03:00.847290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:41:19.956623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837851+0000","last_change":"2024-09-19T08:01:10.772184+0000","last_active":"2024-09-19T08:03:00.837851+0000","last_peered":"2024-09-19T08:03:00.837851+0000","last_clean":"2024-09-19T08:03:00.837851+0000","last_became_active":"2024-09-19T08:01:10.771825+0000","last_became_peered":"2024-09-19T08:01:10.771825+0000","last_unstale":"2024-09-19T08:03:00.837851+0000","last_undegraded":"2024-09-19T08:03:00.837851+0000","last_fullsized":"2024-09-19T08:03:00.837851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:36.764575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841567+0000","last_change":"2024-09-19T08:01:10.779368+0000","last_active":"2024-09-19T08:03:00.841567+0000","last_peered":"2024-09-19T08:03:00.841567+0000","last_clean":"2024-09-19T08:03:00.841567+0000","last_became_active":"2024-09-19T08:01:10.778884+0000","last_became_peered":"2024-09-19T08:01:10.778884+0000","last_unstale":"2024-09-19T08:03:00.841567+0000","last_undegraded":"2024-09-19T08:03:00.841567+0000","last_fullsized":"2024-09-19T08:03:00.841567+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:28:29.630300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780536+0000","last_change":"2024-09-19T08:01:26.741053+0000","last_active":"2024-09-19T08:03:00.780536+0000","last_peered":"2024-09-19T08:03:00.780536+0000","last_clean":"2024-09-19T08:03:00.780536+0000","last_became_active":"2024-09-19T08:01:26.724714+0000","last_became_peered":"2024-09-19T08:01:26.724714+0000","last_unstale":"2024-09-19T08:03:00.780536+0000","last_undegraded":"2024-09-19T08:03:00.780536+0000","last_fullsized":"2024-09-19T08:03:00.780536+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:59.583797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997323+0000","last_change":"2024-09-19T08:01:10.781389+0000","last_active":"2024-09-19T08:03:00.997323+0000","last_peered":"2024-09-19T08:03:00.997323+0000","last_clean":"2024-09-19T08:03:00.997323+0000","last_became_active":"2024-09-19T08:01:10.781080+0000","last_became_peered":"2024-09-19T08:01:10.781080+0000","last_unstale":"2024-09-19T08:03:00.997323+0000","last_undegraded":"2024-09-19T08:03:00.997323+0000","last_fullsized":"2024-09-19T08:03:00.997323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:40:45.225957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847701+0000","last_change":"2024-09-19T08:01:26.768102+0000","last_active":"2024-09-19T08:03:00.847701+0000","last_peered":"2024-09-19T08:03:00.847701+0000","last_clean":"2024-09-19T08:03:00.847701+0000","last_became_active":"2024-09-19T08:01:26.759462+0000","last_became_peered":"2024-09-19T08:01:26.759462+0000","last_unstale":"2024-09-19T08:03:00.847701+0000","last_undegraded":"2024-09-19T08:03:00.847701+0000","last_fullsized":"2024-09-19T08:03:00.847701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:02:06.815238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994921+0000","last_change":"2024-09-19T08:01:10.771031+0000","last_active":"2024-09-19T08:03:00.994921+0000","last_peered":"2024-09-19T08:03:00.994921+0000","last_clean":"2024-09-19T08:03:00.994921+0000","last_became_active":"2024-09-19T08:01:10.770675+0000","last_became_peered":"2024-09-19T08:01:10.770675+0000","last_unstale":"2024-09-19T08:03:00.994921+0000","last_undegraded":"2024-09-19T08:03:00.994921+0000","last_fullsized":"2024-09-19T08:03:00.994921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:42.203848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849256+0000","last_change":"2024-09-19T08:01:09.748622+0000","last_active":"2024-09-19T08:03:00.849256+0000","last_peered":"2024-09-19T08:03:00.849256+0000","last_clean":"2024-09-19T08:03:00.849256+0000","last_became_active":"2024-09-19T08:01:09.748202+0000","last_became_peered":"2024-09-19T08:01:09.748202+0000","last_unstale":"2024-09-19T08:03:00.849256+0000","last_undegraded":"2024-09-19T08:03:00.849256+0000","last_fullsized":"2024-09-19T08:03:00.849256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:17:39.861820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849123+0000","last_change":"2024-09-19T08:01:10.768949+0000","last_active":"2024-09-19T08:03:00.849123+0000","last_peered":"2024-09-19T08:03:00.849123+0000","last_clean":"2024-09-19T08:03:00.849123+0000","last_became_active":"2024-09-19T08:01:10.768159+0000","last_became_peered":"2024-09-19T08:01:10.768159+0000","last_unstale":"2024-09-19T08:03:00.849123+0000","last_undegraded":"2024-09-19T08:03:00.849123+0000","last_fullsized":"2024-09-19T08:03:00.849123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:43.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"81'14","reported_seq":176,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.854932+0000","last_change":"2024-09-19T08:01:26.735877+0000","last_active":"2024-09-19T08:03:10.854932+0000","last_peered":"2024-09-19T08:03:10.854932+0000","last_clean":"2024-09-19T08:03:10.854932+0000","last_became_active":"2024-09-19T08:01:26.723597+0000","last_became_peered":"2024-09-19T08:01:26.723597+0000","last_unstale":"2024-09-19T08:03:10.854932+0000","last_undegraded":"2024-09-19T08:03:10.854932+0000","last_fullsized":"2024-09-19T08:03:10.854932+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:43:42.340832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":110,"num_read_kb":110,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,1],"acting":[9,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":800997,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1499,"num_read_kb":974,"num_write":507,"num_write_kb":2386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1254,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2764800,"data_stored":2402991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645349,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":905216,"data_stored":436392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874282,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33192,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93711960,"statfs":{"total":95995035648,"available":95961047040,"internally_reserved":0,"allocated":1933312,"data_stored":1424044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005103,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32464,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712688,"statfs":{"total":95995035648,"available":95961792512,"internally_reserved":0,"allocated":1310720,"data_stored":827750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234035,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1294336,"data_stored":826874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462968,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1503232,"data_stored":1029567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593790,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36228,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708924,"statfs":{"total":95995035648,"available":95957938176,"internally_reserved":0,"allocated":901120,"data_stored":434644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822722,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28836,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716316,"statfs":{"total":95995035648,"available":95965507584,"internally_reserved":0,"allocated":1855488,"data_stored":1356970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986247,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":901120,"data_stored":434898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215179,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32908,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712244,"statfs":{"total":95995035648,"available":95961337856,"internally_reserved":0,"allocated":1642496,"data_stored":1155524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":507332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607638,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":978944,"data_stored":505622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":921600,"data_stored":436124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":72210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":73920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":753664,"data_stored":721878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":333073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":400147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T08:03:14.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-19T08:03:14.272 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:14.392 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:14 smithi098 ceph-mon[31333]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:14 smithi139 ceph-mon[34874]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:14.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:14 smithi162 ceph-mon[35260]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:14.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.712+0000 7fa6830de640 1 -- 172.21.15.98:0/4114247906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67c100a80 msgr2=0x7fa67c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:14.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.712+0000 7fa6830de640 1 --2- 172.21.15.98:0/4114247906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67c100a80 0x7fa67c100ee0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa668009930 tx=0x7fa66802a420 comp rx=0 tx=0).stop 2024-09-19T08:03:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.712+0000 7fa6830de640 1 -- 172.21.15.98:0/4114247906 shutdown_connections 2024-09-19T08:03:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.712+0000 7fa6830de640 1 --2- 172.21.15.98:0/4114247906 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa67c101420 0x7fa67c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.712+0000 7fa6830de640 1 --2- 172.21.15.98:0/4114247906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67c100a80 0x7fa67c100ee0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.712+0000 7fa6830de640 1 --2- 172.21.15.98:0/4114247906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa67c106a80 0x7fa67c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.712+0000 7fa6830de640 1 -- 172.21.15.98:0/4114247906 >> 172.21.15.98:0/4114247906 conn(0x7fa67c0fc7a0 msgr2=0x7fa67c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.712+0000 7fa6830de640 1 -- 172.21.15.98:0/4114247906 shutdown_connections 2024-09-19T08:03:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.712+0000 7fa6830de640 1 -- 172.21.15.98:0/4114247906 wait complete. 2024-09-19T08:03:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.713+0000 7fa6830de640 1 Processor -- start 2024-09-19T08:03:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.713+0000 7fa6830de640 1 -- start start 2024-09-19T08:03:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.713+0000 7fa6830de640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67c100a80 0x7fa67c10e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.713+0000 7fa6830de640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa67c101420 0x7fa67c10e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.713+0000 7fa6830de640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa67c106a80 0x7fa67c10ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.713+0000 7fa6830de640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c111fc0 con 0x7fa67c100a80 2024-09-19T08:03:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.713+0000 7fa6830de640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c111e40 con 0x7fa67c101420 2024-09-19T08:03:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.713+0000 7fa6830de640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c112140 con 0x7fa67c106a80 2024-09-19T08:03:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.713+0000 7fa680e7a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67c100a80 0x7fa67c10e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.713+0000 7fa680e7a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67c100a80 0x7fa67c10e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47506/0 (socket says 172.21.15.98:47506) 2024-09-19T08:03:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.713+0000 7fa680e7a640 1 -- 172.21.15.98:0/2773930910 learned_addr learned my addr 172.21.15.98:0/2773930910 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.713+0000 7fa68167b640 1 --2- 172.21.15.98:0/2773930910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa67c106a80 0x7fa67c10ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.714+0000 7fa67bfff640 1 --2- 172.21.15.98:0/2773930910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa67c101420 0x7fa67c10e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.714+0000 7fa680e7a640 1 -- 172.21.15.98:0/2773930910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa67c106a80 msgr2=0x7fa67c10ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.714+0000 7fa680e7a640 1 --2- 172.21.15.98:0/2773930910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa67c106a80 0x7fa67c10ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.714+0000 7fa680e7a640 1 -- 172.21.15.98:0/2773930910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa67c101420 msgr2=0x7fa67c10e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.714+0000 7fa680e7a640 1 --2- 172.21.15.98:0/2773930910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa67c101420 0x7fa67c10e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.714+0000 7fa680e7a640 1 -- 172.21.15.98:0/2773930910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67c1a69a0 con 0x7fa67c100a80 2024-09-19T08:03:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.714+0000 7fa680e7a640 1 --2- 172.21.15.98:0/2773930910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67c100a80 0x7fa67c10e3e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa67000d770 tx=0x7fa67000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:14.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.714+0000 7fa679ffb640 1 -- 172.21.15.98:0/2773930910 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa670017070 con 0x7fa67c100a80 2024-09-19T08:03:14.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.714+0000 7fa679ffb640 1 -- 172.21.15.98:0/2773930910 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa670005500 con 0x7fa67c100a80 2024-09-19T08:03:14.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.714+0000 7fa6830de640 1 -- 172.21.15.98:0/2773930910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa67c1a6c90 con 0x7fa67c100a80 2024-09-19T08:03:14.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.714+0000 7fa679ffb640 1 -- 172.21.15.98:0/2773930910 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa670004e00 con 0x7fa67c100a80 2024-09-19T08:03:14.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.714+0000 7fa6830de640 1 -- 172.21.15.98:0/2773930910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa67c1a7150 con 0x7fa67c100a80 2024-09-19T08:03:14.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.715+0000 7fa6830de640 1 -- 172.21.15.98:0/2773930910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa67c100ee0 con 0x7fa67c100a80 2024-09-19T08:03:14.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.716+0000 7fa679ffb640 1 -- 172.21.15.98:0/2773930910 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7fa670004fa0 con 0x7fa67c100a80 2024-09-19T08:03:14.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.717+0000 7fa679ffb640 1 --2- 172.21.15.98:0/2773930910 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa66c0a34d0 0x7fa66c0a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:14.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.717+0000 7fa67bfff640 1 --2- 172.21.15.98:0/2773930910 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa66c0a34d0 0x7fa66c0a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:14.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.717+0000 7fa679ffb640 1 -- 172.21.15.98:0/2773930910 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fa6700cfa20 con 0x7fa67c100a80 2024-09-19T08:03:14.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.717+0000 7fa67bfff640 1 --2- 172.21.15.98:0/2773930910 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa66c0a34d0 0x7fa66c0a5990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa668000c00 tx=0x7fa668005a20 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:14.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.722+0000 7fa679ffb640 1 -- 172.21.15.98:0/2773930910 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fa67009c1e0 con 0x7fa67c100a80 2024-09-19T08:03:14.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.889+0000 7fa6830de640 1 -- 172.21.15.98:0/2773930910 --> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa67c109080 con 0x7fa66c0a34d0 2024-09-19T08:03:14.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.895+0000 7fa679ffb640 1 -- 172.21.15.98:0/2773930910 <== mgr.15252 v2:172.21.15.98:6800/1961446727 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257817 (secure 0 0 0) 0x7fa6680315b0 con 0x7fa66c0a34d0 2024-09-19T08:03:14.899 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:14.899 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-19T08:03:14.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.898+0000 7fa6830de640 1 -- 172.21.15.98:0/2773930910 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa66c0a34d0 msgr2=0x7fa66c0a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:14.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.898+0000 7fa6830de640 1 --2- 172.21.15.98:0/2773930910 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa66c0a34d0 0x7fa66c0a5990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa668000c00 tx=0x7fa668005a20 comp rx=0 tx=0).stop 2024-09-19T08:03:14.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.898+0000 7fa6830de640 1 -- 172.21.15.98:0/2773930910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67c100a80 msgr2=0x7fa67c10e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:14.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.898+0000 7fa6830de640 1 --2- 172.21.15.98:0/2773930910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67c100a80 0x7fa67c10e3e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa67000d770 tx=0x7fa67000dc30 comp rx=0 tx=0).stop 2024-09-19T08:03:14.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.898+0000 7fa6830de640 1 -- 172.21.15.98:0/2773930910 shutdown_connections 2024-09-19T08:03:14.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.898+0000 7fa6830de640 1 --2- 172.21.15.98:0/2773930910 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa66c0a34d0 0x7fa66c0a5990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.898+0000 7fa6830de640 1 --2- 172.21.15.98:0/2773930910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa67c106a80 0x7fa67c10ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.898+0000 7fa6830de640 1 --2- 172.21.15.98:0/2773930910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa67c101420 0x7fa67c10e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.898+0000 7fa6830de640 1 --2- 172.21.15.98:0/2773930910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67c100a80 0x7fa67c10e3e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.898+0000 7fa6830de640 1 -- 172.21.15.98:0/2773930910 >> 172.21.15.98:0/2773930910 conn(0x7fa67c0fc7a0 msgr2=0x7fa67c0fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.898+0000 7fa6830de640 1 -- 172.21.15.98:0/2773930910 shutdown_connections 2024-09-19T08:03:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:14.898+0000 7fa6830de640 1 -- 172.21.15.98:0/2773930910 wait complete. 2024-09-19T08:03:15.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:15 smithi098 ceph-mon[31333]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-09-19T08:03:15.613 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-09-19T08:03:14.773970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1391599,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1736,"num_read_kb":1304,"num_write":710,"num_write_kb":5326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181990,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1036,"ondisk_log_size":1036,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376776,"kb_used_data":14784,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565048,"statfs":{"total":1151940427776,"available":1151554609152,"internally_reserved":0,"allocated":15138816,"data_stored":9375741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":370193724},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":200331,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":461,"num_read_kb":502,"num_write":155,"num_write_kb":923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002625"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780911+0000","last_change":"2024-09-19T08:01:10.768950+0000","last_active":"2024-09-19T08:03:00.780911+0000","last_peered":"2024-09-19T08:03:00.780911+0000","last_clean":"2024-09-19T08:03:00.780911+0000","last_became_active":"2024-09-19T08:01:10.768047+0000","last_became_peered":"2024-09-19T08:01:10.768047+0000","last_unstale":"2024-09-19T08:03:00.780911+0000","last_undegraded":"2024-09-19T08:03:00.780911+0000","last_fullsized":"2024-09-19T08:03:00.780911+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:26.219079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994728+0000","last_change":"2024-09-19T08:01:10.783208+0000","last_active":"2024-09-19T08:03:00.994728+0000","last_peered":"2024-09-19T08:03:00.994728+0000","last_clean":"2024-09-19T08:03:00.994728+0000","last_became_active":"2024-09-19T08:01:10.782888+0000","last_became_peered":"2024-09-19T08:01:10.782888+0000","last_unstale":"2024-09-19T08:03:00.994728+0000","last_undegraded":"2024-09-19T08:03:00.994728+0000","last_fullsized":"2024-09-19T08:03:00.994728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:51:27.603166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994427+0000","last_change":"2024-09-19T08:01:10.770395+0000","last_active":"2024-09-19T08:03:00.994427+0000","last_peered":"2024-09-19T08:03:00.994427+0000","last_clean":"2024-09-19T08:03:00.994427+0000","last_became_active":"2024-09-19T08:01:10.770073+0000","last_became_peered":"2024-09-19T08:01:10.770073+0000","last_unstale":"2024-09-19T08:03:00.994427+0000","last_undegraded":"2024-09-19T08:03:00.994427+0000","last_fullsized":"2024-09-19T08:03:00.994427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:20.047023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761433+0000","last_change":"2024-09-19T08:01:10.777138+0000","last_active":"2024-09-19T08:03:00.761433+0000","last_peered":"2024-09-19T08:03:00.761433+0000","last_clean":"2024-09-19T08:03:00.761433+0000","last_became_active":"2024-09-19T08:01:10.776810+0000","last_became_peered":"2024-09-19T08:01:10.776810+0000","last_unstale":"2024-09-19T08:03:00.761433+0000","last_undegraded":"2024-09-19T08:03:00.761433+0000","last_fullsized":"2024-09-19T08:03:00.761433+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:39.983321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847497+0000","last_change":"2024-09-19T08:01:10.770435+0000","last_active":"2024-09-19T08:03:00.847497+0000","last_peered":"2024-09-19T08:03:00.847497+0000","last_clean":"2024-09-19T08:03:00.847497+0000","last_became_active":"2024-09-19T08:01:10.769686+0000","last_became_peered":"2024-09-19T08:01:10.769686+0000","last_unstale":"2024-09-19T08:03:00.847497+0000","last_undegraded":"2024-09-19T08:03:00.847497+0000","last_fullsized":"2024-09-19T08:03:00.847497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:10:19.610925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839257+0000","last_change":"2024-09-19T08:01:10.782382+0000","last_active":"2024-09-19T08:03:00.839257+0000","last_peered":"2024-09-19T08:03:00.839257+0000","last_clean":"2024-09-19T08:03:00.839257+0000","last_became_active":"2024-09-19T08:01:10.781996+0000","last_became_peered":"2024-09-19T08:01:10.781996+0000","last_unstale":"2024-09-19T08:03:00.839257+0000","last_undegraded":"2024-09-19T08:03:00.839257+0000","last_fullsized":"2024-09-19T08:03:00.839257+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:43:13.574336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837718+0000","last_change":"2024-09-19T08:01:10.773776+0000","last_active":"2024-09-19T08:03:00.837718+0000","last_peered":"2024-09-19T08:03:00.837718+0000","last_clean":"2024-09-19T08:03:00.837718+0000","last_became_active":"2024-09-19T08:01:10.772442+0000","last_became_peered":"2024-09-19T08:01:10.772442+0000","last_unstale":"2024-09-19T08:03:00.837718+0000","last_undegraded":"2024-09-19T08:03:00.837718+0000","last_fullsized":"2024-09-19T08:03:00.837718+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:08:18.155806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848134+0000","last_change":"2024-09-19T08:01:10.781443+0000","last_active":"2024-09-19T08:03:00.848134+0000","last_peered":"2024-09-19T08:03:00.848134+0000","last_clean":"2024-09-19T08:03:00.848134+0000","last_became_active":"2024-09-19T08:01:10.781083+0000","last_became_peered":"2024-09-19T08:01:10.781083+0000","last_unstale":"2024-09-19T08:03:00.848134+0000","last_undegraded":"2024-09-19T08:03:00.848134+0000","last_fullsized":"2024-09-19T08:03:00.848134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:23:22.631865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.998365+0000","last_change":"2024-09-19T08:01:10.767554+0000","last_active":"2024-09-19T08:03:00.998365+0000","last_peered":"2024-09-19T08:03:00.998365+0000","last_clean":"2024-09-19T08:03:00.998365+0000","last_became_active":"2024-09-19T08:01:10.767227+0000","last_became_peered":"2024-09-19T08:01:10.767227+0000","last_unstale":"2024-09-19T08:03:00.998365+0000","last_undegraded":"2024-09-19T08:03:00.998365+0000","last_fullsized":"2024-09-19T08:03:00.998365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:30:13.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839361+0000","last_change":"2024-09-19T08:01:10.783174+0000","last_active":"2024-09-19T08:03:00.839361+0000","last_peered":"2024-09-19T08:03:00.839361+0000","last_clean":"2024-09-19T08:03:00.839361+0000","last_became_active":"2024-09-19T08:01:10.782794+0000","last_became_peered":"2024-09-19T08:01:10.782794+0000","last_unstale":"2024-09-19T08:03:00.839361+0000","last_undegraded":"2024-09-19T08:03:00.839361+0000","last_fullsized":"2024-09-19T08:03:00.839361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:49:19.451107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838191+0000","last_change":"2024-09-19T08:01:10.779638+0000","last_active":"2024-09-19T08:03:00.838191+0000","last_peered":"2024-09-19T08:03:00.838191+0000","last_clean":"2024-09-19T08:03:00.838191+0000","last_became_active":"2024-09-19T08:01:10.779295+0000","last_became_peered":"2024-09-19T08:01:10.779295+0000","last_unstale":"2024-09-19T08:03:00.838191+0000","last_undegraded":"2024-09-19T08:03:00.838191+0000","last_fullsized":"2024-09-19T08:03:00.838191+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:35:10.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761539+0000","last_change":"2024-09-19T08:01:10.784487+0000","last_active":"2024-09-19T08:03:00.761539+0000","last_peered":"2024-09-19T08:03:00.761539+0000","last_clean":"2024-09-19T08:03:00.761539+0000","last_became_active":"2024-09-19T08:01:10.784114+0000","last_became_peered":"2024-09-19T08:01:10.784114+0000","last_unstale":"2024-09-19T08:03:00.761539+0000","last_undegraded":"2024-09-19T08:03:00.761539+0000","last_fullsized":"2024-09-19T08:03:00.761539+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:09:54.819802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758888+0000","last_change":"2024-09-19T08:01:10.783186+0000","last_active":"2024-09-19T08:03:00.758888+0000","last_peered":"2024-09-19T08:03:00.758888+0000","last_clean":"2024-09-19T08:03:00.758888+0000","last_became_active":"2024-09-19T08:01:10.782876+0000","last_became_peered":"2024-09-19T08:01:10.782876+0000","last_unstale":"2024-09-19T08:03:00.758888+0000","last_undegraded":"2024-09-19T08:03:00.758888+0000","last_fullsized":"2024-09-19T08:03:00.758888+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:26:15.252165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994474+0000","last_change":"2024-09-19T08:01:10.783237+0000","last_active":"2024-09-19T08:03:00.994474+0000","last_peered":"2024-09-19T08:03:00.994474+0000","last_clean":"2024-09-19T08:03:00.994474+0000","last_became_active":"2024-09-19T08:01:10.782966+0000","last_became_peered":"2024-09-19T08:01:10.782966+0000","last_unstale":"2024-09-19T08:03:00.994474+0000","last_undegraded":"2024-09-19T08:03:00.994474+0000","last_fullsized":"2024-09-19T08:03:00.994474+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:57:18.771737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994242+0000","last_change":"2024-09-19T08:01:10.771207+0000","last_active":"2024-09-19T08:03:00.994242+0000","last_peered":"2024-09-19T08:03:00.994242+0000","last_clean":"2024-09-19T08:03:00.994242+0000","last_became_active":"2024-09-19T08:01:10.770862+0000","last_became_peered":"2024-09-19T08:01:10.770862+0000","last_unstale":"2024-09-19T08:03:00.994242+0000","last_undegraded":"2024-09-19T08:03:00.994242+0000","last_fullsized":"2024-09-19T08:03:00.994242+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:08:09.313329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997263+0000","last_change":"2024-09-19T08:01:10.775191+0000","last_active":"2024-09-19T08:03:00.997263+0000","last_peered":"2024-09-19T08:03:00.997263+0000","last_clean":"2024-09-19T08:03:00.997263+0000","last_became_active":"2024-09-19T08:01:10.774832+0000","last_became_peered":"2024-09-19T08:01:10.774832+0000","last_unstale":"2024-09-19T08:03:00.997263+0000","last_undegraded":"2024-09-19T08:03:00.997263+0000","last_fullsized":"2024-09-19T08:03:00.997263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:57:29.800787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780968+0000","last_change":"2024-09-19T08:01:10.769115+0000","last_active":"2024-09-19T08:03:00.780968+0000","last_peered":"2024-09-19T08:03:00.780968+0000","last_clean":"2024-09-19T08:03:00.780968+0000","last_became_active":"2024-09-19T08:01:10.768088+0000","last_became_peered":"2024-09-19T08:01:10.768088+0000","last_unstale":"2024-09-19T08:03:00.780968+0000","last_undegraded":"2024-09-19T08:03:00.780968+0000","last_fullsized":"2024-09-19T08:03:00.780968+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:27:16.042722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844124+0000","last_change":"2024-09-19T08:01:10.781426+0000","last_active":"2024-09-19T08:03:00.844124+0000","last_peered":"2024-09-19T08:03:00.844124+0000","last_clean":"2024-09-19T08:03:00.844124+0000","last_became_active":"2024-09-19T08:01:10.781094+0000","last_became_peered":"2024-09-19T08:01:10.781094+0000","last_unstale":"2024-09-19T08:03:00.844124+0000","last_undegraded":"2024-09-19T08:03:00.844124+0000","last_fullsized":"2024-09-19T08:03:00.844124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:43:31.336610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760020+0000","last_change":"2024-09-19T08:01:10.766466+0000","last_active":"2024-09-19T08:03:00.760020+0000","last_peered":"2024-09-19T08:03:00.760020+0000","last_clean":"2024-09-19T08:03:00.760020+0000","last_became_active":"2024-09-19T08:01:10.766123+0000","last_became_peered":"2024-09-19T08:01:10.766123+0000","last_unstale":"2024-09-19T08:03:00.760020+0000","last_undegraded":"2024-09-19T08:03:00.760020+0000","last_fullsized":"2024-09-19T08:03:00.760020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:25.848603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780633+0000","last_change":"2024-09-19T08:01:10.772782+0000","last_active":"2024-09-19T08:03:00.780633+0000","last_peered":"2024-09-19T08:03:00.780633+0000","last_clean":"2024-09-19T08:03:00.780633+0000","last_became_active":"2024-09-19T08:01:10.772445+0000","last_became_peered":"2024-09-19T08:01:10.772445+0000","last_unstale":"2024-09-19T08:03:00.780633+0000","last_undegraded":"2024-09-19T08:03:00.780633+0000","last_fullsized":"2024-09-19T08:03:00.780633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:09.174005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759047+0000","last_change":"2024-09-19T08:01:10.771944+0000","last_active":"2024-09-19T08:03:00.759047+0000","last_peered":"2024-09-19T08:03:00.759047+0000","last_clean":"2024-09-19T08:03:00.759047+0000","last_became_active":"2024-09-19T08:01:10.771643+0000","last_became_peered":"2024-09-19T08:01:10.771643+0000","last_unstale":"2024-09-19T08:03:00.759047+0000","last_undegraded":"2024-09-19T08:03:00.759047+0000","last_fullsized":"2024-09-19T08:03:00.759047+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:46:27.121722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997451+0000","last_change":"2024-09-19T08:01:10.775144+0000","last_active":"2024-09-19T08:03:00.997451+0000","last_peered":"2024-09-19T08:03:00.997451+0000","last_clean":"2024-09-19T08:03:00.997451+0000","last_became_active":"2024-09-19T08:01:10.774769+0000","last_became_peered":"2024-09-19T08:01:10.774769+0000","last_unstale":"2024-09-19T08:03:00.997451+0000","last_undegraded":"2024-09-19T08:03:00.997451+0000","last_fullsized":"2024-09-19T08:03:00.997451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:59:47.053993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848212+0000","last_change":"2024-09-19T08:01:10.766750+0000","last_active":"2024-09-19T08:03:00.848212+0000","last_peered":"2024-09-19T08:03:00.848212+0000","last_clean":"2024-09-19T08:03:00.848212+0000","last_became_active":"2024-09-19T08:01:10.766450+0000","last_became_peered":"2024-09-19T08:01:10.766450+0000","last_unstale":"2024-09-19T08:03:00.848212+0000","last_undegraded":"2024-09-19T08:03:00.848212+0000","last_fullsized":"2024-09-19T08:03:00.848212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:47:58.133815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841186+0000","last_change":"2024-09-19T08:01:10.768430+0000","last_active":"2024-09-19T08:03:00.841186+0000","last_peered":"2024-09-19T08:03:00.841186+0000","last_clean":"2024-09-19T08:03:00.841186+0000","last_became_active":"2024-09-19T08:01:10.768092+0000","last_became_peered":"2024-09-19T08:01:10.768092+0000","last_unstale":"2024-09-19T08:03:00.841186+0000","last_undegraded":"2024-09-19T08:03:00.841186+0000","last_fullsized":"2024-09-19T08:03:00.841186+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:16:42.343138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849206+0000","last_change":"2024-09-19T08:01:10.771131+0000","last_active":"2024-09-19T08:03:00.849206+0000","last_peered":"2024-09-19T08:03:00.849206+0000","last_clean":"2024-09-19T08:03:00.849206+0000","last_became_active":"2024-09-19T08:01:10.770851+0000","last_became_peered":"2024-09-19T08:01:10.770851+0000","last_unstale":"2024-09-19T08:03:00.849206+0000","last_undegraded":"2024-09-19T08:03:00.849206+0000","last_fullsized":"2024-09-19T08:03:00.849206+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:07:15.830476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760273+0000","last_change":"2024-09-19T08:01:10.776220+0000","last_active":"2024-09-19T08:03:00.760273+0000","last_peered":"2024-09-19T08:03:00.760273+0000","last_clean":"2024-09-19T08:03:00.760273+0000","last_became_active":"2024-09-19T08:01:10.775889+0000","last_became_peered":"2024-09-19T08:01:10.775889+0000","last_unstale":"2024-09-19T08:03:00.760273+0000","last_undegraded":"2024-09-19T08:03:00.760273+0000","last_fullsized":"2024-09-19T08:03:00.760273+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:23.714675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780731+0000","last_change":"2024-09-19T08:01:10.776370+0000","last_active":"2024-09-19T08:03:00.780731+0000","last_peered":"2024-09-19T08:03:00.780731+0000","last_clean":"2024-09-19T08:03:00.780731+0000","last_became_active":"2024-09-19T08:01:10.776027+0000","last_became_peered":"2024-09-19T08:01:10.776027+0000","last_unstale":"2024-09-19T08:03:00.780731+0000","last_undegraded":"2024-09-19T08:03:00.780731+0000","last_fullsized":"2024-09-19T08:03:00.780731+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:48:46.293404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758951+0000","last_change":"2024-09-19T08:01:26.758971+0000","last_active":"2024-09-19T08:03:00.758951+0000","last_peered":"2024-09-19T08:03:00.758951+0000","last_clean":"2024-09-19T08:03:00.758951+0000","last_became_active":"2024-09-19T08:01:26.758524+0000","last_became_peered":"2024-09-19T08:01:26.758524+0000","last_unstale":"2024-09-19T08:03:00.758951+0000","last_undegraded":"2024-09-19T08:03:00.758951+0000","last_fullsized":"2024-09-19T08:03:00.758951+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:34:22.439327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994656+0000","last_change":"2024-09-19T08:01:10.767445+0000","last_active":"2024-09-19T08:03:00.994656+0000","last_peered":"2024-09-19T08:03:00.994656+0000","last_clean":"2024-09-19T08:03:00.994656+0000","last_became_active":"2024-09-19T08:01:10.767128+0000","last_became_peered":"2024-09-19T08:01:10.767128+0000","last_unstale":"2024-09-19T08:03:00.994656+0000","last_undegraded":"2024-09-19T08:03:00.994656+0000","last_fullsized":"2024-09-19T08:03:00.994656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:10.273948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759852+0000","last_change":"2024-09-19T08:01:10.768954+0000","last_active":"2024-09-19T08:03:00.759852+0000","last_peered":"2024-09-19T08:03:00.759852+0000","last_clean":"2024-09-19T08:03:00.759852+0000","last_became_active":"2024-09-19T08:01:10.768577+0000","last_became_peered":"2024-09-19T08:01:10.768577+0000","last_unstale":"2024-09-19T08:03:00.759852+0000","last_undegraded":"2024-09-19T08:03:00.759852+0000","last_fullsized":"2024-09-19T08:03:00.759852+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:20:21.812531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759843+0000","last_change":"2024-09-19T08:01:10.774104+0000","last_active":"2024-09-19T08:03:00.759843+0000","last_peered":"2024-09-19T08:03:00.759843+0000","last_clean":"2024-09-19T08:03:00.759843+0000","last_became_active":"2024-09-19T08:01:10.773721+0000","last_became_peered":"2024-09-19T08:01:10.773721+0000","last_unstale":"2024-09-19T08:03:00.759843+0000","last_undegraded":"2024-09-19T08:03:00.759843+0000","last_fullsized":"2024-09-19T08:03:00.759843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:45:34.945994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780513+0000","last_change":"2024-09-19T08:01:26.727296+0000","last_active":"2024-09-19T08:03:00.780513+0000","last_peered":"2024-09-19T08:03:00.780513+0000","last_clean":"2024-09-19T08:03:00.780513+0000","last_became_active":"2024-09-19T08:01:26.726631+0000","last_became_peered":"2024-09-19T08:01:26.726631+0000","last_unstale":"2024-09-19T08:03:00.780513+0000","last_undegraded":"2024-09-19T08:03:00.780513+0000","last_fullsized":"2024-09-19T08:03:00.780513+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:19:24.807050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847424+0000","last_change":"2024-09-19T08:01:10.765282+0000","last_active":"2024-09-19T08:03:00.847424+0000","last_peered":"2024-09-19T08:03:00.847424+0000","last_clean":"2024-09-19T08:03:00.847424+0000","last_became_active":"2024-09-19T08:01:10.764928+0000","last_became_peered":"2024-09-19T08:01:10.764928+0000","last_unstale":"2024-09-19T08:03:00.847424+0000","last_undegraded":"2024-09-19T08:03:00.847424+0000","last_fullsized":"2024-09-19T08:03:00.847424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:00:37.395110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"81'11","reported_seq":152,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:09.854344+0000","last_change":"2024-09-19T08:01:08.760686+0000","last_active":"2024-09-19T08:03:09.854344+0000","last_peered":"2024-09-19T08:03:09.854344+0000","last_clean":"2024-09-19T08:03:09.854344+0000","last_became_active":"2024-09-19T08:01:08.759912+0000","last_became_peered":"2024-09-19T08:01:08.759912+0000","last_unstale":"2024-09-19T08:03:09.854344+0000","last_undegraded":"2024-09-19T08:03:09.854344+0000","last_fullsized":"2024-09-19T08:03:09.854344+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:46:01.879611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2643,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":7,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841395+0000","last_change":"2024-09-19T08:01:10.779668+0000","last_active":"2024-09-19T08:03:00.841395+0000","last_peered":"2024-09-19T08:03:00.841395+0000","last_clean":"2024-09-19T08:03:00.841395+0000","last_became_active":"2024-09-19T08:01:10.779265+0000","last_became_peered":"2024-09-19T08:01:10.779265+0000","last_unstale":"2024-09-19T08:03:00.841395+0000","last_undegraded":"2024-09-19T08:03:00.841395+0000","last_fullsized":"2024-09-19T08:03:00.841395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:06.177700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838967+0000","last_change":"2024-09-19T08:01:08.762668+0000","last_active":"2024-09-19T08:03:00.838967+0000","last_peered":"2024-09-19T08:03:00.838967+0000","last_clean":"2024-09-19T08:03:00.838967+0000","last_became_active":"2024-09-19T08:01:08.762277+0000","last_became_peered":"2024-09-19T08:01:08.762277+0000","last_unstale":"2024-09-19T08:03:00.838967+0000","last_undegraded":"2024-09-19T08:03:00.838967+0000","last_fullsized":"2024-09-19T08:03:00.838967+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:36:59.316655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758585+0000","last_change":"2024-09-19T08:01:08.757401+0000","last_active":"2024-09-19T08:03:00.758585+0000","last_peered":"2024-09-19T08:03:00.758585+0000","last_clean":"2024-09-19T08:03:00.758585+0000","last_became_active":"2024-09-19T08:01:08.757067+0000","last_became_peered":"2024-09-19T08:01:08.757067+0000","last_unstale":"2024-09-19T08:03:00.758585+0000","last_undegraded":"2024-09-19T08:03:00.758585+0000","last_fullsized":"2024-09-19T08:03:00.758585+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:19.617830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848289+0000","last_change":"2024-09-19T08:01:10.765149+0000","last_active":"2024-09-19T08:03:00.848289+0000","last_peered":"2024-09-19T08:03:00.848289+0000","last_clean":"2024-09-19T08:03:00.848289+0000","last_became_active":"2024-09-19T08:01:10.764840+0000","last_became_peered":"2024-09-19T08:01:10.764840+0000","last_unstale":"2024-09-19T08:03:00.848289+0000","last_undegraded":"2024-09-19T08:03:00.848289+0000","last_fullsized":"2024-09-19T08:03:00.848289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:15.539307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841355+0000","last_change":"2024-09-19T08:01:08.759435+0000","last_active":"2024-09-19T08:03:00.841355+0000","last_peered":"2024-09-19T08:03:00.841355+0000","last_clean":"2024-09-19T08:03:00.841355+0000","last_became_active":"2024-09-19T08:01:08.759046+0000","last_became_peered":"2024-09-19T08:01:08.759046+0000","last_unstale":"2024-09-19T08:03:00.841355+0000","last_undegraded":"2024-09-19T08:03:00.841355+0000","last_fullsized":"2024-09-19T08:03:00.841355+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:21:24.077626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.843896+0000","last_change":"2024-09-19T08:01:10.778432+0000","last_active":"2024-09-19T08:03:00.843896+0000","last_peered":"2024-09-19T08:03:00.843896+0000","last_clean":"2024-09-19T08:03:00.843896+0000","last_became_active":"2024-09-19T08:01:10.778098+0000","last_became_peered":"2024-09-19T08:01:10.778098+0000","last_unstale":"2024-09-19T08:03:00.843896+0000","last_undegraded":"2024-09-19T08:03:00.843896+0000","last_fullsized":"2024-09-19T08:03:00.843896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:40:12.921165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":250,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.855304+0000","last_change":"2024-09-19T08:01:08.764117+0000","last_active":"2024-09-19T08:03:10.855304+0000","last_peered":"2024-09-19T08:03:10.855304+0000","last_clean":"2024-09-19T08:03:10.855304+0000","last_became_active":"2024-09-19T08:01:08.763308+0000","last_became_peered":"2024-09-19T08:01:08.763308+0000","last_unstale":"2024-09-19T08:03:10.855304+0000","last_undegraded":"2024-09-19T08:03:10.855304+0000","last_fullsized":"2024-09-19T08:03:10.855304+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:19:57.014628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":220,"num_read_kb":220,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997921+0000","last_change":"2024-09-19T08:01:10.781771+0000","last_active":"2024-09-19T08:03:00.997921+0000","last_peered":"2024-09-19T08:03:00.997921+0000","last_clean":"2024-09-19T08:03:00.997921+0000","last_became_active":"2024-09-19T08:01:10.781432+0000","last_became_peered":"2024-09-19T08:01:10.781432+0000","last_unstale":"2024-09-19T08:03:00.997921+0000","last_undegraded":"2024-09-19T08:03:00.997921+0000","last_fullsized":"2024-09-19T08:03:00.997921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:25:15.067850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838023+0000","last_change":"2024-09-19T08:01:26.771171+0000","last_active":"2024-09-19T08:03:00.838023+0000","last_peered":"2024-09-19T08:03:00.838023+0000","last_clean":"2024-09-19T08:03:00.838023+0000","last_became_active":"2024-09-19T08:01:26.759274+0000","last_became_peered":"2024-09-19T08:01:26.759274+0000","last_unstale":"2024-09-19T08:03:00.838023+0000","last_undegraded":"2024-09-19T08:03:00.838023+0000","last_fullsized":"2024-09-19T08:03:00.838023+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:20:54.603420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994926+0000","last_change":"2024-09-19T08:01:10.768318+0000","last_active":"2024-09-19T08:03:00.994926+0000","last_peered":"2024-09-19T08:03:00.994926+0000","last_clean":"2024-09-19T08:03:00.994926+0000","last_became_active":"2024-09-19T08:01:10.768022+0000","last_became_peered":"2024-09-19T08:01:10.768022+0000","last_unstale":"2024-09-19T08:03:00.994926+0000","last_undegraded":"2024-09-19T08:03:00.994926+0000","last_fullsized":"2024-09-19T08:03:00.994926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:00:45.328638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841594+0000","last_change":"2024-09-19T08:01:09.750530+0000","last_active":"2024-09-19T08:03:00.841594+0000","last_peered":"2024-09-19T08:03:00.841594+0000","last_clean":"2024-09-19T08:03:00.841594+0000","last_became_active":"2024-09-19T08:01:09.750029+0000","last_became_peered":"2024-09-19T08:01:09.750029+0000","last_unstale":"2024-09-19T08:03:00.841594+0000","last_undegraded":"2024-09-19T08:03:00.841594+0000","last_fullsized":"2024-09-19T08:03:00.841594+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:01:16.062992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997659+0000","last_change":"2024-09-19T08:01:26.724636+0000","last_active":"2024-09-19T08:03:00.997659+0000","last_peered":"2024-09-19T08:03:00.997659+0000","last_clean":"2024-09-19T08:03:00.997659+0000","last_became_active":"2024-09-19T08:01:26.724171+0000","last_became_peered":"2024-09-19T08:01:26.724171+0000","last_unstale":"2024-09-19T08:03:00.997659+0000","last_undegraded":"2024-09-19T08:03:00.997659+0000","last_fullsized":"2024-09-19T08:03:00.997659+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:54:05.998374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'39","reported_seq":300,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.855298+0000","last_change":"2024-09-19T08:01:08.761950+0000","last_active":"2024-09-19T08:03:10.855298+0000","last_peered":"2024-09-19T08:03:10.855298+0000","last_clean":"2024-09-19T08:03:10.855298+0000","last_became_active":"2024-09-19T08:01:08.761533+0000","last_became_peered":"2024-09-19T08:01:08.761533+0000","last_unstale":"2024-09-19T08:03:10.855298+0000","last_undegraded":"2024-09-19T08:03:10.855298+0000","last_fullsized":"2024-09-19T08:03:10.855298+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:07:26.668141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69393,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":237,"num_read_kb":187,"num_write":41,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847631+0000","last_change":"2024-09-19T08:01:10.774280+0000","last_active":"2024-09-19T08:03:00.847631+0000","last_peered":"2024-09-19T08:03:00.847631+0000","last_clean":"2024-09-19T08:03:00.847631+0000","last_became_active":"2024-09-19T08:01:10.773982+0000","last_became_peered":"2024-09-19T08:01:10.773982+0000","last_unstale":"2024-09-19T08:03:00.847631+0000","last_undegraded":"2024-09-19T08:03:00.847631+0000","last_fullsized":"2024-09-19T08:03:00.847631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:21:43.180057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760058+0000","last_change":"2024-09-19T08:01:08.761067+0000","last_active":"2024-09-19T08:03:00.760058+0000","last_peered":"2024-09-19T08:03:00.760058+0000","last_clean":"2024-09-19T08:03:00.760058+0000","last_became_active":"2024-09-19T08:01:08.760664+0000","last_became_peered":"2024-09-19T08:01:08.760664+0000","last_unstale":"2024-09-19T08:03:00.760058+0000","last_undegraded":"2024-09-19T08:03:00.760058+0000","last_fullsized":"2024-09-19T08:03:00.760058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:22:38.086891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758630+0000","last_change":"2024-09-19T08:01:10.771845+0000","last_active":"2024-09-19T08:03:00.758630+0000","last_peered":"2024-09-19T08:03:00.758630+0000","last_clean":"2024-09-19T08:03:00.758630+0000","last_became_active":"2024-09-19T08:01:10.771533+0000","last_became_peered":"2024-09-19T08:01:10.771533+0000","last_unstale":"2024-09-19T08:03:00.758630+0000","last_undegraded":"2024-09-19T08:03:00.758630+0000","last_fullsized":"2024-09-19T08:03:00.758630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:29:35.180763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841267+0000","last_change":"2024-09-19T08:01:08.763869+0000","last_active":"2024-09-19T08:03:00.841267+0000","last_peered":"2024-09-19T08:03:00.841267+0000","last_clean":"2024-09-19T08:03:00.841267+0000","last_became_active":"2024-09-19T08:01:08.763451+0000","last_became_peered":"2024-09-19T08:01:08.763451+0000","last_unstale":"2024-09-19T08:03:00.841267+0000","last_undegraded":"2024-09-19T08:03:00.841267+0000","last_fullsized":"2024-09-19T08:03:00.841267+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:32:40.901049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838028+0000","last_change":"2024-09-19T08:01:10.770823+0000","last_active":"2024-09-19T08:03:00.838028+0000","last_peered":"2024-09-19T08:03:00.838028+0000","last_clean":"2024-09-19T08:03:00.838028+0000","last_became_active":"2024-09-19T08:01:10.770462+0000","last_became_peered":"2024-09-19T08:01:10.770462+0000","last_unstale":"2024-09-19T08:03:00.838028+0000","last_undegraded":"2024-09-19T08:03:00.838028+0000","last_fullsized":"2024-09-19T08:03:00.838028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:04:32.832009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'7","reported_seq":145,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:09.854029+0000","last_change":"2024-09-19T08:01:08.759429+0000","last_active":"2024-09-19T08:03:09.854029+0000","last_peered":"2024-09-19T08:03:09.854029+0000","last_clean":"2024-09-19T08:03:09.854029+0000","last_became_active":"2024-09-19T08:01:08.759062+0000","last_became_peered":"2024-09-19T08:01:08.759062+0000","last_unstale":"2024-09-19T08:03:09.854029+0000","last_undegraded":"2024-09-19T08:03:09.854029+0000","last_fullsized":"2024-09-19T08:03:09.854029+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:16:51.582435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1374,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837660+0000","last_change":"2024-09-19T08:01:10.784735+0000","last_active":"2024-09-19T08:03:00.837660+0000","last_peered":"2024-09-19T08:03:00.837660+0000","last_clean":"2024-09-19T08:03:00.837660+0000","last_became_active":"2024-09-19T08:01:10.784346+0000","last_became_peered":"2024-09-19T08:01:10.784346+0000","last_unstale":"2024-09-19T08:03:00.837660+0000","last_undegraded":"2024-09-19T08:03:00.837660+0000","last_fullsized":"2024-09-19T08:03:00.837660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:27:54.580391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847399+0000","last_change":"2024-09-19T08:01:09.749498+0000","last_active":"2024-09-19T08:03:00.847399+0000","last_peered":"2024-09-19T08:03:00.847399+0000","last_clean":"2024-09-19T08:03:00.847399+0000","last_became_active":"2024-09-19T08:01:09.749091+0000","last_became_peered":"2024-09-19T08:01:09.749091+0000","last_unstale":"2024-09-19T08:03:00.847399+0000","last_undegraded":"2024-09-19T08:03:00.847399+0000","last_fullsized":"2024-09-19T08:03:00.847399+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:56:13.644931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841217+0000","last_change":"2024-09-19T08:01:10.779599+0000","last_active":"2024-09-19T08:03:00.841217+0000","last_peered":"2024-09-19T08:03:00.841217+0000","last_clean":"2024-09-19T08:03:00.841217+0000","last_became_active":"2024-09-19T08:01:10.779219+0000","last_became_peered":"2024-09-19T08:01:10.779219+0000","last_unstale":"2024-09-19T08:03:00.841217+0000","last_undegraded":"2024-09-19T08:03:00.841217+0000","last_fullsized":"2024-09-19T08:03:00.841217+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:20.193753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847308+0000","last_change":"2024-09-19T08:01:08.761432+0000","last_active":"2024-09-19T08:03:00.847308+0000","last_peered":"2024-09-19T08:03:00.847308+0000","last_clean":"2024-09-19T08:03:00.847308+0000","last_became_active":"2024-09-19T08:01:08.761044+0000","last_became_peered":"2024-09-19T08:01:08.761044+0000","last_unstale":"2024-09-19T08:03:00.847308+0000","last_undegraded":"2024-09-19T08:03:00.847308+0000","last_fullsized":"2024-09-19T08:03:00.847308+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:12.237334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780765+0000","last_change":"2024-09-19T08:01:10.772559+0000","last_active":"2024-09-19T08:03:00.780765+0000","last_peered":"2024-09-19T08:03:00.780765+0000","last_clean":"2024-09-19T08:03:00.780765+0000","last_became_active":"2024-09-19T08:01:10.772176+0000","last_became_peered":"2024-09-19T08:01:10.772176+0000","last_unstale":"2024-09-19T08:03:00.780765+0000","last_undegraded":"2024-09-19T08:03:00.780765+0000","last_fullsized":"2024-09-19T08:03:00.780765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:27:16.296127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758996+0000","last_change":"2024-09-19T08:01:08.759989+0000","last_active":"2024-09-19T08:03:00.758996+0000","last_peered":"2024-09-19T08:03:00.758996+0000","last_clean":"2024-09-19T08:03:00.758996+0000","last_became_active":"2024-09-19T08:01:08.759705+0000","last_became_peered":"2024-09-19T08:01:08.759705+0000","last_unstale":"2024-09-19T08:03:00.758996+0000","last_undegraded":"2024-09-19T08:03:00.758996+0000","last_fullsized":"2024-09-19T08:03:00.758996+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:33:25.237567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838150+0000","last_change":"2024-09-19T08:01:10.776528+0000","last_active":"2024-09-19T08:03:00.838150+0000","last_peered":"2024-09-19T08:03:00.838150+0000","last_clean":"2024-09-19T08:03:00.838150+0000","last_became_active":"2024-09-19T08:01:10.776222+0000","last_became_peered":"2024-09-19T08:01:10.776222+0000","last_unstale":"2024-09-19T08:03:00.838150+0000","last_undegraded":"2024-09-19T08:03:00.838150+0000","last_fullsized":"2024-09-19T08:03:00.838150+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.787492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'67","reported_seq":122,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:11.300163+0000","last_change":"2024-09-19T08:01:08.759631+0000","last_active":"2024-09-19T08:03:11.300163+0000","last_peered":"2024-09-19T08:03:11.300163+0000","last_clean":"2024-09-19T08:03:11.300163+0000","last_became_active":"2024-09-19T08:01:08.759156+0000","last_became_peered":"2024-09-19T08:01:08.759156+0000","last_unstale":"2024-09-19T08:03:11.300163+0000","last_undegraded":"2024-09-19T08:03:11.300163+0000","last_fullsized":"2024-09-19T08:03:11.300163+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:48:50.910824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837969+0000","last_change":"2024-09-19T08:01:10.772660+0000","last_active":"2024-09-19T08:03:00.837969+0000","last_peered":"2024-09-19T08:03:00.837969+0000","last_clean":"2024-09-19T08:03:00.837969+0000","last_became_active":"2024-09-19T08:01:10.771816+0000","last_became_peered":"2024-09-19T08:01:10.771816+0000","last_unstale":"2024-09-19T08:03:00.837969+0000","last_undegraded":"2024-09-19T08:03:00.837969+0000","last_fullsized":"2024-09-19T08:03:00.837969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:04:32.667107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'7","reported_seq":144,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:09.883643+0000","last_change":"2024-09-19T08:01:09.749844+0000","last_active":"2024-09-19T08:03:09.883643+0000","last_peered":"2024-09-19T08:03:09.883643+0000","last_clean":"2024-09-19T08:03:09.883643+0000","last_became_active":"2024-09-19T08:01:09.749410+0000","last_became_peered":"2024-09-19T08:01:09.749410+0000","last_unstale":"2024-09-19T08:03:09.883643+0000","last_undegraded":"2024-09-19T08:03:09.883643+0000","last_fullsized":"2024-09-19T08:03:09.883643+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:03:14.989814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1704,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":113,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838196+0000","last_change":"2024-09-19T08:01:10.773782+0000","last_active":"2024-09-19T08:03:00.838196+0000","last_peered":"2024-09-19T08:03:00.838196+0000","last_clean":"2024-09-19T08:03:00.838196+0000","last_became_active":"2024-09-19T08:01:10.773471+0000","last_became_peered":"2024-09-19T08:01:10.773471+0000","last_unstale":"2024-09-19T08:03:00.838196+0000","last_undegraded":"2024-09-19T08:03:00.838196+0000","last_fullsized":"2024-09-19T08:03:00.838196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:15.780259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"81'148","reported_seq":388,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:02.852179+0000","last_change":"2024-09-19T08:00:01.677517+0000","last_active":"2024-09-19T08:03:02.852179+0000","last_peered":"2024-09-19T08:03:02.852179+0000","last_clean":"2024-09-19T08:03:02.852179+0000","last_became_active":"2024-09-19T08:00:01.677089+0000","last_became_peered":"2024-09-19T08:00:01.677089+0000","last_unstale":"2024-09-19T08:03:02.852179+0000","last_undegraded":"2024-09-19T08:03:02.852179+0000","last_fullsized":"2024-09-19T08:03:02.852179+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_clean_scrub_stamp":"2024-09-19T07:56:43.962142+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:10:36.158811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761731+0000","last_change":"2024-09-19T08:01:08.759819+0000","last_active":"2024-09-19T08:03:00.761731+0000","last_peered":"2024-09-19T08:03:00.761731+0000","last_clean":"2024-09-19T08:03:00.761731+0000","last_became_active":"2024-09-19T08:01:08.759420+0000","last_became_peered":"2024-09-19T08:01:08.759420+0000","last_unstale":"2024-09-19T08:03:00.761731+0000","last_undegraded":"2024-09-19T08:03:00.761731+0000","last_fullsized":"2024-09-19T08:03:00.761731+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:06:16.852323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.843903+0000","last_change":"2024-09-19T08:01:26.724260+0000","last_active":"2024-09-19T08:03:00.843903+0000","last_peered":"2024-09-19T08:03:00.843903+0000","last_clean":"2024-09-19T08:03:00.843903+0000","last_became_active":"2024-09-19T08:01:26.723851+0000","last_became_peered":"2024-09-19T08:01:26.723851+0000","last_unstale":"2024-09-19T08:03:00.843903+0000","last_undegraded":"2024-09-19T08:03:00.843903+0000","last_fullsized":"2024-09-19T08:03:00.843903+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:18:46.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837872+0000","last_change":"2024-09-19T08:01:08.761291+0000","last_active":"2024-09-19T08:03:00.837872+0000","last_peered":"2024-09-19T08:03:00.837872+0000","last_clean":"2024-09-19T08:03:00.837872+0000","last_became_active":"2024-09-19T08:01:08.760238+0000","last_became_peered":"2024-09-19T08:01:08.760238+0000","last_unstale":"2024-09-19T08:03:00.837872+0000","last_undegraded":"2024-09-19T08:03:00.837872+0000","last_fullsized":"2024-09-19T08:03:00.837872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:26:09.980573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758639+0000","last_change":"2024-09-19T08:01:10.773056+0000","last_active":"2024-09-19T08:03:00.758639+0000","last_peered":"2024-09-19T08:03:00.758639+0000","last_clean":"2024-09-19T08:03:00.758639+0000","last_became_active":"2024-09-19T08:01:10.772431+0000","last_became_peered":"2024-09-19T08:01:10.772431+0000","last_unstale":"2024-09-19T08:03:00.758639+0000","last_undegraded":"2024-09-19T08:03:00.758639+0000","last_fullsized":"2024-09-19T08:03:00.758639+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:39:06.878086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":143,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.854713+0000","last_change":"2024-09-19T08:01:09.749707+0000","last_active":"2024-09-19T08:03:10.854713+0000","last_peered":"2024-09-19T08:03:10.854713+0000","last_clean":"2024-09-19T08:03:10.854713+0000","last_became_active":"2024-09-19T08:01:09.749013+0000","last_became_peered":"2024-09-19T08:01:09.749013+0000","last_unstale":"2024-09-19T08:03:10.854713+0000","last_undegraded":"2024-09-19T08:03:10.854713+0000","last_fullsized":"2024-09-19T08:03:10.854713+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:40:12.682158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841250+0000","last_change":"2024-09-19T08:01:10.774101+0000","last_active":"2024-09-19T08:03:00.841250+0000","last_peered":"2024-09-19T08:03:00.841250+0000","last_clean":"2024-09-19T08:03:00.841250+0000","last_became_active":"2024-09-19T08:01:10.773822+0000","last_became_peered":"2024-09-19T08:01:10.773822+0000","last_unstale":"2024-09-19T08:03:00.841250+0000","last_undegraded":"2024-09-19T08:03:00.841250+0000","last_fullsized":"2024-09-19T08:03:00.841250+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:29.571118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"81'6","reported_seq":142,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.855374+0000","last_change":"2024-09-19T08:01:08.761675+0000","last_active":"2024-09-19T08:03:10.855374+0000","last_peered":"2024-09-19T08:03:10.855374+0000","last_clean":"2024-09-19T08:03:10.855374+0000","last_became_active":"2024-09-19T08:01:08.761346+0000","last_became_peered":"2024-09-19T08:01:08.761346+0000","last_unstale":"2024-09-19T08:03:10.855374+0000","last_undegraded":"2024-09-19T08:03:10.855374+0000","last_fullsized":"2024-09-19T08:03:10.855374+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:51:19.237458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839233+0000","last_change":"2024-09-19T08:01:10.781565+0000","last_active":"2024-09-19T08:03:00.839233+0000","last_peered":"2024-09-19T08:03:00.839233+0000","last_clean":"2024-09-19T08:03:00.839233+0000","last_became_active":"2024-09-19T08:01:10.781301+0000","last_became_peered":"2024-09-19T08:01:10.781301+0000","last_unstale":"2024-09-19T08:03:00.839233+0000","last_undegraded":"2024-09-19T08:03:00.839233+0000","last_fullsized":"2024-09-19T08:03:00.839233+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:55:52.932009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844577+0000","last_change":"2024-09-19T08:01:09.748918+0000","last_active":"2024-09-19T08:03:00.844577+0000","last_peered":"2024-09-19T08:03:00.844577+0000","last_clean":"2024-09-19T08:03:00.844577+0000","last_became_active":"2024-09-19T08:01:09.748497+0000","last_became_peered":"2024-09-19T08:01:09.748497+0000","last_unstale":"2024-09-19T08:03:00.844577+0000","last_undegraded":"2024-09-19T08:03:00.844577+0000","last_fullsized":"2024-09-19T08:03:00.844577+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:20:31.030133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839231+0000","last_change":"2024-09-19T08:01:10.781388+0000","last_active":"2024-09-19T08:03:00.839231+0000","last_peered":"2024-09-19T08:03:00.839231+0000","last_clean":"2024-09-19T08:03:00.839231+0000","last_became_active":"2024-09-19T08:01:10.781063+0000","last_became_peered":"2024-09-19T08:01:10.781063+0000","last_unstale":"2024-09-19T08:03:00.839231+0000","last_undegraded":"2024-09-19T08:03:00.839231+0000","last_fullsized":"2024-09-19T08:03:00.839231+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:12:08.852717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847755+0000","last_change":"2024-09-19T08:01:09.750080+0000","last_active":"2024-09-19T08:03:00.847755+0000","last_peered":"2024-09-19T08:03:00.847755+0000","last_clean":"2024-09-19T08:03:00.847755+0000","last_became_active":"2024-09-19T08:01:09.749053+0000","last_became_peered":"2024-09-19T08:01:09.749053+0000","last_unstale":"2024-09-19T08:03:00.847755+0000","last_undegraded":"2024-09-19T08:03:00.847755+0000","last_fullsized":"2024-09-19T08:03:00.847755+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:15:45.292714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759144+0000","last_change":"2024-09-19T08:01:10.767070+0000","last_active":"2024-09-19T08:03:00.759144+0000","last_peered":"2024-09-19T08:03:00.759144+0000","last_clean":"2024-09-19T08:03:00.759144+0000","last_became_active":"2024-09-19T08:01:10.766760+0000","last_became_peered":"2024-09-19T08:01:10.766760+0000","last_unstale":"2024-09-19T08:03:00.759144+0000","last_undegraded":"2024-09-19T08:03:00.759144+0000","last_fullsized":"2024-09-19T08:03:00.759144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:17.841135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'105","reported_seq":221,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:11.632933+0000","last_change":"2024-09-19T08:01:08.756143+0000","last_active":"2024-09-19T08:03:11.632933+0000","last_peered":"2024-09-19T08:03:11.632933+0000","last_clean":"2024-09-19T08:03:11.632933+0000","last_became_active":"2024-09-19T08:01:08.755777+0000","last_became_peered":"2024-09-19T08:01:08.755777+0000","last_unstale":"2024-09-19T08:03:11.632933+0000","last_undegraded":"2024-09-19T08:03:11.632933+0000","last_fullsized":"2024-09-19T08:03:11.632933+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:21.430673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":191,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849237+0000","last_change":"2024-09-19T08:01:10.774140+0000","last_active":"2024-09-19T08:03:00.849237+0000","last_peered":"2024-09-19T08:03:00.849237+0000","last_clean":"2024-09-19T08:03:00.849237+0000","last_became_active":"2024-09-19T08:01:10.773819+0000","last_became_peered":"2024-09-19T08:01:10.773819+0000","last_unstale":"2024-09-19T08:03:00.849237+0000","last_undegraded":"2024-09-19T08:03:00.849237+0000","last_fullsized":"2024-09-19T08:03:00.849237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:15:13.174013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761607+0000","last_change":"2024-09-19T08:01:09.169129+0000","last_active":"2024-09-19T08:03:00.761607+0000","last_peered":"2024-09-19T08:03:00.761607+0000","last_clean":"2024-09-19T08:03:00.761607+0000","last_became_active":"2024-09-19T08:01:09.168734+0000","last_became_peered":"2024-09-19T08:01:09.168734+0000","last_unstale":"2024-09-19T08:03:00.761607+0000","last_undegraded":"2024-09-19T08:03:00.761607+0000","last_fullsized":"2024-09-19T08:03:00.761607+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:01:52.232391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994759+0000","last_change":"2024-09-19T08:01:10.768592+0000","last_active":"2024-09-19T08:03:00.994759+0000","last_peered":"2024-09-19T08:03:00.994759+0000","last_clean":"2024-09-19T08:03:00.994759+0000","last_became_active":"2024-09-19T08:01:10.767645+0000","last_became_peered":"2024-09-19T08:01:10.767645+0000","last_unstale":"2024-09-19T08:03:00.994759+0000","last_undegraded":"2024-09-19T08:03:00.994759+0000","last_fullsized":"2024-09-19T08:03:00.994759+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:08:46.227461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997364+0000","last_change":"2024-09-19T08:01:09.749240+0000","last_active":"2024-09-19T08:03:00.997364+0000","last_peered":"2024-09-19T08:03:00.997364+0000","last_clean":"2024-09-19T08:03:00.997364+0000","last_became_active":"2024-09-19T08:01:09.748945+0000","last_became_peered":"2024-09-19T08:01:09.748945+0000","last_unstale":"2024-09-19T08:03:00.997364+0000","last_undegraded":"2024-09-19T08:03:00.997364+0000","last_fullsized":"2024-09-19T08:03:00.997364+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:25:13.791290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847326+0000","last_change":"2024-09-19T08:01:10.783133+0000","last_active":"2024-09-19T08:03:00.847326+0000","last_peered":"2024-09-19T08:03:00.847326+0000","last_clean":"2024-09-19T08:03:00.847326+0000","last_became_active":"2024-09-19T08:01:10.782797+0000","last_became_peered":"2024-09-19T08:01:10.782797+0000","last_unstale":"2024-09-19T08:03:00.847326+0000","last_undegraded":"2024-09-19T08:03:00.847326+0000","last_fullsized":"2024-09-19T08:03:00.847326+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:30:43.551877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848489+0000","last_change":"2024-09-19T08:01:08.757964+0000","last_active":"2024-09-19T08:03:00.848489+0000","last_peered":"2024-09-19T08:03:00.848489+0000","last_clean":"2024-09-19T08:03:00.848489+0000","last_became_active":"2024-09-19T08:01:08.757526+0000","last_became_peered":"2024-09-19T08:01:08.757526+0000","last_unstale":"2024-09-19T08:03:00.848489+0000","last_undegraded":"2024-09-19T08:03:00.848489+0000","last_fullsized":"2024-09-19T08:03:00.848489+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:03.565702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780557+0000","last_change":"2024-09-19T08:01:10.769974+0000","last_active":"2024-09-19T08:03:00.780557+0000","last_peered":"2024-09-19T08:03:00.780557+0000","last_clean":"2024-09-19T08:03:00.780557+0000","last_became_active":"2024-09-19T08:01:10.769650+0000","last_became_peered":"2024-09-19T08:01:10.769650+0000","last_unstale":"2024-09-19T08:03:00.780557+0000","last_undegraded":"2024-09-19T08:03:00.780557+0000","last_fullsized":"2024-09-19T08:03:00.780557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:01.755198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"81'4","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760043+0000","last_change":"2024-09-19T08:01:08.763476+0000","last_active":"2024-09-19T08:03:00.760043+0000","last_peered":"2024-09-19T08:03:00.760043+0000","last_clean":"2024-09-19T08:03:00.760043+0000","last_became_active":"2024-09-19T08:01:08.763154+0000","last_became_peered":"2024-09-19T08:01:08.763154+0000","last_unstale":"2024-09-19T08:03:00.760043+0000","last_undegraded":"2024-09-19T08:03:00.760043+0000","last_fullsized":"2024-09-19T08:03:00.760043+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:06.202452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848751+0000","last_change":"2024-09-19T08:01:10.784201+0000","last_active":"2024-09-19T08:03:00.848751+0000","last_peered":"2024-09-19T08:03:00.848751+0000","last_clean":"2024-09-19T08:03:00.848751+0000","last_became_active":"2024-09-19T08:01:10.783867+0000","last_became_peered":"2024-09-19T08:01:10.783867+0000","last_unstale":"2024-09-19T08:03:00.848751+0000","last_undegraded":"2024-09-19T08:03:00.848751+0000","last_fullsized":"2024-09-19T08:03:00.848751+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:43:50.082183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847290+0000","last_change":"2024-09-19T08:01:08.760479+0000","last_active":"2024-09-19T08:03:00.847290+0000","last_peered":"2024-09-19T08:03:00.847290+0000","last_clean":"2024-09-19T08:03:00.847290+0000","last_became_active":"2024-09-19T08:01:08.760004+0000","last_became_peered":"2024-09-19T08:01:08.760004+0000","last_unstale":"2024-09-19T08:03:00.847290+0000","last_undegraded":"2024-09-19T08:03:00.847290+0000","last_fullsized":"2024-09-19T08:03:00.847290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:41:19.956623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837851+0000","last_change":"2024-09-19T08:01:10.772184+0000","last_active":"2024-09-19T08:03:00.837851+0000","last_peered":"2024-09-19T08:03:00.837851+0000","last_clean":"2024-09-19T08:03:00.837851+0000","last_became_active":"2024-09-19T08:01:10.771825+0000","last_became_peered":"2024-09-19T08:01:10.771825+0000","last_unstale":"2024-09-19T08:03:00.837851+0000","last_undegraded":"2024-09-19T08:03:00.837851+0000","last_fullsized":"2024-09-19T08:03:00.837851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:36.764575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841567+0000","last_change":"2024-09-19T08:01:10.779368+0000","last_active":"2024-09-19T08:03:00.841567+0000","last_peered":"2024-09-19T08:03:00.841567+0000","last_clean":"2024-09-19T08:03:00.841567+0000","last_became_active":"2024-09-19T08:01:10.778884+0000","last_became_peered":"2024-09-19T08:01:10.778884+0000","last_unstale":"2024-09-19T08:03:00.841567+0000","last_undegraded":"2024-09-19T08:03:00.841567+0000","last_fullsized":"2024-09-19T08:03:00.841567+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:28:29.630300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780536+0000","last_change":"2024-09-19T08:01:26.741053+0000","last_active":"2024-09-19T08:03:00.780536+0000","last_peered":"2024-09-19T08:03:00.780536+0000","last_clean":"2024-09-19T08:03:00.780536+0000","last_became_active":"2024-09-19T08:01:26.724714+0000","last_became_peered":"2024-09-19T08:01:26.724714+0000","last_unstale":"2024-09-19T08:03:00.780536+0000","last_undegraded":"2024-09-19T08:03:00.780536+0000","last_fullsized":"2024-09-19T08:03:00.780536+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:59.583797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997323+0000","last_change":"2024-09-19T08:01:10.781389+0000","last_active":"2024-09-19T08:03:00.997323+0000","last_peered":"2024-09-19T08:03:00.997323+0000","last_clean":"2024-09-19T08:03:00.997323+0000","last_became_active":"2024-09-19T08:01:10.781080+0000","last_became_peered":"2024-09-19T08:01:10.781080+0000","last_unstale":"2024-09-19T08:03:00.997323+0000","last_undegraded":"2024-09-19T08:03:00.997323+0000","last_fullsized":"2024-09-19T08:03:00.997323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:40:45.225957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847701+0000","last_change":"2024-09-19T08:01:26.768102+0000","last_active":"2024-09-19T08:03:00.847701+0000","last_peered":"2024-09-19T08:03:00.847701+0000","last_clean":"2024-09-19T08:03:00.847701+0000","last_became_active":"2024-09-19T08:01:26.759462+0000","last_became_peered":"2024-09-19T08:01:26.759462+0000","last_unstale":"2024-09-19T08:03:00.847701+0000","last_undegraded":"2024-09-19T08:03:00.847701+0000","last_fullsized":"2024-09-19T08:03:00.847701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:02:06.815238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994921+0000","last_change":"2024-09-19T08:01:10.771031+0000","last_active":"2024-09-19T08:03:00.994921+0000","last_peered":"2024-09-19T08:03:00.994921+0000","last_clean":"2024-09-19T08:03:00.994921+0000","last_became_active":"2024-09-19T08:01:10.770675+0000","last_became_peered":"2024-09-19T08:01:10.770675+0000","last_unstale":"2024-09-19T08:03:00.994921+0000","last_undegraded":"2024-09-19T08:03:00.994921+0000","last_fullsized":"2024-09-19T08:03:00.994921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:42.203848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849256+0000","last_change":"2024-09-19T08:01:09.748622+0000","last_active":"2024-09-19T08:03:00.849256+0000","last_peered":"2024-09-19T08:03:00.849256+0000","last_clean":"2024-09-19T08:03:00.849256+0000","last_became_active":"2024-09-19T08:01:09.748202+0000","last_became_peered":"2024-09-19T08:01:09.748202+0000","last_unstale":"2024-09-19T08:03:00.849256+0000","last_undegraded":"2024-09-19T08:03:00.849256+0000","last_fullsized":"2024-09-19T08:03:00.849256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:17:39.861820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849123+0000","last_change":"2024-09-19T08:01:10.768949+0000","last_active":"2024-09-19T08:03:00.849123+0000","last_peered":"2024-09-19T08:03:00.849123+0000","last_clean":"2024-09-19T08:03:00.849123+0000","last_became_active":"2024-09-19T08:01:10.768159+0000","last_became_peered":"2024-09-19T08:01:10.768159+0000","last_unstale":"2024-09-19T08:03:00.849123+0000","last_undegraded":"2024-09-19T08:03:00.849123+0000","last_fullsized":"2024-09-19T08:03:00.849123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:43.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"81'14","reported_seq":176,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.854932+0000","last_change":"2024-09-19T08:01:26.735877+0000","last_active":"2024-09-19T08:03:10.854932+0000","last_peered":"2024-09-19T08:03:10.854932+0000","last_clean":"2024-09-19T08:03:10.854932+0000","last_became_active":"2024-09-19T08:01:26.723597+0000","last_became_peered":"2024-09-19T08:01:26.723597+0000","last_unstale":"2024-09-19T08:03:10.854932+0000","last_undegraded":"2024-09-19T08:03:10.854932+0000","last_fullsized":"2024-09-19T08:03:10.854932+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:43:42.340832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":110,"num_read_kb":110,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,1],"acting":[9,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":800997,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1499,"num_read_kb":974,"num_write":507,"num_write_kb":2386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1254,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2764800,"data_stored":2402991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645349,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":905216,"data_stored":436392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874282,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33192,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93711960,"statfs":{"total":95995035648,"available":95961047040,"internally_reserved":0,"allocated":1933312,"data_stored":1424044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005103,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32464,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712688,"statfs":{"total":95995035648,"available":95961792512,"internally_reserved":0,"allocated":1310720,"data_stored":827750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234035,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1294336,"data_stored":826874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462968,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1503232,"data_stored":1029567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593790,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36228,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708924,"statfs":{"total":95995035648,"available":95957938176,"internally_reserved":0,"allocated":901120,"data_stored":434644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822722,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28836,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716316,"statfs":{"total":95995035648,"available":95965507584,"internally_reserved":0,"allocated":1855488,"data_stored":1356970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986247,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":901120,"data_stored":434898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215179,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32908,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712244,"statfs":{"total":95995035648,"available":95961337856,"internally_reserved":0,"allocated":1642496,"data_stored":1155524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":507332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607638,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":978944,"data_stored":505622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":921600,"data_stored":436124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":72210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":73920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":753664,"data_stored":721878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":333073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":400147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T08:03:15.617 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-19T08:03:15.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-19T08:03:15.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:15 smithi162 ceph-mon[35260]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-09-19T08:03:15.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:15 smithi139 ceph-mon[34874]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-09-19T08:03:15.869 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:16.307 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:16 smithi098 ceph-mon[31333]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:16.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.306+0000 7fee27382640 1 -- 172.21.15.98:0/3374656138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee20105f40 msgr2=0x7fee201063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:16.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.306+0000 7fee27382640 1 --2- 172.21.15.98:0/3374656138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee20105f40 0x7fee201063d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fee10009900 tx=0x7fee1002a470 comp rx=0 tx=0).stop 2024-09-19T08:03:16.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.306+0000 7fee27382640 1 -- 172.21.15.98:0/3374656138 shutdown_connections 2024-09-19T08:03:16.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.306+0000 7fee27382640 1 --2- 172.21.15.98:0/3374656138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee20105f40 0x7fee201063d0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:16.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.306+0000 7fee27382640 1 --2- 172.21.15.98:0/3374656138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee20101450 0x7fee20105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:16.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.306+0000 7fee27382640 1 --2- 172.21.15.98:0/3374656138 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee20100b30 0x7fee20100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:16.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.306+0000 7fee27382640 1 -- 172.21.15.98:0/3374656138 >> 172.21.15.98:0/3374656138 conn(0x7fee200fc7a0 msgr2=0x7fee200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:16.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.306+0000 7fee27382640 1 -- 172.21.15.98:0/3374656138 shutdown_connections 2024-09-19T08:03:16.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.306+0000 7fee27382640 1 -- 172.21.15.98:0/3374656138 wait complete. 2024-09-19T08:03:16.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.307+0000 7fee27382640 1 Processor -- start 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.307+0000 7fee27382640 1 -- start start 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.307+0000 7fee27382640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee20100b30 0x7fee201a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.307+0000 7fee27382640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee20101450 0x7fee201a4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.307+0000 7fee27382640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee20105f40 0x7fee2019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.307+0000 7fee27382640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fee201185c0 con 0x7fee20105f40 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.307+0000 7fee27382640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fee20118440 con 0x7fee20100b30 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.307+0000 7fee27382640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fee20118740 con 0x7fee20101450 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.307+0000 7fee2591f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee20105f40 0x7fee2019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.307+0000 7fee2591f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee20105f40 0x7fee2019e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56064/0 (socket says 172.21.15.98:56064) 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.307+0000 7fee2591f640 1 -- 172.21.15.98:0/4069230884 learned_addr learned my addr 172.21.15.98:0/4069230884 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.307+0000 7fee2491d640 1 --2- 172.21.15.98:0/4069230884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee20101450 0x7fee201a4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.307+0000 7fee2511e640 1 --2- 172.21.15.98:0/4069230884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee20100b30 0x7fee201a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.308+0000 7fee2591f640 1 -- 172.21.15.98:0/4069230884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee20101450 msgr2=0x7fee201a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.308+0000 7fee2591f640 1 --2- 172.21.15.98:0/4069230884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee20101450 0x7fee201a4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:16.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.308+0000 7fee2591f640 1 -- 172.21.15.98:0/4069230884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee20100b30 msgr2=0x7fee201a4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.308+0000 7fee2591f640 1 --2- 172.21.15.98:0/4069230884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee20100b30 0x7fee201a4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.308+0000 7fee2591f640 1 -- 172.21.15.98:0/4069230884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2019ef90 con 0x7fee20105f40 2024-09-19T08:03:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.308+0000 7fee2591f640 1 --2- 172.21.15.98:0/4069230884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee20105f40 0x7fee2019e7f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fee10009230 tx=0x7fee100042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.308+0000 7fee1e7fc640 1 -- 172.21.15.98:0/4069230884 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1003f030 con 0x7fee20105f40 2024-09-19T08:03:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.308+0000 7fee1e7fc640 1 -- 172.21.15.98:0/4069230884 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee10003ce0 con 0x7fee20105f40 2024-09-19T08:03:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.308+0000 7fee27382640 1 -- 172.21.15.98:0/4069230884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee2019f220 con 0x7fee20105f40 2024-09-19T08:03:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.308+0000 7fee1e7fc640 1 -- 172.21.15.98:0/4069230884 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee10035070 con 0x7fee20105f40 2024-09-19T08:03:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.308+0000 7fee27382640 1 -- 172.21.15.98:0/4069230884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee201ab4d0 con 0x7fee20105f40 2024-09-19T08:03:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.309+0000 7fee27382640 1 -- 172.21.15.98:0/4069230884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee2006b070 con 0x7fee20105f40 2024-09-19T08:03:16.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.311+0000 7fee1e7fc640 1 -- 172.21.15.98:0/4069230884 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7fee10042030 con 0x7fee20105f40 2024-09-19T08:03:16.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.312+0000 7fee1e7fc640 1 --2- 172.21.15.98:0/4069230884 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fee080a34d0 0x7fee080a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:16.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.312+0000 7fee2511e640 1 --2- 172.21.15.98:0/4069230884 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fee080a34d0 0x7fee080a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:16.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.312+0000 7fee1e7fc640 1 -- 172.21.15.98:0/4069230884 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fee100f0850 con 0x7fee20105f40 2024-09-19T08:03:16.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.312+0000 7fee2511e640 1 --2- 172.21.15.98:0/4069230884 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fee080a34d0 0x7fee080a5990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fee18007350 tx=0x7fee180072e0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:16.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.315+0000 7fee1e7fc640 1 -- 172.21.15.98:0/4069230884 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fee100bd0e0 con 0x7fee20105f40 2024-09-19T08:03:16.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.476+0000 7fee27382640 1 -- 172.21.15.98:0/4069230884 --> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fee20104fe0 con 0x7fee080a34d0 2024-09-19T08:03:16.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.482+0000 7fee1e7fc640 1 -- 172.21.15.98:0/4069230884 <== mgr.15252 v2:172.21.15.98:6800/1961446727 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257817 (secure 0 0 0) 0x7fee20104fe0 con 0x7fee080a34d0 2024-09-19T08:03:16.486 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:16.487 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-19T08:03:16.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.485+0000 7fee27382640 1 -- 172.21.15.98:0/4069230884 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fee080a34d0 msgr2=0x7fee080a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:16.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.485+0000 7fee27382640 1 --2- 172.21.15.98:0/4069230884 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fee080a34d0 0x7fee080a5990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fee18007350 tx=0x7fee180072e0 comp rx=0 tx=0).stop 2024-09-19T08:03:16.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.485+0000 7fee27382640 1 -- 172.21.15.98:0/4069230884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee20105f40 msgr2=0x7fee2019e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:16.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.485+0000 7fee27382640 1 --2- 172.21.15.98:0/4069230884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee20105f40 0x7fee2019e7f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fee10009230 tx=0x7fee100042f0 comp rx=0 tx=0).stop 2024-09-19T08:03:16.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.485+0000 7fee27382640 1 -- 172.21.15.98:0/4069230884 shutdown_connections 2024-09-19T08:03:16.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.485+0000 7fee27382640 1 --2- 172.21.15.98:0/4069230884 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fee080a34d0 0x7fee080a5990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:16.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.485+0000 7fee27382640 1 --2- 172.21.15.98:0/4069230884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee20105f40 0x7fee2019e7f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:16.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.485+0000 7fee27382640 1 --2- 172.21.15.98:0/4069230884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee20101450 0x7fee201a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:16.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.485+0000 7fee27382640 1 --2- 172.21.15.98:0/4069230884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee20100b30 0x7fee201a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:16.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.485+0000 7fee27382640 1 -- 172.21.15.98:0/4069230884 >> 172.21.15.98:0/4069230884 conn(0x7fee200fc7a0 msgr2=0x7fee201038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:16.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.485+0000 7fee27382640 1 -- 172.21.15.98:0/4069230884 shutdown_connections 2024-09-19T08:03:16.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:16.485+0000 7fee27382640 1 -- 172.21.15.98:0/4069230884 wait complete. 2024-09-19T08:03:16.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:16 smithi162 ceph-mon[35260]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:16.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:16 smithi139 ceph-mon[34874]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:17.194 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-09-19T08:03:14.773970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1391599,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1736,"num_read_kb":1304,"num_write":710,"num_write_kb":5326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181990,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1036,"ondisk_log_size":1036,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376776,"kb_used_data":14784,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565048,"statfs":{"total":1151940427776,"available":1151554609152,"internally_reserved":0,"allocated":15138816,"data_stored":9375741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":370193724},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":200331,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":461,"num_read_kb":502,"num_write":155,"num_write_kb":923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002625"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780911+0000","last_change":"2024-09-19T08:01:10.768950+0000","last_active":"2024-09-19T08:03:00.780911+0000","last_peered":"2024-09-19T08:03:00.780911+0000","last_clean":"2024-09-19T08:03:00.780911+0000","last_became_active":"2024-09-19T08:01:10.768047+0000","last_became_peered":"2024-09-19T08:01:10.768047+0000","last_unstale":"2024-09-19T08:03:00.780911+0000","last_undegraded":"2024-09-19T08:03:00.780911+0000","last_fullsized":"2024-09-19T08:03:00.780911+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:26.219079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994728+0000","last_change":"2024-09-19T08:01:10.783208+0000","last_active":"2024-09-19T08:03:00.994728+0000","last_peered":"2024-09-19T08:03:00.994728+0000","last_clean":"2024-09-19T08:03:00.994728+0000","last_became_active":"2024-09-19T08:01:10.782888+0000","last_became_peered":"2024-09-19T08:01:10.782888+0000","last_unstale":"2024-09-19T08:03:00.994728+0000","last_undegraded":"2024-09-19T08:03:00.994728+0000","last_fullsized":"2024-09-19T08:03:00.994728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:51:27.603166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994427+0000","last_change":"2024-09-19T08:01:10.770395+0000","last_active":"2024-09-19T08:03:00.994427+0000","last_peered":"2024-09-19T08:03:00.994427+0000","last_clean":"2024-09-19T08:03:00.994427+0000","last_became_active":"2024-09-19T08:01:10.770073+0000","last_became_peered":"2024-09-19T08:01:10.770073+0000","last_unstale":"2024-09-19T08:03:00.994427+0000","last_undegraded":"2024-09-19T08:03:00.994427+0000","last_fullsized":"2024-09-19T08:03:00.994427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:20.047023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761433+0000","last_change":"2024-09-19T08:01:10.777138+0000","last_active":"2024-09-19T08:03:00.761433+0000","last_peered":"2024-09-19T08:03:00.761433+0000","last_clean":"2024-09-19T08:03:00.761433+0000","last_became_active":"2024-09-19T08:01:10.776810+0000","last_became_peered":"2024-09-19T08:01:10.776810+0000","last_unstale":"2024-09-19T08:03:00.761433+0000","last_undegraded":"2024-09-19T08:03:00.761433+0000","last_fullsized":"2024-09-19T08:03:00.761433+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:39.983321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847497+0000","last_change":"2024-09-19T08:01:10.770435+0000","last_active":"2024-09-19T08:03:00.847497+0000","last_peered":"2024-09-19T08:03:00.847497+0000","last_clean":"2024-09-19T08:03:00.847497+0000","last_became_active":"2024-09-19T08:01:10.769686+0000","last_became_peered":"2024-09-19T08:01:10.769686+0000","last_unstale":"2024-09-19T08:03:00.847497+0000","last_undegraded":"2024-09-19T08:03:00.847497+0000","last_fullsized":"2024-09-19T08:03:00.847497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:10:19.610925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839257+0000","last_change":"2024-09-19T08:01:10.782382+0000","last_active":"2024-09-19T08:03:00.839257+0000","last_peered":"2024-09-19T08:03:00.839257+0000","last_clean":"2024-09-19T08:03:00.839257+0000","last_became_active":"2024-09-19T08:01:10.781996+0000","last_became_peered":"2024-09-19T08:01:10.781996+0000","last_unstale":"2024-09-19T08:03:00.839257+0000","last_undegraded":"2024-09-19T08:03:00.839257+0000","last_fullsized":"2024-09-19T08:03:00.839257+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:43:13.574336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837718+0000","last_change":"2024-09-19T08:01:10.773776+0000","last_active":"2024-09-19T08:03:00.837718+0000","last_peered":"2024-09-19T08:03:00.837718+0000","last_clean":"2024-09-19T08:03:00.837718+0000","last_became_active":"2024-09-19T08:01:10.772442+0000","last_became_peered":"2024-09-19T08:01:10.772442+0000","last_unstale":"2024-09-19T08:03:00.837718+0000","last_undegraded":"2024-09-19T08:03:00.837718+0000","last_fullsized":"2024-09-19T08:03:00.837718+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:08:18.155806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848134+0000","last_change":"2024-09-19T08:01:10.781443+0000","last_active":"2024-09-19T08:03:00.848134+0000","last_peered":"2024-09-19T08:03:00.848134+0000","last_clean":"2024-09-19T08:03:00.848134+0000","last_became_active":"2024-09-19T08:01:10.781083+0000","last_became_peered":"2024-09-19T08:01:10.781083+0000","last_unstale":"2024-09-19T08:03:00.848134+0000","last_undegraded":"2024-09-19T08:03:00.848134+0000","last_fullsized":"2024-09-19T08:03:00.848134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:23:22.631865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.998365+0000","last_change":"2024-09-19T08:01:10.767554+0000","last_active":"2024-09-19T08:03:00.998365+0000","last_peered":"2024-09-19T08:03:00.998365+0000","last_clean":"2024-09-19T08:03:00.998365+0000","last_became_active":"2024-09-19T08:01:10.767227+0000","last_became_peered":"2024-09-19T08:01:10.767227+0000","last_unstale":"2024-09-19T08:03:00.998365+0000","last_undegraded":"2024-09-19T08:03:00.998365+0000","last_fullsized":"2024-09-19T08:03:00.998365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:30:13.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839361+0000","last_change":"2024-09-19T08:01:10.783174+0000","last_active":"2024-09-19T08:03:00.839361+0000","last_peered":"2024-09-19T08:03:00.839361+0000","last_clean":"2024-09-19T08:03:00.839361+0000","last_became_active":"2024-09-19T08:01:10.782794+0000","last_became_peered":"2024-09-19T08:01:10.782794+0000","last_unstale":"2024-09-19T08:03:00.839361+0000","last_undegraded":"2024-09-19T08:03:00.839361+0000","last_fullsized":"2024-09-19T08:03:00.839361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:49:19.451107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838191+0000","last_change":"2024-09-19T08:01:10.779638+0000","last_active":"2024-09-19T08:03:00.838191+0000","last_peered":"2024-09-19T08:03:00.838191+0000","last_clean":"2024-09-19T08:03:00.838191+0000","last_became_active":"2024-09-19T08:01:10.779295+0000","last_became_peered":"2024-09-19T08:01:10.779295+0000","last_unstale":"2024-09-19T08:03:00.838191+0000","last_undegraded":"2024-09-19T08:03:00.838191+0000","last_fullsized":"2024-09-19T08:03:00.838191+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:35:10.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761539+0000","last_change":"2024-09-19T08:01:10.784487+0000","last_active":"2024-09-19T08:03:00.761539+0000","last_peered":"2024-09-19T08:03:00.761539+0000","last_clean":"2024-09-19T08:03:00.761539+0000","last_became_active":"2024-09-19T08:01:10.784114+0000","last_became_peered":"2024-09-19T08:01:10.784114+0000","last_unstale":"2024-09-19T08:03:00.761539+0000","last_undegraded":"2024-09-19T08:03:00.761539+0000","last_fullsized":"2024-09-19T08:03:00.761539+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:09:54.819802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758888+0000","last_change":"2024-09-19T08:01:10.783186+0000","last_active":"2024-09-19T08:03:00.758888+0000","last_peered":"2024-09-19T08:03:00.758888+0000","last_clean":"2024-09-19T08:03:00.758888+0000","last_became_active":"2024-09-19T08:01:10.782876+0000","last_became_peered":"2024-09-19T08:01:10.782876+0000","last_unstale":"2024-09-19T08:03:00.758888+0000","last_undegraded":"2024-09-19T08:03:00.758888+0000","last_fullsized":"2024-09-19T08:03:00.758888+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:26:15.252165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994474+0000","last_change":"2024-09-19T08:01:10.783237+0000","last_active":"2024-09-19T08:03:00.994474+0000","last_peered":"2024-09-19T08:03:00.994474+0000","last_clean":"2024-09-19T08:03:00.994474+0000","last_became_active":"2024-09-19T08:01:10.782966+0000","last_became_peered":"2024-09-19T08:01:10.782966+0000","last_unstale":"2024-09-19T08:03:00.994474+0000","last_undegraded":"2024-09-19T08:03:00.994474+0000","last_fullsized":"2024-09-19T08:03:00.994474+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:57:18.771737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994242+0000","last_change":"2024-09-19T08:01:10.771207+0000","last_active":"2024-09-19T08:03:00.994242+0000","last_peered":"2024-09-19T08:03:00.994242+0000","last_clean":"2024-09-19T08:03:00.994242+0000","last_became_active":"2024-09-19T08:01:10.770862+0000","last_became_peered":"2024-09-19T08:01:10.770862+0000","last_unstale":"2024-09-19T08:03:00.994242+0000","last_undegraded":"2024-09-19T08:03:00.994242+0000","last_fullsized":"2024-09-19T08:03:00.994242+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:08:09.313329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997263+0000","last_change":"2024-09-19T08:01:10.775191+0000","last_active":"2024-09-19T08:03:00.997263+0000","last_peered":"2024-09-19T08:03:00.997263+0000","last_clean":"2024-09-19T08:03:00.997263+0000","last_became_active":"2024-09-19T08:01:10.774832+0000","last_became_peered":"2024-09-19T08:01:10.774832+0000","last_unstale":"2024-09-19T08:03:00.997263+0000","last_undegraded":"2024-09-19T08:03:00.997263+0000","last_fullsized":"2024-09-19T08:03:00.997263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:57:29.800787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780968+0000","last_change":"2024-09-19T08:01:10.769115+0000","last_active":"2024-09-19T08:03:00.780968+0000","last_peered":"2024-09-19T08:03:00.780968+0000","last_clean":"2024-09-19T08:03:00.780968+0000","last_became_active":"2024-09-19T08:01:10.768088+0000","last_became_peered":"2024-09-19T08:01:10.768088+0000","last_unstale":"2024-09-19T08:03:00.780968+0000","last_undegraded":"2024-09-19T08:03:00.780968+0000","last_fullsized":"2024-09-19T08:03:00.780968+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:27:16.042722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844124+0000","last_change":"2024-09-19T08:01:10.781426+0000","last_active":"2024-09-19T08:03:00.844124+0000","last_peered":"2024-09-19T08:03:00.844124+0000","last_clean":"2024-09-19T08:03:00.844124+0000","last_became_active":"2024-09-19T08:01:10.781094+0000","last_became_peered":"2024-09-19T08:01:10.781094+0000","last_unstale":"2024-09-19T08:03:00.844124+0000","last_undegraded":"2024-09-19T08:03:00.844124+0000","last_fullsized":"2024-09-19T08:03:00.844124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:43:31.336610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760020+0000","last_change":"2024-09-19T08:01:10.766466+0000","last_active":"2024-09-19T08:03:00.760020+0000","last_peered":"2024-09-19T08:03:00.760020+0000","last_clean":"2024-09-19T08:03:00.760020+0000","last_became_active":"2024-09-19T08:01:10.766123+0000","last_became_peered":"2024-09-19T08:01:10.766123+0000","last_unstale":"2024-09-19T08:03:00.760020+0000","last_undegraded":"2024-09-19T08:03:00.760020+0000","last_fullsized":"2024-09-19T08:03:00.760020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:25.848603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780633+0000","last_change":"2024-09-19T08:01:10.772782+0000","last_active":"2024-09-19T08:03:00.780633+0000","last_peered":"2024-09-19T08:03:00.780633+0000","last_clean":"2024-09-19T08:03:00.780633+0000","last_became_active":"2024-09-19T08:01:10.772445+0000","last_became_peered":"2024-09-19T08:01:10.772445+0000","last_unstale":"2024-09-19T08:03:00.780633+0000","last_undegraded":"2024-09-19T08:03:00.780633+0000","last_fullsized":"2024-09-19T08:03:00.780633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:09.174005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759047+0000","last_change":"2024-09-19T08:01:10.771944+0000","last_active":"2024-09-19T08:03:00.759047+0000","last_peered":"2024-09-19T08:03:00.759047+0000","last_clean":"2024-09-19T08:03:00.759047+0000","last_became_active":"2024-09-19T08:01:10.771643+0000","last_became_peered":"2024-09-19T08:01:10.771643+0000","last_unstale":"2024-09-19T08:03:00.759047+0000","last_undegraded":"2024-09-19T08:03:00.759047+0000","last_fullsized":"2024-09-19T08:03:00.759047+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:46:27.121722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997451+0000","last_change":"2024-09-19T08:01:10.775144+0000","last_active":"2024-09-19T08:03:00.997451+0000","last_peered":"2024-09-19T08:03:00.997451+0000","last_clean":"2024-09-19T08:03:00.997451+0000","last_became_active":"2024-09-19T08:01:10.774769+0000","last_became_peered":"2024-09-19T08:01:10.774769+0000","last_unstale":"2024-09-19T08:03:00.997451+0000","last_undegraded":"2024-09-19T08:03:00.997451+0000","last_fullsized":"2024-09-19T08:03:00.997451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:59:47.053993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848212+0000","last_change":"2024-09-19T08:01:10.766750+0000","last_active":"2024-09-19T08:03:00.848212+0000","last_peered":"2024-09-19T08:03:00.848212+0000","last_clean":"2024-09-19T08:03:00.848212+0000","last_became_active":"2024-09-19T08:01:10.766450+0000","last_became_peered":"2024-09-19T08:01:10.766450+0000","last_unstale":"2024-09-19T08:03:00.848212+0000","last_undegraded":"2024-09-19T08:03:00.848212+0000","last_fullsized":"2024-09-19T08:03:00.848212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:47:58.133815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841186+0000","last_change":"2024-09-19T08:01:10.768430+0000","last_active":"2024-09-19T08:03:00.841186+0000","last_peered":"2024-09-19T08:03:00.841186+0000","last_clean":"2024-09-19T08:03:00.841186+0000","last_became_active":"2024-09-19T08:01:10.768092+0000","last_became_peered":"2024-09-19T08:01:10.768092+0000","last_unstale":"2024-09-19T08:03:00.841186+0000","last_undegraded":"2024-09-19T08:03:00.841186+0000","last_fullsized":"2024-09-19T08:03:00.841186+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:16:42.343138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849206+0000","last_change":"2024-09-19T08:01:10.771131+0000","last_active":"2024-09-19T08:03:00.849206+0000","last_peered":"2024-09-19T08:03:00.849206+0000","last_clean":"2024-09-19T08:03:00.849206+0000","last_became_active":"2024-09-19T08:01:10.770851+0000","last_became_peered":"2024-09-19T08:01:10.770851+0000","last_unstale":"2024-09-19T08:03:00.849206+0000","last_undegraded":"2024-09-19T08:03:00.849206+0000","last_fullsized":"2024-09-19T08:03:00.849206+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:07:15.830476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760273+0000","last_change":"2024-09-19T08:01:10.776220+0000","last_active":"2024-09-19T08:03:00.760273+0000","last_peered":"2024-09-19T08:03:00.760273+0000","last_clean":"2024-09-19T08:03:00.760273+0000","last_became_active":"2024-09-19T08:01:10.775889+0000","last_became_peered":"2024-09-19T08:01:10.775889+0000","last_unstale":"2024-09-19T08:03:00.760273+0000","last_undegraded":"2024-09-19T08:03:00.760273+0000","last_fullsized":"2024-09-19T08:03:00.760273+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:23.714675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780731+0000","last_change":"2024-09-19T08:01:10.776370+0000","last_active":"2024-09-19T08:03:00.780731+0000","last_peered":"2024-09-19T08:03:00.780731+0000","last_clean":"2024-09-19T08:03:00.780731+0000","last_became_active":"2024-09-19T08:01:10.776027+0000","last_became_peered":"2024-09-19T08:01:10.776027+0000","last_unstale":"2024-09-19T08:03:00.780731+0000","last_undegraded":"2024-09-19T08:03:00.780731+0000","last_fullsized":"2024-09-19T08:03:00.780731+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:48:46.293404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758951+0000","last_change":"2024-09-19T08:01:26.758971+0000","last_active":"2024-09-19T08:03:00.758951+0000","last_peered":"2024-09-19T08:03:00.758951+0000","last_clean":"2024-09-19T08:03:00.758951+0000","last_became_active":"2024-09-19T08:01:26.758524+0000","last_became_peered":"2024-09-19T08:01:26.758524+0000","last_unstale":"2024-09-19T08:03:00.758951+0000","last_undegraded":"2024-09-19T08:03:00.758951+0000","last_fullsized":"2024-09-19T08:03:00.758951+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:34:22.439327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994656+0000","last_change":"2024-09-19T08:01:10.767445+0000","last_active":"2024-09-19T08:03:00.994656+0000","last_peered":"2024-09-19T08:03:00.994656+0000","last_clean":"2024-09-19T08:03:00.994656+0000","last_became_active":"2024-09-19T08:01:10.767128+0000","last_became_peered":"2024-09-19T08:01:10.767128+0000","last_unstale":"2024-09-19T08:03:00.994656+0000","last_undegraded":"2024-09-19T08:03:00.994656+0000","last_fullsized":"2024-09-19T08:03:00.994656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:10.273948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759852+0000","last_change":"2024-09-19T08:01:10.768954+0000","last_active":"2024-09-19T08:03:00.759852+0000","last_peered":"2024-09-19T08:03:00.759852+0000","last_clean":"2024-09-19T08:03:00.759852+0000","last_became_active":"2024-09-19T08:01:10.768577+0000","last_became_peered":"2024-09-19T08:01:10.768577+0000","last_unstale":"2024-09-19T08:03:00.759852+0000","last_undegraded":"2024-09-19T08:03:00.759852+0000","last_fullsized":"2024-09-19T08:03:00.759852+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:20:21.812531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759843+0000","last_change":"2024-09-19T08:01:10.774104+0000","last_active":"2024-09-19T08:03:00.759843+0000","last_peered":"2024-09-19T08:03:00.759843+0000","last_clean":"2024-09-19T08:03:00.759843+0000","last_became_active":"2024-09-19T08:01:10.773721+0000","last_became_peered":"2024-09-19T08:01:10.773721+0000","last_unstale":"2024-09-19T08:03:00.759843+0000","last_undegraded":"2024-09-19T08:03:00.759843+0000","last_fullsized":"2024-09-19T08:03:00.759843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:45:34.945994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780513+0000","last_change":"2024-09-19T08:01:26.727296+0000","last_active":"2024-09-19T08:03:00.780513+0000","last_peered":"2024-09-19T08:03:00.780513+0000","last_clean":"2024-09-19T08:03:00.780513+0000","last_became_active":"2024-09-19T08:01:26.726631+0000","last_became_peered":"2024-09-19T08:01:26.726631+0000","last_unstale":"2024-09-19T08:03:00.780513+0000","last_undegraded":"2024-09-19T08:03:00.780513+0000","last_fullsized":"2024-09-19T08:03:00.780513+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:19:24.807050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847424+0000","last_change":"2024-09-19T08:01:10.765282+0000","last_active":"2024-09-19T08:03:00.847424+0000","last_peered":"2024-09-19T08:03:00.847424+0000","last_clean":"2024-09-19T08:03:00.847424+0000","last_became_active":"2024-09-19T08:01:10.764928+0000","last_became_peered":"2024-09-19T08:01:10.764928+0000","last_unstale":"2024-09-19T08:03:00.847424+0000","last_undegraded":"2024-09-19T08:03:00.847424+0000","last_fullsized":"2024-09-19T08:03:00.847424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:00:37.395110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"81'11","reported_seq":152,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:09.854344+0000","last_change":"2024-09-19T08:01:08.760686+0000","last_active":"2024-09-19T08:03:09.854344+0000","last_peered":"2024-09-19T08:03:09.854344+0000","last_clean":"2024-09-19T08:03:09.854344+0000","last_became_active":"2024-09-19T08:01:08.759912+0000","last_became_peered":"2024-09-19T08:01:08.759912+0000","last_unstale":"2024-09-19T08:03:09.854344+0000","last_undegraded":"2024-09-19T08:03:09.854344+0000","last_fullsized":"2024-09-19T08:03:09.854344+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:46:01.879611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2643,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":7,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841395+0000","last_change":"2024-09-19T08:01:10.779668+0000","last_active":"2024-09-19T08:03:00.841395+0000","last_peered":"2024-09-19T08:03:00.841395+0000","last_clean":"2024-09-19T08:03:00.841395+0000","last_became_active":"2024-09-19T08:01:10.779265+0000","last_became_peered":"2024-09-19T08:01:10.779265+0000","last_unstale":"2024-09-19T08:03:00.841395+0000","last_undegraded":"2024-09-19T08:03:00.841395+0000","last_fullsized":"2024-09-19T08:03:00.841395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:06.177700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838967+0000","last_change":"2024-09-19T08:01:08.762668+0000","last_active":"2024-09-19T08:03:00.838967+0000","last_peered":"2024-09-19T08:03:00.838967+0000","last_clean":"2024-09-19T08:03:00.838967+0000","last_became_active":"2024-09-19T08:01:08.762277+0000","last_became_peered":"2024-09-19T08:01:08.762277+0000","last_unstale":"2024-09-19T08:03:00.838967+0000","last_undegraded":"2024-09-19T08:03:00.838967+0000","last_fullsized":"2024-09-19T08:03:00.838967+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:36:59.316655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758585+0000","last_change":"2024-09-19T08:01:08.757401+0000","last_active":"2024-09-19T08:03:00.758585+0000","last_peered":"2024-09-19T08:03:00.758585+0000","last_clean":"2024-09-19T08:03:00.758585+0000","last_became_active":"2024-09-19T08:01:08.757067+0000","last_became_peered":"2024-09-19T08:01:08.757067+0000","last_unstale":"2024-09-19T08:03:00.758585+0000","last_undegraded":"2024-09-19T08:03:00.758585+0000","last_fullsized":"2024-09-19T08:03:00.758585+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:19.617830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848289+0000","last_change":"2024-09-19T08:01:10.765149+0000","last_active":"2024-09-19T08:03:00.848289+0000","last_peered":"2024-09-19T08:03:00.848289+0000","last_clean":"2024-09-19T08:03:00.848289+0000","last_became_active":"2024-09-19T08:01:10.764840+0000","last_became_peered":"2024-09-19T08:01:10.764840+0000","last_unstale":"2024-09-19T08:03:00.848289+0000","last_undegraded":"2024-09-19T08:03:00.848289+0000","last_fullsized":"2024-09-19T08:03:00.848289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:51:15.539307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841355+0000","last_change":"2024-09-19T08:01:08.759435+0000","last_active":"2024-09-19T08:03:00.841355+0000","last_peered":"2024-09-19T08:03:00.841355+0000","last_clean":"2024-09-19T08:03:00.841355+0000","last_became_active":"2024-09-19T08:01:08.759046+0000","last_became_peered":"2024-09-19T08:01:08.759046+0000","last_unstale":"2024-09-19T08:03:00.841355+0000","last_undegraded":"2024-09-19T08:03:00.841355+0000","last_fullsized":"2024-09-19T08:03:00.841355+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:21:24.077626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.843896+0000","last_change":"2024-09-19T08:01:10.778432+0000","last_active":"2024-09-19T08:03:00.843896+0000","last_peered":"2024-09-19T08:03:00.843896+0000","last_clean":"2024-09-19T08:03:00.843896+0000","last_became_active":"2024-09-19T08:01:10.778098+0000","last_became_peered":"2024-09-19T08:01:10.778098+0000","last_unstale":"2024-09-19T08:03:00.843896+0000","last_undegraded":"2024-09-19T08:03:00.843896+0000","last_fullsized":"2024-09-19T08:03:00.843896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:40:12.921165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":250,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.855304+0000","last_change":"2024-09-19T08:01:08.764117+0000","last_active":"2024-09-19T08:03:10.855304+0000","last_peered":"2024-09-19T08:03:10.855304+0000","last_clean":"2024-09-19T08:03:10.855304+0000","last_became_active":"2024-09-19T08:01:08.763308+0000","last_became_peered":"2024-09-19T08:01:08.763308+0000","last_unstale":"2024-09-19T08:03:10.855304+0000","last_undegraded":"2024-09-19T08:03:10.855304+0000","last_fullsized":"2024-09-19T08:03:10.855304+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:19:57.014628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":220,"num_read_kb":220,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997921+0000","last_change":"2024-09-19T08:01:10.781771+0000","last_active":"2024-09-19T08:03:00.997921+0000","last_peered":"2024-09-19T08:03:00.997921+0000","last_clean":"2024-09-19T08:03:00.997921+0000","last_became_active":"2024-09-19T08:01:10.781432+0000","last_became_peered":"2024-09-19T08:01:10.781432+0000","last_unstale":"2024-09-19T08:03:00.997921+0000","last_undegraded":"2024-09-19T08:03:00.997921+0000","last_fullsized":"2024-09-19T08:03:00.997921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:25:15.067850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838023+0000","last_change":"2024-09-19T08:01:26.771171+0000","last_active":"2024-09-19T08:03:00.838023+0000","last_peered":"2024-09-19T08:03:00.838023+0000","last_clean":"2024-09-19T08:03:00.838023+0000","last_became_active":"2024-09-19T08:01:26.759274+0000","last_became_peered":"2024-09-19T08:01:26.759274+0000","last_unstale":"2024-09-19T08:03:00.838023+0000","last_undegraded":"2024-09-19T08:03:00.838023+0000","last_fullsized":"2024-09-19T08:03:00.838023+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:20:54.603420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994926+0000","last_change":"2024-09-19T08:01:10.768318+0000","last_active":"2024-09-19T08:03:00.994926+0000","last_peered":"2024-09-19T08:03:00.994926+0000","last_clean":"2024-09-19T08:03:00.994926+0000","last_became_active":"2024-09-19T08:01:10.768022+0000","last_became_peered":"2024-09-19T08:01:10.768022+0000","last_unstale":"2024-09-19T08:03:00.994926+0000","last_undegraded":"2024-09-19T08:03:00.994926+0000","last_fullsized":"2024-09-19T08:03:00.994926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:00:45.328638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841594+0000","last_change":"2024-09-19T08:01:09.750530+0000","last_active":"2024-09-19T08:03:00.841594+0000","last_peered":"2024-09-19T08:03:00.841594+0000","last_clean":"2024-09-19T08:03:00.841594+0000","last_became_active":"2024-09-19T08:01:09.750029+0000","last_became_peered":"2024-09-19T08:01:09.750029+0000","last_unstale":"2024-09-19T08:03:00.841594+0000","last_undegraded":"2024-09-19T08:03:00.841594+0000","last_fullsized":"2024-09-19T08:03:00.841594+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:01:16.062992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997659+0000","last_change":"2024-09-19T08:01:26.724636+0000","last_active":"2024-09-19T08:03:00.997659+0000","last_peered":"2024-09-19T08:03:00.997659+0000","last_clean":"2024-09-19T08:03:00.997659+0000","last_became_active":"2024-09-19T08:01:26.724171+0000","last_became_peered":"2024-09-19T08:01:26.724171+0000","last_unstale":"2024-09-19T08:03:00.997659+0000","last_undegraded":"2024-09-19T08:03:00.997659+0000","last_fullsized":"2024-09-19T08:03:00.997659+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:54:05.998374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'39","reported_seq":300,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.855298+0000","last_change":"2024-09-19T08:01:08.761950+0000","last_active":"2024-09-19T08:03:10.855298+0000","last_peered":"2024-09-19T08:03:10.855298+0000","last_clean":"2024-09-19T08:03:10.855298+0000","last_became_active":"2024-09-19T08:01:08.761533+0000","last_became_peered":"2024-09-19T08:01:08.761533+0000","last_unstale":"2024-09-19T08:03:10.855298+0000","last_undegraded":"2024-09-19T08:03:10.855298+0000","last_fullsized":"2024-09-19T08:03:10.855298+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:07:26.668141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69393,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":237,"num_read_kb":187,"num_write":41,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847631+0000","last_change":"2024-09-19T08:01:10.774280+0000","last_active":"2024-09-19T08:03:00.847631+0000","last_peered":"2024-09-19T08:03:00.847631+0000","last_clean":"2024-09-19T08:03:00.847631+0000","last_became_active":"2024-09-19T08:01:10.773982+0000","last_became_peered":"2024-09-19T08:01:10.773982+0000","last_unstale":"2024-09-19T08:03:00.847631+0000","last_undegraded":"2024-09-19T08:03:00.847631+0000","last_fullsized":"2024-09-19T08:03:00.847631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:21:43.180057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760058+0000","last_change":"2024-09-19T08:01:08.761067+0000","last_active":"2024-09-19T08:03:00.760058+0000","last_peered":"2024-09-19T08:03:00.760058+0000","last_clean":"2024-09-19T08:03:00.760058+0000","last_became_active":"2024-09-19T08:01:08.760664+0000","last_became_peered":"2024-09-19T08:01:08.760664+0000","last_unstale":"2024-09-19T08:03:00.760058+0000","last_undegraded":"2024-09-19T08:03:00.760058+0000","last_fullsized":"2024-09-19T08:03:00.760058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:22:38.086891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758630+0000","last_change":"2024-09-19T08:01:10.771845+0000","last_active":"2024-09-19T08:03:00.758630+0000","last_peered":"2024-09-19T08:03:00.758630+0000","last_clean":"2024-09-19T08:03:00.758630+0000","last_became_active":"2024-09-19T08:01:10.771533+0000","last_became_peered":"2024-09-19T08:01:10.771533+0000","last_unstale":"2024-09-19T08:03:00.758630+0000","last_undegraded":"2024-09-19T08:03:00.758630+0000","last_fullsized":"2024-09-19T08:03:00.758630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:29:35.180763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841267+0000","last_change":"2024-09-19T08:01:08.763869+0000","last_active":"2024-09-19T08:03:00.841267+0000","last_peered":"2024-09-19T08:03:00.841267+0000","last_clean":"2024-09-19T08:03:00.841267+0000","last_became_active":"2024-09-19T08:01:08.763451+0000","last_became_peered":"2024-09-19T08:01:08.763451+0000","last_unstale":"2024-09-19T08:03:00.841267+0000","last_undegraded":"2024-09-19T08:03:00.841267+0000","last_fullsized":"2024-09-19T08:03:00.841267+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:32:40.901049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838028+0000","last_change":"2024-09-19T08:01:10.770823+0000","last_active":"2024-09-19T08:03:00.838028+0000","last_peered":"2024-09-19T08:03:00.838028+0000","last_clean":"2024-09-19T08:03:00.838028+0000","last_became_active":"2024-09-19T08:01:10.770462+0000","last_became_peered":"2024-09-19T08:01:10.770462+0000","last_unstale":"2024-09-19T08:03:00.838028+0000","last_undegraded":"2024-09-19T08:03:00.838028+0000","last_fullsized":"2024-09-19T08:03:00.838028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:04:32.832009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'7","reported_seq":145,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:09.854029+0000","last_change":"2024-09-19T08:01:08.759429+0000","last_active":"2024-09-19T08:03:09.854029+0000","last_peered":"2024-09-19T08:03:09.854029+0000","last_clean":"2024-09-19T08:03:09.854029+0000","last_became_active":"2024-09-19T08:01:08.759062+0000","last_became_peered":"2024-09-19T08:01:08.759062+0000","last_unstale":"2024-09-19T08:03:09.854029+0000","last_undegraded":"2024-09-19T08:03:09.854029+0000","last_fullsized":"2024-09-19T08:03:09.854029+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:16:51.582435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1374,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837660+0000","last_change":"2024-09-19T08:01:10.784735+0000","last_active":"2024-09-19T08:03:00.837660+0000","last_peered":"2024-09-19T08:03:00.837660+0000","last_clean":"2024-09-19T08:03:00.837660+0000","last_became_active":"2024-09-19T08:01:10.784346+0000","last_became_peered":"2024-09-19T08:01:10.784346+0000","last_unstale":"2024-09-19T08:03:00.837660+0000","last_undegraded":"2024-09-19T08:03:00.837660+0000","last_fullsized":"2024-09-19T08:03:00.837660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:27:54.580391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847399+0000","last_change":"2024-09-19T08:01:09.749498+0000","last_active":"2024-09-19T08:03:00.847399+0000","last_peered":"2024-09-19T08:03:00.847399+0000","last_clean":"2024-09-19T08:03:00.847399+0000","last_became_active":"2024-09-19T08:01:09.749091+0000","last_became_peered":"2024-09-19T08:01:09.749091+0000","last_unstale":"2024-09-19T08:03:00.847399+0000","last_undegraded":"2024-09-19T08:03:00.847399+0000","last_fullsized":"2024-09-19T08:03:00.847399+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:56:13.644931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841217+0000","last_change":"2024-09-19T08:01:10.779599+0000","last_active":"2024-09-19T08:03:00.841217+0000","last_peered":"2024-09-19T08:03:00.841217+0000","last_clean":"2024-09-19T08:03:00.841217+0000","last_became_active":"2024-09-19T08:01:10.779219+0000","last_became_peered":"2024-09-19T08:01:10.779219+0000","last_unstale":"2024-09-19T08:03:00.841217+0000","last_undegraded":"2024-09-19T08:03:00.841217+0000","last_fullsized":"2024-09-19T08:03:00.841217+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:20.193753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847308+0000","last_change":"2024-09-19T08:01:08.761432+0000","last_active":"2024-09-19T08:03:00.847308+0000","last_peered":"2024-09-19T08:03:00.847308+0000","last_clean":"2024-09-19T08:03:00.847308+0000","last_became_active":"2024-09-19T08:01:08.761044+0000","last_became_peered":"2024-09-19T08:01:08.761044+0000","last_unstale":"2024-09-19T08:03:00.847308+0000","last_undegraded":"2024-09-19T08:03:00.847308+0000","last_fullsized":"2024-09-19T08:03:00.847308+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:12.237334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780765+0000","last_change":"2024-09-19T08:01:10.772559+0000","last_active":"2024-09-19T08:03:00.780765+0000","last_peered":"2024-09-19T08:03:00.780765+0000","last_clean":"2024-09-19T08:03:00.780765+0000","last_became_active":"2024-09-19T08:01:10.772176+0000","last_became_peered":"2024-09-19T08:01:10.772176+0000","last_unstale":"2024-09-19T08:03:00.780765+0000","last_undegraded":"2024-09-19T08:03:00.780765+0000","last_fullsized":"2024-09-19T08:03:00.780765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:27:16.296127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758996+0000","last_change":"2024-09-19T08:01:08.759989+0000","last_active":"2024-09-19T08:03:00.758996+0000","last_peered":"2024-09-19T08:03:00.758996+0000","last_clean":"2024-09-19T08:03:00.758996+0000","last_became_active":"2024-09-19T08:01:08.759705+0000","last_became_peered":"2024-09-19T08:01:08.759705+0000","last_unstale":"2024-09-19T08:03:00.758996+0000","last_undegraded":"2024-09-19T08:03:00.758996+0000","last_fullsized":"2024-09-19T08:03:00.758996+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:33:25.237567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838150+0000","last_change":"2024-09-19T08:01:10.776528+0000","last_active":"2024-09-19T08:03:00.838150+0000","last_peered":"2024-09-19T08:03:00.838150+0000","last_clean":"2024-09-19T08:03:00.838150+0000","last_became_active":"2024-09-19T08:01:10.776222+0000","last_became_peered":"2024-09-19T08:01:10.776222+0000","last_unstale":"2024-09-19T08:03:00.838150+0000","last_undegraded":"2024-09-19T08:03:00.838150+0000","last_fullsized":"2024-09-19T08:03:00.838150+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.787492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'67","reported_seq":122,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:11.300163+0000","last_change":"2024-09-19T08:01:08.759631+0000","last_active":"2024-09-19T08:03:11.300163+0000","last_peered":"2024-09-19T08:03:11.300163+0000","last_clean":"2024-09-19T08:03:11.300163+0000","last_became_active":"2024-09-19T08:01:08.759156+0000","last_became_peered":"2024-09-19T08:01:08.759156+0000","last_unstale":"2024-09-19T08:03:11.300163+0000","last_undegraded":"2024-09-19T08:03:11.300163+0000","last_fullsized":"2024-09-19T08:03:11.300163+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:48:50.910824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837969+0000","last_change":"2024-09-19T08:01:10.772660+0000","last_active":"2024-09-19T08:03:00.837969+0000","last_peered":"2024-09-19T08:03:00.837969+0000","last_clean":"2024-09-19T08:03:00.837969+0000","last_became_active":"2024-09-19T08:01:10.771816+0000","last_became_peered":"2024-09-19T08:01:10.771816+0000","last_unstale":"2024-09-19T08:03:00.837969+0000","last_undegraded":"2024-09-19T08:03:00.837969+0000","last_fullsized":"2024-09-19T08:03:00.837969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:04:32.667107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'7","reported_seq":144,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:09.883643+0000","last_change":"2024-09-19T08:01:09.749844+0000","last_active":"2024-09-19T08:03:09.883643+0000","last_peered":"2024-09-19T08:03:09.883643+0000","last_clean":"2024-09-19T08:03:09.883643+0000","last_became_active":"2024-09-19T08:01:09.749410+0000","last_became_peered":"2024-09-19T08:01:09.749410+0000","last_unstale":"2024-09-19T08:03:09.883643+0000","last_undegraded":"2024-09-19T08:03:09.883643+0000","last_fullsized":"2024-09-19T08:03:09.883643+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:03:14.989814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1704,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":113,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.838196+0000","last_change":"2024-09-19T08:01:10.773782+0000","last_active":"2024-09-19T08:03:00.838196+0000","last_peered":"2024-09-19T08:03:00.838196+0000","last_clean":"2024-09-19T08:03:00.838196+0000","last_became_active":"2024-09-19T08:01:10.773471+0000","last_became_peered":"2024-09-19T08:01:10.773471+0000","last_unstale":"2024-09-19T08:03:00.838196+0000","last_undegraded":"2024-09-19T08:03:00.838196+0000","last_fullsized":"2024-09-19T08:03:00.838196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:15.780259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"81'148","reported_seq":388,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:02.852179+0000","last_change":"2024-09-19T08:00:01.677517+0000","last_active":"2024-09-19T08:03:02.852179+0000","last_peered":"2024-09-19T08:03:02.852179+0000","last_clean":"2024-09-19T08:03:02.852179+0000","last_became_active":"2024-09-19T08:00:01.677089+0000","last_became_peered":"2024-09-19T08:00:01.677089+0000","last_unstale":"2024-09-19T08:03:02.852179+0000","last_undegraded":"2024-09-19T08:03:02.852179+0000","last_fullsized":"2024-09-19T08:03:02.852179+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:56:43.962142+0000","last_clean_scrub_stamp":"2024-09-19T07:56:43.962142+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:10:36.158811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761731+0000","last_change":"2024-09-19T08:01:08.759819+0000","last_active":"2024-09-19T08:03:00.761731+0000","last_peered":"2024-09-19T08:03:00.761731+0000","last_clean":"2024-09-19T08:03:00.761731+0000","last_became_active":"2024-09-19T08:01:08.759420+0000","last_became_peered":"2024-09-19T08:01:08.759420+0000","last_unstale":"2024-09-19T08:03:00.761731+0000","last_undegraded":"2024-09-19T08:03:00.761731+0000","last_fullsized":"2024-09-19T08:03:00.761731+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:06:16.852323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.843903+0000","last_change":"2024-09-19T08:01:26.724260+0000","last_active":"2024-09-19T08:03:00.843903+0000","last_peered":"2024-09-19T08:03:00.843903+0000","last_clean":"2024-09-19T08:03:00.843903+0000","last_became_active":"2024-09-19T08:01:26.723851+0000","last_became_peered":"2024-09-19T08:01:26.723851+0000","last_unstale":"2024-09-19T08:03:00.843903+0000","last_undegraded":"2024-09-19T08:03:00.843903+0000","last_fullsized":"2024-09-19T08:03:00.843903+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:18:46.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837872+0000","last_change":"2024-09-19T08:01:08.761291+0000","last_active":"2024-09-19T08:03:00.837872+0000","last_peered":"2024-09-19T08:03:00.837872+0000","last_clean":"2024-09-19T08:03:00.837872+0000","last_became_active":"2024-09-19T08:01:08.760238+0000","last_became_peered":"2024-09-19T08:01:08.760238+0000","last_unstale":"2024-09-19T08:03:00.837872+0000","last_undegraded":"2024-09-19T08:03:00.837872+0000","last_fullsized":"2024-09-19T08:03:00.837872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:26:09.980573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.758639+0000","last_change":"2024-09-19T08:01:10.773056+0000","last_active":"2024-09-19T08:03:00.758639+0000","last_peered":"2024-09-19T08:03:00.758639+0000","last_clean":"2024-09-19T08:03:00.758639+0000","last_became_active":"2024-09-19T08:01:10.772431+0000","last_became_peered":"2024-09-19T08:01:10.772431+0000","last_unstale":"2024-09-19T08:03:00.758639+0000","last_undegraded":"2024-09-19T08:03:00.758639+0000","last_fullsized":"2024-09-19T08:03:00.758639+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:39:06.878086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":143,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.854713+0000","last_change":"2024-09-19T08:01:09.749707+0000","last_active":"2024-09-19T08:03:10.854713+0000","last_peered":"2024-09-19T08:03:10.854713+0000","last_clean":"2024-09-19T08:03:10.854713+0000","last_became_active":"2024-09-19T08:01:09.749013+0000","last_became_peered":"2024-09-19T08:01:09.749013+0000","last_unstale":"2024-09-19T08:03:10.854713+0000","last_undegraded":"2024-09-19T08:03:10.854713+0000","last_fullsized":"2024-09-19T08:03:10.854713+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:40:12.682158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841250+0000","last_change":"2024-09-19T08:01:10.774101+0000","last_active":"2024-09-19T08:03:00.841250+0000","last_peered":"2024-09-19T08:03:00.841250+0000","last_clean":"2024-09-19T08:03:00.841250+0000","last_became_active":"2024-09-19T08:01:10.773822+0000","last_became_peered":"2024-09-19T08:01:10.773822+0000","last_unstale":"2024-09-19T08:03:00.841250+0000","last_undegraded":"2024-09-19T08:03:00.841250+0000","last_fullsized":"2024-09-19T08:03:00.841250+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:25:29.571118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"81'6","reported_seq":142,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.855374+0000","last_change":"2024-09-19T08:01:08.761675+0000","last_active":"2024-09-19T08:03:10.855374+0000","last_peered":"2024-09-19T08:03:10.855374+0000","last_clean":"2024-09-19T08:03:10.855374+0000","last_became_active":"2024-09-19T08:01:08.761346+0000","last_became_peered":"2024-09-19T08:01:08.761346+0000","last_unstale":"2024-09-19T08:03:10.855374+0000","last_undegraded":"2024-09-19T08:03:10.855374+0000","last_fullsized":"2024-09-19T08:03:10.855374+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:51:19.237458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839233+0000","last_change":"2024-09-19T08:01:10.781565+0000","last_active":"2024-09-19T08:03:00.839233+0000","last_peered":"2024-09-19T08:03:00.839233+0000","last_clean":"2024-09-19T08:03:00.839233+0000","last_became_active":"2024-09-19T08:01:10.781301+0000","last_became_peered":"2024-09-19T08:01:10.781301+0000","last_unstale":"2024-09-19T08:03:00.839233+0000","last_undegraded":"2024-09-19T08:03:00.839233+0000","last_fullsized":"2024-09-19T08:03:00.839233+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:55:52.932009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.844577+0000","last_change":"2024-09-19T08:01:09.748918+0000","last_active":"2024-09-19T08:03:00.844577+0000","last_peered":"2024-09-19T08:03:00.844577+0000","last_clean":"2024-09-19T08:03:00.844577+0000","last_became_active":"2024-09-19T08:01:09.748497+0000","last_became_peered":"2024-09-19T08:01:09.748497+0000","last_unstale":"2024-09-19T08:03:00.844577+0000","last_undegraded":"2024-09-19T08:03:00.844577+0000","last_fullsized":"2024-09-19T08:03:00.844577+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:20:31.030133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.839231+0000","last_change":"2024-09-19T08:01:10.781388+0000","last_active":"2024-09-19T08:03:00.839231+0000","last_peered":"2024-09-19T08:03:00.839231+0000","last_clean":"2024-09-19T08:03:00.839231+0000","last_became_active":"2024-09-19T08:01:10.781063+0000","last_became_peered":"2024-09-19T08:01:10.781063+0000","last_unstale":"2024-09-19T08:03:00.839231+0000","last_undegraded":"2024-09-19T08:03:00.839231+0000","last_fullsized":"2024-09-19T08:03:00.839231+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:12:08.852717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847755+0000","last_change":"2024-09-19T08:01:09.750080+0000","last_active":"2024-09-19T08:03:00.847755+0000","last_peered":"2024-09-19T08:03:00.847755+0000","last_clean":"2024-09-19T08:03:00.847755+0000","last_became_active":"2024-09-19T08:01:09.749053+0000","last_became_peered":"2024-09-19T08:01:09.749053+0000","last_unstale":"2024-09-19T08:03:00.847755+0000","last_undegraded":"2024-09-19T08:03:00.847755+0000","last_fullsized":"2024-09-19T08:03:00.847755+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:15:45.292714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.759144+0000","last_change":"2024-09-19T08:01:10.767070+0000","last_active":"2024-09-19T08:03:00.759144+0000","last_peered":"2024-09-19T08:03:00.759144+0000","last_clean":"2024-09-19T08:03:00.759144+0000","last_became_active":"2024-09-19T08:01:10.766760+0000","last_became_peered":"2024-09-19T08:01:10.766760+0000","last_unstale":"2024-09-19T08:03:00.759144+0000","last_undegraded":"2024-09-19T08:03:00.759144+0000","last_fullsized":"2024-09-19T08:03:00.759144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:17.841135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'105","reported_seq":221,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:11.632933+0000","last_change":"2024-09-19T08:01:08.756143+0000","last_active":"2024-09-19T08:03:11.632933+0000","last_peered":"2024-09-19T08:03:11.632933+0000","last_clean":"2024-09-19T08:03:11.632933+0000","last_became_active":"2024-09-19T08:01:08.755777+0000","last_became_peered":"2024-09-19T08:01:08.755777+0000","last_unstale":"2024-09-19T08:03:11.632933+0000","last_undegraded":"2024-09-19T08:03:11.632933+0000","last_fullsized":"2024-09-19T08:03:11.632933+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:21.430673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":191,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849237+0000","last_change":"2024-09-19T08:01:10.774140+0000","last_active":"2024-09-19T08:03:00.849237+0000","last_peered":"2024-09-19T08:03:00.849237+0000","last_clean":"2024-09-19T08:03:00.849237+0000","last_became_active":"2024-09-19T08:01:10.773819+0000","last_became_peered":"2024-09-19T08:01:10.773819+0000","last_unstale":"2024-09-19T08:03:00.849237+0000","last_undegraded":"2024-09-19T08:03:00.849237+0000","last_fullsized":"2024-09-19T08:03:00.849237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:15:13.174013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.761607+0000","last_change":"2024-09-19T08:01:09.169129+0000","last_active":"2024-09-19T08:03:00.761607+0000","last_peered":"2024-09-19T08:03:00.761607+0000","last_clean":"2024-09-19T08:03:00.761607+0000","last_became_active":"2024-09-19T08:01:09.168734+0000","last_became_peered":"2024-09-19T08:01:09.168734+0000","last_unstale":"2024-09-19T08:03:00.761607+0000","last_undegraded":"2024-09-19T08:03:00.761607+0000","last_fullsized":"2024-09-19T08:03:00.761607+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:01:52.232391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994759+0000","last_change":"2024-09-19T08:01:10.768592+0000","last_active":"2024-09-19T08:03:00.994759+0000","last_peered":"2024-09-19T08:03:00.994759+0000","last_clean":"2024-09-19T08:03:00.994759+0000","last_became_active":"2024-09-19T08:01:10.767645+0000","last_became_peered":"2024-09-19T08:01:10.767645+0000","last_unstale":"2024-09-19T08:03:00.994759+0000","last_undegraded":"2024-09-19T08:03:00.994759+0000","last_fullsized":"2024-09-19T08:03:00.994759+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:08:46.227461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997364+0000","last_change":"2024-09-19T08:01:09.749240+0000","last_active":"2024-09-19T08:03:00.997364+0000","last_peered":"2024-09-19T08:03:00.997364+0000","last_clean":"2024-09-19T08:03:00.997364+0000","last_became_active":"2024-09-19T08:01:09.748945+0000","last_became_peered":"2024-09-19T08:01:09.748945+0000","last_unstale":"2024-09-19T08:03:00.997364+0000","last_undegraded":"2024-09-19T08:03:00.997364+0000","last_fullsized":"2024-09-19T08:03:00.997364+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:25:13.791290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847326+0000","last_change":"2024-09-19T08:01:10.783133+0000","last_active":"2024-09-19T08:03:00.847326+0000","last_peered":"2024-09-19T08:03:00.847326+0000","last_clean":"2024-09-19T08:03:00.847326+0000","last_became_active":"2024-09-19T08:01:10.782797+0000","last_became_peered":"2024-09-19T08:01:10.782797+0000","last_unstale":"2024-09-19T08:03:00.847326+0000","last_undegraded":"2024-09-19T08:03:00.847326+0000","last_fullsized":"2024-09-19T08:03:00.847326+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:30:43.551877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848489+0000","last_change":"2024-09-19T08:01:08.757964+0000","last_active":"2024-09-19T08:03:00.848489+0000","last_peered":"2024-09-19T08:03:00.848489+0000","last_clean":"2024-09-19T08:03:00.848489+0000","last_became_active":"2024-09-19T08:01:08.757526+0000","last_became_peered":"2024-09-19T08:01:08.757526+0000","last_unstale":"2024-09-19T08:03:00.848489+0000","last_undegraded":"2024-09-19T08:03:00.848489+0000","last_fullsized":"2024-09-19T08:03:00.848489+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:03.565702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780557+0000","last_change":"2024-09-19T08:01:10.769974+0000","last_active":"2024-09-19T08:03:00.780557+0000","last_peered":"2024-09-19T08:03:00.780557+0000","last_clean":"2024-09-19T08:03:00.780557+0000","last_became_active":"2024-09-19T08:01:10.769650+0000","last_became_peered":"2024-09-19T08:01:10.769650+0000","last_unstale":"2024-09-19T08:03:00.780557+0000","last_undegraded":"2024-09-19T08:03:00.780557+0000","last_fullsized":"2024-09-19T08:03:00.780557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:01.755198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"81'4","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.760043+0000","last_change":"2024-09-19T08:01:08.763476+0000","last_active":"2024-09-19T08:03:00.760043+0000","last_peered":"2024-09-19T08:03:00.760043+0000","last_clean":"2024-09-19T08:03:00.760043+0000","last_became_active":"2024-09-19T08:01:08.763154+0000","last_became_peered":"2024-09-19T08:01:08.763154+0000","last_unstale":"2024-09-19T08:03:00.760043+0000","last_undegraded":"2024-09-19T08:03:00.760043+0000","last_fullsized":"2024-09-19T08:03:00.760043+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:06.202452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.848751+0000","last_change":"2024-09-19T08:01:10.784201+0000","last_active":"2024-09-19T08:03:00.848751+0000","last_peered":"2024-09-19T08:03:00.848751+0000","last_clean":"2024-09-19T08:03:00.848751+0000","last_became_active":"2024-09-19T08:01:10.783867+0000","last_became_peered":"2024-09-19T08:01:10.783867+0000","last_unstale":"2024-09-19T08:03:00.848751+0000","last_undegraded":"2024-09-19T08:03:00.848751+0000","last_fullsized":"2024-09-19T08:03:00.848751+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:43:50.082183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847290+0000","last_change":"2024-09-19T08:01:08.760479+0000","last_active":"2024-09-19T08:03:00.847290+0000","last_peered":"2024-09-19T08:03:00.847290+0000","last_clean":"2024-09-19T08:03:00.847290+0000","last_became_active":"2024-09-19T08:01:08.760004+0000","last_became_peered":"2024-09-19T08:01:08.760004+0000","last_unstale":"2024-09-19T08:03:00.847290+0000","last_undegraded":"2024-09-19T08:03:00.847290+0000","last_fullsized":"2024-09-19T08:03:00.847290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:41:19.956623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.837851+0000","last_change":"2024-09-19T08:01:10.772184+0000","last_active":"2024-09-19T08:03:00.837851+0000","last_peered":"2024-09-19T08:03:00.837851+0000","last_clean":"2024-09-19T08:03:00.837851+0000","last_became_active":"2024-09-19T08:01:10.771825+0000","last_became_peered":"2024-09-19T08:01:10.771825+0000","last_unstale":"2024-09-19T08:03:00.837851+0000","last_undegraded":"2024-09-19T08:03:00.837851+0000","last_fullsized":"2024-09-19T08:03:00.837851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:36.764575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.841567+0000","last_change":"2024-09-19T08:01:10.779368+0000","last_active":"2024-09-19T08:03:00.841567+0000","last_peered":"2024-09-19T08:03:00.841567+0000","last_clean":"2024-09-19T08:03:00.841567+0000","last_became_active":"2024-09-19T08:01:10.778884+0000","last_became_peered":"2024-09-19T08:01:10.778884+0000","last_unstale":"2024-09-19T08:03:00.841567+0000","last_undegraded":"2024-09-19T08:03:00.841567+0000","last_fullsized":"2024-09-19T08:03:00.841567+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:28:29.630300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.780536+0000","last_change":"2024-09-19T08:01:26.741053+0000","last_active":"2024-09-19T08:03:00.780536+0000","last_peered":"2024-09-19T08:03:00.780536+0000","last_clean":"2024-09-19T08:03:00.780536+0000","last_became_active":"2024-09-19T08:01:26.724714+0000","last_became_peered":"2024-09-19T08:01:26.724714+0000","last_unstale":"2024-09-19T08:03:00.780536+0000","last_undegraded":"2024-09-19T08:03:00.780536+0000","last_fullsized":"2024-09-19T08:03:00.780536+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:49:59.583797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.997323+0000","last_change":"2024-09-19T08:01:10.781389+0000","last_active":"2024-09-19T08:03:00.997323+0000","last_peered":"2024-09-19T08:03:00.997323+0000","last_clean":"2024-09-19T08:03:00.997323+0000","last_became_active":"2024-09-19T08:01:10.781080+0000","last_became_peered":"2024-09-19T08:01:10.781080+0000","last_unstale":"2024-09-19T08:03:00.997323+0000","last_undegraded":"2024-09-19T08:03:00.997323+0000","last_fullsized":"2024-09-19T08:03:00.997323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:40:45.225957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.847701+0000","last_change":"2024-09-19T08:01:26.768102+0000","last_active":"2024-09-19T08:03:00.847701+0000","last_peered":"2024-09-19T08:03:00.847701+0000","last_clean":"2024-09-19T08:03:00.847701+0000","last_became_active":"2024-09-19T08:01:26.759462+0000","last_became_peered":"2024-09-19T08:01:26.759462+0000","last_unstale":"2024-09-19T08:03:00.847701+0000","last_undegraded":"2024-09-19T08:03:00.847701+0000","last_fullsized":"2024-09-19T08:03:00.847701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:02:06.815238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.994921+0000","last_change":"2024-09-19T08:01:10.771031+0000","last_active":"2024-09-19T08:03:00.994921+0000","last_peered":"2024-09-19T08:03:00.994921+0000","last_clean":"2024-09-19T08:03:00.994921+0000","last_became_active":"2024-09-19T08:01:10.770675+0000","last_became_peered":"2024-09-19T08:01:10.770675+0000","last_unstale":"2024-09-19T08:03:00.994921+0000","last_undegraded":"2024-09-19T08:03:00.994921+0000","last_fullsized":"2024-09-19T08:03:00.994921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:42.203848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849256+0000","last_change":"2024-09-19T08:01:09.748622+0000","last_active":"2024-09-19T08:03:00.849256+0000","last_peered":"2024-09-19T08:03:00.849256+0000","last_clean":"2024-09-19T08:03:00.849256+0000","last_became_active":"2024-09-19T08:01:09.748202+0000","last_became_peered":"2024-09-19T08:01:09.748202+0000","last_unstale":"2024-09-19T08:03:00.849256+0000","last_undegraded":"2024-09-19T08:03:00.849256+0000","last_fullsized":"2024-09-19T08:03:00.849256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:17:39.861820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:00.849123+0000","last_change":"2024-09-19T08:01:10.768949+0000","last_active":"2024-09-19T08:03:00.849123+0000","last_peered":"2024-09-19T08:03:00.849123+0000","last_clean":"2024-09-19T08:03:00.849123+0000","last_became_active":"2024-09-19T08:01:10.768159+0000","last_became_peered":"2024-09-19T08:01:10.768159+0000","last_unstale":"2024-09-19T08:03:00.849123+0000","last_undegraded":"2024-09-19T08:03:00.849123+0000","last_fullsized":"2024-09-19T08:03:00.849123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:09.747432+0000","last_clean_scrub_stamp":"2024-09-19T08:01:09.747432+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:20:43.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"81'14","reported_seq":176,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-19T08:03:10.854932+0000","last_change":"2024-09-19T08:01:26.735877+0000","last_active":"2024-09-19T08:03:10.854932+0000","last_peered":"2024-09-19T08:03:10.854932+0000","last_clean":"2024-09-19T08:03:10.854932+0000","last_became_active":"2024-09-19T08:01:26.723597+0000","last_became_peered":"2024-09-19T08:01:26.723597+0000","last_unstale":"2024-09-19T08:03:10.854932+0000","last_undegraded":"2024-09-19T08:03:10.854932+0000","last_fullsized":"2024-09-19T08:03:10.854932+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T08:01:07.743976+0000","last_clean_scrub_stamp":"2024-09-19T08:01:07.743976+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:43:42.340832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":110,"num_read_kb":110,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,1],"acting":[9,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":800997,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1499,"num_read_kb":974,"num_write":507,"num_write_kb":2386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1254,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2764800,"data_stored":2402991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645349,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":905216,"data_stored":436392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874282,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33192,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93711960,"statfs":{"total":95995035648,"available":95961047040,"internally_reserved":0,"allocated":1933312,"data_stored":1424044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005103,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32464,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712688,"statfs":{"total":95995035648,"available":95961792512,"internally_reserved":0,"allocated":1310720,"data_stored":827750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234035,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1294336,"data_stored":826874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462968,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1503232,"data_stored":1029567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593790,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36228,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708924,"statfs":{"total":95995035648,"available":95957938176,"internally_reserved":0,"allocated":901120,"data_stored":434644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822722,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28836,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716316,"statfs":{"total":95995035648,"available":95965507584,"internally_reserved":0,"allocated":1855488,"data_stored":1356970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986247,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":901120,"data_stored":434898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215179,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32908,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712244,"statfs":{"total":95995035648,"available":95961337856,"internally_reserved":0,"allocated":1642496,"data_stored":1155524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":507332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607638,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":978944,"data_stored":505622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":921600,"data_stored":436124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":72210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":73920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":753664,"data_stored":721878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":333073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":400147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T08:03:17.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:03:17.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:17 smithi098 ceph-mon[31333]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:17.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:17 smithi098 ceph-mon[31333]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-09-19T08:03:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:17.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:17 smithi162 ceph-mon[35260]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:17.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:17 smithi162 ceph-mon[35260]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-09-19T08:03:17.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:17 smithi139 ceph-mon[34874]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T08:03:17.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:17 smithi139 ceph-mon[34874]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-09-19T08:03:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.864+0000 7f2757e28640 1 -- 172.21.15.98:0/2191577830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2750100b30 msgr2=0x7f2750100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.864+0000 7f2757e28640 1 --2- 172.21.15.98:0/2191577830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2750100b30 0x7f2750100f10 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2744009900 tx=0x7f274402a470 comp rx=0 tx=0).stop 2024-09-19T08:03:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.864+0000 7f2757e28640 1 -- 172.21.15.98:0/2191577830 shutdown_connections 2024-09-19T08:03:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.864+0000 7f2757e28640 1 --2- 172.21.15.98:0/2191577830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2750105f40 0x7f27501063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.864+0000 7f2757e28640 1 --2- 172.21.15.98:0/2191577830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2750101450 0x7f2750105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.864+0000 7f2757e28640 1 --2- 172.21.15.98:0/2191577830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2750100b30 0x7f2750100f10 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.864+0000 7f2757e28640 1 -- 172.21.15.98:0/2191577830 >> 172.21.15.98:0/2191577830 conn(0x7f27500fc7a0 msgr2=0x7f27500febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.865+0000 7f2757e28640 1 -- 172.21.15.98:0/2191577830 shutdown_connections 2024-09-19T08:03:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.865+0000 7f2757e28640 1 -- 172.21.15.98:0/2191577830 wait complete. 2024-09-19T08:03:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.865+0000 7f2757e28640 1 Processor -- start 2024-09-19T08:03:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.865+0000 7f2757e28640 1 -- start start 2024-09-19T08:03:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f2757e28640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2750100b30 0x7f27501a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f2757e28640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2750101450 0x7f27501a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f2757e28640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2750105f40 0x7f275019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f2757e28640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2750118570 con 0x7f2750105f40 2024-09-19T08:03:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f2757e28640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f27501183f0 con 0x7f2750100b30 2024-09-19T08:03:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f2757e28640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f27501186f0 con 0x7f2750101450 2024-09-19T08:03:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f27563c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2750105f40 0x7f275019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f27563c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2750105f40 0x7f275019e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56082/0 (socket says 172.21.15.98:56082) 2024-09-19T08:03:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f27563c5640 1 -- 172.21.15.98:0/3954443212 learned_addr learned my addr 172.21.15.98:0/3954443212 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f27553c3640 1 --2- 172.21.15.98:0/3954443212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2750101450 0x7f27501a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f2755bc4640 1 --2- 172.21.15.98:0/3954443212 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2750100b30 0x7f27501a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f27563c5640 1 -- 172.21.15.98:0/3954443212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2750101450 msgr2=0x7f27501a4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f27563c5640 1 --2- 172.21.15.98:0/3954443212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2750101450 0x7f27501a4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f27563c5640 1 -- 172.21.15.98:0/3954443212 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2750100b30 msgr2=0x7f27501a4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f27563c5640 1 --2- 172.21.15.98:0/3954443212 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2750100b30 0x7f27501a4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f27563c5640 1 -- 172.21.15.98:0/3954443212 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275019f050 con 0x7f2750105f40 2024-09-19T08:03:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f27563c5640 1 --2- 172.21.15.98:0/3954443212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2750105f40 0x7f275019e920 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f274800eaa0 tx=0x7f274800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.866+0000 7f2742ffd640 1 -- 172.21.15.98:0/3954443212 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274800cdb0 con 0x7f2750105f40 2024-09-19T08:03:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.867+0000 7f2742ffd640 1 -- 172.21.15.98:0/3954443212 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2748005020 con 0x7f2750105f40 2024-09-19T08:03:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.867+0000 7f2742ffd640 1 -- 172.21.15.98:0/3954443212 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27480116b0 con 0x7f2750105f40 2024-09-19T08:03:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.867+0000 7f2757e28640 1 -- 172.21.15.98:0/3954443212 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f275019f340 con 0x7f2750105f40 2024-09-19T08:03:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.867+0000 7f2757e28640 1 -- 172.21.15.98:0/3954443212 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27501ab630 con 0x7f2750105f40 2024-09-19T08:03:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.868+0000 7f2757e28640 1 -- 172.21.15.98:0/3954443212 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2750105920 con 0x7f2750105f40 2024-09-19T08:03:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.869+0000 7f2742ffd640 1 -- 172.21.15.98:0/3954443212 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f2748005560 con 0x7f2750105f40 2024-09-19T08:03:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.870+0000 7f2742ffd640 1 --2- 172.21.15.98:0/3954443212 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f273c0a3550 0x7f273c0a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.870+0000 7f2755bc4640 1 --2- 172.21.15.98:0/3954443212 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f273c0a3550 0x7f273c0a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.870+0000 7f2742ffd640 1 -- 172.21.15.98:0/3954443212 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f27480d0050 con 0x7f2750105f40 2024-09-19T08:03:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.870+0000 7f2755bc4640 1 --2- 172.21.15.98:0/3954443212 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f273c0a3550 0x7f273c0a5a10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f275019fa60 tx=0x7f27440058d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:17.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:17.874+0000 7f2742ffd640 1 -- 172.21.15.98:0/3954443212 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2748016030 con 0x7f2750105f40 2024-09-19T08:03:18.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:18.075+0000 7f2757e28640 1 -- 172.21.15.98:0/3954443212 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f275019fe50 con 0x7f2750105f40 2024-09-19T08:03:18.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:18.077+0000 7f2742ffd640 1 -- 172.21.15.98:0/3954443212 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11983 (secure 0 0 0) 0x7f274809c1c0 con 0x7f2750105f40 2024-09-19T08:03:18.077 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:18.077 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"btime":"2024-09-19T08:02:59:855935+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:02:59.805764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:03:18.078 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-09-19T08:03:18.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:18.079+0000 7f2757e28640 1 -- 172.21.15.98:0/3954443212 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f273c0a3550 msgr2=0x7f273c0a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:18.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:18.079+0000 7f2757e28640 1 --2- 172.21.15.98:0/3954443212 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f273c0a3550 0x7f273c0a5a10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f275019fa60 tx=0x7f27440058d0 comp rx=0 tx=0).stop 2024-09-19T08:03:18.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:18.079+0000 7f2757e28640 1 -- 172.21.15.98:0/3954443212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2750105f40 msgr2=0x7f275019e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:18.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:18.079+0000 7f2757e28640 1 --2- 172.21.15.98:0/3954443212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2750105f40 0x7f275019e920 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f274800eaa0 tx=0x7f274800ef60 comp rx=0 tx=0).stop 2024-09-19T08:03:18.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:18.079+0000 7f2757e28640 1 -- 172.21.15.98:0/3954443212 shutdown_connections 2024-09-19T08:03:18.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:18.079+0000 7f2757e28640 1 --2- 172.21.15.98:0/3954443212 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f273c0a3550 0x7f273c0a5a10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:18.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:18.079+0000 7f2757e28640 1 --2- 172.21.15.98:0/3954443212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2750105f40 0x7f275019e920 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:18.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:18.079+0000 7f2757e28640 1 --2- 172.21.15.98:0/3954443212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2750101450 0x7f27501a4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:18.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:18.079+0000 7f2757e28640 1 --2- 172.21.15.98:0/3954443212 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2750100b30 0x7f27501a4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:18.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:18.079+0000 7f2757e28640 1 -- 172.21.15.98:0/3954443212 >> 172.21.15.98:0/3954443212 conn(0x7f27500fc7a0 msgr2=0x7f27500fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:18.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:18.079+0000 7f2757e28640 1 -- 172.21.15.98:0/3954443212 shutdown_connections 2024-09-19T08:03:18.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:18.079+0000 7f2757e28640 1 -- 172.21.15.98:0/3954443212 wait complete. 2024-09-19T08:03:18.560 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:18 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3954443212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:18.648 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:18 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3954443212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:18.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:18 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3954443212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:18.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:03:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:19.485 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:19 smithi098 ceph-mon[31333]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-09-19T08:03:19.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.564+0000 7f003c151640 1 -- 172.21.15.98:0/2219198303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0034101de0 msgr2=0x7f00341021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:19.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.564+0000 7f003c151640 1 --2- 172.21.15.98:0/2219198303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0034101de0 0x7f00341021c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0024009900 tx=0x7f002402a410 comp rx=0 tx=0).stop 2024-09-19T08:03:19.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.564+0000 7f003c151640 1 -- 172.21.15.98:0/2219198303 shutdown_connections 2024-09-19T08:03:19.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.564+0000 7f003c151640 1 --2- 172.21.15.98:0/2219198303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0034106f10 0x7f003410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:19.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.564+0000 7f003c151640 1 --2- 172.21.15.98:0/2219198303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0034102790 0x7f00341067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:19.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.564+0000 7f003c151640 1 --2- 172.21.15.98:0/2219198303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0034101de0 0x7f00341021c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:19.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.564+0000 7f003c151640 1 -- 172.21.15.98:0/2219198303 >> 172.21.15.98:0/2219198303 conn(0x7f00340fd8c0 msgr2=0x7f00340ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:19.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.564+0000 7f003c151640 1 -- 172.21.15.98:0/2219198303 shutdown_connections 2024-09-19T08:03:19.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.564+0000 7f003c151640 1 -- 172.21.15.98:0/2219198303 wait complete. 2024-09-19T08:03:19.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f003c151640 1 Processor -- start 2024-09-19T08:03:19.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f003c151640 1 -- start start 2024-09-19T08:03:19.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f003c151640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0034101de0 0x7f0034198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:19.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f003c151640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0034102790 0x7f0034198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:19.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f003c151640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0034106f10 0x7f003419cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:19.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f003c151640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0034111f30 con 0x7f0034101de0 2024-09-19T08:03:19.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f003c151640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0034111db0 con 0x7f0034102790 2024-09-19T08:03:19.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f003c151640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f00341120b0 con 0x7f0034106f10 2024-09-19T08:03:19.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f0039eed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0034101de0 0x7f0034198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f0039eed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0034101de0 0x7f0034198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56096/0 (socket says 172.21.15.98:56096) 2024-09-19T08:03:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f0039eed640 1 -- 172.21.15.98:0/899594094 learned_addr learned my addr 172.21.15.98:0/899594094 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f00396ec640 1 --2- 172.21.15.98:0/899594094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0034102790 0x7f0034198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f0039eed640 1 -- 172.21.15.98:0/899594094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0034106f10 msgr2=0x7f003419cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f003a6ee640 1 --2- 172.21.15.98:0/899594094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0034106f10 0x7f003419cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f0039eed640 1 --2- 172.21.15.98:0/899594094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0034106f10 0x7f003419cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f0039eed640 1 -- 172.21.15.98:0/899594094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0034102790 msgr2=0x7f0034198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f0039eed640 1 --2- 172.21.15.98:0/899594094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0034102790 0x7f0034198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f0039eed640 1 -- 172.21.15.98:0/899594094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003419d4c0 con 0x7f0034101de0 2024-09-19T08:03:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.565+0000 7f0039eed640 1 --2- 172.21.15.98:0/899594094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0034101de0 0x7f0034198510 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f0024006010 tx=0x7f00240042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.566+0000 7f002affd640 1 -- 172.21.15.98:0/899594094 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002403f030 con 0x7f0034101de0 2024-09-19T08:03:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.566+0000 7f002affd640 1 -- 172.21.15.98:0/899594094 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0024003ce0 con 0x7f0034101de0 2024-09-19T08:03:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.566+0000 7f003c151640 1 -- 172.21.15.98:0/899594094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f003419d750 con 0x7f0034101de0 2024-09-19T08:03:19.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.566+0000 7f002affd640 1 -- 172.21.15.98:0/899594094 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0024035070 con 0x7f0034101de0 2024-09-19T08:03:19.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.566+0000 7f003c151640 1 -- 172.21.15.98:0/899594094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00341a5090 con 0x7f0034101de0 2024-09-19T08:03:19.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.567+0000 7f003c151640 1 -- 172.21.15.98:0/899594094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00341032c0 con 0x7f0034101de0 2024-09-19T08:03:19.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.568+0000 7f002affd640 1 -- 172.21.15.98:0/899594094 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f0024042030 con 0x7f0034101de0 2024-09-19T08:03:19.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.569+0000 7f002affd640 1 --2- 172.21.15.98:0/899594094 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f00200a34d0 0x7f00200a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:19.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.569+0000 7f00396ec640 1 --2- 172.21.15.98:0/899594094 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f00200a34d0 0x7f00200a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:19.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.569+0000 7f002affd640 1 -- 172.21.15.98:0/899594094 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f00240f0830 con 0x7f0034101de0 2024-09-19T08:03:19.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.570+0000 7f00396ec640 1 --2- 172.21.15.98:0/899594094 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f00200a34d0 0x7f00200a5990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00341087b0 tx=0x7f003000a2d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:19.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.572+0000 7f002affd640 1 -- 172.21.15.98:0/899594094 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f00240bd180 con 0x7f0034101de0 2024-09-19T08:03:19.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:19 smithi162 ceph-mon[35260]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-09-19T08:03:19.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:19 smithi139 ceph-mon[34874]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-09-19T08:03:19.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.774+0000 7f003c151640 1 -- 172.21.15.98:0/899594094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00341021c0 con 0x7f0034101de0 2024-09-19T08:03:19.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.775+0000 7f002affd640 1 -- 172.21.15.98:0/899594094 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11983 (secure 0 0 0) 0x7f00240bca80 con 0x7f0034101de0 2024-09-19T08:03:19.776 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:19.776 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"btime":"2024-09-19T08:02:59:855935+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:02:59.805764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:03:19.777 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-09-19T08:03:19.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.777+0000 7f003c151640 1 -- 172.21.15.98:0/899594094 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f00200a34d0 msgr2=0x7f00200a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:19.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.777+0000 7f003c151640 1 --2- 172.21.15.98:0/899594094 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f00200a34d0 0x7f00200a5990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00341087b0 tx=0x7f003000a2d0 comp rx=0 tx=0).stop 2024-09-19T08:03:19.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.777+0000 7f003c151640 1 -- 172.21.15.98:0/899594094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0034101de0 msgr2=0x7f0034198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:19.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.777+0000 7f003c151640 1 --2- 172.21.15.98:0/899594094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0034101de0 0x7f0034198510 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f0024006010 tx=0x7f00240042f0 comp rx=0 tx=0).stop 2024-09-19T08:03:19.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.777+0000 7f003c151640 1 -- 172.21.15.98:0/899594094 shutdown_connections 2024-09-19T08:03:19.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.777+0000 7f003c151640 1 --2- 172.21.15.98:0/899594094 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f00200a34d0 0x7f00200a5990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:19.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.777+0000 7f003c151640 1 --2- 172.21.15.98:0/899594094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0034106f10 0x7f003419cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:19.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.777+0000 7f003c151640 1 --2- 172.21.15.98:0/899594094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0034102790 0x7f0034198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:19.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.777+0000 7f003c151640 1 --2- 172.21.15.98:0/899594094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0034101de0 0x7f0034198510 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:19.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.777+0000 7f003c151640 1 -- 172.21.15.98:0/899594094 >> 172.21.15.98:0/899594094 conn(0x7f00340fd8c0 msgr2=0x7f00340fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:19.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.777+0000 7f003c151640 1 -- 172.21.15.98:0/899594094 shutdown_connections 2024-09-19T08:03:19.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:19.777+0000 7f003c151640 1 -- 172.21.15.98:0/899594094 wait complete. 2024-09-19T08:03:20.381 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:20 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/899594094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:20.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T08:03:20.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:20 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/899594094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:20.660 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:20.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:20 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/899594094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:21.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.166+0000 7f7043798640 1 -- 172.21.15.98:0/33588591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f703c102570 msgr2=0x7f703c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:21.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.166+0000 7f7043798640 1 --2- 172.21.15.98:0/33588591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f703c102570 0x7f703c1029b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7030009900 tx=0x7f703002a480 comp rx=0 tx=0).stop 2024-09-19T08:03:21.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.166+0000 7f7043798640 1 -- 172.21.15.98:0/33588591 shutdown_connections 2024-09-19T08:03:21.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.166+0000 7f7043798640 1 --2- 172.21.15.98:0/33588591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f703c102ef0 0x7f703c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:21.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.166+0000 7f7043798640 1 --2- 172.21.15.98:0/33588591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f703c102570 0x7f703c1029b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:21.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.166+0000 7f7043798640 1 --2- 172.21.15.98:0/33588591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f703c100e20 0x7f703c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:21.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.166+0000 7f7043798640 1 -- 172.21.15.98:0/33588591 >> 172.21.15.98:0/33588591 conn(0x7f703c0fc7d0 msgr2=0x7f703c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:21.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.166+0000 7f7043798640 1 -- 172.21.15.98:0/33588591 shutdown_connections 2024-09-19T08:03:21.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.166+0000 7f7043798640 1 -- 172.21.15.98:0/33588591 wait complete. 2024-09-19T08:03:21.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7043798640 1 Processor -- start 2024-09-19T08:03:21.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7043798640 1 -- start start 2024-09-19T08:03:21.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7043798640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f703c100e20 0x7f703c197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:21.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7043798640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f703c102570 0x7f703c197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:21.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7043798640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f703c102ef0 0x7f703c197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:21.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7043798640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f703c110f80 con 0x7f703c102570 2024-09-19T08:03:21.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7043798640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f703c110e00 con 0x7f703c102ef0 2024-09-19T08:03:21.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7043798640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f703c111100 con 0x7f703c100e20 2024-09-19T08:03:21.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7040d33640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f703c102570 0x7f703c197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:21.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7040d33640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f703c102570 0x7f703c197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56108/0 (socket says 172.21.15.98:56108) 2024-09-19T08:03:21.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7040d33640 1 -- 172.21.15.98:0/1269163408 learned_addr learned my addr 172.21.15.98:0/1269163408 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:21.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7041d35640 1 --2- 172.21.15.98:0/1269163408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f703c102ef0 0x7f703c197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:21.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7041534640 1 --2- 172.21.15.98:0/1269163408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f703c100e20 0x7f703c197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:21.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7040d33640 1 -- 172.21.15.98:0/1269163408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f703c100e20 msgr2=0x7f703c197100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:21.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7040d33640 1 --2- 172.21.15.98:0/1269163408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f703c100e20 0x7f703c197100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:21.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7040d33640 1 -- 172.21.15.98:0/1269163408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f703c102ef0 msgr2=0x7f703c197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:21.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7040d33640 1 --2- 172.21.15.98:0/1269163408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f703c102ef0 0x7f703c197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:21.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7040d33640 1 -- 172.21.15.98:0/1269163408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703c1a6970 con 0x7f703c102570 2024-09-19T08:03:21.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.167+0000 7f7040d33640 1 --2- 172.21.15.98:0/1269163408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f703c102570 0x7f703c197640 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7030006010 tx=0x7f7030003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:21.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.168+0000 7f703a7fc640 1 -- 172.21.15.98:0/1269163408 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703003f070 con 0x7f703c102570 2024-09-19T08:03:21.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.168+0000 7f7043798640 1 -- 172.21.15.98:0/1269163408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f703c1a6c00 con 0x7f703c102570 2024-09-19T08:03:21.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.168+0000 7f703a7fc640 1 -- 172.21.15.98:0/1269163408 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7030031c30 con 0x7f703c102570 2024-09-19T08:03:21.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.168+0000 7f703a7fc640 1 -- 172.21.15.98:0/1269163408 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7030035030 con 0x7f703c102570 2024-09-19T08:03:21.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.168+0000 7f7043798640 1 -- 172.21.15.98:0/1269163408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f703c1a7130 con 0x7f703c102570 2024-09-19T08:03:21.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.169+0000 7f7043798640 1 -- 172.21.15.98:0/1269163408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f703c101200 con 0x7f703c102570 2024-09-19T08:03:21.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.170+0000 7f703a7fc640 1 -- 172.21.15.98:0/1269163408 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f7030042070 con 0x7f703c102570 2024-09-19T08:03:21.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.171+0000 7f703a7fc640 1 --2- 172.21.15.98:0/1269163408 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f70240a3550 0x7f70240a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:21.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.171+0000 7f7041534640 1 --2- 172.21.15.98:0/1269163408 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f70240a3550 0x7f70240a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:21.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.171+0000 7f703a7fc640 1 -- 172.21.15.98:0/1269163408 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f70300f0ab0 con 0x7f703c102570 2024-09-19T08:03:21.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.171+0000 7f7041534640 1 --2- 172.21.15.98:0/1269163408 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f70240a3550 0x7f70240a5a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f70340058c0 tx=0x7f703400a4c0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:21.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.174+0000 7f703a7fc640 1 -- 172.21.15.98:0/1269163408 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f7030031de0 con 0x7f703c102570 2024-09-19T08:03:21.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.347+0000 7f7043798640 1 -- 172.21.15.98:0/1269163408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f703c1a7390 con 0x7f703c102570 2024-09-19T08:03:21.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.348+0000 7f703a7fc640 1 -- 172.21.15.98:0/1269163408 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+23128 (secure 0 0 0) 0x7f70300bd400 con 0x7f703c102570 2024-09-19T08:03:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:21.349 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","created":"2024-09-19T07:53:39.127398+0000","modified":"2024-09-19T08:03:00.753313+0000","last_up_change":"2024-09-19T08:00:22.467777+0000","last_in_change":"2024-09-19T07:59:58.538492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:56:43.773892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T08:01:07.693796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T08:01:09.518028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d01ce6-165d-493d-ae98-d8ad9e5d4134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6803","nonce":1719360533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6805","nonce":1719360533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6809","nonce":1719360533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6807","nonce":1719360533}]},"public_addr":"172.21.15.98:6803/1719360533","cluster_addr":"172.21.15.98:6805/1719360533","heartbeat_back_addr":"172.21.15.98:6809/1719360533","heartbeat_front_addr":"172.21.15.98:6807/1719360533","state":["exists","up"]},{"osd":1,"uuid":"34724bd5-0738-4473-b087-c09031c1ee2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036962626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036962626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036962626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036962626}]},"public_addr":"172.21.15.139:6801/4036962626","cluster_addr":"172.21.15.139:6803/4036962626","heartbeat_back_addr":"172.21.15.139:6807/4036962626","heartbeat_front_addr":"172.21.15.139:6805/4036962626","state":["exists","up"]},{"osd":2,"uuid":"16985ebe-0e32-42ee-bded-b7838e290afc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6801","nonce":2811412891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6803","nonce":2811412891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6807","nonce":2811412891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6805","nonce":2811412891}]},"public_addr":"172.21.15.162:6801/2811412891","cluster_addr":"172.21.15.162:6803/2811412891","heartbeat_back_addr":"172.21.15.162:6807/2811412891","heartbeat_front_addr":"172.21.15.162:6805/2811412891","state":["exists","up"]},{"osd":3,"uuid":"b140e466-3119-49cb-aec2-ba6ca4d10d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6811","nonce":1383392014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6813","nonce":1383392014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6817","nonce":1383392014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6815","nonce":1383392014}]},"public_addr":"172.21.15.98:6811/1383392014","cluster_addr":"172.21.15.98:6813/1383392014","heartbeat_back_addr":"172.21.15.98:6817/1383392014","heartbeat_front_addr":"172.21.15.98:6815/1383392014","state":["exists","up"]},{"osd":4,"uuid":"b4412813-df57-4ac6-8fb7-f57b69c74768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6809","nonce":116453520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6811","nonce":116453520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6815","nonce":116453520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6813","nonce":116453520}]},"public_addr":"172.21.15.139:6809/116453520","cluster_addr":"172.21.15.139:6811/116453520","heartbeat_back_addr":"172.21.15.139:6815/116453520","heartbeat_front_addr":"172.21.15.139:6813/116453520","state":["exists","up"]},{"osd":5,"uuid":"d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6809","nonce":3769530297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6811","nonce":3769530297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6815","nonce":3769530297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6813","nonce":3769530297}]},"public_addr":"172.21.15.162:6809/3769530297","cluster_addr":"172.21.15.162:6811/3769530297","heartbeat_back_addr":"172.21.15.162:6815/3769530297","heartbeat_front_addr":"172.21.15.162:6813/3769530297","state":["exists","up"]},{"osd":6,"uuid":"ba1c1590-6aea-40dc-90c8-e21b88896909","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6819","nonce":692373267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6821","nonce":692373267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6825","nonce":692373267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6823","nonce":692373267}]},"public_addr":"172.21.15.98:6819/692373267","cluster_addr":"172.21.15.98:6821/692373267","heartbeat_back_addr":"172.21.15.98:6825/692373267","heartbeat_front_addr":"172.21.15.98:6823/692373267","state":["exists","up"]},{"osd":7,"uuid":"eb9093ad-a068-4a63-ab87-0b50522cbc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6817","nonce":1963811650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6819","nonce":1963811650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6823","nonce":1963811650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6821","nonce":1963811650}]},"public_addr":"172.21.15.139:6817/1963811650","cluster_addr":"172.21.15.139:6819/1963811650","heartbeat_back_addr":"172.21.15.139:6823/1963811650","heartbeat_front_addr":"172.21.15.139:6821/1963811650","state":["exists","up"]},{"osd":8,"uuid":"31ca330a-fc45-466d-81c2-ef4386b3cc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6817","nonce":3607966458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6819","nonce":3607966458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6823","nonce":3607966458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6821","nonce":3607966458}]},"public_addr":"172.21.15.162:6817/3607966458","cluster_addr":"172.21.15.162:6819/3607966458","heartbeat_back_addr":"172.21.15.162:6823/3607966458","heartbeat_front_addr":"172.21.15.162:6821/3607966458","state":["exists","up"]},{"osd":9,"uuid":"9c4f216f-aef1-4367-9c58-b913b0c4f7db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6827","nonce":2050391244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6829","nonce":2050391244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6833","nonce":2050391244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6831","nonce":2050391244}]},"public_addr":"172.21.15.98:6827/2050391244","cluster_addr":"172.21.15.98:6829/2050391244","heartbeat_back_addr":"172.21.15.98:6833/2050391244","heartbeat_front_addr":"172.21.15.98:6831/2050391244","state":["exists","up"]},{"osd":10,"uuid":"03cc7df6-84ae-4d9a-a3d7-74c91e1744e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6825","nonce":2336761639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6827","nonce":2336761639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6831","nonce":2336761639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6829","nonce":2336761639}]},"public_addr":"172.21.15.139:6825/2336761639","cluster_addr":"172.21.15.139:6827/2336761639","heartbeat_back_addr":"172.21.15.139:6831/2336761639","heartbeat_front_addr":"172.21.15.139:6829/2336761639","state":["exists","up"]},{"osd":11,"uuid":"29e95498-4952-46d4-8fa6-6d8e2b862111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6825","nonce":1231011554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6827","nonce":1231011554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6831","nonce":1231011554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6829","nonce":1231011554}]},"public_addr":"172.21.15.162:6825/1231011554","cluster_addr":"172.21.15.162:6827/1231011554","heartbeat_back_addr":"172.21.15.162:6831/1231011554","heartbeat_front_addr":"172.21.15.162:6829/1231011554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:55:49.995244+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:15.655543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:40.331165+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:04.688881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:29.174966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:53.663161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:17.407551+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:42.584296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:06.317219+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:31.078625+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:55.689269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T08:00:19.659748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1},{"from":7,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1252058007":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/921776066":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/336479555":"2024-09-20T08:03:00.753287+0000","172.21.15.98:6800/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6800/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/1497156658":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6801/470997812":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/1238193420":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6800/470997812":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/3838590070":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3842627255":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3866031018":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3194864054":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3067331617":"2024-09-20T08:03:00.753287+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T08:03:21.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.351+0000 7f7043798640 1 -- 172.21.15.98:0/1269163408 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f70240a3550 msgr2=0x7f70240a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:21.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.351+0000 7f7043798640 1 --2- 172.21.15.98:0/1269163408 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f70240a3550 0x7f70240a5a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f70340058c0 tx=0x7f703400a4c0 comp rx=0 tx=0).stop 2024-09-19T08:03:21.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.351+0000 7f7043798640 1 -- 172.21.15.98:0/1269163408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f703c102570 msgr2=0x7f703c197640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:21.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.351+0000 7f7043798640 1 --2- 172.21.15.98:0/1269163408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f703c102570 0x7f703c197640 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7030006010 tx=0x7f7030003c10 comp rx=0 tx=0).stop 2024-09-19T08:03:21.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.351+0000 7f7043798640 1 -- 172.21.15.98:0/1269163408 shutdown_connections 2024-09-19T08:03:21.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.351+0000 7f7043798640 1 --2- 172.21.15.98:0/1269163408 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f70240a3550 0x7f70240a5a10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:21.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.351+0000 7f7043798640 1 --2- 172.21.15.98:0/1269163408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f703c102ef0 0x7f703c197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:21.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.351+0000 7f7043798640 1 --2- 172.21.15.98:0/1269163408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f703c102570 0x7f703c197640 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:21.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.351+0000 7f7043798640 1 --2- 172.21.15.98:0/1269163408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f703c100e20 0x7f703c197100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:21.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.351+0000 7f7043798640 1 -- 172.21.15.98:0/1269163408 >> 172.21.15.98:0/1269163408 conn(0x7f703c0fc7d0 msgr2=0x7f703c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:21.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.351+0000 7f7043798640 1 -- 172.21.15.98:0/1269163408 shutdown_connections 2024-09-19T08:03:21.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:21.351+0000 7f7043798640 1 -- 172.21.15.98:0/1269163408 wait complete. 2024-09-19T08:03:21.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:21 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:03:21.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:21 smithi162 ceph-mon[35260]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-09-19T08:03:21.663 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:21 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:03:21.663 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:21 smithi098 ceph-mon[31333]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-09-19T08:03:21.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:21 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:03:21.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:21 smithi139 ceph-mon[34874]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-09-19T08:03:21.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-09-19T08:03:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:22.353 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:22 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1269163408' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:03:22.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:22 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1269163408' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:03:22.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:22 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1269163408' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:03:23.422 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:23 smithi098 ceph-mon[31333]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-09-19T08:03:23.450 INFO:teuthology.orchestra.run.smithi098.stdout:{"run_dir":"/var/run/ceph"} 2024-09-19T08:03:23.451 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-09-19T08:03:23.451 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 1747 2024-09-19T08:03:23.451 DEBUG:tasks.quiescer:Yielding 2024-09-19T08:03:23.451 INFO:teuthology.run_tasks:Running task workunit... 2024-09-19T08:03:23.465 INFO:tasks.workunit:Pulling workunits from ref 6fad5dd73c93545e52f57f681e140a9c3529c11e 2024-09-19T08:03:23.466 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-19T08:03:23.466 INFO:tasks.workunit:timeout=3h 2024-09-19T08:03:23.466 INFO:tasks.workunit:cleanup=True 2024-09-19T08:03:23.466 DEBUG:teuthology.orchestra.run.smithi098:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-19T08:03:23.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:03:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-19T08:03:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-09-19T08:03:23.484 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6ch/108d Inode: 1099511628283 Links: 2 2024-09-19T08:03:23.484 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T08:03:23.484 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-19T08:03:23.484 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-19 08:01:29.070025621 +0000 2024-09-19T08:03:23.484 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-19 08:01:29.070025621 +0000 2024-09-19T08:03:23.484 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-19 08:02:59.087984227 +0000 2024-09-19T08:03:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-09-19 08:01:29.070025621 +0000 2024-09-19T08:03:23.485 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-19T08:03:23.485 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-19T08:03:23.554 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 6fad5dd73c93545e52f57f681e140a9c3529c11e 2024-09-19T08:03:23.613 INFO:tasks.workunit.client.0.smithi098.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-19T08:03:23.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:23 smithi162 ceph-mon[35260]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-09-19T08:03:23.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:23 smithi139 ceph-mon[34874]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-09-19T08:03:23.721 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:24.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.200+0000 7fda93c15640 1 -- 172.21.15.98:0/4071304026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda8c102d30 msgr2=0x7fda8c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:24.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.200+0000 7fda93c15640 1 --2- 172.21.15.98:0/4071304026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda8c102d30 0x7fda8c103110 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fda88009930 tx=0x7fda8802a420 comp rx=0 tx=0).stop 2024-09-19T08:03:24.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.201+0000 7fda93c15640 1 -- 172.21.15.98:0/4071304026 shutdown_connections 2024-09-19T08:03:24.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.201+0000 7fda93c15640 1 --2- 172.21.15.98:0/4071304026 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda8c1082a0 0x7fda8c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:24.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.201+0000 7fda93c15640 1 --2- 172.21.15.98:0/4071304026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda8c1036e0 0x7fda8c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:24.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.201+0000 7fda93c15640 1 --2- 172.21.15.98:0/4071304026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda8c102d30 0x7fda8c103110 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.201+0000 7fda93c15640 1 -- 172.21.15.98:0/4071304026 >> 172.21.15.98:0/4071304026 conn(0x7fda8c0fc7d0 msgr2=0x7fda8c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.201+0000 7fda93c15640 1 -- 172.21.15.98:0/4071304026 shutdown_connections 2024-09-19T08:03:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.201+0000 7fda93c15640 1 -- 172.21.15.98:0/4071304026 wait complete. 2024-09-19T08:03:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.202+0000 7fda93c15640 1 Processor -- start 2024-09-19T08:03:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.202+0000 7fda93c15640 1 -- start start 2024-09-19T08:03:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.202+0000 7fda93c15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda8c102d30 0x7fda8c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.202+0000 7fda93c15640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda8c1036e0 0x7fda8c198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.202+0000 7fda93c15640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda8c1082a0 0x7fda8c19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.202+0000 7fda93c15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c1120a0 con 0x7fda8c102d30 2024-09-19T08:03:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.202+0000 7fda93c15640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c111f20 con 0x7fda8c1036e0 2024-09-19T08:03:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.202+0000 7fda93c15640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c112220 con 0x7fda8c1082a0 2024-09-19T08:03:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.202+0000 7fda919b1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda8c102d30 0x7fda8c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:24.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.202+0000 7fda919b1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda8c102d30 0x7fda8c198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56128/0 (socket says 172.21.15.98:56128) 2024-09-19T08:03:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.202+0000 7fda919b1640 1 -- 172.21.15.98:0/113494457 learned_addr learned my addr 172.21.15.98:0/113494457 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.202+0000 7fda921b2640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda8c1082a0 0x7fda8c19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.202+0000 7fda911b0640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda8c1036e0 0x7fda8c198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.203+0000 7fda919b1640 1 -- 172.21.15.98:0/113494457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda8c1082a0 msgr2=0x7fda8c19ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.203+0000 7fda919b1640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda8c1082a0 0x7fda8c19ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.203+0000 7fda919b1640 1 -- 172.21.15.98:0/113494457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda8c1036e0 msgr2=0x7fda8c198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.203+0000 7fda919b1640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda8c1036e0 0x7fda8c198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.203+0000 7fda919b1640 1 -- 172.21.15.98:0/113494457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8c19d4e0 con 0x7fda8c102d30 2024-09-19T08:03:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.203+0000 7fda911b0640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda8c1036e0 0x7fda8c198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.203+0000 7fda919b1640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda8c102d30 0x7fda8c198530 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fda88009230 tx=0x7fda88004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.203+0000 7fda82ffd640 1 -- 172.21.15.98:0/113494457 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8803f030 con 0x7fda8c102d30 2024-09-19T08:03:24.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.203+0000 7fda82ffd640 1 -- 172.21.15.98:0/113494457 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda88003ad0 con 0x7fda8c102d30 2024-09-19T08:03:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.203+0000 7fda93c15640 1 -- 172.21.15.98:0/113494457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda8c19d770 con 0x7fda8c102d30 2024-09-19T08:03:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.203+0000 7fda82ffd640 1 -- 172.21.15.98:0/113494457 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88035070 con 0x7fda8c102d30 2024-09-19T08:03:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.203+0000 7fda93c15640 1 -- 172.21.15.98:0/113494457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda8c1a50b0 con 0x7fda8c102d30 2024-09-19T08:03:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.204+0000 7fda93c15640 1 -- 172.21.15.98:0/113494457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda8c103110 con 0x7fda8c102d30 2024-09-19T08:03:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.206+0000 7fda82ffd640 1 -- 172.21.15.98:0/113494457 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7fda88042030 con 0x7fda8c102d30 2024-09-19T08:03:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.206+0000 7fda82ffd640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fda780a34d0 0x7fda780a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.207+0000 7fda911b0640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fda780a34d0 0x7fda780a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.207+0000 7fda82ffd640 1 -- 172.21.15.98:0/113494457 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fda880f0a40 con 0x7fda8c102d30 2024-09-19T08:03:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.207+0000 7fda911b0640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fda780a34d0 0x7fda780a5990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fda8c0fedf0 tx=0x7fda7c005e50 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:24.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.210+0000 7fda82ffd640 1 -- 172.21.15.98:0/113494457 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fda880bd2d0 con 0x7fda8c102d30 2024-09-19T08:03:24.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.417+0000 7fda93c15640 1 -- 172.21.15.98:0/113494457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda8c10a2b0 con 0x7fda8c102d30 2024-09-19T08:03:24.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.418+0000 7fda82ffd640 1 -- 172.21.15.98:0/113494457 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11983 (secure 0 0 0) 0x7fda880bcbd0 con 0x7fda8c102d30 2024-09-19T08:03:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:24.419 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"btime":"2024-09-19T08:02:59:855935+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:02:59.805764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:03:24.419 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-09-19T08:03:24.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.421+0000 7fda93c15640 1 -- 172.21.15.98:0/113494457 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fda780a34d0 msgr2=0x7fda780a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:24.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.421+0000 7fda93c15640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fda780a34d0 0x7fda780a5990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fda8c0fedf0 tx=0x7fda7c005e50 comp rx=0 tx=0).stop 2024-09-19T08:03:24.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.421+0000 7fda93c15640 1 -- 172.21.15.98:0/113494457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda8c102d30 msgr2=0x7fda8c198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:24.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.421+0000 7fda93c15640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda8c102d30 0x7fda8c198530 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fda88009230 tx=0x7fda88004270 comp rx=0 tx=0).stop 2024-09-19T08:03:24.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.421+0000 7fda93c15640 1 -- 172.21.15.98:0/113494457 shutdown_connections 2024-09-19T08:03:24.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.421+0000 7fda93c15640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fda780a34d0 0x7fda780a5990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.421+0000 7fda93c15640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda8c1082a0 0x7fda8c19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.421+0000 7fda93c15640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda8c1036e0 0x7fda8c198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.421+0000 7fda93c15640 1 --2- 172.21.15.98:0/113494457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda8c102d30 0x7fda8c198530 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.421+0000 7fda93c15640 1 -- 172.21.15.98:0/113494457 >> 172.21.15.98:0/113494457 conn(0x7fda8c0fc7d0 msgr2=0x7fda8c105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.421+0000 7fda93c15640 1 -- 172.21.15.98:0/113494457 shutdown_connections 2024-09-19T08:03:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:24.421+0000 7fda93c15640 1 -- 172.21.15.98:0/113494457 wait complete. 2024-09-19T08:03:24.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:24 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/113494457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:24.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:24 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/113494457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:24.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:24 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/113494457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:25.093 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-19T08:01:11.307169+0000', 'modified': '2024-09-19T08:02:59.805764+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24485, 'mds_1': 24461, 'mds_2': 24437, 'mds_3': 24406, 'mds_4': 24376}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14631': {'gid': 14631, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6839/3417988465', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 3417988465}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 3417988465}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14643': {'gid': 14643, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6841/1339458935', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6840', 'nonce': 1339458935}, {'type': 'v1', 'addr': '172.21.15.98:6841', 'nonce': 1339458935}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24376': {'gid': 24376, 'name': 'l', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.162:6833/3959364141', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6832', 'nonce': 3959364141}, {'type': 'v1', 'addr': '172.21.15.162:6833', 'nonce': 3959364141}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24406': {'gid': 24406, 'name': 'f', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.162:6835/1149502202', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6834', 'nonce': 1149502202}, {'type': 'v1', 'addr': '172.21.15.162:6835', 'nonce': 1149502202}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24413': {'gid': 24413, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6833/2267154969', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6832', 'nonce': 2267154969}, {'type': 'v1', 'addr': '172.21.15.139:6833', 'nonce': 2267154969}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24415': {'gid': 24415, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6837/3969500353', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6836', 'nonce': 3969500353}, {'type': 'v1', 'addr': '172.21.15.162:6837', 'nonce': 3969500353}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6839/3810709211', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6838', 'nonce': 3810709211}, {'type': 'v1', 'addr': '172.21.15.162:6839', 'nonce': 3810709211}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.139:6835/1275838513', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6834', 'nonce': 1275838513}, {'type': 'v1', 'addr': '172.21.15.139:6835', 'nonce': 1275838513}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.139:6837/1885641254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6836', 'nonce': 1885641254}, {'type': 'v1', 'addr': '172.21.15.139:6837', 'nonce': 1885641254}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.139:6839/1001534713', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6838', 'nonce': 1001534713}, {'type': 'v1', 'addr': '172.21.15.139:6839', 'nonce': 1001534713}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24485, 'qdb_cluster': [24376, 24406, 24437, 24461, 24485]} 2024-09-19T08:03:25.094 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-19T08:03:25.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.l status 2024-09-19T08:03:25.379 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:25.786 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:25 smithi098 ceph-mon[31333]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:03:25.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.866+0000 7feb196c4640 1 -- 172.21.15.98:0/2827445078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb14102d30 msgr2=0x7feb14103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:25.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.866+0000 7feb196c4640 1 --2- 172.21.15.98:0/2827445078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb14102d30 0x7feb14103110 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7feb04009930 tx=0x7feb0402a420 comp rx=0 tx=0).stop 2024-09-19T08:03:25.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.866+0000 7feb196c4640 1 -- 172.21.15.98:0/2827445078 shutdown_connections 2024-09-19T08:03:25.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.866+0000 7feb196c4640 1 --2- 172.21.15.98:0/2827445078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb141082a0 0x7feb1410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.866+0000 7feb196c4640 1 --2- 172.21.15.98:0/2827445078 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb141036e0 0x7feb14107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.866+0000 7feb196c4640 1 --2- 172.21.15.98:0/2827445078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb14102d30 0x7feb14103110 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.866+0000 7feb196c4640 1 -- 172.21.15.98:0/2827445078 >> 172.21.15.98:0/2827445078 conn(0x7feb140fc7d0 msgr2=0x7feb140febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:25.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.866+0000 7feb196c4640 1 -- 172.21.15.98:0/2827445078 shutdown_connections 2024-09-19T08:03:25.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.866+0000 7feb196c4640 1 -- 172.21.15.98:0/2827445078 wait complete. 2024-09-19T08:03:25.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.867+0000 7feb196c4640 1 Processor -- start 2024-09-19T08:03:25.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.867+0000 7feb196c4640 1 -- start start 2024-09-19T08:03:25.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.867+0000 7feb196c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb14102d30 0x7feb14198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.867+0000 7feb196c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb141036e0 0x7feb14198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.867+0000 7feb196c4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb141082a0 0x7feb1419ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.867+0000 7feb196c4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb141120a0 con 0x7feb141036e0 2024-09-19T08:03:25.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.867+0000 7feb196c4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feb14111f20 con 0x7feb141082a0 2024-09-19T08:03:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.867+0000 7feb196c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb14112220 con 0x7feb14102d30 2024-09-19T08:03:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.867+0000 7feb127fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb141036e0 0x7feb14198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.867+0000 7feb127fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb141036e0 0x7feb14198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56140/0 (socket says 172.21.15.98:56140) 2024-09-19T08:03:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.867+0000 7feb127fc640 1 -- 172.21.15.98:0/3473165942 learned_addr learned my addr 172.21.15.98:0/3473165942 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.867+0000 7feb12ffd640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb14102d30 0x7feb14198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.867+0000 7feb137fe640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb141082a0 0x7feb1419ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.868+0000 7feb127fc640 1 -- 172.21.15.98:0/3473165942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb14102d30 msgr2=0x7feb14198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.868+0000 7feb127fc640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb14102d30 0x7feb14198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.868+0000 7feb127fc640 1 -- 172.21.15.98:0/3473165942 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb141082a0 msgr2=0x7feb1419ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.868+0000 7feb127fc640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb141082a0 0x7feb1419ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.868+0000 7feb127fc640 1 -- 172.21.15.98:0/3473165942 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1419d530 con 0x7feb141036e0 2024-09-19T08:03:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.868+0000 7feb12ffd640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb14102d30 0x7feb14198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.868+0000 7feb127fc640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb141036e0 0x7feb14198ac0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7feb0800c8c0 tx=0x7feb0800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.868+0000 7feafbfff640 1 -- 172.21.15.98:0/3473165942 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08007bb0 con 0x7feb141036e0 2024-09-19T08:03:25.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.868+0000 7feafbfff640 1 -- 172.21.15.98:0/3473165942 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb08007d50 con 0x7feb141036e0 2024-09-19T08:03:25.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.868+0000 7feb196c4640 1 -- 172.21.15.98:0/3473165942 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1419d820 con 0x7feb141036e0 2024-09-19T08:03:25.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.868+0000 7feafbfff640 1 -- 172.21.15.98:0/3473165942 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08018830 con 0x7feb141036e0 2024-09-19T08:03:25.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.868+0000 7feb196c4640 1 -- 172.21.15.98:0/3473165942 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb141a5100 con 0x7feb141036e0 2024-09-19T08:03:25.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:25 smithi162 ceph-mon[35260]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:03:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feafbfff640 1 -- 172.21.15.98:0/3473165942 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7feb08018a10 con 0x7feb141036e0 2024-09-19T08:03:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feb1ba9d740 1 Processor -- start 2024-09-19T08:03:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feb1ba9d740 1 -- start start 2024-09-19T08:03:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feb1ba9d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c599d0 0x555b00c59db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feb1ba9d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c5a2f0 0x555b00c64670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feb1ba9d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c64bb0 0x555b00c66fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feb1ba9d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555b00bc0090 con 0x555b00c599d0 2024-09-19T08:03:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feb1ba9d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555b00b67a70 con 0x555b00c64bb0 2024-09-19T08:03:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feb1ba9d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555b00bbc050 con 0x555b00c5a2f0 2024-09-19T08:03:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feb137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c599d0 0x555b00c59db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feb137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c599d0 0x555b00c59db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56142/0 (socket says 172.21.15.98:56142) 2024-09-19T08:03:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feb137fe640 1 -- 172.21.15.98:0/2124024374 learned_addr learned my addr 172.21.15.98:0/2124024374 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feafbfff640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feafc0a34d0 0x7feafc0a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feb12ffd640 1 --2- 172.21.15.98:0/2124024374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c5a2f0 0x555b00c64670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.871+0000 7feb127fc640 1 --2- 172.21.15.98:0/2124024374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c64bb0 0x555b00c66fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.872+0000 7feb137fe640 1 -- 172.21.15.98:0/2124024374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c5a2f0 msgr2=0x555b00c64670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.872+0000 7feb137fe640 1 --2- 172.21.15.98:0/2124024374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c5a2f0 0x555b00c64670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.872+0000 7feb137fe640 1 -- 172.21.15.98:0/2124024374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c64bb0 msgr2=0x555b00c66fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.872+0000 7feb137fe640 1 --2- 172.21.15.98:0/2124024374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c64bb0 0x555b00c66fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.872+0000 7feb137fe640 1 -- 172.21.15.98:0/2124024374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b00b74130 con 0x555b00c599d0 2024-09-19T08:03:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.872+0000 7feafbfff640 1 -- 172.21.15.98:0/3473165942 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7feb080d00e0 con 0x7feb141036e0 2024-09-19T08:03:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.872+0000 7feb137fe640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feafc0a34d0 0x7feafc0a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.872+0000 7feb137fe640 1 --2- 172.21.15.98:0/2124024374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c599d0 0x555b00c59db0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7feb0c008ec0 tx=0x7feb0c00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6791d5d890974c90 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.872+0000 7feaf97fa640 1 -- 172.21.15.98:0/2124024374 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c022070 con 0x555b00c599d0 2024-09-19T08:03:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.872+0000 7feaf97fa640 1 -- 172.21.15.98:0/2124024374 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0c01c470 con 0x555b00c599d0 2024-09-19T08:03:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.872+0000 7feaf97fa640 1 -- 172.21.15.98:0/2124024374 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c01a620 con 0x555b00c599d0 2024-09-19T08:03:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.872+0000 7feb137fe640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feafc0a34d0 0x7feafc0a5990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7feb0c01f000 tx=0x7feb0c020000 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.873+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2124024374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c599d0 msgr2=0x555b00c59db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.873+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2124024374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c599d0 0x555b00c59db0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7feb0c008ec0 tx=0x7feb0c00e720 comp rx=0 tx=0).stop 2024-09-19T08:03:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.873+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2124024374 shutdown_connections 2024-09-19T08:03:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.873+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2124024374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c64bb0 0x555b00c66fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.873+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2124024374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c5a2f0 0x555b00c64670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.873+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2124024374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c599d0 0x555b00c59db0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.873+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2124024374 >> 172.21.15.98:0/2124024374 conn(0x555b00c55f60 msgr2=0x555b00c56340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.873+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2124024374 shutdown_connections 2024-09-19T08:03:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.873+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2124024374 wait complete. 2024-09-19T08:03:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.873+0000 7feb1ba9d740 1 -- start start 2024-09-19T08:03:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.873+0000 7feb1ba9d740 1 Processor -- start 2024-09-19T08:03:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb1ba9d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c599d0 0x555b00c64480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb1ba9d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c5a2f0 0x555b00c617d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb1ba9d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c64bb0 0x555b00c61d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb1ba9d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555b00c6bb90 con 0x555b00c5a2f0 2024-09-19T08:03:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb1ba9d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555b00b7af20 con 0x555b00c599d0 2024-09-19T08:03:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb1ba9d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555b00c6bd10 con 0x555b00c64bb0 2024-09-19T08:03:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb1ba9d740 10 client.0 _ll_register_callbacks cb 0x555b00b44030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb127fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c5a2f0 0x555b00c617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb127fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c5a2f0 0x555b00c617d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56150/0 (socket says 172.21.15.98:56150) 2024-09-19T08:03:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb127fc640 1 -- 172.21.15.98:0/2431582691 learned_addr learned my addr 172.21.15.98:0/2431582691 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb12ffd640 1 --2- 172.21.15.98:0/2431582691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c599d0 0x555b00c64480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb137fe640 1 --2- 172.21.15.98:0/2431582691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c64bb0 0x555b00c61d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb127fc640 1 -- 172.21.15.98:0/2431582691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c64bb0 msgr2=0x555b00c61d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb127fc640 1 --2- 172.21.15.98:0/2431582691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c64bb0 0x555b00c61d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb127fc640 1 -- 172.21.15.98:0/2431582691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c599d0 msgr2=0x555b00c64480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb127fc640 1 --2- 172.21.15.98:0/2431582691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c599d0 0x555b00c64480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb127fc640 1 -- 172.21.15.98:0/2431582691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b009312b0 con 0x555b00c5a2f0 2024-09-19T08:03:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb12ffd640 1 --2- 172.21.15.98:0/2431582691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c599d0 0x555b00c64480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb127fc640 1 --2- 172.21.15.98:0/2431582691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c5a2f0 0x555b00c617d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7feb080cd890 tx=0x7feb0809c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a5f718796c560f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feaf6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feaf6ff5640 1 -- 172.21.15.98:0/2431582691 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0801f300 con 0x555b00c5a2f0 2024-09-19T08:03:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feaf6ff5640 1 -- 172.21.15.98:0/2431582691 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7feb080bc0b0 con 0x555b00c5a2f0 2024-09-19T08:03:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.874+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2431582691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb0c01c470 con 0x555b00c5a2f0 2024-09-19T08:03:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.875+0000 7feaf6ff5640 1 -- 172.21.15.98:0/2431582691 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7feb080bc2b0 con 0x555b00c5a2f0 2024-09-19T08:03:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.875+0000 7feb1ba9d740 10 client.15510 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.875+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2431582691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555b00c774e0 con 0x555b00c5a2f0 2024-09-19T08:03:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.875+0000 7feaf6ff5640 1 -- 172.21.15.98:0/2431582691 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0c01c470 con 0x555b00c5a2f0 2024-09-19T08:03:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.875+0000 7feaf6ff5640 1 -- 172.21.15.98:0/2431582691 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7feb0809c4a0 con 0x555b00c5a2f0 2024-09-19T08:03:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.875+0000 7feb1ba9d740 10 client.15510 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.875+0000 7feb1ba9d740 10 client.15510 resolve_mds: resolved name 'l' to daemon mds.l 2024-09-19T08:03:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.875+0000 7feb1ba9d740 20 client.15510 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.875+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2431582691 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x555b00c79a20 0x555b00c7ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.875+0000 7feb1ba9d740 4 client.15510 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.875+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2431582691 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b00bcd600 con 0x555b00c79a20 2024-09-19T08:03:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.876+0000 7feb12ffd640 1 --2- 172.21.15.98:0/2431582691 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x555b00c79a20 0x555b00c7ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.876+0000 7feb12ffd640 1 --2- 172.21.15.98:0/2431582691 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x555b00c79a20 0x555b00c7ae90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d2956294d0a42691 server_cookie=422982bddde80dbd in_seq=0 out_seq=0 2024-09-19T08:03:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.876+0000 7feaf6ff5640 10 client.15510 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:03:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feaf6ff5640 1 -- 172.21.15.98:0/2431582691 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555b00bcd600 con 0x555b00c79a20 2024-09-19T08:03:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feaf6ff5640 10 client.15510 handle_command_reply: tid=0 2024-09-19T08:03:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feb1ba9d740 1 client.15510 shutdown 2024-09-19T08:03:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2431582691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c5a2f0 msgr2=0x555b00c617d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2431582691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c5a2f0 0x555b00c617d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7feb080cd890 tx=0x7feb0809c140 comp rx=0 tx=0).stop 2024-09-19T08:03:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2431582691 shutdown_connections 2024-09-19T08:03:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2431582691 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x555b00c79a20 0x555b00c7ae90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2431582691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c64bb0 0x555b00c61d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2431582691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c5a2f0 0x555b00c617d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2431582691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c599d0 0x555b00c64480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2431582691 >> 172.21.15.98:0/2431582691 conn(0x555b00c55f60 msgr2=0x555b00c5bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:25.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feaf6ff5640 0 client.15510 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:03:25.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2431582691 shutdown_connections 2024-09-19T08:03:25.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2431582691 wait complete. 2024-09-19T08:03:25.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.877+0000 7feb1ba9d740 20 client.15510 trim_cache size 0 max 16384 2024-09-19T08:03:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.894+0000 7feb1ba9d740 1 Processor -- start 2024-09-19T08:03:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb1ba9d740 1 -- start start 2024-09-19T08:03:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb1ba9d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c599d0 0x555b00c79a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb1ba9d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c5a2f0 0x555b00c56340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb1ba9d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c64bb0 0x555b00c56880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb1ba9d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555b00c791d0 con 0x555b00c64bb0 2024-09-19T08:03:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb1ba9d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555b00c798a0 con 0x555b00c599d0 2024-09-19T08:03:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb1ba9d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555b00c79f70 con 0x555b00c5a2f0 2024-09-19T08:03:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c64bb0 0x555b00c56880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c64bb0 0x555b00c56880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56158/0 (socket says 172.21.15.98:56158) 2024-09-19T08:03:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb137fe640 1 -- 172.21.15.98:0/2980027982 learned_addr learned my addr 172.21.15.98:0/2980027982 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb12ffd640 1 --2- 172.21.15.98:0/2980027982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c599d0 0x555b00c79a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb127fc640 1 --2- 172.21.15.98:0/2980027982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c5a2f0 0x555b00c56340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb137fe640 1 -- 172.21.15.98:0/2980027982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c5a2f0 msgr2=0x555b00c56340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb137fe640 1 --2- 172.21.15.98:0/2980027982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c5a2f0 0x555b00c56340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb137fe640 1 -- 172.21.15.98:0/2980027982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c599d0 msgr2=0x555b00c79a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb137fe640 1 --2- 172.21.15.98:0/2980027982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c599d0 0x555b00c79a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb137fe640 1 -- 172.21.15.98:0/2980027982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b00c647f0 con 0x555b00c64bb0 2024-09-19T08:03:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.895+0000 7feb137fe640 1 --2- 172.21.15.98:0/2980027982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c64bb0 0x555b00c56880 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7feb0c013fe0 tx=0x7feb0c0085b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7db03b518387148 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.896+0000 7feaf8ff9640 1 -- 172.21.15.98:0/2980027982 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c035070 con 0x555b00c64bb0 2024-09-19T08:03:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.896+0000 7feaf8ff9640 1 -- 172.21.15.98:0/2980027982 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0c011d10 con 0x555b00c64bb0 2024-09-19T08:03:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.896+0000 7feaf8ff9640 1 -- 172.21.15.98:0/2980027982 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c01ce60 con 0x555b00c64bb0 2024-09-19T08:03:25.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.896+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2980027982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c64bb0 msgr2=0x555b00c56880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:25.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.896+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2980027982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c64bb0 0x555b00c56880 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7feb0c013fe0 tx=0x7feb0c0085b0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.896+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2980027982 shutdown_connections 2024-09-19T08:03:25.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.896+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2980027982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c64bb0 0x555b00c56880 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.896+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2980027982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c5a2f0 0x555b00c56340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.896+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2980027982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c599d0 0x555b00c79a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.896+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2980027982 >> 172.21.15.98:0/2980027982 conn(0x555b00c55f60 msgr2=0x555b00c50ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.896+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2980027982 shutdown_connections 2024-09-19T08:03:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.896+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2980027982 wait complete. 2024-09-19T08:03:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb1ba9d740 1 -- start start 2024-09-19T08:03:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb1ba9d740 1 Processor -- start 2024-09-19T08:03:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb1ba9d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c599d0 0x555b00c7e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb1ba9d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c5a2f0 0x555b00c7e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb1ba9d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c64bb0 0x555b00ce2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb1ba9d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555b00c7ad10 con 0x555b00c5a2f0 2024-09-19T08:03:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb1ba9d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555b00c7a640 con 0x555b00c64bb0 2024-09-19T08:03:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb1ba9d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555b00c7b3e0 con 0x555b00c599d0 2024-09-19T08:03:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb1ba9d740 10 client.0 _ll_register_callbacks cb 0x7feb0800fd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb127fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c5a2f0 0x555b00c7e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb127fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c5a2f0 0x555b00c7e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56168/0 (socket says 172.21.15.98:56168) 2024-09-19T08:03:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb127fc640 1 -- 172.21.15.98:0/2100483903 learned_addr learned my addr 172.21.15.98:0/2100483903 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb137fe640 1 --2- 172.21.15.98:0/2100483903 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c64bb0 0x555b00ce2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb12ffd640 1 --2- 172.21.15.98:0/2100483903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c599d0 0x555b00c7e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb127fc640 1 -- 172.21.15.98:0/2100483903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c599d0 msgr2=0x555b00c7e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb127fc640 1 --2- 172.21.15.98:0/2100483903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c599d0 0x555b00c7e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb127fc640 1 -- 172.21.15.98:0/2100483903 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c64bb0 msgr2=0x555b00ce2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb127fc640 1 --2- 172.21.15.98:0/2100483903 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c64bb0 0x555b00ce2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb127fc640 1 -- 172.21.15.98:0/2100483903 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b00c7f2b0 con 0x555b00c5a2f0 2024-09-19T08:03:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb137fe640 1 --2- 172.21.15.98:0/2100483903 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c64bb0 0x555b00ce2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.897+0000 7feb127fc640 1 --2- 172.21.15.98:0/2100483903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c5a2f0 0x555b00c7e9c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7feb0809bf40 tx=0x7feb0809c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9ad8f9d1cac39be server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feaf6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feaf6ff5640 1 -- 172.21.15.98:0/2100483903 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb080b9020 con 0x555b00c5a2f0 2024-09-19T08:03:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feaf6ff5640 1 -- 172.21.15.98:0/2100483903 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7feb080a0120 con 0x555b00c5a2f0 2024-09-19T08:03:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2100483903 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb0c011d10 con 0x555b00c5a2f0 2024-09-19T08:03:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feaf6ff5640 1 -- 172.21.15.98:0/2100483903 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7feb080984e0 con 0x555b00c5a2f0 2024-09-19T08:03:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feb1ba9d740 10 client.15516 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2100483903 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555b00ce3790 con 0x555b00c5a2f0 2024-09-19T08:03:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feaf6ff5640 1 -- 172.21.15.98:0/2100483903 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0c011d10 con 0x555b00c5a2f0 2024-09-19T08:03:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feaf6ff5640 1 -- 172.21.15.98:0/2100483903 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7feb0809b690 con 0x555b00c5a2f0 2024-09-19T08:03:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feb1ba9d740 10 client.15516 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feb1ba9d740 10 client.15516 resolve_mds: resolved name 'l' to daemon mds.l 2024-09-19T08:03:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feb1ba9d740 20 client.15516 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2100483903 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x555b00ce8390 0x555b00ce70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feb1ba9d740 4 client.15516 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:03:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.898+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2100483903 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "status"}) -- 0x555b00bc7b10 con 0x555b00ce8390 2024-09-19T08:03:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feb12ffd640 1 --2- 172.21.15.98:0/2100483903 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x555b00ce8390 0x555b00ce70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feb12ffd640 1 --2- 172.21.15.98:0/2100483903 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x555b00ce8390 0x555b00ce70a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=31034fac5cbb02f0 server_cookie=a766dc24866508f4 in_seq=0 out_seq=0 2024-09-19T08:03:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feaf6ff5640 10 client.15516 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:03:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feaf6ff5640 1 -- 172.21.15.98:0/2100483903 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x555b00bc7b10 con 0x555b00ce8390 2024-09-19T08:03:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feaf6ff5640 10 client.15516 handle_command_reply: tid=0 2024-09-19T08:03:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feb1ba9d740 1 client.15516 shutdown 2024-09-19T08:03:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2100483903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c5a2f0 msgr2=0x555b00c7e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2100483903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c5a2f0 0x555b00c7e9c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7feb0809bf40 tx=0x7feb0809c860 comp rx=0 tx=0).stop 2024-09-19T08:03:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2100483903 shutdown_connections 2024-09-19T08:03:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2100483903 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x555b00ce8390 0x555b00ce70a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2100483903 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b00c64bb0 0x555b00ce2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2100483903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b00c5a2f0 0x555b00c7e9c0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feb1ba9d740 1 --2- 172.21.15.98:0/2100483903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b00c599d0 0x555b00c7e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feaf6ff5640 0 client.15516 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:03:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.899+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2100483903 >> 172.21.15.98:0/2100483903 conn(0x555b00c55f60 msgr2=0x555b00c56340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:25.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2100483903 shutdown_connections 2024-09-19T08:03:25.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb1ba9d740 1 -- 172.21.15.98:0/2100483903 wait complete. 2024-09-19T08:03:25.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb1ba9d740 20 client.15516 trim_cache size 0 max 16384 2024-09-19T08:03:25.907 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:03:25.907 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:03:25.907 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 4, 2024-09-19T08:03:25.907 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24376, 2024-09-19T08:03:25.907 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:03:25.907 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:03:25.908 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:03:25.908 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 124.209041602, 2024-09-19T08:03:25.908 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-09-19T08:03:25.908 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 81, 2024-09-19T08:03:25.908 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 80, 2024-09-19T08:03:25.908 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 175.960506608 2024-09-19T08:03:25.908 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:03:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb196c4640 1 -- 172.21.15.98:0/3473165942 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feafc0a34d0 msgr2=0x7feafc0a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb196c4640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feafc0a34d0 0x7feafc0a5990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7feb0c01f000 tx=0x7feb0c020000 comp rx=0 tx=0).stop 2024-09-19T08:03:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb196c4640 1 -- 172.21.15.98:0/3473165942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb141036e0 msgr2=0x7feb14198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb196c4640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb141036e0 0x7feb14198ac0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7feb0800c8c0 tx=0x7feb0800cd80 comp rx=0 tx=0).stop 2024-09-19T08:03:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb196c4640 1 -- 172.21.15.98:0/3473165942 shutdown_connections 2024-09-19T08:03:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb196c4640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feafc0a34d0 0x7feafc0a5990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb196c4640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb141082a0 0x7feb1419ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb196c4640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb141036e0 0x7feb14198ac0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb196c4640 1 --2- 172.21.15.98:0/3473165942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb14102d30 0x7feb14198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:25.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb196c4640 1 -- 172.21.15.98:0/3473165942 >> 172.21.15.98:0/3473165942 conn(0x7feb140fc7d0 msgr2=0x7feb14105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:25.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb196c4640 1 -- 172.21.15.98:0/3473165942 shutdown_connections 2024-09-19T08:03:25.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:25.900+0000 7feb196c4640 1 -- 172.21.15.98:0/3473165942 wait complete. 2024-09-19T08:03:25.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:25 smithi139 ceph-mon[34874]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:03:26.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.f status 2024-09-19T08:03:26.852 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:27.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:26 smithi098 ceph-mon[31333]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-19T08:03:27.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:26 smithi162 ceph-mon[35260]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-19T08:03:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:26 smithi139 ceph-mon[34874]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-19T08:03:27.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.351+0000 7f9c9f5cc640 1 -- 172.21.15.98:0/1175884530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c98102790 msgr2=0x7f9c981067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:27.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.351+0000 7f9c9f5cc640 1 --2- 172.21.15.98:0/1175884530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c98102790 0x7f9c981067e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009900 tx=0x7f9c8802a410 comp rx=0 tx=0).stop 2024-09-19T08:03:27.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.351+0000 7f9c9f5cc640 1 -- 172.21.15.98:0/1175884530 shutdown_connections 2024-09-19T08:03:27.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.351+0000 7f9c9f5cc640 1 --2- 172.21.15.98:0/1175884530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c98106f10 0x7f9c9810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.351+0000 7f9c9f5cc640 1 --2- 172.21.15.98:0/1175884530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c98102790 0x7f9c981067e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.351+0000 7f9c9f5cc640 1 --2- 172.21.15.98:0/1175884530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c98101de0 0x7f9c981021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.351+0000 7f9c9f5cc640 1 -- 172.21.15.98:0/1175884530 >> 172.21.15.98:0/1175884530 conn(0x7f9c980fd8c0 msgr2=0x7f9c980ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:27.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.351+0000 7f9c9f5cc640 1 -- 172.21.15.98:0/1175884530 shutdown_connections 2024-09-19T08:03:27.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.351+0000 7f9c9f5cc640 1 -- 172.21.15.98:0/1175884530 wait complete. 2024-09-19T08:03:27.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.352+0000 7f9c9f5cc640 1 Processor -- start 2024-09-19T08:03:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.352+0000 7f9c9f5cc640 1 -- start start 2024-09-19T08:03:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.352+0000 7f9c9f5cc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c98101de0 0x7f9c98198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.352+0000 7f9c9f5cc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c98102790 0x7f9c981989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.352+0000 7f9c9f5cc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c98106f10 0x7f9c9819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.352+0000 7f9c9f5cc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9c98111f90 con 0x7f9c98106f10 2024-09-19T08:03:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.352+0000 7f9c9f5cc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9c98111e10 con 0x7f9c98101de0 2024-09-19T08:03:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.352+0000 7f9c9f5cc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9c98112110 con 0x7f9c98102790 2024-09-19T08:03:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c9db69640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c98106f10 0x7f9c9819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c9db69640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c98106f10 0x7f9c9819cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45136/0 (socket says 172.21.15.98:45136) 2024-09-19T08:03:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c9db69640 1 -- 172.21.15.98:0/309524769 learned_addr learned my addr 172.21.15.98:0/309524769 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c98102790 0x7f9c981989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c9d368640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c98101de0 0x7f9c98198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c9db69640 1 -- 172.21.15.98:0/309524769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c98102790 msgr2=0x7f9c981989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:27.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c9db69640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c98102790 0x7f9c981989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c9db69640 1 -- 172.21.15.98:0/309524769 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c98101de0 msgr2=0x7f9c98198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:27.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c9db69640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c98101de0 0x7f9c98198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c9db69640 1 -- 172.21.15.98:0/309524769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c9819d420 con 0x7f9c98106f10 2024-09-19T08:03:27.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c98102790 0x7f9c981989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:27.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c9db69640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c98106f10 0x7f9c9819cd40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000b970 tx=0x7f9c9000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:27.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c8e7fc640 1 -- 172.21.15.98:0/309524769 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90019070 con 0x7f9c98106f10 2024-09-19T08:03:27.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c8e7fc640 1 -- 172.21.15.98:0/309524769 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c9000cb40 con 0x7f9c98106f10 2024-09-19T08:03:27.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c9f5cc640 1 -- 172.21.15.98:0/309524769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c9819d710 con 0x7f9c98106f10 2024-09-19T08:03:27.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c8e7fc640 1 -- 172.21.15.98:0/309524769 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90007650 con 0x7f9c98106f10 2024-09-19T08:03:27.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.353+0000 7f9c9f5cc640 1 -- 172.21.15.98:0/309524769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c981a4ff0 con 0x7f9c98106f10 2024-09-19T08:03:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.356+0000 7f9c8e7fc640 1 -- 172.21.15.98:0/309524769 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f9c9001c070 con 0x7f9c98106f10 2024-09-19T08:03:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.356+0000 7f9ca19a5740 1 Processor -- start 2024-09-19T08:03:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.356+0000 7f9ca19a5740 1 -- start start 2024-09-19T08:03:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.356+0000 7f9ca19a5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af8890 0x5604d3af8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.356+0000 7f9ca19a5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af91b0 0x5604d3b07a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.356+0000 7f9ca19a5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3b07fc0 0x5604d3b0a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.356+0000 7f9ca19a5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5604d3a5f090 con 0x5604d3b07fc0 2024-09-19T08:03:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.356+0000 7f9ca19a5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5604d3a06a70 con 0x5604d3af91b0 2024-09-19T08:03:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.356+0000 7f9ca19a5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5604d3a5b050 con 0x5604d3af8890 2024-09-19T08:03:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.356+0000 7f9c9cb67640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3b07fc0 0x5604d3b0a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.356+0000 7f9c9cb67640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3b07fc0 0x5604d3b0a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45138/0 (socket says 172.21.15.98:45138) 2024-09-19T08:03:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.356+0000 7f9c9cb67640 1 -- 172.21.15.98:0/410802278 learned_addr learned my addr 172.21.15.98:0/410802278 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c8e7fc640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9c840a34d0 0x7f9c840a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c9d368640 1 --2- 172.21.15.98:0/410802278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af91b0 0x5604d3b07a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/410802278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af8890 0x5604d3af8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c9db69640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9c840a34d0 0x7f9c840a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c9cb67640 1 -- 172.21.15.98:0/410802278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af8890 msgr2=0x5604d3af8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/410802278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af8890 0x5604d3af8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c9cb67640 1 -- 172.21.15.98:0/410802278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af91b0 msgr2=0x5604d3b07a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/410802278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af91b0 0x5604d3b07a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c9cb67640 1 -- 172.21.15.98:0/410802278 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604d3a13130 con 0x5604d3b07fc0 2024-09-19T08:03:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c8e7fc640 1 -- 172.21.15.98:0/309524769 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f9c900cf4a0 con 0x7f9c98106f10 2024-09-19T08:03:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/410802278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3b07fc0 0x5604d3b0a3b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9c880096c0 tx=0x7f9c88031440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b2cc04dc163eb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c9db69640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9c840a34d0 0x7f9c840a5990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9c900ccc10 tx=0x7f9c9006e4a0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c837fe640 1 -- 172.21.15.98:0/410802278 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88045070 con 0x5604d3b07fc0 2024-09-19T08:03:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c837fe640 1 -- 172.21.15.98:0/410802278 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c88005040 con 0x5604d3b07fc0 2024-09-19T08:03:27.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.357+0000 7f9c837fe640 1 -- 172.21.15.98:0/410802278 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88035050 con 0x5604d3b07fc0 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.358+0000 7f9ca19a5740 1 -- 172.21.15.98:0/410802278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3b07fc0 msgr2=0x5604d3b0a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.358+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/410802278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3b07fc0 0x5604d3b0a3b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9c880096c0 tx=0x7f9c88031440 comp rx=0 tx=0).stop 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.358+0000 7f9ca19a5740 1 -- 172.21.15.98:0/410802278 shutdown_connections 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.358+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/410802278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3b07fc0 0x5604d3b0a3b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.358+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/410802278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af91b0 0x5604d3b07a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.358+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/410802278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af8890 0x5604d3af8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.358+0000 7f9ca19a5740 1 -- 172.21.15.98:0/410802278 >> 172.21.15.98:0/410802278 conn(0x5604d3af4f60 msgr2=0x5604d3af5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.358+0000 7f9ca19a5740 1 -- 172.21.15.98:0/410802278 shutdown_connections 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.358+0000 7f9ca19a5740 1 -- 172.21.15.98:0/410802278 wait complete. 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.358+0000 7f9ca19a5740 1 -- start start 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.358+0000 7f9ca19a5740 1 Processor -- start 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9ca19a5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af8890 0x5604d3b078b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9ca19a5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af91b0 0x5604d3b04c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9ca19a5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3b07fc0 0x5604d3b05140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9ca19a5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5604d3b0ab30 con 0x5604d3af8890 2024-09-19T08:03:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9ca19a5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5604d3a19f20 con 0x5604d3b07fc0 2024-09-19T08:03:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9ca19a5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5604d3b0acb0 con 0x5604d3af91b0 2024-09-19T08:03:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9ca19a5740 10 client.0 _ll_register_callbacks cb 0x5604d39e3030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9c9cb67640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af8890 0x5604d3b078b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9c9cb67640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af8890 0x5604d3b078b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45150/0 (socket says 172.21.15.98:45150) 2024-09-19T08:03:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9c9cb67640 1 -- 172.21.15.98:0/103442364 learned_addr learned my addr 172.21.15.98:0/103442364 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9c9d368640 1 --2- 172.21.15.98:0/103442364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af91b0 0x5604d3b04c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9c9cb67640 1 -- 172.21.15.98:0/103442364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af91b0 msgr2=0x5604d3b04c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/103442364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af91b0 0x5604d3b04c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9c9cb67640 1 -- 172.21.15.98:0/103442364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3b07fc0 msgr2=0x5604d3b05140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/103442364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3b07fc0 0x5604d3b05140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9c9cb67640 1 -- 172.21.15.98:0/103442364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604d37d02b0 con 0x5604d3af8890 2024-09-19T08:03:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/103442364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af8890 0x5604d3b078b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9c88033000 tx=0x7f9c880056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa435e37b0b25e61 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.359+0000 7f9c80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.360+0000 7f9c80ff9640 1 -- 172.21.15.98:0/103442364 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88000e60 con 0x5604d3af8890 2024-09-19T08:03:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.360+0000 7f9ca19a5740 1 -- 172.21.15.98:0/103442364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c88005040 con 0x5604d3af8890 2024-09-19T08:03:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.360+0000 7f9c80ff9640 1 -- 172.21.15.98:0/103442364 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f9c88004ad0 con 0x5604d3af8890 2024-09-19T08:03:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.360+0000 7f9c80ff9640 1 -- 172.21.15.98:0/103442364 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9c88045070 con 0x5604d3af8890 2024-09-19T08:03:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.360+0000 7f9ca19a5740 10 client.15528 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.360+0000 7f9ca19a5740 1 -- 172.21.15.98:0/103442364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5604d3b164e0 con 0x5604d3af8890 2024-09-19T08:03:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.360+0000 7f9c80ff9640 1 -- 172.21.15.98:0/103442364 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c88005040 con 0x5604d3af8890 2024-09-19T08:03:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.360+0000 7f9c80ff9640 1 -- 172.21.15.98:0/103442364 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9c88049070 con 0x5604d3af8890 2024-09-19T08:03:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.360+0000 7f9ca19a5740 10 client.15528 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.360+0000 7f9ca19a5740 10 client.15528 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.360+0000 7f9ca19a5740 20 client.15528 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.360+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/103442364 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604d3b17830 0x5604d3b19a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.360+0000 7f9ca19a5740 4 client.15528 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.360+0000 7f9ca19a5740 1 -- 172.21.15.98:0/103442364 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604d3a6c600 con 0x5604d3b17830 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.361+0000 7f9c9d368640 1 --2- 172.21.15.98:0/103442364 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604d3b17830 0x5604d3b19a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.361+0000 7f9c9d368640 1 --2- 172.21.15.98:0/103442364 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604d3b17830 0x5604d3b19a20 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=86e73142cd0b9aa4 server_cookie=bd91d3376b3f442f in_seq=0 out_seq=0 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.361+0000 7f9c80ff9640 10 client.15528 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9c80ff9640 1 -- 172.21.15.98:0/103442364 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5604d3a6c600 con 0x5604d3b17830 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9c80ff9640 10 client.15528 handle_command_reply: tid=0 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9ca19a5740 1 client.15528 shutdown 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9ca19a5740 1 -- 172.21.15.98:0/103442364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af8890 msgr2=0x5604d3b078b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/103442364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af8890 0x5604d3b078b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9c88033000 tx=0x7f9c880056d0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9ca19a5740 1 -- 172.21.15.98:0/103442364 shutdown_connections 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/103442364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3b07fc0 0x5604d3b05140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/103442364 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604d3b17830 0x5604d3b19a20 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/103442364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af91b0 0x5604d3b04c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/103442364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af8890 0x5604d3b078b0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9c80ff9640 0 client.15528 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:03:27.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9ca19a5740 1 -- 172.21.15.98:0/103442364 >> 172.21.15.98:0/103442364 conn(0x5604d3af4f60 msgr2=0x5604d3afacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:27.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9ca19a5740 1 -- 172.21.15.98:0/103442364 shutdown_connections 2024-09-19T08:03:27.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9ca19a5740 1 -- 172.21.15.98:0/103442364 wait complete. 2024-09-19T08:03:27.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.362+0000 7f9ca19a5740 20 client.15528 trim_cache size 0 max 16384 2024-09-19T08:03:27.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.384+0000 7f9ca19a5740 1 Processor -- start 2024-09-19T08:03:27.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.384+0000 7f9ca19a5740 1 -- start start 2024-09-19T08:03:27.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9ca19a5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af4f60 0x5604d3b044c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9ca19a5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af8890 0x5604d3af47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9ca19a5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af91b0 0x5604d3b21d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9ca19a5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5604d3b17d60 con 0x5604d3af91b0 2024-09-19T08:03:27.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9ca19a5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5604d3b18430 con 0x5604d3af8890 2024-09-19T08:03:27.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9ca19a5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5604d3b18b00 con 0x5604d3af4f60 2024-09-19T08:03:27.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c9cb67640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af91b0 0x5604d3b21d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c9cb67640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af91b0 0x5604d3b21d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45156/0 (socket says 172.21.15.98:45156) 2024-09-19T08:03:27.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c9cb67640 1 -- 172.21.15.98:0/1003255759 learned_addr learned my addr 172.21.15.98:0/1003255759 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:27.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/1003255759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af4f60 0x5604d3b044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c9d368640 1 --2- 172.21.15.98:0/1003255759 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af8890 0x5604d3af47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c9cb67640 1 -- 172.21.15.98:0/1003255759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af4f60 msgr2=0x5604d3b044c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:27.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/1003255759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af4f60 0x5604d3b044c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c9cb67640 1 -- 172.21.15.98:0/1003255759 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af8890 msgr2=0x5604d3af47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:27.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/1003255759 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af8890 0x5604d3af47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c9cb67640 1 -- 172.21.15.98:0/1003255759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604d3afb3d0 con 0x5604d3af91b0 2024-09-19T08:03:27.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c9d368640 1 --2- 172.21.15.98:0/1003255759 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af8890 0x5604d3af47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:27.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/1003255759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af91b0 0x5604d3b21d00 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9c88001eb0 tx=0x7f9c88005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8d2442985c8f73 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:27.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c82ffd640 1 -- 172.21.15.98:0/1003255759 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88059070 con 0x5604d3af91b0 2024-09-19T08:03:27.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c82ffd640 1 -- 172.21.15.98:0/1003255759 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c88046040 con 0x5604d3af91b0 2024-09-19T08:03:27.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.385+0000 7f9c82ffd640 1 -- 172.21.15.98:0/1003255759 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88041030 con 0x5604d3af91b0 2024-09-19T08:03:27.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.386+0000 7f9ca19a5740 1 -- 172.21.15.98:0/1003255759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af91b0 msgr2=0x5604d3b21d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:27.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.386+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/1003255759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af91b0 0x5604d3b21d00 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9c88001eb0 tx=0x7f9c88005740 comp rx=0 tx=0).stop 2024-09-19T08:03:27.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.386+0000 7f9ca19a5740 1 -- 172.21.15.98:0/1003255759 shutdown_connections 2024-09-19T08:03:27.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.386+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/1003255759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af91b0 0x5604d3b21d00 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.386+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/1003255759 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af8890 0x5604d3af47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.386+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/1003255759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af4f60 0x5604d3b044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.386+0000 7f9ca19a5740 1 -- 172.21.15.98:0/1003255759 >> 172.21.15.98:0/1003255759 conn(0x5604d3b052a0 msgr2=0x5604d3b00240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:27.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.386+0000 7f9ca19a5740 1 -- 172.21.15.98:0/1003255759 shutdown_connections 2024-09-19T08:03:27.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.386+0000 7f9ca19a5740 1 -- 172.21.15.98:0/1003255759 wait complete. 2024-09-19T08:03:27.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.387+0000 7f9ca19a5740 1 -- start start 2024-09-19T08:03:27.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.387+0000 7f9ca19a5740 1 Processor -- start 2024-09-19T08:03:27.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.387+0000 7f9ca19a5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af4f60 0x5604d3b22220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.387+0000 7f9ca19a5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af8890 0x5604d3b1b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.387+0000 7f9ca19a5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af91b0 0x5604d3b1ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.387+0000 7f9ca19a5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5604d3b198a0 con 0x5604d3af4f60 2024-09-19T08:03:27.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.387+0000 7f9ca19a5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5604d3b191d0 con 0x5604d3af91b0 2024-09-19T08:03:27.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.387+0000 7f9ca19a5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5604d3b19f70 con 0x5604d3af8890 2024-09-19T08:03:27.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.387+0000 7f9ca19a5740 10 client.0 _ll_register_callbacks cb 0x7f9c88043070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:27.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.387+0000 7f9c9cb67640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af4f60 0x5604d3b22220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.387+0000 7f9c9cb67640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af4f60 0x5604d3b22220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45160/0 (socket says 172.21.15.98:45160) 2024-09-19T08:03:27.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.387+0000 7f9c9cb67640 1 -- 172.21.15.98:0/3611074732 learned_addr learned my addr 172.21.15.98:0/3611074732 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:27.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.387+0000 7f9c9d368640 1 --2- 172.21.15.98:0/3611074732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af8890 0x5604d3b1b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/3611074732 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af91b0 0x5604d3b1ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9c9cb67640 1 -- 172.21.15.98:0/3611074732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af8890 msgr2=0x5604d3b1b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:27.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/3611074732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af8890 0x5604d3b1b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9c9cb67640 1 -- 172.21.15.98:0/3611074732 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af91b0 msgr2=0x5604d3b1ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/3611074732 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af91b0 0x5604d3b1ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9c9cb67640 1 -- 172.21.15.98:0/3611074732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604d3b1c0d0 con 0x5604d3af4f60 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9c9cb67640 1 --2- 172.21.15.98:0/3611074732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af4f60 0x5604d3b22220 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9c88004970 tx=0x7f9c88000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b639d97048b6f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9c80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9c80ff9640 1 -- 172.21.15.98:0/3611074732 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88064060 con 0x5604d3af4f60 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9c80ff9640 1 -- 172.21.15.98:0/3611074732 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f9c88052aa0 con 0x5604d3af4f60 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9ca19a5740 1 -- 172.21.15.98:0/3611074732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c88046040 con 0x5604d3af4f60 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9c80ff9640 1 -- 172.21.15.98:0/3611074732 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9c88059070 con 0x5604d3af4f60 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9ca19a5740 10 client.15534 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9ca19a5740 1 -- 172.21.15.98:0/3611074732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5604d3b20b20 con 0x5604d3af4f60 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.388+0000 7f9c80ff9640 1 -- 172.21.15.98:0/3611074732 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c88046040 con 0x5604d3af4f60 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.389+0000 7f9c80ff9640 1 -- 172.21.15.98:0/3611074732 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9c88049070 con 0x5604d3af4f60 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.389+0000 7f9ca19a5740 10 client.15534 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.389+0000 7f9ca19a5740 10 client.15534 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-19T08:03:27.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.389+0000 7f9ca19a5740 20 client.15534 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.389+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/3611074732 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604d3b07fc0 0x5604d3b86080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.389+0000 7f9ca19a5740 4 client.15534 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.389+0000 7f9ca19a5740 1 -- 172.21.15.98:0/3611074732 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "status"}) -- 0x5604d3a66b10 con 0x5604d3b07fc0 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.389+0000 7f9c9d368640 1 --2- 172.21.15.98:0/3611074732 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604d3b07fc0 0x5604d3b86080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.389+0000 7f9c80ff9640 10 client.15534 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.389+0000 7f9c9d368640 1 --2- 172.21.15.98:0/3611074732 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604d3b07fc0 0x5604d3b86080 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=71c1e6df1b329664 server_cookie=894dec5210769e16 in_seq=0 out_seq=0 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9c80ff9640 1 -- 172.21.15.98:0/3611074732 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5604d3a66b10 con 0x5604d3b07fc0 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9c80ff9640 10 client.15534 handle_command_reply: tid=0 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9ca19a5740 1 client.15534 shutdown 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9ca19a5740 1 -- 172.21.15.98:0/3611074732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af4f60 msgr2=0x5604d3b22220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/3611074732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af4f60 0x5604d3b22220 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9c88004970 tx=0x7f9c88000bc0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9ca19a5740 1 -- 172.21.15.98:0/3611074732 shutdown_connections 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/3611074732 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604d3af91b0 0x5604d3b1ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/3611074732 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604d3b07fc0 0x5604d3b86080 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/3611074732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604d3af8890 0x5604d3b1b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9ca19a5740 1 --2- 172.21.15.98:0/3611074732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604d3af4f60 0x5604d3b22220 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9ca19a5740 1 -- 172.21.15.98:0/3611074732 >> 172.21.15.98:0/3611074732 conn(0x5604d3b052a0 msgr2=0x5604d3b034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:27.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9c80ff9640 0 client.15534 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:03:27.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9ca19a5740 1 -- 172.21.15.98:0/3611074732 shutdown_connections 2024-09-19T08:03:27.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9ca19a5740 1 -- 172.21.15.98:0/3611074732 wait complete. 2024-09-19T08:03:27.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.390+0000 7f9ca19a5740 20 client.15534 trim_cache size 0 max 16384 2024-09-19T08:03:27.396 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:03:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:03:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 3, 2024-09-19T08:03:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24406, 2024-09-19T08:03:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:03:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:03:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:03:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 126.70594534999999, 2024-09-19T08:03:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-09-19T08:03:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 81, 2024-09-19T08:03:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 80, 2024-09-19T08:03:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 163.36463553999999 2024-09-19T08:03:27.398 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:03:27.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.391+0000 7f9c9f5cc640 1 -- 172.21.15.98:0/309524769 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9c840a34d0 msgr2=0x7f9c840a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:27.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.391+0000 7f9c9f5cc640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9c840a34d0 0x7f9c840a5990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9c900ccc10 tx=0x7f9c9006e4a0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.391+0000 7f9c9f5cc640 1 -- 172.21.15.98:0/309524769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c98106f10 msgr2=0x7f9c9819cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:27.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.391+0000 7f9c9f5cc640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c98106f10 0x7f9c9819cd40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000b970 tx=0x7f9c9000be30 comp rx=0 tx=0).stop 2024-09-19T08:03:27.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.391+0000 7f9c9f5cc640 1 -- 172.21.15.98:0/309524769 shutdown_connections 2024-09-19T08:03:27.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.391+0000 7f9c9f5cc640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9c840a34d0 0x7f9c840a5990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.391+0000 7f9c9f5cc640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c98106f10 0x7f9c9819cd40 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.391+0000 7f9c9f5cc640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c98102790 0x7f9c981989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.391+0000 7f9c9f5cc640 1 --2- 172.21.15.98:0/309524769 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c98101de0 0x7f9c98198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:27.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.391+0000 7f9c9f5cc640 1 -- 172.21.15.98:0/309524769 >> 172.21.15.98:0/309524769 conn(0x7f9c980fd8c0 msgr2=0x7f9c980fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:27.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.391+0000 7f9c9f5cc640 1 -- 172.21.15.98:0/309524769 shutdown_connections 2024-09-19T08:03:27.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:27.391+0000 7f9c9f5cc640 1 -- 172.21.15.98:0/309524769 wait complete. 2024-09-19T08:03:28.067 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.e status 2024-09-19T08:03:28.312 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:28.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.791+0000 7f8ffffc3640 1 -- 172.21.15.98:0/705808350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8100a80 msgr2=0x7f8ff8100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:28.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.791+0000 7f8ffffc3640 1 --2- 172.21.15.98:0/705808350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8100a80 0x7f8ff8100ee0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009930 tx=0x7f8fe802a420 comp rx=0 tx=0).stop 2024-09-19T08:03:28.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.791+0000 7f8ffffc3640 1 -- 172.21.15.98:0/705808350 shutdown_connections 2024-09-19T08:03:28.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.791+0000 7f8ffffc3640 1 --2- 172.21.15.98:0/705808350 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ff8101420 0x7f8ff810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.791+0000 7f8ffffc3640 1 --2- 172.21.15.98:0/705808350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8100a80 0x7f8ff8100ee0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.791+0000 7f8ffffc3640 1 --2- 172.21.15.98:0/705808350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ff8106a80 0x7f8ff8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.791+0000 7f8ffffc3640 1 -- 172.21.15.98:0/705808350 >> 172.21.15.98:0/705808350 conn(0x7f8ff80fc7a0 msgr2=0x7f8ff80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:28.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.791+0000 7f8ffffc3640 1 -- 172.21.15.98:0/705808350 shutdown_connections 2024-09-19T08:03:28.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.791+0000 7f8ffffc3640 1 -- 172.21.15.98:0/705808350 wait complete. 2024-09-19T08:03:28.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.792+0000 7f8ffffc3640 1 Processor -- start 2024-09-19T08:03:28.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.792+0000 7f8ffffc3640 1 -- start start 2024-09-19T08:03:28.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.792+0000 7f8ffffc3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ff8100a80 0x7f8ff8199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.792+0000 7f8ffffc3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8101420 0x7f8ff81997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.792+0000 7f8ffffc3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ff8106a80 0x7f8ff81a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.792+0000 7f8ffffc3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8110ea0 con 0x7f8ff8101420 2024-09-19T08:03:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.792+0000 7f8ffffc3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8110d20 con 0x7f8ff8106a80 2024-09-19T08:03:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.792+0000 7f8ffffc3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8111020 con 0x7f8ff8100a80 2024-09-19T08:03:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.792+0000 7f8ffd55e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8101420 0x7f8ff81997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.792+0000 7f8ffd55e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8101420 0x7f8ff81997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45186/0 (socket says 172.21.15.98:45186) 2024-09-19T08:03:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.792+0000 7f8ffd55e640 1 -- 172.21.15.98:0/2477426396 learned_addr learned my addr 172.21.15.98:0/2477426396 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.792+0000 7f8ffdd5f640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ff8100a80 0x7f8ff8199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.792+0000 7f8ffe560640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ff8106a80 0x7f8ff81a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.792+0000 7f8ffd55e640 1 -- 172.21.15.98:0/2477426396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ff8100a80 msgr2=0x7f8ff8199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.793+0000 7f8ffd55e640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ff8100a80 0x7f8ff8199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.793+0000 7f8ffd55e640 1 -- 172.21.15.98:0/2477426396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ff8106a80 msgr2=0x7f8ff81a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.793+0000 7f8ffd55e640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ff8106a80 0x7f8ff81a4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.793+0000 7f8ffd55e640 1 -- 172.21.15.98:0/2477426396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff81a4850 con 0x7f8ff8101420 2024-09-19T08:03:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.793+0000 7f8ffe560640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ff8106a80 0x7f8ff81a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.793+0000 7f8ffd55e640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8101420 0x7f8ff81997d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009a50 tx=0x7f8fe8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:28.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.793+0000 7f8feeffd640 1 -- 172.21.15.98:0/2477426396 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe803f030 con 0x7f8ff8101420 2024-09-19T08:03:28.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.793+0000 7f8feeffd640 1 -- 172.21.15.98:0/2477426396 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fe8003ad0 con 0x7f8ff8101420 2024-09-19T08:03:28.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.793+0000 7f8ffffc3640 1 -- 172.21.15.98:0/2477426396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff81a4a20 con 0x7f8ff8101420 2024-09-19T08:03:28.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.793+0000 7f8feeffd640 1 -- 172.21.15.98:0/2477426396 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8035070 con 0x7f8ff8101420 2024-09-19T08:03:28.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.793+0000 7f8ffffc3640 1 -- 172.21.15.98:0/2477426396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff81a4e70 con 0x7f8ff8101420 2024-09-19T08:03:28.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.795+0000 7f8feeffd640 1 -- 172.21.15.98:0/2477426396 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f8fe8042030 con 0x7f8ff8101420 2024-09-19T08:03:28.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.795+0000 7f900239c740 1 Processor -- start 2024-09-19T08:03:28.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f900239c740 1 -- start start 2024-09-19T08:03:28.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f900239c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 0x56550fcc5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f900239c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcc62f0 0x56550fcd0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f900239c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcd0bb0 0x56550fcd2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f900239c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56550fc2c090 con 0x56550fcc62f0 2024-09-19T08:03:28.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f900239c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56550fbd3a70 con 0x56550fcd0bb0 2024-09-19T08:03:28.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f900239c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56550fc28050 con 0x56550fcc59d0 2024-09-19T08:03:28.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f8ffdd5f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcc62f0 0x56550fcd0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f8ffdd5f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcc62f0 0x56550fcd0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45202/0 (socket says 172.21.15.98:45202) 2024-09-19T08:03:28.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f8ffdd5f640 1 -- 172.21.15.98:0/4047291316 learned_addr learned my addr 172.21.15.98:0/4047291316 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:28.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f8feeffd640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8fe40a34d0 0x7f8fe40a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f8ffd55e640 1 --2- 172.21.15.98:0/4047291316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcd0bb0 0x56550fcd2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f8ffe560640 1 --2- 172.21.15.98:0/4047291316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 0x56550fcc5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f8ffe560640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8fe40a34d0 0x7f8fe40a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f8ffdd5f640 1 -- 172.21.15.98:0/4047291316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 msgr2=0x56550fcc5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:28.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f8ffdd5f640 1 --2- 172.21.15.98:0/4047291316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 0x56550fcc5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f8ffdd5f640 1 -- 172.21.15.98:0/4047291316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcd0bb0 msgr2=0x56550fcd2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:28.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f8ffdd5f640 1 --2- 172.21.15.98:0/4047291316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcd0bb0 0x56550fcd2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f8ffdd5f640 1 -- 172.21.15.98:0/4047291316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56550fbe0130 con 0x56550fcc62f0 2024-09-19T08:03:28.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.796+0000 7f8ffe560640 1 --2- 172.21.15.98:0/4047291316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 0x56550fcc5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:28.841 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:03:28.841 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:03:28.841 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2024-09-19T08:03:28.841 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24437, 2024-09-19T08:03:28.842 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:03:28.842 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:03:28.842 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:03:28.842 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 128.99649693399999, 2024-09-19T08:03:28.842 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-09-19T08:03:28.842 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 81, 2024-09-19T08:03:28.842 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 80, 2024-09-19T08:03:28.842 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 167.28138898 2024-09-19T08:03:28.842 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:03:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.797+0000 7f8feeffd640 1 -- 172.21.15.98:0/2477426396 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f8fe80f0710 con 0x7f8ff8101420 2024-09-19T08:03:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.797+0000 7f8ffd55e640 1 --2- 172.21.15.98:0/4047291316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcd0bb0 0x56550fcd2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.797+0000 7f8ffdd5f640 1 --2- 172.21.15.98:0/4047291316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcc62f0 0x56550fcd0670 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400ad00 tx=0x7f8ff400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a8db18b6ed8675 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.797+0000 7f8fe3fff640 1 -- 172.21.15.98:0/4047291316 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff401d070 con 0x56550fcc62f0 2024-09-19T08:03:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.797+0000 7f8ffe560640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8fe40a34d0 0x7f8fe40a5990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009fd0 tx=0x7f8ff000c040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.797+0000 7f8fe3fff640 1 -- 172.21.15.98:0/4047291316 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ff4018b20 con 0x56550fcc62f0 2024-09-19T08:03:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.797+0000 7f8fe3fff640 1 -- 172.21.15.98:0/4047291316 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4010bc0 con 0x56550fcc62f0 2024-09-19T08:03:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.798+0000 7f900239c740 1 -- 172.21.15.98:0/4047291316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcc62f0 msgr2=0x56550fcd0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.798+0000 7f900239c740 1 --2- 172.21.15.98:0/4047291316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcc62f0 0x56550fcd0670 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400ad00 tx=0x7f8ff400f790 comp rx=0 tx=0).stop 2024-09-19T08:03:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.798+0000 7f900239c740 1 -- 172.21.15.98:0/4047291316 shutdown_connections 2024-09-19T08:03:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.798+0000 7f900239c740 1 --2- 172.21.15.98:0/4047291316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcd0bb0 0x56550fcd2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.798+0000 7f900239c740 1 --2- 172.21.15.98:0/4047291316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcc62f0 0x56550fcd0670 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.798+0000 7f900239c740 1 --2- 172.21.15.98:0/4047291316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 0x56550fcc5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.798+0000 7f900239c740 1 -- 172.21.15.98:0/4047291316 >> 172.21.15.98:0/4047291316 conn(0x56550fcc1f60 msgr2=0x56550fcc2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.798+0000 7f900239c740 1 -- 172.21.15.98:0/4047291316 shutdown_connections 2024-09-19T08:03:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.798+0000 7f900239c740 1 -- 172.21.15.98:0/4047291316 wait complete. 2024-09-19T08:03:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.798+0000 7f900239c740 1 -- start start 2024-09-19T08:03:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.798+0000 7f900239c740 1 Processor -- start 2024-09-19T08:03:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f900239c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 0x56550fcd03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f900239c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc62f0 0x56550fccd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f900239c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fccdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f900239c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56550fcd3780 con 0x56550fcd0bb0 2024-09-19T08:03:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f900239c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56550fbe6f20 con 0x56550fcc62f0 2024-09-19T08:03:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f900239c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56550fcd3900 con 0x56550fcc59d0 2024-09-19T08:03:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f900239c740 10 client.0 _ll_register_callbacks cb 0x56550fbb0030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f8ffe560640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fccdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f8ffe560640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fccdc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45214/0 (socket says 172.21.15.98:45214) 2024-09-19T08:03:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f8ffe560640 1 -- 172.21.15.98:0/3385559675 learned_addr learned my addr 172.21.15.98:0/3385559675 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f8ffdd5f640 1 --2- 172.21.15.98:0/3385559675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 0x56550fcd03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f8ffd55e640 1 --2- 172.21.15.98:0/3385559675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc62f0 0x56550fccd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f8ffe560640 1 -- 172.21.15.98:0/3385559675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 msgr2=0x56550fcd03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f8ffe560640 1 --2- 172.21.15.98:0/3385559675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 0x56550fcd03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f8ffe560640 1 -- 172.21.15.98:0/3385559675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc62f0 msgr2=0x56550fccd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f8ffe560640 1 --2- 172.21.15.98:0/3385559675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc62f0 0x56550fccd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f8ffe560640 1 -- 172.21.15.98:0/3385559675 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56550f99d2b0 con 0x56550fcd0bb0 2024-09-19T08:03:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f8ffe560640 1 --2- 172.21.15.98:0/3385559675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fccdc40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009fa0 tx=0x7f8ff0004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ae1edbf422d9d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f8fe17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f8fe17fa640 1 -- 172.21.15.98:0/3385559675 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0027060 con 0x56550fcd0bb0 2024-09-19T08:03:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f8fe17fa640 1 -- 172.21.15.98:0/3385559675 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f8ff0022400 con 0x56550fcd0bb0 2024-09-19T08:03:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.799+0000 7f900239c740 1 -- 172.21.15.98:0/3385559675 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ff4018b20 con 0x56550fcd0bb0 2024-09-19T08:03:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.800+0000 7f8fe17fa640 1 -- 172.21.15.98:0/3385559675 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8ff0013040 con 0x56550fcd0bb0 2024-09-19T08:03:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.800+0000 7f900239c740 10 client.15546 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.800+0000 7f900239c740 1 -- 172.21.15.98:0/3385559675 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56550fcdf0c0 con 0x56550fcd0bb0 2024-09-19T08:03:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.800+0000 7f8fe17fa640 1 -- 172.21.15.98:0/3385559675 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ff4018b20 con 0x56550fcd0bb0 2024-09-19T08:03:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.800+0000 7f8fe17fa640 1 -- 172.21.15.98:0/3385559675 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8ff0021660 con 0x56550fcd0bb0 2024-09-19T08:03:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.800+0000 7f900239c740 10 client.15546 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.800+0000 7f900239c740 10 client.15546 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-19T08:03:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.800+0000 7f900239c740 20 client.15546 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.800+0000 7f900239c740 1 --2- 172.21.15.98:0/3385559675 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x56550fce1720 0x56550fce2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.800+0000 7f900239c740 4 client.15546 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.800+0000 7f900239c740 1 -- 172.21.15.98:0/3385559675 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56550fc39600 con 0x56550fce1720 2024-09-19T08:03:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.800+0000 7f8ffdd5f640 1 --2- 172.21.15.98:0/3385559675 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x56550fce1720 0x56550fce2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.801+0000 7f8fe17fa640 10 client.15546 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:03:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.801+0000 7f8ffdd5f640 1 --2- 172.21.15.98:0/3385559675 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x56550fce1720 0x56550fce2b90 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5ae6bbb458c10384 server_cookie=23ce51516d409c70 in_seq=0 out_seq=0 2024-09-19T08:03:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.801+0000 7f8fe17fa640 1 -- 172.21.15.98:0/3385559675 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56550fc39600 con 0x56550fce1720 2024-09-19T08:03:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.801+0000 7f8fe17fa640 10 client.15546 handle_command_reply: tid=0 2024-09-19T08:03:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.801+0000 7f900239c740 1 client.15546 shutdown 2024-09-19T08:03:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.801+0000 7f900239c740 1 -- 172.21.15.98:0/3385559675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 msgr2=0x56550fccdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.801+0000 7f900239c740 1 --2- 172.21.15.98:0/3385559675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fccdc40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009fa0 tx=0x7f8ff0004b50 comp rx=0 tx=0).stop 2024-09-19T08:03:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.802+0000 7f900239c740 1 -- 172.21.15.98:0/3385559675 shutdown_connections 2024-09-19T08:03:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.802+0000 7f900239c740 1 --2- 172.21.15.98:0/3385559675 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x56550fce1720 0x56550fce2b90 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.802+0000 7f900239c740 1 --2- 172.21.15.98:0/3385559675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fccdc40 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.802+0000 7f8fe17fa640 0 client.15546 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:03:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.802+0000 7f900239c740 1 --2- 172.21.15.98:0/3385559675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc62f0 0x56550fccd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.802+0000 7f900239c740 1 --2- 172.21.15.98:0/3385559675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 0x56550fcd03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.802+0000 7f900239c740 1 -- 172.21.15.98:0/3385559675 >> 172.21.15.98:0/3385559675 conn(0x56550fcc1f60 msgr2=0x56550fcc7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.802+0000 7f900239c740 1 -- 172.21.15.98:0/3385559675 shutdown_connections 2024-09-19T08:03:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.802+0000 7f900239c740 1 -- 172.21.15.98:0/3385559675 wait complete. 2024-09-19T08:03:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.802+0000 7f900239c740 20 client.15546 trim_cache size 0 max 16384 2024-09-19T08:03:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.819+0000 7f900239c740 1 Processor -- start 2024-09-19T08:03:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.819+0000 7f900239c740 1 -- start start 2024-09-19T08:03:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f900239c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc59d0 0x56550fce2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f900239c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc62f0 0x56550fccd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f900239c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fccdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f900239c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56550fce0ed0 con 0x56550fcd0bb0 2024-09-19T08:03:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f900239c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56550fce15a0 con 0x56550fcc59d0 2024-09-19T08:03:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f900239c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56550fce1c70 con 0x56550fcc62f0 2024-09-19T08:03:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8ffe560640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fccdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8ffe560640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fccdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45230/0 (socket says 172.21.15.98:45230) 2024-09-19T08:03:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8ffe560640 1 -- 172.21.15.98:0/3709194821 learned_addr learned my addr 172.21.15.98:0/3709194821 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8ffdd5f640 1 --2- 172.21.15.98:0/3709194821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc59d0 0x56550fce2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8ffd55e640 1 --2- 172.21.15.98:0/3709194821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc62f0 0x56550fccd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8ffe560640 1 -- 172.21.15.98:0/3709194821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc62f0 msgr2=0x56550fccd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8ffe560640 1 --2- 172.21.15.98:0/3709194821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc62f0 0x56550fccd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8ffe560640 1 -- 172.21.15.98:0/3709194821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc59d0 msgr2=0x56550fce2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8ffe560640 1 --2- 172.21.15.98:0/3709194821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc59d0 0x56550fce2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8ffe560640 1 -- 172.21.15.98:0/3709194821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56550fcc1d90 con 0x56550fcd0bb0 2024-09-19T08:03:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8ffdd5f640 1 --2- 172.21.15.98:0/3709194821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc59d0 0x56550fce2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8ffe560640 1 --2- 172.21.15.98:0/3709194821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fccdc00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009440 tx=0x7f8ff0009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeaa58d7a8b7bbbd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8fe37fe640 1 -- 172.21.15.98:0/3709194821 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff003a070 con 0x56550fcd0bb0 2024-09-19T08:03:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8fe37fe640 1 -- 172.21.15.98:0/3709194821 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ff0021ca0 con 0x56550fcd0bb0 2024-09-19T08:03:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.820+0000 7f8fe37fe640 1 -- 172.21.15.98:0/3709194821 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0035a60 con 0x56550fcd0bb0 2024-09-19T08:03:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.821+0000 7f900239c740 1 -- 172.21.15.98:0/3709194821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 msgr2=0x56550fccdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.821+0000 7f900239c740 1 --2- 172.21.15.98:0/3709194821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fccdc00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009440 tx=0x7f8ff0009710 comp rx=0 tx=0).stop 2024-09-19T08:03:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.821+0000 7f900239c740 1 -- 172.21.15.98:0/3709194821 shutdown_connections 2024-09-19T08:03:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.821+0000 7f900239c740 1 --2- 172.21.15.98:0/3709194821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fccdc00 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.821+0000 7f900239c740 1 --2- 172.21.15.98:0/3709194821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc62f0 0x56550fccd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.821+0000 7f900239c740 1 --2- 172.21.15.98:0/3709194821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc59d0 0x56550fce2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.821+0000 7f900239c740 1 -- 172.21.15.98:0/3709194821 >> 172.21.15.98:0/3709194821 conn(0x56550fcc1f60 msgr2=0x56550fcccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.821+0000 7f900239c740 1 -- 172.21.15.98:0/3709194821 shutdown_connections 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.821+0000 7f900239c740 1 -- 172.21.15.98:0/3709194821 wait complete. 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.821+0000 7f900239c740 1 -- start start 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.821+0000 7f900239c740 1 Processor -- start 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f900239c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 0x56550fce41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f900239c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc62f0 0x56550fce4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f900239c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fd49e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f900239c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56550fce2a10 con 0x56550fcd0bb0 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f900239c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56550fce2340 con 0x56550fcc62f0 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f900239c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56550fce30e0 con 0x56550fcc59d0 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f900239c740 10 client.0 _ll_register_callbacks cb 0x7f8ff00178c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f8ffe560640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fd49e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f8ffe560640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fd49e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45236/0 (socket says 172.21.15.98:45236) 2024-09-19T08:03:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f8ffe560640 1 -- 172.21.15.98:0/2780224575 learned_addr learned my addr 172.21.15.98:0/2780224575 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f8ffdd5f640 1 --2- 172.21.15.98:0/2780224575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 0x56550fce41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f8ffe560640 1 -- 172.21.15.98:0/2780224575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 msgr2=0x56550fce41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f8ffe560640 1 --2- 172.21.15.98:0/2780224575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 0x56550fce41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f8ffe560640 1 -- 172.21.15.98:0/2780224575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc62f0 msgr2=0x56550fce4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f8ffd55e640 1 --2- 172.21.15.98:0/2780224575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc62f0 0x56550fce4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f8ffe560640 1 --2- 172.21.15.98:0/2780224575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc62f0 0x56550fce4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f8ffe560640 1 -- 172.21.15.98:0/2780224575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56550fd4a360 con 0x56550fcd0bb0 2024-09-19T08:03:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f8ffe560640 1 --2- 172.21.15.98:0/2780224575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fd49e20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0021a30 tx=0x7f8ff0035480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac66c8b2ee6c5ac6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f8fe17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f8fe17fa640 1 -- 172.21.15.98:0/2780224575 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff003a070 con 0x56550fcd0bb0 2024-09-19T08:03:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f8fe17fa640 1 -- 172.21.15.98:0/2780224575 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f8ff00217b0 con 0x56550fcd0bb0 2024-09-19T08:03:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.822+0000 7f900239c740 1 -- 172.21.15.98:0/2780224575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ff0021ca0 con 0x56550fcd0bb0 2024-09-19T08:03:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.823+0000 7f8fe17fa640 1 -- 172.21.15.98:0/2780224575 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8ff0027040 con 0x56550fcd0bb0 2024-09-19T08:03:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.823+0000 7f900239c740 10 client.15552 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.823+0000 7f900239c740 1 -- 172.21.15.98:0/2780224575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56550fd4b530 con 0x56550fcd0bb0 2024-09-19T08:03:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.823+0000 7f8fe17fa640 1 -- 172.21.15.98:0/2780224575 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ff0021ca0 con 0x56550fcd0bb0 2024-09-19T08:03:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.823+0000 7f8fe17fa640 1 -- 172.21.15.98:0/2780224575 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8ff002c860 con 0x56550fcd0bb0 2024-09-19T08:03:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.823+0000 7f900239c740 10 client.15552 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.823+0000 7f900239c740 10 client.15552 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-19T08:03:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.823+0000 7f900239c740 20 client.15552 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.823+0000 7f900239c740 1 --2- 172.21.15.98:0/2780224575 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x56550fd4fd20 0x56550fd4ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.823+0000 7f900239c740 4 client.15552 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:03:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.823+0000 7f900239c740 1 -- 172.21.15.98:0/2780224575 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "status"}) -- 0x56550fc33b10 con 0x56550fd4fd20 2024-09-19T08:03:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.823+0000 7f8ffdd5f640 1 --2- 172.21.15.98:0/2780224575 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x56550fd4fd20 0x56550fd4ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f8fe17fa640 10 client.15552 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:03:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f8ffdd5f640 1 --2- 172.21.15.98:0/2780224575 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x56550fd4fd20 0x56550fd4ea30 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e4a4e035f6682e47 server_cookie=753224ce89560fa6 in_seq=0 out_seq=0 2024-09-19T08:03:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f8fe17fa640 1 -- 172.21.15.98:0/2780224575 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x56550fc33b10 con 0x56550fd4fd20 2024-09-19T08:03:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f8fe17fa640 10 client.15552 handle_command_reply: tid=0 2024-09-19T08:03:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f900239c740 1 client.15552 shutdown 2024-09-19T08:03:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f900239c740 1 -- 172.21.15.98:0/2780224575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 msgr2=0x56550fd49e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f900239c740 1 --2- 172.21.15.98:0/2780224575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fd49e20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0021a30 tx=0x7f8ff0035480 comp rx=0 tx=0).stop 2024-09-19T08:03:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f900239c740 1 -- 172.21.15.98:0/2780224575 shutdown_connections 2024-09-19T08:03:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f900239c740 1 --2- 172.21.15.98:0/2780224575 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x56550fd4fd20 0x56550fd4ea30 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f900239c740 1 --2- 172.21.15.98:0/2780224575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56550fcd0bb0 0x56550fd49e20 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f900239c740 1 --2- 172.21.15.98:0/2780224575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56550fcc62f0 0x56550fce4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f8fe17fa640 0 client.15552 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:03:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f900239c740 1 --2- 172.21.15.98:0/2780224575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56550fcc59d0 0x56550fce41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f900239c740 1 -- 172.21.15.98:0/2780224575 >> 172.21.15.98:0/2780224575 conn(0x56550fcc1f60 msgr2=0x56550fccd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f900239c740 1 -- 172.21.15.98:0/2780224575 shutdown_connections 2024-09-19T08:03:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f900239c740 1 -- 172.21.15.98:0/2780224575 wait complete. 2024-09-19T08:03:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.824+0000 7f900239c740 20 client.15552 trim_cache size 0 max 16384 2024-09-19T08:03:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.825+0000 7f8ffffc3640 1 -- 172.21.15.98:0/2477426396 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8fe40a34d0 msgr2=0x7f8fe40a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.825+0000 7f8ffffc3640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8fe40a34d0 0x7f8fe40a5990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009fd0 tx=0x7f8ff000c040 comp rx=0 tx=0).stop 2024-09-19T08:03:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.825+0000 7f8ffffc3640 1 -- 172.21.15.98:0/2477426396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8101420 msgr2=0x7f8ff81997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.825+0000 7f8ffffc3640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8101420 0x7f8ff81997d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009a50 tx=0x7f8fe8004270 comp rx=0 tx=0).stop 2024-09-19T08:03:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.825+0000 7f8ffffc3640 1 -- 172.21.15.98:0/2477426396 shutdown_connections 2024-09-19T08:03:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.825+0000 7f8ffffc3640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8fe40a34d0 0x7f8fe40a5990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.825+0000 7f8ffffc3640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ff8106a80 0x7f8ff81a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.825+0000 7f8ffffc3640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ff8101420 0x7f8ff81997d0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.825+0000 7f8ffffc3640 1 --2- 172.21.15.98:0/2477426396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ff8100a80 0x7f8ff8199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.825+0000 7f8ffffc3640 1 -- 172.21.15.98:0/2477426396 >> 172.21.15.98:0/2477426396 conn(0x7f8ff80fc7a0 msgr2=0x7f8ff80fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:28.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.825+0000 7f8ffffc3640 1 -- 172.21.15.98:0/2477426396 shutdown_connections 2024-09-19T08:03:28.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:28.825+0000 7f8ffffc3640 1 -- 172.21.15.98:0/2477426396 wait complete. 2024-09-19T08:03:29.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:28 smithi162 ceph-mon[35260]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 12 op/s 2024-09-19T08:03:29.122 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:28 smithi098 ceph-mon[31333]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 12 op/s 2024-09-19T08:03:29.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:28 smithi139 ceph-mon[34874]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 12 op/s 2024-09-19T08:03:29.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-19T08:03:29.726 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:30.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.190+0000 7fa05f792640 1 -- 172.21.15.98:0/591497741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa05810c170 msgr2=0x7fa05810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:30.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.190+0000 7fa05f792640 1 --2- 172.21.15.98:0/591497741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa05810c170 0x7fa05810e610 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa050009ca0 tx=0x7fa05002a950 comp rx=0 tx=0).stop 2024-09-19T08:03:30.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.190+0000 7fa05f792640 1 -- 172.21.15.98:0/591497741 shutdown_connections 2024-09-19T08:03:30.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.190+0000 7fa05f792640 1 --2- 172.21.15.98:0/591497741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa05810c170 0x7fa05810e610 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.190+0000 7fa05f792640 1 --2- 172.21.15.98:0/591497741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa058103660 0x7fa05810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.190+0000 7fa05f792640 1 --2- 172.21.15.98:0/591497741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa058102d40 0x7fa058103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.190+0000 7fa05f792640 1 -- 172.21.15.98:0/591497741 >> 172.21.15.98:0/591497741 conn(0x7fa0580fc7a0 msgr2=0x7fa0580febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:30.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.190+0000 7fa05f792640 1 -- 172.21.15.98:0/591497741 shutdown_connections 2024-09-19T08:03:30.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.190+0000 7fa05f792640 1 -- 172.21.15.98:0/591497741 wait complete. 2024-09-19T08:03:30.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05f792640 1 Processor -- start 2024-09-19T08:03:30.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05f792640 1 -- start start 2024-09-19T08:03:30.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05f792640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa058102d40 0x7fa058198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05f792640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa058103660 0x7fa058198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05f792640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa05810c170 0x7fa05819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05f792640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa058111f70 con 0x7fa058102d40 2024-09-19T08:03:30.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05f792640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa058111df0 con 0x7fa05810c170 2024-09-19T08:03:30.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05f792640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa0581120f0 con 0x7fa058103660 2024-09-19T08:03:30.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05d52e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa058102d40 0x7fa058198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05d52e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa058102d40 0x7fa058198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45240/0 (socket says 172.21.15.98:45240) 2024-09-19T08:03:30.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05d52e640 1 -- 172.21.15.98:0/3982037643 learned_addr learned my addr 172.21.15.98:0/3982037643 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:30.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05dd2f640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa05810c170 0x7fa05819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05cd2d640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa058103660 0x7fa058198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05d52e640 1 -- 172.21.15.98:0/3982037643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa058103660 msgr2=0x7fa058198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:30.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05d52e640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa058103660 0x7fa058198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05d52e640 1 -- 172.21.15.98:0/3982037643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa05810c170 msgr2=0x7fa05819cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:30.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05d52e640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa05810c170 0x7fa05819cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05d52e640 1 -- 172.21.15.98:0/3982037643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05819d3c0 con 0x7fa058102d40 2024-09-19T08:03:30.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.191+0000 7fa05dd2f640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa05810c170 0x7fa05819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:30.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.192+0000 7fa05d52e640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa058102d40 0x7fa058198410 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa04800e9f0 tx=0x7fa04800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:30.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.192+0000 7fa04e7fc640 1 -- 172.21.15.98:0/3982037643 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04800cd80 con 0x7fa058102d40 2024-09-19T08:03:30.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.192+0000 7fa04e7fc640 1 -- 172.21.15.98:0/3982037643 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa048005500 con 0x7fa058102d40 2024-09-19T08:03:30.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.192+0000 7fa05f792640 1 -- 172.21.15.98:0/3982037643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa05819d6b0 con 0x7fa058102d40 2024-09-19T08:03:30.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.192+0000 7fa04e7fc640 1 -- 172.21.15.98:0/3982037643 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa048010690 con 0x7fa058102d40 2024-09-19T08:03:30.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.192+0000 7fa05f792640 1 -- 172.21.15.98:0/3982037643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0581a4f90 con 0x7fa058102d40 2024-09-19T08:03:30.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.194+0000 7fa04e7fc640 1 -- 172.21.15.98:0/3982037643 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7fa0480046c0 con 0x7fa058102d40 2024-09-19T08:03:30.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.194+0000 7fa061b6b740 1 Processor -- start 2024-09-19T08:03:30.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.194+0000 7fa061b6b740 1 -- start start 2024-09-19T08:03:30.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa061b6b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684579de0 0x55b68457a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa061b6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b68457a700 0x55b684584a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa061b6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b684584fc0 0x55b6845873b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa061b6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b6844dc090 con 0x55b68457a700 2024-09-19T08:03:30.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa061b6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b684483a70 con 0x55b684584fc0 2024-09-19T08:03:30.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa061b6b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b6844d8050 con 0x55b684579de0 2024-09-19T08:03:30.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa05d52e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b68457a700 0x55b684584a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa05d52e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b68457a700 0x55b684584a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45246/0 (socket says 172.21.15.98:45246) 2024-09-19T08:03:30.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa05d52e640 1 -- 172.21.15.98:0/1734645859 learned_addr learned my addr 172.21.15.98:0/1734645859 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:30.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa05cd2d640 1 --2- 172.21.15.98:0/1734645859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b684584fc0 0x55b6845873b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa05dd2f640 1 --2- 172.21.15.98:0/1734645859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684579de0 0x55b68457a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa04e7fc640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa0440a34d0 0x7fa0440a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa05d52e640 1 -- 172.21.15.98:0/1734645859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684579de0 msgr2=0x55b68457a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:30.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa05d52e640 1 --2- 172.21.15.98:0/1734645859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684579de0 0x55b68457a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa05d52e640 1 -- 172.21.15.98:0/1734645859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b684584fc0 msgr2=0x55b6845873b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:30.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa05d52e640 1 --2- 172.21.15.98:0/1734645859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b684584fc0 0x55b6845873b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa05d52e640 1 -- 172.21.15.98:0/1734645859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b684490130 con 0x55b68457a700 2024-09-19T08:03:30.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa05dd2f640 1 --2- 172.21.15.98:0/1734645859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684579de0 0x55b68457a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:30.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa05dd2f640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa0440a34d0 0x7fa0440a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa04e7fc640 1 -- 172.21.15.98:0/3982037643 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fa0480d0540 con 0x7fa058102d40 2024-09-19T08:03:30.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.195+0000 7fa05d52e640 1 --2- 172.21.15.98:0/1734645859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b68457a700 0x55b684584a80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa0480cdb70 tx=0x7fa048004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e219128b2fc0373c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:30.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.196+0000 7fa0437fe640 1 -- 172.21.15.98:0/1734645859 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04809c370 con 0x55b68457a700 2024-09-19T08:03:30.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.196+0000 7fa0437fe640 1 -- 172.21.15.98:0/1734645859 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa04809cca0 con 0x55b68457a700 2024-09-19T08:03:30.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.196+0000 7fa0437fe640 1 -- 172.21.15.98:0/1734645859 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04800ad80 con 0x55b68457a700 2024-09-19T08:03:30.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.196+0000 7fa061b6b740 1 -- 172.21.15.98:0/1734645859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b68457a700 msgr2=0x55b684584a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:30.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.196+0000 7fa05dd2f640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa0440a34d0 0x7fa0440a5990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa050030040 tx=0x7fa0500095c0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:30.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.196+0000 7fa061b6b740 1 --2- 172.21.15.98:0/1734645859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b68457a700 0x55b684584a80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa0480cdb70 tx=0x7fa048004a40 comp rx=0 tx=0).stop 2024-09-19T08:03:30.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.196+0000 7fa061b6b740 1 -- 172.21.15.98:0/1734645859 shutdown_connections 2024-09-19T08:03:30.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.196+0000 7fa061b6b740 1 --2- 172.21.15.98:0/1734645859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b684584fc0 0x55b6845873b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.196+0000 7fa061b6b740 1 --2- 172.21.15.98:0/1734645859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b68457a700 0x55b684584a80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.196+0000 7fa061b6b740 1 --2- 172.21.15.98:0/1734645859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684579de0 0x55b68457a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.196+0000 7fa061b6b740 1 -- 172.21.15.98:0/1734645859 >> 172.21.15.98:0/1734645859 conn(0x55b684576370 msgr2=0x55b684576750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:30.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.196+0000 7fa061b6b740 1 -- 172.21.15.98:0/1734645859 shutdown_connections 2024-09-19T08:03:30.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.196+0000 7fa061b6b740 1 -- 172.21.15.98:0/1734645859 wait complete. 2024-09-19T08:03:30.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.197+0000 7fa061b6b740 1 -- start start 2024-09-19T08:03:30.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.197+0000 7fa061b6b740 1 Processor -- start 2024-09-19T08:03:30.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.197+0000 7fa061b6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b684579de0 0x55b684584800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.197+0000 7fa061b6b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b68457a700 0x55b684581b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.197+0000 7fa061b6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684584fc0 0x55b684582090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.197+0000 7fa061b6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b684587b30 con 0x55b684584fc0 2024-09-19T08:03:30.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.197+0000 7fa061b6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b684496f20 con 0x55b684579de0 2024-09-19T08:03:30.236 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2024-09-19T08:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24461, 2024-09-19T08:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:03:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 131.406503625, 2024-09-19T08:03:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-09-19T08:03:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 81, 2024-09-19T08:03:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 80, 2024-09-19T08:03:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 161.452995012 2024-09-19T08:03:30.239 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:03:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.197+0000 7fa061b6b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b684587cb0 con 0x55b68457a700 2024-09-19T08:03:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.197+0000 7fa061b6b740 10 client.0 _ll_register_callbacks cb 0x55b684460030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.197+0000 7fa05dd2f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684584fc0 0x55b684582090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.197+0000 7fa05dd2f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684584fc0 0x55b684582090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45250/0 (socket says 172.21.15.98:45250) 2024-09-19T08:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.197+0000 7fa05dd2f640 1 -- 172.21.15.98:0/919917274 learned_addr learned my addr 172.21.15.98:0/919917274 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.197+0000 7fa05cd2d640 1 --2- 172.21.15.98:0/919917274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b68457a700 0x55b684581b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.197+0000 7fa05d52e640 1 --2- 172.21.15.98:0/919917274 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b684579de0 0x55b684584800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.198+0000 7fa05dd2f640 1 -- 172.21.15.98:0/919917274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b68457a700 msgr2=0x55b684581b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.198+0000 7fa05dd2f640 1 --2- 172.21.15.98:0/919917274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b68457a700 0x55b684581b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.198+0000 7fa05dd2f640 1 -- 172.21.15.98:0/919917274 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b684579de0 msgr2=0x55b684584800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.198+0000 7fa05dd2f640 1 --2- 172.21.15.98:0/919917274 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b684579de0 0x55b684584800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.198+0000 7fa05dd2f640 1 -- 172.21.15.98:0/919917274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b68424d2b0 con 0x55b684584fc0 2024-09-19T08:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.198+0000 7fa05dd2f640 1 --2- 172.21.15.98:0/919917274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684584fc0 0x55b684582090 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa05002f4c0 tx=0x7fa05002fd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b599c58c2e91a6f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.198+0000 7fa040ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.198+0000 7fa040ff9640 1 -- 172.21.15.98:0/919917274 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa050034040 con 0x55b684584fc0 2024-09-19T08:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.198+0000 7fa040ff9640 1 -- 172.21.15.98:0/919917274 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fa050003dc0 con 0x55b684584fc0 2024-09-19T08:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.198+0000 7fa061b6b740 1 -- 172.21.15.98:0/919917274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa04809cca0 con 0x55b684584fc0 2024-09-19T08:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.198+0000 7fa040ff9640 1 -- 172.21.15.98:0/919917274 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa050004040 con 0x55b684584fc0 2024-09-19T08:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.198+0000 7fa061b6b740 10 client.15564 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.198+0000 7fa061b6b740 1 -- 172.21.15.98:0/919917274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b684593390 con 0x55b684584fc0 2024-09-19T08:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.198+0000 7fa040ff9640 1 -- 172.21.15.98:0/919917274 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa04809cca0 con 0x55b684584fc0 2024-09-19T08:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.199+0000 7fa040ff9640 1 -- 172.21.15.98:0/919917274 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa050059b00 con 0x55b684584fc0 2024-09-19T08:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.199+0000 7fa061b6b740 10 client.15564 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.199+0000 7fa061b6b740 10 client.15564 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-19T08:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.199+0000 7fa061b6b740 20 client.15564 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.199+0000 7fa061b6b740 1 --2- 172.21.15.98:0/919917274 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55b684595a00 0x55b684596e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.199+0000 7fa061b6b740 4 client.15564 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.199+0000 7fa061b6b740 1 -- 172.21.15.98:0/919917274 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6844e9600 con 0x55b684595a00 2024-09-19T08:03:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.199+0000 7fa05d52e640 1 --2- 172.21.15.98:0/919917274 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55b684595a00 0x55b684596e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.200+0000 7fa040ff9640 10 client.15564 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:03:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.200+0000 7fa05d52e640 1 --2- 172.21.15.98:0/919917274 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55b684595a00 0x55b684596e70 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ac4a258fbeaa07e5 server_cookie=92a95a688e39fdb0 in_seq=0 out_seq=0 2024-09-19T08:03:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.200+0000 7fa040ff9640 1 -- 172.21.15.98:0/919917274 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b6844e9600 con 0x55b684595a00 2024-09-19T08:03:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.200+0000 7fa040ff9640 10 client.15564 handle_command_reply: tid=0 2024-09-19T08:03:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.200+0000 7fa061b6b740 1 client.15564 shutdown 2024-09-19T08:03:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.200+0000 7fa061b6b740 1 -- 172.21.15.98:0/919917274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684584fc0 msgr2=0x55b684582090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.201+0000 7fa061b6b740 1 --2- 172.21.15.98:0/919917274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684584fc0 0x55b684582090 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa05002f4c0 tx=0x7fa05002fd50 comp rx=0 tx=0).stop 2024-09-19T08:03:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.201+0000 7fa061b6b740 1 -- 172.21.15.98:0/919917274 shutdown_connections 2024-09-19T08:03:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.201+0000 7fa061b6b740 1 --2- 172.21.15.98:0/919917274 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55b684595a00 0x55b684596e70 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.201+0000 7fa061b6b740 1 --2- 172.21.15.98:0/919917274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684584fc0 0x55b684582090 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.201+0000 7fa061b6b740 1 --2- 172.21.15.98:0/919917274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b68457a700 0x55b684581b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.201+0000 7fa061b6b740 1 --2- 172.21.15.98:0/919917274 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b684579de0 0x55b684584800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.201+0000 7fa061b6b740 1 -- 172.21.15.98:0/919917274 >> 172.21.15.98:0/919917274 conn(0x55b684576370 msgr2=0x55b68457c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.201+0000 7fa040ff9640 0 client.15564 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:03:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.201+0000 7fa061b6b740 1 -- 172.21.15.98:0/919917274 shutdown_connections 2024-09-19T08:03:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.201+0000 7fa061b6b740 1 -- 172.21.15.98:0/919917274 wait complete. 2024-09-19T08:03:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.201+0000 7fa061b6b740 20 client.15564 trim_cache size 0 max 16384 2024-09-19T08:03:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.218+0000 7fa061b6b740 1 Processor -- start 2024-09-19T08:03:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.218+0000 7fa061b6b740 1 -- start start 2024-09-19T08:03:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.218+0000 7fa061b6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684579de0 0x55b684595a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.218+0000 7fa061b6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b68457a700 0x55b684581410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.218+0000 7fa061b6b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684584fc0 0x55b684580420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.218+0000 7fa061b6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b6845951b0 con 0x55b684579de0 2024-09-19T08:03:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.218+0000 7fa061b6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b684595880 con 0x55b68457a700 2024-09-19T08:03:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.218+0000 7fa061b6b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b684595f50 con 0x55b684584fc0 2024-09-19T08:03:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.218+0000 7fa05d52e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684579de0 0x55b684595a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.218+0000 7fa05d52e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684579de0 0x55b684595a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45256/0 (socket says 172.21.15.98:45256) 2024-09-19T08:03:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.218+0000 7fa05d52e640 1 -- 172.21.15.98:0/1363807071 learned_addr learned my addr 172.21.15.98:0/1363807071 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.219+0000 7fa05dd2f640 1 --2- 172.21.15.98:0/1363807071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684584fc0 0x55b684580420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.219+0000 7fa05cd2d640 1 --2- 172.21.15.98:0/1363807071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b68457a700 0x55b684581410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.219+0000 7fa05d52e640 1 -- 172.21.15.98:0/1363807071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684584fc0 msgr2=0x55b684580420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.219+0000 7fa05d52e640 1 --2- 172.21.15.98:0/1363807071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684584fc0 0x55b684580420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.219+0000 7fa05d52e640 1 -- 172.21.15.98:0/1363807071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b68457a700 msgr2=0x55b684581410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.219+0000 7fa05d52e640 1 --2- 172.21.15.98:0/1363807071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b68457a700 0x55b684581410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.219+0000 7fa05d52e640 1 -- 172.21.15.98:0/1363807071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6845761a0 con 0x55b684579de0 2024-09-19T08:03:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.219+0000 7fa05cd2d640 1 --2- 172.21.15.98:0/1363807071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b68457a700 0x55b684581410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.219+0000 7fa05d52e640 1 --2- 172.21.15.98:0/1363807071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684579de0 0x55b684595a00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa0480049d0 tx=0x7fa048098240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beeb4f12197e23e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.219+0000 7fa042ffd640 1 -- 172.21.15.98:0/1363807071 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04809bf00 con 0x55b684579de0 2024-09-19T08:03:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.219+0000 7fa042ffd640 1 -- 172.21.15.98:0/1363807071 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa04809c5a0 con 0x55b684579de0 2024-09-19T08:03:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.219+0000 7fa042ffd640 1 -- 172.21.15.98:0/1363807071 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04809c0e0 con 0x55b684579de0 2024-09-19T08:03:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.220+0000 7fa061b6b740 1 -- 172.21.15.98:0/1363807071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684579de0 msgr2=0x55b684595a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.220+0000 7fa061b6b740 1 --2- 172.21.15.98:0/1363807071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684579de0 0x55b684595a00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa0480049d0 tx=0x7fa048098240 comp rx=0 tx=0).stop 2024-09-19T08:03:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.220+0000 7fa061b6b740 1 -- 172.21.15.98:0/1363807071 shutdown_connections 2024-09-19T08:03:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.220+0000 7fa061b6b740 1 --2- 172.21.15.98:0/1363807071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684584fc0 0x55b684580420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.220+0000 7fa061b6b740 1 --2- 172.21.15.98:0/1363807071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b68457a700 0x55b684581410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.220+0000 7fa061b6b740 1 --2- 172.21.15.98:0/1363807071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684579de0 0x55b684595a00 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.220+0000 7fa061b6b740 1 -- 172.21.15.98:0/1363807071 >> 172.21.15.98:0/1363807071 conn(0x55b684576370 msgr2=0x55b684581b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.220+0000 7fa061b6b740 1 -- 172.21.15.98:0/1363807071 shutdown_connections 2024-09-19T08:03:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.220+0000 7fa061b6b740 1 -- 172.21.15.98:0/1363807071 wait complete. 2024-09-19T08:03:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.220+0000 7fa061b6b740 1 -- start start 2024-09-19T08:03:30.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.220+0000 7fa061b6b740 1 Processor -- start 2024-09-19T08:03:30.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.220+0000 7fa061b6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684579de0 0x55b68459f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa061b6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b68457a700 0x55b684599cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa061b6b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684584fc0 0x55b68459a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa061b6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b684596cf0 con 0x55b684579de0 2024-09-19T08:03:30.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa061b6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b684596620 con 0x55b68457a700 2024-09-19T08:03:30.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa061b6b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b6845973c0 con 0x55b684584fc0 2024-09-19T08:03:30.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa061b6b740 10 client.0 _ll_register_callbacks cb 0x7fa048005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa05d52e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684579de0 0x55b68459f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa05d52e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684579de0 0x55b68459f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45266/0 (socket says 172.21.15.98:45266) 2024-09-19T08:03:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa05d52e640 1 -- 172.21.15.98:0/3733972272 learned_addr learned my addr 172.21.15.98:0/3733972272 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa05cd2d640 1 --2- 172.21.15.98:0/3733972272 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b68457a700 0x55b684599cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa05dd2f640 1 --2- 172.21.15.98:0/3733972272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684584fc0 0x55b68459a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa05d52e640 1 -- 172.21.15.98:0/3733972272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684584fc0 msgr2=0x55b68459a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa05d52e640 1 --2- 172.21.15.98:0/3733972272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684584fc0 0x55b68459a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa05d52e640 1 -- 172.21.15.98:0/3733972272 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b68457a700 msgr2=0x55b684599cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa05d52e640 1 --2- 172.21.15.98:0/3733972272 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b68457a700 0x55b684599cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa05d52e640 1 -- 172.21.15.98:0/3733972272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6845fe100 con 0x55b684579de0 2024-09-19T08:03:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa05d52e640 1 --2- 172.21.15.98:0/3733972272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684579de0 0x55b68459f9a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa04809c530 tx=0x7fa04809a940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c383a535e19b7b00 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa040ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa040ff9640 1 -- 172.21.15.98:0/3733972272 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0480ba060 con 0x55b684579de0 2024-09-19T08:03:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa040ff9640 1 -- 172.21.15.98:0/3733972272 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fa04809a640 con 0x55b684579de0 2024-09-19T08:03:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa061b6b740 1 -- 172.21.15.98:0/3733972272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b6844e3b10 con 0x55b684579de0 2024-09-19T08:03:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa040ff9640 1 -- 172.21.15.98:0/3733972272 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa0480b4050 con 0x55b684579de0 2024-09-19T08:03:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa061b6b740 10 client.15570 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.221+0000 7fa061b6b740 1 -- 172.21.15.98:0/3733972272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b6845ff1f0 con 0x55b684579de0 2024-09-19T08:03:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.222+0000 7fa040ff9640 1 -- 172.21.15.98:0/3733972272 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b6844e3b10 con 0x55b684579de0 2024-09-19T08:03:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.222+0000 7fa040ff9640 1 -- 172.21.15.98:0/3733972272 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa0480982f0 con 0x55b684579de0 2024-09-19T08:03:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.222+0000 7fa061b6b740 10 client.15570 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.222+0000 7fa061b6b740 10 client.15570 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-19T08:03:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.222+0000 7fa061b6b740 20 client.15570 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.222+0000 7fa061b6b740 1 --2- 172.21.15.98:0/3733972272 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55b684603540 0x55b684602250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.222+0000 7fa061b6b740 4 client.15570 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:03:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.222+0000 7fa061b6b740 1 -- 172.21.15.98:0/3733972272 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "status"}) -- 0x55b6844600e0 con 0x55b684603540 2024-09-19T08:03:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.222+0000 7fa05cd2d640 1 --2- 172.21.15.98:0/3733972272 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55b684603540 0x55b684602250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa05cd2d640 1 --2- 172.21.15.98:0/3733972272 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55b684603540 0x55b684602250 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=db855d543352a997 server_cookie=6c53ad8621fd7218 in_seq=0 out_seq=0 2024-09-19T08:03:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa040ff9640 10 client.15570 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:03:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa040ff9640 1 -- 172.21.15.98:0/3733972272 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55b6844600e0 con 0x55b684603540 2024-09-19T08:03:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa040ff9640 10 client.15570 handle_command_reply: tid=0 2024-09-19T08:03:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa061b6b740 1 client.15570 shutdown 2024-09-19T08:03:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa061b6b740 1 -- 172.21.15.98:0/3733972272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684579de0 msgr2=0x55b68459f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa061b6b740 1 --2- 172.21.15.98:0/3733972272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684579de0 0x55b68459f9a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa04809c530 tx=0x7fa04809a940 comp rx=0 tx=0).stop 2024-09-19T08:03:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa061b6b740 1 -- 172.21.15.98:0/3733972272 shutdown_connections 2024-09-19T08:03:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa061b6b740 1 --2- 172.21.15.98:0/3733972272 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55b684603540 0x55b684602250 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa061b6b740 1 --2- 172.21.15.98:0/3733972272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b684584fc0 0x55b68459a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa040ff9640 0 client.15570 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:03:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa061b6b740 1 --2- 172.21.15.98:0/3733972272 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b68457a700 0x55b684599cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa061b6b740 1 --2- 172.21.15.98:0/3733972272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b684579de0 0x55b68459f9a0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa061b6b740 1 -- 172.21.15.98:0/3733972272 >> 172.21.15.98:0/3733972272 conn(0x55b684576370 msgr2=0x55b684580020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa061b6b740 1 -- 172.21.15.98:0/3733972272 shutdown_connections 2024-09-19T08:03:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa061b6b740 1 -- 172.21.15.98:0/3733972272 wait complete. 2024-09-19T08:03:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.223+0000 7fa061b6b740 20 client.15570 trim_cache size 0 max 16384 2024-09-19T08:03:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.224+0000 7fa05f792640 1 -- 172.21.15.98:0/3982037643 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa0440a34d0 msgr2=0x7fa0440a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.224+0000 7fa05f792640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa0440a34d0 0x7fa0440a5990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa050030040 tx=0x7fa0500095c0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.224+0000 7fa05f792640 1 -- 172.21.15.98:0/3982037643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa058102d40 msgr2=0x7fa058198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.224+0000 7fa05f792640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa058102d40 0x7fa058198410 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa04800e9f0 tx=0x7fa04800edf0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.224+0000 7fa05f792640 1 -- 172.21.15.98:0/3982037643 shutdown_connections 2024-09-19T08:03:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.224+0000 7fa05f792640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa0440a34d0 0x7fa0440a5990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.224+0000 7fa05f792640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa05810c170 0x7fa05819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.224+0000 7fa05f792640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa058103660 0x7fa058198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.224+0000 7fa05f792640 1 --2- 172.21.15.98:0/3982037643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa058102d40 0x7fa058198410 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.224+0000 7fa05f792640 1 -- 172.21.15.98:0/3982037643 >> 172.21.15.98:0/3982037643 conn(0x7fa0580fc7a0 msgr2=0x7fa0580fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.224+0000 7fa05f792640 1 -- 172.21.15.98:0/3982037643 shutdown_connections 2024-09-19T08:03:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:30.224+0000 7fa05f792640 1 -- 172.21.15.98:0/3982037643 wait complete. 2024-09-19T08:03:30.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-19T08:03:31.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:30 smithi162 ceph-mon[35260]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-09-19T08:03:31.134 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:31.165 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:30 smithi098 ceph-mon[31333]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-09-19T08:03:31.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:30 smithi139 ceph-mon[34874]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-09-19T08:03:31.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.633+0000 7f54b4c8e640 1 -- 172.21.15.98:0/3223804119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b0101450 msgr2=0x7f54b0105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:31.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.633+0000 7f54b4c8e640 1 --2- 172.21.15.98:0/3223804119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b0101450 0x7f54b0105920 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5498009900 tx=0x7f549802a470 comp rx=0 tx=0).stop 2024-09-19T08:03:31.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.633+0000 7f54b4c8e640 1 -- 172.21.15.98:0/3223804119 shutdown_connections 2024-09-19T08:03:31.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.633+0000 7f54b4c8e640 1 --2- 172.21.15.98:0/3223804119 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54b0105f40 0x7f54b01063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.633+0000 7f54b4c8e640 1 --2- 172.21.15.98:0/3223804119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b0101450 0x7f54b0105920 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.633+0000 7f54b4c8e640 1 --2- 172.21.15.98:0/3223804119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54b0100b30 0x7f54b0100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.633+0000 7f54b4c8e640 1 -- 172.21.15.98:0/3223804119 >> 172.21.15.98:0/3223804119 conn(0x7f54b00fc7a0 msgr2=0x7f54b00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.633+0000 7f54b4c8e640 1 -- 172.21.15.98:0/3223804119 shutdown_connections 2024-09-19T08:03:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.633+0000 7f54b4c8e640 1 -- 172.21.15.98:0/3223804119 wait complete. 2024-09-19T08:03:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.634+0000 7f54b4c8e640 1 Processor -- start 2024-09-19T08:03:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.634+0000 7f54b4c8e640 1 -- start start 2024-09-19T08:03:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.634+0000 7f54b4c8e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b0100b30 0x7f54b01a4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.634+0000 7f54b4c8e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54b0101450 0x7f54b01a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.634+0000 7f54b4c8e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54b0105f40 0x7f54b019e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.634+0000 7f54b4c8e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f54b0118630 con 0x7f54b0100b30 2024-09-19T08:03:31.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.634+0000 7f54b4c8e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f54b01184b0 con 0x7f54b0101450 2024-09-19T08:03:31.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.634+0000 7f54b4c8e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f54b01187b0 con 0x7f54b0105f40 2024-09-19T08:03:31.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.634+0000 7f54ae59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b0100b30 0x7f54b01a4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.634+0000 7f54ae59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b0100b30 0x7f54b01a4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45278/0 (socket says 172.21.15.98:45278) 2024-09-19T08:03:31.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.634+0000 7f54ae59c640 1 -- 172.21.15.98:0/1768163557 learned_addr learned my addr 172.21.15.98:0/1768163557 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:31.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.634+0000 7f54a7fff640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54b0101450 0x7f54b01a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.634+0000 7f54aed9d640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54b0105f40 0x7f54b019e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.635+0000 7f54ae59c640 1 -- 172.21.15.98:0/1768163557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54b0105f40 msgr2=0x7f54b019e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.635+0000 7f54ae59c640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54b0105f40 0x7f54b019e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.635+0000 7f54ae59c640 1 -- 172.21.15.98:0/1768163557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54b0101450 msgr2=0x7f54b01a4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.635+0000 7f54ae59c640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54b0101450 0x7f54b01a4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.635+0000 7f54ae59c640 1 -- 172.21.15.98:0/1768163557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b019f030 con 0x7f54b0100b30 2024-09-19T08:03:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.635+0000 7f54a7fff640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54b0101450 0x7f54b01a4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.635+0000 7f54aed9d640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54b0105f40 0x7f54b019e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.635+0000 7f54ae59c640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b0100b30 0x7f54b01a4740 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f54a000d770 tx=0x7f54a000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.635+0000 7f54a77fe640 1 -- 172.21.15.98:0/1768163557 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a0017070 con 0x7f54b0100b30 2024-09-19T08:03:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.635+0000 7f54b4c8e640 1 -- 172.21.15.98:0/1768163557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b019f320 con 0x7f54b0100b30 2024-09-19T08:03:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.635+0000 7f54a77fe640 1 -- 172.21.15.98:0/1768163557 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54a0005500 con 0x7f54b0100b30 2024-09-19T08:03:31.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.635+0000 7f54a77fe640 1 -- 172.21.15.98:0/1768163557 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a0004e00 con 0x7f54b0100b30 2024-09-19T08:03:31.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.635+0000 7f54b4c8e640 1 -- 172.21.15.98:0/1768163557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b01ab610 con 0x7f54b0100b30 2024-09-19T08:03:31.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.637+0000 7f54a77fe640 1 -- 172.21.15.98:0/1768163557 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f54a0004fa0 con 0x7f54b0100b30 2024-09-19T08:03:31.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.637+0000 7f54b7067740 1 Processor -- start 2024-09-19T08:03:31.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54b7067740 1 -- start start 2024-09-19T08:03:31.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54b7067740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b24f9d0 0x55e88b24fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54b7067740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b2502f0 0x55e88b25a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54b7067740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b25abb0 0x55e88b25cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54b7067740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e88b1b6090 con 0x55e88b25abb0 2024-09-19T08:03:31.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54b7067740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e88b15da70 con 0x55e88b2502f0 2024-09-19T08:03:31.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54b7067740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e88b1b2050 con 0x55e88b24f9d0 2024-09-19T08:03:31.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54a77fe640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f549c0a34d0 0x7f549c0a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54a7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b25abb0 0x55e88b25cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54a7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b25abb0 0x55e88b25cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45284/0 (socket says 172.21.15.98:45284) 2024-09-19T08:03:31.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54a7fff640 1 -- 172.21.15.98:0/3705138660 learned_addr learned my addr 172.21.15.98:0/3705138660 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:31.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54aed9d640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f549c0a34d0 0x7f549c0a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54a77fe640 1 -- 172.21.15.98:0/1768163557 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f54a00cfad0 con 0x7f54b0100b30 2024-09-19T08:03:31.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54aed9d640 1 --2- 172.21.15.98:0/3705138660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b24f9d0 0x55e88b24fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54ae59c640 1 --2- 172.21.15.98:0/3705138660 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b2502f0 0x55e88b25a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54a7fff640 1 -- 172.21.15.98:0/3705138660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b24f9d0 msgr2=0x55e88b24fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:31.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54a7fff640 1 --2- 172.21.15.98:0/3705138660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b24f9d0 0x55e88b24fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54a7fff640 1 -- 172.21.15.98:0/3705138660 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b2502f0 msgr2=0x55e88b25a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54a7fff640 1 --2- 172.21.15.98:0/3705138660 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b2502f0 0x55e88b25a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54a7fff640 1 -- 172.21.15.98:0/3705138660 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e88b16a130 con 0x55e88b25abb0 2024-09-19T08:03:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.638+0000 7f54ae59c640 1 --2- 172.21.15.98:0/3705138660 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b2502f0 0x55e88b25a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.639+0000 7f54a7fff640 1 --2- 172.21.15.98:0/3705138660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b25abb0 0x55e88b25cfa0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f549802a990 tx=0x7f54980040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584050086b8cdd14 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.639+0000 7f54aed9d640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f549c0a34d0 0x7f549c0a5990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f54a800afd0 tx=0x7f54a800ad00 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:31.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.639+0000 7f54a4ff9640 1 -- 172.21.15.98:0/3705138660 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549803f070 con 0x55e88b25abb0 2024-09-19T08:03:31.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.639+0000 7f54a4ff9640 1 -- 172.21.15.98:0/3705138660 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5498005520 con 0x55e88b25abb0 2024-09-19T08:03:31.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.639+0000 7f54a4ff9640 1 -- 172.21.15.98:0/3705138660 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5498035030 con 0x55e88b25abb0 2024-09-19T08:03:31.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 -- 172.21.15.98:0/3705138660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b25abb0 msgr2=0x55e88b25cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:31.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 --2- 172.21.15.98:0/3705138660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b25abb0 0x55e88b25cfa0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f549802a990 tx=0x7f54980040c0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 -- 172.21.15.98:0/3705138660 shutdown_connections 2024-09-19T08:03:31.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 --2- 172.21.15.98:0/3705138660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b25abb0 0x55e88b25cfa0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 --2- 172.21.15.98:0/3705138660 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b2502f0 0x55e88b25a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 --2- 172.21.15.98:0/3705138660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b24f9d0 0x55e88b24fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 -- 172.21.15.98:0/3705138660 >> 172.21.15.98:0/3705138660 conn(0x55e88b24bf60 msgr2=0x55e88b24c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:31.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 -- 172.21.15.98:0/3705138660 shutdown_connections 2024-09-19T08:03:31.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 -- 172.21.15.98:0/3705138660 wait complete. 2024-09-19T08:03:31.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 -- start start 2024-09-19T08:03:31.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 Processor -- start 2024-09-19T08:03:31.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b24f9d0 0x55e88b25a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b2502f0 0x55e88b257760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b25abb0 0x55e88b257ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e88b25d780 con 0x55e88b24f9d0 2024-09-19T08:03:31.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e88b170f20 con 0x55e88b25abb0 2024-09-19T08:03:31.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e88b25d900 con 0x55e88b2502f0 2024-09-19T08:03:31.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54b7067740 10 client.0 _ll_register_callbacks cb 0x55e88b13a030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:31.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54ae59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b24f9d0 0x55e88b25a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.640+0000 7f54ae59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b24f9d0 0x55e88b25a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45300/0 (socket says 172.21.15.98:45300) 2024-09-19T08:03:31.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f54ae59c640 1 -- 172.21.15.98:0/300694000 learned_addr learned my addr 172.21.15.98:0/300694000 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:31.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f54a7fff640 1 --2- 172.21.15.98:0/300694000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b2502f0 0x55e88b257760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f54aed9d640 1 --2- 172.21.15.98:0/300694000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b25abb0 0x55e88b257ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f54ae59c640 1 -- 172.21.15.98:0/300694000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b2502f0 msgr2=0x55e88b257760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:31.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f54ae59c640 1 --2- 172.21.15.98:0/300694000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b2502f0 0x55e88b257760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f54ae59c640 1 -- 172.21.15.98:0/300694000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b25abb0 msgr2=0x55e88b257ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:31.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f54ae59c640 1 --2- 172.21.15.98:0/300694000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b25abb0 0x55e88b257ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f54ae59c640 1 -- 172.21.15.98:0/300694000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e88af272b0 con 0x55e88b24f9d0 2024-09-19T08:03:31.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f54a7fff640 1 --2- 172.21.15.98:0/300694000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b2502f0 0x55e88b257760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:31.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f54ae59c640 1 --2- 172.21.15.98:0/300694000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b24f9d0 0x55e88b25a410 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f54a00cd850 tx=0x7f54a0004780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdd3f366d4e922f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:31.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f5495ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:31.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f5495ffb640 1 -- 172.21.15.98:0/300694000 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a0020040 con 0x55e88b24f9d0 2024-09-19T08:03:31.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f5495ffb640 1 -- 172.21.15.98:0/300694000 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f54a0098200 con 0x55e88b24f9d0 2024-09-19T08:03:31.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f54b7067740 1 -- 172.21.15.98:0/300694000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5498005520 con 0x55e88b24f9d0 2024-09-19T08:03:31.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f5495ffb640 1 -- 172.21.15.98:0/300694000 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f54a009bb90 con 0x55e88b24f9d0 2024-09-19T08:03:31.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f54b7067740 10 client.15582 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:31.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f54b7067740 1 -- 172.21.15.98:0/300694000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e88b268fe0 con 0x55e88b24f9d0 2024-09-19T08:03:31.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f5495ffb640 1 -- 172.21.15.98:0/300694000 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5498005520 con 0x55e88b24f9d0 2024-09-19T08:03:31.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.641+0000 7f5495ffb640 1 -- 172.21.15.98:0/300694000 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f54a00a4370 con 0x55e88b24f9d0 2024-09-19T08:03:31.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.642+0000 7f54b7067740 10 client.15582 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:31.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.642+0000 7f54b7067740 10 client.15582 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-19T08:03:31.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.642+0000 7f54b7067740 20 client.15582 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:31.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.642+0000 7f54b7067740 1 --2- 172.21.15.98:0/300694000 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e88b26b610 0x55e88b26ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.642+0000 7f54b7067740 4 client.15582 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:31.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.642+0000 7f54b7067740 1 -- 172.21.15.98:0/300694000 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e88b1c3600 con 0x55e88b26b610 2024-09-19T08:03:31.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.642+0000 7f54a7fff640 1 --2- 172.21.15.98:0/300694000 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e88b26b610 0x55e88b26ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.642+0000 7f5495ffb640 10 client.15582 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:31.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.642+0000 7f54a7fff640 1 --2- 172.21.15.98:0/300694000 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e88b26b610 0x55e88b26ca80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e7b460c047b1672 server_cookie=c33e48c1cf3f6531 in_seq=0 out_seq=0 2024-09-19T08:03:31.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f5495ffb640 1 -- 172.21.15.98:0/300694000 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e88b1c3600 con 0x55e88b26b610 2024-09-19T08:03:31.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f5495ffb640 10 client.15582 handle_command_reply: tid=0 2024-09-19T08:03:31.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f54b7067740 1 client.15582 shutdown 2024-09-19T08:03:31.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f54b7067740 1 -- 172.21.15.98:0/300694000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b24f9d0 msgr2=0x55e88b25a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:31.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f54b7067740 1 --2- 172.21.15.98:0/300694000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b24f9d0 0x55e88b25a410 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f54a00cd850 tx=0x7f54a0004780 comp rx=0 tx=0).stop 2024-09-19T08:03:31.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f54b7067740 1 -- 172.21.15.98:0/300694000 shutdown_connections 2024-09-19T08:03:31.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f54b7067740 1 --2- 172.21.15.98:0/300694000 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e88b26b610 0x55e88b26ca80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f54b7067740 1 --2- 172.21.15.98:0/300694000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b25abb0 0x55e88b257ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f54b7067740 1 --2- 172.21.15.98:0/300694000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b2502f0 0x55e88b257760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f5495ffb640 0 client.15582 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:31.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f54b7067740 1 --2- 172.21.15.98:0/300694000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b24f9d0 0x55e88b25a410 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f54b7067740 1 -- 172.21.15.98:0/300694000 >> 172.21.15.98:0/300694000 conn(0x55e88b24bf60 msgr2=0x55e88b251c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:31.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f54b7067740 1 -- 172.21.15.98:0/300694000 shutdown_connections 2024-09-19T08:03:31.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f54b7067740 1 -- 172.21.15.98:0/300694000 wait complete. 2024-09-19T08:03:31.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.643+0000 7f54b7067740 20 client.15582 trim_cache size 0 max 16384 2024-09-19T08:03:31.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54b7067740 1 Processor -- start 2024-09-19T08:03:31.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54b7067740 1 -- start start 2024-09-19T08:03:31.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54b7067740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b24f9d0 0x55e88b26ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54b7067740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b2502f0 0x55e88b256ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54b7067740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b25abb0 0x55e88b256030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54b7067740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e88b26adc0 con 0x55e88b2502f0 2024-09-19T08:03:31.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54b7067740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e88b26b490 con 0x55e88b25abb0 2024-09-19T08:03:31.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54b7067740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e88b26bb60 con 0x55e88b24f9d0 2024-09-19T08:03:31.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54a7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b2502f0 0x55e88b256ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54a7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b2502f0 0x55e88b256ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45304/0 (socket says 172.21.15.98:45304) 2024-09-19T08:03:31.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54a7fff640 1 -- 172.21.15.98:0/488925094 learned_addr learned my addr 172.21.15.98:0/488925094 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:31.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54aed9d640 1 --2- 172.21.15.98:0/488925094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b25abb0 0x55e88b256030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54ae59c640 1 --2- 172.21.15.98:0/488925094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b24f9d0 0x55e88b26ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54a7fff640 1 -- 172.21.15.98:0/488925094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b24f9d0 msgr2=0x55e88b26ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:31.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54a7fff640 1 --2- 172.21.15.98:0/488925094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b24f9d0 0x55e88b26ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54a7fff640 1 -- 172.21.15.98:0/488925094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b25abb0 msgr2=0x55e88b256030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:31.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54a7fff640 1 --2- 172.21.15.98:0/488925094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b25abb0 0x55e88b256030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54a7fff640 1 -- 172.21.15.98:0/488925094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e88b25d890 con 0x55e88b2502f0 2024-09-19T08:03:31.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54aed9d640 1 --2- 172.21.15.98:0/488925094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b25abb0 0x55e88b256030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:31.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.661+0000 7f54a7fff640 1 --2- 172.21.15.98:0/488925094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b2502f0 0x55e88b256ca0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f549802a990 tx=0x7f54980314d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e47140ad7cd9371d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:31.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.662+0000 7f5497fff640 1 -- 172.21.15.98:0/488925094 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549803f040 con 0x55e88b2502f0 2024-09-19T08:03:31.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.662+0000 7f5497fff640 1 -- 172.21.15.98:0/488925094 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5498050910 con 0x55e88b2502f0 2024-09-19T08:03:31.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.662+0000 7f5497fff640 1 -- 172.21.15.98:0/488925094 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5498035040 con 0x55e88b2502f0 2024-09-19T08:03:31.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.662+0000 7f54b7067740 1 -- 172.21.15.98:0/488925094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b2502f0 msgr2=0x55e88b256ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:31.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.662+0000 7f54b7067740 1 --2- 172.21.15.98:0/488925094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b2502f0 0x55e88b256ca0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f549802a990 tx=0x7f54980314d0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.662+0000 7f54b7067740 1 -- 172.21.15.98:0/488925094 shutdown_connections 2024-09-19T08:03:31.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.662+0000 7f54b7067740 1 --2- 172.21.15.98:0/488925094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b25abb0 0x55e88b256030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.662+0000 7f54b7067740 1 --2- 172.21.15.98:0/488925094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b2502f0 0x55e88b256ca0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.662+0000 7f54b7067740 1 --2- 172.21.15.98:0/488925094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b24f9d0 0x55e88b26ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.662+0000 7f54b7067740 1 -- 172.21.15.98:0/488925094 >> 172.21.15.98:0/488925094 conn(0x55e88b24bf60 msgr2=0x55e88b24c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:31.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.662+0000 7f54b7067740 1 -- 172.21.15.98:0/488925094 shutdown_connections 2024-09-19T08:03:31.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.662+0000 7f54b7067740 1 -- 172.21.15.98:0/488925094 wait complete. 2024-09-19T08:03:31.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54b7067740 1 -- start start 2024-09-19T08:03:31.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54b7067740 1 Processor -- start 2024-09-19T08:03:31.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54b7067740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b24f9d0 0x55e88b270480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54b7067740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b2502f0 0x55e88b2709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54b7067740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b25abb0 0x55e88b2cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54b7067740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e88b26c900 con 0x55e88b25abb0 2024-09-19T08:03:31.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54b7067740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e88b26c230 con 0x55e88b24f9d0 2024-09-19T08:03:31.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54b7067740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e88b26cfd0 con 0x55e88b2502f0 2024-09-19T08:03:31.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54b7067740 10 client.0 _ll_register_callbacks cb 0x7f54a000d6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:31.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54aed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b25abb0 0x55e88b2cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54aed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b25abb0 0x55e88b2cddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45312/0 (socket says 172.21.15.98:45312) 2024-09-19T08:03:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54aed9d640 1 -- 172.21.15.98:0/3363669529 learned_addr learned my addr 172.21.15.98:0/3363669529 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54ae59c640 1 --2- 172.21.15.98:0/3363669529 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b24f9d0 0x55e88b270480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54a7fff640 1 --2- 172.21.15.98:0/3363669529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b2502f0 0x55e88b2709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54aed9d640 1 -- 172.21.15.98:0/3363669529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b2502f0 msgr2=0x55e88b2709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54aed9d640 1 --2- 172.21.15.98:0/3363669529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b2502f0 0x55e88b2709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54aed9d640 1 -- 172.21.15.98:0/3363669529 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b24f9d0 msgr2=0x55e88b270480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:31.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54aed9d640 1 --2- 172.21.15.98:0/3363669529 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b24f9d0 0x55e88b270480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54aed9d640 1 -- 172.21.15.98:0/3363669529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e88b2ce400 con 0x55e88b25abb0 2024-09-19T08:03:31.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.663+0000 7f54a7fff640 1 --2- 172.21.15.98:0/3363669529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b2502f0 0x55e88b2709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:31.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.664+0000 7f54aed9d640 1 --2- 172.21.15.98:0/3363669529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b25abb0 0x55e88b2cddc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f54a800a710 tx=0x7f54a80012a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8228848c5bb381c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:31.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.664+0000 7f5495ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:31.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.664+0000 7f5495ffb640 1 -- 172.21.15.98:0/3363669529 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a800fad0 con 0x55e88b25abb0 2024-09-19T08:03:31.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.664+0000 7f5495ffb640 1 -- 172.21.15.98:0/3363669529 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f54a800fc70 con 0x55e88b25abb0 2024-09-19T08:03:31.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.664+0000 7f54b7067740 1 -- 172.21.15.98:0/3363669529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5498050910 con 0x55e88b25abb0 2024-09-19T08:03:31.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.664+0000 7f5495ffb640 1 -- 172.21.15.98:0/3363669529 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f54a8029400 con 0x55e88b25abb0 2024-09-19T08:03:31.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.664+0000 7f54b7067740 10 client.15588 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:31.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.664+0000 7f54b7067740 1 -- 172.21.15.98:0/3363669529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e88b2d5700 con 0x55e88b25abb0 2024-09-19T08:03:31.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.664+0000 7f5495ffb640 1 -- 172.21.15.98:0/3363669529 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5498050910 con 0x55e88b25abb0 2024-09-19T08:03:31.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.664+0000 7f5495ffb640 1 -- 172.21.15.98:0/3363669529 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f54a8010040 con 0x55e88b25abb0 2024-09-19T08:03:31.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.664+0000 7f54b7067740 10 client.15588 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:31.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.664+0000 7f54b7067740 10 client.15588 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-19T08:03:31.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.664+0000 7f54b7067740 20 client.15588 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:31.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.664+0000 7f54b7067740 1 --2- 172.21.15.98:0/3363669529 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e88b2da260 0x55e88b2d8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:31.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.665+0000 7f54b7067740 4 client.15588 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T08:03:31.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.665+0000 7f54b7067740 1 -- 172.21.15.98:0/3363669529 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "status"}) -- 0x55e88b1bdb10 con 0x55e88b2da260 2024-09-19T08:03:31.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.665+0000 7f54ae59c640 1 --2- 172.21.15.98:0/3363669529 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e88b2da260 0x55e88b2d8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:31.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.665+0000 7f5495ffb640 10 client.15588 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:31.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.665+0000 7f54ae59c640 1 --2- 172.21.15.98:0/3363669529 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e88b2da260 0x55e88b2d8f70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b00092db2b749c6 server_cookie=221d6f8a6c96822e in_seq=0 out_seq=0 2024-09-19T08:03:31.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.665+0000 7f5495ffb640 1 -- 172.21.15.98:0/3363669529 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55e88b1bdb10 con 0x55e88b2da260 2024-09-19T08:03:31.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.665+0000 7f5495ffb640 10 client.15588 handle_command_reply: tid=0 2024-09-19T08:03:31.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.665+0000 7f54b7067740 1 client.15588 shutdown 2024-09-19T08:03:31.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b7067740 1 -- 172.21.15.98:0/3363669529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b25abb0 msgr2=0x55e88b2cddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:31.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b7067740 1 --2- 172.21.15.98:0/3363669529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b25abb0 0x55e88b2cddc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f54a800a710 tx=0x7f54a80012a0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b7067740 1 -- 172.21.15.98:0/3363669529 shutdown_connections 2024-09-19T08:03:31.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b7067740 1 --2- 172.21.15.98:0/3363669529 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e88b2da260 0x55e88b2d8f70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b7067740 1 --2- 172.21.15.98:0/3363669529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e88b25abb0 0x55e88b2cddc0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b7067740 1 --2- 172.21.15.98:0/3363669529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e88b2502f0 0x55e88b2709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b7067740 1 --2- 172.21.15.98:0/3363669529 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e88b24f9d0 0x55e88b270480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f5495ffb640 0 client.15588 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:31.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b7067740 1 -- 172.21.15.98:0/3363669529 >> 172.21.15.98:0/3363669529 conn(0x55e88b24bf60 msgr2=0x55e88b255f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:31.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b7067740 1 -- 172.21.15.98:0/3363669529 shutdown_connections 2024-09-19T08:03:31.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b7067740 1 -- 172.21.15.98:0/3363669529 wait complete. 2024-09-19T08:03:31.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b7067740 20 client.15588 trim_cache size 0 max 16384 2024-09-19T08:03:31.695 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:03:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "1a02525a-765c-11ef-bceb-c7b262605968", 2024-09-19T08:03:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-09-19T08:03:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24485, 2024-09-19T08:03:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-09-19T08:03:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-09-19T08:03:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-09-19T08:03:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 139.884086475, 2024-09-19T08:03:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-09-19T08:03:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 81, 2024-09-19T08:03:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 80, 2024-09-19T08:03:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 151.27631686999999 2024-09-19T08:03:31.698 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:03:31.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b4c8e640 1 -- 172.21.15.98:0/1768163557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f549c0a34d0 msgr2=0x7f549c0a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:31.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b4c8e640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f549c0a34d0 0x7f549c0a5990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f54a800afd0 tx=0x7f54a800ad00 comp rx=0 tx=0).stop 2024-09-19T08:03:31.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b4c8e640 1 -- 172.21.15.98:0/1768163557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b0100b30 msgr2=0x7f54b01a4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:31.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b4c8e640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b0100b30 0x7f54b01a4740 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f54a000d770 tx=0x7f54a000dc30 comp rx=0 tx=0).stop 2024-09-19T08:03:31.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b4c8e640 1 -- 172.21.15.98:0/1768163557 shutdown_connections 2024-09-19T08:03:31.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b4c8e640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f549c0a34d0 0x7f549c0a5990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b4c8e640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54b0105f40 0x7f54b019e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b4c8e640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54b0101450 0x7f54b01a4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b4c8e640 1 --2- 172.21.15.98:0/1768163557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b0100b30 0x7f54b01a4740 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:31.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b4c8e640 1 -- 172.21.15.98:0/1768163557 >> 172.21.15.98:0/1768163557 conn(0x7f54b00fc7a0 msgr2=0x7f54b01038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:31.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.666+0000 7f54b4c8e640 1 -- 172.21.15.98:0/1768163557 shutdown_connections 2024-09-19T08:03:31.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:31.667+0000 7f54b4c8e640 1 -- 172.21.15.98:0/1768163557 wait complete. 2024-09-19T08:03:32.352 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-09-19T08:03:33.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:32 smithi162 ceph-mon[35260]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 767 B/s wr, 13 op/s 2024-09-19T08:03:33.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:32 smithi139 ceph-mon[34874]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 767 B/s wr, 13 op/s 2024-09-19T08:03:33.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:32 smithi098 ceph-mon[31333]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 767 B/s wr, 13 op/s 2024-09-19T08:03:35.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:34 smithi162 ceph-mon[35260]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-09-19T08:03:35.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:34 smithi139 ceph-mon[34874]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-09-19T08:03:35.184 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:34 smithi098 ceph-mon[31333]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-09-19T08:03:37.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:36 smithi162 ceph-mon[35260]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 767 B/s wr, 12 op/s 2024-09-19T08:03:37.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:36 smithi139 ceph-mon[34874]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 767 B/s wr, 12 op/s 2024-09-19T08:03:37.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:36 smithi098 ceph-mon[31333]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 767 B/s wr, 12 op/s 2024-09-19T08:03:39.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:38 smithi162 ceph-mon[35260]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-19T08:03:39.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:38 smithi139 ceph-mon[34874]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-19T08:03:39.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:38 smithi098 ceph-mon[31333]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-19T08:03:41.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:40 smithi162 ceph-mon[35260]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:03:41.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:40 smithi139 ceph-mon[34874]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:03:41.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:40 smithi098 ceph-mon[31333]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:03:42.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:03:42.780 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:43.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:42 smithi162 ceph-mon[35260]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:03:43.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:42 smithi139 ceph-mon[34874]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:03:43.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:42 smithi098 ceph-mon[31333]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:03:43.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.270+0000 7f04d3349640 1 -- 172.21.15.98:0/692424044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04cc102570 msgr2=0x7f04cc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:43.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.270+0000 7f04d3349640 1 --2- 172.21.15.98:0/692424044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04cc102570 0x7f04cc1029b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009900 tx=0x7f04bc02a480 comp rx=0 tx=0).stop 2024-09-19T08:03:43.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.270+0000 7f04d3349640 1 -- 172.21.15.98:0/692424044 shutdown_connections 2024-09-19T08:03:43.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.270+0000 7f04d3349640 1 --2- 172.21.15.98:0/692424044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04cc102ef0 0x7f04cc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.270+0000 7f04d3349640 1 --2- 172.21.15.98:0/692424044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04cc102570 0x7f04cc1029b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.270+0000 7f04d3349640 1 --2- 172.21.15.98:0/692424044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04cc100e20 0x7f04cc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.270+0000 7f04d3349640 1 -- 172.21.15.98:0/692424044 >> 172.21.15.98:0/692424044 conn(0x7f04cc0fc7d0 msgr2=0x7f04cc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:43.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.270+0000 7f04d3349640 1 -- 172.21.15.98:0/692424044 shutdown_connections 2024-09-19T08:03:43.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.270+0000 7f04d3349640 1 -- 172.21.15.98:0/692424044 wait complete. 2024-09-19T08:03:43.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.270+0000 7f04d3349640 1 Processor -- start 2024-09-19T08:03:43.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d3349640 1 -- start start 2024-09-19T08:03:43.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d3349640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04cc100e20 0x7f04cc0fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d3349640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04cc102570 0x7f04cc0fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d3349640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04cc102ef0 0x7f04cc0fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d3349640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc110f80 con 0x7f04cc102ef0 2024-09-19T08:03:43.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d3349640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc110e00 con 0x7f04cc102570 2024-09-19T08:03:43.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d3349640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc111100 con 0x7f04cc100e20 2024-09-19T08:03:43.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d18e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04cc102ef0 0x7f04cc0fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d18e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04cc102ef0 0x7f04cc0fd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59862/0 (socket says 172.21.15.98:59862) 2024-09-19T08:03:43.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d18e6640 1 -- 172.21.15.98:0/2756796399 learned_addr learned my addr 172.21.15.98:0/2756796399 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:43.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d08e4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04cc102570 0x7f04cc0fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d10e5640 1 --2- 172.21.15.98:0/2756796399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04cc100e20 0x7f04cc0fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d18e6640 1 -- 172.21.15.98:0/2756796399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04cc100e20 msgr2=0x7f04cc0fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:43.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d18e6640 1 --2- 172.21.15.98:0/2756796399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04cc100e20 0x7f04cc0fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d18e6640 1 -- 172.21.15.98:0/2756796399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04cc102570 msgr2=0x7f04cc0fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:43.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d18e6640 1 --2- 172.21.15.98:0/2756796399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04cc102570 0x7f04cc0fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d18e6640 1 -- 172.21.15.98:0/2756796399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04cc1a6a30 con 0x7f04cc102ef0 2024-09-19T08:03:43.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.271+0000 7f04d18e6640 1 --2- 172.21.15.98:0/2756796399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04cc102ef0 0x7f04cc0fd700 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f04c400cad0 tx=0x7f04c400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:43.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.272+0000 7f04c27fc640 1 -- 172.21.15.98:0/2756796399 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c400ae20 con 0x7f04cc102ef0 2024-09-19T08:03:43.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.272+0000 7f04c27fc640 1 -- 172.21.15.98:0/2756796399 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04c4005020 con 0x7f04cc102ef0 2024-09-19T08:03:43.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.272+0000 7f04d3349640 1 -- 172.21.15.98:0/2756796399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04cc1a6d20 con 0x7f04cc102ef0 2024-09-19T08:03:43.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.272+0000 7f04c27fc640 1 -- 172.21.15.98:0/2756796399 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c400f6f0 con 0x7f04cc102ef0 2024-09-19T08:03:43.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.272+0000 7f04d3349640 1 -- 172.21.15.98:0/2756796399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04cc1a7230 con 0x7f04cc102ef0 2024-09-19T08:03:43.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.274+0000 7f04c27fc640 1 -- 172.21.15.98:0/2756796399 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f04c40051d0 con 0x7f04cc102ef0 2024-09-19T08:03:43.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.274+0000 7f04d5722740 1 Processor -- start 2024-09-19T08:03:43.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.274+0000 7f04d5722740 1 -- start start 2024-09-19T08:03:43.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d5722740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6c4a0 0x56204fa6c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d5722740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa6cdc0 0x56204fa77140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d5722740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa77680 0x56204fa79a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d5722740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56204f9d22e0 con 0x56204fa77680 2024-09-19T08:03:43.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d5722740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56204f979ce0 con 0x56204fa6c4a0 2024-09-19T08:03:43.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d5722740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56204f9ce2a0 con 0x56204fa6cdc0 2024-09-19T08:03:43.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d08e4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa77680 0x56204fa79a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d10e5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa6cdc0 0x56204fa77140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d08e4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa77680 0x56204fa79a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59878/0 (socket says 172.21.15.98:59878) 2024-09-19T08:03:43.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d08e4640 1 -- 172.21.15.98:0/1807028005 learned_addr learned my addr 172.21.15.98:0/1807028005 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:43.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d08e4640 1 --2- 172.21.15.98:0/1807028005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6c4a0 0x56204fa6c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04c27fc640 1 --2- 172.21.15.98:0/2756796399 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f04b80a3550 0x7f04b80a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d10e5640 1 -- 172.21.15.98:0/1807028005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6c4a0 msgr2=0x56204fa6c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:43.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d10e5640 1 --2- 172.21.15.98:0/1807028005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6c4a0 0x56204fa6c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d10e5640 1 -- 172.21.15.98:0/1807028005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa77680 msgr2=0x56204fa79a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:43.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d10e5640 1 --2- 172.21.15.98:0/1807028005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa77680 0x56204fa79a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d18e6640 1 --2- 172.21.15.98:0/2756796399 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f04b80a3550 0x7f04b80a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d10e5640 1 -- 172.21.15.98:0/1807028005 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56204f95e440 con 0x56204fa6cdc0 2024-09-19T08:03:43.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04c27fc640 1 -- 172.21.15.98:0/2756796399 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f04c40cfe60 con 0x7f04cc102ef0 2024-09-19T08:03:43.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.275+0000 7f04d10e5640 1 --2- 172.21.15.98:0/1807028005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa6cdc0 0x56204fa77140 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f04c800ab40 tx=0x7f04c800f3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2abccd63f7ae3a0b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:43.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.276+0000 7f04b77fe640 1 -- 172.21.15.98:0/1807028005 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c8004020 con 0x56204fa6cdc0 2024-09-19T08:03:43.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.276+0000 7f04d18e6640 1 --2- 172.21.15.98:0/2756796399 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f04b80a3550 0x7f04b80a5a10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f04c40cd4c0 tx=0x7f04c40bc040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:43.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.276+0000 7f04b77fe640 1 -- 172.21.15.98:0/1807028005 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04c80041c0 con 0x56204fa6cdc0 2024-09-19T08:03:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.276+0000 7f04b77fe640 1 -- 172.21.15.98:0/1807028005 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c8018770 con 0x56204fa6cdc0 2024-09-19T08:03:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.276+0000 7f04d5722740 1 -- 172.21.15.98:0/1807028005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa6cdc0 msgr2=0x56204fa77140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.276+0000 7f04d5722740 1 --2- 172.21.15.98:0/1807028005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa6cdc0 0x56204fa77140 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f04c800ab40 tx=0x7f04c800f3c0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.276+0000 7f04d5722740 1 -- 172.21.15.98:0/1807028005 shutdown_connections 2024-09-19T08:03:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.276+0000 7f04d5722740 1 --2- 172.21.15.98:0/1807028005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa77680 0x56204fa79a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.276+0000 7f04d5722740 1 --2- 172.21.15.98:0/1807028005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa6cdc0 0x56204fa77140 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.276+0000 7f04d5722740 1 --2- 172.21.15.98:0/1807028005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6c4a0 0x56204fa6c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.276+0000 7f04d5722740 1 -- 172.21.15.98:0/1807028005 >> 172.21.15.98:0/1807028005 conn(0x56204fa68a30 msgr2=0x56204fa68e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.277+0000 7f04d5722740 1 -- 172.21.15.98:0/1807028005 shutdown_connections 2024-09-19T08:03:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.277+0000 7f04d5722740 1 -- 172.21.15.98:0/1807028005 wait complete. 2024-09-19T08:03:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.277+0000 7f04d5722740 1 -- start start 2024-09-19T08:03:43.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.277+0000 7f04d5722740 1 Processor -- start 2024-09-19T08:03:43.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.277+0000 7f04d5722740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa6c4a0 0x56204fa76f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.277+0000 7f04d5722740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6cdc0 0x56204fa741e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.277+0000 7f04d5722740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa77680 0x56204fa74720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.277+0000 7f04d5722740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56204fa7a250 con 0x56204fa6cdc0 2024-09-19T08:03:43.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.277+0000 7f04d5722740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56204f98d050 con 0x56204fa77680 2024-09-19T08:03:43.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.277+0000 7f04d5722740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56204fa7a3d0 con 0x56204fa6c4a0 2024-09-19T08:03:43.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.277+0000 7f04d5722740 10 client.0 _ll_register_callbacks cb 0x56204f9cd160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:43.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04d10e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6cdc0 0x56204fa741e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04d10e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6cdc0 0x56204fa741e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59894/0 (socket says 172.21.15.98:59894) 2024-09-19T08:03:43.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04d10e5640 1 -- 172.21.15.98:0/3530969619 learned_addr learned my addr 172.21.15.98:0/3530969619 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:43.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04d08e4640 1 --2- 172.21.15.98:0/3530969619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa6c4a0 0x56204fa76f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04d08e4640 1 --2- 172.21.15.98:0/3530969619 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa77680 0x56204fa74720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04d10e5640 1 -- 172.21.15.98:0/3530969619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa6c4a0 msgr2=0x56204fa76f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04d10e5640 1 --2- 172.21.15.98:0/3530969619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa6c4a0 0x56204fa76f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04d10e5640 1 -- 172.21.15.98:0/3530969619 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa77680 msgr2=0x56204fa74720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04d10e5640 1 --2- 172.21.15.98:0/3530969619 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa77680 0x56204fa74720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04d10e5640 1 -- 172.21.15.98:0/3530969619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56204f7433c0 con 0x56204fa6cdc0 2024-09-19T08:03:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04d10e5640 1 --2- 172.21.15.98:0/3530969619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6cdc0 0x56204fa741e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f04c801a040 tx=0x7f04c800ffb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90220fb797e5d0ee server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04b4ff9640 1 -- 172.21.15.98:0/3530969619 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c8027020 con 0x56204fa6cdc0 2024-09-19T08:03:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04b4ff9640 1 -- 172.21.15.98:0/3530969619 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f04c80035f0 con 0x56204fa6cdc0 2024-09-19T08:03:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04d5722740 1 -- 172.21.15.98:0/3530969619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04c80041c0 con 0x56204fa6cdc0 2024-09-19T08:03:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04b4ff9640 1 -- 172.21.15.98:0/3530969619 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f04c8032050 con 0x56204fa6cdc0 2024-09-19T08:03:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04d5722740 10 client.15600 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.278+0000 7f04d5722740 1 -- 172.21.15.98:0/3530969619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56204fa85ba0 con 0x56204fa6cdc0 2024-09-19T08:03:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.279+0000 7f04b4ff9640 1 -- 172.21.15.98:0/3530969619 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04c80041c0 con 0x56204fa6cdc0 2024-09-19T08:03:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.279+0000 7f04b4ff9640 1 -- 172.21.15.98:0/3530969619 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f04c8004900 con 0x56204fa6cdc0 2024-09-19T08:03:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.279+0000 7f04d5722740 10 client.15600 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.279+0000 7f04d5722740 10 client.15600 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:03:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.279+0000 7f04d5722740 20 client.15600 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.279+0000 7f04d5722740 1 --2- 172.21.15.98:0/3530969619 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56204fa87000 0x56204fa893f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.279+0000 7f04d5722740 4 client.15600 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.279+0000 7f04d5722740 1 -- 172.21.15.98:0/3530969619 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56204f9df850 con 0x56204fa87000 2024-09-19T08:03:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.279+0000 7f04d08e4640 1 --2- 172.21.15.98:0/3530969619 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56204fa87000 0x56204fa893f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.279+0000 7f04b4ff9640 10 client.15600 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.279+0000 7f04d08e4640 1 --2- 172.21.15.98:0/3530969619 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56204fa87000 0x56204fa893f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3d58868008a0fd7 server_cookie=b576e4ffcb8ff954 in_seq=0 out_seq=0 2024-09-19T08:03:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.280+0000 7f04b4ff9640 1 -- 172.21.15.98:0/3530969619 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56204f9df850 con 0x56204fa87000 2024-09-19T08:03:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.280+0000 7f04b4ff9640 10 client.15600 handle_command_reply: tid=0 2024-09-19T08:03:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.280+0000 7f04d5722740 1 client.15600 shutdown 2024-09-19T08:03:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.280+0000 7f04d5722740 1 -- 172.21.15.98:0/3530969619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6cdc0 msgr2=0x56204fa741e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.280+0000 7f04d5722740 1 --2- 172.21.15.98:0/3530969619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6cdc0 0x56204fa741e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f04c801a040 tx=0x7f04c800ffb0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.280+0000 7f04d5722740 1 -- 172.21.15.98:0/3530969619 shutdown_connections 2024-09-19T08:03:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.280+0000 7f04d5722740 1 --2- 172.21.15.98:0/3530969619 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56204fa87000 0x56204fa893f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.280+0000 7f04d5722740 1 --2- 172.21.15.98:0/3530969619 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa77680 0x56204fa74720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.280+0000 7f04d5722740 1 --2- 172.21.15.98:0/3530969619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6cdc0 0x56204fa741e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.280+0000 7f04b4ff9640 0 client.15600 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:43.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.280+0000 7f04d5722740 1 --2- 172.21.15.98:0/3530969619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa6c4a0 0x56204fa76f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.281+0000 7f04d5722740 1 -- 172.21.15.98:0/3530969619 >> 172.21.15.98:0/3530969619 conn(0x56204fa68a30 msgr2=0x56204fa6e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.281+0000 7f04d5722740 1 -- 172.21.15.98:0/3530969619 shutdown_connections 2024-09-19T08:03:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.281+0000 7f04d5722740 1 -- 172.21.15.98:0/3530969619 wait complete. 2024-09-19T08:03:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.281+0000 7f04d5722740 20 client.15600 trim_cache size 0 max 16384 2024-09-19T08:03:43.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.298+0000 7f04d5722740 1 Processor -- start 2024-09-19T08:03:43.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.298+0000 7f04d5722740 1 -- start start 2024-09-19T08:03:43.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.298+0000 7f04d5722740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6c4a0 0x56204fa76510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.298+0000 7f04d5722740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6cdc0 0x56204fa8eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.298+0000 7f04d5722740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa77680 0x56204fa8f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.298+0000 7f04d5722740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc1af9d0 con 0x56204fa6cdc0 2024-09-19T08:03:43.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.298+0000 7f04d5722740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc1b1510 con 0x56204fa6c4a0 2024-09-19T08:03:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.298+0000 7f04d5722740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc1b00a0 con 0x56204fa77680 2024-09-19T08:03:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.298+0000 7f04d10e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6cdc0 0x56204fa8eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.298+0000 7f04d10e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6cdc0 0x56204fa8eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59898/0 (socket says 172.21.15.98:59898) 2024-09-19T08:03:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.298+0000 7f04d10e5640 1 -- 172.21.15.98:0/245152598 learned_addr learned my addr 172.21.15.98:0/245152598 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.298+0000 7f04d08e4640 1 --2- 172.21.15.98:0/245152598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa77680 0x56204fa8f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.299+0000 7f04d08e4640 1 --2- 172.21.15.98:0/245152598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6c4a0 0x56204fa76510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.299+0000 7f04d10e5640 1 -- 172.21.15.98:0/245152598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa77680 msgr2=0x56204fa8f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.299+0000 7f04d10e5640 1 --2- 172.21.15.98:0/245152598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa77680 0x56204fa8f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.299+0000 7f04d10e5640 1 -- 172.21.15.98:0/245152598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6c4a0 msgr2=0x56204fa76510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.299+0000 7f04d10e5640 1 --2- 172.21.15.98:0/245152598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6c4a0 0x56204fa76510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.299+0000 7f04d10e5640 1 -- 172.21.15.98:0/245152598 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56204fa684d0 con 0x56204fa6cdc0 2024-09-19T08:03:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.299+0000 7f04d08e4640 1 --2- 172.21.15.98:0/245152598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa77680 0x56204fa8f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:43.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.299+0000 7f04d10e5640 1 --2- 172.21.15.98:0/245152598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6cdc0 0x56204fa8eae0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f04c8004da0 tx=0x7f04c8005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2b37cba00e772 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.299+0000 7f04b6ffd640 1 -- 172.21.15.98:0/245152598 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c8027040 con 0x56204fa6cdc0 2024-09-19T08:03:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.299+0000 7f04b6ffd640 1 -- 172.21.15.98:0/245152598 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04c800a9c0 con 0x56204fa6cdc0 2024-09-19T08:03:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.299+0000 7f04b6ffd640 1 -- 172.21.15.98:0/245152598 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c8005280 con 0x56204fa6cdc0 2024-09-19T08:03:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.300+0000 7f04d5722740 1 -- 172.21.15.98:0/245152598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6cdc0 msgr2=0x56204fa8eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.300+0000 7f04d5722740 1 --2- 172.21.15.98:0/245152598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6cdc0 0x56204fa8eae0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f04c8004da0 tx=0x7f04c8005e20 comp rx=0 tx=0).stop 2024-09-19T08:03:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.300+0000 7f04d5722740 1 -- 172.21.15.98:0/245152598 shutdown_connections 2024-09-19T08:03:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.300+0000 7f04d5722740 1 --2- 172.21.15.98:0/245152598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa77680 0x56204fa8f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.300+0000 7f04d5722740 1 --2- 172.21.15.98:0/245152598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6cdc0 0x56204fa8eae0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.300+0000 7f04d5722740 1 --2- 172.21.15.98:0/245152598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6c4a0 0x56204fa76510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.300+0000 7f04d5722740 1 -- 172.21.15.98:0/245152598 >> 172.21.15.98:0/245152598 conn(0x56204fa68a30 msgr2=0x56204fa67b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.300+0000 7f04d5722740 1 -- 172.21.15.98:0/245152598 shutdown_connections 2024-09-19T08:03:43.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.300+0000 7f04d5722740 1 -- 172.21.15.98:0/245152598 wait complete. 2024-09-19T08:03:43.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.300+0000 7f04d5722740 1 -- start start 2024-09-19T08:03:43.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.300+0000 7f04d5722740 1 Processor -- start 2024-09-19T08:03:43.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d5722740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6c4a0 0x56204fa7f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d5722740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6cdc0 0x56204fa7fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d5722740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa77680 0x56204fa7cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d5722740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc1b0e40 con 0x56204fa6c4a0 2024-09-19T08:03:43.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d5722740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc1b0770 con 0x56204fa6cdc0 2024-09-19T08:03:43.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d5722740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc1ad9b0 con 0x56204fa77680 2024-09-19T08:03:43.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d5722740 10 client.0 _ll_register_callbacks cb 0x7f04bc004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:43.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d08e4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6c4a0 0x56204fa7f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d08e4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6c4a0 0x56204fa7f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59912/0 (socket says 172.21.15.98:59912) 2024-09-19T08:03:43.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d10e5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6cdc0 0x56204fa7fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d08e4640 1 -- 172.21.15.98:0/617463428 learned_addr learned my addr 172.21.15.98:0/617463428 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:43.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d08e4640 1 --2- 172.21.15.98:0/617463428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa77680 0x56204fa7cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d08e4640 1 -- 172.21.15.98:0/617463428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa77680 msgr2=0x56204fa7cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:43.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d08e4640 1 --2- 172.21.15.98:0/617463428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa77680 0x56204fa7cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d08e4640 1 -- 172.21.15.98:0/617463428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6cdc0 msgr2=0x56204fa7fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:43.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d08e4640 1 --2- 172.21.15.98:0/617463428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6cdc0 0x56204fa7fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d08e4640 1 -- 172.21.15.98:0/617463428 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56204fa7d7d0 con 0x56204fa6c4a0 2024-09-19T08:03:43.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d10e5640 1 --2- 172.21.15.98:0/617463428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6cdc0 0x56204fa7fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:43.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04d08e4640 1 --2- 172.21.15.98:0/617463428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6c4a0 0x56204fa7f9b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f04bc02a9a0 tx=0x7f04bc005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f8b55888b47423 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:43.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.301+0000 7f04b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:43.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.302+0000 7f04b4ff9640 1 -- 172.21.15.98:0/617463428 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc050070 con 0x56204fa6c4a0 2024-09-19T08:03:43.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.302+0000 7f04b4ff9640 1 -- 172.21.15.98:0/617463428 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f04bc031540 con 0x56204fa6c4a0 2024-09-19T08:03:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.302+0000 7f04d5722740 1 -- 172.21.15.98:0/617463428 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04c800a9c0 con 0x56204fa6c4a0 2024-09-19T08:03:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.302+0000 7f04b4ff9640 1 -- 172.21.15.98:0/617463428 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f04bc035040 con 0x56204fa6c4a0 2024-09-19T08:03:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.302+0000 7f04d5722740 10 client.15606 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.302+0000 7f04d5722740 1 -- 172.21.15.98:0/617463428 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56204faf16c0 con 0x56204fa6c4a0 2024-09-19T08:03:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.302+0000 7f04b4ff9640 1 -- 172.21.15.98:0/617463428 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04c800a9c0 con 0x56204fa6c4a0 2024-09-19T08:03:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.302+0000 7f04b4ff9640 1 -- 172.21.15.98:0/617463428 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f04bc055030 con 0x56204fa6c4a0 2024-09-19T08:03:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.302+0000 7f04d5722740 10 client.15606 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.302+0000 7f04d5722740 10 client.15606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:03:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.302+0000 7f04d5722740 20 client.15606 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.302+0000 7f04d5722740 1 --2- 172.21.15.98:0/617463428 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56204fa87000 0x56204faf3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.302+0000 7f04d5722740 4 client.15606 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:03:43.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.302+0000 7f04d5722740 1 -- 172.21.15.98:0/617463428 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56204f9d9d60 con 0x56204fa87000 2024-09-19T08:03:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04d10e5640 1 --2- 172.21.15.98:0/617463428 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56204fa87000 0x56204faf3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04b4ff9640 10 client.15606 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04d10e5640 1 --2- 172.21.15.98:0/617463428 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56204fa87000 0x56204faf3e00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=756680e66398f6cc server_cookie=72a0612bb41716e5 in_seq=0 out_seq=0 2024-09-19T08:03:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04b4ff9640 1 -- 172.21.15.98:0/617463428 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56204f9d9d60 con 0x56204fa87000 2024-09-19T08:03:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04b4ff9640 10 client.15606 handle_command_reply: tid=0 2024-09-19T08:03:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04d5722740 1 client.15606 shutdown 2024-09-19T08:03:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04d5722740 1 -- 172.21.15.98:0/617463428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6c4a0 msgr2=0x56204fa7f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04d5722740 1 --2- 172.21.15.98:0/617463428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6c4a0 0x56204fa7f9b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f04bc02a9a0 tx=0x7f04bc005b50 comp rx=0 tx=0).stop 2024-09-19T08:03:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04d5722740 1 -- 172.21.15.98:0/617463428 shutdown_connections 2024-09-19T08:03:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04d5722740 1 --2- 172.21.15.98:0/617463428 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56204fa87000 0x56204faf3e00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04d5722740 1 --2- 172.21.15.98:0/617463428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56204fa77680 0x56204fa7cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04d5722740 1 --2- 172.21.15.98:0/617463428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56204fa6cdc0 0x56204fa7fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04d5722740 1 --2- 172.21.15.98:0/617463428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56204fa6c4a0 0x56204fa7f9b0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04d5722740 1 -- 172.21.15.98:0/617463428 >> 172.21.15.98:0/617463428 conn(0x56204fa68a30 msgr2=0x56204fa67350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:43.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.303+0000 7f04b4ff9640 0 client.15606 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:43.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.304+0000 7f04d5722740 1 -- 172.21.15.98:0/617463428 shutdown_connections 2024-09-19T08:03:43.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.304+0000 7f04d5722740 1 -- 172.21.15.98:0/617463428 wait complete. 2024-09-19T08:03:43.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.304+0000 7f04d5722740 20 client.15606 trim_cache size 0 max 16384 2024-09-19T08:03:43.321 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:03:43.321 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:03:43.321 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:03:43.321 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:03:43.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.304+0000 7f04d3349640 1 -- 172.21.15.98:0/2756796399 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f04b80a3550 msgr2=0x7f04b80a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.304+0000 7f04d3349640 1 --2- 172.21.15.98:0/2756796399 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f04b80a3550 0x7f04b80a5a10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f04c40cd4c0 tx=0x7f04c40bc040 comp rx=0 tx=0).stop 2024-09-19T08:03:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.304+0000 7f04d3349640 1 -- 172.21.15.98:0/2756796399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04cc102ef0 msgr2=0x7f04cc0fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.304+0000 7f04d3349640 1 --2- 172.21.15.98:0/2756796399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04cc102ef0 0x7f04cc0fd700 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f04c400cad0 tx=0x7f04c400cf90 comp rx=0 tx=0).stop 2024-09-19T08:03:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.304+0000 7f04d3349640 1 -- 172.21.15.98:0/2756796399 shutdown_connections 2024-09-19T08:03:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.304+0000 7f04d3349640 1 --2- 172.21.15.98:0/2756796399 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f04b80a3550 0x7f04b80a5a10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.304+0000 7f04d3349640 1 --2- 172.21.15.98:0/2756796399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04cc102ef0 0x7f04cc0fd700 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.304+0000 7f04d3349640 1 --2- 172.21.15.98:0/2756796399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04cc102570 0x7f04cc0fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.304+0000 7f04d3349640 1 --2- 172.21.15.98:0/2756796399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04cc100e20 0x7f04cc0fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:43.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.304+0000 7f04d3349640 1 -- 172.21.15.98:0/2756796399 >> 172.21.15.98:0/2756796399 conn(0x7f04cc0fc7d0 msgr2=0x7f04cc06a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:43.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.304+0000 7f04d3349640 1 -- 172.21.15.98:0/2756796399 shutdown_connections 2024-09-19T08:03:43.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:43.305+0000 7f04d3349640 1 -- 172.21.15.98:0/2756796399 wait complete. 2024-09-19T08:03:44.020 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:03:44.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:03:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:44.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.767+0000 7f412534d640 1 -- 172.21.15.98:0/2204179139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4120102ef0 msgr2=0x7f412010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:44.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.767+0000 7f412534d640 1 --2- 172.21.15.98:0/2204179139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4120102ef0 0x7f412010da40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4118009ca0 tx=0x7f411802a950 comp rx=0 tx=0).stop 2024-09-19T08:03:44.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 -- 172.21.15.98:0/2204179139 shutdown_connections 2024-09-19T08:03:44.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 --2- 172.21.15.98:0/2204179139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4120102ef0 0x7f412010da40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:44.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 --2- 172.21.15.98:0/2204179139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4120102570 0x7f41201029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:44.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 --2- 172.21.15.98:0/2204179139 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4120100e20 0x7f4120101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:44.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 -- 172.21.15.98:0/2204179139 >> 172.21.15.98:0/2204179139 conn(0x7f41200fc7d0 msgr2=0x7f41200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:44.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 -- 172.21.15.98:0/2204179139 shutdown_connections 2024-09-19T08:03:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 -- 172.21.15.98:0/2204179139 wait complete. 2024-09-19T08:03:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 Processor -- start 2024-09-19T08:03:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 -- start start 2024-09-19T08:03:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4120100e20 0x7f4120198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4120102570 0x7f4120198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4120102ef0 0x7f412019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4120110f80 con 0x7f4120102ef0 2024-09-19T08:03:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4120110e00 con 0x7f4120100e20 2024-09-19T08:03:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.768+0000 7f412534d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4120111100 con 0x7f4120102570 2024-09-19T08:03:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f411f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4120102ef0 0x7f412019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f411f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4120102ef0 0x7f412019cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59938/0 (socket says 172.21.15.98:59938) 2024-09-19T08:03:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f411f7fe640 1 -- 172.21.15.98:0/403345578 learned_addr learned my addr 172.21.15.98:0/403345578 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f4117fff640 1 --2- 172.21.15.98:0/403345578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4120102570 0x7f4120198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f411effd640 1 --2- 172.21.15.98:0/403345578 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4120100e20 0x7f4120198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f411f7fe640 1 -- 172.21.15.98:0/403345578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4120102570 msgr2=0x7f4120198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f411f7fe640 1 --2- 172.21.15.98:0/403345578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4120102570 0x7f4120198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f411f7fe640 1 -- 172.21.15.98:0/403345578 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4120100e20 msgr2=0x7f4120198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f411f7fe640 1 --2- 172.21.15.98:0/403345578 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4120100e20 0x7f4120198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f411f7fe640 1 -- 172.21.15.98:0/403345578 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412019d4b0 con 0x7f4120102ef0 2024-09-19T08:03:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f411f7fe640 1 --2- 172.21.15.98:0/403345578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4120102ef0 0x7f412019cdd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f411802ae70 tx=0x7f411802f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f411cff9640 1 -- 172.21.15.98:0/403345578 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4118034040 con 0x7f4120102ef0 2024-09-19T08:03:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f412534d640 1 -- 172.21.15.98:0/403345578 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f412019d740 con 0x7f4120102ef0 2024-09-19T08:03:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f411cff9640 1 -- 172.21.15.98:0/403345578 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4118030920 con 0x7f4120102ef0 2024-09-19T08:03:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f411cff9640 1 -- 172.21.15.98:0/403345578 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4118045480 con 0x7f4120102ef0 2024-09-19T08:03:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.769+0000 7f412534d640 1 -- 172.21.15.98:0/403345578 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41201a5080 con 0x7f4120102ef0 2024-09-19T08:03:44.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.771+0000 7f412534d640 1 -- 172.21.15.98:0/403345578 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4120101200 con 0x7f4120102ef0 2024-09-19T08:03:44.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.771+0000 7f411cff9640 1 -- 172.21.15.98:0/403345578 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f4118037020 con 0x7f4120102ef0 2024-09-19T08:03:44.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.772+0000 7f411cff9640 1 --2- 172.21.15.98:0/403345578 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f410c0a34d0 0x7f410c0a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:44.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.772+0000 7f411effd640 1 --2- 172.21.15.98:0/403345578 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f410c0a34d0 0x7f410c0a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:44.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.772+0000 7f411cff9640 1 -- 172.21.15.98:0/403345578 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f41180f0ca0 con 0x7f4120102ef0 2024-09-19T08:03:44.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.773+0000 7f411effd640 1 --2- 172.21.15.98:0/403345578 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f410c0a34d0 0x7f410c0a5990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f411000bbe0 tx=0x7f411000b390 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:44.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:44.777+0000 7f411cff9640 1 -- 172.21.15.98:0/403345578 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f41180bd570 con 0x7f4120102ef0 2024-09-19T08:03:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:45.010+0000 7f412534d640 1 -- 172.21.15.98:0/403345578 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f412010a0a0 con 0x7f4120102ef0 2024-09-19T08:03:45.010 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:44 smithi098 ceph-mon[31333]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:03:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:45.011+0000 7f411cff9640 1 -- 172.21.15.98:0/403345578 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11983 (secure 0 0 0) 0x7f41180bce70 con 0x7f4120102ef0 2024-09-19T08:03:45.011 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:45.011 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"btime":"2024-09-19T08:02:59:855935+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:02:59.805764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:03:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-09-19T08:03:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:45.013+0000 7f412534d640 1 -- 172.21.15.98:0/403345578 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f410c0a34d0 msgr2=0x7f410c0a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:45.013+0000 7f412534d640 1 --2- 172.21.15.98:0/403345578 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f410c0a34d0 0x7f410c0a5990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f411000bbe0 tx=0x7f411000b390 comp rx=0 tx=0).stop 2024-09-19T08:03:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:45.013+0000 7f412534d640 1 -- 172.21.15.98:0/403345578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4120102ef0 msgr2=0x7f412019cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:45.013+0000 7f412534d640 1 --2- 172.21.15.98:0/403345578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4120102ef0 0x7f412019cdd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f411802ae70 tx=0x7f411802f5d0 comp rx=0 tx=0).stop 2024-09-19T08:03:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:45.013+0000 7f412534d640 1 -- 172.21.15.98:0/403345578 shutdown_connections 2024-09-19T08:03:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:45.013+0000 7f412534d640 1 --2- 172.21.15.98:0/403345578 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f410c0a34d0 0x7f410c0a5990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:45.013+0000 7f412534d640 1 --2- 172.21.15.98:0/403345578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4120102ef0 0x7f412019cdd0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:45.013+0000 7f412534d640 1 --2- 172.21.15.98:0/403345578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4120102570 0x7f4120198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:45.013+0000 7f412534d640 1 --2- 172.21.15.98:0/403345578 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4120100e20 0x7f4120198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:45.013+0000 7f412534d640 1 -- 172.21.15.98:0/403345578 >> 172.21.15.98:0/403345578 conn(0x7f41200fc7d0 msgr2=0x7f412006a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:45.013+0000 7f412534d640 1 -- 172.21.15.98:0/403345578 shutdown_connections 2024-09-19T08:03:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:45.013+0000 7f412534d640 1 -- 172.21.15.98:0/403345578 wait complete. 2024-09-19T08:03:45.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:44 smithi162 ceph-mon[35260]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:03:45.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:44 smithi139 ceph-mon[34874]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:03:45.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:03:46.007 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:46.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:45 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/403345578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:46.127 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:45 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/403345578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:46.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:45 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/403345578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.546+0000 7fc5d2f50640 1 -- 172.21.15.98:0/3225961694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc106a60 msgr2=0x7fc5cc106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.546+0000 7fc5d2f50640 1 --2- 172.21.15.98:0/3225961694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc106a60 0x7fc5cc106e40 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009930 tx=0x7fc5c002a420 comp rx=0 tx=0).stop 2024-09-19T08:03:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.546+0000 7fc5d2f50640 1 -- 172.21.15.98:0/3225961694 shutdown_connections 2024-09-19T08:03:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.546+0000 7fc5d2f50640 1 --2- 172.21.15.98:0/3225961694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5cc101400 0x7fc5cc10d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.546+0000 7fc5d2f50640 1 --2- 172.21.15.98:0/3225961694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5cc100a60 0x7fc5cc100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.546+0000 7fc5d2f50640 1 --2- 172.21.15.98:0/3225961694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc106a60 0x7fc5cc106e40 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.546+0000 7fc5d2f50640 1 -- 172.21.15.98:0/3225961694 >> 172.21.15.98:0/3225961694 conn(0x7fc5cc0fc780 msgr2=0x7fc5cc0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.546+0000 7fc5d2f50640 1 -- 172.21.15.98:0/3225961694 shutdown_connections 2024-09-19T08:03:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.546+0000 7fc5d2f50640 1 -- 172.21.15.98:0/3225961694 wait complete. 2024-09-19T08:03:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.546+0000 7fc5d2f50640 1 Processor -- start 2024-09-19T08:03:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d2f50640 1 -- start start 2024-09-19T08:03:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d2f50640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5cc100a60 0x7fc5cc199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d2f50640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5cc101400 0x7fc5cc199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d2f50640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc106a60 0x7fc5cc1a43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d2f50640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc110e80 con 0x7fc5cc106a60 2024-09-19T08:03:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d2f50640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc110d00 con 0x7fc5cc100a60 2024-09-19T08:03:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d2f50640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc111000 con 0x7fc5cc101400 2024-09-19T08:03:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d14ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc106a60 0x7fc5cc1a43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d14ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc106a60 0x7fc5cc1a43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41250/0 (socket says 172.21.15.98:41250) 2024-09-19T08:03:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d14ed640 1 -- 172.21.15.98:0/1872705913 learned_addr learned my addr 172.21.15.98:0/1872705913 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d0cec640 1 --2- 172.21.15.98:0/1872705913 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5cc100a60 0x7fc5cc199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/1872705913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5cc101400 0x7fc5cc199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d14ed640 1 -- 172.21.15.98:0/1872705913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5cc101400 msgr2=0x7fc5cc199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d14ed640 1 --2- 172.21.15.98:0/1872705913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5cc101400 0x7fc5cc199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d14ed640 1 -- 172.21.15.98:0/1872705913 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5cc100a60 msgr2=0x7fc5cc199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d14ed640 1 --2- 172.21.15.98:0/1872705913 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5cc100a60 0x7fc5cc199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d14ed640 1 -- 172.21.15.98:0/1872705913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5cc1a4900 con 0x7fc5cc106a60 2024-09-19T08:03:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.547+0000 7fc5d14ed640 1 --2- 172.21.15.98:0/1872705913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc106a60 0x7fc5cc1a43c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400b970 tx=0x7fc5c400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.548+0000 7fc5c9ffb640 1 -- 172.21.15.98:0/1872705913 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4019070 con 0x7fc5cc106a60 2024-09-19T08:03:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.548+0000 7fc5c9ffb640 1 -- 172.21.15.98:0/1872705913 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5c400cb40 con 0x7fc5cc106a60 2024-09-19T08:03:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.548+0000 7fc5c9ffb640 1 -- 172.21.15.98:0/1872705913 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4007650 con 0x7fc5cc106a60 2024-09-19T08:03:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.548+0000 7fc5d2f50640 1 -- 172.21.15.98:0/1872705913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5cc1a4ad0 con 0x7fc5cc106a60 2024-09-19T08:03:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.548+0000 7fc5d2f50640 1 -- 172.21.15.98:0/1872705913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5cc1a4f90 con 0x7fc5cc106a60 2024-09-19T08:03:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.551+0000 7fc5d5329740 1 Processor -- start 2024-09-19T08:03:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.551+0000 7fc5c9ffb640 1 -- 172.21.15.98:0/1872705913 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7fc5c401c070 con 0x7fc5cc106a60 2024-09-19T08:03:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.551+0000 7fc5d5329740 1 -- start start 2024-09-19T08:03:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.551+0000 7fc5d5329740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd2b40 0x5590e3dd1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.551+0000 7fc5d5329740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd15d0 0x5590e3dd1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.551+0000 7fc5d5329740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.551+0000 7fc5d5329740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5590e3d372f0 con 0x5590e3de0780 2024-09-19T08:03:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.551+0000 7fc5d5329740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5590e3cdecf0 con 0x5590e3dd15d0 2024-09-19T08:03:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.551+0000 7fc5d5329740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5590e3d332b0 con 0x5590e3dd2b40 2024-09-19T08:03:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.551+0000 7fc5cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.551+0000 7fc5cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41266/0 (socket says 172.21.15.98:41266) 2024-09-19T08:03:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.551+0000 7fc5cbfff640 1 -- 172.21.15.98:0/1113921303 learned_addr learned my addr 172.21.15.98:0/1113921303 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.552+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/1113921303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd2b40 0x5590e3dd1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.552+0000 7fc5d0cec640 1 --2- 172.21.15.98:0/1113921303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd15d0 0x5590e3dd1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.552+0000 7fc5cbfff640 1 -- 172.21.15.98:0/1113921303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd2b40 msgr2=0x5590e3dd1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.552+0000 7fc5c9ffb640 1 --2- 172.21.15.98:0/1872705913 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc5bc0a3550 0x7fc5bc0a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.552+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/1113921303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd2b40 0x5590e3dd1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.552+0000 7fc5cbfff640 1 -- 172.21.15.98:0/1113921303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd15d0 msgr2=0x5590e3dd1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:46.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.552+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/1113921303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd15d0 0x5590e3dd1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.552+0000 7fc5cbfff640 1 -- 172.21.15.98:0/1113921303 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590e3cc33a0 con 0x5590e3de0780 2024-09-19T08:03:46.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.552+0000 7fc5d14ed640 1 --2- 172.21.15.98:0/1872705913 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc5bc0a3550 0x7fc5bc0a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.552+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/1113921303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de2b70 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8012540 tx=0x7fc5b800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e899440daa3d76b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:46.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.552+0000 7fc5c9ffb640 1 -- 172.21.15.98:0/1872705913 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fc5c40cf3e0 con 0x7fc5cc106a60 2024-09-19T08:03:46.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.552+0000 7fc5b6ffd640 1 -- 172.21.15.98:0/1113921303 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b80106b0 con 0x5590e3de0780 2024-09-19T08:03:46.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.552+0000 7fc5b6ffd640 1 -- 172.21.15.98:0/1113921303 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b8010d30 con 0x5590e3de0780 2024-09-19T08:03:46.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.552+0000 7fc5b6ffd640 1 -- 172.21.15.98:0/1113921303 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b801dd80 con 0x5590e3de0780 2024-09-19T08:03:46.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.553+0000 7fc5d5329740 1 -- 172.21.15.98:0/1113921303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 msgr2=0x5590e3de2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:46.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.553+0000 7fc5d14ed640 1 --2- 172.21.15.98:0/1872705913 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc5bc0a3550 0x7fc5bc0a5a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40ccb50 tx=0x7fc5c4009970 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:46.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.553+0000 7fc5d5329740 1 --2- 172.21.15.98:0/1113921303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de2b70 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8012540 tx=0x7fc5b800b870 comp rx=0 tx=0).stop 2024-09-19T08:03:46.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.553+0000 7fc5d5329740 1 -- 172.21.15.98:0/1113921303 shutdown_connections 2024-09-19T08:03:46.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.553+0000 7fc5d5329740 1 --2- 172.21.15.98:0/1113921303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de2b70 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.553+0000 7fc5d5329740 1 --2- 172.21.15.98:0/1113921303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd15d0 0x5590e3dd1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.553+0000 7fc5d5329740 1 --2- 172.21.15.98:0/1113921303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd2b40 0x5590e3dd1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.553+0000 7fc5d5329740 1 -- 172.21.15.98:0/1113921303 >> 172.21.15.98:0/1113921303 conn(0x5590e3d9a310 msgr2=0x5590e3dcdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:46.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.553+0000 7fc5d5329740 1 -- 172.21.15.98:0/1113921303 shutdown_connections 2024-09-19T08:03:46.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.553+0000 7fc5d5329740 1 -- 172.21.15.98:0/1113921303 wait complete. 2024-09-19T08:03:46.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.554+0000 7fc5d5329740 1 -- start start 2024-09-19T08:03:46.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.554+0000 7fc5d5329740 1 Processor -- start 2024-09-19T08:03:46.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.554+0000 7fc5d5329740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd15d0 0x5590e3de0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.554+0000 7fc5d5329740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd2b40 0x5590e3ddd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.554+0000 7fc5d5329740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3ddd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.554+0000 7fc5d5329740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5590e3de3480 con 0x5590e3de0780 2024-09-19T08:03:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.554+0000 7fc5d5329740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5590e3cf2060 con 0x5590e3dd15d0 2024-09-19T08:03:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.554+0000 7fc5d5329740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5590e3de3600 con 0x5590e3dd2b40 2024-09-19T08:03:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.554+0000 7fc5d5329740 10 client.0 _ll_register_callbacks cb 0x5590e3a458c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.554+0000 7fc5cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3ddd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.554+0000 7fc5cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3ddd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41276/0 (socket says 172.21.15.98:41276) 2024-09-19T08:03:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.554+0000 7fc5cbfff640 1 -- 172.21.15.98:0/2642171646 learned_addr learned my addr 172.21.15.98:0/2642171646 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.554+0000 7fc5d0cec640 1 --2- 172.21.15.98:0/2642171646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd2b40 0x5590e3ddd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.554+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/2642171646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd15d0 0x5590e3de0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5cbfff640 1 -- 172.21.15.98:0/2642171646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd2b40 msgr2=0x5590e3ddd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/2642171646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd2b40 0x5590e3ddd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5cbfff640 1 -- 172.21.15.98:0/2642171646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd15d0 msgr2=0x5590e3de0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/2642171646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd15d0 0x5590e3de0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5cbfff640 1 -- 172.21.15.98:0/2642171646 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590e3aa83c0 con 0x5590e3de0780 2024-09-19T08:03:46.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/2642171646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3ddd9e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8001eb0 tx=0x7fc5b801d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3786538b49954514 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:46.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5abfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:46.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5abfff640 1 -- 172.21.15.98:0/2642171646 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8005050 con 0x5590e3de0780 2024-09-19T08:03:46.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5abfff640 1 -- 172.21.15.98:0/2642171646 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fc5b80051f0 con 0x5590e3de0780 2024-09-19T08:03:46.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5d5329740 1 -- 172.21.15.98:0/2642171646 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5b8010d30 con 0x5590e3de0780 2024-09-19T08:03:46.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5abfff640 1 -- 172.21.15.98:0/2642171646 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc5b80053f0 con 0x5590e3de0780 2024-09-19T08:03:46.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5d5329740 10 client.15624 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:46.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5d5329740 1 -- 172.21.15.98:0/2642171646 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5590e3deef00 con 0x5590e3de0780 2024-09-19T08:03:46.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5abfff640 1 -- 172.21.15.98:0/2642171646 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b8010d30 con 0x5590e3de0780 2024-09-19T08:03:46.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.555+0000 7fc5abfff640 1 -- 172.21.15.98:0/2642171646 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc5b801c070 con 0x5590e3de0780 2024-09-19T08:03:46.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.556+0000 7fc5d5329740 10 client.15624 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:46.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.556+0000 7fc5d5329740 10 client.15624 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:03:46.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.556+0000 7fc5d5329740 20 client.15624 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:46.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.556+0000 7fc5d5329740 1 --2- 172.21.15.98:0/2642171646 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5590e3df0300 0x5590e3df26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.556+0000 7fc5d5329740 4 client.15624 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:46.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.556+0000 7fc5d5329740 1 -- 172.21.15.98:0/2642171646 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590e3d44860 con 0x5590e3df0300 2024-09-19T08:03:46.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.556+0000 7fc5d0cec640 1 --2- 172.21.15.98:0/2642171646 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5590e3df0300 0x5590e3df26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.556+0000 7fc5d0cec640 1 --2- 172.21.15.98:0/2642171646 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5590e3df0300 0x5590e3df26f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b65cc404af2200b2 server_cookie=518cb08e11050d4b in_seq=0 out_seq=0 2024-09-19T08:03:46.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.556+0000 7fc5abfff640 10 client.15624 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:03:46.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.557+0000 7fc5abfff640 1 -- 172.21.15.98:0/2642171646 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5590e3d44860 con 0x5590e3df0300 2024-09-19T08:03:46.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.557+0000 7fc5abfff640 10 client.15624 handle_command_reply: tid=0 2024-09-19T08:03:46.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.557+0000 7fc5d5329740 1 client.15624 shutdown 2024-09-19T08:03:46.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.557+0000 7fc5d5329740 1 -- 172.21.15.98:0/2642171646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 msgr2=0x5590e3ddd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:46.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.557+0000 7fc5d5329740 1 --2- 172.21.15.98:0/2642171646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3ddd9e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8001eb0 tx=0x7fc5b801d960 comp rx=0 tx=0).stop 2024-09-19T08:03:46.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.557+0000 7fc5d5329740 1 -- 172.21.15.98:0/2642171646 shutdown_connections 2024-09-19T08:03:46.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.557+0000 7fc5d5329740 1 --2- 172.21.15.98:0/2642171646 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5590e3df0300 0x5590e3df26f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.557+0000 7fc5d5329740 1 --2- 172.21.15.98:0/2642171646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3ddd9e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.557+0000 7fc5d5329740 1 --2- 172.21.15.98:0/2642171646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd2b40 0x5590e3ddd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.557+0000 7fc5abfff640 0 client.15624 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:03:46.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.557+0000 7fc5d5329740 1 --2- 172.21.15.98:0/2642171646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd15d0 0x5590e3de0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.557+0000 7fc5d5329740 1 -- 172.21.15.98:0/2642171646 >> 172.21.15.98:0/2642171646 conn(0x5590e3d9a310 msgr2=0x5590e3dd3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:46.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.558+0000 7fc5d5329740 1 -- 172.21.15.98:0/2642171646 shutdown_connections 2024-09-19T08:03:46.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.558+0000 7fc5d5329740 1 -- 172.21.15.98:0/2642171646 wait complete. 2024-09-19T08:03:46.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.558+0000 7fc5d5329740 20 client.15624 trim_cache size 0 max 16384 2024-09-19T08:03:46.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.570+0000 7fc5d5329740 1 Processor -- start 2024-09-19T08:03:46.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5d5329740 1 -- start start 2024-09-19T08:03:46.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5d5329740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd15d0 0x5590e3dec720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5d5329740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd2b40 0x5590e3ddccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5d5329740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5d5329740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc1ab6c0 con 0x5590e3de0780 2024-09-19T08:03:46.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5d5329740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc1ad200 con 0x5590e3dd2b40 2024-09-19T08:03:46.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5d5329740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc1abd90 con 0x5590e3dd15d0 2024-09-19T08:03:46.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41280/0 (socket says 172.21.15.98:41280) 2024-09-19T08:03:46.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5cbfff640 1 -- 172.21.15.98:0/859514163 learned_addr learned my addr 172.21.15.98:0/859514163 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:46.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/859514163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd15d0 0x5590e3dec720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5d0cec640 1 --2- 172.21.15.98:0/859514163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd2b40 0x5590e3ddccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5cbfff640 1 -- 172.21.15.98:0/859514163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd15d0 msgr2=0x5590e3dec720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/859514163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd15d0 0x5590e3dec720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5cbfff640 1 -- 172.21.15.98:0/859514163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd2b40 msgr2=0x5590e3ddccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/859514163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd2b40 0x5590e3ddccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5cbfff640 1 -- 172.21.15.98:0/859514163 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590e3dd4120 con 0x5590e3de0780 2024-09-19T08:03:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.571+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/859514163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de1b70 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800be60 tx=0x7fc5b8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9893e6e649eb1a2e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.572+0000 7fc5b67fc640 1 -- 172.21.15.98:0/859514163 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b801d9d0 con 0x5590e3de0780 2024-09-19T08:03:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.572+0000 7fc5b67fc640 1 -- 172.21.15.98:0/859514163 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b8005590 con 0x5590e3de0780 2024-09-19T08:03:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.572+0000 7fc5b67fc640 1 -- 172.21.15.98:0/859514163 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8010bd0 con 0x5590e3de0780 2024-09-19T08:03:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.572+0000 7fc5d5329740 1 -- 172.21.15.98:0/859514163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 msgr2=0x5590e3de1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.572+0000 7fc5d5329740 1 --2- 172.21.15.98:0/859514163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de1b70 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800be60 tx=0x7fc5b8000bc0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.572+0000 7fc5d5329740 1 -- 172.21.15.98:0/859514163 shutdown_connections 2024-09-19T08:03:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.572+0000 7fc5d5329740 1 --2- 172.21.15.98:0/859514163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de1b70 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.572+0000 7fc5d5329740 1 --2- 172.21.15.98:0/859514163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd2b40 0x5590e3ddccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.572+0000 7fc5d5329740 1 --2- 172.21.15.98:0/859514163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd15d0 0x5590e3dec720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.572+0000 7fc5d5329740 1 -- 172.21.15.98:0/859514163 >> 172.21.15.98:0/859514163 conn(0x5590e3d9a310 msgr2=0x5590e3ddd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:46.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.573+0000 7fc5d5329740 1 -- 172.21.15.98:0/859514163 shutdown_connections 2024-09-19T08:03:46.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.573+0000 7fc5d5329740 1 -- 172.21.15.98:0/859514163 wait complete. 2024-09-19T08:03:46.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.573+0000 7fc5d5329740 1 -- start start 2024-09-19T08:03:46.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.573+0000 7fc5d5329740 1 Processor -- start 2024-09-19T08:03:46.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.573+0000 7fc5d5329740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd15d0 0x5590e3df7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.573+0000 7fc5d5329740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd2b40 0x5590e3df7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.573+0000 7fc5d5329740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.573+0000 7fc5d5329740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc1acb30 con 0x5590e3de0780 2024-09-19T08:03:46.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.573+0000 7fc5d5329740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc1ac460 con 0x5590e3dd2b40 2024-09-19T08:03:46.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.573+0000 7fc5d5329740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc1a96a0 con 0x5590e3dd15d0 2024-09-19T08:03:46.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.573+0000 7fc5d5329740 10 client.0 _ll_register_callbacks cb 0x7fc5b80123f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:46.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.573+0000 7fc5cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.573+0000 7fc5cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41296/0 (socket says 172.21.15.98:41296) 2024-09-19T08:03:46.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.573+0000 7fc5cbfff640 1 -- 172.21.15.98:0/2763344800 learned_addr learned my addr 172.21.15.98:0/2763344800 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:46.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5d0cec640 1 --2- 172.21.15.98:0/2763344800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd2b40 0x5590e3df7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/2763344800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd15d0 0x5590e3df7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5cbfff640 1 -- 172.21.15.98:0/2763344800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd15d0 msgr2=0x5590e3df7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:46.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/2763344800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd15d0 0x5590e3df7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5cbfff640 1 -- 172.21.15.98:0/2763344800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd2b40 msgr2=0x5590e3df7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:46.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/2763344800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd2b40 0x5590e3df7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5cbfff640 1 -- 172.21.15.98:0/2763344800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590e3de7f30 con 0x5590e3de0780 2024-09-19T08:03:46.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/2763344800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de7890 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800b6b0 tx=0x7fc5b80124a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89143107f4ef9c77 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:46.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5abfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:46.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5abfff640 1 -- 172.21.15.98:0/2763344800 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b801d5f0 con 0x5590e3de0780 2024-09-19T08:03:46.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5abfff640 1 -- 172.21.15.98:0/2763344800 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fc5b8004640 con 0x5590e3de0780 2024-09-19T08:03:46.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5d5329740 1 -- 172.21.15.98:0/2763344800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5b8005590 con 0x5590e3de0780 2024-09-19T08:03:46.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5abfff640 1 -- 172.21.15.98:0/2763344800 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc5b80120c0 con 0x5590e3de0780 2024-09-19T08:03:46.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5d5329740 10 client.15630 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:46.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.574+0000 7fc5d5329740 1 -- 172.21.15.98:0/2763344800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5590e3de9230 con 0x5590e3de0780 2024-09-19T08:03:46.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.575+0000 7fc5abfff640 1 -- 172.21.15.98:0/2763344800 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b8005590 con 0x5590e3de0780 2024-09-19T08:03:46.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.575+0000 7fc5abfff640 1 -- 172.21.15.98:0/2763344800 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc5b801c030 con 0x5590e3de0780 2024-09-19T08:03:46.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.575+0000 7fc5d5329740 10 client.15630 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.575+0000 7fc5d5329740 10 client.15630 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:03:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.575+0000 7fc5d5329740 20 client.15630 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.575+0000 7fc5d5329740 1 --2- 172.21.15.98:0/2763344800 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5590e3e5ae50 0x5590e3e5d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.575+0000 7fc5d5329740 4 client.15630 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:03:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.575+0000 7fc5d5329740 1 -- 172.21.15.98:0/2763344800 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5590e3d3ed70 con 0x5590e3e5ae50 2024-09-19T08:03:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.575+0000 7fc5d0cec640 1 --2- 172.21.15.98:0/2763344800 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5590e3e5ae50 0x5590e3e5d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5d0cec640 1 --2- 172.21.15.98:0/2763344800 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5590e3e5ae50 0x5590e3e5d240 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bbfdebcd47131fed server_cookie=1b70f3bf3b61d1a8 in_seq=0 out_seq=0 2024-09-19T08:03:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5abfff640 10 client.15630 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:03:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5abfff640 1 -- 172.21.15.98:0/2763344800 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5590e3d3ed70 con 0x5590e3e5ae50 2024-09-19T08:03:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5abfff640 10 client.15630 handle_command_reply: tid=0 2024-09-19T08:03:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5d5329740 1 client.15630 shutdown 2024-09-19T08:03:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5d5329740 1 -- 172.21.15.98:0/2763344800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 msgr2=0x5590e3de7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5d5329740 1 --2- 172.21.15.98:0/2763344800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de7890 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800b6b0 tx=0x7fc5b80124a0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5d5329740 1 -- 172.21.15.98:0/2763344800 shutdown_connections 2024-09-19T08:03:46.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5d5329740 1 --2- 172.21.15.98:0/2763344800 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5590e3e5ae50 0x5590e3e5d240 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5d5329740 1 --2- 172.21.15.98:0/2763344800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590e3de0780 0x5590e3de7890 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5d5329740 1 --2- 172.21.15.98:0/2763344800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590e3dd2b40 0x5590e3df7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5d5329740 1 --2- 172.21.15.98:0/2763344800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590e3dd15d0 0x5590e3df7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5d5329740 1 -- 172.21.15.98:0/2763344800 >> 172.21.15.98:0/2763344800 conn(0x5590e3d9a310 msgr2=0x5590e3dd3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:46.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5abfff640 0 client.15630 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:03:46.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5d5329740 1 -- 172.21.15.98:0/2763344800 shutdown_connections 2024-09-19T08:03:46.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.576+0000 7fc5d5329740 1 -- 172.21.15.98:0/2763344800 wait complete. 2024-09-19T08:03:46.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.577+0000 7fc5d5329740 20 client.15630 trim_cache size 0 max 16384 2024-09-19T08:03:46.584 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:03:46.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.577+0000 7fc5d2f50640 1 -- 172.21.15.98:0/1872705913 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc5bc0a3550 msgr2=0x7fc5bc0a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:46.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.577+0000 7fc5d2f50640 1 --2- 172.21.15.98:0/1872705913 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc5bc0a3550 0x7fc5bc0a5a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40ccb50 tx=0x7fc5c4009970 comp rx=0 tx=0).stop 2024-09-19T08:03:46.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.577+0000 7fc5d2f50640 1 -- 172.21.15.98:0/1872705913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc106a60 msgr2=0x7fc5cc1a43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:46.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.577+0000 7fc5d2f50640 1 --2- 172.21.15.98:0/1872705913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc106a60 0x7fc5cc1a43c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400b970 tx=0x7fc5c400be30 comp rx=0 tx=0).stop 2024-09-19T08:03:46.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.577+0000 7fc5d2f50640 1 -- 172.21.15.98:0/1872705913 shutdown_connections 2024-09-19T08:03:46.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.577+0000 7fc5d2f50640 1 --2- 172.21.15.98:0/1872705913 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc5bc0a3550 0x7fc5bc0a5a10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.577+0000 7fc5d2f50640 1 --2- 172.21.15.98:0/1872705913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc106a60 0x7fc5cc1a43c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.577+0000 7fc5d2f50640 1 --2- 172.21.15.98:0/1872705913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5cc101400 0x7fc5cc199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.577+0000 7fc5d2f50640 1 --2- 172.21.15.98:0/1872705913 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5cc100a60 0x7fc5cc199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:46.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.577+0000 7fc5d2f50640 1 -- 172.21.15.98:0/1872705913 >> 172.21.15.98:0/1872705913 conn(0x7fc5cc0fc780 msgr2=0x7fc5cc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:46.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.577+0000 7fc5d2f50640 1 -- 172.21.15.98:0/1872705913 shutdown_connections 2024-09-19T08:03:46.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:46.577+0000 7fc5d2f50640 1 -- 172.21.15.98:0/1872705913 wait complete. 2024-09-19T08:03:47.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:46 smithi162 ceph-mon[35260]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:03:47.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:46 smithi139 ceph-mon[34874]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:03:47.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:46 smithi098 ceph-mon[31333]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:03:47.353 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 11.2 2024-09-19T08:03:47.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:03:47.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:03:47.629 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:48.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.138+0000 7fcb43b63640 1 -- 172.21.15.98:0/876191921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c102ef0 msgr2=0x7fcb3c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:48.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.138+0000 7fcb43b63640 1 --2- 172.21.15.98:0/876191921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c102ef0 0x7fcb3c10da40 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fcb34009ca0 tx=0x7fcb3402a950 comp rx=0 tx=0).stop 2024-09-19T08:03:48.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.138+0000 7fcb43b63640 1 -- 172.21.15.98:0/876191921 shutdown_connections 2024-09-19T08:03:48.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.138+0000 7fcb43b63640 1 --2- 172.21.15.98:0/876191921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c102ef0 0x7fcb3c10da40 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.138+0000 7fcb43b63640 1 --2- 172.21.15.98:0/876191921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb3c102570 0x7fcb3c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.138+0000 7fcb43b63640 1 --2- 172.21.15.98:0/876191921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb3c100e20 0x7fcb3c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.138+0000 7fcb43b63640 1 -- 172.21.15.98:0/876191921 >> 172.21.15.98:0/876191921 conn(0x7fcb3c0fc7d0 msgr2=0x7fcb3c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:48.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.138+0000 7fcb43b63640 1 -- 172.21.15.98:0/876191921 shutdown_connections 2024-09-19T08:03:48.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.138+0000 7fcb43b63640 1 -- 172.21.15.98:0/876191921 wait complete. 2024-09-19T08:03:48.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.139+0000 7fcb43b63640 1 Processor -- start 2024-09-19T08:03:48.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.139+0000 7fcb43b63640 1 -- start start 2024-09-19T08:03:48.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.139+0000 7fcb43b63640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c100e20 0x7fcb3c199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.139+0000 7fcb43b63640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb3c102570 0x7fcb3c199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.139+0000 7fcb43b63640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb3c102ef0 0x7fcb3c1a4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.139+0000 7fcb43b63640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c110f80 con 0x7fcb3c100e20 2024-09-19T08:03:48.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.139+0000 7fcb43b63640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c110e00 con 0x7fcb3c102570 2024-09-19T08:03:48.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.139+0000 7fcb43b63640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c111100 con 0x7fcb3c102ef0 2024-09-19T08:03:48.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.139+0000 7fcb418ff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c100e20 0x7fcb3c199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.139+0000 7fcb418ff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c100e20 0x7fcb3c199210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41320/0 (socket says 172.21.15.98:41320) 2024-09-19T08:03:48.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.139+0000 7fcb418ff640 1 -- 172.21.15.98:0/559804881 learned_addr learned my addr 172.21.15.98:0/559804881 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:48.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.140+0000 7fcb42100640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb3c102ef0 0x7fcb3c1a4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.140+0000 7fcb410fe640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb3c102570 0x7fcb3c199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.140+0000 7fcb418ff640 1 -- 172.21.15.98:0/559804881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb3c102ef0 msgr2=0x7fcb3c1a4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:48.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.140+0000 7fcb418ff640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb3c102ef0 0x7fcb3c1a4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.140+0000 7fcb418ff640 1 -- 172.21.15.98:0/559804881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb3c102570 msgr2=0x7fcb3c199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:48.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.140+0000 7fcb418ff640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb3c102570 0x7fcb3c199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.140+0000 7fcb418ff640 1 -- 172.21.15.98:0/559804881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb3c1a47d0 con 0x7fcb3c100e20 2024-09-19T08:03:48.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.140+0000 7fcb410fe640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb3c102570 0x7fcb3c199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:48.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.140+0000 7fcb418ff640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c100e20 0x7fcb3c199210 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800c920 tx=0x7fcb3800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:48.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.140+0000 7fcb32ffd640 1 -- 172.21.15.98:0/559804881 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb38007c70 con 0x7fcb3c100e20 2024-09-19T08:03:48.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.140+0000 7fcb32ffd640 1 -- 172.21.15.98:0/559804881 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb38007e10 con 0x7fcb3c100e20 2024-09-19T08:03:48.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.140+0000 7fcb43b63640 1 -- 172.21.15.98:0/559804881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb3c1a49a0 con 0x7fcb3c100e20 2024-09-19T08:03:48.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.140+0000 7fcb32ffd640 1 -- 172.21.15.98:0/559804881 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb38018890 con 0x7fcb3c100e20 2024-09-19T08:03:48.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.140+0000 7fcb43b63640 1 -- 172.21.15.98:0/559804881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb3c1a4ee0 con 0x7fcb3c100e20 2024-09-19T08:03:48.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.141+0000 7fcb43b63640 1 -- 172.21.15.98:0/559804881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb3c101200 con 0x7fcb3c100e20 2024-09-19T08:03:48.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.143+0000 7fcb32ffd640 1 -- 172.21.15.98:0/559804881 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7fcb38018a50 con 0x7fcb3c100e20 2024-09-19T08:03:48.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.144+0000 7fcb32ffd640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcb280a3550 0x7fcb280a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.144+0000 7fcb410fe640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcb280a3550 0x7fcb280a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.144+0000 7fcb32ffd640 1 -- 172.21.15.98:0/559804881 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fcb380d03f0 con 0x7fcb3c100e20 2024-09-19T08:03:48.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.144+0000 7fcb410fe640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcb280a3550 0x7fcb280a5a10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c005fd0 tx=0x7fcb2c005850 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:48.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.147+0000 7fcb32ffd640 1 -- 172.21.15.98:0/559804881 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fcb38014030 con 0x7fcb3c100e20 2024-09-19T08:03:48.196 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.380+0000 7fcb43b63640 1 -- 172.21.15.98:0/559804881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb3c1a5170 con 0x7fcb3c100e20 2024-09-19T08:03:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.380+0000 7fcb32ffd640 1 -- 172.21.15.98:0/559804881 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11983 (secure 0 0 0) 0x7fcb3809cc60 con 0x7fcb3c100e20 2024-09-19T08:03:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:03:48.381 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"btime":"2024-09-19T08:02:59:855935+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:02:59.805764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:03:48.381 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-09-19T08:03:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.383+0000 7fcb43b63640 1 -- 172.21.15.98:0/559804881 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcb280a3550 msgr2=0x7fcb280a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.383+0000 7fcb43b63640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcb280a3550 0x7fcb280a5a10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c005fd0 tx=0x7fcb2c005850 comp rx=0 tx=0).stop 2024-09-19T08:03:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.383+0000 7fcb43b63640 1 -- 172.21.15.98:0/559804881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c100e20 msgr2=0x7fcb3c199210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.383+0000 7fcb43b63640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c100e20 0x7fcb3c199210 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800c920 tx=0x7fcb3800cde0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.383+0000 7fcb43b63640 1 -- 172.21.15.98:0/559804881 shutdown_connections 2024-09-19T08:03:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.383+0000 7fcb43b63640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcb280a3550 0x7fcb280a5a10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.383+0000 7fcb43b63640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb3c102ef0 0x7fcb3c1a4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.383+0000 7fcb43b63640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb3c102570 0x7fcb3c199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.383+0000 7fcb43b63640 1 --2- 172.21.15.98:0/559804881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c100e20 0x7fcb3c199210 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.383+0000 7fcb43b63640 1 -- 172.21.15.98:0/559804881 >> 172.21.15.98:0/559804881 conn(0x7fcb3c0fc7d0 msgr2=0x7fcb3c1067c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.384+0000 7fcb43b63640 1 -- 172.21.15.98:0/559804881 shutdown_connections 2024-09-19T08:03:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.384+0000 7fcb43b63640 1 -- 172.21.15.98:0/559804881 wait complete. 2024-09-19T08:03:48.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:48 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/559804881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:48.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:48 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/559804881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.887+0000 7fface99f640 1 -- 172.21.15.98:0/3651635495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffac8102b90 msgr2=0x7ffac8102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.887+0000 7fface99f640 1 --2- 172.21.15.98:0/3651635495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffac8102b90 0x7ffac8102f70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ffac4009900 tx=0x7ffac402a480 comp rx=0 tx=0).stop 2024-09-19T08:03:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.887+0000 7fface99f640 1 -- 172.21.15.98:0/3651635495 shutdown_connections 2024-09-19T08:03:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.887+0000 7fface99f640 1 --2- 172.21.15.98:0/3651635495 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffac810bfc0 0x7ffac810e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.887+0000 7fface99f640 1 --2- 172.21.15.98:0/3651635495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffac81034b0 0x7ffac810b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.887+0000 7fface99f640 1 --2- 172.21.15.98:0/3651635495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffac8102b90 0x7ffac8102f70 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.887+0000 7fface99f640 1 -- 172.21.15.98:0/3651635495 >> 172.21.15.98:0/3651635495 conn(0x7ffac80fc610 msgr2=0x7ffac80fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.887+0000 7fface99f640 1 -- 172.21.15.98:0/3651635495 shutdown_connections 2024-09-19T08:03:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.887+0000 7fface99f640 1 -- 172.21.15.98:0/3651635495 wait complete. 2024-09-19T08:03:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.888+0000 7fface99f640 1 Processor -- start 2024-09-19T08:03:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.888+0000 7fface99f640 1 -- start start 2024-09-19T08:03:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.888+0000 7fface99f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffac8102b90 0x7ffac81007f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.888+0000 7fface99f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffac81034b0 0x7ffac8100d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.888+0000 7fface99f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffac810bfc0 0x7ffac8101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.888+0000 7fface99f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffac8111dc0 con 0x7ffac8102b90 2024-09-19T08:03:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.888+0000 7fface99f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ffac8111c40 con 0x7ffac81034b0 2024-09-19T08:03:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.888+0000 7fface99f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffac8111f40 con 0x7ffac810bfc0 2024-09-19T08:03:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.888+0000 7ffacd99d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffac8102b90 0x7ffac81007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.888+0000 7ffacd99d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffac8102b90 0x7ffac81007f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41340/0 (socket says 172.21.15.98:41340) 2024-09-19T08:03:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.888+0000 7ffacd99d640 1 -- 172.21.15.98:0/3244344527 learned_addr learned my addr 172.21.15.98:0/3244344527 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.889+0000 7ffacd19c640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffac81034b0 0x7ffac8100d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.889+0000 7ffacd99d640 1 -- 172.21.15.98:0/3244344527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffac810bfc0 msgr2=0x7ffac8101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.889+0000 7fface19e640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffac810bfc0 0x7ffac8101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.889+0000 7ffacd99d640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffac810bfc0 0x7ffac8101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.889+0000 7ffacd99d640 1 -- 172.21.15.98:0/3244344527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffac81034b0 msgr2=0x7ffac8100d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.889+0000 7ffacd99d640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffac81034b0 0x7ffac8100d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.889+0000 7ffacd99d640 1 -- 172.21.15.98:0/3244344527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac81a6700 con 0x7ffac8102b90 2024-09-19T08:03:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.889+0000 7ffacd19c640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffac81034b0 0x7ffac8100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.889+0000 7ffacd99d640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffac8102b90 0x7ffac81007f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ffac4006010 tx=0x7ffac40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.889+0000 7ffabeffd640 1 -- 172.21.15.98:0/3244344527 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac403f030 con 0x7ffac8102b90 2024-09-19T08:03:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.889+0000 7fface99f640 1 -- 172.21.15.98:0/3244344527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffac81a6990 con 0x7ffac8102b90 2024-09-19T08:03:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.889+0000 7ffabeffd640 1 -- 172.21.15.98:0/3244344527 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffac4003ce0 con 0x7ffac8102b90 2024-09-19T08:03:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.889+0000 7ffabeffd640 1 -- 172.21.15.98:0/3244344527 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac4035070 con 0x7ffac8102b90 2024-09-19T08:03:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.889+0000 7fface99f640 1 -- 172.21.15.98:0/3244344527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffac81a6ec0 con 0x7ffac8102b90 2024-09-19T08:03:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.892+0000 7ffabeffd640 1 -- 172.21.15.98:0/3244344527 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7ffac4042030 con 0x7ffac8102b90 2024-09-19T08:03:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.892+0000 7ffabeffd640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ffab40a34d0 0x7ffab40a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.892+0000 7ffabeffd640 1 -- 172.21.15.98:0/3244344527 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ffac40f0640 con 0x7ffac8102b90 2024-09-19T08:03:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.892+0000 7ffacd19c640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ffab40a34d0 0x7ffab40a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.893+0000 7ffad1fda740 1 Processor -- start 2024-09-19T08:03:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.893+0000 7ffacd19c640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ffab40a34d0 0x7ffab40a5990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffac8102280 tx=0x7ffab8009670 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.893+0000 7ffad1fda740 1 -- start start 2024-09-19T08:03:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.893+0000 7ffad1fda740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b3d20 0x556b207b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.893+0000 7ffad1fda740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b27b0 0x556b207b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.893+0000 7ffad1fda740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207bd550 0x556b207bf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.893+0000 7ffad1fda740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556b207182f0 con 0x556b207b3d20 2024-09-19T08:03:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.893+0000 7ffad1fda740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556b206bfcf0 con 0x556b207bd550 2024-09-19T08:03:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.893+0000 7ffad1fda740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556b207142b0 con 0x556b207b27b0 2024-09-19T08:03:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.893+0000 7ffacd99d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b3d20 0x556b207b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.893+0000 7ffacd99d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b3d20 0x556b207b2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41342/0 (socket says 172.21.15.98:41342) 2024-09-19T08:03:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.893+0000 7ffacd99d640 1 -- 172.21.15.98:0/946655442 learned_addr learned my addr 172.21.15.98:0/946655442 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.893+0000 7ffacd19c640 1 --2- 172.21.15.98:0/946655442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b27b0 0x556b207b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.894+0000 7fface19e640 1 --2- 172.21.15.98:0/946655442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207bd550 0x556b207bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.894+0000 7ffacd99d640 1 -- 172.21.15.98:0/946655442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b27b0 msgr2=0x556b207b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.894+0000 7ffacd99d640 1 --2- 172.21.15.98:0/946655442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b27b0 0x556b207b2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.894+0000 7ffacd99d640 1 -- 172.21.15.98:0/946655442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207bd550 msgr2=0x556b207bf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.894+0000 7ffacd99d640 1 --2- 172.21.15.98:0/946655442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207bd550 0x556b207bf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.894+0000 7ffacd99d640 1 -- 172.21.15.98:0/946655442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b206a43a0 con 0x556b207b3d20 2024-09-19T08:03:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.894+0000 7ffacd19c640 1 --2- 172.21.15.98:0/946655442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b27b0 0x556b207b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.894+0000 7fface19e640 1 --2- 172.21.15.98:0/946655442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207bd550 0x556b207bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.894+0000 7ffacd99d640 1 --2- 172.21.15.98:0/946655442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b3d20 0x556b207b2270 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ffac40ee3c0 tx=0x7ffac40bc450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65e943cbd352af4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.894+0000 7ffab3fff640 1 -- 172.21.15.98:0/946655442 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac4049070 con 0x556b207b3d20 2024-09-19T08:03:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.894+0000 7ffab3fff640 1 -- 172.21.15.98:0/946655442 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffac403b580 con 0x556b207b3d20 2024-09-19T08:03:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.894+0000 7ffab3fff640 1 -- 172.21.15.98:0/946655442 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac40c0c40 con 0x556b207b3d20 2024-09-19T08:03:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.895+0000 7ffad1fda740 1 -- 172.21.15.98:0/946655442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b3d20 msgr2=0x556b207b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.895+0000 7ffad1fda740 1 --2- 172.21.15.98:0/946655442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b3d20 0x556b207b2270 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ffac40ee3c0 tx=0x7ffac40bc450 comp rx=0 tx=0).stop 2024-09-19T08:03:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.895+0000 7ffad1fda740 1 -- 172.21.15.98:0/946655442 shutdown_connections 2024-09-19T08:03:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.895+0000 7ffad1fda740 1 --2- 172.21.15.98:0/946655442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207bd550 0x556b207bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.895+0000 7ffad1fda740 1 --2- 172.21.15.98:0/946655442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b27b0 0x556b207b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.895+0000 7ffad1fda740 1 --2- 172.21.15.98:0/946655442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b3d20 0x556b207b2270 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.895+0000 7ffad1fda740 1 -- 172.21.15.98:0/946655442 >> 172.21.15.98:0/946655442 conn(0x556b2077b310 msgr2=0x556b207aee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.895+0000 7ffad1fda740 1 -- 172.21.15.98:0/946655442 shutdown_connections 2024-09-19T08:03:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.895+0000 7ffad1fda740 1 -- 172.21.15.98:0/946655442 wait complete. 2024-09-19T08:03:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.896+0000 7ffad1fda740 1 -- start start 2024-09-19T08:03:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.896+0000 7ffad1fda740 1 Processor -- start 2024-09-19T08:03:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.896+0000 7ffad1fda740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b27b0 0x556b207bcf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.896+0000 7ffad1fda740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b3d20 0x556b207ba1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.896+0000 7ffad1fda740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207bd550 0x556b207ba7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.896+0000 7ffad1fda740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556b207c0010 con 0x556b207b27b0 2024-09-19T08:03:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.896+0000 7ffad1fda740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556b206d3060 con 0x556b207bd550 2024-09-19T08:03:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.896+0000 7ffad1fda740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556b207c0190 con 0x556b207b3d20 2024-09-19T08:03:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.896+0000 7ffad1fda740 10 client.0 _ll_register_callbacks cb 0x556b204268c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.896+0000 7ffacd99d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b27b0 0x556b207bcf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.896+0000 7ffacd99d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b27b0 0x556b207bcf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41356/0 (socket says 172.21.15.98:41356) 2024-09-19T08:03:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.896+0000 7ffacd99d640 1 -- 172.21.15.98:0/987077319 learned_addr learned my addr 172.21.15.98:0/987077319 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.896+0000 7fface19e640 1 --2- 172.21.15.98:0/987077319 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207bd550 0x556b207ba7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffacd19c640 1 --2- 172.21.15.98:0/987077319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b3d20 0x556b207ba1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffacd99d640 1 -- 172.21.15.98:0/987077319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b3d20 msgr2=0x556b207ba1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffacd99d640 1 --2- 172.21.15.98:0/987077319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b3d20 0x556b207ba1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffacd99d640 1 -- 172.21.15.98:0/987077319 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207bd550 msgr2=0x556b207ba7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffacd99d640 1 --2- 172.21.15.98:0/987077319 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207bd550 0x556b207ba7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffacd99d640 1 -- 172.21.15.98:0/987077319 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b204893c0 con 0x556b207b27b0 2024-09-19T08:03:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffacd99d640 1 --2- 172.21.15.98:0/987077319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b27b0 0x556b207bcf30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ffac40c4fd0 tx=0x7ffac403bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=131bea17cedc4831 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffab17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffab17fa640 1 -- 172.21.15.98:0/987077319 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac4086060 con 0x556b207b27b0 2024-09-19T08:03:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffab17fa640 1 -- 172.21.15.98:0/987077319 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ffac40c5070 con 0x556b207b27b0 2024-09-19T08:03:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffad1fda740 1 -- 172.21.15.98:0/987077319 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556b20725860 con 0x556b207b27b0 2024-09-19T08:03:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffab17fa640 1 -- 172.21.15.98:0/987077319 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffac4049070 con 0x556b207b27b0 2024-09-19T08:03:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffad1fda740 10 client.15648 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffad1fda740 1 -- 172.21.15.98:0/987077319 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556b207c0010 con 0x556b207b27b0 2024-09-19T08:03:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.897+0000 7ffab17fa640 1 -- 172.21.15.98:0/987077319 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556b20725860 con 0x556b207b27b0 2024-09-19T08:03:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.898+0000 7ffab17fa640 1 -- 172.21.15.98:0/987077319 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ffac40bdbc0 con 0x556b207b27b0 2024-09-19T08:03:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.898+0000 7ffad1fda740 10 client.15648 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.898+0000 7ffad1fda740 10 client.15648 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:03:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.898+0000 7ffad1fda740 20 client.15648 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.898+0000 7ffad1fda740 1 --2- 172.21.15.98:0/987077319 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556b207ccb80 0x556b207cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.898+0000 7ffad1fda740 4 client.15648 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.898+0000 7ffad1fda740 1 -- 172.21.15.98:0/987077319 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b2071fd70 con 0x556b207ccb80 2024-09-19T08:03:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.898+0000 7ffacd19c640 1 --2- 172.21.15.98:0/987077319 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556b207ccb80 0x556b207cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.898+0000 7ffab17fa640 10 client.15648 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:03:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.898+0000 7ffacd19c640 1 --2- 172.21.15.98:0/987077319 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556b207ccb80 0x556b207cefa0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5793fd7d8b7fdf35 server_cookie=a6b4be86adc09e91 in_seq=0 out_seq=0 2024-09-19T08:03:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.899+0000 7ffab17fa640 1 -- 172.21.15.98:0/987077319 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556b2071fd70 con 0x556b207ccb80 2024-09-19T08:03:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.899+0000 7ffab17fa640 10 client.15648 handle_command_reply: tid=0 2024-09-19T08:03:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.899+0000 7ffad1fda740 1 client.15648 shutdown 2024-09-19T08:03:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.899+0000 7ffad1fda740 1 -- 172.21.15.98:0/987077319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b27b0 msgr2=0x556b207bcf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.899+0000 7ffad1fda740 1 --2- 172.21.15.98:0/987077319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b27b0 0x556b207bcf30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ffac40c4fd0 tx=0x7ffac403bcb0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.899+0000 7ffad1fda740 1 -- 172.21.15.98:0/987077319 shutdown_connections 2024-09-19T08:03:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.899+0000 7ffad1fda740 1 --2- 172.21.15.98:0/987077319 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207bd550 0x556b207ba7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.899+0000 7ffad1fda740 1 --2- 172.21.15.98:0/987077319 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556b207ccb80 0x556b207cefa0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.899+0000 7ffad1fda740 1 --2- 172.21.15.98:0/987077319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b3d20 0x556b207ba1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.899+0000 7ffad1fda740 1 --2- 172.21.15.98:0/987077319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b27b0 0x556b207bcf30 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.899+0000 7ffab17fa640 0 client.15648 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:03:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.899+0000 7ffad1fda740 1 -- 172.21.15.98:0/987077319 >> 172.21.15.98:0/987077319 conn(0x556b2077b310 msgr2=0x556b207b4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.900+0000 7ffad1fda740 1 -- 172.21.15.98:0/987077319 shutdown_connections 2024-09-19T08:03:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.900+0000 7ffad1fda740 1 -- 172.21.15.98:0/987077319 wait complete. 2024-09-19T08:03:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.900+0000 7ffad1fda740 20 client.15648 trim_cache size 0 max 16384 2024-09-19T08:03:48.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.913+0000 7ffad1fda740 1 Processor -- start 2024-09-19T08:03:48.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.913+0000 7ffad1fda740 1 -- start start 2024-09-19T08:03:48.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.913+0000 7ffad1fda740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b27b0 0x556b207c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.913+0000 7ffad1fda740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207b3d20 0x556b207b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.913+0000 7ffad1fda740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207bd550 0x556b207b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.913+0000 7ffad1fda740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffac81af830 con 0x556b207b27b0 2024-09-19T08:03:48.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.913+0000 7ffad1fda740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ffac81b1370 con 0x556b207b3d20 2024-09-19T08:03:48.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.913+0000 7ffad1fda740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffac81aff00 con 0x556b207bd550 2024-09-19T08:03:48.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.913+0000 7ffacd99d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b27b0 0x556b207c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.913+0000 7ffacd99d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b27b0 0x556b207c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41358/0 (socket says 172.21.15.98:41358) 2024-09-19T08:03:48.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.913+0000 7ffacd99d640 1 -- 172.21.15.98:0/736041545 learned_addr learned my addr 172.21.15.98:0/736041545 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:48.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.913+0000 7ffacd19c640 1 --2- 172.21.15.98:0/736041545 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207b3d20 0x556b207b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.913+0000 7fface19e640 1 --2- 172.21.15.98:0/736041545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207bd550 0x556b207b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.914+0000 7ffacd99d640 1 -- 172.21.15.98:0/736041545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207bd550 msgr2=0x556b207b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:48.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.914+0000 7ffacd99d640 1 --2- 172.21.15.98:0/736041545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207bd550 0x556b207b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.914+0000 7ffacd99d640 1 -- 172.21.15.98:0/736041545 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207b3d20 msgr2=0x556b207b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:48.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.914+0000 7ffacd99d640 1 --2- 172.21.15.98:0/736041545 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207b3d20 0x556b207b5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.914+0000 7ffacd99d640 1 -- 172.21.15.98:0/736041545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b207b3940 con 0x556b207b27b0 2024-09-19T08:03:48.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.914+0000 7fface19e640 1 --2- 172.21.15.98:0/736041545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207bd550 0x556b207b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:48.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.914+0000 7ffacd99d640 1 --2- 172.21.15.98:0/736041545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b27b0 0x556b207c9270 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ffac4005ab0 tx=0x7ffac40bd0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e9e1507be7f8db server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:48.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.914+0000 7ffab37fe640 1 -- 172.21.15.98:0/736041545 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac4086020 con 0x556b207b27b0 2024-09-19T08:03:48.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.914+0000 7ffab37fe640 1 -- 172.21.15.98:0/736041545 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffac4073e50 con 0x556b207b27b0 2024-09-19T08:03:48.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.914+0000 7ffab37fe640 1 -- 172.21.15.98:0/736041545 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac40db050 con 0x556b207b27b0 2024-09-19T08:03:48.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.915+0000 7ffad1fda740 1 -- 172.21.15.98:0/736041545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b27b0 msgr2=0x556b207c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:48.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.915+0000 7ffad1fda740 1 --2- 172.21.15.98:0/736041545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b27b0 0x556b207c9270 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ffac4005ab0 tx=0x7ffac40bd0f0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.915+0000 7ffad1fda740 1 -- 172.21.15.98:0/736041545 shutdown_connections 2024-09-19T08:03:48.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.915+0000 7ffad1fda740 1 --2- 172.21.15.98:0/736041545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207bd550 0x556b207b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.915+0000 7ffad1fda740 1 --2- 172.21.15.98:0/736041545 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207b3d20 0x556b207b5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.915+0000 7ffad1fda740 1 --2- 172.21.15.98:0/736041545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207b27b0 0x556b207c9270 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.915+0000 7ffad1fda740 1 -- 172.21.15.98:0/736041545 >> 172.21.15.98:0/736041545 conn(0x556b2077b310 msgr2=0x556b207b92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:48.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.915+0000 7ffad1fda740 1 -- 172.21.15.98:0/736041545 shutdown_connections 2024-09-19T08:03:48.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.915+0000 7ffad1fda740 1 -- 172.21.15.98:0/736041545 wait complete. 2024-09-19T08:03:48.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.915+0000 7ffad1fda740 1 -- start start 2024-09-19T08:03:48.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.915+0000 7ffad1fda740 1 Processor -- start 2024-09-19T08:03:48.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7ffad1fda740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b27b0 0x556b207c5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7ffad1fda740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207b3d20 0x556b207c0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7ffad1fda740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207bd550 0x556b207c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7ffad1fda740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffac81b0ca0 con 0x556b207bd550 2024-09-19T08:03:48.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7ffad1fda740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ffac81b05d0 con 0x556b207b3d20 2024-09-19T08:03:48.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7ffad1fda740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffac81ad810 con 0x556b207b27b0 2024-09-19T08:03:48.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7ffad1fda740 10 client.0 _ll_register_callbacks cb 0x7ffac40bc6f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:48.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7fface19e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207bd550 0x556b207c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7ffacd99d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b27b0 0x556b207c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7fface19e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207bd550 0x556b207c10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41370/0 (socket says 172.21.15.98:41370) 2024-09-19T08:03:48.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7fface19e640 1 -- 172.21.15.98:0/435084113 learned_addr learned my addr 172.21.15.98:0/435084113 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:48.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7ffacd19c640 1 --2- 172.21.15.98:0/435084113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207b3d20 0x556b207c0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7fface19e640 1 -- 172.21.15.98:0/435084113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b27b0 msgr2=0x556b207c5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:48.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7fface19e640 1 --2- 172.21.15.98:0/435084113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b27b0 0x556b207c5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7fface19e640 1 -- 172.21.15.98:0/435084113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207b3d20 msgr2=0x556b207c0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:48.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7fface19e640 1 --2- 172.21.15.98:0/435084113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207b3d20 0x556b207c0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7fface19e640 1 -- 172.21.15.98:0/435084113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b207c1610 con 0x556b207bd550 2024-09-19T08:03:48.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.916+0000 7ffacd19c640 1 --2- 172.21.15.98:0/435084113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207b3d20 0x556b207c0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:48.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7fface19e640 1 --2- 172.21.15.98:0/435084113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207bd550 0x556b207c10d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ffac000fa00 tx=0x7ffac000faf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732e67178a2c91af server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:48.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffab17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:48.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffab17fa640 1 -- 172.21.15.98:0/435084113 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac001e060 con 0x556b207bd550 2024-09-19T08:03:48.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffab17fa640 1 -- 172.21.15.98:0/435084113 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ffac000ead0 con 0x556b207bd550 2024-09-19T08:03:48.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffad1fda740 1 -- 172.21.15.98:0/435084113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffac4073e50 con 0x556b207bd550 2024-09-19T08:03:48.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffab17fa640 1 -- 172.21.15.98:0/435084113 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffac000ecd0 con 0x556b207bd550 2024-09-19T08:03:48.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffad1fda740 10 client.15654 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:48.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffad1fda740 1 -- 172.21.15.98:0/435084113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556b2083fde0 con 0x556b207bd550 2024-09-19T08:03:48.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffab17fa640 1 -- 172.21.15.98:0/435084113 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffac4073e50 con 0x556b207bd550 2024-09-19T08:03:48.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffab17fa640 1 -- 172.21.15.98:0/435084113 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ffac0018070 con 0x556b207bd550 2024-09-19T08:03:48.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffad1fda740 10 client.15654 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:48.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffad1fda740 10 client.15654 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:03:48.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffad1fda740 20 client.15654 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:48.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffad1fda740 1 --2- 172.21.15.98:0/435084113 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556b20840530 0x556b20842950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:48.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffad1fda740 4 client.15654 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:03:48.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.917+0000 7ffad1fda740 1 -- 172.21.15.98:0/435084113 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556b2069c170 con 0x556b20840530 2024-09-19T08:03:48.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.918+0000 7ffacd99d640 1 --2- 172.21.15.98:0/435084113 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556b20840530 0x556b20842950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:48.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.918+0000 7ffab17fa640 10 client.15654 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:03:48.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.918+0000 7ffacd99d640 1 --2- 172.21.15.98:0/435084113 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556b20840530 0x556b20842950 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=21ed68e2cb4cdf12 server_cookie=e232d7ce672485b4 in_seq=0 out_seq=0 2024-09-19T08:03:48.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.918+0000 7ffab17fa640 1 -- 172.21.15.98:0/435084113 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556b2069c170 con 0x556b20840530 2024-09-19T08:03:48.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.918+0000 7ffab17fa640 10 client.15654 handle_command_reply: tid=0 2024-09-19T08:03:48.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.918+0000 7ffad1fda740 1 client.15654 shutdown 2024-09-19T08:03:48.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.918+0000 7ffad1fda740 1 -- 172.21.15.98:0/435084113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207bd550 msgr2=0x556b207c10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:48.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.918+0000 7ffad1fda740 1 --2- 172.21.15.98:0/435084113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207bd550 0x556b207c10d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ffac000fa00 tx=0x7ffac000faf0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.919+0000 7ffad1fda740 1 -- 172.21.15.98:0/435084113 shutdown_connections 2024-09-19T08:03:48.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.919+0000 7ffad1fda740 1 --2- 172.21.15.98:0/435084113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556b207bd550 0x556b207c10d0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.919+0000 7ffad1fda740 1 --2- 172.21.15.98:0/435084113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556b207b3d20 0x556b207c0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.919+0000 7ffad1fda740 1 --2- 172.21.15.98:0/435084113 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556b20840530 0x556b20842950 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.919+0000 7ffad1fda740 1 --2- 172.21.15.98:0/435084113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b207b27b0 0x556b207c5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.919+0000 7ffad1fda740 1 -- 172.21.15.98:0/435084113 >> 172.21.15.98:0/435084113 conn(0x556b2077b310 msgr2=0x556b207bba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:48.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.919+0000 7ffab17fa640 0 client.15654 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:03:48.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.919+0000 7ffad1fda740 1 -- 172.21.15.98:0/435084113 shutdown_connections 2024-09-19T08:03:48.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.919+0000 7ffad1fda740 1 -- 172.21.15.98:0/435084113 wait complete. 2024-09-19T08:03:48.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.919+0000 7ffad1fda740 20 client.15654 trim_cache size 0 max 16384 2024-09-19T08:03:48.924 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:03:48.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.919+0000 7fface99f640 1 -- 172.21.15.98:0/3244344527 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ffab40a34d0 msgr2=0x7ffab40a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:48.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.919+0000 7fface99f640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ffab40a34d0 0x7ffab40a5990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffac8102280 tx=0x7ffab8009670 comp rx=0 tx=0).stop 2024-09-19T08:03:48.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.919+0000 7fface99f640 1 -- 172.21.15.98:0/3244344527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffac8102b90 msgr2=0x7ffac81007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:48.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.919+0000 7fface99f640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffac8102b90 0x7ffac81007f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ffac4006010 tx=0x7ffac40042f0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.920+0000 7fface99f640 1 -- 172.21.15.98:0/3244344527 shutdown_connections 2024-09-19T08:03:48.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.920+0000 7fface99f640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ffab40a34d0 0x7ffab40a5990 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.920+0000 7fface99f640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffac810bfc0 0x7ffac8101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.920+0000 7fface99f640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffac81034b0 0x7ffac8100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.920+0000 7fface99f640 1 --2- 172.21.15.98:0/3244344527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffac8102b90 0x7ffac81007f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:48.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.920+0000 7fface99f640 1 -- 172.21.15.98:0/3244344527 >> 172.21.15.98:0/3244344527 conn(0x7ffac80fc610 msgr2=0x7ffac80fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:48.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.920+0000 7fface99f640 1 -- 172.21.15.98:0/3244344527 shutdown_connections 2024-09-19T08:03:48.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:48.920+0000 7fface99f640 1 -- 172.21.15.98:0/3244344527 wait complete. 2024-09-19T08:03:48.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:48 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/559804881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:03:49.363 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 131.2 --await-for 45.0' 2024-09-19T08:03:49.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 131.2 --await-for 45.0 2024-09-19T08:03:49.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:49 smithi098 ceph-mon[31333]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:03:49.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:49 smithi162 ceph-mon[35260]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:03:49.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:49 smithi139 ceph-mon[34874]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T08:03:50.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:03:50.061 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:50.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.567+0000 7fdd540b9640 1 -- 172.21.15.98:0/3787413705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd4c103660 msgr2=0x7fdd4c10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:50.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.567+0000 7fdd540b9640 1 --2- 172.21.15.98:0/3787413705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd4c103660 0x7fdd4c10bb50 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009900 tx=0x7fdd3c02a480 comp rx=0 tx=0).stop 2024-09-19T08:03:50.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.567+0000 7fdd540b9640 1 -- 172.21.15.98:0/3787413705 shutdown_connections 2024-09-19T08:03:50.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.567+0000 7fdd540b9640 1 --2- 172.21.15.98:0/3787413705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd4c10c170 0x7fdd4c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.567+0000 7fdd540b9640 1 --2- 172.21.15.98:0/3787413705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd4c103660 0x7fdd4c10bb50 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.567+0000 7fdd540b9640 1 --2- 172.21.15.98:0/3787413705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd4c102d40 0x7fdd4c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.567+0000 7fdd540b9640 1 -- 172.21.15.98:0/3787413705 >> 172.21.15.98:0/3787413705 conn(0x7fdd4c0fc7a0 msgr2=0x7fdd4c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:50.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.567+0000 7fdd540b9640 1 -- 172.21.15.98:0/3787413705 shutdown_connections 2024-09-19T08:03:50.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.567+0000 7fdd540b9640 1 -- 172.21.15.98:0/3787413705 wait complete. 2024-09-19T08:03:50.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.568+0000 7fdd540b9640 1 Processor -- start 2024-09-19T08:03:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.568+0000 7fdd540b9640 1 -- start start 2024-09-19T08:03:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.568+0000 7fdd540b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd4c102d40 0x7fdd4c1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.568+0000 7fdd540b9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd4c10c170 0x7fdd4c100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.568+0000 7fdd540b9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd4c101420 0x7fdd4c1a6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.568+0000 7fdd540b9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c111f70 con 0x7fdd4c10c170 2024-09-19T08:03:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.568+0000 7fdd540b9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c111df0 con 0x7fdd4c101420 2024-09-19T08:03:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.568+0000 7fdd540b9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c1120f0 con 0x7fdd4c102d40 2024-09-19T08:03:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.569+0000 7fdd51654640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd4c10c170 0x7fdd4c100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.569+0000 7fdd51654640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd4c10c170 0x7fdd4c100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41396/0 (socket says 172.21.15.98:41396) 2024-09-19T08:03:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.569+0000 7fdd51654640 1 -- 172.21.15.98:0/3909424794 learned_addr learned my addr 172.21.15.98:0/3909424794 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.569+0000 7fdd51e55640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd4c102d40 0x7fdd4c1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.569+0000 7fdd52656640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd4c101420 0x7fdd4c1a6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.569+0000 7fdd51654640 1 -- 172.21.15.98:0/3909424794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd4c102d40 msgr2=0x7fdd4c1009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:50.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.569+0000 7fdd51654640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd4c102d40 0x7fdd4c1009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.569+0000 7fdd51654640 1 -- 172.21.15.98:0/3909424794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd4c101420 msgr2=0x7fdd4c1a6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:50.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.569+0000 7fdd51654640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd4c101420 0x7fdd4c1a6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.569+0000 7fdd51654640 1 -- 172.21.15.98:0/3909424794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4c1a6ee0 con 0x7fdd4c10c170 2024-09-19T08:03:50.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.569+0000 7fdd52656640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd4c101420 0x7fdd4c1a6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:50.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.569+0000 7fdd51654640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd4c10c170 0x7fdd4c100ee0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c005940 tx=0x7fdd3c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:50.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.570+0000 7fdd42ffd640 1 -- 172.21.15.98:0/3909424794 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c03f030 con 0x7fdd4c10c170 2024-09-19T08:03:50.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.570+0000 7fdd42ffd640 1 -- 172.21.15.98:0/3909424794 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd3c004440 con 0x7fdd4c10c170 2024-09-19T08:03:50.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.570+0000 7fdd540b9640 1 -- 172.21.15.98:0/3909424794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4c1a7170 con 0x7fdd4c10c170 2024-09-19T08:03:50.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.570+0000 7fdd42ffd640 1 -- 172.21.15.98:0/3909424794 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c035070 con 0x7fdd4c10c170 2024-09-19T08:03:50.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.570+0000 7fdd540b9640 1 -- 172.21.15.98:0/3909424794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd4c1a7590 con 0x7fdd4c10c170 2024-09-19T08:03:50.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.573+0000 7fdd42ffd640 1 -- 172.21.15.98:0/3909424794 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7fdd3c042030 con 0x7fdd4c10c170 2024-09-19T08:03:50.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.573+0000 7fdd56492740 1 Processor -- start 2024-09-19T08:03:50.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.574+0000 7fdd56492740 1 -- start start 2024-09-19T08:03:50.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.574+0000 7fdd56492740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a5e760 0x558e16a5eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.574+0000 7fdd42ffd640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdd380a3550 0x7fdd380a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.574+0000 7fdd56492740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5f080 0x558e16a69400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.574+0000 7fdd51e55640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdd380a3550 0x7fdd380a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.574+0000 7fdd42ffd640 1 -- 172.21.15.98:0/3909424794 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fdd3c0f0170 con 0x7fdd4c10c170 2024-09-19T08:03:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.574+0000 7fdd52656640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a5e760 0x558e16a5eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.574+0000 7fdd52656640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a5e760 0x558e16a5eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:34404/0 (socket says 172.21.15.98:34404) 2024-09-19T08:03:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.575+0000 7fdd56492740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a69940 0x558e16a6bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.575+0000 7fdd56492740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558e169c42a0 con 0x558e16a5f080 2024-09-19T08:03:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.575+0000 7fdd51654640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5f080 0x558e16a69400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.575+0000 7fdd51654640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5f080 0x558e16a69400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41410/0 (socket says 172.21.15.98:41410) 2024-09-19T08:03:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.575+0000 7fdd51654640 1 -- 172.21.15.98:0/2630987356 learned_addr learned my addr 172.21.15.98:0/2630987356 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.575+0000 7fdd51e55640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdd380a3550 0x7fdd380a5a10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800abe0 tx=0x7fdd4800a2d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.575+0000 7fdd56492740 1 -- 172.21.15.98:0/2630987356 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558e1696bb10 con 0x558e16a5e760 2024-09-19T08:03:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.575+0000 7fdd56492740 1 -- 172.21.15.98:0/2630987356 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558e169c0260 con 0x558e16a69940 2024-09-19T08:03:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.575+0000 7fdd51654640 1 -- 172.21.15.98:0/2630987356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a69940 msgr2=0x558e16a6bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.575+0000 7fdd51654640 1 --2- 172.21.15.98:0/2630987356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a69940 0x558e16a6bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.575+0000 7fdd51654640 1 -- 172.21.15.98:0/2630987356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a5e760 msgr2=0x558e16a5eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.575+0000 7fdd51654640 1 --2- 172.21.15.98:0/2630987356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a5e760 0x558e16a5eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.575+0000 7fdd51654640 1 -- 172.21.15.98:0/2630987356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e16735b90 con 0x558e16a5f080 2024-09-19T08:03:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.576+0000 7fdd51654640 1 --2- 172.21.15.98:0/2630987356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5f080 0x558e16a69400 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c000c00 tx=0x7fdd3c0bbde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dae128407cd4307 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.576+0000 7fdd37fff640 1 -- 172.21.15.98:0/2630987356 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c049070 con 0x558e16a5f080 2024-09-19T08:03:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.576+0000 7fdd37fff640 1 -- 172.21.15.98:0/2630987356 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd3c0bc9a0 con 0x558e16a5f080 2024-09-19T08:03:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.576+0000 7fdd37fff640 1 -- 172.21.15.98:0/2630987356 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c049070 con 0x558e16a5f080 2024-09-19T08:03:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.577+0000 7fdd56492740 1 -- 172.21.15.98:0/2630987356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5f080 msgr2=0x558e16a69400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.577+0000 7fdd56492740 1 --2- 172.21.15.98:0/2630987356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5f080 0x558e16a69400 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c000c00 tx=0x7fdd3c0bbde0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.577+0000 7fdd56492740 1 -- 172.21.15.98:0/2630987356 shutdown_connections 2024-09-19T08:03:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.577+0000 7fdd56492740 1 --2- 172.21.15.98:0/2630987356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a69940 0x558e16a6bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.577+0000 7fdd56492740 1 --2- 172.21.15.98:0/2630987356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5f080 0x558e16a69400 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.577+0000 7fdd56492740 1 --2- 172.21.15.98:0/2630987356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a5e760 0x558e16a5eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.577+0000 7fdd56492740 1 -- 172.21.15.98:0/2630987356 >> 172.21.15.98:0/2630987356 conn(0x558e16a5acf0 msgr2=0x558e16a5b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.577+0000 7fdd56492740 1 -- 172.21.15.98:0/2630987356 shutdown_connections 2024-09-19T08:03:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.578+0000 7fdd56492740 1 -- 172.21.15.98:0/2630987356 wait complete. 2024-09-19T08:03:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.578+0000 7fdd56492740 1 -- start start 2024-09-19T08:03:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.578+0000 7fdd56492740 1 Processor -- start 2024-09-19T08:03:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.579+0000 7fdd56492740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5e760 0x558e16a66870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.579+0000 7fdd56492740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5f080 0x558e16a66db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.579+0000 7fdd51e55640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5e760 0x558e16a66870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.579+0000 7fdd51654640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5f080 0x558e16a66db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.579+0000 7fdd51e55640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5e760 0x558e16a66870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:58402/0 (socket says 172.21.15.98:58402) 2024-09-19T08:03:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.579+0000 7fdd51654640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5f080 0x558e16a66db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41418/0 (socket says 172.21.15.98:41418) 2024-09-19T08:03:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.579+0000 7fdd51e55640 1 -- 172.21.15.98:0/1094744306 learned_addr learned my addr 172.21.15.98:0/1094744306 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.579+0000 7fdd56492740 1 --2- 172.21.15.98:0/1094744306 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a69940 0x558e16a672f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.579+0000 7fdd56492740 1 -- 172.21.15.98:0/1094744306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558e16a6c2d0 con 0x558e16a5f080 2024-09-19T08:03:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.579+0000 7fdd56492740 1 -- 172.21.15.98:0/1094744306 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558e1694c6e0 con 0x558e16a69940 2024-09-19T08:03:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.579+0000 7fdd56492740 1 -- 172.21.15.98:0/1094744306 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558e16a6c450 con 0x558e16a5e760 2024-09-19T08:03:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.580+0000 7fdd52656640 1 --2- 172.21.15.98:0/1094744306 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a69940 0x558e16a672f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.580+0000 7fdd51e55640 1 -- 172.21.15.98:0/1094744306 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a69940 msgr2=0x558e16a672f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.580+0000 7fdd51e55640 1 --2- 172.21.15.98:0/1094744306 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a69940 0x558e16a672f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.580+0000 7fdd51e55640 1 -- 172.21.15.98:0/1094744306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5f080 msgr2=0x558e16a66db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.580+0000 7fdd51e55640 1 --2- 172.21.15.98:0/1094744306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5f080 0x558e16a66db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.580+0000 7fdd51e55640 1 -- 172.21.15.98:0/1094744306 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e16a75700 con 0x558e16a5e760 2024-09-19T08:03:50.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.580+0000 7fdd51654640 1 --2- 172.21.15.98:0/1094744306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5f080 0x558e16a66db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:50.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.580+0000 7fdd56492740 10 client.0 _ll_register_callbacks cb 0x558e16970a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:50.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.580+0000 7fdd56492740 1 -- 172.21.15.98:0/1094744306 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd3c0bc9a0 con 0x558e16a5e760 2024-09-19T08:03:50.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.580+0000 7fdd357fa640 10 client.25084 ms_handle_connect on v2:172.21.15.162:3300/0 2024-09-19T08:03:50.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.580+0000 7fdd51e55640 1 --2- 172.21.15.98:0/1094744306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5e760 0x558e16a66870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdd48010e00 tx=0x7fdd48004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9307dd498684f61b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:50.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.581+0000 7fdd357fa640 1 -- 172.21.15.98:0/1094744306 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd48016070 con 0x558e16a5e760 2024-09-19T08:03:50.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.581+0000 7fdd357fa640 1 -- 172.21.15.98:0/1094744306 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fdd4800fc40 con 0x558e16a5e760 2024-09-19T08:03:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.581+0000 7fdd357fa640 1 -- 172.21.15.98:0/1094744306 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdd48016070 con 0x558e16a5e760 2024-09-19T08:03:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.581+0000 7fdd56492740 10 client.25084 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.581+0000 7fdd56492740 1 -- 172.21.15.98:0/1094744306 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({fsmap=28}) -- 0x558e16a77ef0 con 0x558e16a5e760 2024-09-19T08:03:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.582+0000 7fdd357fa640 1 -- 172.21.15.98:0/1094744306 <== mon.1 v2:172.21.15.162:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdd48031420 con 0x558e16a5e760 2024-09-19T08:03:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.582+0000 7fdd56492740 10 client.25084 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.582+0000 7fdd56492740 10 client.25084 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-19T08:03:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.582+0000 7fdd56492740 20 client.25084 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.582+0000 7fdd56492740 1 --2- 172.21.15.98:0/1094744306 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558e16a79240 0x558e16a7b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.582+0000 7fdd56492740 4 client.25084 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.582+0000 7fdd56492740 1 -- 172.21.15.98:0/1094744306 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e169d19d0 con 0x558e16a79240 2024-09-19T08:03:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.582+0000 7fdd51654640 1 --2- 172.21.15.98:0/1094744306 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558e16a79240 0x558e16a7b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.582+0000 7fdd357fa640 10 client.25084 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.582+0000 7fdd51654640 1 --2- 172.21.15.98:0/1094744306 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558e16a79240 0x558e16a7b6d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9c11ed9e4a03aed server_cookie=f9bc0cee6afd9abe in_seq=0 out_seq=0 2024-09-19T08:03:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.583+0000 7fdd357fa640 1 -- 172.21.15.98:0/1094744306 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558e169d19d0 con 0x558e16a79240 2024-09-19T08:03:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.583+0000 7fdd357fa640 10 client.25084 handle_command_reply: tid=0 2024-09-19T08:03:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.583+0000 7fdd56492740 1 client.25084 shutdown 2024-09-19T08:03:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.583+0000 7fdd56492740 1 -- 172.21.15.98:0/1094744306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5e760 msgr2=0x558e16a66870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.583+0000 7fdd56492740 1 --2- 172.21.15.98:0/1094744306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5e760 0x558e16a66870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdd48010e00 tx=0x7fdd48004730 comp rx=0 tx=0).stop 2024-09-19T08:03:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.584+0000 7fdd56492740 1 -- 172.21.15.98:0/1094744306 shutdown_connections 2024-09-19T08:03:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.584+0000 7fdd56492740 1 --2- 172.21.15.98:0/1094744306 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558e16a79240 0x558e16a7b6d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.584+0000 7fdd56492740 1 --2- 172.21.15.98:0/1094744306 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a69940 0x558e16a672f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.584+0000 7fdd56492740 1 --2- 172.21.15.98:0/1094744306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5f080 0x558e16a66db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.584+0000 7fdd56492740 1 --2- 172.21.15.98:0/1094744306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5e760 0x558e16a66870 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.584+0000 7fdd56492740 1 -- 172.21.15.98:0/1094744306 >> 172.21.15.98:0/1094744306 conn(0x558e16a5acf0 msgr2=0x558e16a59e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.584+0000 7fdd357fa640 0 client.25084 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.584+0000 7fdd56492740 1 -- 172.21.15.98:0/1094744306 shutdown_connections 2024-09-19T08:03:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.584+0000 7fdd56492740 1 -- 172.21.15.98:0/1094744306 wait complete. 2024-09-19T08:03:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.584+0000 7fdd56492740 20 client.25084 trim_cache size 0 max 16384 2024-09-19T08:03:50.614 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.617+0000 7fdd56492740 1 Processor -- start 2024-09-19T08:03:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.617+0000 7fdd56492740 1 -- start start 2024-09-19T08:03:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.617+0000 7fdd56492740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5e760 0x558e16a61210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.617+0000 7fdd56492740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5f080 0x558e16a59d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.617+0000 7fdd56492740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a69940 0x558e16a80fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.617+0000 7fdd56492740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c1b0180 con 0x558e16a5e760 2024-09-19T08:03:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.617+0000 7fdd56492740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c1b1cc0 con 0x558e16a69940 2024-09-19T08:03:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.617+0000 7fdd56492740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c1b0850 con 0x558e16a5f080 2024-09-19T08:03:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.617+0000 7fdd51e55640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5e760 0x558e16a61210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.617+0000 7fdd51e55640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5e760 0x558e16a61210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41430/0 (socket says 172.21.15.98:41430) 2024-09-19T08:03:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.617+0000 7fdd51e55640 1 -- 172.21.15.98:0/3611464803 learned_addr learned my addr 172.21.15.98:0/3611464803 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.617+0000 7fdd51654640 1 --2- 172.21.15.98:0/3611464803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5f080 0x558e16a59d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.617+0000 7fdd52656640 1 --2- 172.21.15.98:0/3611464803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a69940 0x558e16a80fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.618+0000 7fdd51e55640 1 -- 172.21.15.98:0/3611464803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5f080 msgr2=0x558e16a59d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.618+0000 7fdd51e55640 1 --2- 172.21.15.98:0/3611464803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5f080 0x558e16a59d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.618+0000 7fdd51e55640 1 -- 172.21.15.98:0/3611464803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a69940 msgr2=0x558e16a80fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.618+0000 7fdd51e55640 1 --2- 172.21.15.98:0/3611464803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a69940 0x558e16a80fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.618+0000 7fdd51e55640 1 -- 172.21.15.98:0/3611464803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e16a65880 con 0x558e16a5e760 2024-09-19T08:03:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.618+0000 7fdd51e55640 1 --2- 172.21.15.98:0/3611464803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5e760 0x558e16a61210 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800f8f0 tx=0x7fdd4801e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc980e37f33e268a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.618+0000 7fdd377fe640 1 -- 172.21.15.98:0/3611464803 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd48016070 con 0x558e16a5e760 2024-09-19T08:03:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.618+0000 7fdd377fe640 1 -- 172.21.15.98:0/3611464803 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd3c0bc9a0 con 0x558e16a5e760 2024-09-19T08:03:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.618+0000 7fdd377fe640 1 -- 172.21.15.98:0/3611464803 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd48030e00 con 0x558e16a5e760 2024-09-19T08:03:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.619+0000 7fdd56492740 1 -- 172.21.15.98:0/3611464803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5e760 msgr2=0x558e16a61210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.619+0000 7fdd56492740 1 --2- 172.21.15.98:0/3611464803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5e760 0x558e16a61210 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800f8f0 tx=0x7fdd4801e070 comp rx=0 tx=0).stop 2024-09-19T08:03:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.619+0000 7fdd56492740 1 -- 172.21.15.98:0/3611464803 shutdown_connections 2024-09-19T08:03:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.619+0000 7fdd56492740 1 --2- 172.21.15.98:0/3611464803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a69940 0x558e16a80fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.619+0000 7fdd56492740 1 --2- 172.21.15.98:0/3611464803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5f080 0x558e16a59d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.619+0000 7fdd56492740 1 --2- 172.21.15.98:0/3611464803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a5e760 0x558e16a61210 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.619+0000 7fdd56492740 1 -- 172.21.15.98:0/3611464803 >> 172.21.15.98:0/3611464803 conn(0x558e16a5acf0 msgr2=0x558e16a59650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.619+0000 7fdd56492740 1 -- 172.21.15.98:0/3611464803 shutdown_connections 2024-09-19T08:03:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.619+0000 7fdd56492740 1 -- 172.21.15.98:0/3611464803 wait complete. 2024-09-19T08:03:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.620+0000 7fdd56492740 1 -- start start 2024-09-19T08:03:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.620+0000 7fdd56492740 1 Processor -- start 2024-09-19T08:03:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.620+0000 7fdd56492740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a5e760 0x558e16a71b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.620+0000 7fdd56492740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5f080 0x558e16a6e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.620+0000 7fdd56492740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a69940 0x558e16a6eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.620+0000 7fdd56492740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c1b15f0 con 0x558e16a69940 2024-09-19T08:03:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.620+0000 7fdd56492740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c1b0f20 con 0x558e16a5e760 2024-09-19T08:03:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.620+0000 7fdd56492740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c1ae100 con 0x558e16a5f080 2024-09-19T08:03:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.620+0000 7fdd56492740 10 client.0 _ll_register_callbacks cb 0x7fdd48010ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.620+0000 7fdd52656640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a69940 0x558e16a6eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.620+0000 7fdd52656640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a69940 0x558e16a6eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41442/0 (socket says 172.21.15.98:41442) 2024-09-19T08:03:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.620+0000 7fdd52656640 1 -- 172.21.15.98:0/3769217468 learned_addr learned my addr 172.21.15.98:0/3769217468 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.621+0000 7fdd51654640 1 --2- 172.21.15.98:0/3769217468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5f080 0x558e16a6e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.621+0000 7fdd51e55640 1 --2- 172.21.15.98:0/3769217468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a5e760 0x558e16a71b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.621+0000 7fdd52656640 1 -- 172.21.15.98:0/3769217468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5f080 msgr2=0x558e16a6e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.621+0000 7fdd52656640 1 --2- 172.21.15.98:0/3769217468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5f080 0x558e16a6e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.621+0000 7fdd52656640 1 -- 172.21.15.98:0/3769217468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a5e760 msgr2=0x558e16a71b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.621+0000 7fdd52656640 1 --2- 172.21.15.98:0/3769217468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a5e760 0x558e16a71b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.621+0000 7fdd52656640 1 -- 172.21.15.98:0/3769217468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e16a6f150 con 0x558e16a69940 2024-09-19T08:03:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.621+0000 7fdd52656640 1 --2- 172.21.15.98:0/3769217468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a69940 0x558e16a6eae0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdd44011d10 tx=0x7fdd44010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfbeed91c0c58624 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.621+0000 7fdd357fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.621+0000 7fdd357fa640 1 -- 172.21.15.98:0/3769217468 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd44004d20 con 0x558e16a69940 2024-09-19T08:03:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.621+0000 7fdd357fa640 1 -- 172.21.15.98:0/3769217468 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fdd44005020 con 0x558e16a69940 2024-09-19T08:03:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.621+0000 7fdd56492740 1 -- 172.21.15.98:0/3769217468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd3c0bc9a0 con 0x558e16a69940 2024-09-19T08:03:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.622+0000 7fdd357fa640 1 -- 172.21.15.98:0/3769217468 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdd44005220 con 0x558e16a69940 2024-09-19T08:03:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.622+0000 7fdd56492740 10 client.15672 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.622+0000 7fdd56492740 1 -- 172.21.15.98:0/3769217468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558e16ae4e70 con 0x558e16a69940 2024-09-19T08:03:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.622+0000 7fdd357fa640 1 -- 172.21.15.98:0/3769217468 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd3c0bc9a0 con 0x558e16a69940 2024-09-19T08:03:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.622+0000 7fdd357fa640 1 -- 172.21.15.98:0/3769217468 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdd4401b070 con 0x558e16a69940 2024-09-19T08:03:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.622+0000 7fdd56492740 10 client.15672 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.622+0000 7fdd56492740 10 client.15672 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-19T08:03:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.622+0000 7fdd56492740 20 client.15672 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.622+0000 7fdd56492740 1 --2- 172.21.15.98:0/3769217468 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558e16ae5650 0x558e16ae7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.622+0000 7fdd56492740 4 client.15672 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 131.2, "await_for": 45.0}] 2024-09-19T08:03:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.622+0000 7fdd56492740 1 -- 172.21.15.98:0/3769217468 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 131.2, "await_for": 45.0}) -- 0x558e169cbd20 con 0x558e16ae5650 2024-09-19T08:03:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.623+0000 7fdd51e55640 1 --2- 172.21.15.98:0/3769217468 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558e16ae5650 0x558e16ae7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.623+0000 7fdd357fa640 10 client.15672 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.623+0000 7fdd51e55640 1 --2- 172.21.15.98:0/3769217468 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558e16ae5650 0x558e16ae7a70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bea7ae09ea8a3da5 server_cookie=c83a509cb81ec09c in_seq=0 out_seq=0 2024-09-19T08:03:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.635+0000 7fdd357fa640 1 -- 172.21.15.98:0/3769217468 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x558e169cbd20 con 0x558e16ae5650 2024-09-19T08:03:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.635+0000 7fdd357fa640 10 client.15672 handle_command_reply: tid=0 2024-09-19T08:03:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.635+0000 7fdd56492740 1 client.15672 shutdown 2024-09-19T08:03:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.636+0000 7fdd56492740 1 -- 172.21.15.98:0/3769217468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a69940 msgr2=0x558e16a6eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.636+0000 7fdd56492740 1 --2- 172.21.15.98:0/3769217468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a69940 0x558e16a6eae0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdd44011d10 tx=0x7fdd44010800 comp rx=0 tx=0).stop 2024-09-19T08:03:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.636+0000 7fdd56492740 1 -- 172.21.15.98:0/3769217468 shutdown_connections 2024-09-19T08:03:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.636+0000 7fdd56492740 1 --2- 172.21.15.98:0/3769217468 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558e16ae5650 0x558e16ae7a70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.636+0000 7fdd56492740 1 --2- 172.21.15.98:0/3769217468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e16a69940 0x558e16a6eae0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.636+0000 7fdd56492740 1 --2- 172.21.15.98:0/3769217468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e16a5f080 0x558e16a6e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.636+0000 7fdd56492740 1 --2- 172.21.15.98:0/3769217468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558e16a5e760 0x558e16a71b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.636+0000 7fdd56492740 1 -- 172.21.15.98:0/3769217468 >> 172.21.15.98:0/3769217468 conn(0x558e16a5acf0 msgr2=0x558e16a67050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.636+0000 7fdd357fa640 0 client.15672 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.636+0000 7fdd56492740 1 -- 172.21.15.98:0/3769217468 shutdown_connections 2024-09-19T08:03:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.636+0000 7fdd56492740 1 -- 172.21.15.98:0/3769217468 wait complete. 2024-09-19T08:03:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.636+0000 7fdd56492740 20 client.15672 trim_cache size 0 max 16384 2024-09-19T08:03:50.637 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:03:50.637 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 28, 2024-09-19T08:03:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24485, 2024-09-19T08:03:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 2, 2024-09-19T08:03:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-09-19T08:03:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "f1380945": { 2024-09-19T08:03:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 2, 2024-09-19T08:03:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-09-19T08:03:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-09-19T08:03:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-09-19T08:03:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-09-19T08:03:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:03:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 90.0, 2024-09-19T08:03:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 131.2, 2024-09-19T08:03:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-09-19T08:03:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-09-19T08:03:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-09-19T08:03:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-09-19T08:03:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-09-19T08:03:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-09-19T08:03:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:03:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:03:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:03:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:03:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:03:50.639 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:03:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.637+0000 7fdd540b9640 1 -- 172.21.15.98:0/3909424794 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdd380a3550 msgr2=0x7fdd380a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.637+0000 7fdd540b9640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdd380a3550 0x7fdd380a5a10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800abe0 tx=0x7fdd4800a2d0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.637+0000 7fdd540b9640 1 -- 172.21.15.98:0/3909424794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd4c10c170 msgr2=0x7fdd4c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.637+0000 7fdd540b9640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd4c10c170 0x7fdd4c100ee0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c005940 tx=0x7fdd3c004040 comp rx=0 tx=0).stop 2024-09-19T08:03:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.637+0000 7fdd540b9640 1 -- 172.21.15.98:0/3909424794 shutdown_connections 2024-09-19T08:03:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.637+0000 7fdd540b9640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdd380a3550 0x7fdd380a5a10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.637+0000 7fdd540b9640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd4c101420 0x7fdd4c1a6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.637+0000 7fdd540b9640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd4c10c170 0x7fdd4c100ee0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.637+0000 7fdd540b9640 1 --2- 172.21.15.98:0/3909424794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd4c102d40 0x7fdd4c1009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.637+0000 7fdd540b9640 1 -- 172.21.15.98:0/3909424794 >> 172.21.15.98:0/3909424794 conn(0x7fdd4c0fc7a0 msgr2=0x7fdd4c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.637+0000 7fdd540b9640 1 -- 172.21.15.98:0/3909424794 shutdown_connections 2024-09-19T08:03:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:50.637+0000 7fdd540b9640 1 -- 172.21.15.98:0/3909424794 wait complete. 2024-09-19T08:03:51.106 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:50 smithi098 ceph-mon[31333]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:03:51.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:50 smithi162 ceph-mon[35260]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:03:51.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:50 smithi139 ceph-mon[34874]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:03:51.582 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'f1380945', quiesce took 4.2 seconds. Will release after: 6.999999999999999 2024-09-19T08:03:51.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.767+0000 7f25c0547640 1 -- 172.21.15.98:0/2534273513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b8101e10 msgr2=0x7f25b81021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:51.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.767+0000 7f25c0547640 1 --2- 172.21.15.98:0/2534273513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b8101e10 0x7f25b81021f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f25b4009900 tx=0x7f25b402a410 comp rx=0 tx=0).stop 2024-09-19T08:03:51.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.767+0000 7f25c0547640 1 -- 172.21.15.98:0/2534273513 shutdown_connections 2024-09-19T08:03:51.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.767+0000 7f25c0547640 1 --2- 172.21.15.98:0/2534273513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25b8106f40 0x7f25b810eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.767+0000 7f25c0547640 1 --2- 172.21.15.98:0/2534273513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25b81027c0 0x7f25b8106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.767+0000 7f25c0547640 1 --2- 172.21.15.98:0/2534273513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b8101e10 0x7f25b81021f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.767+0000 7f25c0547640 1 -- 172.21.15.98:0/2534273513 >> 172.21.15.98:0/2534273513 conn(0x7f25b80fd8d0 msgr2=0x7f25b80ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:51.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.767+0000 7f25c0547640 1 -- 172.21.15.98:0/2534273513 shutdown_connections 2024-09-19T08:03:51.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.767+0000 7f25c0547640 1 -- 172.21.15.98:0/2534273513 wait complete. 2024-09-19T08:03:51.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.768+0000 7f25c0547640 1 Processor -- start 2024-09-19T08:03:51.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.768+0000 7f25c0547640 1 -- start start 2024-09-19T08:03:51.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.768+0000 7f25c0547640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b8101e10 0x7f25b8198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.768+0000 7f25c0547640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25b81027c0 0x7f25b8198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.768+0000 7f25c0547640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25b8106f40 0x7f25b819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.768+0000 7f25c0547640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25b8111fa0 con 0x7f25b8101e10 2024-09-19T08:03:51.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.768+0000 7f25c0547640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f25b8111e20 con 0x7f25b81027c0 2024-09-19T08:03:51.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.768+0000 7f25c0547640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25b8112120 con 0x7f25b8106f40 2024-09-19T08:03:51.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.768+0000 7f25be2e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b8101e10 0x7f25b8198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.768+0000 7f25be2e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b8101e10 0x7f25b8198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41458/0 (socket says 172.21.15.98:41458) 2024-09-19T08:03:51.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.768+0000 7f25be2e3640 1 -- 172.21.15.98:0/2166908041 learned_addr learned my addr 172.21.15.98:0/2166908041 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:51.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.768+0000 7f25bdae2640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25b81027c0 0x7f25b8198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.769+0000 7f25beae4640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25b8106f40 0x7f25b819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.769+0000 7f25be2e3640 1 -- 172.21.15.98:0/2166908041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25b8106f40 msgr2=0x7f25b819cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:51.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.769+0000 7f25be2e3640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25b8106f40 0x7f25b819cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.769+0000 7f25be2e3640 1 -- 172.21.15.98:0/2166908041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25b81027c0 msgr2=0x7f25b8198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:51.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.769+0000 7f25be2e3640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25b81027c0 0x7f25b8198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.769+0000 7f25be2e3640 1 -- 172.21.15.98:0/2166908041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b819d3c0 con 0x7f25b8101e10 2024-09-19T08:03:51.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.769+0000 7f25beae4640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25b8106f40 0x7f25b819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:51.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.769+0000 7f25bdae2640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25b81027c0 0x7f25b8198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:51.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.769+0000 7f25be2e3640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b8101e10 0x7f25b8198410 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f25b4006010 tx=0x7f25b40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:51.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.769+0000 7f25af7fe640 1 -- 172.21.15.98:0/2166908041 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b403f030 con 0x7f25b8101e10 2024-09-19T08:03:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.769+0000 7f25c0547640 1 -- 172.21.15.98:0/2166908041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b819d650 con 0x7f25b8101e10 2024-09-19T08:03:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.769+0000 7f25af7fe640 1 -- 172.21.15.98:0/2166908041 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25b4003ce0 con 0x7f25b8101e10 2024-09-19T08:03:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.769+0000 7f25af7fe640 1 -- 172.21.15.98:0/2166908041 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b4035070 con 0x7f25b8101e10 2024-09-19T08:03:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.769+0000 7f25c0547640 1 -- 172.21.15.98:0/2166908041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b81a4f90 con 0x7f25b8101e10 2024-09-19T08:03:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.772+0000 7f25c2920740 1 Processor -- start 2024-09-19T08:03:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.772+0000 7f25af7fe640 1 -- 172.21.15.98:0/2166908041 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f25b4042030 con 0x7f25b8101e10 2024-09-19T08:03:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.772+0000 7f25c2920740 1 -- start start 2024-09-19T08:03:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.772+0000 7f25c2920740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e28eb40 0x560e5e28d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.772+0000 7f25c2920740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28d5d0 0x560e5e28da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.772+0000 7f25c2920740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e298370 0x560e5e29a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.772+0000 7f25c2920740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560e5e1f32f0 con 0x560e5e298370 2024-09-19T08:03:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.772+0000 7f25c2920740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560e5e19acf0 con 0x560e5e28d5d0 2024-09-19T08:03:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.772+0000 7f25c2920740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560e5e1ef2b0 con 0x560e5e28eb40 2024-09-19T08:03:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25bdae2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e298370 0x560e5e29a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25bdae2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e298370 0x560e5e29a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41470/0 (socket says 172.21.15.98:41470) 2024-09-19T08:03:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25bdae2640 1 -- 172.21.15.98:0/2981948424 learned_addr learned my addr 172.21.15.98:0/2981948424 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25beae4640 1 --2- 172.21.15.98:0/2981948424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e28eb40 0x560e5e28d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25be2e3640 1 --2- 172.21.15.98:0/2981948424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28d5d0 0x560e5e28da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25af7fe640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f25a40a34d0 0x7f25a40a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25bdae2640 1 -- 172.21.15.98:0/2981948424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e28eb40 msgr2=0x560e5e28d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25bdae2640 1 --2- 172.21.15.98:0/2981948424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e28eb40 0x560e5e28d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25bdae2640 1 -- 172.21.15.98:0/2981948424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28d5d0 msgr2=0x560e5e28da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25bdae2640 1 --2- 172.21.15.98:0/2981948424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28d5d0 0x560e5e28da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25bdae2640 1 -- 172.21.15.98:0/2981948424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e5e17f3a0 con 0x560e5e298370 2024-09-19T08:03:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25be2e3640 1 --2- 172.21.15.98:0/2981948424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28d5d0 0x560e5e28da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25beae4640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f25a40a34d0 0x7f25a40a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25af7fe640 1 -- 172.21.15.98:0/2166908041 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f25b40f04b0 con 0x7f25b8101e10 2024-09-19T08:03:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25bdae2640 1 --2- 172.21.15.98:0/2981948424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e298370 0x560e5e29a760 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f25a80114f0 tx=0x7f25a80119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb170a13bed4a76a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.773+0000 7f25acff9640 1 -- 172.21.15.98:0/2981948424 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a801e070 con 0x560e5e298370 2024-09-19T08:03:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.774+0000 7f25acff9640 1 -- 172.21.15.98:0/2981948424 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25a80035f0 con 0x560e5e298370 2024-09-19T08:03:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.774+0000 7f25acff9640 1 -- 172.21.15.98:0/2981948424 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a800f7c0 con 0x560e5e298370 2024-09-19T08:03:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.774+0000 7f25beae4640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f25a40a34d0 0x7f25a40a5990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f25b0005fd0 tx=0x7f25b0004a40 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.774+0000 7f25c2920740 1 -- 172.21.15.98:0/2981948424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e298370 msgr2=0x560e5e29a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.774+0000 7f25c2920740 1 --2- 172.21.15.98:0/2981948424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e298370 0x560e5e29a760 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f25a80114f0 tx=0x7f25a80119b0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.774+0000 7f25c2920740 1 -- 172.21.15.98:0/2981948424 shutdown_connections 2024-09-19T08:03:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.774+0000 7f25c2920740 1 --2- 172.21.15.98:0/2981948424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e298370 0x560e5e29a760 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.774+0000 7f25c2920740 1 --2- 172.21.15.98:0/2981948424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28d5d0 0x560e5e28da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.774+0000 7f25c2920740 1 --2- 172.21.15.98:0/2981948424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e28eb40 0x560e5e28d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.774+0000 7f25c2920740 1 -- 172.21.15.98:0/2981948424 >> 172.21.15.98:0/2981948424 conn(0x560e5e256310 msgr2=0x560e5e289c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25c2920740 1 -- 172.21.15.98:0/2981948424 shutdown_connections 2024-09-19T08:03:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25c2920740 1 -- 172.21.15.98:0/2981948424 wait complete. 2024-09-19T08:03:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25c2920740 1 -- start start 2024-09-19T08:03:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25c2920740 1 Processor -- start 2024-09-19T08:03:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25c2920740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28d5d0 0x560e5e297d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25c2920740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28eb40 0x560e5e294fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25c2920740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e298370 0x560e5e295570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25c2920740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560e5e29b070 con 0x560e5e28d5d0 2024-09-19T08:03:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25c2920740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560e5e1ae060 con 0x560e5e28eb40 2024-09-19T08:03:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25c2920740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560e5e29b1f0 con 0x560e5e298370 2024-09-19T08:03:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25c2920740 10 client.0 _ll_register_callbacks cb 0x560e5df018c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25be2e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28d5d0 0x560e5e297d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25beae4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e298370 0x560e5e295570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25be2e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28d5d0 0x560e5e297d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41482/0 (socket says 172.21.15.98:41482) 2024-09-19T08:03:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.775+0000 7f25be2e3640 1 -- 172.21.15.98:0/754650935 learned_addr learned my addr 172.21.15.98:0/754650935 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25bdae2640 1 --2- 172.21.15.98:0/754650935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28eb40 0x560e5e294fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25be2e3640 1 -- 172.21.15.98:0/754650935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e298370 msgr2=0x560e5e295570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25be2e3640 1 --2- 172.21.15.98:0/754650935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e298370 0x560e5e295570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25be2e3640 1 -- 172.21.15.98:0/754650935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28eb40 msgr2=0x560e5e294fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25be2e3640 1 --2- 172.21.15.98:0/754650935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28eb40 0x560e5e294fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25be2e3640 1 -- 172.21.15.98:0/754650935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e5df643c0 con 0x560e5e28d5d0 2024-09-19T08:03:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25bdae2640 1 --2- 172.21.15.98:0/754650935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28eb40 0x560e5e294fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25be2e3640 1 --2- 172.21.15.98:0/754650935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28d5d0 0x560e5e297d20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f25b40b8b60 tx=0x7f25b40b8ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1551df510f26f08 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25a1ffb640 1 -- 172.21.15.98:0/754650935 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b4049070 con 0x560e5e28d5d0 2024-09-19T08:03:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25a1ffb640 1 -- 172.21.15.98:0/754650935 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f25b40bc960 con 0x560e5e28d5d0 2024-09-19T08:03:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25c2920740 1 -- 172.21.15.98:0/754650935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25a80035f0 con 0x560e5e28d5d0 2024-09-19T08:03:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25a1ffb640 1 -- 172.21.15.98:0/754650935 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f25b40bc4d0 con 0x560e5e28d5d0 2024-09-19T08:03:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25c2920740 10 client.15684 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.776+0000 7f25c2920740 1 -- 172.21.15.98:0/754650935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560e5e2a6ac0 con 0x560e5e28d5d0 2024-09-19T08:03:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.777+0000 7f25a1ffb640 1 -- 172.21.15.98:0/754650935 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25a80035f0 con 0x560e5e28d5d0 2024-09-19T08:03:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.777+0000 7f25a1ffb640 1 -- 172.21.15.98:0/754650935 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f25b40c4ee0 con 0x560e5e28d5d0 2024-09-19T08:03:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.777+0000 7f25c2920740 10 client.15684 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.777+0000 7f25c2920740 10 client.15684 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:03:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.777+0000 7f25c2920740 20 client.15684 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.777+0000 7f25c2920740 1 --2- 172.21.15.98:0/754650935 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560e5e2a7e10 0x560e5e2aa220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.777+0000 7f25c2920740 4 client.15684 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.777+0000 7f25c2920740 1 -- 172.21.15.98:0/754650935 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e5e200860 con 0x560e5e2a7e10 2024-09-19T08:03:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.777+0000 7f25bdae2640 1 --2- 172.21.15.98:0/754650935 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560e5e2a7e10 0x560e5e2aa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.778+0000 7f25bdae2640 1 --2- 172.21.15.98:0/754650935 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560e5e2a7e10 0x560e5e2aa220 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cad1fb249439a695 server_cookie=523268b68e29032a in_seq=0 out_seq=0 2024-09-19T08:03:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.778+0000 7f25a1ffb640 10 client.15684 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:03:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.778+0000 7f25a1ffb640 1 -- 172.21.15.98:0/754650935 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560e5e200860 con 0x560e5e2a7e10 2024-09-19T08:03:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.778+0000 7f25a1ffb640 10 client.15684 handle_command_reply: tid=0 2024-09-19T08:03:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.778+0000 7f25c2920740 1 client.15684 shutdown 2024-09-19T08:03:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.779+0000 7f25c2920740 1 -- 172.21.15.98:0/754650935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28d5d0 msgr2=0x560e5e297d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.779+0000 7f25c2920740 1 --2- 172.21.15.98:0/754650935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28d5d0 0x560e5e297d20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f25b40b8b60 tx=0x7f25b40b8ad0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.779+0000 7f25c2920740 1 -- 172.21.15.98:0/754650935 shutdown_connections 2024-09-19T08:03:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.779+0000 7f25c2920740 1 --2- 172.21.15.98:0/754650935 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560e5e2a7e10 0x560e5e2aa220 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.779+0000 7f25c2920740 1 --2- 172.21.15.98:0/754650935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e298370 0x560e5e295570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.779+0000 7f25c2920740 1 --2- 172.21.15.98:0/754650935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28eb40 0x560e5e294fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.779+0000 7f25a1ffb640 0 client.15684 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:03:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.779+0000 7f25c2920740 1 --2- 172.21.15.98:0/754650935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28d5d0 0x560e5e297d20 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.779+0000 7f25c2920740 1 -- 172.21.15.98:0/754650935 >> 172.21.15.98:0/754650935 conn(0x560e5e256310 msgr2=0x560e5e28f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.779+0000 7f25c2920740 1 -- 172.21.15.98:0/754650935 shutdown_connections 2024-09-19T08:03:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.779+0000 7f25c2920740 1 -- 172.21.15.98:0/754650935 wait complete. 2024-09-19T08:03:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.779+0000 7f25c2920740 20 client.15684 trim_cache size 0 max 16384 2024-09-19T08:03:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.792+0000 7f25c2920740 1 Processor -- start 2024-09-19T08:03:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25c2920740 1 -- start start 2024-09-19T08:03:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25c2920740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28d5d0 0x560e5e2a4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25c2920740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28eb40 0x560e5e288bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25c2920740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e298370 0x560e5e289130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25c2920740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25b81ad7d0 con 0x560e5e28eb40 2024-09-19T08:03:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25c2920740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f25b81af310 con 0x560e5e28d5d0 2024-09-19T08:03:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25c2920740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25b81adea0 con 0x560e5e298370 2024-09-19T08:03:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25bdae2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28eb40 0x560e5e288bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25bdae2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28eb40 0x560e5e288bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41492/0 (socket says 172.21.15.98:41492) 2024-09-19T08:03:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25bdae2640 1 -- 172.21.15.98:0/659242195 learned_addr learned my addr 172.21.15.98:0/659242195 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25be2e3640 1 --2- 172.21.15.98:0/659242195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28d5d0 0x560e5e2a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25beae4640 1 --2- 172.21.15.98:0/659242195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e298370 0x560e5e289130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25bdae2640 1 -- 172.21.15.98:0/659242195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e298370 msgr2=0x560e5e289130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25bdae2640 1 --2- 172.21.15.98:0/659242195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e298370 0x560e5e289130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25bdae2640 1 -- 172.21.15.98:0/659242195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28d5d0 msgr2=0x560e5e2a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25bdae2640 1 --2- 172.21.15.98:0/659242195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28d5d0 0x560e5e2a4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25bdae2640 1 -- 172.21.15.98:0/659242195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e5e298090 con 0x560e5e28eb40 2024-09-19T08:03:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25beae4640 1 --2- 172.21.15.98:0/659242195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e298370 0x560e5e289130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.793+0000 7f25be2e3640 1 --2- 172.21.15.98:0/659242195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28d5d0 0x560e5e2a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.794+0000 7f25bdae2640 1 --2- 172.21.15.98:0/659242195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28eb40 0x560e5e288bf0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x560e5e28e070 tx=0x7f25a8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d4f5ada9470c136 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.794+0000 7f25a3fff640 1 -- 172.21.15.98:0/659242195 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a801e040 con 0x560e5e28eb40 2024-09-19T08:03:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.794+0000 7f25a3fff640 1 -- 172.21.15.98:0/659242195 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25a8005520 con 0x560e5e28eb40 2024-09-19T08:03:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.794+0000 7f25a3fff640 1 -- 172.21.15.98:0/659242195 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a8004d10 con 0x560e5e28eb40 2024-09-19T08:03:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 -- 172.21.15.98:0/659242195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28eb40 msgr2=0x560e5e288bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 --2- 172.21.15.98:0/659242195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28eb40 0x560e5e288bf0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x560e5e28e070 tx=0x7f25a8000bc0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 -- 172.21.15.98:0/659242195 shutdown_connections 2024-09-19T08:03:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 --2- 172.21.15.98:0/659242195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e298370 0x560e5e289130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 --2- 172.21.15.98:0/659242195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28eb40 0x560e5e288bf0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 --2- 172.21.15.98:0/659242195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e28d5d0 0x560e5e2a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 -- 172.21.15.98:0/659242195 >> 172.21.15.98:0/659242195 conn(0x560e5e256310 msgr2=0x560e5e2884e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 -- 172.21.15.98:0/659242195 shutdown_connections 2024-09-19T08:03:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 -- 172.21.15.98:0/659242195 wait complete. 2024-09-19T08:03:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 -- start start 2024-09-19T08:03:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 Processor -- start 2024-09-19T08:03:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28d5d0 0x560e5e2a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e28eb40 0x560e5e2a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e298370 0x560e5e29cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25b81aec40 con 0x560e5e28d5d0 2024-09-19T08:03:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f25b81ae570 con 0x560e5e298370 2024-09-19T08:03:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.795+0000 7f25c2920740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25b81ab7b0 con 0x560e5e28eb40 2024-09-19T08:03:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25c2920740 10 client.0 _ll_register_callbacks cb 0x7f25b4005750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25be2e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28d5d0 0x560e5e2a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25bdae2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e28eb40 0x560e5e2a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25bdae2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e28eb40 0x560e5e2a0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:58478/0 (socket says 172.21.15.98:58478) 2024-09-19T08:03:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25be2e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28d5d0 0x560e5e2a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41506/0 (socket says 172.21.15.98:41506) 2024-09-19T08:03:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25bdae2640 1 -- 172.21.15.98:0/3487307266 learned_addr learned my addr 172.21.15.98:0/3487307266 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25beae4640 1 --2- 172.21.15.98:0/3487307266 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e298370 0x560e5e29cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25be2e3640 1 -- 172.21.15.98:0/3487307266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e28eb40 msgr2=0x560e5e2a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25be2e3640 1 --2- 172.21.15.98:0/3487307266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e28eb40 0x560e5e2a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25be2e3640 1 -- 172.21.15.98:0/3487307266 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e298370 msgr2=0x560e5e29cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25be2e3640 1 --2- 172.21.15.98:0/3487307266 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e298370 0x560e5e29cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25be2e3640 1 -- 172.21.15.98:0/3487307266 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e5e29d5b0 con 0x560e5e28d5d0 2024-09-19T08:03:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25be2e3640 1 --2- 172.21.15.98:0/3487307266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28d5d0 0x560e5e2a07f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f25b40ee230 tx=0x7f25b40bcb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6cabb4b3495b42 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25a1ffb640 1 -- 172.21.15.98:0/3487307266 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b4049040 con 0x560e5e28d5d0 2024-09-19T08:03:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.796+0000 7f25a1ffb640 1 -- 172.21.15.98:0/3487307266 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f25b403bdb0 con 0x560e5e28d5d0 2024-09-19T08:03:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.797+0000 7f25c2920740 1 -- 172.21.15.98:0/3487307266 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25a8005520 con 0x560e5e28d5d0 2024-09-19T08:03:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.797+0000 7f25a1ffb640 1 -- 172.21.15.98:0/3487307266 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f25b40044e0 con 0x560e5e28d5d0 2024-09-19T08:03:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.797+0000 7f25c2920740 10 client.15690 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.797+0000 7f25c2920740 1 -- 172.21.15.98:0/3487307266 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560e5e3136b0 con 0x560e5e28d5d0 2024-09-19T08:03:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.797+0000 7f25a1ffb640 1 -- 172.21.15.98:0/3487307266 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25b40a8080 con 0x560e5e28d5d0 2024-09-19T08:03:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.797+0000 7f25a1ffb640 1 -- 172.21.15.98:0/3487307266 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f25b40edd90 con 0x560e5e28d5d0 2024-09-19T08:03:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.797+0000 7f25c2920740 10 client.15690 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.797+0000 7f25c2920740 10 client.15690 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:03:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.797+0000 7f25c2920740 20 client.15690 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.797+0000 7f25c2920740 1 --2- 172.21.15.98:0/3487307266 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560e5e313c90 0x560e5e316080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.797+0000 7f25c2920740 4 client.15690 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:03:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.797+0000 7f25c2920740 1 -- 172.21.15.98:0/3487307266 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f25a8005520 con 0x560e5e313c90 2024-09-19T08:03:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.798+0000 7f25bdae2640 1 --2- 172.21.15.98:0/3487307266 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560e5e313c90 0x560e5e316080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.798+0000 7f25bdae2640 1 --2- 172.21.15.98:0/3487307266 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560e5e313c90 0x560e5e316080 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d3093f473c114a21 server_cookie=d515fbf9584eedaf in_seq=0 out_seq=0 2024-09-19T08:03:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.798+0000 7f25a1ffb640 10 client.15690 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:03:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.798+0000 7f25a1ffb640 1 -- 172.21.15.98:0/3487307266 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f25a8005520 con 0x560e5e313c90 2024-09-19T08:03:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.798+0000 7f25a1ffb640 10 client.15690 handle_command_reply: tid=0 2024-09-19T08:03:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.798+0000 7f25c2920740 1 client.15690 shutdown 2024-09-19T08:03:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.798+0000 7f25c2920740 1 -- 172.21.15.98:0/3487307266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28d5d0 msgr2=0x560e5e2a07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.798+0000 7f25c2920740 1 --2- 172.21.15.98:0/3487307266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28d5d0 0x560e5e2a07f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f25b40ee230 tx=0x7f25b40bcb90 comp rx=0 tx=0).stop 2024-09-19T08:03:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c2920740 1 -- 172.21.15.98:0/3487307266 shutdown_connections 2024-09-19T08:03:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c2920740 1 --2- 172.21.15.98:0/3487307266 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560e5e313c90 0x560e5e316080 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c2920740 1 --2- 172.21.15.98:0/3487307266 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e5e298370 0x560e5e29cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c2920740 1 --2- 172.21.15.98:0/3487307266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e5e28eb40 0x560e5e2a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c2920740 1 --2- 172.21.15.98:0/3487307266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e5e28d5d0 0x560e5e2a07f0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c2920740 1 -- 172.21.15.98:0/3487307266 >> 172.21.15.98:0/3487307266 conn(0x560e5e256310 msgr2=0x560e5e2884e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25a1ffb640 0 client.15690 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:03:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c2920740 1 -- 172.21.15.98:0/3487307266 shutdown_connections 2024-09-19T08:03:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c2920740 1 -- 172.21.15.98:0/3487307266 wait complete. 2024-09-19T08:03:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c2920740 20 client.15690 trim_cache size 0 max 16384 2024-09-19T08:03:51.817 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:03:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c0547640 1 -- 172.21.15.98:0/2166908041 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f25a40a34d0 msgr2=0x7f25a40a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c0547640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f25a40a34d0 0x7f25a40a5990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f25b0005fd0 tx=0x7f25b0004a40 comp rx=0 tx=0).stop 2024-09-19T08:03:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c0547640 1 -- 172.21.15.98:0/2166908041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b8101e10 msgr2=0x7f25b8198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c0547640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b8101e10 0x7f25b8198410 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f25b4006010 tx=0x7f25b40042f0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c0547640 1 -- 172.21.15.98:0/2166908041 shutdown_connections 2024-09-19T08:03:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.799+0000 7f25c0547640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f25a40a34d0 0x7f25a40a5990 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.800+0000 7f25c0547640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25b8106f40 0x7f25b819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.800+0000 7f25c0547640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25b81027c0 0x7f25b8198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.800+0000 7f25c0547640 1 --2- 172.21.15.98:0/2166908041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b8101e10 0x7f25b8198410 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.800+0000 7f25c0547640 1 -- 172.21.15.98:0/2166908041 >> 172.21.15.98:0/2166908041 conn(0x7f25b80fd8d0 msgr2=0x7f25b80fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.800+0000 7f25c0547640 1 -- 172.21.15.98:0/2166908041 shutdown_connections 2024-09-19T08:03:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:51.800+0000 7f25c0547640 1 -- 172.21.15.98:0/2166908041 wait complete. 2024-09-19T08:03:52.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:03:52.745 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:53.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:52 smithi162 ceph-mon[35260]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-09-19T08:03:53.169 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:52 smithi098 ceph-mon[31333]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-09-19T08:03:53.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:52 smithi139 ceph-mon[34874]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-09-19T08:03:53.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.254+0000 7fce1b8df640 1 -- 172.21.15.98:0/4022235391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce14108290 msgr2=0x7fce1410ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:53.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.254+0000 7fce1b8df640 1 --2- 172.21.15.98:0/4022235391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce14108290 0x7fce1410ec30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fce0c009cd0 tx=0x7fce0c02a8e0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 -- 172.21.15.98:0/4022235391 shutdown_connections 2024-09-19T08:03:53.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 --2- 172.21.15.98:0/4022235391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce14108290 0x7fce1410ec30 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 --2- 172.21.15.98:0/4022235391 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce14103740 0x7fce14107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 --2- 172.21.15.98:0/4022235391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce14102d90 0x7fce14103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 -- 172.21.15.98:0/4022235391 >> 172.21.15.98:0/4022235391 conn(0x7fce140fc7d0 msgr2=0x7fce140febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:53.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 -- 172.21.15.98:0/4022235391 shutdown_connections 2024-09-19T08:03:53.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 -- 172.21.15.98:0/4022235391 wait complete. 2024-09-19T08:03:53.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 Processor -- start 2024-09-19T08:03:53.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 -- start start 2024-09-19T08:03:53.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce14102d90 0x7fce141985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce14103740 0x7fce14198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce14108290 0x7fce1419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fce141120f0 con 0x7fce14102d90 2024-09-19T08:03:53.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fce14111f70 con 0x7fce14103740 2024-09-19T08:03:53.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.255+0000 7fce1b8df640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce14112270 con 0x7fce14108290 2024-09-19T08:03:53.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce1967b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce14102d90 0x7fce141985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce1967b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce14102d90 0x7fce141985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41526/0 (socket says 172.21.15.98:41526) 2024-09-19T08:03:53.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce1967b640 1 -- 172.21.15.98:0/1621684379 learned_addr learned my addr 172.21.15.98:0/1621684379 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:53.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce18e7a640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce14103740 0x7fce14198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce19e7c640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce14108290 0x7fce1419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce1967b640 1 -- 172.21.15.98:0/1621684379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce14108290 msgr2=0x7fce1419cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:53.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce1967b640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce14108290 0x7fce1419cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce1967b640 1 -- 172.21.15.98:0/1621684379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce14103740 msgr2=0x7fce14198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:53.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce1967b640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce14103740 0x7fce14198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce1967b640 1 -- 172.21.15.98:0/1621684379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce1419d580 con 0x7fce14102d90 2024-09-19T08:03:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce18e7a640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce14103740 0x7fce14198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce19e7c640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce14108290 0x7fce1419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce1967b640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce14102d90 0x7fce141985d0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fce1000c970 tx=0x7fce1000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce0a7fc640 1 -- 172.21.15.98:0/1621684379 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce10007bb0 con 0x7fce14102d90 2024-09-19T08:03:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce1b8df640 1 -- 172.21.15.98:0/1621684379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce1419d870 con 0x7fce14102d90 2024-09-19T08:03:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce0a7fc640 1 -- 172.21.15.98:0/1621684379 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce10007d50 con 0x7fce14102d90 2024-09-19T08:03:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce0a7fc640 1 -- 172.21.15.98:0/1621684379 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1000f450 con 0x7fce14102d90 2024-09-19T08:03:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.256+0000 7fce1b8df640 1 -- 172.21.15.98:0/1621684379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce141a5150 con 0x7fce14102d90 2024-09-19T08:03:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.259+0000 7fce0a7fc640 1 -- 172.21.15.98:0/1621684379 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7fce10005020 con 0x7fce14102d90 2024-09-19T08:03:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.259+0000 7fce1dcb8740 1 Processor -- start 2024-09-19T08:03:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.259+0000 7fce1dcb8740 1 -- start start 2024-09-19T08:03:53.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.259+0000 7fce0a7fc640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fce000a34d0 0x7fce000a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.259+0000 7fce1dcb8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffc5b40 0x55d9fffc4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce1dcb8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc45d0 0x55d9fffc4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce1dcb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffd3780 0x55d9fffd5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce1dcb8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d9fff2a2f0 con 0x55d9fffd3780 2024-09-19T08:03:53.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce1dcb8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d9ffed1cf0 con 0x55d9fffc45d0 2024-09-19T08:03:53.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce1dcb8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d9fff262b0 con 0x55d9fffc5b40 2024-09-19T08:03:53.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce19e7c640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fce000a34d0 0x7fce000a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce0a7fc640 1 -- 172.21.15.98:0/1621684379 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fce100cfe30 con 0x7fce14102d90 2024-09-19T08:03:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce19e7c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffd3780 0x55d9fffd5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce19e7c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffd3780 0x55d9fffd5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41534/0 (socket says 172.21.15.98:41534) 2024-09-19T08:03:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce19e7c640 1 -- 172.21.15.98:0/280851625 learned_addr learned my addr 172.21.15.98:0/280851625 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce1967b640 1 --2- 172.21.15.98:0/280851625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffc5b40 0x55d9fffc4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce18e7a640 1 --2- 172.21.15.98:0/280851625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc45d0 0x55d9fffc4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce1967b640 1 -- 172.21.15.98:0/280851625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc45d0 msgr2=0x55d9fffc4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce1967b640 1 --2- 172.21.15.98:0/280851625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc45d0 0x55d9fffc4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce1967b640 1 -- 172.21.15.98:0/280851625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffd3780 msgr2=0x55d9fffd5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce1967b640 1 --2- 172.21.15.98:0/280851625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffd3780 0x55d9fffd5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce1967b640 1 -- 172.21.15.98:0/280851625 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9ffeb63a0 con 0x55d9fffc5b40 2024-09-19T08:03:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce19e7c640 1 --2- 172.21.15.98:0/280851625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffd3780 0x55d9fffd5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce19e7c640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fce000a34d0 0x7fce000a5990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fce0c009a60 tx=0x7fce0c0099f0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.260+0000 7fce1967b640 1 --2- 172.21.15.98:0/280851625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffc5b40 0x55d9fffc4090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fce100cd430 tx=0x7fce1009b850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2a5027813f8e31f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.261+0000 7fcdff7fe640 1 -- 172.21.15.98:0/280851625 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce10014bc0 con 0x55d9fffc5b40 2024-09-19T08:03:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.261+0000 7fcdff7fe640 1 -- 172.21.15.98:0/280851625 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce100137e0 con 0x55d9fffc5b40 2024-09-19T08:03:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.261+0000 7fcdff7fe640 1 -- 172.21.15.98:0/280851625 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1009a1f0 con 0x55d9fffc5b40 2024-09-19T08:03:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.262+0000 7fce1dcb8740 1 -- 172.21.15.98:0/280851625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffc5b40 msgr2=0x55d9fffc4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.262+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/280851625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffc5b40 0x55d9fffc4090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fce100cd430 tx=0x7fce1009b850 comp rx=0 tx=0).stop 2024-09-19T08:03:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.262+0000 7fce1dcb8740 1 -- 172.21.15.98:0/280851625 shutdown_connections 2024-09-19T08:03:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.262+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/280851625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffd3780 0x55d9fffd5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.262+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/280851625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc45d0 0x55d9fffc4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.262+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/280851625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffc5b40 0x55d9fffc4090 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.262+0000 7fce1dcb8740 1 -- 172.21.15.98:0/280851625 >> 172.21.15.98:0/280851625 conn(0x55d9fff8d310 msgr2=0x55d9fffc0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.262+0000 7fce1dcb8740 1 -- 172.21.15.98:0/280851625 shutdown_connections 2024-09-19T08:03:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.262+0000 7fce1dcb8740 1 -- 172.21.15.98:0/280851625 wait complete. 2024-09-19T08:03:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.262+0000 7fce1dcb8740 1 -- start start 2024-09-19T08:03:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.262+0000 7fce1dcb8740 1 Processor -- start 2024-09-19T08:03:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.262+0000 7fce1dcb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc45d0 0x55d9fffd31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.262+0000 7fce1dcb8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffc5b40 0x55d9fffd0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.262+0000 7fce1dcb8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffd3780 0x55d9fffd0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce1dcb8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d9fffd6420 con 0x55d9fffc45d0 2024-09-19T08:03:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce1dcb8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d9ffee5060 con 0x55d9fffd3780 2024-09-19T08:03:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce1dcb8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d9fffd65a0 con 0x55d9fffc5b40 2024-09-19T08:03:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce1dcb8740 10 client.0 _ll_register_callbacks cb 0x55d9ffc388c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce1967b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc45d0 0x55d9fffd31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce1967b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc45d0 0x55d9fffd31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41536/0 (socket says 172.21.15.98:41536) 2024-09-19T08:03:53.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce1967b640 1 -- 172.21.15.98:0/3810182454 learned_addr learned my addr 172.21.15.98:0/3810182454 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:53.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce18e7a640 1 --2- 172.21.15.98:0/3810182454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffc5b40 0x55d9fffd0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce19e7c640 1 --2- 172.21.15.98:0/3810182454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffd3780 0x55d9fffd0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce1967b640 1 -- 172.21.15.98:0/3810182454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffc5b40 msgr2=0x55d9fffd0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:53.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce1967b640 1 --2- 172.21.15.98:0/3810182454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffc5b40 0x55d9fffd0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce1967b640 1 -- 172.21.15.98:0/3810182454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffd3780 msgr2=0x55d9fffd0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:53.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce1967b640 1 --2- 172.21.15.98:0/3810182454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffd3780 0x55d9fffd0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce1967b640 1 -- 172.21.15.98:0/3810182454 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9ffc9b3c0 con 0x55d9fffc45d0 2024-09-19T08:03:53.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce19e7c640 1 --2- 172.21.15.98:0/3810182454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffd3780 0x55d9fffd0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:53.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce1967b640 1 --2- 172.21.15.98:0/3810182454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc45d0 0x55d9fffd31d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fce100cd490 tx=0x7fce100cd3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32160106bce397b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:53.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fcdfcff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:53.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fcdfcff9640 1 -- 172.21.15.98:0/3810182454 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce100b4070 con 0x55d9fffc45d0 2024-09-19T08:03:53.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fcdfcff9640 1 -- 172.21.15.98:0/3810182454 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fce1009a1f0 con 0x55d9fffc45d0 2024-09-19T08:03:53.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.263+0000 7fce1dcb8740 1 -- 172.21.15.98:0/3810182454 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce100137e0 con 0x55d9fffc45d0 2024-09-19T08:03:53.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.264+0000 7fcdfcff9640 1 -- 172.21.15.98:0/3810182454 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fce10014ea0 con 0x55d9fffc45d0 2024-09-19T08:03:53.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.264+0000 7fce1dcb8740 10 client.15702 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:53.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.264+0000 7fce1dcb8740 1 -- 172.21.15.98:0/3810182454 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d9fffe1f30 con 0x55d9fffc45d0 2024-09-19T08:03:53.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.264+0000 7fcdfcff9640 1 -- 172.21.15.98:0/3810182454 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce100137e0 con 0x55d9fffc45d0 2024-09-19T08:03:53.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.264+0000 7fcdfcff9640 1 -- 172.21.15.98:0/3810182454 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fce100cd7a0 con 0x55d9fffc45d0 2024-09-19T08:03:53.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.264+0000 7fce1dcb8740 10 client.15702 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:53.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.264+0000 7fce1dcb8740 10 client.15702 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:03:53.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.264+0000 7fce1dcb8740 20 client.15702 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:53.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.264+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/3810182454 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55d9fffe3280 0x55d9fffe5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.264+0000 7fce1dcb8740 4 client.15702 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:53.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.264+0000 7fce1dcb8740 1 -- 172.21.15.98:0/3810182454 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9fff37860 con 0x55d9fffe3280 2024-09-19T08:03:53.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.264+0000 7fce18e7a640 1 --2- 172.21.15.98:0/3810182454 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55d9fffe3280 0x55d9fffe5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.265+0000 7fcdfcff9640 10 client.15702 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:03:53.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.265+0000 7fce18e7a640 1 --2- 172.21.15.98:0/3810182454 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55d9fffe3280 0x55d9fffe5670 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5c955e3db2685d7d server_cookie=59f382c29c4ef83c in_seq=0 out_seq=0 2024-09-19T08:03:53.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.265+0000 7fcdfcff9640 1 -- 172.21.15.98:0/3810182454 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d9fff37860 con 0x55d9fffe3280 2024-09-19T08:03:53.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.265+0000 7fcdfcff9640 10 client.15702 handle_command_reply: tid=0 2024-09-19T08:03:53.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.266+0000 7fce1dcb8740 1 client.15702 shutdown 2024-09-19T08:03:53.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.266+0000 7fce1dcb8740 1 -- 172.21.15.98:0/3810182454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc45d0 msgr2=0x55d9fffd31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:53.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.266+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/3810182454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc45d0 0x55d9fffd31d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fce100cd490 tx=0x7fce100cd3e0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.266+0000 7fce1dcb8740 1 -- 172.21.15.98:0/3810182454 shutdown_connections 2024-09-19T08:03:53.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.266+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/3810182454 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55d9fffe3280 0x55d9fffe5670 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.266+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/3810182454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffd3780 0x55d9fffd0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.266+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/3810182454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffc5b40 0x55d9fffd0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.266+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/3810182454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc45d0 0x55d9fffd31d0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.266+0000 7fcdfcff9640 0 client.15702 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:03:53.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.266+0000 7fce1dcb8740 1 -- 172.21.15.98:0/3810182454 >> 172.21.15.98:0/3810182454 conn(0x55d9fff8d310 msgr2=0x55d9fffc6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:53.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.266+0000 7fce1dcb8740 1 -- 172.21.15.98:0/3810182454 shutdown_connections 2024-09-19T08:03:53.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.266+0000 7fce1dcb8740 1 -- 172.21.15.98:0/3810182454 wait complete. 2024-09-19T08:03:53.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.266+0000 7fce1dcb8740 20 client.15702 trim_cache size 0 max 16384 2024-09-19T08:03:53.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.279+0000 7fce1dcb8740 1 Processor -- start 2024-09-19T08:03:53.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.279+0000 7fce1dcb8740 1 -- start start 2024-09-19T08:03:53.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.279+0000 7fce1dcb8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc45d0 0x55d9fffc5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.279+0000 7fce1dcb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc5b40 0x55d9fffe8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.279+0000 7fce1dcb8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffd3780 0x55d9fffe8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.279+0000 7fce1dcb8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fce141ab600 con 0x55d9fffc5b40 2024-09-19T08:03:53.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.279+0000 7fce1dcb8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fce141ad140 con 0x55d9fffc45d0 2024-09-19T08:03:53.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.279+0000 7fce1dcb8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce141abcd0 con 0x55d9fffd3780 2024-09-19T08:03:53.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.280+0000 7fce18e7a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc5b40 0x55d9fffe8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.280+0000 7fce18e7a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc5b40 0x55d9fffe8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41538/0 (socket says 172.21.15.98:41538) 2024-09-19T08:03:53.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.280+0000 7fce18e7a640 1 -- 172.21.15.98:0/3097643234 learned_addr learned my addr 172.21.15.98:0/3097643234 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:53.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.280+0000 7fce19e7c640 1 --2- 172.21.15.98:0/3097643234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffd3780 0x55d9fffe8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.280+0000 7fce1967b640 1 --2- 172.21.15.98:0/3097643234 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc45d0 0x55d9fffc5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.280+0000 7fce18e7a640 1 -- 172.21.15.98:0/3097643234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffd3780 msgr2=0x55d9fffe8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:53.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.280+0000 7fce18e7a640 1 --2- 172.21.15.98:0/3097643234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffd3780 0x55d9fffe8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.280+0000 7fce18e7a640 1 -- 172.21.15.98:0/3097643234 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc45d0 msgr2=0x55d9fffc5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:53.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.280+0000 7fce18e7a640 1 --2- 172.21.15.98:0/3097643234 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc45d0 0x55d9fffc5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.280+0000 7fce18e7a640 1 -- 172.21.15.98:0/3097643234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9fffd0f80 con 0x55d9fffc5b40 2024-09-19T08:03:53.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.280+0000 7fce1967b640 1 --2- 172.21.15.98:0/3097643234 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc45d0 0x55d9fffc5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:53.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.280+0000 7fce19e7c640 1 --2- 172.21.15.98:0/3097643234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffd3780 0x55d9fffe8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:53.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.280+0000 7fce18e7a640 1 --2- 172.21.15.98:0/3097643234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc5b40 0x55d9fffe8690 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x55d9fffc1ac0 tx=0x7fce04008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b9c3dce4ed07b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:53.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.280+0000 7fcdfeffd640 1 -- 172.21.15.98:0/3097643234 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04016aa0 con 0x55d9fffc5b40 2024-09-19T08:03:53.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.281+0000 7fcdfeffd640 1 -- 172.21.15.98:0/3097643234 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce04020030 con 0x55d9fffc5b40 2024-09-19T08:03:53.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.281+0000 7fcdfeffd640 1 -- 172.21.15.98:0/3097643234 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0401c250 con 0x55d9fffc5b40 2024-09-19T08:03:53.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.281+0000 7fce1dcb8740 1 -- 172.21.15.98:0/3097643234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc5b40 msgr2=0x55d9fffe8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:53.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.281+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/3097643234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc5b40 0x55d9fffe8690 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x55d9fffc1ac0 tx=0x7fce04008040 comp rx=0 tx=0).stop 2024-09-19T08:03:53.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.281+0000 7fce1dcb8740 1 -- 172.21.15.98:0/3097643234 shutdown_connections 2024-09-19T08:03:53.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.281+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/3097643234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffd3780 0x55d9fffe8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.281+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/3097643234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc5b40 0x55d9fffe8690 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.281+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/3097643234 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc45d0 0x55d9fffc5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.281+0000 7fce1dcb8740 1 -- 172.21.15.98:0/3097643234 >> 172.21.15.98:0/3097643234 conn(0x55d9fff8d310 msgr2=0x55d9fffbba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:53.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.281+0000 7fce1dcb8740 1 -- 172.21.15.98:0/3097643234 shutdown_connections 2024-09-19T08:03:53.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.281+0000 7fce1dcb8740 1 -- 172.21.15.98:0/3097643234 wait complete. 2024-09-19T08:03:53.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.282+0000 7fce1dcb8740 1 -- start start 2024-09-19T08:03:53.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.282+0000 7fce1dcb8740 1 Processor -- start 2024-09-19T08:03:53.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.282+0000 7fce1dcb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc45d0 0x55d9fffe8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.282+0000 7fce1dcb8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc5b40 0x55d9fffe5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.282+0000 7fce1dcb8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffd3780 0x55d9fffe6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.282+0000 7fce1dcb8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fce141aca70 con 0x55d9fffc45d0 2024-09-19T08:03:53.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.282+0000 7fce1dcb8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fce141ac3a0 con 0x55d9fffc5b40 2024-09-19T08:03:53.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.282+0000 7fce1dcb8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce141a95e0 con 0x55d9fffd3780 2024-09-19T08:03:53.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.282+0000 7fce1dcb8740 10 client.0 _ll_register_callbacks cb 0x7fce040071a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:53.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.282+0000 7fce1967b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc45d0 0x55d9fffe8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.282+0000 7fce1967b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc45d0 0x55d9fffe8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41544/0 (socket says 172.21.15.98:41544) 2024-09-19T08:03:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.282+0000 7fce1967b640 1 -- 172.21.15.98:0/1299267848 learned_addr learned my addr 172.21.15.98:0/1299267848 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fce19e7c640 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffd3780 0x55d9fffe6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fce18e7a640 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc5b40 0x55d9fffe5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fce1967b640 1 -- 172.21.15.98:0/1299267848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffd3780 msgr2=0x55d9fffe6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fce1967b640 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffd3780 0x55d9fffe6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fce1967b640 1 -- 172.21.15.98:0/1299267848 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc5b40 msgr2=0x55d9fffe5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fce1967b640 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc5b40 0x55d9fffe5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fce1967b640 1 -- 172.21.15.98:0/1299267848 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9fffe6a10 con 0x55d9fffc45d0 2024-09-19T08:03:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fce19e7c640 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffd3780 0x55d9fffe6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:03:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fce18e7a640 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc5b40 0x55d9fffe5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fce1967b640 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc45d0 0x55d9fffe8bc0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fce10007590 tx=0x7fce1009ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f391d1b206892aa server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:53.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fcdfcff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:53.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fcdfcff9640 1 -- 172.21.15.98:0/1299267848 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce100b4040 con 0x55d9fffc45d0 2024-09-19T08:03:53.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fcdfcff9640 1 -- 172.21.15.98:0/1299267848 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fce100cd710 con 0x55d9fffc45d0 2024-09-19T08:03:53.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fce1dcb8740 1 -- 172.21.15.98:0/1299267848 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce04020030 con 0x55d9fffc45d0 2024-09-19T08:03:53.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fcdfcff9640 1 -- 172.21.15.98:0/1299267848 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fce100a9050 con 0x55d9fffc45d0 2024-09-19T08:03:53.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fce1dcb8740 10 client.15708 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:53.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fce1dcb8740 1 -- 172.21.15.98:0/1299267848 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d9fffe6a10 con 0x55d9fffc45d0 2024-09-19T08:03:53.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.283+0000 7fcdfcff9640 1 -- 172.21.15.98:0/1299267848 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce04020030 con 0x55d9fffc45d0 2024-09-19T08:03:53.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.284+0000 7fcdfcff9640 1 -- 172.21.15.98:0/1299267848 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fce10065040 con 0x55d9fffc45d0 2024-09-19T08:03:53.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.284+0000 7fce1dcb8740 10 client.15708 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:53.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.284+0000 7fce1dcb8740 10 client.15708 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:03:53.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.284+0000 7fce1dcb8740 20 client.15708 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:53.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.284+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55da0004dce0 0x55da00050100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:53.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.284+0000 7fce1dcb8740 4 client.15708 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:03:53.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.284+0000 7fce1dcb8740 1 -- 172.21.15.98:0/1299267848 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d9fff31d70 con 0x55da0004dce0 2024-09-19T08:03:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.284+0000 7fce18e7a640 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55da0004dce0 0x55da00050100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.284+0000 7fce18e7a640 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55da0004dce0 0x55da00050100 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dd09abccd7b9037a server_cookie=3a10eaac70fada39 in_seq=0 out_seq=0 2024-09-19T08:03:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.284+0000 7fcdfcff9640 10 client.15708 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:03:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fcdfcff9640 1 -- 172.21.15.98:0/1299267848 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d9fff31d70 con 0x55da0004dce0 2024-09-19T08:03:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fcdfcff9640 10 client.15708 handle_command_reply: tid=0 2024-09-19T08:03:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1dcb8740 1 client.15708 shutdown 2024-09-19T08:03:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1dcb8740 1 -- 172.21.15.98:0/1299267848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc45d0 msgr2=0x55d9fffe8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc45d0 0x55d9fffe8bc0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fce10007590 tx=0x7fce1009ba30 comp rx=0 tx=0).stop 2024-09-19T08:03:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1dcb8740 1 -- 172.21.15.98:0/1299267848 shutdown_connections 2024-09-19T08:03:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55da0004dce0 0x55da00050100 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.314 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:03:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9fffd3780 0x55d9fffe6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d9fffc5b40 0x55d9fffe5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1dcb8740 1 --2- 172.21.15.98:0/1299267848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9fffc45d0 0x55d9fffe8bc0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1dcb8740 1 -- 172.21.15.98:0/1299267848 >> 172.21.15.98:0/1299267848 conn(0x55d9fff8d310 msgr2=0x55d9fffde830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fcdfcff9640 0 client.15708 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:03:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1dcb8740 1 -- 172.21.15.98:0/1299267848 shutdown_connections 2024-09-19T08:03:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1dcb8740 1 -- 172.21.15.98:0/1299267848 wait complete. 2024-09-19T08:03:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1dcb8740 20 client.15708 trim_cache size 0 max 16384 2024-09-19T08:03:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1b8df640 1 -- 172.21.15.98:0/1621684379 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fce000a34d0 msgr2=0x7fce000a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1b8df640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fce000a34d0 0x7fce000a5990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fce0c009a60 tx=0x7fce0c0099f0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1b8df640 1 -- 172.21.15.98:0/1621684379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce14102d90 msgr2=0x7fce141985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.285+0000 7fce1b8df640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce14102d90 0x7fce141985d0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fce1000c970 tx=0x7fce1000ce30 comp rx=0 tx=0).stop 2024-09-19T08:03:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.286+0000 7fce1b8df640 1 -- 172.21.15.98:0/1621684379 shutdown_connections 2024-09-19T08:03:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.286+0000 7fce1b8df640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fce000a34d0 0x7fce000a5990 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.286+0000 7fce1b8df640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce14108290 0x7fce1419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.286+0000 7fce1b8df640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce14103740 0x7fce14198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.286+0000 7fce1b8df640 1 --2- 172.21.15.98:0/1621684379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce14102d90 0x7fce141985d0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.286+0000 7fce1b8df640 1 -- 172.21.15.98:0/1621684379 >> 172.21.15.98:0/1621684379 conn(0x7fce140fc7d0 msgr2=0x7fce14106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.286+0000 7fce1b8df640 1 -- 172.21.15.98:0/1621684379 shutdown_connections 2024-09-19T08:03:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:53.286+0000 7fce1b8df640 1 -- 172.21.15.98:0/1621684379 wait complete. 2024-09-19T08:03:54.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:03:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.790+0000 7f606d7a2640 1 -- 172.21.15.98:0/107402918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f606810bff0 msgr2=0x7f606810e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.790+0000 7f606d7a2640 1 --2- 172.21.15.98:0/107402918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f606810bff0 0x7f606810e460 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6060009ca0 tx=0x7f606002a950 comp rx=0 tx=0).stop 2024-09-19T08:03:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.790+0000 7f606d7a2640 1 -- 172.21.15.98:0/107402918 shutdown_connections 2024-09-19T08:03:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.790+0000 7f606d7a2640 1 --2- 172.21.15.98:0/107402918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f606810bff0 0x7f606810e460 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.790+0000 7f606d7a2640 1 --2- 172.21.15.98:0/107402918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6068103570 0x7f606810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.790+0000 7f606d7a2640 1 --2- 172.21.15.98:0/107402918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6068102c50 0x7f6068103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.790+0000 7f606d7a2640 1 -- 172.21.15.98:0/107402918 >> 172.21.15.98:0/107402918 conn(0x7f60680fc610 msgr2=0x7f60680fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.790+0000 7f606d7a2640 1 -- 172.21.15.98:0/107402918 shutdown_connections 2024-09-19T08:03:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.790+0000 7f606d7a2640 1 -- 172.21.15.98:0/107402918 wait complete. 2024-09-19T08:03:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.790+0000 7f606d7a2640 1 Processor -- start 2024-09-19T08:03:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.790+0000 7f606d7a2640 1 -- start start 2024-09-19T08:03:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f606d7a2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6068102c50 0x7f60681008b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f606d7a2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6068103570 0x7f6068100df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f606d7a2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f606810bff0 0x7f6068101330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f606d7a2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6068111d90 con 0x7f6068102c50 2024-09-19T08:03:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f606d7a2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6068111c10 con 0x7f606810bff0 2024-09-19T08:03:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f606d7a2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6068111f10 con 0x7f6068103570 2024-09-19T08:03:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f6067fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6068102c50 0x7f60681008b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f6067fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6068102c50 0x7f60681008b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41562/0 (socket says 172.21.15.98:41562) 2024-09-19T08:03:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f6067fff640 1 -- 172.21.15.98:0/1941375722 learned_addr learned my addr 172.21.15.98:0/1941375722 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f60677fe640 1 --2- 172.21.15.98:0/1941375722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6068103570 0x7f6068100df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f606cfa1640 1 --2- 172.21.15.98:0/1941375722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f606810bff0 0x7f6068101330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f6067fff640 1 -- 172.21.15.98:0/1941375722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6068103570 msgr2=0x7f6068100df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f6067fff640 1 --2- 172.21.15.98:0/1941375722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6068103570 0x7f6068100df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f6067fff640 1 -- 172.21.15.98:0/1941375722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f606810bff0 msgr2=0x7f6068101330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f6067fff640 1 --2- 172.21.15.98:0/1941375722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f606810bff0 0x7f6068101330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.791+0000 7f6067fff640 1 -- 172.21.15.98:0/1941375722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60681a6730 con 0x7f6068102c50 2024-09-19T08:03:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.792+0000 7f6067fff640 1 --2- 172.21.15.98:0/1941375722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6068102c50 0x7f60681008b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f605800d950 tx=0x7f605800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.792+0000 7f60657fa640 1 -- 172.21.15.98:0/1941375722 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058017070 con 0x7f6068102c50 2024-09-19T08:03:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.792+0000 7f606d7a2640 1 -- 172.21.15.98:0/1941375722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60681a6a20 con 0x7f6068102c50 2024-09-19T08:03:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.792+0000 7f60657fa640 1 -- 172.21.15.98:0/1941375722 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6058009c30 con 0x7f6068102c50 2024-09-19T08:03:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.792+0000 7f60657fa640 1 -- 172.21.15.98:0/1941375722 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605800f6b0 con 0x7f6068102c50 2024-09-19T08:03:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.792+0000 7f606d7a2640 1 -- 172.21.15.98:0/1941375722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60681a6f60 con 0x7f6068102c50 2024-09-19T08:03:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.794+0000 7f60657fa640 1 -- 172.21.15.98:0/1941375722 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f60580109d0 con 0x7f6068102c50 2024-09-19T08:03:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.794+0000 7f6070ddd740 1 Processor -- start 2024-09-19T08:03:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.794+0000 7f6070ddd740 1 -- start start 2024-09-19T08:03:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f6070ddd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c7473d20 0x5598c7472270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f6070ddd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c74727b0 0x5598c7472bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f606cfa1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c7473d20 0x5598c7472270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f6070ddd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c747d550 0x5598c747f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f6070ddd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5598c73d82f0 con 0x5598c7473d20 2024-09-19T08:03:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f6070ddd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5598c737fcf0 con 0x5598c74727b0 2024-09-19T08:03:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f606cfa1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c7473d20 0x5598c7472270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41568/0 (socket says 172.21.15.98:41568) 2024-09-19T08:03:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f606cfa1640 1 -- 172.21.15.98:0/3859321771 learned_addr learned my addr 172.21.15.98:0/3859321771 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f6070ddd740 1 -- 172.21.15.98:0/3859321771 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5598c73d42b0 con 0x5598c747d550 2024-09-19T08:03:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f6067fff640 1 --2- 172.21.15.98:0/3859321771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c74727b0 0x5598c7472bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f60677fe640 1 --2- 172.21.15.98:0/3859321771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c747d550 0x5598c747f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f60657fa640 1 --2- 172.21.15.98:0/1941375722 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f60500a34d0 0x7f60500a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f6067fff640 1 -- 172.21.15.98:0/3859321771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c747d550 msgr2=0x5598c747f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f6067fff640 1 --2- 172.21.15.98:0/3859321771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c747d550 0x5598c747f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f6067fff640 1 -- 172.21.15.98:0/3859321771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c7473d20 msgr2=0x5598c7472270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f6067fff640 1 --2- 172.21.15.98:0/3859321771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c7473d20 0x5598c7472270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f6067fff640 1 -- 172.21.15.98:0/3859321771 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598c73643a0 con 0x5598c74727b0 2024-09-19T08:03:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f606cfa1640 1 --2- 172.21.15.98:0/3859321771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c7473d20 0x5598c7472270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.795+0000 7f606cfa1640 1 --2- 172.21.15.98:0/1941375722 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f60500a34d0 0x7f60500a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.796+0000 7f6067fff640 1 --2- 172.21.15.98:0/3859321771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c74727b0 0x5598c7472bf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f60580cd210 tx=0x7f60580d3e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2cb9b417f50bdbc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.796+0000 7f604e7fc640 1 -- 172.21.15.98:0/3859321771 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605809baa0 con 0x5598c74727b0 2024-09-19T08:03:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.796+0000 7f60657fa640 1 -- 172.21.15.98:0/1941375722 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f60580cfc10 con 0x7f6068102c50 2024-09-19T08:03:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.796+0000 7f606cfa1640 1 --2- 172.21.15.98:0/1941375722 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f60500a34d0 0x7f60500a5990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6060009430 tx=0x7f60600095c0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.796+0000 7f604e7fc640 1 -- 172.21.15.98:0/3859321771 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6058097bc0 con 0x5598c74727b0 2024-09-19T08:03:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.796+0000 7f604e7fc640 1 -- 172.21.15.98:0/3859321771 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058009de0 con 0x5598c74727b0 2024-09-19T08:03:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.797+0000 7f6070ddd740 1 -- 172.21.15.98:0/3859321771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c74727b0 msgr2=0x5598c7472bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.797+0000 7f6070ddd740 1 --2- 172.21.15.98:0/3859321771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c74727b0 0x5598c7472bf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f60580cd210 tx=0x7f60580d3e10 comp rx=0 tx=0).stop 2024-09-19T08:03:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.797+0000 7f6070ddd740 1 -- 172.21.15.98:0/3859321771 shutdown_connections 2024-09-19T08:03:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.797+0000 7f6070ddd740 1 --2- 172.21.15.98:0/3859321771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c747d550 0x5598c747f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.797+0000 7f6070ddd740 1 --2- 172.21.15.98:0/3859321771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c74727b0 0x5598c7472bf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.797+0000 7f6070ddd740 1 --2- 172.21.15.98:0/3859321771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c7473d20 0x5598c7472270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.797+0000 7f6070ddd740 1 -- 172.21.15.98:0/3859321771 >> 172.21.15.98:0/3859321771 conn(0x5598c743b310 msgr2=0x5598c746ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.797+0000 7f6070ddd740 1 -- 172.21.15.98:0/3859321771 shutdown_connections 2024-09-19T08:03:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.797+0000 7f6070ddd740 1 -- 172.21.15.98:0/3859321771 wait complete. 2024-09-19T08:03:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.797+0000 7f6070ddd740 1 -- start start 2024-09-19T08:03:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.797+0000 7f6070ddd740 1 Processor -- start 2024-09-19T08:03:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f6070ddd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c74727b0 0x5598c747ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f6070ddd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c7473d20 0x5598c747a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f6070ddd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c747d550 0x5598c747a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f6070ddd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5598c7480250 con 0x5598c747d550 2024-09-19T08:03:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f6070ddd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5598c7393060 con 0x5598c7473d20 2024-09-19T08:03:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f6070ddd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5598c74803d0 con 0x5598c74727b0 2024-09-19T08:03:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f6070ddd740 10 client.0 _ll_register_callbacks cb 0x5598c70e68c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f606cfa1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c747d550 0x5598c747a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f606cfa1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c747d550 0x5598c747a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41574/0 (socket says 172.21.15.98:41574) 2024-09-19T08:03:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f606cfa1640 1 -- 172.21.15.98:0/2498591597 learned_addr learned my addr 172.21.15.98:0/2498591597 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f60677fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c7473d20 0x5598c747a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f6067fff640 1 --2- 172.21.15.98:0/2498591597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c74727b0 0x5598c747ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f606cfa1640 1 -- 172.21.15.98:0/2498591597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c74727b0 msgr2=0x5598c747ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f606cfa1640 1 --2- 172.21.15.98:0/2498591597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c74727b0 0x5598c747ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f606cfa1640 1 -- 172.21.15.98:0/2498591597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c7473d20 msgr2=0x5598c747a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f606cfa1640 1 --2- 172.21.15.98:0/2498591597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c7473d20 0x5598c747a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f606cfa1640 1 -- 172.21.15.98:0/2498591597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598c71493c0 con 0x5598c747d550 2024-09-19T08:03:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f60677fe640 1 --2- 172.21.15.98:0/2498591597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c7473d20 0x5598c747a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f606cfa1640 1 --2- 172.21.15.98:0/2498591597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c747d550 0x5598c747a690 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6060006010 tx=0x7f606002a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3536c598691e381 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.798+0000 7f60477fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.799+0000 7f60477fe640 1 -- 172.21.15.98:0/2498591597 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6060034040 con 0x5598c747d550 2024-09-19T08:03:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.799+0000 7f60477fe640 1 -- 172.21.15.98:0/2498591597 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f6060030dd0 con 0x5598c747d550 2024-09-19T08:03:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.799+0000 7f6070ddd740 1 -- 172.21.15.98:0/2498591597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6058097bc0 con 0x5598c747d550 2024-09-19T08:03:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.799+0000 7f60477fe640 1 -- 172.21.15.98:0/2498591597 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6060003970 con 0x5598c747d550 2024-09-19T08:03:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.799+0000 7f6070ddd740 10 client.15720 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.799+0000 7f6070ddd740 1 -- 172.21.15.98:0/2498591597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5598c748bca0 con 0x5598c747d550 2024-09-19T08:03:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.799+0000 7f60477fe640 1 -- 172.21.15.98:0/2498591597 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6058097bc0 con 0x5598c747d550 2024-09-19T08:03:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.799+0000 7f60477fe640 1 -- 172.21.15.98:0/2498591597 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f606003a430 con 0x5598c747d550 2024-09-19T08:03:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.799+0000 7f6070ddd740 10 client.15720 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.799+0000 7f6070ddd740 10 client.15720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:03:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.799+0000 7f6070ddd740 20 client.15720 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.799+0000 7f6070ddd740 1 --2- 172.21.15.98:0/2498591597 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5598c748d170 0x5598c748f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.799+0000 7f6070ddd740 4 client.15720 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.799+0000 7f6070ddd740 1 -- 172.21.15.98:0/2498591597 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598c73e5860 con 0x5598c748d170 2024-09-19T08:03:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.800+0000 7f6067fff640 1 --2- 172.21.15.98:0/2498591597 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5598c748d170 0x5598c748f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.800+0000 7f6067fff640 1 --2- 172.21.15.98:0/2498591597 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5598c748d170 0x5598c748f5b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=559950873287fc12 server_cookie=7258691d4b5a3890 in_seq=0 out_seq=0 2024-09-19T08:03:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.800+0000 7f60477fe640 10 client.15720 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.801+0000 7f60477fe640 1 -- 172.21.15.98:0/2498591597 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5598c73e5860 con 0x5598c748d170 2024-09-19T08:03:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.801+0000 7f60477fe640 10 client.15720 handle_command_reply: tid=0 2024-09-19T08:03:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.801+0000 7f6070ddd740 1 client.15720 shutdown 2024-09-19T08:03:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.801+0000 7f6070ddd740 1 -- 172.21.15.98:0/2498591597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c747d550 msgr2=0x5598c747a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.801+0000 7f6070ddd740 1 --2- 172.21.15.98:0/2498591597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c747d550 0x5598c747a690 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6060006010 tx=0x7f606002a950 comp rx=0 tx=0).stop 2024-09-19T08:03:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.801+0000 7f6070ddd740 1 -- 172.21.15.98:0/2498591597 shutdown_connections 2024-09-19T08:03:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.801+0000 7f6070ddd740 1 --2- 172.21.15.98:0/2498591597 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5598c748d170 0x5598c748f5b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.801+0000 7f6070ddd740 1 --2- 172.21.15.98:0/2498591597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c747d550 0x5598c747a690 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.801+0000 7f60477fe640 0 client.15720 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.801+0000 7f6070ddd740 1 --2- 172.21.15.98:0/2498591597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c7473d20 0x5598c747a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.801+0000 7f6070ddd740 1 --2- 172.21.15.98:0/2498591597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c74727b0 0x5598c747ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.801+0000 7f6070ddd740 1 -- 172.21.15.98:0/2498591597 >> 172.21.15.98:0/2498591597 conn(0x5598c743b310 msgr2=0x5598c7474b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:54.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.801+0000 7f6070ddd740 1 -- 172.21.15.98:0/2498591597 shutdown_connections 2024-09-19T08:03:54.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.802+0000 7f6070ddd740 1 -- 172.21.15.98:0/2498591597 wait complete. 2024-09-19T08:03:54.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.802+0000 7f6070ddd740 20 client.15720 trim_cache size 0 max 16384 2024-09-19T08:03:54.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.817+0000 7f6070ddd740 1 Processor -- start 2024-09-19T08:03:54.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.817+0000 7f6070ddd740 1 -- start start 2024-09-19T08:03:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6070ddd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c74727b0 0x5598c7489550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6070ddd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c7473d20 0x5598c746e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6070ddd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c747d550 0x5598c746ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6070ddd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f60681af910 con 0x5598c74727b0 2024-09-19T08:03:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6070ddd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f60681b1450 con 0x5598c7473d20 2024-09-19T08:03:54.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6070ddd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f60681affe0 con 0x5598c747d550 2024-09-19T08:03:54.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6067fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c74727b0 0x5598c7489550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6067fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c74727b0 0x5598c7489550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41578/0 (socket says 172.21.15.98:41578) 2024-09-19T08:03:54.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6067fff640 1 -- 172.21.15.98:0/276884510 learned_addr learned my addr 172.21.15.98:0/276884510 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:54.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f606cfa1640 1 --2- 172.21.15.98:0/276884510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c747d550 0x5598c746ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f60677fe640 1 --2- 172.21.15.98:0/276884510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c7473d20 0x5598c746e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6067fff640 1 -- 172.21.15.98:0/276884510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c747d550 msgr2=0x5598c746ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:54.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6067fff640 1 --2- 172.21.15.98:0/276884510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c747d550 0x5598c746ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6067fff640 1 -- 172.21.15.98:0/276884510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c7473d20 msgr2=0x5598c746e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:54.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6067fff640 1 --2- 172.21.15.98:0/276884510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c7473d20 0x5598c746e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6067fff640 1 -- 172.21.15.98:0/276884510 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598c747b4d0 con 0x5598c74727b0 2024-09-19T08:03:54.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f6067fff640 1 --2- 172.21.15.98:0/276884510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c74727b0 0x5598c7489550 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f605800dfe0 tx=0x7f60580a4940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5321816e30ba88 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:54.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.818+0000 7f604dffb640 1 -- 172.21.15.98:0/276884510 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058020040 con 0x5598c74727b0 2024-09-19T08:03:54.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.819+0000 7f604dffb640 1 -- 172.21.15.98:0/276884510 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60580cd150 con 0x5598c74727b0 2024-09-19T08:03:54.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.819+0000 7f604dffb640 1 -- 172.21.15.98:0/276884510 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058010460 con 0x5598c74727b0 2024-09-19T08:03:54.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.819+0000 7f6070ddd740 1 -- 172.21.15.98:0/276884510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c74727b0 msgr2=0x5598c7489550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:54.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.819+0000 7f6070ddd740 1 --2- 172.21.15.98:0/276884510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c74727b0 0x5598c7489550 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f605800dfe0 tx=0x7f60580a4940 comp rx=0 tx=0).stop 2024-09-19T08:03:54.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.819+0000 7f6070ddd740 1 -- 172.21.15.98:0/276884510 shutdown_connections 2024-09-19T08:03:54.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.819+0000 7f6070ddd740 1 --2- 172.21.15.98:0/276884510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c747d550 0x5598c746ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.819+0000 7f6070ddd740 1 --2- 172.21.15.98:0/276884510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c7473d20 0x5598c746e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.819+0000 7f6070ddd740 1 --2- 172.21.15.98:0/276884510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c74727b0 0x5598c7489550 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.819+0000 7f6070ddd740 1 -- 172.21.15.98:0/276884510 >> 172.21.15.98:0/276884510 conn(0x5598c743b310 msgr2=0x5598c7479240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:54.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.819+0000 7f6070ddd740 1 -- 172.21.15.98:0/276884510 shutdown_connections 2024-09-19T08:03:54.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.819+0000 7f6070ddd740 1 -- 172.21.15.98:0/276884510 wait complete. 2024-09-19T08:03:54.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.820+0000 7f6070ddd740 1 -- start start 2024-09-19T08:03:54.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.820+0000 7f6070ddd740 1 Processor -- start 2024-09-19T08:03:54.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.820+0000 7f6070ddd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c7473d20 0x5598c7485d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.820+0000 7f6070ddd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c747d550 0x5598c7481020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.820+0000 7f6070ddd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c7481560 0x5598c74819e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.820+0000 7f6070ddd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f60681b0d80 con 0x5598c7473d20 2024-09-19T08:03:54.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.820+0000 7f6070ddd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f60681b06b0 con 0x5598c747d550 2024-09-19T08:03:54.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.820+0000 7f6070ddd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f60681ad8f0 con 0x5598c7481560 2024-09-19T08:03:54.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.820+0000 7f6070ddd740 10 client.0 _ll_register_callbacks cb 0x7f606002fcc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:54.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.820+0000 7f6067fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c7473d20 0x5598c7485d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.820+0000 7f6067fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c7473d20 0x5598c7485d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41590/0 (socket says 172.21.15.98:41590) 2024-09-19T08:03:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.820+0000 7f6067fff640 1 -- 172.21.15.98:0/1837670414 learned_addr learned my addr 172.21.15.98:0/1837670414 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f60677fe640 1 --2- 172.21.15.98:0/1837670414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c747d550 0x5598c7481020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f606cfa1640 1 --2- 172.21.15.98:0/1837670414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c7481560 0x5598c74819e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f6067fff640 1 -- 172.21.15.98:0/1837670414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c7481560 msgr2=0x5598c74819e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f6067fff640 1 --2- 172.21.15.98:0/1837670414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c7481560 0x5598c74819e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f6067fff640 1 -- 172.21.15.98:0/1837670414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c747d550 msgr2=0x5598c7481020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f6067fff640 1 --2- 172.21.15.98:0/1837670414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c747d550 0x5598c7481020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f6067fff640 1 -- 172.21.15.98:0/1837670414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598c74f6f40 con 0x5598c7473d20 2024-09-19T08:03:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f606cfa1640 1 --2- 172.21.15.98:0/1837670414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c7481560 0x5598c74819e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f6067fff640 1 --2- 172.21.15.98:0/1837670414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c7473d20 0x5598c7485d90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6058084fd0 tx=0x7f60580a4a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3650cbf30d2427 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f60477fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f60477fe640 1 -- 172.21.15.98:0/1837670414 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058020070 con 0x5598c7473d20 2024-09-19T08:03:54.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f60477fe640 1 -- 172.21.15.98:0/1837670414 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f6058099fd0 con 0x5598c7473d20 2024-09-19T08:03:54.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f6070ddd740 1 -- 172.21.15.98:0/1837670414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60580cd150 con 0x5598c7473d20 2024-09-19T08:03:54.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f60477fe640 1 -- 172.21.15.98:0/1837670414 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f605800f850 con 0x5598c7473d20 2024-09-19T08:03:54.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f6070ddd740 10 client.15726 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:54.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.821+0000 7f6070ddd740 1 -- 172.21.15.98:0/1837670414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5598c74f8200 con 0x5598c7473d20 2024-09-19T08:03:54.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.822+0000 7f60477fe640 1 -- 172.21.15.98:0/1837670414 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60580cd150 con 0x5598c7473d20 2024-09-19T08:03:54.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.822+0000 7f60477fe640 1 -- 172.21.15.98:0/1837670414 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6058098340 con 0x5598c7473d20 2024-09-19T08:03:54.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.822+0000 7f6070ddd740 10 client.15726 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:54.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.822+0000 7f6070ddd740 10 client.15726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:03:54.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.822+0000 7f6070ddd740 20 client.15726 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:54.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.822+0000 7f6070ddd740 1 --2- 172.21.15.98:0/1837670414 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5598c74f83d0 0x5598c74fa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:54.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.822+0000 7f6070ddd740 4 client.15726 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:03:54.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.822+0000 7f6070ddd740 1 -- 172.21.15.98:0/1837670414 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5598c73dfd70 con 0x5598c74f83d0 2024-09-19T08:03:54.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.822+0000 7f60677fe640 1 --2- 172.21.15.98:0/1837670414 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5598c74f83d0 0x5598c74fa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:54.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f60477fe640 10 client.15726 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:54.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f60677fe640 1 --2- 172.21.15.98:0/1837670414 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5598c74f83d0 0x5598c74fa7c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7211e37820c7f4a server_cookie=8fd8ffc31f9d1676 in_seq=0 out_seq=0 2024-09-19T08:03:54.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f60477fe640 1 -- 172.21.15.98:0/1837670414 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5598c73dfd70 con 0x5598c74f83d0 2024-09-19T08:03:54.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f60477fe640 10 client.15726 handle_command_reply: tid=0 2024-09-19T08:03:54.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f6070ddd740 1 client.15726 shutdown 2024-09-19T08:03:54.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f6070ddd740 1 -- 172.21.15.98:0/1837670414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c7473d20 msgr2=0x5598c7485d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:54.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f6070ddd740 1 --2- 172.21.15.98:0/1837670414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c7473d20 0x5598c7485d90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6058084fd0 tx=0x7f60580a4a80 comp rx=0 tx=0).stop 2024-09-19T08:03:54.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f6070ddd740 1 -- 172.21.15.98:0/1837670414 shutdown_connections 2024-09-19T08:03:54.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f6070ddd740 1 --2- 172.21.15.98:0/1837670414 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5598c74f83d0 0x5598c74fa7c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f60477fe640 0 client.15726 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:54.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f6070ddd740 1 --2- 172.21.15.98:0/1837670414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5598c7481560 0x5598c74819e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f6070ddd740 1 --2- 172.21.15.98:0/1837670414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598c747d550 0x5598c7481020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f6070ddd740 1 --2- 172.21.15.98:0/1837670414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598c7473d20 0x5598c7485d90 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f6070ddd740 1 -- 172.21.15.98:0/1837670414 >> 172.21.15.98:0/1837670414 conn(0x5598c743b310 msgr2=0x5598c747b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:54.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f6070ddd740 1 -- 172.21.15.98:0/1837670414 shutdown_connections 2024-09-19T08:03:54.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f6070ddd740 1 -- 172.21.15.98:0/1837670414 wait complete. 2024-09-19T08:03:54.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.823+0000 7f6070ddd740 20 client.15726 trim_cache size 0 max 16384 2024-09-19T08:03:54.828 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:03:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.824+0000 7f606d7a2640 1 -- 172.21.15.98:0/1941375722 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f60500a34d0 msgr2=0x7f60500a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.824+0000 7f606d7a2640 1 --2- 172.21.15.98:0/1941375722 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f60500a34d0 0x7f60500a5990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6060009430 tx=0x7f60600095c0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.824+0000 7f606d7a2640 1 -- 172.21.15.98:0/1941375722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6068102c50 msgr2=0x7f60681008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.824+0000 7f606d7a2640 1 --2- 172.21.15.98:0/1941375722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6068102c50 0x7f60681008b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f605800d950 tx=0x7f605800de10 comp rx=0 tx=0).stop 2024-09-19T08:03:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.824+0000 7f606d7a2640 1 -- 172.21.15.98:0/1941375722 shutdown_connections 2024-09-19T08:03:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.824+0000 7f606d7a2640 1 --2- 172.21.15.98:0/1941375722 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f60500a34d0 0x7f60500a5990 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.824+0000 7f606d7a2640 1 --2- 172.21.15.98:0/1941375722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f606810bff0 0x7f6068101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.824+0000 7f606d7a2640 1 --2- 172.21.15.98:0/1941375722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6068103570 0x7f6068100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.824+0000 7f606d7a2640 1 --2- 172.21.15.98:0/1941375722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6068102c50 0x7f60681008b0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.824+0000 7f606d7a2640 1 -- 172.21.15.98:0/1941375722 >> 172.21.15.98:0/1941375722 conn(0x7f60680fc610 msgr2=0x7f60680fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.824+0000 7f606d7a2640 1 -- 172.21.15.98:0/1941375722 shutdown_connections 2024-09-19T08:03:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:54.824+0000 7f606d7a2640 1 -- 172.21.15.98:0/1941375722 wait complete. 2024-09-19T08:03:55.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:54 smithi162 ceph-mon[35260]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-09-19T08:03:55.149 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:54 smithi098 ceph-mon[31333]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-09-19T08:03:55.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:54 smithi139 ceph-mon[34874]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-09-19T08:03:56.584 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:03:56.585 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:03:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:56.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:56 smithi098 ceph-mon[31333]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-09-19T08:03:57.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:56 smithi162 ceph-mon[35260]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-09-19T08:03:57.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:56 smithi139 ceph-mon[34874]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-09-19T08:03:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.418+0000 7f122485b640 1 -- 172.21.15.98:0/3610463709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1220102570 msgr2=0x7f12201029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.418+0000 7f122485b640 1 --2- 172.21.15.98:0/3610463709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1220102570 0x7f12201029b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1214009900 tx=0x7f121402a480 comp rx=0 tx=0).stop 2024-09-19T08:03:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.418+0000 7f122485b640 1 -- 172.21.15.98:0/3610463709 shutdown_connections 2024-09-19T08:03:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.418+0000 7f122485b640 1 --2- 172.21.15.98:0/3610463709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1220102ef0 0x7f122010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.418+0000 7f122485b640 1 --2- 172.21.15.98:0/3610463709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1220102570 0x7f12201029b0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.418+0000 7f122485b640 1 --2- 172.21.15.98:0/3610463709 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1220100e20 0x7f1220101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.418+0000 7f122485b640 1 -- 172.21.15.98:0/3610463709 >> 172.21.15.98:0/3610463709 conn(0x7f12200fc7d0 msgr2=0x7f12200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.418+0000 7f122485b640 1 -- 172.21.15.98:0/3610463709 shutdown_connections 2024-09-19T08:03:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.418+0000 7f122485b640 1 -- 172.21.15.98:0/3610463709 wait complete. 2024-09-19T08:03:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.419+0000 7f122485b640 1 Processor -- start 2024-09-19T08:03:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.419+0000 7f122485b640 1 -- start start 2024-09-19T08:03:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.419+0000 7f122485b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1220100e20 0x7f1220197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.419+0000 7f122485b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1220102570 0x7f1220197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f122485b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1220102ef0 0x7f1220197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f122485b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1220110f80 con 0x7f1220100e20 2024-09-19T08:03:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f122485b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1220110e00 con 0x7f1220102570 2024-09-19T08:03:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f122485b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1220111100 con 0x7f1220102ef0 2024-09-19T08:03:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f121e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1220100e20 0x7f1220197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f121e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1220100e20 0x7f1220197100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48648/0 (socket says 172.21.15.98:48648) 2024-09-19T08:03:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f121e59c640 1 -- 172.21.15.98:0/3818955766 learned_addr learned my addr 172.21.15.98:0/3818955766 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f121ed9d640 1 --2- 172.21.15.98:0/3818955766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1220102ef0 0x7f1220197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f121dd9b640 1 --2- 172.21.15.98:0/3818955766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1220102570 0x7f1220197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f121e59c640 1 -- 172.21.15.98:0/3818955766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1220102ef0 msgr2=0x7f1220197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f121e59c640 1 --2- 172.21.15.98:0/3818955766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1220102ef0 0x7f1220197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f121e59c640 1 -- 172.21.15.98:0/3818955766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1220102570 msgr2=0x7f1220197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f121e59c640 1 --2- 172.21.15.98:0/3818955766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1220102570 0x7f1220197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f121e59c640 1 -- 172.21.15.98:0/3818955766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12201a6970 con 0x7f1220100e20 2024-09-19T08:03:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.420+0000 7f121e59c640 1 --2- 172.21.15.98:0/3818955766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1220100e20 0x7f1220197100 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f121000d950 tx=0x7f121000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.421+0000 7f120f7fe640 1 -- 172.21.15.98:0/3818955766 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210017070 con 0x7f1220100e20 2024-09-19T08:03:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.421+0000 7f120f7fe640 1 -- 172.21.15.98:0/3818955766 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1210009c30 con 0x7f1220100e20 2024-09-19T08:03:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.421+0000 7f122485b640 1 -- 172.21.15.98:0/3818955766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12201a6c60 con 0x7f1220100e20 2024-09-19T08:03:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.421+0000 7f120f7fe640 1 -- 172.21.15.98:0/3818955766 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210017070 con 0x7f1220100e20 2024-09-19T08:03:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.421+0000 7f122485b640 1 -- 172.21.15.98:0/3818955766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12201a71a0 con 0x7f1220100e20 2024-09-19T08:03:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.423+0000 7f1226c34740 1 Processor -- start 2024-09-19T08:03:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.424+0000 7f1226c34740 1 -- start start 2024-09-19T08:03:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.424+0000 7f1226c34740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b03b60 0x558799b03f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.424+0000 7f1226c34740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b04480 0x558799b0e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.424+0000 7f1226c34740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b11130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.424+0000 7f1226c34740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558799a69280 con 0x558799b0ed40 2024-09-19T08:03:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.424+0000 7f1226c34740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558799a10b10 con 0x558799b04480 2024-09-19T08:03:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.424+0000 7f1226c34740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558799a65240 con 0x558799b03b60 2024-09-19T08:03:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.424+0000 7f121dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b11130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.424+0000 7f121dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b11130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48660/0 (socket says 172.21.15.98:48660) 2024-09-19T08:03:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.424+0000 7f121dd9b640 1 -- 172.21.15.98:0/3803568346 learned_addr learned my addr 172.21.15.98:0/3803568346 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.424+0000 7f121ed9d640 1 --2- 172.21.15.98:0/3803568346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b03b60 0x558799b03f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.425+0000 7f121dd9b640 1 -- 172.21.15.98:0/3803568346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b03b60 msgr2=0x558799b03f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.425+0000 7f121dd9b640 1 --2- 172.21.15.98:0/3803568346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b03b60 0x558799b03f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.425+0000 7f121dd9b640 1 -- 172.21.15.98:0/3803568346 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b04480 msgr2=0x558799b0e800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T08:03:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.425+0000 7f121dd9b640 1 --2- 172.21.15.98:0/3803568346 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b04480 0x558799b0e800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.425+0000 7f121dd9b640 1 -- 172.21.15.98:0/3803568346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558799a1d1f0 con 0x558799b0ed40 2024-09-19T08:03:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.425+0000 7f121dd9b640 1 --2- 172.21.15.98:0/3803568346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b11130 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f121402a9a0 tx=0x7f1214005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d8399addfbb670 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.425+0000 7f120cff9640 1 -- 172.21.15.98:0/3803568346 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214035030 con 0x558799b0ed40 2024-09-19T08:03:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.425+0000 7f120cff9640 1 -- 172.21.15.98:0/3803568346 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1214005520 con 0x558799b0ed40 2024-09-19T08:03:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.425+0000 7f120cff9640 1 -- 172.21.15.98:0/3803568346 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12140035f0 con 0x558799b0ed40 2024-09-19T08:03:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.425+0000 7f120f7fe640 1 -- 172.21.15.98:0/3818955766 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f12100109d0 con 0x7f1220100e20 2024-09-19T08:03:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.426+0000 7f1226c34740 1 -- 172.21.15.98:0/3803568346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 msgr2=0x558799b11130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.426+0000 7f1226c34740 1 --2- 172.21.15.98:0/3803568346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b11130 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f121402a9a0 tx=0x7f1214005e60 comp rx=0 tx=0).stop 2024-09-19T08:03:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.426+0000 7f1226c34740 1 -- 172.21.15.98:0/3803568346 shutdown_connections 2024-09-19T08:03:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.426+0000 7f1226c34740 1 --2- 172.21.15.98:0/3803568346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b11130 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.426+0000 7f1226c34740 1 --2- 172.21.15.98:0/3803568346 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b04480 0x558799b0e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.426+0000 7f1226c34740 1 --2- 172.21.15.98:0/3803568346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b03b60 0x558799b03f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.426+0000 7f1226c34740 1 -- 172.21.15.98:0/3803568346 >> 172.21.15.98:0/3803568346 conn(0x558799b000f0 msgr2=0x558799b004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.426+0000 7f1226c34740 1 -- 172.21.15.98:0/3803568346 shutdown_connections 2024-09-19T08:03:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.427+0000 7f1226c34740 1 -- 172.21.15.98:0/3803568346 wait complete. 2024-09-19T08:03:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.427+0000 7f120f7fe640 1 --2- 172.21.15.98:0/3818955766 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f12080a37e0 0x7f12080a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.427+0000 7f120f7fe640 1 -- 172.21.15.98:0/3818955766 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f1210009a30 con 0x7f1220100e20 2024-09-19T08:03:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.427+0000 7f1226c34740 1 -- start start 2024-09-19T08:03:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.427+0000 7f1226c34740 1 Processor -- start 2024-09-19T08:03:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.427+0000 7f121dd9b640 1 --2- 172.21.15.98:0/3818955766 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f12080a37e0 0x7f12080a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.427+0000 7f1226c34740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b03b60 0x558799b0bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.427+0000 7f1226c34740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b04480 0x558799b0c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.427+0000 7f1226c34740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b0c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f121ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b0c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f121ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b0c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48664/0 (socket says 172.21.15.98:48664) 2024-09-19T08:03:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f121ed9d640 1 -- 172.21.15.98:0/2750759844 learned_addr learned my addr 172.21.15.98:0/2750759844 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f1226c34740 1 -- 172.21.15.98:0/2750759844 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558799b119b0 con 0x558799b0ed40 2024-09-19T08:03:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f1226c34740 1 -- 172.21.15.98:0/2750759844 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558799a23f10 con 0x558799b04480 2024-09-19T08:03:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f1226c34740 1 -- 172.21.15.98:0/2750759844 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558799b11b30 con 0x558799b03b60 2024-09-19T08:03:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f121dd9b640 1 --2- 172.21.15.98:0/3818955766 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f12080a37e0 0x7f12080a5ca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f12140314e0 tx=0x7f121402a480 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f121dd9b640 1 --2- 172.21.15.98:0/2750759844 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b04480 0x558799b0c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f1226c34740 10 client.0 _ll_register_callbacks cb 0x5587999eca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f121e59c640 1 --2- 172.21.15.98:0/2750759844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b03b60 0x558799b0bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f121ed9d640 1 -- 172.21.15.98:0/2750759844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b03b60 msgr2=0x558799b0bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f121ed9d640 1 --2- 172.21.15.98:0/2750759844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b03b60 0x558799b0bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f121ed9d640 1 -- 172.21.15.98:0/2750759844 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b04480 msgr2=0x558799b0c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f121ed9d640 1 --2- 172.21.15.98:0/2750759844 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b04480 0x558799b0c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f121ed9d640 1 -- 172.21.15.98:0/2750759844 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587997da6a0 con 0x558799b0ed40 2024-09-19T08:03:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f1226c34740 1 -- 172.21.15.98:0/2750759844 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1214005520 con 0x558799b0ed40 2024-09-19T08:03:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f121ed9d640 1 --2- 172.21.15.98:0/2750759844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b0c690 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1218011fa0 tx=0x7f121800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54e3659ce6e391f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.428+0000 7f1205ffb640 10 client.15738 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.429+0000 7f1205ffb640 1 -- 172.21.15.98:0/2750759844 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121801e020 con 0x558799b0ed40 2024-09-19T08:03:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.429+0000 7f1205ffb640 1 -- 172.21.15.98:0/2750759844 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f1218004b20 con 0x558799b0ed40 2024-09-19T08:03:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.429+0000 7f1205ffb640 1 -- 172.21.15.98:0/2750759844 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1218004d60 con 0x558799b0ed40 2024-09-19T08:03:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.429+0000 7f1226c34740 10 client.15738 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.429+0000 7f1226c34740 1 -- 172.21.15.98:0/2750759844 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=28}) -- 0x558799b1d540 con 0x558799b0ed40 2024-09-19T08:03:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.429+0000 7f1205ffb640 1 -- 172.21.15.98:0/2750759844 <== mon.0 v2:172.21.15.98:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1218019070 con 0x558799b0ed40 2024-09-19T08:03:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.429+0000 7f1226c34740 10 client.15738 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.429+0000 7f1226c34740 10 client.15738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:03:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.429+0000 7f1226c34740 20 client.15738 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.429+0000 7f1226c34740 1 --2- 172.21.15.98:0/2750759844 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558799b1e9b0 0x558799b20e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.429+0000 7f1226c34740 4 client.15738 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.429+0000 7f1226c34740 1 -- 172.21.15.98:0/2750759844 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558799a767f0 con 0x558799b1e9b0 2024-09-19T08:03:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.430+0000 7f121e59c640 1 --2- 172.21.15.98:0/2750759844 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558799b1e9b0 0x558799b20e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.430+0000 7f121e59c640 1 --2- 172.21.15.98:0/2750759844 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558799b1e9b0 0x558799b20e20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=328731be498993b2 server_cookie=ae60d78ff64fbfaf in_seq=0 out_seq=0 2024-09-19T08:03:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.430+0000 7f1205ffb640 10 client.15738 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1205ffb640 1 -- 172.21.15.98:0/2750759844 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558799a767f0 con 0x558799b1e9b0 2024-09-19T08:03:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1205ffb640 10 client.15738 handle_command_reply: tid=0 2024-09-19T08:03:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1226c34740 1 client.15738 shutdown 2024-09-19T08:03:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1226c34740 1 -- 172.21.15.98:0/2750759844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 msgr2=0x558799b0c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1226c34740 1 --2- 172.21.15.98:0/2750759844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b0c690 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1218011fa0 tx=0x7f121800f790 comp rx=0 tx=0).stop 2024-09-19T08:03:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1226c34740 1 -- 172.21.15.98:0/2750759844 shutdown_connections 2024-09-19T08:03:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1226c34740 1 --2- 172.21.15.98:0/2750759844 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558799b1e9b0 0x558799b20e20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1226c34740 1 --2- 172.21.15.98:0/2750759844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b0c690 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1226c34740 1 --2- 172.21.15.98:0/2750759844 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b04480 0x558799b0c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1226c34740 1 --2- 172.21.15.98:0/2750759844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b03b60 0x558799b0bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1226c34740 1 -- 172.21.15.98:0/2750759844 >> 172.21.15.98:0/2750759844 conn(0x558799b000f0 msgr2=0x558799b05dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1205ffb640 0 client.15738 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1226c34740 1 -- 172.21.15.98:0/2750759844 shutdown_connections 2024-09-19T08:03:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1226c34740 1 -- 172.21.15.98:0/2750759844 wait complete. 2024-09-19T08:03:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.431+0000 7f1226c34740 20 client.15738 trim_cache size 0 max 16384 2024-09-19T08:03:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.449+0000 7f1226c34740 1 Processor -- start 2024-09-19T08:03:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.449+0000 7f1226c34740 1 -- start start 2024-09-19T08:03:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.449+0000 7f1226c34740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b03b60 0x558799b26630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.449+0000 7f1226c34740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b04480 0x558799b26b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.449+0000 7f1226c34740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b16da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.449+0000 7f1226c34740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f12201af930 con 0x558799b0ed40 2024-09-19T08:03:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.449+0000 7f1226c34740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f12201b1470 con 0x558799b03b60 2024-09-19T08:03:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.449+0000 7f1226c34740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f12201b0000 con 0x558799b04480 2024-09-19T08:03:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.449+0000 7f121ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b16da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.449+0000 7f121ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b16da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48670/0 (socket says 172.21.15.98:48670) 2024-09-19T08:03:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.449+0000 7f121ed9d640 1 -- 172.21.15.98:0/1397423694 learned_addr learned my addr 172.21.15.98:0/1397423694 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.449+0000 7f121dd9b640 1 --2- 172.21.15.98:0/1397423694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b04480 0x558799b26b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.449+0000 7f121e59c640 1 --2- 172.21.15.98:0/1397423694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b03b60 0x558799b26630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.450+0000 7f121ed9d640 1 -- 172.21.15.98:0/1397423694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b04480 msgr2=0x558799b26b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.450+0000 7f121ed9d640 1 --2- 172.21.15.98:0/1397423694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b04480 0x558799b26b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.450+0000 7f121ed9d640 1 -- 172.21.15.98:0/1397423694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b03b60 msgr2=0x558799b26630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.450+0000 7f121ed9d640 1 --2- 172.21.15.98:0/1397423694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b03b60 0x558799b26630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.450+0000 7f121ed9d640 1 -- 172.21.15.98:0/1397423694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558799b21400 con 0x558799b0ed40 2024-09-19T08:03:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.450+0000 7f121e59c640 1 --2- 172.21.15.98:0/1397423694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b03b60 0x558799b26630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.450+0000 7f121ed9d640 1 --2- 172.21.15.98:0/1397423694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b16da0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f121800f7d0 tx=0x7f1218018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5648586c762a73f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.450+0000 7f1207fff640 1 -- 172.21.15.98:0/1397423694 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121801e070 con 0x558799b0ed40 2024-09-19T08:03:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.450+0000 7f1207fff640 1 -- 172.21.15.98:0/1397423694 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1214005520 con 0x558799b0ed40 2024-09-19T08:03:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.450+0000 7f1207fff640 1 -- 172.21.15.98:0/1397423694 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218028060 con 0x558799b0ed40 2024-09-19T08:03:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.451+0000 7f1226c34740 1 -- 172.21.15.98:0/1397423694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 msgr2=0x558799b16da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.451+0000 7f1226c34740 1 --2- 172.21.15.98:0/1397423694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b16da0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f121800f7d0 tx=0x7f1218018000 comp rx=0 tx=0).stop 2024-09-19T08:03:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.451+0000 7f1226c34740 1 -- 172.21.15.98:0/1397423694 shutdown_connections 2024-09-19T08:03:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.451+0000 7f1226c34740 1 --2- 172.21.15.98:0/1397423694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b0ed40 0x558799b16da0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.451+0000 7f1226c34740 1 --2- 172.21.15.98:0/1397423694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b04480 0x558799b26b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.451+0000 7f1226c34740 1 --2- 172.21.15.98:0/1397423694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b03b60 0x558799b26630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.451+0000 7f1226c34740 1 -- 172.21.15.98:0/1397423694 >> 172.21.15.98:0/1397423694 conn(0x558799b000f0 msgr2=0x558799b05dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.451+0000 7f1226c34740 1 -- 172.21.15.98:0/1397423694 shutdown_connections 2024-09-19T08:03:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.451+0000 7f1226c34740 1 -- 172.21.15.98:0/1397423694 wait complete. 2024-09-19T08:03:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.452+0000 7f1226c34740 1 -- start start 2024-09-19T08:03:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.452+0000 7f1226c34740 1 Processor -- start 2024-09-19T08:03:57.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.452+0000 7f1226c34740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b03b60 0x558799b14f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.452+0000 7f1226c34740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b04480 0x558799b15450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.452+0000 7f1226c34740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b0ed40 0x558799b90e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.452+0000 7f1226c34740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f12201b0da0 con 0x558799b04480 2024-09-19T08:03:57.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.452+0000 7f1226c34740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f12201b06d0 con 0x558799b03b60 2024-09-19T08:03:57.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.452+0000 7f1226c34740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f12201ad910 con 0x558799b0ed40 2024-09-19T08:03:57.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.452+0000 7f1226c34740 10 client.0 _ll_register_callbacks cb 0x7f1218009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:57.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.452+0000 7f121dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b04480 0x558799b15450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.452+0000 7f121dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b04480 0x558799b15450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48676/0 (socket says 172.21.15.98:48676) 2024-09-19T08:03:57.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.452+0000 7f121dd9b640 1 -- 172.21.15.98:0/2226152156 learned_addr learned my addr 172.21.15.98:0/2226152156 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:57.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.452+0000 7f121e59c640 1 --2- 172.21.15.98:0/2226152156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b03b60 0x558799b14f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.452+0000 7f121ed9d640 1 --2- 172.21.15.98:0/2226152156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b0ed40 0x558799b90e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.453+0000 7f121dd9b640 1 -- 172.21.15.98:0/2226152156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b0ed40 msgr2=0x558799b90e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:57.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.453+0000 7f121dd9b640 1 --2- 172.21.15.98:0/2226152156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b0ed40 0x558799b90e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.453+0000 7f121dd9b640 1 -- 172.21.15.98:0/2226152156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b03b60 msgr2=0x558799b14f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.453+0000 7f121dd9b640 1 --2- 172.21.15.98:0/2226152156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b03b60 0x558799b14f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.453+0000 7f121dd9b640 1 -- 172.21.15.98:0/2226152156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558799b91360 con 0x558799b04480 2024-09-19T08:03:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.453+0000 7f121dd9b640 1 --2- 172.21.15.98:0/2226152156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b04480 0x558799b15450 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1214009900 tx=0x7f1214004ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5489e45f7fba3f6b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.453+0000 7f1205ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.453+0000 7f1205ffb640 1 -- 172.21.15.98:0/2226152156 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121405f060 con 0x558799b04480 2024-09-19T08:03:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.453+0000 7f1205ffb640 1 -- 172.21.15.98:0/2226152156 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f1214003800 con 0x558799b04480 2024-09-19T08:03:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.453+0000 7f1226c34740 1 -- 172.21.15.98:0/2226152156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1214005520 con 0x558799b04480 2024-09-19T08:03:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.454+0000 7f1205ffb640 1 -- 172.21.15.98:0/2226152156 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1214035070 con 0x558799b04480 2024-09-19T08:03:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.454+0000 7f1226c34740 10 client.15744 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.454+0000 7f1226c34740 1 -- 172.21.15.98:0/2226152156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558799b924a0 con 0x558799b04480 2024-09-19T08:03:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.454+0000 7f1205ffb640 1 -- 172.21.15.98:0/2226152156 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1214005520 con 0x558799b04480 2024-09-19T08:03:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.454+0000 7f1205ffb640 1 -- 172.21.15.98:0/2226152156 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1214031aa0 con 0x558799b04480 2024-09-19T08:03:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.454+0000 7f1226c34740 10 client.15744 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:57.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.454+0000 7f1226c34740 10 client.15744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:03:57.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.454+0000 7f1226c34740 20 client.15744 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:57.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.454+0000 7f1226c34740 1 --2- 172.21.15.98:0/2226152156 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558799b92770 0x558799b94b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:57.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.454+0000 7f1226c34740 4 client.15744 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:03:57.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.454+0000 7f1226c34740 1 -- 172.21.15.98:0/2226152156 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558799b07850 con 0x558799b92770 2024-09-19T08:03:57.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.455+0000 7f121e59c640 1 --2- 172.21.15.98:0/2226152156 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558799b92770 0x558799b94b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:57.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.455+0000 7f121e59c640 1 --2- 172.21.15.98:0/2226152156 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558799b92770 0x558799b94b60 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1893831296b9464 server_cookie=ff7d080a55458abd in_seq=0 out_seq=0 2024-09-19T08:03:57.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.455+0000 7f1205ffb640 10 client.15744 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:57.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.456+0000 7f1205ffb640 1 -- 172.21.15.98:0/2226152156 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558799b07850 con 0x558799b92770 2024-09-19T08:03:57.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.456+0000 7f1205ffb640 10 client.15744 handle_command_reply: tid=0 2024-09-19T08:03:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.456+0000 7f1226c34740 1 client.15744 shutdown 2024-09-19T08:03:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.456+0000 7f1226c34740 1 -- 172.21.15.98:0/2226152156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b04480 msgr2=0x558799b15450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.456+0000 7f1226c34740 1 --2- 172.21.15.98:0/2226152156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b04480 0x558799b15450 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1214009900 tx=0x7f1214004ba0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.457+0000 7f1226c34740 1 -- 172.21.15.98:0/2226152156 shutdown_connections 2024-09-19T08:03:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.457+0000 7f1226c34740 1 --2- 172.21.15.98:0/2226152156 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558799b92770 0x558799b94b60 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.457+0000 7f1226c34740 1 --2- 172.21.15.98:0/2226152156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558799b0ed40 0x558799b90e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.457+0000 7f1205ffb640 0 client.15744 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.457+0000 7f1226c34740 1 --2- 172.21.15.98:0/2226152156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558799b04480 0x558799b15450 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.457+0000 7f1226c34740 1 --2- 172.21.15.98:0/2226152156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558799b03b60 0x558799b14f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.457+0000 7f1226c34740 1 -- 172.21.15.98:0/2226152156 >> 172.21.15.98:0/2226152156 conn(0x558799b000f0 msgr2=0x558799afea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.457+0000 7f1226c34740 1 -- 172.21.15.98:0/2226152156 shutdown_connections 2024-09-19T08:03:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.457+0000 7f1226c34740 1 -- 172.21.15.98:0/2226152156 wait complete. 2024-09-19T08:03:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.457+0000 7f1226c34740 20 client.15744 trim_cache size 0 max 16384 2024-09-19T08:03:57.461 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:03:57.461 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:03:57.461 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "a346458c-274b-4ab6-8c2c-cdb733a89881", 2024-09-19T08:03:57.461 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:03:57.461 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:03:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.458+0000 7f122485b640 1 -- 172.21.15.98:0/3818955766 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f12080a37e0 msgr2=0x7f12080a5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.458+0000 7f122485b640 1 --2- 172.21.15.98:0/3818955766 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f12080a37e0 0x7f12080a5ca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f12140314e0 tx=0x7f121402a480 comp rx=0 tx=0).stop 2024-09-19T08:03:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.458+0000 7f122485b640 1 -- 172.21.15.98:0/3818955766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1220100e20 msgr2=0x7f1220197100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.458+0000 7f122485b640 1 --2- 172.21.15.98:0/3818955766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1220100e20 0x7f1220197100 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f121000d950 tx=0x7f121000de10 comp rx=0 tx=0).stop 2024-09-19T08:03:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.458+0000 7f122485b640 1 -- 172.21.15.98:0/3818955766 shutdown_connections 2024-09-19T08:03:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.459+0000 7f122485b640 1 --2- 172.21.15.98:0/3818955766 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f12080a37e0 0x7f12080a5ca0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.459+0000 7f122485b640 1 --2- 172.21.15.98:0/3818955766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1220102ef0 0x7f1220197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.459+0000 7f122485b640 1 --2- 172.21.15.98:0/3818955766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1220102570 0x7f1220197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.459+0000 7f122485b640 1 --2- 172.21.15.98:0/3818955766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1220100e20 0x7f1220197100 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.459+0000 7f122485b640 1 -- 172.21.15.98:0/3818955766 >> 172.21.15.98:0/3818955766 conn(0x7f12200fc7d0 msgr2=0x7f12200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.459+0000 7f122485b640 1 -- 172.21.15.98:0/3818955766 shutdown_connections 2024-09-19T08:03:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:57.459+0000 7f122485b640 1 -- 172.21.15.98:0/3818955766 wait complete. 2024-09-19T08:03:58.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:03:58.379 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:58.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:03:58.763 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:58 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:03:58.764 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:58 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:03:58.764 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:58 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:03:58.764 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:58 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:03:58.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:58 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:03:58.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:58 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:03:58.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:58 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:03:58.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:58 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:03:58.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.876+0000 7f5abda7c640 1 -- 172.21.15.98:0/1347802871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ab8102d90 msgr2=0x7f5ab8103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:58.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.876+0000 7f5abda7c640 1 --2- 172.21.15.98:0/1347802871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ab8102d90 0x7f5ab8103170 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800fb40 tx=0x7f5aa80329b0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.876+0000 7f5abda7c640 1 -- 172.21.15.98:0/1347802871 shutdown_connections 2024-09-19T08:03:58.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.876+0000 7f5abda7c640 1 --2- 172.21.15.98:0/1347802871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ab8108290 0x7f5ab810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.876+0000 7f5abda7c640 1 --2- 172.21.15.98:0/1347802871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ab8103740 0x7f5ab8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.876+0000 7f5abda7c640 1 --2- 172.21.15.98:0/1347802871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ab8102d90 0x7f5ab8103170 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.876+0000 7f5abda7c640 1 -- 172.21.15.98:0/1347802871 >> 172.21.15.98:0/1347802871 conn(0x7f5ab80fc7d0 msgr2=0x7f5ab80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:58.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.876+0000 7f5abda7c640 1 -- 172.21.15.98:0/1347802871 shutdown_connections 2024-09-19T08:03:58.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.876+0000 7f5abda7c640 1 -- 172.21.15.98:0/1347802871 wait complete. 2024-09-19T08:03:58.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.877+0000 7f5abda7c640 1 Processor -- start 2024-09-19T08:03:58.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.877+0000 7f5abda7c640 1 -- start start 2024-09-19T08:03:58.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.877+0000 7f5abda7c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ab8102d90 0x7f5ab8073980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.877+0000 7f5abda7c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ab8103740 0x7f5ab8073ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.877+0000 7f5abda7c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ab8108290 0x7f5ab806cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.877+0000 7f5abda7c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab810ff30 con 0x7f5ab8103740 2024-09-19T08:03:58.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.877+0000 7f5abda7c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab810fdb0 con 0x7f5ab8102d90 2024-09-19T08:03:58.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.877+0000 7f5abda7c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab81100b0 con 0x7f5ab8108290 2024-09-19T08:03:58.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.877+0000 7f5ab6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ab8103740 0x7f5ab8073ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.877+0000 7f5ab6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ab8103740 0x7f5ab8073ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48688/0 (socket says 172.21.15.98:48688) 2024-09-19T08:03:58.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.877+0000 7f5ab6ffd640 1 -- 172.21.15.98:0/2712843029 learned_addr learned my addr 172.21.15.98:0/2712843029 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:58.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.877+0000 7f5ab7fff640 1 --2- 172.21.15.98:0/2712843029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ab8108290 0x7f5ab806cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.877+0000 7f5ab77fe640 1 --2- 172.21.15.98:0/2712843029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ab8102d90 0x7f5ab8073980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.878+0000 7f5ab6ffd640 1 -- 172.21.15.98:0/2712843029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ab8108290 msgr2=0x7f5ab806cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:58.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.878+0000 7f5ab6ffd640 1 --2- 172.21.15.98:0/2712843029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ab8108290 0x7f5ab806cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.878+0000 7f5ab6ffd640 1 -- 172.21.15.98:0/2712843029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ab8102d90 msgr2=0x7f5ab8073980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:58.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.878+0000 7f5ab6ffd640 1 --2- 172.21.15.98:0/2712843029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ab8102d90 0x7f5ab8073980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.878+0000 7f5ab6ffd640 1 -- 172.21.15.98:0/2712843029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab806d490 con 0x7f5ab8103740 2024-09-19T08:03:58.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.878+0000 7f5ab6ffd640 1 --2- 172.21.15.98:0/2712843029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ab8103740 0x7f5ab8073ec0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00e9f0 tx=0x7f5aac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:58.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.878+0000 7f5ab4ff9640 1 -- 172.21.15.98:0/2712843029 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac00ccc0 con 0x7f5ab8103740 2024-09-19T08:03:58.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.878+0000 7f5ab4ff9640 1 -- 172.21.15.98:0/2712843029 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aac005500 con 0x7f5ab8103740 2024-09-19T08:03:58.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.878+0000 7f5abda7c640 1 -- 172.21.15.98:0/2712843029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab806d780 con 0x7f5ab8103740 2024-09-19T08:03:58.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.878+0000 7f5abda7c640 1 -- 172.21.15.98:0/2712843029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab8071790 con 0x7f5ab8103740 2024-09-19T08:03:58.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.878+0000 7f5ab4ff9640 1 -- 172.21.15.98:0/2712843029 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac010690 con 0x7f5ab8103740 2024-09-19T08:03:58.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.881+0000 7f5abfe55740 1 Processor -- start 2024-09-19T08:03:58.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.881+0000 7f5abfe55740 1 -- start start 2024-09-19T08:03:58.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.882+0000 7f5abfe55740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b1705880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.882+0000 7f5abfe55740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1710140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.882+0000 7f5abfe55740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 0x55c7b1712a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.882+0000 7f5abfe55740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c7b166b2e0 con 0x55c7b1705dc0 2024-09-19T08:03:58.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.882+0000 7f5abfe55740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c7b1612ce0 con 0x55c7b1710680 2024-09-19T08:03:58.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.882+0000 7f5abfe55740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c7b16672a0 con 0x55c7b17054a0 2024-09-19T08:03:58.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.882+0000 7f5ab77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1710140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.882+0000 7f5ab77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1710140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48704/0 (socket says 172.21.15.98:48704) 2024-09-19T08:03:58.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.882+0000 7f5ab77fe640 1 -- 172.21.15.98:0/3255818890 learned_addr learned my addr 172.21.15.98:0/3255818890 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:58.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.882+0000 7f5ab7fff640 1 --2- 172.21.15.98:0/3255818890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b1705880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.883+0000 7f5ab4ff9640 1 -- 172.21.15.98:0/2712843029 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f5aac0046c0 con 0x7f5ab8103740 2024-09-19T08:03:58.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.883+0000 7f5ab77fe640 1 -- 172.21.15.98:0/3255818890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 msgr2=0x55c7b1705880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:58.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.883+0000 7f5ab77fe640 1 --2- 172.21.15.98:0/3255818890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b1705880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.883+0000 7f5ab77fe640 1 -- 172.21.15.98:0/3255818890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 msgr2=0x55c7b1712a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:58.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.883+0000 7f5ab6ffd640 1 --2- 172.21.15.98:0/3255818890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 0x55c7b1712a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.883+0000 7f5ab77fe640 1 --2- 172.21.15.98:0/3255818890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 0x55c7b1712a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.883+0000 7f5ab77fe640 1 -- 172.21.15.98:0/3255818890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7b15f7440 con 0x55c7b1705dc0 2024-09-19T08:03:58.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.883+0000 7f5ab7fff640 1 --2- 172.21.15.98:0/3255818890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b1705880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:58.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.883+0000 7f5ab6ffd640 1 --2- 172.21.15.98:0/3255818890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 0x55c7b1712a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:03:58.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.883+0000 7f5ab77fe640 1 --2- 172.21.15.98:0/3255818890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1710140 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800fb40 tx=0x7f5aa80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607527de5545cb5b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:58.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.884+0000 7f5a9dffb640 1 -- 172.21.15.98:0/3255818890 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8004040 con 0x55c7b1705dc0 2024-09-19T08:03:58.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.884+0000 7f5a9dffb640 1 -- 172.21.15.98:0/3255818890 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aa8047070 con 0x55c7b1705dc0 2024-09-19T08:03:58.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.884+0000 7f5a9dffb640 1 -- 172.21.15.98:0/3255818890 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa800f590 con 0x55c7b1705dc0 2024-09-19T08:03:58.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.884+0000 7f5ab4ff9640 1 --2- 172.21.15.98:0/2712843029 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5aa00a3550 0x7f5aa00a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.885+0000 7f5abfe55740 1 -- 172.21.15.98:0/3255818890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 msgr2=0x55c7b1710140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:58.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.885+0000 7f5abfe55740 1 --2- 172.21.15.98:0/3255818890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1710140 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800fb40 tx=0x7f5aa80047c0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.885+0000 7f5ab7fff640 1 --2- 172.21.15.98:0/2712843029 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5aa00a3550 0x7f5aa00a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.885+0000 7f5ab4ff9640 1 -- 172.21.15.98:0/2712843029 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5aac0d0360 con 0x7f5ab8103740 2024-09-19T08:03:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.886+0000 7f5ab7fff640 1 --2- 172.21.15.98:0/2712843029 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5aa00a3550 0x7f5aa00a5a10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000a620 tx=0x7f5ab000d040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.886+0000 7f5abfe55740 1 -- 172.21.15.98:0/3255818890 shutdown_connections 2024-09-19T08:03:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.886+0000 7f5abfe55740 1 --2- 172.21.15.98:0/3255818890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 0x55c7b1712a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.886+0000 7f5abfe55740 1 --2- 172.21.15.98:0/3255818890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1710140 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.886+0000 7f5abfe55740 1 --2- 172.21.15.98:0/3255818890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b1705880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.886+0000 7f5abfe55740 1 -- 172.21.15.98:0/3255818890 >> 172.21.15.98:0/3255818890 conn(0x55c7b1701a30 msgr2=0x55c7b1701e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:58.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.886+0000 7f5abfe55740 1 -- 172.21.15.98:0/3255818890 shutdown_connections 2024-09-19T08:03:58.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.886+0000 7f5abfe55740 1 -- 172.21.15.98:0/3255818890 wait complete. 2024-09-19T08:03:58.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.886+0000 7f5abfe55740 1 -- start start 2024-09-19T08:03:58.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.887+0000 7f5abfe55740 1 Processor -- start 2024-09-19T08:03:58.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.887+0000 7f5abfe55740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b170ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.887+0000 7f5abfe55740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b170d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.887+0000 7f5abfe55740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 0x55c7b170d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.887+0000 7f5abfe55740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c7b1713250 con 0x55c7b1705dc0 2024-09-19T08:03:58.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.887+0000 7f5abfe55740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c7b1626050 con 0x55c7b1710680 2024-09-19T08:03:58.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.887+0000 7f5abfe55740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c7b17133d0 con 0x55c7b17054a0 2024-09-19T08:03:58.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.887+0000 7f5abfe55740 10 client.0 _ll_register_callbacks cb 0x55c7b1666160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:58.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.888+0000 7f5ab6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b170d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.888+0000 7f5ab6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b170d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48712/0 (socket says 172.21.15.98:48712) 2024-09-19T08:03:58.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.888+0000 7f5ab6ffd640 1 -- 172.21.15.98:0/113914590 learned_addr learned my addr 172.21.15.98:0/113914590 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:58.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.888+0000 7f5ab77fe640 1 --2- 172.21.15.98:0/113914590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b170ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.888+0000 7f5ab7fff640 1 --2- 172.21.15.98:0/113914590 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 0x55c7b170d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.888+0000 7f5ab77fe640 1 -- 172.21.15.98:0/113914590 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 msgr2=0x55c7b170d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:58.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.888+0000 7f5ab77fe640 1 --2- 172.21.15.98:0/113914590 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 0x55c7b170d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.888+0000 7f5ab77fe640 1 -- 172.21.15.98:0/113914590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 msgr2=0x55c7b170d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:58.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.888+0000 7f5ab77fe640 1 --2- 172.21.15.98:0/113914590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b170d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.888+0000 7f5ab77fe640 1 -- 172.21.15.98:0/113914590 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7b13dc3c0 con 0x55c7b17054a0 2024-09-19T08:03:58.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.888+0000 7f5ab77fe640 1 --2- 172.21.15.98:0/113914590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b170ff30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8006010 tx=0x7f5aa800eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1de1f721450372f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:58.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.888+0000 7f5a96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-09-19T08:03:58.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.889+0000 7f5a96ffd640 1 -- 172.21.15.98:0/113914590 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8054070 con 0x55c7b17054a0 2024-09-19T08:03:58.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.889+0000 7f5abfe55740 1 -- 172.21.15.98:0/113914590 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5aa8047070 con 0x55c7b17054a0 2024-09-19T08:03:58.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.889+0000 7f5a96ffd640 1 -- 172.21.15.98:0/113914590 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5aa8038480 con 0x55c7b17054a0 2024-09-19T08:03:58.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.889+0000 7f5a96ffd640 1 -- 172.21.15.98:0/113914590 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5aa800f760 con 0x55c7b17054a0 2024-09-19T08:03:58.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.889+0000 7f5abfe55740 10 client.25159 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:58.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.889+0000 7f5abfe55740 1 -- 172.21.15.98:0/113914590 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c7b1713250 con 0x55c7b17054a0 2024-09-19T08:03:58.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.889+0000 7f5a96ffd640 1 -- 172.21.15.98:0/113914590 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aa8047070 con 0x55c7b17054a0 2024-09-19T08:03:58.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.889+0000 7f5a96ffd640 1 -- 172.21.15.98:0/113914590 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5aa8057020 con 0x55c7b17054a0 2024-09-19T08:03:58.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.889+0000 7f5abfe55740 10 client.25159 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:58.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.889+0000 7f5abfe55740 10 client.25159 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:03:58.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.889+0000 7f5abfe55740 20 client.25159 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:58.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.889+0000 7f5abfe55740 1 --2- 172.21.15.98:0/113914590 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55c7b171fc90 0x55c7b1722080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.889+0000 7f5abfe55740 4 client.25159 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:03:58.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.889+0000 7f5abfe55740 1 -- 172.21.15.98:0/113914590 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7b1678850 con 0x55c7b171fc90 2024-09-19T08:03:58.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.890+0000 7f5ab6ffd640 1 --2- 172.21.15.98:0/113914590 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55c7b171fc90 0x55c7b1722080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.890+0000 7f5a96ffd640 10 client.25159 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:58.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.890+0000 7f5ab6ffd640 1 --2- 172.21.15.98:0/113914590 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55c7b171fc90 0x55c7b1722080 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1282cc73c86eca2c server_cookie=716f308baffd276c in_seq=0 out_seq=0 2024-09-19T08:03:58.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.891+0000 7f5a96ffd640 1 -- 172.21.15.98:0/113914590 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c7b1678850 con 0x55c7b171fc90 2024-09-19T08:03:58.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.891+0000 7f5a96ffd640 10 client.25159 handle_command_reply: tid=0 2024-09-19T08:03:58.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.891+0000 7f5abfe55740 1 client.25159 shutdown 2024-09-19T08:03:58.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.891+0000 7f5abfe55740 1 -- 172.21.15.98:0/113914590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 msgr2=0x55c7b170ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:58.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.891+0000 7f5abfe55740 1 --2- 172.21.15.98:0/113914590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b170ff30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8006010 tx=0x7f5aa800eb90 comp rx=0 tx=0).stop 2024-09-19T08:03:58.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.892+0000 7f5abfe55740 1 -- 172.21.15.98:0/113914590 shutdown_connections 2024-09-19T08:03:58.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.892+0000 7f5abfe55740 1 --2- 172.21.15.98:0/113914590 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55c7b171fc90 0x55c7b1722080 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.892+0000 7f5abfe55740 1 --2- 172.21.15.98:0/113914590 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 0x55c7b170d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.892+0000 7f5abfe55740 1 --2- 172.21.15.98:0/113914590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b170d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.892+0000 7f5abfe55740 1 --2- 172.21.15.98:0/113914590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b170ff30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.892+0000 7f5abfe55740 1 -- 172.21.15.98:0/113914590 >> 172.21.15.98:0/113914590 conn(0x55c7b1701a30 msgr2=0x55c7b1700b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:58.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.892+0000 7f5a96ffd640 0 client.25159 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:58.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.892+0000 7f5abfe55740 1 -- 172.21.15.98:0/113914590 shutdown_connections 2024-09-19T08:03:58.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.892+0000 7f5abfe55740 1 -- 172.21.15.98:0/113914590 wait complete. 2024-09-19T08:03:58.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.892+0000 7f5abfe55740 20 client.25159 trim_cache size 0 max 16384 2024-09-19T08:03:58.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.925+0000 7f5abfe55740 1 Processor -- start 2024-09-19T08:03:58.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.925+0000 7f5abfe55740 1 -- start start 2024-09-19T08:03:58.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.925+0000 7f5abfe55740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b17054a0 0x55c7b171c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.925+0000 7f5abfe55740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1706e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.925+0000 7f5ab77fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b17054a0 0x55c7b171c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.925+0000 7f5ab77fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b17054a0 0x55c7b171c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:53002/0 (socket says 172.21.15.98:53002) 2024-09-19T08:03:58.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5ab77fe640 1 -- 172.21.15.98:0/1506757206 learned_addr learned my addr 172.21.15.98:0/1506757206 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:58.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5abfe55740 1 --2- 172.21.15.98:0/1506757206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b1710680 0x55c7b1727790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5abfe55740 1 -- 172.21.15.98:0/1506757206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab81b3e00 con 0x55c7b1705dc0 2024-09-19T08:03:58.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5abfe55740 1 -- 172.21.15.98:0/1506757206 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab81b5940 con 0x55c7b17054a0 2024-09-19T08:03:58.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5abfe55740 1 -- 172.21.15.98:0/1506757206 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab81b44d0 con 0x55c7b1710680 2024-09-19T08:03:58.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5ab6ffd640 1 --2- 172.21.15.98:0/1506757206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1706e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5ab77fe640 1 -- 172.21.15.98:0/1506757206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b1710680 msgr2=0x55c7b1727790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:58.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5ab77fe640 1 --2- 172.21.15.98:0/1506757206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b1710680 0x55c7b1727790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5ab77fe640 1 -- 172.21.15.98:0/1506757206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 msgr2=0x55c7b1706e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:58.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5ab77fe640 1 --2- 172.21.15.98:0/1506757206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1706e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5ab77fe640 1 -- 172.21.15.98:0/1506757206 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7b17014d0 con 0x55c7b17054a0 2024-09-19T08:03:58.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5ab77fe640 1 --2- 172.21.15.98:0/1506757206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b17054a0 0x55c7b171c4e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80329f0 tx=0x7f5aa800f9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a016e30b7ccef394 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:58 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:03:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:58 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:03:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:58 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:03:58.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:58 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:03:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5a9d7fa640 1 -- 172.21.15.98:0/1506757206 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8054040 con 0x55c7b17054a0 2024-09-19T08:03:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5a9d7fa640 1 -- 172.21.15.98:0/1506757206 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aa8047070 con 0x55c7b17054a0 2024-09-19T08:03:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.926+0000 7f5a9d7fa640 1 -- 172.21.15.98:0/1506757206 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8038bc0 con 0x55c7b17054a0 2024-09-19T08:03:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.927+0000 7f5abfe55740 1 -- 172.21.15.98:0/1506757206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b17054a0 msgr2=0x55c7b171c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.927+0000 7f5abfe55740 1 --2- 172.21.15.98:0/1506757206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b17054a0 0x55c7b171c4e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80329f0 tx=0x7f5aa800f9d0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.928+0000 7f5abfe55740 1 -- 172.21.15.98:0/1506757206 shutdown_connections 2024-09-19T08:03:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.928+0000 7f5abfe55740 1 --2- 172.21.15.98:0/1506757206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b1710680 0x55c7b1727790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.928+0000 7f5abfe55740 1 --2- 172.21.15.98:0/1506757206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1706e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.928+0000 7f5abfe55740 1 --2- 172.21.15.98:0/1506757206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b17054a0 0x55c7b171c4e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.928+0000 7f5abfe55740 1 -- 172.21.15.98:0/1506757206 >> 172.21.15.98:0/1506757206 conn(0x55c7b1701a30 msgr2=0x55c7b170c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.928+0000 7f5abfe55740 1 -- 172.21.15.98:0/1506757206 shutdown_connections 2024-09-19T08:03:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.928+0000 7f5abfe55740 1 -- 172.21.15.98:0/1506757206 wait complete. 2024-09-19T08:03:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.928+0000 7f5abfe55740 1 -- start start 2024-09-19T08:03:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.928+0000 7f5abfe55740 1 Processor -- start 2024-09-19T08:03:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.929+0000 7f5abfe55740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b1718f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.929+0000 7f5abfe55740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1714270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.929+0000 7f5abfe55740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 0x55c7b17149b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.929+0000 7f5ab6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1714270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.929+0000 7f5ab77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b1718f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.929+0000 7f5ab77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b1718f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:59430/0 (socket says 172.21.15.98:59430) 2024-09-19T08:03:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.929+0000 7f5ab77fe640 1 -- 172.21.15.98:0/1809259453 learned_addr learned my addr 172.21.15.98:0/1809259453 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:03:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.929+0000 7f5abfe55740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab81b5270 con 0x55c7b1705dc0 2024-09-19T08:03:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.929+0000 7f5abfe55740 1 -- 172.21.15.98:0/1809259453 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab81b4ba0 con 0x55c7b1710680 2024-09-19T08:03:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.929+0000 7f5abfe55740 1 -- 172.21.15.98:0/1809259453 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab81b1de0 con 0x55c7b17054a0 2024-09-19T08:03:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.929+0000 7f5ab7fff640 1 --2- 172.21.15.98:0/1809259453 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 0x55c7b17149b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5ab6ffd640 1 -- 172.21.15.98:0/1809259453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 msgr2=0x55c7b1718f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5ab6ffd640 1 --2- 172.21.15.98:0/1809259453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b1718f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5ab6ffd640 1 -- 172.21.15.98:0/1809259453 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 msgr2=0x55c7b17149b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:03:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5ab6ffd640 1 --2- 172.21.15.98:0/1809259453 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 0x55c7b17149b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5ab6ffd640 1 -- 172.21.15.98:0/1809259453 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7b1789760 con 0x55c7b1705dc0 2024-09-19T08:03:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5ab7fff640 1 --2- 172.21.15.98:0/1809259453 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 0x55c7b17149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:03:58.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5a96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:03:58.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5ab6ffd640 1 --2- 172.21.15.98:0/1809259453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1714270 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00c6a0 tx=0x7f5aac005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ded7280a83b7ca4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:03:58.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5abfe55740 10 client.0 _ll_register_callbacks cb 0x7f5ab0009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:03:58.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5abfe55740 1 -- 172.21.15.98:0/1809259453 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5aa8047070 con 0x55c7b1705dc0 2024-09-19T08:03:58.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5a96ffd640 1 -- 172.21.15.98:0/1809259453 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac09cb20 con 0x55c7b1705dc0 2024-09-19T08:03:58.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5a96ffd640 1 -- 172.21.15.98:0/1809259453 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5aac0a4c00 con 0x55c7b1705dc0 2024-09-19T08:03:58.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5a96ffd640 1 -- 172.21.15.98:0/1809259453 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5aac0a4e00 con 0x55c7b1705dc0 2024-09-19T08:03:58.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5abfe55740 10 client.15762 fetch_fsmap learned FSMap version 28 2024-09-19T08:03:58.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.930+0000 7f5abfe55740 1 -- 172.21.15.98:0/1809259453 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55c7b178a710 con 0x55c7b1705dc0 2024-09-19T08:03:58.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.931+0000 7f5a96ffd640 1 -- 172.21.15.98:0/1809259453 <== mon.0 v2:172.21.15.98:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5aac0b5180 con 0x55c7b1705dc0 2024-09-19T08:03:58.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.931+0000 7f5abfe55740 10 client.15762 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:03:58.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.931+0000 7f5abfe55740 10 client.15762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:03:58.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.931+0000 7f5abfe55740 20 client.15762 populate_metadata read hostname 'smithi098' 2024-09-19T08:03:58.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.931+0000 7f5abfe55740 1 --2- 172.21.15.98:0/1809259453 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55c7b178adb0 0x55c7b178d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:03:58.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.931+0000 7f5abfe55740 4 client.15762 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:03:58.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.931+0000 7f5abfe55740 1 -- 172.21.15.98:0/1809259453 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c7b1672d60 con 0x55c7b178adb0 2024-09-19T08:03:58.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.931+0000 7f5ab77fe640 1 --2- 172.21.15.98:0/1809259453 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55c7b178adb0 0x55c7b178d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:03:58.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5a96ffd640 10 client.15762 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:58.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5ab77fe640 1 --2- 172.21.15.98:0/1809259453 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55c7b178adb0 0x55c7b178d270 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77c9a5ded02cd294 server_cookie=2717baecf183fae8 in_seq=0 out_seq=0 2024-09-19T08:03:58.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5a96ffd640 1 -- 172.21.15.98:0/1809259453 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c7b1672d60 con 0x55c7b178adb0 2024-09-19T08:03:58.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5a96ffd640 10 client.15762 handle_command_reply: tid=0 2024-09-19T08:03:58.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5abfe55740 1 client.15762 shutdown 2024-09-19T08:03:58.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5abfe55740 1 -- 172.21.15.98:0/1809259453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 msgr2=0x55c7b1714270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:58.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5abfe55740 1 --2- 172.21.15.98:0/1809259453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1714270 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00c6a0 tx=0x7f5aac005f10 comp rx=0 tx=0).stop 2024-09-19T08:03:58.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5abfe55740 1 -- 172.21.15.98:0/1809259453 shutdown_connections 2024-09-19T08:03:58.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5abfe55740 1 --2- 172.21.15.98:0/1809259453 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55c7b178adb0 0x55c7b178d270 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5abfe55740 1 --2- 172.21.15.98:0/1809259453 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7b1710680 0x55c7b17149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5abfe55740 1 --2- 172.21.15.98:0/1809259453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7b1705dc0 0x55c7b1714270 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5abfe55740 1 --2- 172.21.15.98:0/1809259453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7b17054a0 0x55c7b1718f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5abfe55740 1 -- 172.21.15.98:0/1809259453 >> 172.21.15.98:0/1809259453 conn(0x55c7b1701a30 msgr2=0x55c7b170ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:58.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5a96ffd640 0 client.15762 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:03:58.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5abfe55740 1 -- 172.21.15.98:0/1809259453 shutdown_connections 2024-09-19T08:03:58.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.932+0000 7f5abfe55740 1 -- 172.21.15.98:0/1809259453 wait complete. 2024-09-19T08:03:58.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.933+0000 7f5abfe55740 20 client.15762 trim_cache size 0 max 16384 2024-09-19T08:03:58.938 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:03:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-19T08:03:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-09-19T08:03:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: "a346458c-274b-4ab6-8c2c-cdb733a89881": { 2024-09-19T08:03:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-09-19T08:03:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "a346458c-274b-4ab6-8c2c-cdb733a89881", 2024-09-19T08:03:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-09-19T08:03:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:03:58.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:03:58.938 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:03:58.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.933+0000 7f5abda7c640 1 -- 172.21.15.98:0/2712843029 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5aa00a3550 msgr2=0x7f5aa00a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:58.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.933+0000 7f5abda7c640 1 --2- 172.21.15.98:0/2712843029 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5aa00a3550 0x7f5aa00a5a10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000a620 tx=0x7f5ab000d040 comp rx=0 tx=0).stop 2024-09-19T08:03:58.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.933+0000 7f5abda7c640 1 -- 172.21.15.98:0/2712843029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ab8103740 msgr2=0x7f5ab8073ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:03:58.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.933+0000 7f5abda7c640 1 --2- 172.21.15.98:0/2712843029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ab8103740 0x7f5ab8073ec0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00e9f0 tx=0x7f5aac00eeb0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.934+0000 7f5abda7c640 1 -- 172.21.15.98:0/2712843029 shutdown_connections 2024-09-19T08:03:58.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.934+0000 7f5abda7c640 1 --2- 172.21.15.98:0/2712843029 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5aa00a3550 0x7f5aa00a5a10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.934+0000 7f5abda7c640 1 --2- 172.21.15.98:0/2712843029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ab8108290 0x7f5ab806cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.934+0000 7f5abda7c640 1 --2- 172.21.15.98:0/2712843029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ab8103740 0x7f5ab8073ec0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.934+0000 7f5abda7c640 1 --2- 172.21.15.98:0/2712843029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ab8102d90 0x7f5ab8073980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:03:58.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.934+0000 7f5abda7c640 1 -- 172.21.15.98:0/2712843029 >> 172.21.15.98:0/2712843029 conn(0x7f5ab80fc7d0 msgr2=0x7f5ab8105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:03:58.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.934+0000 7f5abda7c640 1 -- 172.21.15.98:0/2712843029 shutdown_connections 2024-09-19T08:03:58.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:03:58.934+0000 7f5abda7c640 1 -- 172.21.15.98:0/2712843029 wait complete. 2024-09-19T08:03:59.293 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:03:59.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:03:59 smithi098 ceph-mon[31333]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-09-19T08:03:59.730 INFO:tasks.cephfs.filesystem:scrub status for tag:a346458c-274b-4ab6-8c2c-cdb733a89881 - {'path': '/', 'tag': 'a346458c-274b-4ab6-8c2c-cdb733a89881', 'options': 'recursive,force'} 2024-09-19T08:03:59.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:03:59 smithi162 ceph-mon[35260]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-09-19T08:03:59.948 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:03:59 smithi139 ceph-mon[34874]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-09-19T08:04:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.199+0000 7f335600b640 1 -- 172.21.15.98:0/3615838324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3350108290 msgr2=0x7f335010ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.199+0000 7f335600b640 1 --2- 172.21.15.98:0/3615838324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3350108290 0x7f335010ec30 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3348009cd0 tx=0x7f334802a8e0 comp rx=0 tx=0).stop 2024-09-19T08:04:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.199+0000 7f335600b640 1 -- 172.21.15.98:0/3615838324 shutdown_connections 2024-09-19T08:04:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.199+0000 7f335600b640 1 --2- 172.21.15.98:0/3615838324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3350108290 0x7f335010ec30 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.199+0000 7f335600b640 1 --2- 172.21.15.98:0/3615838324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3350103740 0x7f3350107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.199+0000 7f335600b640 1 --2- 172.21.15.98:0/3615838324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3350102d90 0x7f3350103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.199+0000 7f335600b640 1 -- 172.21.15.98:0/3615838324 >> 172.21.15.98:0/3615838324 conn(0x7f33500fc7d0 msgr2=0x7f33500febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:00.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.199+0000 7f335600b640 1 -- 172.21.15.98:0/3615838324 shutdown_connections 2024-09-19T08:04:00.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.199+0000 7f335600b640 1 -- 172.21.15.98:0/3615838324 wait complete. 2024-09-19T08:04:00.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.199+0000 7f335600b640 1 Processor -- start 2024-09-19T08:04:00.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f335600b640 1 -- start start 2024-09-19T08:04:00.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f335600b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3350102d90 0x7f33501984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:00.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f335600b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3350103740 0x7f33501989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:00.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f335600b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3350108290 0x7f335019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:00.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f335600b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33501120a0 con 0x7f3350103740 2024-09-19T08:04:00.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f335600b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3350111f20 con 0x7f3350102d90 2024-09-19T08:04:00.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f335600b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3350112220 con 0x7f3350108290 2024-09-19T08:04:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f334effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3350103740 0x7f33501989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f334effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3350103740 0x7f33501989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48756/0 (socket says 172.21.15.98:48756) 2024-09-19T08:04:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f334effd640 1 -- 172.21.15.98:0/3268323432 learned_addr learned my addr 172.21.15.98:0/3268323432 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f334ffff640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3350108290 0x7f335019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f334f7fe640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3350102d90 0x7f33501984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:00.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f334effd640 1 -- 172.21.15.98:0/3268323432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3350108290 msgr2=0x7f335019cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:00.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f334effd640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3350108290 0x7f335019cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:00.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f334effd640 1 -- 172.21.15.98:0/3268323432 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3350102d90 msgr2=0x7f33501984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:00.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f334effd640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3350102d90 0x7f33501984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:00.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f334effd640 1 -- 172.21.15.98:0/3268323432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f335019d450 con 0x7f3350103740 2024-09-19T08:04:00.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.200+0000 7f334f7fe640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3350102d90 0x7f33501984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:00.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.201+0000 7f334effd640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3350103740 0x7f33501989e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f334400c970 tx=0x7f334400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:00.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.201+0000 7f334cff9640 1 -- 172.21.15.98:0/3268323432 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3344014070 con 0x7f3350103740 2024-09-19T08:04:00.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.201+0000 7f334cff9640 1 -- 172.21.15.98:0/3268323432 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3344004440 con 0x7f3350103740 2024-09-19T08:04:00.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.201+0000 7f335600b640 1 -- 172.21.15.98:0/3268323432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f335019d740 con 0x7f3350103740 2024-09-19T08:04:00.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.201+0000 7f334cff9640 1 -- 172.21.15.98:0/3268323432 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334400f450 con 0x7f3350103740 2024-09-19T08:04:00.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.201+0000 7f335600b640 1 -- 172.21.15.98:0/3268323432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3350100920 con 0x7f3350103740 2024-09-19T08:04:00.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.202+0000 7f335600b640 1 -- 172.21.15.98:0/3268323432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3350103170 con 0x7f3350103740 2024-09-19T08:04:00.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.203+0000 7f334cff9640 1 -- 172.21.15.98:0/3268323432 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f3344017070 con 0x7f3350103740 2024-09-19T08:04:00.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.204+0000 7f334cff9640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f33380a34d0 0x7f33380a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:00.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.204+0000 7f334f7fe640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f33380a34d0 0x7f33380a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:00.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.205+0000 7f334cff9640 1 -- 172.21.15.98:0/3268323432 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f33440cfd90 con 0x7f3350103740 2024-09-19T08:04:00.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.205+0000 7f334f7fe640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f33380a34d0 0x7f33380a5990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f33400058c0 tx=0x7f3340005ec0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:00.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.212+0000 7f334cff9640 1 -- 172.21.15.98:0/3268323432 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f334401e020 con 0x7f3350103740 2024-09-19T08:04:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.478+0000 7f335600b640 1 -- 172.21.15.98:0/3268323432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3350199800 con 0x7f3350103740 2024-09-19T08:04:00.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.479+0000 7f334cff9640 1 -- 172.21.15.98:0/3268323432 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11983 (secure 0 0 0) 0x7f3350199800 con 0x7f3350103740 2024-09-19T08:04:00.479 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:04:00.479 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"btime":"2024-09-19T08:02:59:855935+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:02:59.805764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:04:00.479 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-09-19T08:04:00.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.482+0000 7f335600b640 1 -- 172.21.15.98:0/3268323432 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f33380a34d0 msgr2=0x7f33380a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:00.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.482+0000 7f335600b640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f33380a34d0 0x7f33380a5990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f33400058c0 tx=0x7f3340005ec0 comp rx=0 tx=0).stop 2024-09-19T08:04:00.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.482+0000 7f335600b640 1 -- 172.21.15.98:0/3268323432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3350103740 msgr2=0x7f33501989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:00.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.482+0000 7f335600b640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3350103740 0x7f33501989e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f334400c970 tx=0x7f334400ce30 comp rx=0 tx=0).stop 2024-09-19T08:04:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.482+0000 7f335600b640 1 -- 172.21.15.98:0/3268323432 shutdown_connections 2024-09-19T08:04:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.482+0000 7f335600b640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f33380a34d0 0x7f33380a5990 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.482+0000 7f335600b640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3350108290 0x7f335019cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.482+0000 7f335600b640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3350103740 0x7f33501989e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.482+0000 7f335600b640 1 --2- 172.21.15.98:0/3268323432 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3350102d90 0x7f33501984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.482+0000 7f335600b640 1 -- 172.21.15.98:0/3268323432 >> 172.21.15.98:0/3268323432 conn(0x7f33500fc7d0 msgr2=0x7f33500fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.483+0000 7f335600b640 1 -- 172.21.15.98:0/3268323432 shutdown_connections 2024-09-19T08:04:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:00.483+0000 7f335600b640 1 -- 172.21.15.98:0/3268323432 wait complete. 2024-09-19T08:04:00.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:00 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:04:00.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:00 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:04:00.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:00 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:04:01.138 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:04:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:04:00.932+0000 7f85cae30640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:04:01.138 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:04:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:04:00.932+0000 7f85cae30640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:04:01.138 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:04:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:04:00.932+0000 7f85cae30640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:04:01.138 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:04:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:04:00.932+0000 7f85cae30640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:04:01.138 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:04:00 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:04:00.932+0000 7f85cae30640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T08:04:01.263 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id f1380945 --release --await' 2024-09-19T08:04:01.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id f1380945 --release --await 2024-09-19T08:04:01.519 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:01.743 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:01 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3268323432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:01.743 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:01 smithi098 ceph-mon[31333]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-09-19T08:04:01.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:01 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3268323432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:01.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:01 smithi162 ceph-mon[35260]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-09-19T08:04:01.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:01 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3268323432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:01.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:01 smithi139 ceph-mon[34874]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-09-19T08:04:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.989+0000 7f0304079640 1 -- 172.21.15.98:0/3169459544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02fc1036e0 msgr2=0x7f02fc107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.989+0000 7f0304079640 1 --2- 172.21.15.98:0/3169459544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02fc1036e0 0x7f02fc107b90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009930 tx=0x7f02ec02a420 comp rx=0 tx=0).stop 2024-09-19T08:04:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.989+0000 7f0304079640 1 -- 172.21.15.98:0/3169459544 shutdown_connections 2024-09-19T08:04:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.989+0000 7f0304079640 1 --2- 172.21.15.98:0/3169459544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02fc1082a0 0x7f02fc10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.989+0000 7f0304079640 1 --2- 172.21.15.98:0/3169459544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02fc1036e0 0x7f02fc107b90 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.989+0000 7f0304079640 1 --2- 172.21.15.98:0/3169459544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f02fc102d30 0x7f02fc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.989+0000 7f0304079640 1 -- 172.21.15.98:0/3169459544 >> 172.21.15.98:0/3169459544 conn(0x7f02fc0fc7d0 msgr2=0x7f02fc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.990+0000 7f0304079640 1 -- 172.21.15.98:0/3169459544 shutdown_connections 2024-09-19T08:04:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.990+0000 7f0304079640 1 -- 172.21.15.98:0/3169459544 wait complete. 2024-09-19T08:04:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.990+0000 7f0304079640 1 Processor -- start 2024-09-19T08:04:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.990+0000 7f0304079640 1 -- start start 2024-09-19T08:04:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0304079640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02fc102d30 0x7f02fc198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0304079640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f02fc1036e0 0x7f02fc198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0304079640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02fc1082a0 0x7f02fc19cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0301e15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02fc102d30 0x7f02fc198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0304079640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc112060 con 0x7f02fc102d30 2024-09-19T08:04:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0304079640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc111ee0 con 0x7f02fc1036e0 2024-09-19T08:04:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0301e15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02fc102d30 0x7f02fc198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48794/0 (socket says 172.21.15.98:48794) 2024-09-19T08:04:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0304079640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc1121e0 con 0x7f02fc1082a0 2024-09-19T08:04:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0301e15640 1 -- 172.21.15.98:0/2898605962 learned_addr learned my addr 172.21.15.98:0/2898605962 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0301614640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f02fc1036e0 0x7f02fc198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0302616640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02fc1082a0 0x7f02fc19cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0301e15640 1 -- 172.21.15.98:0/2898605962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02fc1082a0 msgr2=0x7f02fc19cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0301e15640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02fc1082a0 0x7f02fc19cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0301e15640 1 -- 172.21.15.98:0/2898605962 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f02fc1036e0 msgr2=0x7f02fc198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0301e15640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f02fc1036e0 0x7f02fc198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0301e15640 1 -- 172.21.15.98:0/2898605962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02fc19d4d0 con 0x7f02fc102d30 2024-09-19T08:04:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0301614640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f02fc1036e0 0x7f02fc198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:04:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.991+0000 7f0302616640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02fc1082a0 0x7f02fc19cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.992+0000 7f0301e15640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02fc102d30 0x7f02fc198520 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f02f800d950 tx=0x7f02f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.992+0000 7f02f2ffd640 1 -- 172.21.15.98:0/2898605962 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f8017030 con 0x7f02fc102d30 2024-09-19T08:04:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.992+0000 7f0304079640 1 -- 172.21.15.98:0/2898605962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02fc19d7c0 con 0x7f02fc102d30 2024-09-19T08:04:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.992+0000 7f0304079640 1 -- 172.21.15.98:0/2898605962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02fc1a50a0 con 0x7f02fc102d30 2024-09-19T08:04:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.992+0000 7f02f2ffd640 1 -- 172.21.15.98:0/2898605962 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f8004020 con 0x7f02fc102d30 2024-09-19T08:04:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.992+0000 7f02f2ffd640 1 -- 172.21.15.98:0/2898605962 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f80047a0 con 0x7f02fc102d30 2024-09-19T08:04:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.995+0000 7f0306452740 1 Processor -- start 2024-09-19T08:04:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.995+0000 7f0306452740 1 -- start start 2024-09-19T08:04:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.995+0000 7f02f2ffd640 1 -- 172.21.15.98:0/2898605962 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138582+0+0 (secure 0 0 0) 0x7f02f80041d0 con 0x7f02fc102d30 2024-09-19T08:04:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0306452740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61bf40 0x55b85c61c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0306452740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 0x55b85c62b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0306452740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c62b650 0x55b85c62da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0306452740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b85c581620 con 0x55b85c61bf40 2024-09-19T08:04:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0306452740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b85c528e50 con 0x55b85c61c860 2024-09-19T08:04:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0306452740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b85c57d5e0 con 0x55b85c62b650 2024-09-19T08:04:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0302616640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61bf40 0x55b85c61c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0302616640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61bf40 0x55b85c61c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48802/0 (socket says 172.21.15.98:48802) 2024-09-19T08:04:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0302616640 1 -- 172.21.15.98:0/3361425427 learned_addr learned my addr 172.21.15.98:0/3361425427 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0301e15640 1 --2- 172.21.15.98:0/3361425427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 0x55b85c62b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0302616640 1 -- 172.21.15.98:0/3361425427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c62b650 msgr2=0x55b85c62da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0302616640 1 --2- 172.21.15.98:0/3361425427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c62b650 0x55b85c62da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0302616640 1 -- 172.21.15.98:0/3361425427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 msgr2=0x55b85c62b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0302616640 1 --2- 172.21.15.98:0/3361425427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 0x55b85c62b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.996+0000 7f0302616640 1 -- 172.21.15.98:0/3361425427 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b85c519d40 con 0x55b85c61bf40 2024-09-19T08:04:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.997+0000 7f02f2ffd640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f02e80a3550 0x7f02e80a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.997+0000 7f0302616640 1 --2- 172.21.15.98:0/3361425427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61bf40 0x55b85c61c320 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f02f4007e60 tx=0x7f02f400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4989333a76e11070 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.997+0000 7f02f2ffd640 1 -- 172.21.15.98:0/2898605962 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f02f80cf790 con 0x7f02fc102d30 2024-09-19T08:04:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.997+0000 7f02e7fff640 1 -- 172.21.15.98:0/3361425427 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4014070 con 0x55b85c61bf40 2024-09-19T08:04:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.997+0000 7f02e7fff640 1 -- 172.21.15.98:0/3361425427 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f4003e50 con 0x55b85c61bf40 2024-09-19T08:04:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.997+0000 7f02e7fff640 1 -- 172.21.15.98:0/3361425427 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4010640 con 0x55b85c61bf40 2024-09-19T08:04:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.997+0000 7f0301e15640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f02e80a3550 0x7f02e80a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.998+0000 7f0306452740 1 -- 172.21.15.98:0/3361425427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61bf40 msgr2=0x55b85c61c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.998+0000 7f0306452740 1 --2- 172.21.15.98:0/3361425427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61bf40 0x55b85c61c320 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f02f4007e60 tx=0x7f02f400e720 comp rx=0 tx=0).stop 2024-09-19T08:04:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.998+0000 7f0301e15640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f02e80a3550 0x7f02e80a5a10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f02f80ccdf0 tx=0x7f02f80057b0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.998+0000 7f0306452740 1 -- 172.21.15.98:0/3361425427 shutdown_connections 2024-09-19T08:04:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.998+0000 7f0306452740 1 --2- 172.21.15.98:0/3361425427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c62b650 0x55b85c62da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.998+0000 7f0306452740 1 --2- 172.21.15.98:0/3361425427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 0x55b85c62b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.998+0000 7f0306452740 1 --2- 172.21.15.98:0/3361425427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61bf40 0x55b85c61c320 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.998+0000 7f0306452740 1 -- 172.21.15.98:0/3361425427 >> 172.21.15.98:0/3361425427 conn(0x55b85c6185f0 msgr2=0x55b85c6189d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.998+0000 7f0306452740 1 -- 172.21.15.98:0/3361425427 shutdown_connections 2024-09-19T08:04:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.998+0000 7f0306452740 1 -- 172.21.15.98:0/3361425427 wait complete. 2024-09-19T08:04:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.999+0000 7f0306452740 1 -- start start 2024-09-19T08:04:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.999+0000 7f0306452740 1 Processor -- start 2024-09-19T08:04:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.999+0000 7f0306452740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61bf40 0x55b85c628570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.999+0000 7f0306452740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 0x55b85c628ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.999+0000 7f0306452740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c62b650 0x55b85c628ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.999+0000 7f0306452740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b85c62e220 con 0x55b85c61bf40 2024-09-19T08:04:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.999+0000 7f0306452740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b85c509f10 con 0x55b85c61c860 2024-09-19T08:04:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.999+0000 7f0306452740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b85c62e3a0 con 0x55b85c62b650 2024-09-19T08:04:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:01.999+0000 7f0306452740 10 client.0 _ll_register_callbacks cb 0x55b85c52ddb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.000+0000 7f0302616640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61bf40 0x55b85c628570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.000+0000 7f0302616640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61bf40 0x55b85c628570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48806/0 (socket says 172.21.15.98:48806) 2024-09-19T08:04:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.000+0000 7f0302616640 1 -- 172.21.15.98:0/626424203 learned_addr learned my addr 172.21.15.98:0/626424203 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.000+0000 7f0301614640 1 --2- 172.21.15.98:0/626424203 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 0x55b85c628ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.000+0000 7f0302616640 1 -- 172.21.15.98:0/626424203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c62b650 msgr2=0x55b85c628ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.000+0000 7f0302616640 1 --2- 172.21.15.98:0/626424203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c62b650 0x55b85c628ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.000+0000 7f0302616640 1 -- 172.21.15.98:0/626424203 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 msgr2=0x55b85c628ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.000+0000 7f0302616640 1 --2- 172.21.15.98:0/626424203 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 0x55b85c628ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.000+0000 7f0302616640 1 -- 172.21.15.98:0/626424203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b85c290580 con 0x55b85c61bf40 2024-09-19T08:04:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.000+0000 7f0302616640 1 --2- 172.21.15.98:0/626424203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61bf40 0x55b85c628570 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f02f4004350 tx=0x7f02f4004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f83a242182e82b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.000+0000 7f02e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.000+0000 7f02e57fa640 1 -- 172.21.15.98:0/626424203 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4014070 con 0x55b85c61bf40 2024-09-19T08:04:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.001+0000 7f02e57fa640 1 -- 172.21.15.98:0/626424203 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f02f4010640 con 0x55b85c61bf40 2024-09-19T08:04:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.001+0000 7f0306452740 1 -- 172.21.15.98:0/626424203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02f4003e50 con 0x55b85c61bf40 2024-09-19T08:04:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.001+0000 7f02e57fa640 1 -- 172.21.15.98:0/626424203 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f02f4010840 con 0x55b85c61bf40 2024-09-19T08:04:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.001+0000 7f0306452740 10 client.15786 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.001+0000 7f0306452740 1 -- 172.21.15.98:0/626424203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b85c35a960 con 0x55b85c61bf40 2024-09-19T08:04:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.001+0000 7f02e57fa640 1 -- 172.21.15.98:0/626424203 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f4020070 con 0x55b85c61bf40 2024-09-19T08:04:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.001+0000 7f02e57fa640 1 -- 172.21.15.98:0/626424203 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f02f4025b10 con 0x55b85c61bf40 2024-09-19T08:04:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.001+0000 7f0306452740 10 client.15786 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.001+0000 7f0306452740 10 client.15786 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-19T08:04:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.001+0000 7f0306452740 20 client.15786 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.001+0000 7f0306452740 1 --2- 172.21.15.98:0/626424203 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b85c63b180 0x55b85c63d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.001+0000 7f0306452740 4 client.15786 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:04:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.001+0000 7f0306452740 1 -- 172.21.15.98:0/626424203 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f02f4003e50 con 0x55b85c63b180 2024-09-19T08:04:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.002+0000 7f0301614640 1 --2- 172.21.15.98:0/626424203 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b85c63b180 0x55b85c63d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.002+0000 7f02e57fa640 10 client.15786 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.002+0000 7f0301614640 1 --2- 172.21.15.98:0/626424203 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b85c63b180 0x55b85c63d570 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a51c5367b8e0c8f8 server_cookie=99f8fac6ea518e3 in_seq=0 out_seq=0 2024-09-19T08:04:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.003+0000 7f02e57fa640 1 -- 172.21.15.98:0/626424203 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f02f4003e50 con 0x55b85c63b180 2024-09-19T08:04:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.003+0000 7f02e57fa640 10 client.15786 handle_command_reply: tid=0 2024-09-19T08:04:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.003+0000 7f0306452740 1 client.15786 shutdown 2024-09-19T08:04:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.003+0000 7f0306452740 1 -- 172.21.15.98:0/626424203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61bf40 msgr2=0x55b85c628570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.003+0000 7f0306452740 1 --2- 172.21.15.98:0/626424203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61bf40 0x55b85c628570 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f02f4004350 tx=0x7f02f4004170 comp rx=0 tx=0).stop 2024-09-19T08:04:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.004+0000 7f0306452740 1 -- 172.21.15.98:0/626424203 shutdown_connections 2024-09-19T08:04:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.004+0000 7f0306452740 1 --2- 172.21.15.98:0/626424203 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b85c63b180 0x55b85c63d570 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.004+0000 7f02e57fa640 0 client.15786 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.004+0000 7f0306452740 1 --2- 172.21.15.98:0/626424203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c62b650 0x55b85c628ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.004+0000 7f0306452740 1 --2- 172.21.15.98:0/626424203 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 0x55b85c628ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.004+0000 7f0306452740 1 --2- 172.21.15.98:0/626424203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61bf40 0x55b85c628570 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.004+0000 7f0306452740 1 -- 172.21.15.98:0/626424203 >> 172.21.15.98:0/626424203 conn(0x55b85c6185f0 msgr2=0x55b85c61e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.004+0000 7f0306452740 1 -- 172.21.15.98:0/626424203 shutdown_connections 2024-09-19T08:04:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.004+0000 7f0306452740 1 -- 172.21.15.98:0/626424203 wait complete. 2024-09-19T08:04:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.004+0000 7f0306452740 20 client.15786 trim_cache size 0 max 16384 2024-09-19T08:04:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0306452740 1 Processor -- start 2024-09-19T08:04:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0306452740 1 -- start start 2024-09-19T08:04:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0306452740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c61bf40 0x55b85c61e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0306452740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 0x55b85c61e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0306452740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c62b650 0x55b85c642b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0306452740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02e000e6d0 con 0x55b85c62b650 2024-09-19T08:04:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0306452740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f02e0010210 con 0x55b85c61c860 2024-09-19T08:04:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0306452740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02e000eda0 con 0x55b85c61bf40 2024-09-19T08:04:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0302616640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c62b650 0x55b85c642b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0302616640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c62b650 0x55b85c642b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48814/0 (socket says 172.21.15.98:48814) 2024-09-19T08:04:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0302616640 1 -- 172.21.15.98:0/2062776785 learned_addr learned my addr 172.21.15.98:0/2062776785 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0301614640 1 --2- 172.21.15.98:0/2062776785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 0x55b85c61e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0302616640 1 --2- 172.21.15.98:0/2062776785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c61bf40 0x55b85c61e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0302616640 1 -- 172.21.15.98:0/2062776785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c61bf40 msgr2=0x55b85c61e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.022+0000 7f0302616640 1 --2- 172.21.15.98:0/2062776785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c61bf40 0x55b85c61e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.023+0000 7f0302616640 1 -- 172.21.15.98:0/2062776785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 msgr2=0x55b85c61e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.023+0000 7f0302616640 1 --2- 172.21.15.98:0/2062776785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 0x55b85c61e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.023+0000 7f0302616640 1 -- 172.21.15.98:0/2062776785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b85c627520 con 0x55b85c62b650 2024-09-19T08:04:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.023+0000 7f0302616640 1 --2- 172.21.15.98:0/2062776785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c62b650 0x55b85c642b90 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f02f4002230 tx=0x7f02f400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f56b9630ff8284 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.023+0000 7f02e77fe640 1 -- 172.21.15.98:0/2062776785 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4014030 con 0x55b85c62b650 2024-09-19T08:04:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.023+0000 7f02e77fe640 1 -- 172.21.15.98:0/2062776785 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f4020070 con 0x55b85c62b650 2024-09-19T08:04:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.023+0000 7f02e77fe640 1 -- 172.21.15.98:0/2062776785 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4003dc0 con 0x55b85c62b650 2024-09-19T08:04:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.024+0000 7f0306452740 1 -- 172.21.15.98:0/2062776785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c62b650 msgr2=0x55b85c642b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.024+0000 7f0306452740 1 --2- 172.21.15.98:0/2062776785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c62b650 0x55b85c642b90 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f02f4002230 tx=0x7f02f400e720 comp rx=0 tx=0).stop 2024-09-19T08:04:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.024+0000 7f0306452740 1 -- 172.21.15.98:0/2062776785 shutdown_connections 2024-09-19T08:04:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.024+0000 7f0306452740 1 --2- 172.21.15.98:0/2062776785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c62b650 0x55b85c642b90 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.024+0000 7f0306452740 1 --2- 172.21.15.98:0/2062776785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61c860 0x55b85c61e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.024+0000 7f0306452740 1 --2- 172.21.15.98:0/2062776785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c61bf40 0x55b85c61e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.024+0000 7f0306452740 1 -- 172.21.15.98:0/2062776785 >> 172.21.15.98:0/2062776785 conn(0x55b85c6185f0 msgr2=0x55b85c61da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.024+0000 7f0306452740 1 -- 172.21.15.98:0/2062776785 shutdown_connections 2024-09-19T08:04:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.024+0000 7f0306452740 1 -- 172.21.15.98:0/2062776785 wait complete. 2024-09-19T08:04:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.024+0000 7f0306452740 1 -- start start 2024-09-19T08:04:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.024+0000 7f0306452740 1 Processor -- start 2024-09-19T08:04:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.025+0000 7f0306452740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61bf40 0x55b85c6338c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.025+0000 7f0306452740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61c860 0x55b85c633e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.025+0000 7f0306452740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c62b650 0x55b85c630660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.025+0000 7f0306452740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02e000fb40 con 0x55b85c61c860 2024-09-19T08:04:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.025+0000 7f0301614640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61c860 0x55b85c633e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.025+0000 7f0301614640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61c860 0x55b85c633e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48824/0 (socket says 172.21.15.98:48824) 2024-09-19T08:04:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.025+0000 7f0302616640 1 --2- 172.21.15.98:0/1482422058 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61bf40 0x55b85c6338c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.025+0000 7f0301614640 1 -- 172.21.15.98:0/1482422058 learned_addr learned my addr 172.21.15.98:0/1482422058 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.025+0000 7f0306452740 1 -- 172.21.15.98:0/1482422058 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f02e000f470 con 0x55b85c61bf40 2024-09-19T08:04:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.025+0000 7f0306452740 1 -- 172.21.15.98:0/1482422058 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02e000c6b0 con 0x55b85c62b650 2024-09-19T08:04:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.025+0000 7f0302616640 1 --2- 172.21.15.98:0/1482422058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c62b650 0x55b85c630660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.026+0000 7f0301614640 1 -- 172.21.15.98:0/1482422058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c62b650 msgr2=0x55b85c630660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.026+0000 7f0301614640 1 --2- 172.21.15.98:0/1482422058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c62b650 0x55b85c630660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.026+0000 7f0301614640 1 -- 172.21.15.98:0/1482422058 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61bf40 msgr2=0x55b85c6338c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.026+0000 7f0306452740 10 client.0 _ll_register_callbacks cb 0x7f02f400ef30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.026+0000 7f0301614640 1 --2- 172.21.15.98:0/1482422058 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61bf40 0x55b85c6338c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.026+0000 7f0301614640 1 -- 172.21.15.98:0/1482422058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b85c630e90 con 0x55b85c61c860 2024-09-19T08:04:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.026+0000 7f0306452740 1 -- 172.21.15.98:0/1482422058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02f4020070 con 0x55b85c61c860 2024-09-19T08:04:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.026+0000 7f0301614640 1 --2- 172.21.15.98:0/1482422058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61c860 0x55b85c633e00 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f02ec005940 tx=0x7f02ec009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39c315a386b0d9f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.026+0000 7f02e57fa640 10 client.15792 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.026+0000 7f02e57fa640 1 -- 172.21.15.98:0/1482422058 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec035040 con 0x55b85c61c860 2024-09-19T08:04:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.026+0000 7f02e57fa640 1 -- 172.21.15.98:0/1482422058 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f02ec005450 con 0x55b85c61c860 2024-09-19T08:04:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.027+0000 7f02e57fa640 1 -- 172.21.15.98:0/1482422058 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f02ec004490 con 0x55b85c61c860 2024-09-19T08:04:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.027+0000 7f0306452740 10 client.15792 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.027+0000 7f0306452740 1 -- 172.21.15.98:0/1482422058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55b85c6a66a0 con 0x55b85c61c860 2024-09-19T08:04:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.027+0000 7f02e57fa640 1 -- 172.21.15.98:0/1482422058 <== mon.0 v2:172.21.15.98:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f02ec031830 con 0x55b85c61c860 2024-09-19T08:04:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.027+0000 7f0306452740 10 client.15792 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.027+0000 7f0306452740 10 client.15792 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-19T08:04:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.027+0000 7f0306452740 20 client.15792 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.027+0000 7f0306452740 1 --2- 172.21.15.98:0/1482422058 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b85c63b180 0x55b85c6a8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.027+0000 7f0306452740 4 client.15792 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "f1380945", "release": true, "await": true}] 2024-09-19T08:04:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.027+0000 7f0306452740 1 -- 172.21.15.98:0/1482422058 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "quiesce db", "set_id": "f1380945", "release": true, "await": true}) -- 0x55b85c58eb90 con 0x55b85c63b180 2024-09-19T08:04:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.027+0000 7f0302616640 1 --2- 172.21.15.98:0/1482422058 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b85c63b180 0x55b85c6a8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.028+0000 7f0302616640 1 --2- 172.21.15.98:0/1482422058 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b85c63b180 0x55b85c6a8cd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cfba8b048128ae8 server_cookie=b8b9807f52632e07 in_seq=0 out_seq=0 2024-09-19T08:04:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.028+0000 7f02e57fa640 10 client.15792 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.030+0000 7f02e57fa640 1 -- 172.21.15.98:0/1482422058 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55b85c58eb90 con 0x55b85c63b180 2024-09-19T08:04:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.030+0000 7f02e57fa640 10 client.15792 handle_command_reply: tid=0 2024-09-19T08:04:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.030+0000 7f0306452740 1 client.15792 shutdown 2024-09-19T08:04:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.030+0000 7f0306452740 1 -- 172.21.15.98:0/1482422058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61c860 msgr2=0x55b85c633e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.030+0000 7f0306452740 1 --2- 172.21.15.98:0/1482422058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61c860 0x55b85c633e00 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f02ec005940 tx=0x7f02ec009260 comp rx=0 tx=0).stop 2024-09-19T08:04:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.030+0000 7f0306452740 1 -- 172.21.15.98:0/1482422058 shutdown_connections 2024-09-19T08:04:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.030+0000 7f0306452740 1 --2- 172.21.15.98:0/1482422058 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b85c63b180 0x55b85c6a8cd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.030+0000 7f02e57fa640 0 client.15792 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.030+0000 7f0306452740 1 --2- 172.21.15.98:0/1482422058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b85c62b650 0x55b85c630660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.030+0000 7f0306452740 1 --2- 172.21.15.98:0/1482422058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b85c61c860 0x55b85c633e00 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.030+0000 7f0306452740 1 --2- 172.21.15.98:0/1482422058 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b85c61bf40 0x55b85c6338c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.030+0000 7f0306452740 1 -- 172.21.15.98:0/1482422058 >> 172.21.15.98:0/1482422058 conn(0x55b85c6185f0 msgr2=0x55b85c61e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.031+0000 7f0306452740 1 -- 172.21.15.98:0/1482422058 shutdown_connections 2024-09-19T08:04:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.031+0000 7f0306452740 1 -- 172.21.15.98:0/1482422058 wait complete. 2024-09-19T08:04:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.031+0000 7f0306452740 20 client.15792 trim_cache size 0 max 16384 2024-09-19T08:04:02.033 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:04:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 28, 2024-09-19T08:04:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24485, 2024-09-19T08:04:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 4, 2024-09-19T08:04:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-09-19T08:04:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "f1380945": { 2024-09-19T08:04:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 4, 2024-09-19T08:04:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-09-19T08:04:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-09-19T08:04:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-09-19T08:04:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-09-19T08:04:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:04:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 90.0, 2024-09-19T08:04:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 131.2, 2024-09-19T08:04:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-09-19T08:04:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-09-19T08:04:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-09-19T08:04:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-09-19T08:04:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-09-19T08:04:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-09-19T08:04:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:02.035 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:04:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.031+0000 7f0304079640 1 -- 172.21.15.98:0/2898605962 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f02e80a3550 msgr2=0x7f02e80a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.031+0000 7f0304079640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f02e80a3550 0x7f02e80a5a10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f02f80ccdf0 tx=0x7f02f80057b0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.031+0000 7f0304079640 1 -- 172.21.15.98:0/2898605962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02fc102d30 msgr2=0x7f02fc198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.031+0000 7f0304079640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02fc102d30 0x7f02fc198520 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f02f800d950 tx=0x7f02f800de10 comp rx=0 tx=0).stop 2024-09-19T08:04:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.032+0000 7f0304079640 1 -- 172.21.15.98:0/2898605962 shutdown_connections 2024-09-19T08:04:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.032+0000 7f0304079640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f02e80a3550 0x7f02e80a5a10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.032+0000 7f0304079640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02fc1082a0 0x7f02fc19cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.032+0000 7f0304079640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f02fc1036e0 0x7f02fc198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.032+0000 7f0304079640 1 --2- 172.21.15.98:0/2898605962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02fc102d30 0x7f02fc198520 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.032+0000 7f0304079640 1 -- 172.21.15.98:0/2898605962 >> 172.21.15.98:0/2898605962 conn(0x7f02fc0fc7d0 msgr2=0x7f02fc106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.032+0000 7f0304079640 1 -- 172.21.15.98:0/2898605962 shutdown_connections 2024-09-19T08:04:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:02.032+0000 7f0304079640 1 -- 172.21.15.98:0/2898605962 wait complete. 2024-09-19T08:04:02.743 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'f1380945', total seconds elapsed: 15.4 2024-09-19T08:04:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:02 smithi098 ceph-mon[31333]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-09-19T08:04:02.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:02 smithi098 ceph-mon[31333]: mgrmap e23: x(active, since 62s), standbys: z, y 2024-09-19T08:04:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:02 smithi162 ceph-mon[35260]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-09-19T08:04:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:02 smithi162 ceph-mon[35260]: mgrmap e23: x(active, since 62s), standbys: z, y 2024-09-19T08:04:03.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:02 smithi139 ceph-mon[34874]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-09-19T08:04:03.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:02 smithi139 ceph-mon[34874]: mgrmap e23: x(active, since 62s), standbys: z, y 2024-09-19T08:04:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:04 smithi162 ceph-mon[35260]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:04:05.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:04 smithi139 ceph-mon[34874]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:04:05.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:04 smithi098 ceph-mon[31333]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:04:07.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:06 smithi162 ceph-mon[35260]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:04:07.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:06 smithi139 ceph-mon[34874]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:04:07.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:06 smithi098 ceph-mon[31333]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:04:09.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:08 smithi162 ceph-mon[35260]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-19T08:04:09.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:08 smithi139 ceph-mon[34874]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-19T08:04:09.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:08 smithi098 ceph-mon[31333]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-19T08:04:10.619 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:10 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:04:10.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:10 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:04:10.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:10 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:04:11.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:11 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:04:11.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:11 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:04:11.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:11 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:04:11.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:11 smithi162 ceph-mon[35260]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:04:11.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:11 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:04:11.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:11 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:04:11.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:11 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:04:11.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:11 smithi139 ceph-mon[34874]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:04:11.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:11 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:04:11.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:11 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:04:11.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:11 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:04:11.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:11 smithi098 ceph-mon[31333]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:04:13.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:12 smithi162 ceph-mon[35260]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-09-19T08:04:13.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:12 smithi139 ceph-mon[34874]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-09-19T08:04:13.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:12 smithi098 ceph-mon[31333]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-09-19T08:04:15.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:14 smithi162 ceph-mon[35260]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-09-19T08:04:15.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:14 smithi139 ceph-mon[34874]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-09-19T08:04:15.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:14 smithi098 ceph-mon[31333]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-09-19T08:04:17.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:16 smithi162 ceph-mon[35260]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 255 B/s wr, 13 op/s 2024-09-19T08:04:17.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:16 smithi139 ceph-mon[34874]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 255 B/s wr, 13 op/s 2024-09-19T08:04:17.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:16 smithi098 ceph-mon[31333]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 255 B/s wr, 13 op/s 2024-09-19T08:04:19.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:18 smithi162 ceph-mon[35260]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-19T08:04:19.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:18 smithi139 ceph-mon[34874]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-19T08:04:19.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:18 smithi098 ceph-mon[31333]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-19T08:04:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:20 smithi162 ceph-mon[35260]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-19T08:04:21.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:20 smithi139 ceph-mon[34874]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-19T08:04:21.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:20 smithi098 ceph-mon[31333]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-19T08:04:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:22 smithi162 ceph-mon[35260]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-19T08:04:23.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:22 smithi139 ceph-mon[34874]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-19T08:04:23.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:22 smithi098 ceph-mon[31333]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-19T08:04:23.543 INFO:tasks.quiescer.fs.[cephfs]:Total duration (40.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-19T08:04:23.543 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 20.35 2024-09-19T08:04:23.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:04:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:24.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.454+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/4105081136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a78102570 msgr2=0x7f1a781029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:24.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.454+0000 7f1a7f4c5640 1 --2- 172.21.15.98:0/4105081136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a78102570 0x7f1a781029b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1a68009900 tx=0x7f1a6802a480 comp rx=0 tx=0).stop 2024-09-19T08:04:24.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.454+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/4105081136 shutdown_connections 2024-09-19T08:04:24.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.454+0000 7f1a7f4c5640 1 --2- 172.21.15.98:0/4105081136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a78102ef0 0x7f1a7810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:24.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.454+0000 7f1a7f4c5640 1 --2- 172.21.15.98:0/4105081136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a78102570 0x7f1a781029b0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:24.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.454+0000 7f1a7f4c5640 1 --2- 172.21.15.98:0/4105081136 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1a78100e20 0x7f1a78101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:24.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.454+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/4105081136 >> 172.21.15.98:0/4105081136 conn(0x7f1a780fc7d0 msgr2=0x7f1a780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:24.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.455+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/4105081136 shutdown_connections 2024-09-19T08:04:24.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.455+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/4105081136 wait complete. 2024-09-19T08:04:24.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.455+0000 7f1a7f4c5640 1 Processor -- start 2024-09-19T08:04:24.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.455+0000 7f1a7f4c5640 1 -- start start 2024-09-19T08:04:24.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.455+0000 7f1a7f4c5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1a78100e20 0x7f1a78199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:24.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.455+0000 7f1a7f4c5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a78102570 0x7f1a78199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:24.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.455+0000 7f1a7f4c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a78102ef0 0x7f1a781a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:24.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.455+0000 7f1a7f4c5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1a78110fa0 con 0x7f1a78102ef0 2024-09-19T08:04:24.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.455+0000 7f1a7f4c5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1a78110e20 con 0x7f1a78100e20 2024-09-19T08:04:24.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.455+0000 7f1a7f4c5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1a78111120 con 0x7f1a78102570 2024-09-19T08:04:24.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a7da62640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a78102ef0 0x7f1a781a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:24.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a7da62640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a78102ef0 0x7f1a781a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47618/0 (socket says 172.21.15.98:47618) 2024-09-19T08:04:24.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a7da62640 1 -- 172.21.15.98:0/2297706962 learned_addr learned my addr 172.21.15.98:0/2297706962 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:24.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a7d261640 1 --2- 172.21.15.98:0/2297706962 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1a78100e20 0x7f1a78199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:24.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a7ca60640 1 --2- 172.21.15.98:0/2297706962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a78102570 0x7f1a78199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:24.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a7da62640 1 -- 172.21.15.98:0/2297706962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a78102570 msgr2=0x7f1a78199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:24.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a7da62640 1 --2- 172.21.15.98:0/2297706962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a78102570 0x7f1a78199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:24.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a7da62640 1 -- 172.21.15.98:0/2297706962 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1a78100e20 msgr2=0x7f1a78199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:24.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a7da62640 1 --2- 172.21.15.98:0/2297706962 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1a78100e20 0x7f1a78199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:24.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a7da62640 1 -- 172.21.15.98:0/2297706962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a781a4910 con 0x7f1a78102ef0 2024-09-19T08:04:24.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a7da62640 1 --2- 172.21.15.98:0/2297706962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a78102ef0 0x7f1a781a43d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1a7000b970 tx=0x7f1a7000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:24.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a6e7fc640 1 -- 172.21.15.98:0/2297706962 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70019070 con 0x7f1a78102ef0 2024-09-19T08:04:24.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a6e7fc640 1 -- 172.21.15.98:0/2297706962 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a7000cb40 con 0x7f1a78102ef0 2024-09-19T08:04:24.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a6e7fc640 1 -- 172.21.15.98:0/2297706962 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70007650 con 0x7f1a78102ef0 2024-09-19T08:04:24.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/2297706962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a781a4ae0 con 0x7f1a78102ef0 2024-09-19T08:04:24.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.456+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/2297706962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a781a4fd0 con 0x7f1a78102ef0 2024-09-19T08:04:24.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.457+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/2297706962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a78101200 con 0x7f1a78102ef0 2024-09-19T08:04:24.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.459+0000 7f1a6e7fc640 1 -- 172.21.15.98:0/2297706962 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 23) ==== 138639+0+0 (secure 0 0 0) 0x7f1a7001c070 con 0x7f1a78102ef0 2024-09-19T08:04:24.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.459+0000 7f1a6e7fc640 1 --2- 172.21.15.98:0/2297706962 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f1a640a3560 0x7f1a640a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:24.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.460+0000 7f1a7d261640 1 --2- 172.21.15.98:0/2297706962 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f1a640a3560 0x7f1a640a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:24.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.460+0000 7f1a6e7fc640 1 -- 172.21.15.98:0/2297706962 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f1a700cf9b0 con 0x7f1a78102ef0 2024-09-19T08:04:24.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.460+0000 7f1a7d261640 1 --2- 172.21.15.98:0/2297706962 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f1a640a3560 0x7f1a640a5a20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1a740058c0 tx=0x7f1a7400a4c0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:24.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.463+0000 7f1a6e7fc640 1 -- 172.21.15.98:0/2297706962 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f1a7009c290 con 0x7f1a78102ef0 2024-09-19T08:04:24.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.672+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/2297706962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a78109090 con 0x7f1a78102ef0 2024-09-19T08:04:24.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.673+0000 7f1a6e7fc640 1 -- 172.21.15.98:0/2297706962 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11983 (secure 0 0 0) 0x7f1a7009bb90 con 0x7f1a78102ef0 2024-09-19T08:04:24.674 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:04:24.674 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"btime":"2024-09-19T08:02:59:855935+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:02:59.805764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:04:24.674 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-09-19T08:04:24.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.675+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/2297706962 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f1a640a3560 msgr2=0x7f1a640a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:24.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.675+0000 7f1a7f4c5640 1 --2- 172.21.15.98:0/2297706962 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f1a640a3560 0x7f1a640a5a20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1a740058c0 tx=0x7f1a7400a4c0 comp rx=0 tx=0).stop 2024-09-19T08:04:24.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.675+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/2297706962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a78102ef0 msgr2=0x7f1a781a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:24.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.675+0000 7f1a7f4c5640 1 --2- 172.21.15.98:0/2297706962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a78102ef0 0x7f1a781a43d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1a7000b970 tx=0x7f1a7000be30 comp rx=0 tx=0).stop 2024-09-19T08:04:24.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.675+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/2297706962 shutdown_connections 2024-09-19T08:04:24.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.675+0000 7f1a7f4c5640 1 --2- 172.21.15.98:0/2297706962 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f1a640a3560 0x7f1a640a5a20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:24.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.675+0000 7f1a7f4c5640 1 --2- 172.21.15.98:0/2297706962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a78102ef0 0x7f1a781a43d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:24.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.675+0000 7f1a7f4c5640 1 --2- 172.21.15.98:0/2297706962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a78102570 0x7f1a78199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:24.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.675+0000 7f1a7f4c5640 1 --2- 172.21.15.98:0/2297706962 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1a78100e20 0x7f1a78199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:24.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.675+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/2297706962 >> 172.21.15.98:0/2297706962 conn(0x7f1a780fc7d0 msgr2=0x7f1a780febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:24.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.675+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/2297706962 shutdown_connections 2024-09-19T08:04:24.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:24.675+0000 7f1a7f4c5640 1 -- 172.21.15.98:0/2297706962 wait complete. 2024-09-19T08:04:24.867 INFO:tasks.workunit.client.0.smithi098.stderr:Note: switching to '6fad5dd73c93545e52f57f681e140a9c3529c11e'. 2024-09-19T08:04:24.867 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-19T08:04:24.867 INFO:tasks.workunit.client.0.smithi098.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-19T08:04:24.867 INFO:tasks.workunit.client.0.smithi098.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-19T08:04:24.867 INFO:tasks.workunit.client.0.smithi098.stderr:state without impacting any branches by switching back to a branch. 2024-09-19T08:04:24.867 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-19T08:04:24.867 INFO:tasks.workunit.client.0.smithi098.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-19T08:04:24.867 INFO:tasks.workunit.client.0.smithi098.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-19T08:04:24.867 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-19T08:04:24.868 INFO:tasks.workunit.client.0.smithi098.stderr: git switch -c 2024-09-19T08:04:24.868 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-19T08:04:24.868 INFO:tasks.workunit.client.0.smithi098.stderr:Or undo this operation with: 2024-09-19T08:04:24.868 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-19T08:04:24.868 INFO:tasks.workunit.client.0.smithi098.stderr: git switch - 2024-09-19T08:04:24.868 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-19T08:04:24.868 INFO:tasks.workunit.client.0.smithi098.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-19T08:04:24.868 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-19T08:04:24.868 INFO:tasks.workunit.client.0.smithi098.stderr:HEAD is now at 6fad5dd73c9 Merge pull request #59812 from neha-ojha/wip-rtd-squid 2024-09-19T08:04:24.873 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-19T08:04:24.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:24 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2297706962' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:24.949 INFO:tasks.workunit.client.0.smithi098.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-19T08:04:24.951 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-19T08:04:24.951 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-19T08:04:25.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:24 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2297706962' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:25.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:24 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2297706962' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:25.741 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 140.35 --await-for 45.0' 2024-09-19T08:04:25.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 140.35 --await-for 45.0 2024-09-19T08:04:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:26.119 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:25 smithi098 ceph-mon[31333]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:04:26.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:25 smithi162 ceph-mon[35260]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:04:26.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:25 smithi139 ceph-mon[34874]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:04:26.329 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-19T08:04:26.397 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-19T08:04:26.436 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-19T08:04:26.438 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-19T08:04:26.438 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-19T08:04:26.482 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-19T08:04:26.486 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T08:04:26.486 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-19T08:04:26.505 INFO:tasks.workunit:Running workunits matching suites/iozone.sh on client.0... 2024-09-19T08:04:26.506 INFO:tasks.workunit:Running workunit suites/iozone.sh... 2024-09-19T08:04:26.507 DEBUG:teuthology.orchestra.run.smithi098:workunit test suites/iozone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6fad5dd73c93545e52f57f681e140a9c3529c11e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh 2024-09-19T08:04:26.568 INFO:tasks.workunit.client.0.smithi098.stderr:+ iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-09-19T08:04:26.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.594+0000 7f618ef92640 1 -- 172.21.15.98:0/1136336053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6188101df0 msgr2=0x7f61881021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:26.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.594+0000 7f618ef92640 1 --2- 172.21.15.98:0/1136336053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6188101df0 0x7f61881021d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f617c009900 tx=0x7f617c02a410 comp rx=0 tx=0).stop 2024-09-19T08:04:26.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.594+0000 7f618ef92640 1 -- 172.21.15.98:0/1136336053 shutdown_connections 2024-09-19T08:04:26.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.594+0000 7f618ef92640 1 --2- 172.21.15.98:0/1136336053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6188106f20 0x7f618810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.594+0000 7f618ef92640 1 --2- 172.21.15.98:0/1136336053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61881027a0 0x7f61881067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.594+0000 7f618ef92640 1 --2- 172.21.15.98:0/1136336053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6188101df0 0x7f61881021d0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.594+0000 7f618ef92640 1 -- 172.21.15.98:0/1136336053 >> 172.21.15.98:0/1136336053 conn(0x7f61880fd8f0 msgr2=0x7f61880ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.594+0000 7f618ef92640 1 -- 172.21.15.98:0/1136336053 shutdown_connections 2024-09-19T08:04:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.594+0000 7f618ef92640 1 -- 172.21.15.98:0/1136336053 wait complete. 2024-09-19T08:04:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f618ef92640 1 Processor -- start 2024-09-19T08:04:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f618ef92640 1 -- start start 2024-09-19T08:04:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f618ef92640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6188101df0 0x7f6188198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f618ef92640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61881027a0 0x7f61881989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f618ef92640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6188106f20 0x7f618819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f618ef92640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6188111f80 con 0x7f61881027a0 2024-09-19T08:04:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f618ef92640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6188111e00 con 0x7f6188101df0 2024-09-19T08:04:26.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f618ef92640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6188112100 con 0x7f6188106f20 2024-09-19T08:04:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f6187fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61881027a0 0x7f61881989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f6187fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61881027a0 0x7f61881989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33516/0 (socket says 172.21.15.98:33516) 2024-09-19T08:04:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f6187fff640 1 -- 172.21.15.98:0/4064851823 learned_addr learned my addr 172.21.15.98:0/4064851823 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f618cd2e640 1 --2- 172.21.15.98:0/4064851823 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6188101df0 0x7f6188198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f618d52f640 1 --2- 172.21.15.98:0/4064851823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6188106f20 0x7f618819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f6187fff640 1 -- 172.21.15.98:0/4064851823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6188106f20 msgr2=0x7f618819cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f6187fff640 1 --2- 172.21.15.98:0/4064851823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6188106f20 0x7f618819cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f6187fff640 1 -- 172.21.15.98:0/4064851823 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6188101df0 msgr2=0x7f6188198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f6187fff640 1 --2- 172.21.15.98:0/4064851823 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6188101df0 0x7f6188198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.595+0000 7f6187fff640 1 -- 172.21.15.98:0/4064851823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618819d410 con 0x7f61881027a0 2024-09-19T08:04:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.596+0000 7f6187fff640 1 --2- 172.21.15.98:0/4064851823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61881027a0 0x7f61881989a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f617400d770 tx=0x7f617400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:26.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.596+0000 7f6185ffb640 1 -- 172.21.15.98:0/4064851823 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174017070 con 0x7f61881027a0 2024-09-19T08:04:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.596+0000 7f6185ffb640 1 -- 172.21.15.98:0/4064851823 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6174005500 con 0x7f61881027a0 2024-09-19T08:04:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.596+0000 7f618ef92640 1 -- 172.21.15.98:0/4064851823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618819d700 con 0x7f61881027a0 2024-09-19T08:04:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.596+0000 7f6185ffb640 1 -- 172.21.15.98:0/4064851823 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174004e00 con 0x7f61881027a0 2024-09-19T08:04:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.596+0000 7f618ef92640 1 -- 172.21.15.98:0/4064851823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61881a4fe0 con 0x7f61881027a0 2024-09-19T08:04:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.598+0000 7f6185ffb640 1 -- 172.21.15.98:0/4064851823 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 23) ==== 138639+0+0 (secure 0 0 0) 0x7f6174004fa0 con 0x7f61881027a0 2024-09-19T08:04:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f619136b740 1 Processor -- start 2024-09-19T08:04:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f6185ffb640 1 --2- 172.21.15.98:0/4064851823 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f61780a35e0 0x7f61780a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f619136b740 1 -- start start 2024-09-19T08:04:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f6185ffb640 1 -- 172.21.15.98:0/4064851823 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f61740cf8e0 con 0x7f61881027a0 2024-09-19T08:04:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f618d52f640 1 --2- 172.21.15.98:0/4064851823 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f61780a35e0 0x7f61780a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f619136b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffd850 0x55b0b8ffdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f619136b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffe170 0x55b0b90084f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f619136b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b900ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f619136b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b0b8f62160 con 0x55b0b9008a30 2024-09-19T08:04:26.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f619136b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b0b8f09720 con 0x55b0b8ffd850 2024-09-19T08:04:26.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f619136b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b0b8f5e120 con 0x55b0b8ffe170 2024-09-19T08:04:26.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f618d52f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b900ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f618d52f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b900ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33526/0 (socket says 172.21.15.98:33526) 2024-09-19T08:04:26.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f618d52f640 1 -- 172.21.15.98:0/3595917356 learned_addr learned my addr 172.21.15.98:0/3595917356 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:26.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f6187fff640 1 --2- 172.21.15.98:0/3595917356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffe170 0x55b0b90084f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.599+0000 7f618cd2e640 1 --2- 172.21.15.98:0/3595917356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffd850 0x55b0b8ffdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.600+0000 7f618d52f640 1 --2- 172.21.15.98:0/4064851823 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f61780a35e0 0x7f61780a5aa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6180013710 tx=0x7f6180013b10 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:26.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.600+0000 7f618d52f640 1 -- 172.21.15.98:0/3595917356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffe170 msgr2=0x55b0b90084f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:26.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.600+0000 7f618d52f640 1 --2- 172.21.15.98:0/3595917356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffe170 0x55b0b90084f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.600+0000 7f618d52f640 1 -- 172.21.15.98:0/3595917356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffd850 msgr2=0x55b0b8ffdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:26.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.600+0000 7f618d52f640 1 --2- 172.21.15.98:0/3595917356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffd850 0x55b0b8ffdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.600+0000 7f618d52f640 1 -- 172.21.15.98:0/3595917356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0b8cd3ba0 con 0x55b0b9008a30 2024-09-19T08:04:26.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.600+0000 7f618d52f640 1 --2- 172.21.15.98:0/3595917356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b900ae20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f618000ac00 tx=0x7f618001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89fa16935bda5a36 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:26.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.600+0000 7f6172ffd640 1 -- 172.21.15.98:0/3595917356 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6180013220 con 0x55b0b9008a30 2024-09-19T08:04:26.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.600+0000 7f6172ffd640 1 -- 172.21.15.98:0/3595917356 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6180003e50 con 0x55b0b9008a30 2024-09-19T08:04:26.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.600+0000 7f6172ffd640 1 -- 172.21.15.98:0/3595917356 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6180004410 con 0x55b0b9008a30 2024-09-19T08:04:26.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.601+0000 7f619136b740 1 -- 172.21.15.98:0/3595917356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 msgr2=0x55b0b900ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:26.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.601+0000 7f619136b740 1 --2- 172.21.15.98:0/3595917356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b900ae20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f618000ac00 tx=0x7f618001e000 comp rx=0 tx=0).stop 2024-09-19T08:04:26.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.601+0000 7f619136b740 1 -- 172.21.15.98:0/3595917356 shutdown_connections 2024-09-19T08:04:26.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.601+0000 7f619136b740 1 --2- 172.21.15.98:0/3595917356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b900ae20 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.601+0000 7f619136b740 1 --2- 172.21.15.98:0/3595917356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffe170 0x55b0b90084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.601+0000 7f619136b740 1 --2- 172.21.15.98:0/3595917356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffd850 0x55b0b8ffdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.601+0000 7f619136b740 1 -- 172.21.15.98:0/3595917356 >> 172.21.15.98:0/3595917356 conn(0x55b0b8ff9de0 msgr2=0x55b0b8ffa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.601+0000 7f619136b740 1 -- 172.21.15.98:0/3595917356 shutdown_connections 2024-09-19T08:04:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.601+0000 7f619136b740 1 -- 172.21.15.98:0/3595917356 wait complete. 2024-09-19T08:04:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.601+0000 7f619136b740 1 -- start start 2024-09-19T08:04:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.601+0000 7f619136b740 1 Processor -- start 2024-09-19T08:04:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f619136b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffd850 0x55b0b90058e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f619136b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b8ffe170 0x55b0b9005e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f619136b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b9008a30 0x55b0b9006360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f619136b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b0b900fa10 con 0x55b0b8ffe170 2024-09-19T08:04:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f619136b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b0b8eea900 con 0x55b0b9008a30 2024-09-19T08:04:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f619136b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b0b900fb90 con 0x55b0b8ffd850 2024-09-19T08:04:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f619136b740 10 client.0 _ll_register_callbacks cb 0x55b0b8ee8780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f6187fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b8ffe170 0x55b0b9005e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f6187fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b8ffe170 0x55b0b9005e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33540/0 (socket says 172.21.15.98:33540) 2024-09-19T08:04:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f6187fff640 1 -- 172.21.15.98:0/2266526091 learned_addr learned my addr 172.21.15.98:0/2266526091 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f618cd2e640 1 --2- 172.21.15.98:0/2266526091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffd850 0x55b0b90058e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f618d52f640 1 --2- 172.21.15.98:0/2266526091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b9008a30 0x55b0b9006360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f6187fff640 1 -- 172.21.15.98:0/2266526091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffd850 msgr2=0x55b0b90058e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:26.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f6187fff640 1 --2- 172.21.15.98:0/2266526091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffd850 0x55b0b90058e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f6187fff640 1 -- 172.21.15.98:0/2266526091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b9008a30 msgr2=0x55b0b9006360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f6187fff640 1 --2- 172.21.15.98:0/2266526091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b9008a30 0x55b0b9006360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f6187fff640 1 -- 172.21.15.98:0/2266526091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0b8f2ac50 con 0x55b0b8ffe170 2024-09-19T08:04:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f618d52f640 1 --2- 172.21.15.98:0/2266526091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b9008a30 0x55b0b9006360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f6187fff640 1 --2- 172.21.15.98:0/2266526091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b8ffe170 0x55b0b9005e20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6174097e70 tx=0x7f617409ba60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb065eb4d250cc5a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f6167fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f6167fff640 1 -- 172.21.15.98:0/2266526091 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61740b4060 con 0x55b0b8ffe170 2024-09-19T08:04:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f6167fff640 1 -- 172.21.15.98:0/2266526091 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f61740860b0 con 0x55b0b8ffe170 2024-09-19T08:04:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.602+0000 7f619136b740 1 -- 172.21.15.98:0/2266526091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6180003e50 con 0x55b0b8ffe170 2024-09-19T08:04:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.603+0000 7f6167fff640 1 -- 172.21.15.98:0/2266526091 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6174020040 con 0x55b0b8ffe170 2024-09-19T08:04:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.603+0000 7f619136b740 10 client.15810 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:26.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.603+0000 7f619136b740 1 -- 172.21.15.98:0/2266526091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b0b901b6f0 con 0x55b0b8ffe170 2024-09-19T08:04:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.603+0000 7f6167fff640 1 -- 172.21.15.98:0/2266526091 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6180003e50 con 0x55b0b8ffe170 2024-09-19T08:04:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.603+0000 7f6167fff640 1 -- 172.21.15.98:0/2266526091 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f61740b6ce0 con 0x55b0b8ffe170 2024-09-19T08:04:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.603+0000 7f619136b740 10 client.15810 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.603+0000 7f619136b740 10 client.15810 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-19T08:04:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.603+0000 7f619136b740 20 client.15810 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.603+0000 7f619136b740 1 --2- 172.21.15.98:0/2266526091 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b0b901cb00 0x55b0b901ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.603+0000 7f619136b740 4 client.15810 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:04:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.603+0000 7f619136b740 1 -- 172.21.15.98:0/2266526091 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0b8f6f890 con 0x55b0b901cb00 2024-09-19T08:04:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.603+0000 7f618cd2e640 1 --2- 172.21.15.98:0/2266526091 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b0b901cb00 0x55b0b901ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.604+0000 7f6167fff640 10 client.15810 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.604+0000 7f618cd2e640 1 --2- 172.21.15.98:0/2266526091 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b0b901cb00 0x55b0b901ef60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0054f82f93b748d server_cookie=f12afc7c35a4395b in_seq=0 out_seq=0 2024-09-19T08:04:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.604+0000 7f6167fff640 1 -- 172.21.15.98:0/2266526091 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b0b8f6f890 con 0x55b0b901cb00 2024-09-19T08:04:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.604+0000 7f6167fff640 10 client.15810 handle_command_reply: tid=0 2024-09-19T08:04:26.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.604+0000 7f619136b740 1 client.15810 shutdown 2024-09-19T08:04:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.604+0000 7f619136b740 1 -- 172.21.15.98:0/2266526091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b8ffe170 msgr2=0x55b0b9005e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.604+0000 7f619136b740 1 --2- 172.21.15.98:0/2266526091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b8ffe170 0x55b0b9005e20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6174097e70 tx=0x7f617409ba60 comp rx=0 tx=0).stop 2024-09-19T08:04:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.605+0000 7f619136b740 1 -- 172.21.15.98:0/2266526091 shutdown_connections 2024-09-19T08:04:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.605+0000 7f619136b740 1 --2- 172.21.15.98:0/2266526091 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b0b901cb00 0x55b0b901ef60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.605+0000 7f619136b740 1 --2- 172.21.15.98:0/2266526091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b9008a30 0x55b0b9006360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.605+0000 7f619136b740 1 --2- 172.21.15.98:0/2266526091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b8ffe170 0x55b0b9005e20 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.605+0000 7f6167fff640 0 client.15810 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.605+0000 7f619136b740 1 --2- 172.21.15.98:0/2266526091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffd850 0x55b0b90058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.605+0000 7f619136b740 1 -- 172.21.15.98:0/2266526091 >> 172.21.15.98:0/2266526091 conn(0x55b0b8ff9de0 msgr2=0x55b0b8fffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.605+0000 7f619136b740 1 -- 172.21.15.98:0/2266526091 shutdown_connections 2024-09-19T08:04:26.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.605+0000 7f619136b740 1 -- 172.21.15.98:0/2266526091 wait complete. 2024-09-19T08:04:26.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.605+0000 7f619136b740 20 client.15810 trim_cache size 0 max 16384 2024-09-19T08:04:26.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f619136b740 1 Processor -- start 2024-09-19T08:04:26.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f619136b740 1 -- start start 2024-09-19T08:04:26.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f619136b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffd850 0x55b0b9007850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f619136b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffe170 0x55b0b90247e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f619136b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b9024d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f619136b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61881ab650 con 0x55b0b9008a30 2024-09-19T08:04:26.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f619136b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f61881ad190 con 0x55b0b8ffd850 2024-09-19T08:04:26.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f619136b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f61881abd20 con 0x55b0b8ffe170 2024-09-19T08:04:26.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f618d52f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b9024d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f618d52f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b9024d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33546/0 (socket says 172.21.15.98:33546) 2024-09-19T08:04:26.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f618d52f640 1 -- 172.21.15.98:0/4218231157 learned_addr learned my addr 172.21.15.98:0/4218231157 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:26.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f6187fff640 1 --2- 172.21.15.98:0/4218231157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffe170 0x55b0b90247e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f618cd2e640 1 --2- 172.21.15.98:0/4218231157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffd850 0x55b0b9007850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f618d52f640 1 -- 172.21.15.98:0/4218231157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffe170 msgr2=0x55b0b90247e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:26.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f618d52f640 1 --2- 172.21.15.98:0/4218231157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffe170 0x55b0b90247e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f618d52f640 1 -- 172.21.15.98:0/4218231157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffd850 msgr2=0x55b0b9007850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:26.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f618d52f640 1 --2- 172.21.15.98:0/4218231157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffd850 0x55b0b9007850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f618d52f640 1 -- 172.21.15.98:0/4218231157 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0b9008250 con 0x55b0b9008a30 2024-09-19T08:04:26.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.622+0000 7f6187fff640 1 --2- 172.21.15.98:0/4218231157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffe170 0x55b0b90247e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:26.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.623+0000 7f618d52f640 1 --2- 172.21.15.98:0/4218231157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b9024d20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f618000ac00 tx=0x7f6180005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c5e5474987dbe94 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:26.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.623+0000 7f61727fc640 1 -- 172.21.15.98:0/4218231157 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618002d070 con 0x55b0b9008a30 2024-09-19T08:04:26.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.623+0000 7f61727fc640 1 -- 172.21.15.98:0/4218231157 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6180013300 con 0x55b0b9008a30 2024-09-19T08:04:26.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.623+0000 7f61727fc640 1 -- 172.21.15.98:0/4218231157 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618001eb90 con 0x55b0b9008a30 2024-09-19T08:04:26.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.623+0000 7f619136b740 1 -- 172.21.15.98:0/4218231157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 msgr2=0x55b0b9024d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:26.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.623+0000 7f619136b740 1 --2- 172.21.15.98:0/4218231157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b9024d20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f618000ac00 tx=0x7f6180005e80 comp rx=0 tx=0).stop 2024-09-19T08:04:26.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.624+0000 7f619136b740 1 -- 172.21.15.98:0/4218231157 shutdown_connections 2024-09-19T08:04:26.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.624+0000 7f619136b740 1 --2- 172.21.15.98:0/4218231157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b9024d20 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.624+0000 7f619136b740 1 --2- 172.21.15.98:0/4218231157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffe170 0x55b0b90247e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.624+0000 7f619136b740 1 --2- 172.21.15.98:0/4218231157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffd850 0x55b0b9007850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.624+0000 7f619136b740 1 -- 172.21.15.98:0/4218231157 >> 172.21.15.98:0/4218231157 conn(0x55b0b8ff9de0 msgr2=0x55b0b9007140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:26.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.624+0000 7f619136b740 1 -- 172.21.15.98:0/4218231157 shutdown_connections 2024-09-19T08:04:26.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.624+0000 7f619136b740 1 -- 172.21.15.98:0/4218231157 wait complete. 2024-09-19T08:04:26.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.624+0000 7f619136b740 1 -- start start 2024-09-19T08:04:26.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.624+0000 7f619136b740 1 Processor -- start 2024-09-19T08:04:26.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f619136b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffd850 0x55b0b90152f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f619136b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffe170 0x55b0b9010620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f619136b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b9010b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f619136b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61881acac0 con 0x55b0b9008a30 2024-09-19T08:04:26.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f619136b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f61881ac3f0 con 0x55b0b8ffe170 2024-09-19T08:04:26.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f619136b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f61881a9630 con 0x55b0b8ffd850 2024-09-19T08:04:26.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f619136b740 10 client.0 _ll_register_callbacks cb 0x7f617409b490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:26.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f618d52f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b9010b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f618d52f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b9010b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33554/0 (socket says 172.21.15.98:33554) 2024-09-19T08:04:26.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f618d52f640 1 -- 172.21.15.98:0/2456792081 learned_addr learned my addr 172.21.15.98:0/2456792081 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:26.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f6187fff640 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffe170 0x55b0b9010620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f618cd2e640 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffd850 0x55b0b90152f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f618d52f640 1 -- 172.21.15.98:0/2456792081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffd850 msgr2=0x55b0b90152f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:26.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f618d52f640 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffd850 0x55b0b90152f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f618d52f640 1 -- 172.21.15.98:0/2456792081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffe170 msgr2=0x55b0b9010620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:26.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f618d52f640 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffe170 0x55b0b9010620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f618d52f640 1 -- 172.21.15.98:0/2456792081 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0b9011120 con 0x55b0b9008a30 2024-09-19T08:04:26.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f618cd2e640 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffd850 0x55b0b90152f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:26.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f6187fff640 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffe170 0x55b0b9010620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:26.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f618d52f640 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b9010b60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f618000a340 tx=0x7f618001ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a437d2cd8904cd8a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:26.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f6167fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:26.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f6167fff640 1 -- 172.21.15.98:0/2456792081 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618002d070 con 0x55b0b9008a30 2024-09-19T08:04:26.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f6167fff640 1 -- 172.21.15.98:0/2456792081 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f6180004a00 con 0x55b0b9008a30 2024-09-19T08:04:26.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.625+0000 7f619136b740 1 -- 172.21.15.98:0/2456792081 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6180013300 con 0x55b0b9008a30 2024-09-19T08:04:26.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.626+0000 7f6167fff640 1 -- 172.21.15.98:0/2456792081 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6180042050 con 0x55b0b9008a30 2024-09-19T08:04:26.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.626+0000 7f619136b740 10 client.15816 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:26.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.626+0000 7f619136b740 1 -- 172.21.15.98:0/2456792081 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b0b9087560 con 0x55b0b9008a30 2024-09-19T08:04:26.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.626+0000 7f6167fff640 1 -- 172.21.15.98:0/2456792081 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6180013300 con 0x55b0b9008a30 2024-09-19T08:04:26.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.626+0000 7f6167fff640 1 -- 172.21.15.98:0/2456792081 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6180005050 con 0x55b0b9008a30 2024-09-19T08:04:26.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.626+0000 7f619136b740 10 client.15816 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:26.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.626+0000 7f619136b740 10 client.15816 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-19T08:04:26.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.626+0000 7f619136b740 20 client.15816 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:26.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.626+0000 7f619136b740 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b0b9087af0 0x55b0b9089ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:26.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.626+0000 7f619136b740 4 client.15816 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.35, "await_for": 45.0}] 2024-09-19T08:04:26.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.626+0000 7f619136b740 1 -- 172.21.15.98:0/2456792081 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.35, "await_for": 45.0}) -- 0x55b0b8f69be0 con 0x55b0b9087af0 2024-09-19T08:04:26.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.626+0000 7f618cd2e640 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b0b9087af0 0x55b0b9089ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:26.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.627+0000 7f6167fff640 10 client.15816 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:26.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.627+0000 7f618cd2e640 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b0b9087af0 0x55b0b9089ee0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77fde2531a35a851 server_cookie=55e3da44ec8fc5a5 in_seq=0 out_seq=0 2024-09-19T08:04:26.656 INFO:tasks.workunit.client.0.smithi098.stdout: Iozone: Performance Test of File I/O 2024-09-19T08:04:26.656 INFO:tasks.workunit.client.0.smithi098.stdout: Version $Revision: 3.494 $ 2024-09-19T08:04:26.656 INFO:tasks.workunit.client.0.smithi098.stdout: Compiled for 64 bit mode. 2024-09-19T08:04:26.656 INFO:tasks.workunit.client.0.smithi098.stdout: Build: linux-AMD64 2024-09-19T08:04:26.656 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:04:26.656 INFO:tasks.workunit.client.0.smithi098.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-09-19T08:04:26.656 INFO:tasks.workunit.client.0.smithi098.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-09-19T08:04:26.657 INFO:tasks.workunit.client.0.smithi098.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-09-19T08:04:26.657 INFO:tasks.workunit.client.0.smithi098.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-09-19T08:04:26.657 INFO:tasks.workunit.client.0.smithi098.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-09-19T08:04:26.657 INFO:tasks.workunit.client.0.smithi098.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-09-19T08:04:26.657 INFO:tasks.workunit.client.0.smithi098.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-09-19T08:04:26.657 INFO:tasks.workunit.client.0.smithi098.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-09-19T08:04:26.657 INFO:tasks.workunit.client.0.smithi098.stdout: Alexey Skidanov, Sudhir Kumar. 2024-09-19T08:04:26.657 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:04:26.657 INFO:tasks.workunit.client.0.smithi098.stdout: Run began: Thu Sep 19 08:04:26 2024 2024-09-19T08:04:26.657 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:04:26.657 INFO:tasks.workunit.client.0.smithi098.stdout: Include close in write timing 2024-09-19T08:04:26.657 INFO:tasks.workunit.client.0.smithi098.stdout: Include fsync in write timing 2024-09-19T08:04:26.657 INFO:tasks.workunit.client.0.smithi098.stdout: File size set to 1048576 kB 2024-09-19T08:04:26.657 INFO:tasks.workunit.client.0.smithi098.stdout: Record Size 16 kB 2024-09-19T08:04:26.658 INFO:tasks.workunit.client.0.smithi098.stdout: Command line used: iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-09-19T08:04:26.658 INFO:tasks.workunit.client.0.smithi098.stdout: Output is in kBytes/sec 2024-09-19T08:04:26.658 INFO:tasks.workunit.client.0.smithi098.stdout: Time Resolution = 0.000001 seconds. 2024-09-19T08:04:26.658 INFO:tasks.workunit.client.0.smithi098.stdout: Processor cache size set to 1024 kBytes. 2024-09-19T08:04:26.658 INFO:tasks.workunit.client.0.smithi098.stdout: Processor cache line size set to 32 bytes. 2024-09-19T08:04:26.658 INFO:tasks.workunit.client.0.smithi098.stdout: File stride size set to 17 * record size. 2024-09-19T08:04:26.658 INFO:tasks.workunit.client.0.smithi098.stdout: Throughput test with 1 process 2024-09-19T08:04:26.658 INFO:tasks.workunit.client.0.smithi098.stdout: Each process writes a 1048576 kByte file in 16 kByte records 2024-09-19T08:04:26.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.669+0000 7f6167fff640 1 -- 172.21.15.98:0/2456792081 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55b0b8f69be0 con 0x55b0b9087af0 2024-09-19T08:04:26.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.669+0000 7f6167fff640 10 client.15816 handle_command_reply: tid=0 2024-09-19T08:04:26.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.669+0000 7f619136b740 1 client.15816 shutdown 2024-09-19T08:04:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.670+0000 7f619136b740 1 -- 172.21.15.98:0/2456792081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 msgr2=0x55b0b9010b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.670+0000 7f619136b740 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b9010b60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f618000a340 tx=0x7f618001ebe0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.670+0000 7f619136b740 1 -- 172.21.15.98:0/2456792081 shutdown_connections 2024-09-19T08:04:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.670+0000 7f619136b740 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55b0b9087af0 0x55b0b9089ee0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.670+0000 7f619136b740 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0b9008a30 0x55b0b9010b60 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.670+0000 7f619136b740 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b0b8ffe170 0x55b0b9010620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.670+0000 7f619136b740 1 --2- 172.21.15.98:0/2456792081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0b8ffd850 0x55b0b90152f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.670+0000 7f619136b740 1 -- 172.21.15.98:0/2456792081 >> 172.21.15.98:0/2456792081 conn(0x55b0b8ff9de0 msgr2=0x55b0b8fffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:26.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.670+0000 7f6167fff640 0 client.15816 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:26.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.670+0000 7f619136b740 1 -- 172.21.15.98:0/2456792081 shutdown_connections 2024-09-19T08:04:26.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.670+0000 7f619136b740 1 -- 172.21.15.98:0/2456792081 wait complete. 2024-09-19T08:04:26.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.670+0000 7f619136b740 20 client.15816 trim_cache size 0 max 16384 2024-09-19T08:04:26.673 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:04:26.673 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 28, 2024-09-19T08:04:26.673 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24485, 2024-09-19T08:04:26.673 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 6, 2024-09-19T08:04:26.674 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-09-19T08:04:26.674 INFO:teuthology.orchestra.run.smithi098.stdout: "28e1fa50": { 2024-09-19T08:04:26.674 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 6, 2024-09-19T08:04:26.674 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-09-19T08:04:26.674 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-09-19T08:04:26.674 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-09-19T08:04:26.674 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-09-19T08:04:26.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:04:26.674 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 90.0, 2024-09-19T08:04:26.674 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 140.3, 2024-09-19T08:04:26.675 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-09-19T08:04:26.675 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-09-19T08:04:26.675 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-09-19T08:04:26.675 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-09-19T08:04:26.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-09-19T08:04:26.675 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-09-19T08:04:26.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:26.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:26.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:26.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:26.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:26.676 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:04:26.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.671+0000 7f618ef92640 1 -- 172.21.15.98:0/4064851823 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f61780a35e0 msgr2=0x7f61780a5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:26.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.671+0000 7f618ef92640 1 --2- 172.21.15.98:0/4064851823 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f61780a35e0 0x7f61780a5aa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6180013710 tx=0x7f6180013b10 comp rx=0 tx=0).stop 2024-09-19T08:04:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.671+0000 7f618ef92640 1 -- 172.21.15.98:0/4064851823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61881027a0 msgr2=0x7f61881989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.671+0000 7f618ef92640 1 --2- 172.21.15.98:0/4064851823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61881027a0 0x7f61881989a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f617400d770 tx=0x7f617400dc30 comp rx=0 tx=0).stop 2024-09-19T08:04:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.671+0000 7f618ef92640 1 -- 172.21.15.98:0/4064851823 shutdown_connections 2024-09-19T08:04:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.671+0000 7f618ef92640 1 --2- 172.21.15.98:0/4064851823 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f61780a35e0 0x7f61780a5aa0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.671+0000 7f618ef92640 1 --2- 172.21.15.98:0/4064851823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6188106f20 0x7f618819cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.671+0000 7f618ef92640 1 --2- 172.21.15.98:0/4064851823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61881027a0 0x7f61881989a0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.671+0000 7f618ef92640 1 --2- 172.21.15.98:0/4064851823 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6188101df0 0x7f6188198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.671+0000 7f618ef92640 1 -- 172.21.15.98:0/4064851823 >> 172.21.15.98:0/4064851823 conn(0x7f61880fd8f0 msgr2=0x7f61880fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.671+0000 7f618ef92640 1 -- 172.21.15.98:0/4064851823 shutdown_connections 2024-09-19T08:04:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:26.671+0000 7f618ef92640 1 -- 172.21.15.98:0/4064851823 wait complete. 2024-09-19T08:04:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:26 smithi162 ceph-mon[35260]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-19T08:04:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:26 smithi139 ceph-mon[34874]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-19T08:04:27.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:26 smithi098 ceph-mon[31333]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-19T08:04:27.441 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '28e1fa50', quiesce took 3.9 seconds. Will release after: 16.450000000000003 2024-09-19T08:04:29.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:28 smithi162 ceph-mon[35260]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-09-19T08:04:29.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:28 smithi139 ceph-mon[34874]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-09-19T08:04:29.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:28 smithi098 ceph-mon[31333]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-09-19T08:04:29.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:04:29.991 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.468+0000 7f5dc87ec640 1 -- 172.21.15.98:0/2248834037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc0101de0 msgr2=0x7f5dc01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.468+0000 7f5dc87ec640 1 --2- 172.21.15.98:0/2248834037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc0101de0 0x7f5dc01021c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009900 tx=0x7f5dbc02a410 comp rx=0 tx=0).stop 2024-09-19T08:04:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.468+0000 7f5dc87ec640 1 -- 172.21.15.98:0/2248834037 shutdown_connections 2024-09-19T08:04:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.468+0000 7f5dc87ec640 1 --2- 172.21.15.98:0/2248834037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5dc0106f10 0x7f5dc010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.468+0000 7f5dc87ec640 1 --2- 172.21.15.98:0/2248834037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dc0102790 0x7f5dc01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.468+0000 7f5dc87ec640 1 --2- 172.21.15.98:0/2248834037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc0101de0 0x7f5dc01021c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.468+0000 7f5dc87ec640 1 -- 172.21.15.98:0/2248834037 >> 172.21.15.98:0/2248834037 conn(0x7f5dc00fd8c0 msgr2=0x7f5dc00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.468+0000 7f5dc87ec640 1 -- 172.21.15.98:0/2248834037 shutdown_connections 2024-09-19T08:04:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.468+0000 7f5dc87ec640 1 -- 172.21.15.98:0/2248834037 wait complete. 2024-09-19T08:04:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.469+0000 7f5dc87ec640 1 Processor -- start 2024-09-19T08:04:30.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.469+0000 7f5dc87ec640 1 -- start start 2024-09-19T08:04:30.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.469+0000 7f5dc87ec640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dc0101de0 0x7f5dc00744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.469+0000 7f5dc87ec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc0102790 0x7f5dc0074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.469+0000 7f5dc87ec640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5dc0106f10 0x7f5dc0074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.469+0000 7f5dc87ec640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc0112010 con 0x7f5dc0102790 2024-09-19T08:04:30.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.469+0000 7f5dc87ec640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc0111e90 con 0x7f5dc0106f10 2024-09-19T08:04:30.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.469+0000 7f5dc87ec640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc0112190 con 0x7f5dc0101de0 2024-09-19T08:04:30.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.469+0000 7f5dc5d87640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc0102790 0x7f5dc0074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.469+0000 7f5dc5d87640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc0102790 0x7f5dc0074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33580/0 (socket says 172.21.15.98:33580) 2024-09-19T08:04:30.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.469+0000 7f5dc5d87640 1 -- 172.21.15.98:0/1099657149 learned_addr learned my addr 172.21.15.98:0/1099657149 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:30.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.470+0000 7f5dc6588640 1 --2- 172.21.15.98:0/1099657149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dc0101de0 0x7f5dc00744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.470+0000 7f5dc5d87640 1 -- 172.21.15.98:0/1099657149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dc0101de0 msgr2=0x7f5dc00744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.470+0000 7f5dc5d87640 1 --2- 172.21.15.98:0/1099657149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dc0101de0 0x7f5dc00744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.470+0000 7f5dc5d87640 1 -- 172.21.15.98:0/1099657149 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5dc0106f10 msgr2=0x7f5dc0074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.470+0000 7f5dc5d87640 1 --2- 172.21.15.98:0/1099657149 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5dc0106f10 0x7f5dc0074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.470+0000 7f5dc5d87640 1 -- 172.21.15.98:0/1099657149 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc01a6a70 con 0x7f5dc0102790 2024-09-19T08:04:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.470+0000 7f5dc5d87640 1 --2- 172.21.15.98:0/1099657149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc0102790 0x7f5dc0074a30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f5db000b4a0 tx=0x7f5db000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.470+0000 7f5db77fe640 1 -- 172.21.15.98:0/1099657149 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db0014070 con 0x7f5dc0102790 2024-09-19T08:04:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.470+0000 7f5db77fe640 1 -- 172.21.15.98:0/1099657149 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5db0010af0 con 0x7f5dc0102790 2024-09-19T08:04:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.470+0000 7f5dc87ec640 1 -- 172.21.15.98:0/1099657149 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc01a6d60 con 0x7f5dc0102790 2024-09-19T08:04:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.470+0000 7f5db77fe640 1 -- 172.21.15.98:0/1099657149 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db0017d10 con 0x7f5dc0102790 2024-09-19T08:04:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.470+0000 7f5dc87ec640 1 -- 172.21.15.98:0/1099657149 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc01a7260 con 0x7f5dc0102790 2024-09-19T08:04:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.472+0000 7f5db77fe640 1 -- 172.21.15.98:0/1099657149 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 23) ==== 138639+0+0 (secure 0 0 0) 0x7f5db0017070 con 0x7f5dc0102790 2024-09-19T08:04:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.472+0000 7f5dcabc5740 1 Processor -- start 2024-09-19T08:04:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dcabc5740 1 -- start start 2024-09-19T08:04:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dcabc5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a684880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dcabc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a684dc0 0x55ea2a68f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dcabc5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a68f680 0x55ea2a691a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dcabc5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ea2a5ea2e0 con 0x55ea2a6844a0 2024-09-19T08:04:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dcabc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ea2a591ce0 con 0x55ea2a684dc0 2024-09-19T08:04:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dcabc5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ea2a5e62a0 con 0x55ea2a68f680 2024-09-19T08:04:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dc6d89640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a684880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dc6d89640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a684880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33590/0 (socket says 172.21.15.98:33590) 2024-09-19T08:04:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dc6d89640 1 -- 172.21.15.98:0/1776482780 learned_addr learned my addr 172.21.15.98:0/1776482780 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5db77fe640 1 --2- 172.21.15.98:0/1099657149 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5dac0a35e0 0x7f5dac0a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dc6588640 1 --2- 172.21.15.98:0/1776482780 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a684dc0 0x55ea2a68f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dc5d87640 1 --2- 172.21.15.98:0/1776482780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a68f680 0x55ea2a691a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dc6d89640 1 --2- 172.21.15.98:0/1099657149 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5dac0a35e0 0x7f5dac0a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dc6d89640 1 -- 172.21.15.98:0/1776482780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a68f680 msgr2=0x55ea2a691a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:30.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dc6d89640 1 --2- 172.21.15.98:0/1776482780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a68f680 0x55ea2a691a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dc6d89640 1 -- 172.21.15.98:0/1776482780 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a684dc0 msgr2=0x55ea2a68f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:30.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dc6d89640 1 --2- 172.21.15.98:0/1776482780 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a684dc0 0x55ea2a68f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5dc6d89640 1 -- 172.21.15.98:0/1776482780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea2a576440 con 0x55ea2a6844a0 2024-09-19T08:04:30.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.473+0000 7f5db77fe640 1 -- 172.21.15.98:0/1099657149 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5db00cf1f0 con 0x7f5dc0102790 2024-09-19T08:04:30.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.474+0000 7f5dc6d89640 1 --2- 172.21.15.98:0/1776482780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a684880 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5db80126c0 tx=0x7f5db8012b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ade12274781256 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:30.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.474+0000 7f5db4ff9640 1 -- 172.21.15.98:0/1776482780 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db8012050 con 0x55ea2a6844a0 2024-09-19T08:04:30.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.474+0000 7f5db4ff9640 1 -- 172.21.15.98:0/1776482780 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5db8013040 con 0x55ea2a6844a0 2024-09-19T08:04:30.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.474+0000 7f5db4ff9640 1 -- 172.21.15.98:0/1776482780 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db801b070 con 0x55ea2a6844a0 2024-09-19T08:04:30.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.474+0000 7f5dcabc5740 1 -- 172.21.15.98:0/1776482780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 msgr2=0x55ea2a684880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.474+0000 7f5dc6d89640 1 --2- 172.21.15.98:0/1099657149 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5dac0a35e0 0x7f5dac0a5aa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5db801d000 tx=0x7f5db801b000 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.474+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/1776482780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a684880 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5db80126c0 tx=0x7f5db8012b80 comp rx=0 tx=0).stop 2024-09-19T08:04:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.474+0000 7f5dcabc5740 1 -- 172.21.15.98:0/1776482780 shutdown_connections 2024-09-19T08:04:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.474+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/1776482780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a68f680 0x55ea2a691a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.474+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/1776482780 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a684dc0 0x55ea2a68f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.474+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/1776482780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a684880 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.474+0000 7f5dcabc5740 1 -- 172.21.15.98:0/1776482780 >> 172.21.15.98:0/1776482780 conn(0x55ea2a680a30 msgr2=0x55ea2a680e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.475+0000 7f5dcabc5740 1 -- 172.21.15.98:0/1776482780 shutdown_connections 2024-09-19T08:04:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.475+0000 7f5dcabc5740 1 -- 172.21.15.98:0/1776482780 wait complete. 2024-09-19T08:04:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.475+0000 7f5dcabc5740 1 -- start start 2024-09-19T08:04:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.475+0000 7f5dcabc5740 1 Processor -- start 2024-09-19T08:04:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.475+0000 7f5dcabc5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a68ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.475+0000 7f5dcabc5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a684dc0 0x55ea2a68c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.475+0000 7f5dcabc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a68f680 0x55ea2a68c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.475+0000 7f5dcabc5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ea2a692250 con 0x55ea2a6844a0 2024-09-19T08:04:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.475+0000 7f5dcabc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ea2a5a5050 con 0x55ea2a68f680 2024-09-19T08:04:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.475+0000 7f5dcabc5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ea2a6923d0 con 0x55ea2a684dc0 2024-09-19T08:04:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.475+0000 7f5dcabc5740 10 client.0 _ll_register_callbacks cb 0x55ea2a5e5160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.475+0000 7f5dc6588640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a68ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.475+0000 7f5dc6588640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a68ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33596/0 (socket says 172.21.15.98:33596) 2024-09-19T08:04:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.475+0000 7f5dc6588640 1 -- 172.21.15.98:0/3333214725 learned_addr learned my addr 172.21.15.98:0/3333214725 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dc6588640 1 -- 172.21.15.98:0/3333214725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a684dc0 msgr2=0x55ea2a68c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:30.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dc6588640 1 --2- 172.21.15.98:0/3333214725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a684dc0 0x55ea2a68c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dc6588640 1 -- 172.21.15.98:0/3333214725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a68f680 msgr2=0x55ea2a68c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:30.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dc6588640 1 --2- 172.21.15.98:0/3333214725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a68f680 0x55ea2a68c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dc6588640 1 -- 172.21.15.98:0/3333214725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea2a35b3c0 con 0x55ea2a6844a0 2024-09-19T08:04:30.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dc6588640 1 --2- 172.21.15.98:0/3333214725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a68ef90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009900 tx=0x7f5dbc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d8ab182e317752 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:30.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5da9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:30.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5da9ffb640 1 -- 172.21.15.98:0/3333214725 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc035040 con 0x55ea2a6844a0 2024-09-19T08:04:30.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5da9ffb640 1 -- 172.21.15.98:0/3333214725 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5dbc005020 con 0x55ea2a6844a0 2024-09-19T08:04:30.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dcabc5740 1 -- 172.21.15.98:0/3333214725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5db8013040 con 0x55ea2a6844a0 2024-09-19T08:04:30.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5da9ffb640 1 -- 172.21.15.98:0/3333214725 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5dbc005d50 con 0x55ea2a6844a0 2024-09-19T08:04:30.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dcabc5740 10 client.15828 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:30.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dcabc5740 1 -- 172.21.15.98:0/3333214725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ea2a69dbd0 con 0x55ea2a6844a0 2024-09-19T08:04:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5da9ffb640 1 -- 172.21.15.98:0/3333214725 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5db8013040 con 0x55ea2a6844a0 2024-09-19T08:04:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5da9ffb640 1 -- 172.21.15.98:0/3333214725 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5dbc003680 con 0x55ea2a6844a0 2024-09-19T08:04:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dcabc5740 10 client.15828 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dcabc5740 10 client.15828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:04:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dcabc5740 20 client.15828 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/3333214725 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ea2a69efe0 0x55ea2a6a14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dcabc5740 4 client.15828 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:04:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.476+0000 7f5dcabc5740 1 -- 172.21.15.98:0/3333214725 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea2a5f7850 con 0x55ea2a69efe0 2024-09-19T08:04:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.477+0000 7f5dc5d87640 1 --2- 172.21.15.98:0/3333214725 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ea2a69efe0 0x55ea2a6a14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.477+0000 7f5da9ffb640 10 client.15828 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.477+0000 7f5dc5d87640 1 --2- 172.21.15.98:0/3333214725 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ea2a69efe0 0x55ea2a6a14a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75cff53d789d1d5e server_cookie=ce72dd9bf49e4759 in_seq=0 out_seq=0 2024-09-19T08:04:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5da9ffb640 1 -- 172.21.15.98:0/3333214725 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ea2a5f7850 con 0x55ea2a69efe0 2024-09-19T08:04:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5da9ffb640 10 client.15828 handle_command_reply: tid=0 2024-09-19T08:04:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5dcabc5740 1 client.15828 shutdown 2024-09-19T08:04:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5dcabc5740 1 -- 172.21.15.98:0/3333214725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 msgr2=0x55ea2a68ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/3333214725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a68ef90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009900 tx=0x7f5dbc004270 comp rx=0 tx=0).stop 2024-09-19T08:04:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5dcabc5740 1 -- 172.21.15.98:0/3333214725 shutdown_connections 2024-09-19T08:04:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/3333214725 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ea2a69efe0 0x55ea2a6a14a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/3333214725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a68f680 0x55ea2a68c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/3333214725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a684dc0 0x55ea2a68c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/3333214725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a68ef90 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5da9ffb640 0 client.15828 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5dcabc5740 1 -- 172.21.15.98:0/3333214725 >> 172.21.15.98:0/3333214725 conn(0x55ea2a680a30 msgr2=0x55ea2a67fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5dcabc5740 1 -- 172.21.15.98:0/3333214725 shutdown_connections 2024-09-19T08:04:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5dcabc5740 1 -- 172.21.15.98:0/3333214725 wait complete. 2024-09-19T08:04:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.478+0000 7f5dcabc5740 20 client.15828 trim_cache size 0 max 16384 2024-09-19T08:04:30.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.495+0000 7f5dcabc5740 1 Processor -- start 2024-09-19T08:04:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.495+0000 7f5dcabc5740 1 -- start start 2024-09-19T08:04:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dcabc5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a69b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dcabc5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a684dc0 0x55ea2a6a6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dcabc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a68f680 0x55ea2a6a72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dcabc5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc01afb10 con 0x55ea2a6844a0 2024-09-19T08:04:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dcabc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc01b1650 con 0x55ea2a68f680 2024-09-19T08:04:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dcabc5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc01b01e0 con 0x55ea2a684dc0 2024-09-19T08:04:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dc6588640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a69b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dc6588640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a69b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33604/0 (socket says 172.21.15.98:33604) 2024-09-19T08:04:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dc6588640 1 -- 172.21.15.98:0/4213444764 learned_addr learned my addr 172.21.15.98:0/4213444764 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dc5d87640 1 --2- 172.21.15.98:0/4213444764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a684dc0 0x55ea2a6a6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dc6d89640 1 --2- 172.21.15.98:0/4213444764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a68f680 0x55ea2a6a72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dc6588640 1 -- 172.21.15.98:0/4213444764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a684dc0 msgr2=0x55ea2a6a6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dc6588640 1 --2- 172.21.15.98:0/4213444764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a684dc0 0x55ea2a6a6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dc6588640 1 -- 172.21.15.98:0/4213444764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a68f680 msgr2=0x55ea2a6a72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dc6588640 1 --2- 172.21.15.98:0/4213444764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a68f680 0x55ea2a6a72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dc6588640 1 -- 172.21.15.98:0/4213444764 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea2a6842b0 con 0x55ea2a6844a0 2024-09-19T08:04:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dc6588640 1 --2- 172.21.15.98:0/4213444764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a69b4b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc045510 tx=0x7f5dbc004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29bf6084ffef0e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dabfff640 1 -- 172.21.15.98:0/4213444764 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc035070 con 0x55ea2a6844a0 2024-09-19T08:04:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dabfff640 1 -- 172.21.15.98:0/4213444764 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dbc003e70 con 0x55ea2a6844a0 2024-09-19T08:04:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.496+0000 7f5dabfff640 1 -- 172.21.15.98:0/4213444764 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc04e650 con 0x55ea2a6844a0 2024-09-19T08:04:30.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.497+0000 7f5dcabc5740 1 -- 172.21.15.98:0/4213444764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 msgr2=0x55ea2a69b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:30.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.497+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/4213444764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a69b4b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc045510 tx=0x7f5dbc004360 comp rx=0 tx=0).stop 2024-09-19T08:04:30.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.497+0000 7f5dcabc5740 1 -- 172.21.15.98:0/4213444764 shutdown_connections 2024-09-19T08:04:30.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.497+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/4213444764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a68f680 0x55ea2a6a72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.497+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/4213444764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a684dc0 0x55ea2a6a6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.497+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/4213444764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a69b4b0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.497+0000 7f5dcabc5740 1 -- 172.21.15.98:0/4213444764 >> 172.21.15.98:0/4213444764 conn(0x55ea2a680a30 msgr2=0x55ea2a68ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:30.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.497+0000 7f5dcabc5740 1 -- 172.21.15.98:0/4213444764 shutdown_connections 2024-09-19T08:04:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.497+0000 7f5dcabc5740 1 -- 172.21.15.98:0/4213444764 wait complete. 2024-09-19T08:04:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dcabc5740 1 -- start start 2024-09-19T08:04:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dcabc5740 1 Processor -- start 2024-09-19T08:04:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dcabc5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a697620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dcabc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a684dc0 0x55ea2a697b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dcabc5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a68f680 0x55ea2a6945f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dcabc5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc01b0f80 con 0x55ea2a6844a0 2024-09-19T08:04:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dcabc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc01b08b0 con 0x55ea2a684dc0 2024-09-19T08:04:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dcabc5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc01adaf0 con 0x55ea2a68f680 2024-09-19T08:04:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dcabc5740 10 client.0 _ll_register_callbacks cb 0x7f5dbc009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dc6588640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a697620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dc6588640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a697620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33620/0 (socket says 172.21.15.98:33620) 2024-09-19T08:04:30.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dc6588640 1 -- 172.21.15.98:0/2976266037 learned_addr learned my addr 172.21.15.98:0/2976266037 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:30.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dc6d89640 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a68f680 0x55ea2a6945f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dc5d87640 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a684dc0 0x55ea2a697b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dc6588640 1 -- 172.21.15.98:0/2976266037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a68f680 msgr2=0x55ea2a6945f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:30.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dc6588640 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a68f680 0x55ea2a6945f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dc6588640 1 -- 172.21.15.98:0/2976266037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a684dc0 msgr2=0x55ea2a697b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:30.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dc6588640 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a684dc0 0x55ea2a697b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dc6588640 1 -- 172.21.15.98:0/2976266037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea2a698220 con 0x55ea2a6844a0 2024-09-19T08:04:30.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dc6d89640 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a68f680 0x55ea2a6945f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.498+0000 7f5dc5d87640 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a684dc0 0x55ea2a697b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5dc6588640 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a697620 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc003a80 tx=0x7f5dbc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4e60e98841de89 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5da9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5da9ffb640 1 -- 172.21.15.98:0/2976266037 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc035070 con 0x55ea2a6844a0 2024-09-19T08:04:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5da9ffb640 1 -- 172.21.15.98:0/2976266037 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5dbc045790 con 0x55ea2a6844a0 2024-09-19T08:04:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5dcabc5740 1 -- 172.21.15.98:0/2976266037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dbc003e70 con 0x55ea2a6844a0 2024-09-19T08:04:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5da9ffb640 1 -- 172.21.15.98:0/2976266037 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5dbc065050 con 0x55ea2a6844a0 2024-09-19T08:04:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5dcabc5740 10 client.15834 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5dcabc5740 1 -- 172.21.15.98:0/2976266037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ea2a70a320 con 0x55ea2a6844a0 2024-09-19T08:04:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5da9ffb640 1 -- 172.21.15.98:0/2976266037 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dbc003e70 con 0x55ea2a6844a0 2024-09-19T08:04:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5da9ffb640 1 -- 172.21.15.98:0/2976266037 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5dbc04e450 con 0x55ea2a6844a0 2024-09-19T08:04:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5dcabc5740 10 client.15834 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5dcabc5740 10 client.15834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:04:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5dcabc5740 20 client.15834 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ea2a70a970 0x55ea2a70cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:30.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5dcabc5740 4 client.15834 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:04:30.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.499+0000 7f5dcabc5740 1 -- 172.21.15.98:0/2976266037 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ea2a68cb10 con 0x55ea2a70a970 2024-09-19T08:04:30.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5dc5d87640 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ea2a70a970 0x55ea2a70cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:30.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5dc5d87640 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ea2a70a970 0x55ea2a70cd60 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fd2e994e0808f6e server_cookie=565b6eb1401432fd in_seq=0 out_seq=0 2024-09-19T08:04:30.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5da9ffb640 10 client.15834 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:30.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5da9ffb640 1 -- 172.21.15.98:0/2976266037 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ea2a68cb10 con 0x55ea2a70a970 2024-09-19T08:04:30.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5da9ffb640 10 client.15834 handle_command_reply: tid=0 2024-09-19T08:04:30.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5dcabc5740 1 client.15834 shutdown 2024-09-19T08:04:30.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5dcabc5740 1 -- 172.21.15.98:0/2976266037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 msgr2=0x55ea2a697620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:30.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a697620 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc003a80 tx=0x7f5dbc004270 comp rx=0 tx=0).stop 2024-09-19T08:04:30.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5dcabc5740 1 -- 172.21.15.98:0/2976266037 shutdown_connections 2024-09-19T08:04:30.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ea2a70a970 0x55ea2a70cd60 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea2a68f680 0x55ea2a6945f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ea2a684dc0 0x55ea2a697b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5dcabc5740 1 --2- 172.21.15.98:0/2976266037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea2a6844a0 0x55ea2a697620 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5dcabc5740 1 -- 172.21.15.98:0/2976266037 >> 172.21.15.98:0/2976266037 conn(0x55ea2a680a30 msgr2=0x55ea2a67f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:30.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.500+0000 7f5da9ffb640 0 client.15834 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:30.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dcabc5740 1 -- 172.21.15.98:0/2976266037 shutdown_connections 2024-09-19T08:04:30.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dcabc5740 1 -- 172.21.15.98:0/2976266037 wait complete. 2024-09-19T08:04:30.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dcabc5740 20 client.15834 trim_cache size 0 max 16384 2024-09-19T08:04:30.514 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:04:30.514 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:04:30.514 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:04:30.514 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:04:30.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dc87ec640 1 -- 172.21.15.98:0/1099657149 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5dac0a35e0 msgr2=0x7f5dac0a5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:30.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dc87ec640 1 --2- 172.21.15.98:0/1099657149 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5dac0a35e0 0x7f5dac0a5aa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5db801d000 tx=0x7f5db801b000 comp rx=0 tx=0).stop 2024-09-19T08:04:30.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dc87ec640 1 -- 172.21.15.98:0/1099657149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc0102790 msgr2=0x7f5dc0074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:30.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dc87ec640 1 --2- 172.21.15.98:0/1099657149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc0102790 0x7f5dc0074a30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f5db000b4a0 tx=0x7f5db000b960 comp rx=0 tx=0).stop 2024-09-19T08:04:30.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dc87ec640 1 -- 172.21.15.98:0/1099657149 shutdown_connections 2024-09-19T08:04:30.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dc87ec640 1 --2- 172.21.15.98:0/1099657149 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5dac0a35e0 0x7f5dac0a5aa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dc87ec640 1 --2- 172.21.15.98:0/1099657149 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5dc0106f10 0x7f5dc0074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dc87ec640 1 --2- 172.21.15.98:0/1099657149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc0102790 0x7f5dc0074a30 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dc87ec640 1 --2- 172.21.15.98:0/1099657149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dc0101de0 0x7f5dc00744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:30.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dc87ec640 1 -- 172.21.15.98:0/1099657149 >> 172.21.15.98:0/1099657149 conn(0x7f5dc00fd8c0 msgr2=0x7f5dc00fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:30.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dc87ec640 1 -- 172.21.15.98:0/1099657149 shutdown_connections 2024-09-19T08:04:30.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:30.501+0000 7f5dc87ec640 1 -- 172.21.15.98:0/1099657149 wait complete. 2024-09-19T08:04:31.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:30 smithi162 ceph-mon[35260]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-09-19T08:04:31.123 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:30 smithi098 ceph-mon[31333]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-09-19T08:04:31.163 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:04:31.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:04:31.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:30 smithi139 ceph-mon[34874]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-09-19T08:04:31.421 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:31.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.878+0000 7ff87d3be640 1 -- 172.21.15.98:0/2980115610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff878100a80 msgr2=0x7ff878100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.878+0000 7ff87d3be640 1 --2- 172.21.15.98:0/2980115610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff878100a80 0x7ff878100ee0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff86c009930 tx=0x7ff86c02a420 comp rx=0 tx=0).stop 2024-09-19T08:04:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.878+0000 7ff87d3be640 1 -- 172.21.15.98:0/2980115610 shutdown_connections 2024-09-19T08:04:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.878+0000 7ff87d3be640 1 --2- 172.21.15.98:0/2980115610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff878101420 0x7ff87810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.878+0000 7ff87d3be640 1 --2- 172.21.15.98:0/2980115610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff878100a80 0x7ff878100ee0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.878+0000 7ff87d3be640 1 --2- 172.21.15.98:0/2980115610 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff878106a80 0x7ff878106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.878+0000 7ff87d3be640 1 -- 172.21.15.98:0/2980115610 >> 172.21.15.98:0/2980115610 conn(0x7ff8780fc7a0 msgr2=0x7ff8780febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.878+0000 7ff87d3be640 1 -- 172.21.15.98:0/2980115610 shutdown_connections 2024-09-19T08:04:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.878+0000 7ff87d3be640 1 -- 172.21.15.98:0/2980115610 wait complete. 2024-09-19T08:04:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.879+0000 7ff87d3be640 1 Processor -- start 2024-09-19T08:04:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.879+0000 7ff87d3be640 1 -- start start 2024-09-19T08:04:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.879+0000 7ff87d3be640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff878100a80 0x7ff8781984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.879+0000 7ff87d3be640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff878101420 0x7ff8781989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:31.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.879+0000 7ff87d3be640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff878106a80 0x7ff87819cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:31.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.879+0000 7ff87d3be640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff878111fa0 con 0x7ff878106a80 2024-09-19T08:04:31.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.879+0000 7ff87d3be640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff878111e20 con 0x7ff878101420 2024-09-19T08:04:31.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.879+0000 7ff87d3be640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff878112120 con 0x7ff878100a80 2024-09-19T08:04:31.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.879+0000 7ff8777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff878106a80 0x7ff87819cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:31.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.879+0000 7ff8777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff878106a80 0x7ff87819cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33636/0 (socket says 172.21.15.98:33636) 2024-09-19T08:04:31.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff8777fe640 1 -- 172.21.15.98:0/2970529808 learned_addr learned my addr 172.21.15.98:0/2970529808 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:31.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff876ffd640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff878100a80 0x7ff8781984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:31.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff8767fc640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff878101420 0x7ff8781989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:31.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff8777fe640 1 -- 172.21.15.98:0/2970529808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff878100a80 msgr2=0x7ff8781984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:31.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff8777fe640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff878100a80 0x7ff8781984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:31.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff8777fe640 1 -- 172.21.15.98:0/2970529808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff878101420 msgr2=0x7ff8781989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:31.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff8777fe640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff878101420 0x7ff8781989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:31.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff8777fe640 1 -- 172.21.15.98:0/2970529808 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87819d450 con 0x7ff878106a80 2024-09-19T08:04:31.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff8767fc640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff878101420 0x7ff8781989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:31.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff8777fe640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff878106a80 0x7ff87819cd70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff870009fd0 tx=0x7ff87000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:31.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff85ffff640 1 -- 172.21.15.98:0/2970529808 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870011830 con 0x7ff878106a80 2024-09-19T08:04:31.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff87d3be640 1 -- 172.21.15.98:0/2970529808 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff87819d740 con 0x7ff878106a80 2024-09-19T08:04:31.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff85ffff640 1 -- 172.21.15.98:0/2970529808 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff870016070 con 0x7ff878106a80 2024-09-19T08:04:31.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff85ffff640 1 -- 172.21.15.98:0/2970529808 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87000f2e0 con 0x7ff878106a80 2024-09-19T08:04:31.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.880+0000 7ff87d3be640 1 -- 172.21.15.98:0/2970529808 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8781a5020 con 0x7ff878106a80 2024-09-19T08:04:31.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.881+0000 7ff87d3be640 1 -- 172.21.15.98:0/2970529808 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff878100ee0 con 0x7ff878106a80 2024-09-19T08:04:31.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.883+0000 7ff85ffff640 1 -- 172.21.15.98:0/2970529808 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 23) ==== 138639+0+0 (secure 0 0 0) 0x7ff8700119d0 con 0x7ff878106a80 2024-09-19T08:04:31.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.884+0000 7ff85ffff640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff8600a3560 0x7ff8600a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:31.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.884+0000 7ff876ffd640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff8600a3560 0x7ff8600a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:31.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.884+0000 7ff85ffff640 1 -- 172.21.15.98:0/2970529808 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff8700cfc10 con 0x7ff878106a80 2024-09-19T08:04:31.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.884+0000 7ff876ffd640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff8600a3560 0x7ff8600a5a20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff868005fd0 tx=0x7ff868009760 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:31.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:31.887+0000 7ff85ffff640 1 -- 172.21.15.98:0/2970529808 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ff87009c4f0 con 0x7ff878106a80 2024-09-19T08:04:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:32.087+0000 7ff87d3be640 1 -- 172.21.15.98:0/2970529808 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff878109070 con 0x7ff878106a80 2024-09-19T08:04:32.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:32.088+0000 7ff85ffff640 1 -- 172.21.15.98:0/2970529808 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11983 (secure 0 0 0) 0x7ff87009bdf0 con 0x7ff878106a80 2024-09-19T08:04:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:04:32.089 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"btime":"2024-09-19T08:02:59:855935+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:02:59.805764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:04:32.090 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-09-19T08:04:32.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:32.091+0000 7ff87d3be640 1 -- 172.21.15.98:0/2970529808 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff8600a3560 msgr2=0x7ff8600a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:32.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:32.091+0000 7ff87d3be640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff8600a3560 0x7ff8600a5a20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff868005fd0 tx=0x7ff868009760 comp rx=0 tx=0).stop 2024-09-19T08:04:32.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:32.091+0000 7ff87d3be640 1 -- 172.21.15.98:0/2970529808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff878106a80 msgr2=0x7ff87819cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:32.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:32.091+0000 7ff87d3be640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff878106a80 0x7ff87819cd70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff870009fd0 tx=0x7ff87000b8e0 comp rx=0 tx=0).stop 2024-09-19T08:04:32.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:32.091+0000 7ff87d3be640 1 -- 172.21.15.98:0/2970529808 shutdown_connections 2024-09-19T08:04:32.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:32.091+0000 7ff87d3be640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff8600a3560 0x7ff8600a5a20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:32.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:32.091+0000 7ff87d3be640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff878106a80 0x7ff87819cd70 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:32.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:32.091+0000 7ff87d3be640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff878101420 0x7ff8781989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:32.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:32.091+0000 7ff87d3be640 1 --2- 172.21.15.98:0/2970529808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff878100a80 0x7ff8781984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:32.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:32.091+0000 7ff87d3be640 1 -- 172.21.15.98:0/2970529808 >> 172.21.15.98:0/2970529808 conn(0x7ff8780fc7a0 msgr2=0x7ff8780fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:32.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:32.091+0000 7ff87d3be640 1 -- 172.21.15.98:0/2970529808 shutdown_connections 2024-09-19T08:04:32.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:32.091+0000 7ff87d3be640 1 -- 172.21.15.98:0/2970529808 wait complete. 2024-09-19T08:04:32.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:32 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2970529808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:32.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:32 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2970529808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:32.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:32 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2970529808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:32.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:04:33.001 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:33.382 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:33 smithi098 ceph-mon[31333]: pgmap v49: 97 pgs: 97 active+clean; 5.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 344 KiB/s wr, 14 op/s 2024-09-19T08:04:33.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:33 smithi139 ceph-mon[34874]: pgmap v49: 97 pgs: 97 active+clean; 5.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 344 KiB/s wr, 14 op/s 2024-09-19T08:04:33.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.462+0000 7fa4d9bcf640 1 -- 172.21.15.98:0/795823669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4d4100a70 msgr2=0x7fa4d4100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:33.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.462+0000 7fa4d9bcf640 1 --2- 172.21.15.98:0/795823669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4d4100a70 0x7fa4d4100ed0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009930 tx=0x7fa4c802a420 comp rx=0 tx=0).stop 2024-09-19T08:04:33.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.462+0000 7fa4d9bcf640 1 -- 172.21.15.98:0/795823669 shutdown_connections 2024-09-19T08:04:33.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.462+0000 7fa4d9bcf640 1 --2- 172.21.15.98:0/795823669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d4101410 0x7fa4d410d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.462+0000 7fa4d9bcf640 1 --2- 172.21.15.98:0/795823669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4d4100a70 0x7fa4d4100ed0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.462+0000 7fa4d9bcf640 1 --2- 172.21.15.98:0/795823669 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4d4106a70 0x7fa4d4106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.462+0000 7fa4d9bcf640 1 -- 172.21.15.98:0/795823669 >> 172.21.15.98:0/795823669 conn(0x7fa4d40fc790 msgr2=0x7fa4d40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.462+0000 7fa4d9bcf640 1 -- 172.21.15.98:0/795823669 shutdown_connections 2024-09-19T08:04:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.462+0000 7fa4d9bcf640 1 -- 172.21.15.98:0/795823669 wait complete. 2024-09-19T08:04:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.462+0000 7fa4d9bcf640 1 Processor -- start 2024-09-19T08:04:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.462+0000 7fa4d9bcf640 1 -- start start 2024-09-19T08:04:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d9bcf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4d4100a70 0x7fa4d4198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d9bcf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d4101410 0x7fa4d41989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d9bcf640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4d4106a70 0x7fa4d419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d9bcf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d4111f90 con 0x7fa4d4100a70 2024-09-19T08:04:33.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d9bcf640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d4111e10 con 0x7fa4d4106a70 2024-09-19T08:04:33.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d9bcf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d4112110 con 0x7fa4d4101410 2024-09-19T08:04:33.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4d4100a70 0x7fa4d4198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4d4100a70 0x7fa4d4198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33656/0 (socket says 172.21.15.98:33656) 2024-09-19T08:04:33.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d37fe640 1 -- 172.21.15.98:0/2821724444 learned_addr learned my addr 172.21.15.98:0/2821724444 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:33.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d2ffd640 1 --2- 172.21.15.98:0/2821724444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d4101410 0x7fa4d41989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d3fff640 1 --2- 172.21.15.98:0/2821724444 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4d4106a70 0x7fa4d419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d37fe640 1 -- 172.21.15.98:0/2821724444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d4101410 msgr2=0x7fa4d41989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:33.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d37fe640 1 --2- 172.21.15.98:0/2821724444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d4101410 0x7fa4d41989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d37fe640 1 -- 172.21.15.98:0/2821724444 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4d4106a70 msgr2=0x7fa4d419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:33.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d37fe640 1 --2- 172.21.15.98:0/2821724444 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4d4106a70 0x7fa4d419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d37fe640 1 -- 172.21.15.98:0/2821724444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d419d440 con 0x7fa4d4100a70 2024-09-19T08:04:33.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d37fe640 1 --2- 172.21.15.98:0/2821724444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4d4100a70 0x7fa4d4198490 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400eab0 tx=0x7fa4c400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:33.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.463+0000 7fa4d0ff9640 1 -- 172.21.15.98:0/2821724444 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c400ce40 con 0x7fa4d4100a70 2024-09-19T08:04:33.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.464+0000 7fa4d0ff9640 1 -- 172.21.15.98:0/2821724444 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c4011950 con 0x7fa4d4100a70 2024-09-19T08:04:33.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.464+0000 7fa4d9bcf640 1 -- 172.21.15.98:0/2821724444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d419d730 con 0x7fa4d4100a70 2024-09-19T08:04:33.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.464+0000 7fa4d0ff9640 1 -- 172.21.15.98:0/2821724444 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4010ae0 con 0x7fa4d4100a70 2024-09-19T08:04:33.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.464+0000 7fa4d9bcf640 1 -- 172.21.15.98:0/2821724444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d41a5010 con 0x7fa4d4100a70 2024-09-19T08:04:33.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.466+0000 7fa4d0ff9640 1 -- 172.21.15.98:0/2821724444 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fa4c4011b00 con 0x7fa4d4100a70 2024-09-19T08:04:33.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.466+0000 7fa4dbfa8740 1 Processor -- start 2024-09-19T08:04:33.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.466+0000 7fa4dbfa8740 1 -- start start 2024-09-19T08:04:33.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.466+0000 7fa4dbfa8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f6850b40 0x5593f684f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.466+0000 7fa4dbfa8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 0x5593f684fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.466+0000 7fa4dbfa8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f685a370 0x5593f685c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.466+0000 7fa4dbfa8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5593f67b52f0 con 0x5593f685a370 2024-09-19T08:04:33.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.466+0000 7fa4dbfa8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5593f675ccf0 con 0x5593f6850b40 2024-09-19T08:04:33.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.466+0000 7fa4dbfa8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5593f67b12b0 con 0x5593f684f5d0 2024-09-19T08:04:33.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.466+0000 7fa4d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f685a370 0x5593f685c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.466+0000 7fa4d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f685a370 0x5593f685c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33658/0 (socket says 172.21.15.98:33658) 2024-09-19T08:04:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.466+0000 7fa4d2ffd640 1 -- 172.21.15.98:0/252924182 learned_addr learned my addr 172.21.15.98:0/252924182 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4d37fe640 1 --2- 172.21.15.98:0/252924182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 0x5593f684fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4d3fff640 1 --2- 172.21.15.98:0/252924182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f6850b40 0x5593f684f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4d0ff9640 1 --2- 172.21.15.98:0/2821724444 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa4bc0a3430 0x7fa4bc0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4d3fff640 1 --2- 172.21.15.98:0/2821724444 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa4bc0a3430 0x7fa4bc0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4d2ffd640 1 -- 172.21.15.98:0/252924182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 msgr2=0x5593f684fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4d2ffd640 1 --2- 172.21.15.98:0/252924182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 0x5593f684fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4d0ff9640 1 -- 172.21.15.98:0/2821724444 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fa4c40d0210 con 0x7fa4d4100a70 2024-09-19T08:04:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4d2ffd640 1 -- 172.21.15.98:0/252924182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f6850b40 msgr2=0x5593f684f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4d2ffd640 1 --2- 172.21.15.98:0/252924182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f6850b40 0x5593f684f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4d2ffd640 1 -- 172.21.15.98:0/252924182 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593f67413a0 con 0x5593f685a370 2024-09-19T08:04:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4d2ffd640 1 --2- 172.21.15.98:0/252924182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f685a370 0x5593f685c760 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa4c802a940 tx=0x7fa4c8005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=717e75b604cbe545 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4d3fff640 1 --2- 172.21.15.98:0/2821724444 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa4bc0a3430 0x7fa4bc0a58f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009710 tx=0x7fa4cc004730 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4b9ffb640 1 -- 172.21.15.98:0/252924182 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8040070 con 0x5593f685a370 2024-09-19T08:04:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4b9ffb640 1 -- 172.21.15.98:0/252924182 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c80318f0 con 0x5593f685a370 2024-09-19T08:04:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.467+0000 7fa4b9ffb640 1 -- 172.21.15.98:0/252924182 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8035040 con 0x5593f685a370 2024-09-19T08:04:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.468+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/252924182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f685a370 msgr2=0x5593f685c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.468+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/252924182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f685a370 0x5593f685c760 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa4c802a940 tx=0x7fa4c8005c00 comp rx=0 tx=0).stop 2024-09-19T08:04:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.468+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/252924182 shutdown_connections 2024-09-19T08:04:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.468+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/252924182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f685a370 0x5593f685c760 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.468+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/252924182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 0x5593f684fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.468+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/252924182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f6850b40 0x5593f684f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.468+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/252924182 >> 172.21.15.98:0/252924182 conn(0x5593f6818310 msgr2=0x5593f684bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.468+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/252924182 shutdown_connections 2024-09-19T08:04:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.468+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/252924182 wait complete. 2024-09-19T08:04:33.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.468+0000 7fa4dbfa8740 1 -- start start 2024-09-19T08:04:33.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4dbfa8740 1 Processor -- start 2024-09-19T08:04:33.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4dbfa8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 0x5593f6859d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4dbfa8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f6850b40 0x5593f6856fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4dbfa8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f685a370 0x5593f6857570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4dbfa8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5593f685d010 con 0x5593f6850b40 2024-09-19T08:04:33.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4dbfa8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5593f6770060 con 0x5593f685a370 2024-09-19T08:04:33.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4dbfa8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5593f685d190 con 0x5593f684f5d0 2024-09-19T08:04:33.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4dbfa8740 10 client.0 _ll_register_callbacks cb 0x5593f64c38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:33.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f6850b40 0x5593f6856fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f6850b40 0x5593f6856fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33674/0 (socket says 172.21.15.98:33674) 2024-09-19T08:04:33.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4d2ffd640 1 -- 172.21.15.98:0/2594495342 learned_addr learned my addr 172.21.15.98:0/2594495342 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:33.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4d3fff640 1 --2- 172.21.15.98:0/2594495342 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f685a370 0x5593f6857570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4d37fe640 1 --2- 172.21.15.98:0/2594495342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 0x5593f6859d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4d2ffd640 1 -- 172.21.15.98:0/2594495342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 msgr2=0x5593f6859d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:33.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4d2ffd640 1 --2- 172.21.15.98:0/2594495342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 0x5593f6859d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4d2ffd640 1 -- 172.21.15.98:0/2594495342 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f685a370 msgr2=0x5593f6857570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:33.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4d2ffd640 1 --2- 172.21.15.98:0/2594495342 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f685a370 0x5593f6857570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.469+0000 7fa4d2ffd640 1 -- 172.21.15.98:0/2594495342 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593f65263c0 con 0x5593f6850b40 2024-09-19T08:04:33.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4d2ffd640 1 --2- 172.21.15.98:0/2594495342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f6850b40 0x5593f6856fd0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009a50 tx=0x7fa4c80052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b2d5bf551434b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:33.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:33.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4b2ffd640 1 -- 172.21.15.98:0/2594495342 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8040030 con 0x5593f6850b40 2024-09-19T08:04:33.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4b2ffd640 1 -- 172.21.15.98:0/2594495342 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fa4c8004260 con 0x5593f6850b40 2024-09-19T08:04:33.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/2594495342 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4c80318f0 con 0x5593f6850b40 2024-09-19T08:04:33.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4b2ffd640 1 -- 172.21.15.98:0/2594495342 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa4c8056050 con 0x5593f6850b40 2024-09-19T08:04:33.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4dbfa8740 10 client.15852 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:33.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/2594495342 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5593f6868a60 con 0x5593f6850b40 2024-09-19T08:04:33.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4b2ffd640 1 -- 172.21.15.98:0/2594495342 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c80318f0 con 0x5593f6850b40 2024-09-19T08:04:33.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4b2ffd640 1 -- 172.21.15.98:0/2594495342 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa4c8003680 con 0x5593f6850b40 2024-09-19T08:04:33.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4dbfa8740 10 client.15852 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:33.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4dbfa8740 10 client.15852 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:04:33.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4dbfa8740 20 client.15852 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:33.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/2594495342 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5593f6869db0 0x5593f686c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4dbfa8740 4 client.15852 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:04:33.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.470+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/2594495342 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593f67c2860 con 0x5593f6869db0 2024-09-19T08:04:33.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.471+0000 7fa4d37fe640 1 --2- 172.21.15.98:0/2594495342 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5593f6869db0 0x5593f686c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.471+0000 7fa4b2ffd640 10 client.15852 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:04:33.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.471+0000 7fa4d37fe640 1 --2- 172.21.15.98:0/2594495342 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5593f6869db0 0x5593f686c1a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ed3a20285145caae server_cookie=625524347e2501c9 in_seq=0 out_seq=0 2024-09-19T08:04:33.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4b2ffd640 1 -- 172.21.15.98:0/2594495342 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5593f67c2860 con 0x5593f6869db0 2024-09-19T08:04:33.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4b2ffd640 10 client.15852 handle_command_reply: tid=0 2024-09-19T08:04:33.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4dbfa8740 1 client.15852 shutdown 2024-09-19T08:04:33.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/2594495342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f6850b40 msgr2=0x5593f6856fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:33.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/2594495342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f6850b40 0x5593f6856fd0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009a50 tx=0x7fa4c80052d0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/2594495342 shutdown_connections 2024-09-19T08:04:33.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/2594495342 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5593f6869db0 0x5593f686c1a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4b2ffd640 0 client.15852 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:04:33.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/2594495342 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f685a370 0x5593f6857570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/2594495342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f6850b40 0x5593f6856fd0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/2594495342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 0x5593f6859d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/2594495342 >> 172.21.15.98:0/2594495342 conn(0x5593f6818310 msgr2=0x5593f6851960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:33.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/2594495342 shutdown_connections 2024-09-19T08:04:33.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/2594495342 wait complete. 2024-09-19T08:04:33.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.472+0000 7fa4dbfa8740 20 client.15852 trim_cache size 0 max 16384 2024-09-19T08:04:33.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4dbfa8740 1 Processor -- start 2024-09-19T08:04:33.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4dbfa8740 1 -- start start 2024-09-19T08:04:33.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4dbfa8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 0x5593f684abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4dbfa8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f6850b40 0x5593f6851fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4dbfa8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f685a370 0x5593f6852500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4dbfa8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d41ad7b0 con 0x5593f685a370 2024-09-19T08:04:33.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4dbfa8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d41af2f0 con 0x5593f6850b40 2024-09-19T08:04:33.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4dbfa8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d41ade80 con 0x5593f684f5d0 2024-09-19T08:04:33.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4d3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f685a370 0x5593f6852500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4d3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f685a370 0x5593f6852500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33676/0 (socket says 172.21.15.98:33676) 2024-09-19T08:04:33.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4d3fff640 1 -- 172.21.15.98:0/530317683 learned_addr learned my addr 172.21.15.98:0/530317683 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:33.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4d2ffd640 1 --2- 172.21.15.98:0/530317683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f6850b40 0x5593f6851fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4d37fe640 1 --2- 172.21.15.98:0/530317683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 0x5593f684abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4d3fff640 1 -- 172.21.15.98:0/530317683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 msgr2=0x5593f684abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4d3fff640 1 --2- 172.21.15.98:0/530317683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 0x5593f684abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4d3fff640 1 -- 172.21.15.98:0/530317683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f6850b40 msgr2=0x5593f6851fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4d3fff640 1 --2- 172.21.15.98:0/530317683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f6850b40 0x5593f6851fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4d3fff640 1 -- 172.21.15.98:0/530317683 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593f6855520 con 0x5593f685a370 2024-09-19T08:04:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.485+0000 7fa4d2ffd640 1 --2- 172.21.15.98:0/530317683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f6850b40 0x5593f6851fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.486+0000 7fa4d3fff640 1 --2- 172.21.15.98:0/530317683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f685a370 0x5593f6852500 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc004100 tx=0x7fa4cc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cdeb87f90cb96bd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.486+0000 7fa4b97fa640 1 -- 172.21.15.98:0/530317683 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc028020 con 0x5593f685a370 2024-09-19T08:04:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.486+0000 7fa4b97fa640 1 -- 172.21.15.98:0/530317683 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4cc003c30 con 0x5593f685a370 2024-09-19T08:04:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.486+0000 7fa4b97fa640 1 -- 172.21.15.98:0/530317683 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc010720 con 0x5593f685a370 2024-09-19T08:04:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.486+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/530317683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f685a370 msgr2=0x5593f6852500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.486+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/530317683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f685a370 0x5593f6852500 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc004100 tx=0x7fa4cc005850 comp rx=0 tx=0).stop 2024-09-19T08:04:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.486+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/530317683 shutdown_connections 2024-09-19T08:04:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.486+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/530317683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f685a370 0x5593f6852500 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.486+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/530317683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f6850b40 0x5593f6851fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.486+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/530317683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f684f5d0 0x5593f684abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.486+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/530317683 >> 172.21.15.98:0/530317683 conn(0x5593f6818310 msgr2=0x5593f684a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.486+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/530317683 shutdown_connections 2024-09-19T08:04:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.486+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/530317683 wait complete. 2024-09-19T08:04:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4dbfa8740 1 -- start start 2024-09-19T08:04:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4dbfa8740 1 Processor -- start 2024-09-19T08:04:33.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4dbfa8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f684f5d0 0x5593f6862420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4dbfa8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f6850b40 0x5593f6862960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4dbfa8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f685a370 0x5593f685f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4dbfa8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d41aec20 con 0x5593f684f5d0 2024-09-19T08:04:33.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4dbfa8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d41ae550 con 0x5593f685a370 2024-09-19T08:04:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4dbfa8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d41ab790 con 0x5593f6850b40 2024-09-19T08:04:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4dbfa8740 10 client.0 _ll_register_callbacks cb 0x7fa4cc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4d37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f684f5d0 0x5593f6862420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4d37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f684f5d0 0x5593f6862420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33686/0 (socket says 172.21.15.98:33686) 2024-09-19T08:04:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4d37fe640 1 -- 172.21.15.98:0/4189551372 learned_addr learned my addr 172.21.15.98:0/4189551372 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4d2ffd640 1 --2- 172.21.15.98:0/4189551372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f6850b40 0x5593f6862960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4d3fff640 1 --2- 172.21.15.98:0/4189551372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f685a370 0x5593f685f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4d37fe640 1 -- 172.21.15.98:0/4189551372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f6850b40 msgr2=0x5593f6862960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4d37fe640 1 --2- 172.21.15.98:0/4189551372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f6850b40 0x5593f6862960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4d37fe640 1 -- 172.21.15.98:0/4189551372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f685a370 msgr2=0x5593f685f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4d37fe640 1 --2- 172.21.15.98:0/4189551372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f685a370 0x5593f685f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4d37fe640 1 -- 172.21.15.98:0/4189551372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593f685f930 con 0x5593f684f5d0 2024-09-19T08:04:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.487+0000 7fa4d2ffd640 1 --2- 172.21.15.98:0/4189551372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f6850b40 0x5593f6862960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4d37fe640 1 --2- 172.21.15.98:0/4189551372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f684f5d0 0x5593f6862420 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40cdf90 tx=0x7fa4c409c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ee9f99b4a65600 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4b2ffd640 1 -- 172.21.15.98:0/4189551372 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c40a08d0 con 0x5593f684f5d0 2024-09-19T08:04:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4b2ffd640 1 -- 172.21.15.98:0/4189551372 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fa4c4098b90 con 0x5593f684f5d0 2024-09-19T08:04:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/4189551372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4cc003c30 con 0x5593f684f5d0 2024-09-19T08:04:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4b2ffd640 1 -- 172.21.15.98:0/4189551372 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa4c40a5190 con 0x5593f684f5d0 2024-09-19T08:04:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4dbfa8740 10 client.15858 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/4189551372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5593f68d5000 con 0x5593f684f5d0 2024-09-19T08:04:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4b2ffd640 1 -- 172.21.15.98:0/4189551372 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4cc003c30 con 0x5593f684f5d0 2024-09-19T08:04:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4b2ffd640 1 -- 172.21.15.98:0/4189551372 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa4c400f040 con 0x5593f684f5d0 2024-09-19T08:04:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4dbfa8740 10 client.15858 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4dbfa8740 10 client.15858 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:04:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4dbfa8740 20 client.15858 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/4189551372 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5593f6869db0 0x5593f68d76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4dbfa8740 4 client.15858 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:04:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.488+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/4189551372 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5593f67bcd70 con 0x5593f6869db0 2024-09-19T08:04:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.489+0000 7fa4d2ffd640 1 --2- 172.21.15.98:0/4189551372 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5593f6869db0 0x5593f68d76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.489+0000 7fa4d2ffd640 1 --2- 172.21.15.98:0/4189551372 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5593f6869db0 0x5593f68d76c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bcc519b16d2210a1 server_cookie=9c1e9c46cad3ed35 in_seq=0 out_seq=0 2024-09-19T08:04:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.489+0000 7fa4b2ffd640 10 client.15858 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:04:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.489+0000 7fa4b2ffd640 1 -- 172.21.15.98:0/4189551372 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5593f67bcd70 con 0x5593f6869db0 2024-09-19T08:04:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.489+0000 7fa4b2ffd640 10 client.15858 handle_command_reply: tid=0 2024-09-19T08:04:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.489+0000 7fa4dbfa8740 1 client.15858 shutdown 2024-09-19T08:04:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/4189551372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f684f5d0 msgr2=0x5593f6862420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/4189551372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f684f5d0 0x5593f6862420 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40cdf90 tx=0x7fa4c409c3a0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/4189551372 shutdown_connections 2024-09-19T08:04:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/4189551372 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5593f6869db0 0x5593f68d76c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/4189551372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593f685a370 0x5593f685f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/4189551372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f6850b40 0x5593f6862960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4b2ffd640 0 client.15858 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:04:33.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4dbfa8740 1 --2- 172.21.15.98:0/4189551372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593f684f5d0 0x5593f6862420 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/4189551372 >> 172.21.15.98:0/4189551372 conn(0x5593f6818310 msgr2=0x5593f684ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:33.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/4189551372 shutdown_connections 2024-09-19T08:04:33.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4dbfa8740 1 -- 172.21.15.98:0/4189551372 wait complete. 2024-09-19T08:04:33.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4dbfa8740 20 client.15858 trim_cache size 0 max 16384 2024-09-19T08:04:33.497 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:04:33.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4d9bcf640 1 -- 172.21.15.98:0/2821724444 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa4bc0a3430 msgr2=0x7fa4bc0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:33.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4d9bcf640 1 --2- 172.21.15.98:0/2821724444 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa4bc0a3430 0x7fa4bc0a58f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009710 tx=0x7fa4cc004730 comp rx=0 tx=0).stop 2024-09-19T08:04:33.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4d9bcf640 1 -- 172.21.15.98:0/2821724444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4d4100a70 msgr2=0x7fa4d4198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:33.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4d9bcf640 1 --2- 172.21.15.98:0/2821724444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4d4100a70 0x7fa4d4198490 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400eab0 tx=0x7fa4c400ef70 comp rx=0 tx=0).stop 2024-09-19T08:04:33.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4d9bcf640 1 -- 172.21.15.98:0/2821724444 shutdown_connections 2024-09-19T08:04:33.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4d9bcf640 1 --2- 172.21.15.98:0/2821724444 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa4bc0a3430 0x7fa4bc0a58f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4d9bcf640 1 --2- 172.21.15.98:0/2821724444 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4d4106a70 0x7fa4d419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4d9bcf640 1 --2- 172.21.15.98:0/2821724444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d4101410 0x7fa4d41989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4d9bcf640 1 --2- 172.21.15.98:0/2821724444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4d4100a70 0x7fa4d4198490 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:33.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.490+0000 7fa4d9bcf640 1 -- 172.21.15.98:0/2821724444 >> 172.21.15.98:0/2821724444 conn(0x7fa4d40fc790 msgr2=0x7fa4d40fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:33.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.491+0000 7fa4d9bcf640 1 -- 172.21.15.98:0/2821724444 shutdown_connections 2024-09-19T08:04:33.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:33.491+0000 7fa4d9bcf640 1 -- 172.21.15.98:0/2821724444 wait complete. 2024-09-19T08:04:33.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:33 smithi162 ceph-mon[35260]: pgmap v49: 97 pgs: 97 active+clean; 5.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 344 KiB/s wr, 14 op/s 2024-09-19T08:04:34.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:04:34.336 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:34.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:34 smithi139 ceph-mon[34874]: mgrmap e24: x(active, since 92s), standbys: z, y 2024-09-19T08:04:34.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:34 smithi098 ceph-mon[31333]: mgrmap e24: x(active, since 92s), standbys: z, y 2024-09-19T08:04:34.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:34 smithi162 ceph-mon[35260]: mgrmap e24: x(active, since 92s), standbys: z, y 2024-09-19T08:04:34.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.832+0000 7f078e8c6640 1 -- 172.21.15.98:0/4126508668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0788102790 msgr2=0x7f07881067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:34.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.832+0000 7f078e8c6640 1 --2- 172.21.15.98:0/4126508668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0788102790 0x7f07881067e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0770009900 tx=0x7f077002a410 comp rx=0 tx=0).stop 2024-09-19T08:04:34.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.832+0000 7f078e8c6640 1 -- 172.21.15.98:0/4126508668 shutdown_connections 2024-09-19T08:04:34.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.832+0000 7f078e8c6640 1 --2- 172.21.15.98:0/4126508668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0788106f10 0x7f078810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.832+0000 7f078e8c6640 1 --2- 172.21.15.98:0/4126508668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0788102790 0x7f07881067e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.832+0000 7f078e8c6640 1 --2- 172.21.15.98:0/4126508668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0788101de0 0x7f07881021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.832+0000 7f078e8c6640 1 -- 172.21.15.98:0/4126508668 >> 172.21.15.98:0/4126508668 conn(0x7f07880fd8c0 msgr2=0x7f07880ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.832+0000 7f078e8c6640 1 -- 172.21.15.98:0/4126508668 shutdown_connections 2024-09-19T08:04:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.832+0000 7f078e8c6640 1 -- 172.21.15.98:0/4126508668 wait complete. 2024-09-19T08:04:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.833+0000 7f078e8c6640 1 Processor -- start 2024-09-19T08:04:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.833+0000 7f078e8c6640 1 -- start start 2024-09-19T08:04:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.833+0000 7f078e8c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0788101de0 0x7f0788198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.833+0000 7f078e8c6640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0788102790 0x7f07881989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.833+0000 7f078e8c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0788106f10 0x7f078819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.833+0000 7f078e8c6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0788111f70 con 0x7f0788101de0 2024-09-19T08:04:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.833+0000 7f078e8c6640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0788111df0 con 0x7f0788102790 2024-09-19T08:04:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.833+0000 7f078e8c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f07881120f0 con 0x7f0788106f10 2024-09-19T08:04:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.833+0000 7f0787fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0788101de0 0x7f0788198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.833+0000 7f0787fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0788101de0 0x7f0788198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33706/0 (socket says 172.21.15.98:33706) 2024-09-19T08:04:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.833+0000 7f0787fff640 1 -- 172.21.15.98:0/745800824 learned_addr learned my addr 172.21.15.98:0/745800824 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.833+0000 7f078ce63640 1 --2- 172.21.15.98:0/745800824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0788106f10 0x7f078819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.833+0000 7f07877fe640 1 --2- 172.21.15.98:0/745800824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0788102790 0x7f07881989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.834+0000 7f0787fff640 1 -- 172.21.15.98:0/745800824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0788106f10 msgr2=0x7f078819cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.834+0000 7f0787fff640 1 --2- 172.21.15.98:0/745800824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0788106f10 0x7f078819cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.834+0000 7f0787fff640 1 -- 172.21.15.98:0/745800824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0788102790 msgr2=0x7f07881989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.834+0000 7f0787fff640 1 --2- 172.21.15.98:0/745800824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0788102790 0x7f07881989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.834+0000 7f0787fff640 1 -- 172.21.15.98:0/745800824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078819d420 con 0x7f0788101de0 2024-09-19T08:04:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.834+0000 7f0787fff640 1 --2- 172.21.15.98:0/745800824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0788101de0 0x7f0788198470 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f077800d770 tx=0x7f077800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.834+0000 7f07857fa640 1 -- 172.21.15.98:0/745800824 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778017070 con 0x7f0788101de0 2024-09-19T08:04:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.834+0000 7f07857fa640 1 -- 172.21.15.98:0/745800824 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0778005500 con 0x7f0788101de0 2024-09-19T08:04:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.834+0000 7f078e8c6640 1 -- 172.21.15.98:0/745800824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078819d710 con 0x7f0788101de0 2024-09-19T08:04:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.834+0000 7f07857fa640 1 -- 172.21.15.98:0/745800824 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778004e00 con 0x7f0788101de0 2024-09-19T08:04:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.834+0000 7f078e8c6640 1 -- 172.21.15.98:0/745800824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07881a4ff0 con 0x7f0788101de0 2024-09-19T08:04:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.836+0000 7f07857fa640 1 -- 172.21.15.98:0/745800824 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f0778004fa0 con 0x7f0788101de0 2024-09-19T08:04:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.836+0000 7f0790c9f740 1 Processor -- start 2024-09-19T08:04:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.836+0000 7f0790c9f740 1 -- start start 2024-09-19T08:04:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0790c9f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0bbb40 0x56275e0ba090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0790c9f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0ba5d0 0x56275e0baa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0790c9f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0c5370 0x56275e0c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0790c9f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56275e0202f0 con 0x56275e0ba5d0 2024-09-19T08:04:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0790c9f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56275dfc7cf0 con 0x56275e0bbb40 2024-09-19T08:04:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0790c9f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56275e01c2b0 con 0x56275e0c5370 2024-09-19T08:04:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0787fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0ba5d0 0x56275e0baa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0787fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0ba5d0 0x56275e0baa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33712/0 (socket says 172.21.15.98:33712) 2024-09-19T08:04:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0787fff640 1 -- 172.21.15.98:0/3846021406 learned_addr learned my addr 172.21.15.98:0/3846021406 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f07877fe640 1 --2- 172.21.15.98:0/3846021406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0c5370 0x56275e0c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f078ce63640 1 --2- 172.21.15.98:0/3846021406 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0bbb40 0x56275e0ba090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f07857fa640 1 --2- 172.21.15.98:0/745800824 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f07740a33b0 0x7f07740a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0787fff640 1 -- 172.21.15.98:0/3846021406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0c5370 msgr2=0x56275e0c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0787fff640 1 --2- 172.21.15.98:0/3846021406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0c5370 0x56275e0c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0787fff640 1 -- 172.21.15.98:0/3846021406 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0bbb40 msgr2=0x56275e0ba090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0787fff640 1 --2- 172.21.15.98:0/3846021406 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0bbb40 0x56275e0ba090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0787fff640 1 -- 172.21.15.98:0/3846021406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56275dfac3a0 con 0x56275e0ba5d0 2024-09-19T08:04:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f07877fe640 1 --2- 172.21.15.98:0/3846021406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0c5370 0x56275e0c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f078ce63640 1 --2- 172.21.15.98:0/745800824 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f07740a33b0 0x7f07740a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f07857fa640 1 -- 172.21.15.98:0/745800824 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f07780cf720 con 0x7f0788101de0 2024-09-19T08:04:34.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.837+0000 7f0787fff640 1 --2- 172.21.15.98:0/3846021406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0ba5d0 0x56275e0baa10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f07780ccd80 tx=0x7f077806e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9db5f82fb52150 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.838+0000 7f077e7fc640 1 -- 172.21.15.98:0/3846021406 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778020020 con 0x56275e0ba5d0 2024-09-19T08:04:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.838+0000 7f077e7fc640 1 -- 172.21.15.98:0/3846021406 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07780046c0 con 0x56275e0ba5d0 2024-09-19T08:04:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.838+0000 7f077e7fc640 1 -- 172.21.15.98:0/3846021406 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07780a0f10 con 0x56275e0ba5d0 2024-09-19T08:04:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.838+0000 7f078ce63640 1 --2- 172.21.15.98:0/745800824 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f07740a33b0 0x7f07740a5870 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f078000a2c0 tx=0x7f0780004a40 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.838+0000 7f0790c9f740 1 -- 172.21.15.98:0/3846021406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0ba5d0 msgr2=0x56275e0baa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.838+0000 7f0790c9f740 1 --2- 172.21.15.98:0/3846021406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0ba5d0 0x56275e0baa10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f07780ccd80 tx=0x7f077806e8b0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.838+0000 7f0790c9f740 1 -- 172.21.15.98:0/3846021406 shutdown_connections 2024-09-19T08:04:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.838+0000 7f0790c9f740 1 --2- 172.21.15.98:0/3846021406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0c5370 0x56275e0c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.838+0000 7f0790c9f740 1 --2- 172.21.15.98:0/3846021406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0ba5d0 0x56275e0baa10 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.838+0000 7f0790c9f740 1 --2- 172.21.15.98:0/3846021406 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0bbb40 0x56275e0ba090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.838+0000 7f0790c9f740 1 -- 172.21.15.98:0/3846021406 >> 172.21.15.98:0/3846021406 conn(0x56275e083310 msgr2=0x56275e0b6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.838+0000 7f0790c9f740 1 -- 172.21.15.98:0/3846021406 shutdown_connections 2024-09-19T08:04:34.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.838+0000 7f0790c9f740 1 -- 172.21.15.98:0/3846021406 wait complete. 2024-09-19T08:04:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.839+0000 7f0790c9f740 1 -- start start 2024-09-19T08:04:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.839+0000 7f0790c9f740 1 Processor -- start 2024-09-19T08:04:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.839+0000 7f0790c9f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0ba5d0 0x56275e0c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.839+0000 7f0790c9f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0bbb40 0x56275e0c1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.839+0000 7f0790c9f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0c5370 0x56275e0c2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.839+0000 7f0790c9f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56275e0c7fb0 con 0x56275e0bbb40 2024-09-19T08:04:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.839+0000 7f0790c9f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56275dfdb060 con 0x56275e0ba5d0 2024-09-19T08:04:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.839+0000 7f0790c9f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56275e0c8130 con 0x56275e0c5370 2024-09-19T08:04:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.839+0000 7f0790c9f740 10 client.0 _ll_register_callbacks cb 0x56275dd2e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.839+0000 7f07877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0bbb40 0x56275e0c1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.839+0000 7f07877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0bbb40 0x56275e0c1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33724/0 (socket says 172.21.15.98:33724) 2024-09-19T08:04:34.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.839+0000 7f07877fe640 1 -- 172.21.15.98:0/249092569 learned_addr learned my addr 172.21.15.98:0/249092569 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:34.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.839+0000 7f0787fff640 1 --2- 172.21.15.98:0/249092569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0ba5d0 0x56275e0c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.839+0000 7f078ce63640 1 --2- 172.21.15.98:0/249092569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0c5370 0x56275e0c2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f07877fe640 1 -- 172.21.15.98:0/249092569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0c5370 msgr2=0x56275e0c2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:34.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f07877fe640 1 --2- 172.21.15.98:0/249092569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0c5370 0x56275e0c2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f07877fe640 1 -- 172.21.15.98:0/249092569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0ba5d0 msgr2=0x56275e0c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:34.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f07877fe640 1 --2- 172.21.15.98:0/249092569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0ba5d0 0x56275e0c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f07877fe640 1 -- 172.21.15.98:0/249092569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56275dd913c0 con 0x56275e0bbb40 2024-09-19T08:04:34.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f07877fe640 1 --2- 172.21.15.98:0/249092569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0bbb40 0x56275e0c1fd0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f077002a930 tx=0x7f0770031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a9f634de1e44d91 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:34.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f07677fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:34.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f07677fe640 1 -- 172.21.15.98:0/249092569 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0770035040 con 0x56275e0bbb40 2024-09-19T08:04:34.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f07677fe640 1 -- 172.21.15.98:0/249092569 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f0770004380 con 0x56275e0bbb40 2024-09-19T08:04:34.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f0790c9f740 1 -- 172.21.15.98:0/249092569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56275e02d860 con 0x56275e0bbb40 2024-09-19T08:04:34.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f07677fe640 1 -- 172.21.15.98:0/249092569 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0770005020 con 0x56275e0bbb40 2024-09-19T08:04:34.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f0790c9f740 10 client.15870 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:34.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f0790c9f740 1 -- 172.21.15.98:0/249092569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56275e0d3a00 con 0x56275e0bbb40 2024-09-19T08:04:34.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f07677fe640 1 -- 172.21.15.98:0/249092569 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56275e02d860 con 0x56275e0bbb40 2024-09-19T08:04:34.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.840+0000 7f07677fe640 1 -- 172.21.15.98:0/249092569 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f07700035f0 con 0x56275e0bbb40 2024-09-19T08:04:34.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.841+0000 7f0790c9f740 10 client.15870 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:34.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.841+0000 7f0790c9f740 10 client.15870 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:04:34.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.841+0000 7f0790c9f740 20 client.15870 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:34.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.841+0000 7f0790c9f740 1 --2- 172.21.15.98:0/249092569 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x56275e0d4ec0 0x56275e0d7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.841+0000 7f0790c9f740 4 client.15870 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:04:34.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.841+0000 7f0790c9f740 1 -- 172.21.15.98:0/249092569 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56275e027d70 con 0x56275e0d4ec0 2024-09-19T08:04:34.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.841+0000 7f0787fff640 1 --2- 172.21.15.98:0/249092569 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x56275e0d4ec0 0x56275e0d7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.841+0000 7f0787fff640 1 --2- 172.21.15.98:0/249092569 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x56275e0d4ec0 0x56275e0d7320 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=260e2eedc21dc37c server_cookie=ec18284a1b9d47b3 in_seq=0 out_seq=0 2024-09-19T08:04:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.841+0000 7f07677fe640 10 client.15870 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:04:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f07677fe640 1 -- 172.21.15.98:0/249092569 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56275e027d70 con 0x56275e0d4ec0 2024-09-19T08:04:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f07677fe640 10 client.15870 handle_command_reply: tid=0 2024-09-19T08:04:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f0790c9f740 1 client.15870 shutdown 2024-09-19T08:04:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f0790c9f740 1 -- 172.21.15.98:0/249092569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0bbb40 msgr2=0x56275e0c1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f0790c9f740 1 --2- 172.21.15.98:0/249092569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0bbb40 0x56275e0c1fd0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f077002a930 tx=0x7f0770031a30 comp rx=0 tx=0).stop 2024-09-19T08:04:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f0790c9f740 1 -- 172.21.15.98:0/249092569 shutdown_connections 2024-09-19T08:04:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f0790c9f740 1 --2- 172.21.15.98:0/249092569 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x56275e0d4ec0 0x56275e0d7320 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f0790c9f740 1 --2- 172.21.15.98:0/249092569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0c5370 0x56275e0c2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f07677fe640 0 client.15870 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:04:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f0790c9f740 1 --2- 172.21.15.98:0/249092569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0bbb40 0x56275e0c1fd0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f0790c9f740 1 --2- 172.21.15.98:0/249092569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0ba5d0 0x56275e0c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f0790c9f740 1 -- 172.21.15.98:0/249092569 >> 172.21.15.98:0/249092569 conn(0x56275e083310 msgr2=0x56275e0bc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f0790c9f740 1 -- 172.21.15.98:0/249092569 shutdown_connections 2024-09-19T08:04:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f0790c9f740 1 -- 172.21.15.98:0/249092569 wait complete. 2024-09-19T08:04:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.842+0000 7f0790c9f740 20 client.15870 trim_cache size 0 max 16384 2024-09-19T08:04:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 Processor -- start 2024-09-19T08:04:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 -- start start 2024-09-19T08:04:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0ba5d0 0x56275e0bd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0bbb40 0x56275e0bd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0c5370 0x56275e0dcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f07881ad8d0 con 0x56275e0ba5d0 2024-09-19T08:04:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f07881af410 con 0x56275e0c5370 2024-09-19T08:04:34.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f07881adfa0 con 0x56275e0bbb40 2024-09-19T08:04:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0787fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0ba5d0 0x56275e0bd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0787fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0ba5d0 0x56275e0bd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33726/0 (socket says 172.21.15.98:33726) 2024-09-19T08:04:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0787fff640 1 -- 172.21.15.98:0/2887126206 learned_addr learned my addr 172.21.15.98:0/2887126206 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f07877fe640 1 --2- 172.21.15.98:0/2887126206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0bbb40 0x56275e0bd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0787fff640 1 -- 172.21.15.98:0/2887126206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0bbb40 msgr2=0x56275e0bd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0787fff640 1 --2- 172.21.15.98:0/2887126206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0bbb40 0x56275e0bd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0787fff640 1 -- 172.21.15.98:0/2887126206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0c5370 msgr2=0x56275e0dcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0787fff640 1 --2- 172.21.15.98:0/2887126206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0c5370 0x56275e0dcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0787fff640 1 -- 172.21.15.98:0/2887126206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56275e0c4940 con 0x56275e0ba5d0 2024-09-19T08:04:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0787fff640 1 --2- 172.21.15.98:0/2887126206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0ba5d0 0x56275e0bd120 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f077809b780 tx=0x7f07780d5530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f7e749df05cdbb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f077dffb640 1 -- 172.21.15.98:0/2887126206 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778020040 con 0x56275e0ba5d0 2024-09-19T08:04:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f077dffb640 1 -- 172.21.15.98:0/2887126206 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f077806edb0 con 0x56275e0ba5d0 2024-09-19T08:04:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f077dffb640 1 -- 172.21.15.98:0/2887126206 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778005290 con 0x56275e0ba5d0 2024-09-19T08:04:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 -- 172.21.15.98:0/2887126206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0ba5d0 msgr2=0x56275e0bd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 --2- 172.21.15.98:0/2887126206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0ba5d0 0x56275e0bd120 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f077809b780 tx=0x7f07780d5530 comp rx=0 tx=0).stop 2024-09-19T08:04:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 -- 172.21.15.98:0/2887126206 shutdown_connections 2024-09-19T08:04:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 --2- 172.21.15.98:0/2887126206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0c5370 0x56275e0dcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 --2- 172.21.15.98:0/2887126206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0bbb40 0x56275e0bd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 --2- 172.21.15.98:0/2887126206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0ba5d0 0x56275e0bd120 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.856+0000 7f0790c9f740 1 -- 172.21.15.98:0/2887126206 >> 172.21.15.98:0/2887126206 conn(0x56275e083310 msgr2=0x56275e0c4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0790c9f740 1 -- 172.21.15.98:0/2887126206 shutdown_connections 2024-09-19T08:04:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0790c9f740 1 -- 172.21.15.98:0/2887126206 wait complete. 2024-09-19T08:04:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0790c9f740 1 -- start start 2024-09-19T08:04:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0790c9f740 1 Processor -- start 2024-09-19T08:04:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0790c9f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0ba5d0 0x56275e0cd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0790c9f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0bbb40 0x56275e0c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0790c9f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0c5370 0x56275e0ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0790c9f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f07881aed40 con 0x56275e0c5370 2024-09-19T08:04:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0790c9f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f07881ae670 con 0x56275e0bbb40 2024-09-19T08:04:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0790c9f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f07881ab8b0 con 0x56275e0ba5d0 2024-09-19T08:04:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0790c9f740 10 client.0 _ll_register_callbacks cb 0x7f0770009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f078ce63640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0c5370 0x56275e0ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f078ce63640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0c5370 0x56275e0ca0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33730/0 (socket says 172.21.15.98:33730) 2024-09-19T08:04:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f078ce63640 1 -- 172.21.15.98:0/3845488940 learned_addr learned my addr 172.21.15.98:0/3845488940 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f07877fe640 1 --2- 172.21.15.98:0/3845488940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0bbb40 0x56275e0c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0787fff640 1 --2- 172.21.15.98:0/3845488940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0ba5d0 0x56275e0cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f078ce63640 1 -- 172.21.15.98:0/3845488940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0ba5d0 msgr2=0x56275e0cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:34.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f078ce63640 1 --2- 172.21.15.98:0/3845488940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0ba5d0 0x56275e0cd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f078ce63640 1 -- 172.21.15.98:0/3845488940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0bbb40 msgr2=0x56275e0c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:34.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f078ce63640 1 --2- 172.21.15.98:0/3845488940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0bbb40 0x56275e0c9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f078ce63640 1 -- 172.21.15.98:0/3845488940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56275e0ca620 con 0x56275e0c5370 2024-09-19T08:04:34.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f078ce63640 1 --2- 172.21.15.98:0/3845488940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0c5370 0x56275e0ca0e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f078001fe60 tx=0x7f078000a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f3ef4c33fa6f9c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:34.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f07677fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:34.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f07677fe640 1 -- 172.21.15.98:0/3845488940 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078001de20 con 0x56275e0c5370 2024-09-19T08:04:34.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0790c9f740 1 -- 172.21.15.98:0/3845488940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f077806edb0 con 0x56275e0c5370 2024-09-19T08:04:34.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f07677fe640 1 -- 172.21.15.98:0/3845488940 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f0780028440 con 0x56275e0c5370 2024-09-19T08:04:34.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f07677fe640 1 -- 172.21.15.98:0/3845488940 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0780028640 con 0x56275e0c5370 2024-09-19T08:04:34.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0790c9f740 10 client.15876 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:34.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f0790c9f740 1 -- 172.21.15.98:0/3845488940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56275e140980 con 0x56275e0c5370 2024-09-19T08:04:34.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.858+0000 7f07677fe640 1 -- 172.21.15.98:0/3845488940 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f077806edb0 con 0x56275e0c5370 2024-09-19T08:04:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.859+0000 7f07677fe640 1 -- 172.21.15.98:0/3845488940 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0780026020 con 0x56275e0c5370 2024-09-19T08:04:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.859+0000 7f0790c9f740 10 client.15876 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.859+0000 7f0790c9f740 10 client.15876 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:04:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.859+0000 7f0790c9f740 20 client.15876 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.859+0000 7f0790c9f740 1 --2- 172.21.15.98:0/3845488940 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x56275e141100 0x56275e143520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.859+0000 7f0790c9f740 4 client.15876 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:04:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.859+0000 7f0790c9f740 1 -- 172.21.15.98:0/3845488940 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56275dfa4170 con 0x56275e141100 2024-09-19T08:04:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.859+0000 7f0787fff640 1 --2- 172.21.15.98:0/3845488940 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x56275e141100 0x56275e143520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.859+0000 7f07677fe640 10 client.15876 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:04:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.859+0000 7f0787fff640 1 --2- 172.21.15.98:0/3845488940 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x56275e141100 0x56275e143520 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=aea93157912aca24 server_cookie=2ec8fcf95f8d7fe9 in_seq=0 out_seq=0 2024-09-19T08:04:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.859+0000 7f07677fe640 1 -- 172.21.15.98:0/3845488940 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56275dfa4170 con 0x56275e141100 2024-09-19T08:04:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.859+0000 7f07677fe640 10 client.15876 handle_command_reply: tid=0 2024-09-19T08:04:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.859+0000 7f0790c9f740 1 client.15876 shutdown 2024-09-19T08:04:34.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f0790c9f740 1 -- 172.21.15.98:0/3845488940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0c5370 msgr2=0x56275e0ca0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f0790c9f740 1 --2- 172.21.15.98:0/3845488940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0c5370 0x56275e0ca0e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f078001fe60 tx=0x7f078000a4d0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f0790c9f740 1 -- 172.21.15.98:0/3845488940 shutdown_connections 2024-09-19T08:04:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f0790c9f740 1 --2- 172.21.15.98:0/3845488940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56275e0c5370 0x56275e0ca0e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f0790c9f740 1 --2- 172.21.15.98:0/3845488940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56275e0bbb40 0x56275e0c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f0790c9f740 1 --2- 172.21.15.98:0/3845488940 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x56275e141100 0x56275e143520 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f0790c9f740 1 --2- 172.21.15.98:0/3845488940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56275e0ba5d0 0x56275e0cd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f0790c9f740 1 -- 172.21.15.98:0/3845488940 >> 172.21.15.98:0/3845488940 conn(0x56275e083310 msgr2=0x56275e0b54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f07677fe640 0 client.15876 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:04:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f0790c9f740 1 -- 172.21.15.98:0/3845488940 shutdown_connections 2024-09-19T08:04:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f0790c9f740 1 -- 172.21.15.98:0/3845488940 wait complete. 2024-09-19T08:04:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f0790c9f740 20 client.15876 trim_cache size 0 max 16384 2024-09-19T08:04:34.884 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:04:34.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f078e8c6640 1 -- 172.21.15.98:0/745800824 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f07740a33b0 msgr2=0x7f07740a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:34.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f078e8c6640 1 --2- 172.21.15.98:0/745800824 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f07740a33b0 0x7f07740a5870 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f078000a2c0 tx=0x7f0780004a40 comp rx=0 tx=0).stop 2024-09-19T08:04:34.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f078e8c6640 1 -- 172.21.15.98:0/745800824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0788101de0 msgr2=0x7f0788198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:34.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f078e8c6640 1 --2- 172.21.15.98:0/745800824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0788101de0 0x7f0788198470 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f077800d770 tx=0x7f077800dc30 comp rx=0 tx=0).stop 2024-09-19T08:04:34.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f078e8c6640 1 -- 172.21.15.98:0/745800824 shutdown_connections 2024-09-19T08:04:34.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f078e8c6640 1 --2- 172.21.15.98:0/745800824 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f07740a33b0 0x7f07740a5870 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f078e8c6640 1 --2- 172.21.15.98:0/745800824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0788106f10 0x7f078819cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f078e8c6640 1 --2- 172.21.15.98:0/745800824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0788102790 0x7f07881989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f078e8c6640 1 --2- 172.21.15.98:0/745800824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0788101de0 0x7f0788198470 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:34.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f078e8c6640 1 -- 172.21.15.98:0/745800824 >> 172.21.15.98:0/745800824 conn(0x7f07880fd8c0 msgr2=0x7f07880fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:34.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f078e8c6640 1 -- 172.21.15.98:0/745800824 shutdown_connections 2024-09-19T08:04:34.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:34.860+0000 7f078e8c6640 1 -- 172.21.15.98:0/745800824 wait complete. 2024-09-19T08:04:35.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:35 smithi139 ceph-mon[34874]: pgmap v50: 97 pgs: 97 active+clean; 5.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 344 KiB/s wr, 10 op/s 2024-09-19T08:04:35.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:35 smithi098 ceph-mon[31333]: pgmap v50: 97 pgs: 97 active+clean; 5.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 344 KiB/s wr, 10 op/s 2024-09-19T08:04:35.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:35 smithi162 ceph-mon[35260]: pgmap v50: 97 pgs: 97 active+clean; 5.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 344 KiB/s wr, 10 op/s 2024-09-19T08:04:35.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:04:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:36.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.364+0000 7ff73fe22640 1 -- 172.21.15.98:0/2842713884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738074620 msgr2=0x7ff738074a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:36.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.364+0000 7ff73fe22640 1 --2- 172.21.15.98:0/2842713884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738074620 0x7ff738074a00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff730009ca0 tx=0x7ff73002a950 comp rx=0 tx=0).stop 2024-09-19T08:04:36.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.364+0000 7ff73fe22640 1 -- 172.21.15.98:0/2842713884 shutdown_connections 2024-09-19T08:04:36.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.364+0000 7ff73fe22640 1 --2- 172.21.15.98:0/2842713884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff73810e3c0 0x7ff738110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.364+0000 7ff73fe22640 1 --2- 172.21.15.98:0/2842713884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff738074f40 0x7ff73810de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.364+0000 7ff73fe22640 1 --2- 172.21.15.98:0/2842713884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738074620 0x7ff738074a00 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.364+0000 7ff73fe22640 1 -- 172.21.15.98:0/2842713884 >> 172.21.15.98:0/2842713884 conn(0x7ff7380fd8c0 msgr2=0x7ff7380ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:36.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.364+0000 7ff73fe22640 1 -- 172.21.15.98:0/2842713884 shutdown_connections 2024-09-19T08:04:36.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.364+0000 7ff73fe22640 1 -- 172.21.15.98:0/2842713884 wait complete. 2024-09-19T08:04:36.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73fe22640 1 Processor -- start 2024-09-19T08:04:36.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73fe22640 1 -- start start 2024-09-19T08:04:36.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73fe22640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738074620 0x7ff73819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73fe22640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff738074f40 0x7ff73819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73fe22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff73810e3c0 0x7ff73819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73fe22640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7381141a0 con 0x7ff738074620 2024-09-19T08:04:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73fe22640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff738114020 con 0x7ff738074f40 2024-09-19T08:04:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73fe22640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff738114320 con 0x7ff73810e3c0 2024-09-19T08:04:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73dbbe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738074620 0x7ff73819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73dbbe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738074620 0x7ff73819a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50332/0 (socket says 172.21.15.98:50332) 2024-09-19T08:04:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73dbbe640 1 -- 172.21.15.98:0/4115557430 learned_addr learned my addr 172.21.15.98:0/4115557430 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73dbbe640 1 -- 172.21.15.98:0/4115557430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff73810e3c0 msgr2=0x7ff73819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73dbbe640 1 --2- 172.21.15.98:0/4115557430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff73810e3c0 0x7ff73819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73dbbe640 1 -- 172.21.15.98:0/4115557430 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff738074f40 msgr2=0x7ff73819abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.365+0000 7ff73dbbe640 1 --2- 172.21.15.98:0/4115557430 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff738074f40 0x7ff73819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.366+0000 7ff73dbbe640 1 -- 172.21.15.98:0/4115557430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73819f630 con 0x7ff738074620 2024-09-19T08:04:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.366+0000 7ff73dbbe640 1 --2- 172.21.15.98:0/4115557430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738074620 0x7ff73819a680 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff7300098d0 tx=0x7ff730003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.366+0000 7ff72affd640 1 -- 172.21.15.98:0/4115557430 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff730034040 con 0x7ff738074620 2024-09-19T08:04:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.366+0000 7ff72affd640 1 -- 172.21.15.98:0/4115557430 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff73002fe20 con 0x7ff738074620 2024-09-19T08:04:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.366+0000 7ff73fe22640 1 -- 172.21.15.98:0/4115557430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff73819f8c0 con 0x7ff738074620 2024-09-19T08:04:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.366+0000 7ff72affd640 1 -- 172.21.15.98:0/4115557430 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff730004810 con 0x7ff738074620 2024-09-19T08:04:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.366+0000 7ff73fe22640 1 -- 172.21.15.98:0/4115557430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7381a7200 con 0x7ff738074620 2024-09-19T08:04:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.368+0000 7ff72affd640 1 -- 172.21.15.98:0/4115557430 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7ff730037020 con 0x7ff738074620 2024-09-19T08:04:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.368+0000 7ff7421fb740 1 Processor -- start 2024-09-19T08:04:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff7421fb740 1 -- start start 2024-09-19T08:04:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff7421fb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9bc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff7421fb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9bc5d0 0x556bda9bca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff7421fb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9c7370 0x556bda9c9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff7421fb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556bda9222f0 con 0x556bda9c7370 2024-09-19T08:04:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff7421fb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556bda8c9cf0 con 0x556bda9bdb40 2024-09-19T08:04:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff7421fb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556bda91e2b0 con 0x556bda9bc5d0 2024-09-19T08:04:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff73d3bd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9c7370 0x556bda9c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff73d3bd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9c7370 0x556bda9c9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50334/0 (socket says 172.21.15.98:50334) 2024-09-19T08:04:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff73d3bd640 1 -- 172.21.15.98:0/3668196472 learned_addr learned my addr 172.21.15.98:0/3668196472 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff73dbbe640 1 --2- 172.21.15.98:0/3668196472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9bc5d0 0x556bda9bca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff73e3bf640 1 --2- 172.21.15.98:0/3668196472 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9bc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff72affd640 1 --2- 172.21.15.98:0/4115557430 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff7240a3430 0x7ff7240a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff73d3bd640 1 -- 172.21.15.98:0/3668196472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9bc5d0 msgr2=0x556bda9bca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff73d3bd640 1 --2- 172.21.15.98:0/3668196472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9bc5d0 0x556bda9bca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff73d3bd640 1 -- 172.21.15.98:0/3668196472 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 msgr2=0x556bda9bc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff73d3bd640 1 --2- 172.21.15.98:0/3668196472 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9bc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff73d3bd640 1 -- 172.21.15.98:0/3668196472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bda8ae3a0 con 0x556bda9c7370 2024-09-19T08:04:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff73e3bf640 1 --2- 172.21.15.98:0/3668196472 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9bc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff73e3bf640 1 --2- 172.21.15.98:0/4115557430 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff7240a3430 0x7ff7240a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff73d3bd640 1 --2- 172.21.15.98:0/3668196472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9c7370 0x556bda9c9760 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff73400aa40 tx=0x7ff73400af00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1812ec0ea44443af server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.369+0000 7ff72affd640 1 -- 172.21.15.98:0/4115557430 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff7300f0d80 con 0x7ff738074620 2024-09-19T08:04:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.370+0000 7ff723fff640 1 -- 172.21.15.98:0/3668196472 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73400ce40 con 0x556bda9c7370 2024-09-19T08:04:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.370+0000 7ff723fff640 1 -- 172.21.15.98:0/3668196472 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7340035f0 con 0x556bda9c7370 2024-09-19T08:04:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.370+0000 7ff723fff640 1 -- 172.21.15.98:0/3668196472 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73400f670 con 0x556bda9c7370 2024-09-19T08:04:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.370+0000 7ff7421fb740 1 -- 172.21.15.98:0/3668196472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9c7370 msgr2=0x556bda9c9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.370+0000 7ff73e3bf640 1 --2- 172.21.15.98:0/4115557430 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff7240a3430 0x7ff7240a58f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff72c00aaa0 tx=0x7ff72c0046c0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.370+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3668196472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9c7370 0x556bda9c9760 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff73400aa40 tx=0x7ff73400af00 comp rx=0 tx=0).stop 2024-09-19T08:04:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.370+0000 7ff7421fb740 1 -- 172.21.15.98:0/3668196472 shutdown_connections 2024-09-19T08:04:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.370+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3668196472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9c7370 0x556bda9c9760 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.370+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3668196472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9bc5d0 0x556bda9bca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.370+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3668196472 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9bc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.370+0000 7ff7421fb740 1 -- 172.21.15.98:0/3668196472 >> 172.21.15.98:0/3668196472 conn(0x556bda985310 msgr2=0x556bda9b8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.371+0000 7ff7421fb740 1 -- 172.21.15.98:0/3668196472 shutdown_connections 2024-09-19T08:04:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.371+0000 7ff7421fb740 1 -- 172.21.15.98:0/3668196472 wait complete. 2024-09-19T08:04:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.371+0000 7ff7421fb740 1 -- start start 2024-09-19T08:04:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.371+0000 7ff7421fb740 1 Processor -- start 2024-09-19T08:04:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.371+0000 7ff7421fb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.371+0000 7ff7421fb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9bdb40 0x556bda9c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.371+0000 7ff7421fb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9c7370 0x556bda9c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.371+0000 7ff7421fb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556bda9ca070 con 0x556bda9bc5d0 2024-09-19T08:04:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.371+0000 7ff7421fb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556bda8dd060 con 0x556bda9c7370 2024-09-19T08:04:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.371+0000 7ff7421fb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556bda9ca1f0 con 0x556bda9bdb40 2024-09-19T08:04:36.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.371+0000 7ff7421fb740 10 client.0 _ll_register_callbacks cb 0x556bda6308c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:36.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.371+0000 7ff73dbbe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.371+0000 7ff73dbbe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9c6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50338/0 (socket says 172.21.15.98:50338) 2024-09-19T08:04:36.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.371+0000 7ff73dbbe640 1 -- 172.21.15.98:0/3889107769 learned_addr learned my addr 172.21.15.98:0/3889107769 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:36.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff73e3bf640 1 --2- 172.21.15.98:0/3889107769 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9c7370 0x556bda9c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff73d3bd640 1 --2- 172.21.15.98:0/3889107769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9bdb40 0x556bda9c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff73dbbe640 1 -- 172.21.15.98:0/3889107769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9bdb40 msgr2=0x556bda9c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:36.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff73dbbe640 1 --2- 172.21.15.98:0/3889107769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9bdb40 0x556bda9c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff73dbbe640 1 -- 172.21.15.98:0/3889107769 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9c7370 msgr2=0x556bda9c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:36.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff73dbbe640 1 --2- 172.21.15.98:0/3889107769 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9c7370 0x556bda9c44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff73dbbe640 1 -- 172.21.15.98:0/3889107769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bda6933c0 con 0x556bda9bc5d0 2024-09-19T08:04:36.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff73dbbe640 1 --2- 172.21.15.98:0/3889107769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9c6cd0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff730006010 tx=0x7ff7300b9500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a74717f3bbd6a25 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:36.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff7217fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff7217fa640 1 -- 172.21.15.98:0/3889107769 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7300bcb00 con 0x556bda9bc5d0 2024-09-19T08:04:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff7217fa640 1 -- 172.21.15.98:0/3889107769 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff7300ee780 con 0x556bda9bc5d0 2024-09-19T08:04:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff7421fb740 1 -- 172.21.15.98:0/3889107769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7340035f0 con 0x556bda9bc5d0 2024-09-19T08:04:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff7217fa640 1 -- 172.21.15.98:0/3889107769 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff7300bd8a0 con 0x556bda9bc5d0 2024-09-19T08:04:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff7421fb740 10 client.15888 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff7421fb740 1 -- 172.21.15.98:0/3889107769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556bda9ca070 con 0x556bda9bc5d0 2024-09-19T08:04:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff7217fa640 1 -- 172.21.15.98:0/3889107769 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7340035f0 con 0x556bda9bc5d0 2024-09-19T08:04:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.372+0000 7ff7217fa640 1 -- 172.21.15.98:0/3889107769 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff7300b9810 con 0x556bda9bc5d0 2024-09-19T08:04:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.373+0000 7ff7421fb740 10 client.15888 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.373+0000 7ff7421fb740 10 client.15888 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:04:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.373+0000 7ff7421fb740 20 client.15888 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.373+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3889107769 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x556bda9d6f30 0x556bda9d93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.373+0000 7ff7421fb740 4 client.15888 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:04:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.373+0000 7ff7421fb740 1 -- 172.21.15.98:0/3889107769 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bda92f860 con 0x556bda9d6f30 2024-09-19T08:04:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.373+0000 7ff73d3bd640 1 --2- 172.21.15.98:0/3889107769 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x556bda9d6f30 0x556bda9d93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.373+0000 7ff73d3bd640 1 --2- 172.21.15.98:0/3889107769 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x556bda9d6f30 0x556bda9d93c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3fe9bfbd16b02a23 server_cookie=f389d67dee9e465e in_seq=0 out_seq=0 2024-09-19T08:04:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.373+0000 7ff7217fa640 10 client.15888 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:04:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.374+0000 7ff7217fa640 1 -- 172.21.15.98:0/3889107769 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556bda92f860 con 0x556bda9d6f30 2024-09-19T08:04:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.374+0000 7ff7217fa640 10 client.15888 handle_command_reply: tid=0 2024-09-19T08:04:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.374+0000 7ff7421fb740 1 client.15888 shutdown 2024-09-19T08:04:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.374+0000 7ff7421fb740 1 -- 172.21.15.98:0/3889107769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 msgr2=0x556bda9c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.374+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3889107769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9c6cd0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff730006010 tx=0x7ff7300b9500 comp rx=0 tx=0).stop 2024-09-19T08:04:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.374+0000 7ff7421fb740 1 -- 172.21.15.98:0/3889107769 shutdown_connections 2024-09-19T08:04:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.374+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3889107769 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x556bda9d6f30 0x556bda9d93c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.374+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3889107769 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9c7370 0x556bda9c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.374+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3889107769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9bdb40 0x556bda9c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.374+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3889107769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9c6cd0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.374+0000 7ff7217fa640 0 client.15888 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:04:36.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.374+0000 7ff7421fb740 1 -- 172.21.15.98:0/3889107769 >> 172.21.15.98:0/3889107769 conn(0x556bda985310 msgr2=0x556bda9be960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:36.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.375+0000 7ff7421fb740 1 -- 172.21.15.98:0/3889107769 shutdown_connections 2024-09-19T08:04:36.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.375+0000 7ff7421fb740 1 -- 172.21.15.98:0/3889107769 wait complete. 2024-09-19T08:04:36.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.375+0000 7ff7421fb740 20 client.15888 trim_cache size 0 max 16384 2024-09-19T08:04:36.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.387+0000 7ff7421fb740 1 Processor -- start 2024-09-19T08:04:36.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.387+0000 7ff7421fb740 1 -- start start 2024-09-19T08:04:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff7421fb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff7421fb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9c8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff7421fb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9c7370 0x556bda9c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff7421fb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7381af9d0 con 0x556bda9bc5d0 2024-09-19T08:04:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff7421fb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff7381b1510 con 0x556bda9bdb40 2024-09-19T08:04:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff7421fb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff7381b00a0 con 0x556bda9c7370 2024-09-19T08:04:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff73dbbe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff73dbbe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9b7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50342/0 (socket says 172.21.15.98:50342) 2024-09-19T08:04:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff73dbbe640 1 -- 172.21.15.98:0/2722960627 learned_addr learned my addr 172.21.15.98:0/2722960627 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff73e3bf640 1 --2- 172.21.15.98:0/2722960627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9c7370 0x556bda9c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff73d3bd640 1 --2- 172.21.15.98:0/2722960627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff73dbbe640 1 -- 172.21.15.98:0/2722960627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9c7370 msgr2=0x556bda9c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff73dbbe640 1 --2- 172.21.15.98:0/2722960627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9c7370 0x556bda9c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff73dbbe640 1 -- 172.21.15.98:0/2722960627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 msgr2=0x556bda9c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff73dbbe640 1 --2- 172.21.15.98:0/2722960627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff73dbbe640 1 -- 172.21.15.98:0/2722960627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bda9b9960 con 0x556bda9bc5d0 2024-09-19T08:04:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff73d3bd640 1 --2- 172.21.15.98:0/2722960627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff73dbbe640 1 --2- 172.21.15.98:0/2722960627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9b7bf0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff730000f80 tx=0x7ff7300f1110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0fbafa073a2a8a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff7237fe640 1 -- 172.21.15.98:0/2722960627 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7300bcb00 con 0x556bda9bc5d0 2024-09-19T08:04:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff7237fe640 1 -- 172.21.15.98:0/2722960627 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7300c1550 con 0x556bda9bc5d0 2024-09-19T08:04:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.388+0000 7ff7237fe640 1 -- 172.21.15.98:0/2722960627 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73002c040 con 0x556bda9bc5d0 2024-09-19T08:04:36.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.389+0000 7ff7421fb740 1 -- 172.21.15.98:0/2722960627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 msgr2=0x556bda9b7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:36.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.389+0000 7ff7421fb740 1 --2- 172.21.15.98:0/2722960627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9b7bf0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff730000f80 tx=0x7ff7300f1110 comp rx=0 tx=0).stop 2024-09-19T08:04:36.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.389+0000 7ff7421fb740 1 -- 172.21.15.98:0/2722960627 shutdown_connections 2024-09-19T08:04:36.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.389+0000 7ff7421fb740 1 --2- 172.21.15.98:0/2722960627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9c7370 0x556bda9c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.389+0000 7ff7421fb740 1 --2- 172.21.15.98:0/2722960627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.389+0000 7ff7421fb740 1 --2- 172.21.15.98:0/2722960627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9b7bf0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.389+0000 7ff7421fb740 1 -- 172.21.15.98:0/2722960627 >> 172.21.15.98:0/2722960627 conn(0x556bda985310 msgr2=0x556bda9b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:36.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.389+0000 7ff7421fb740 1 -- 172.21.15.98:0/2722960627 shutdown_connections 2024-09-19T08:04:36.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.389+0000 7ff7421fb740 1 -- 172.21.15.98:0/2722960627 wait complete. 2024-09-19T08:04:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.389+0000 7ff7421fb740 1 -- start start 2024-09-19T08:04:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff7421fb740 1 Processor -- start 2024-09-19T08:04:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff7421fb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9dea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff7421fb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9defc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff7421fb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9c7370 0x556bda9df500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff7421fb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7381b0e40 con 0x556bda9bc5d0 2024-09-19T08:04:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff7421fb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff7381b0770 con 0x556bda9bdb40 2024-09-19T08:04:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff7421fb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff7381ad9b0 con 0x556bda9c7370 2024-09-19T08:04:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff7421fb740 10 client.0 _ll_register_callbacks cb 0x7ff730009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff73dbbe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9dea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff73dbbe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9dea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50354/0 (socket says 172.21.15.98:50354) 2024-09-19T08:04:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff73dbbe640 1 -- 172.21.15.98:0/3386723775 learned_addr learned my addr 172.21.15.98:0/3386723775 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff73e3bf640 1 --2- 172.21.15.98:0/3386723775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9c7370 0x556bda9df500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff73d3bd640 1 --2- 172.21.15.98:0/3386723775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9defc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff73dbbe640 1 -- 172.21.15.98:0/3386723775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9c7370 msgr2=0x556bda9df500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff73dbbe640 1 --2- 172.21.15.98:0/3386723775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9c7370 0x556bda9df500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff73dbbe640 1 -- 172.21.15.98:0/3386723775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 msgr2=0x556bda9defc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:36.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff73dbbe640 1 --2- 172.21.15.98:0/3386723775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9defc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff73dbbe640 1 -- 172.21.15.98:0/3386723775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bda9cde20 con 0x556bda9bc5d0 2024-09-19T08:04:36.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.390+0000 7ff73d3bd640 1 --2- 172.21.15.98:0/3386723775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9defc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:36.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff73dbbe640 1 --2- 172.21.15.98:0/3386723775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9dea80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff730004320 tx=0x7ff7300bd4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3157d34634aa26 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:36.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7217fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:36.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7217fa640 1 -- 172.21.15.98:0/3386723775 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7300bd1a0 con 0x556bda9bc5d0 2024-09-19T08:04:36.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7217fa640 1 -- 172.21.15.98:0/3386723775 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff7300f44a0 con 0x556bda9bc5d0 2024-09-19T08:04:36.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7421fb740 1 -- 172.21.15.98:0/3386723775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7300c1550 con 0x556bda9bc5d0 2024-09-19T08:04:36.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7217fa640 1 -- 172.21.15.98:0/3386723775 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff7300dd070 con 0x556bda9bc5d0 2024-09-19T08:04:36.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7421fb740 10 client.15894 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:36.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7421fb740 1 -- 172.21.15.98:0/3386723775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556bda9cde20 con 0x556bda9bc5d0 2024-09-19T08:04:36.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7217fa640 1 -- 172.21.15.98:0/3386723775 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7300c1550 con 0x556bda9bc5d0 2024-09-19T08:04:36.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7217fa640 1 -- 172.21.15.98:0/3386723775 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff7300c5980 con 0x556bda9bc5d0 2024-09-19T08:04:36.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7421fb740 10 client.15894 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:36.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7421fb740 10 client.15894 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:04:36.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7421fb740 20 client.15894 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:36.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3386723775 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x556bdaa42370 0x556bdaa44760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:36.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7421fb740 4 client.15894 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:04:36.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.391+0000 7ff7421fb740 1 -- 172.21.15.98:0/3386723775 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556bda929d70 con 0x556bdaa42370 2024-09-19T08:04:36.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff73d3bd640 1 --2- 172.21.15.98:0/3386723775 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x556bdaa42370 0x556bdaa44760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:36.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff73d3bd640 1 --2- 172.21.15.98:0/3386723775 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x556bdaa42370 0x556bdaa44760 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ab700cc2e4ee3669 server_cookie=ebfc8842f0bd7091 in_seq=0 out_seq=0 2024-09-19T08:04:36.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7217fa640 10 client.15894 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:04:36.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7217fa640 1 -- 172.21.15.98:0/3386723775 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556bda929d70 con 0x556bdaa42370 2024-09-19T08:04:36.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7217fa640 10 client.15894 handle_command_reply: tid=0 2024-09-19T08:04:36.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7421fb740 1 client.15894 shutdown 2024-09-19T08:04:36.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7421fb740 1 -- 172.21.15.98:0/3386723775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 msgr2=0x556bda9dea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:36.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3386723775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9dea80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff730004320 tx=0x7ff7300bd4c0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7421fb740 1 -- 172.21.15.98:0/3386723775 shutdown_connections 2024-09-19T08:04:36.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3386723775 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x556bdaa42370 0x556bdaa44760 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3386723775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556bda9c7370 0x556bda9df500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7217fa640 0 client.15894 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:04:36.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3386723775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556bda9bdb40 0x556bda9defc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7421fb740 1 --2- 172.21.15.98:0/3386723775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556bda9bc5d0 0x556bda9dea80 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7421fb740 1 -- 172.21.15.98:0/3386723775 >> 172.21.15.98:0/3386723775 conn(0x556bda985310 msgr2=0x556bda9b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:36.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7421fb740 1 -- 172.21.15.98:0/3386723775 shutdown_connections 2024-09-19T08:04:36.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.392+0000 7ff7421fb740 1 -- 172.21.15.98:0/3386723775 wait complete. 2024-09-19T08:04:36.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.393+0000 7ff7421fb740 20 client.15894 trim_cache size 0 max 16384 2024-09-19T08:04:36.400 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:04:36.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.393+0000 7ff73fe22640 1 -- 172.21.15.98:0/4115557430 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff7240a3430 msgr2=0x7ff7240a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:36.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.393+0000 7ff73fe22640 1 --2- 172.21.15.98:0/4115557430 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff7240a3430 0x7ff7240a58f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff72c00aaa0 tx=0x7ff72c0046c0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.393+0000 7ff73fe22640 1 -- 172.21.15.98:0/4115557430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738074620 msgr2=0x7ff73819a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:36.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.393+0000 7ff73fe22640 1 --2- 172.21.15.98:0/4115557430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738074620 0x7ff73819a680 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff7300098d0 tx=0x7ff730003f10 comp rx=0 tx=0).stop 2024-09-19T08:04:36.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.393+0000 7ff73fe22640 1 -- 172.21.15.98:0/4115557430 shutdown_connections 2024-09-19T08:04:36.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.393+0000 7ff73fe22640 1 --2- 172.21.15.98:0/4115557430 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff7240a3430 0x7ff7240a58f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.393+0000 7ff73fe22640 1 --2- 172.21.15.98:0/4115557430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff73810e3c0 0x7ff73819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.393+0000 7ff73fe22640 1 --2- 172.21.15.98:0/4115557430 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff738074f40 0x7ff73819abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.393+0000 7ff73fe22640 1 --2- 172.21.15.98:0/4115557430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738074620 0x7ff73819a680 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:36.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.393+0000 7ff73fe22640 1 -- 172.21.15.98:0/4115557430 >> 172.21.15.98:0/4115557430 conn(0x7ff7380fd8c0 msgr2=0x7ff7380fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:36.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.393+0000 7ff73fe22640 1 -- 172.21.15.98:0/4115557430 shutdown_connections 2024-09-19T08:04:36.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:36.393+0000 7ff73fe22640 1 -- 172.21.15.98:0/4115557430 wait complete. 2024-09-19T08:04:36.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:04:37.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:36 smithi162 ceph-mon[35260]: pgmap v51: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 345 KiB/s wr, 15 op/s 2024-09-19T08:04:37.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:36 smithi139 ceph-mon[34874]: pgmap v51: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 345 KiB/s wr, 15 op/s 2024-09-19T08:04:37.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:36 smithi098 ceph-mon[31333]: pgmap v51: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 345 KiB/s wr, 15 op/s 2024-09-19T08:04:37.203 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.698+0000 7f3b366e1640 1 -- 172.21.15.98:0/3032207028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b30101de0 msgr2=0x7f3b301021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.698+0000 7f3b366e1640 1 --2- 172.21.15.98:0/3032207028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b30101de0 0x7f3b301021c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3b20009900 tx=0x7f3b2002a410 comp rx=0 tx=0).stop 2024-09-19T08:04:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.698+0000 7f3b366e1640 1 -- 172.21.15.98:0/3032207028 shutdown_connections 2024-09-19T08:04:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.698+0000 7f3b366e1640 1 --2- 172.21.15.98:0/3032207028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b30106f10 0x7f3b3010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.698+0000 7f3b366e1640 1 --2- 172.21.15.98:0/3032207028 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3b30102790 0x7f3b301067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.698+0000 7f3b366e1640 1 --2- 172.21.15.98:0/3032207028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b30101de0 0x7f3b301021c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.698+0000 7f3b366e1640 1 -- 172.21.15.98:0/3032207028 >> 172.21.15.98:0/3032207028 conn(0x7f3b300fd8c0 msgr2=0x7f3b300ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.698+0000 7f3b366e1640 1 -- 172.21.15.98:0/3032207028 shutdown_connections 2024-09-19T08:04:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.698+0000 7f3b366e1640 1 -- 172.21.15.98:0/3032207028 wait complete. 2024-09-19T08:04:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b366e1640 1 Processor -- start 2024-09-19T08:04:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b366e1640 1 -- start start 2024-09-19T08:04:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b366e1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b30101de0 0x7f3b301983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b366e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b30102790 0x7f3b30198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b366e1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3b30106f10 0x7f3b3019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b366e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b30111f70 con 0x7f3b30102790 2024-09-19T08:04:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b366e1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3b30111df0 con 0x7f3b30106f10 2024-09-19T08:04:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b366e1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301120f0 con 0x7f3b30101de0 2024-09-19T08:04:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b2f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b30102790 0x7f3b30198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b2f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b30102790 0x7f3b30198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50370/0 (socket says 172.21.15.98:50370) 2024-09-19T08:04:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b2f7fe640 1 -- 172.21.15.98:0/1522317727 learned_addr learned my addr 172.21.15.98:0/1522317727 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b2ffff640 1 --2- 172.21.15.98:0/1522317727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b30101de0 0x7f3b301983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b34c7e640 1 --2- 172.21.15.98:0/1522317727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3b30106f10 0x7f3b3019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b2f7fe640 1 -- 172.21.15.98:0/1522317727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b30101de0 msgr2=0x7f3b301983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b2f7fe640 1 --2- 172.21.15.98:0/1522317727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b30101de0 0x7f3b301983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b2f7fe640 1 -- 172.21.15.98:0/1522317727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3b30106f10 msgr2=0x7f3b3019cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b2f7fe640 1 --2- 172.21.15.98:0/1522317727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3b30106f10 0x7f3b3019cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.699+0000 7f3b2f7fe640 1 -- 172.21.15.98:0/1522317727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3019d380 con 0x7f3b30102790 2024-09-19T08:04:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.700+0000 7f3b2f7fe640 1 --2- 172.21.15.98:0/1522317727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b30102790 0x7f3b30198910 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400d950 tx=0x7f3b2400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.700+0000 7f3b2d7fa640 1 -- 172.21.15.98:0/1522317727 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b24017070 con 0x7f3b30102790 2024-09-19T08:04:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.700+0000 7f3b2d7fa640 1 -- 172.21.15.98:0/1522317727 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b24009c30 con 0x7f3b30102790 2024-09-19T08:04:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.700+0000 7f3b366e1640 1 -- 172.21.15.98:0/1522317727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b3019d670 con 0x7f3b30102790 2024-09-19T08:04:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.700+0000 7f3b2d7fa640 1 -- 172.21.15.98:0/1522317727 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2400f6b0 con 0x7f3b30102790 2024-09-19T08:04:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.700+0000 7f3b366e1640 1 -- 172.21.15.98:0/1522317727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b301a4f50 con 0x7f3b30102790 2024-09-19T08:04:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.702+0000 7f3b2d7fa640 1 -- 172.21.15.98:0/1522317727 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f3b240109d0 con 0x7f3b30102790 2024-09-19T08:04:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b38aba740 1 Processor -- start 2024-09-19T08:04:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b38aba740 1 -- start start 2024-09-19T08:04:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b2d7fa640 1 --2- 172.21.15.98:0/1522317727 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3b180a33b0 0x7f3b180a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b38aba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b67b40 0x5613b2b66090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b38aba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b66a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b38aba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 0x5613b2b77b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b38aba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5613b2acc2f0 con 0x5613b2b665d0 2024-09-19T08:04:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b34c7e640 1 --2- 172.21.15.98:0/1522317727 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3b180a33b0 0x7f3b180a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b38aba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5613b2a73cf0 con 0x5613b2b75780 2024-09-19T08:04:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b38aba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5613b2ac82b0 con 0x5613b2b67b40 2024-09-19T08:04:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b2d7fa640 1 -- 172.21.15.98:0/1522317727 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f3b240cf8d0 con 0x7f3b30102790 2024-09-19T08:04:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b2f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b66a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b2f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b66a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50382/0 (socket says 172.21.15.98:50382) 2024-09-19T08:04:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b2f7fe640 1 -- 172.21.15.98:0/667012046 learned_addr learned my addr 172.21.15.98:0/667012046 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b34c7e640 1 --2- 172.21.15.98:0/667012046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 0x5613b2b77b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b2f7fe640 1 -- 172.21.15.98:0/667012046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b67b40 msgr2=0x5613b2b66090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T08:04:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b2f7fe640 1 --2- 172.21.15.98:0/667012046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b67b40 0x5613b2b66090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b2f7fe640 1 -- 172.21.15.98:0/667012046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 msgr2=0x5613b2b77b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.703+0000 7f3b2f7fe640 1 --2- 172.21.15.98:0/667012046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 0x5613b2b77b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.704+0000 7f3b2f7fe640 1 -- 172.21.15.98:0/667012046 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613b2a583a0 con 0x5613b2b665d0 2024-09-19T08:04:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.704+0000 7f3b34c7e640 1 --2- 172.21.15.98:0/667012046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 0x5613b2b77b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.704+0000 7f3b34c7e640 1 --2- 172.21.15.98:0/1522317727 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3b180a33b0 0x7f3b180a5870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3b28007de0 tx=0x7f3b2800f790 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.704+0000 7f3b2f7fe640 1 --2- 172.21.15.98:0/667012046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b66a10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3b240cd650 tx=0x7f3b24010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ac3b465c20fc65 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.704+0000 7f3b167fc640 1 -- 172.21.15.98:0/667012046 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b24020070 con 0x5613b2b665d0 2024-09-19T08:04:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.704+0000 7f3b167fc640 1 -- 172.21.15.98:0/667012046 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b240a4850 con 0x5613b2b665d0 2024-09-19T08:04:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.704+0000 7f3b167fc640 1 -- 172.21.15.98:0/667012046 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b24060390 con 0x5613b2b665d0 2024-09-19T08:04:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.705+0000 7f3b38aba740 1 -- 172.21.15.98:0/667012046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 msgr2=0x5613b2b66a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.705+0000 7f3b38aba740 1 --2- 172.21.15.98:0/667012046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b66a10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3b240cd650 tx=0x7f3b24010d70 comp rx=0 tx=0).stop 2024-09-19T08:04:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.705+0000 7f3b38aba740 1 -- 172.21.15.98:0/667012046 shutdown_connections 2024-09-19T08:04:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.705+0000 7f3b38aba740 1 --2- 172.21.15.98:0/667012046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 0x5613b2b77b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.705+0000 7f3b38aba740 1 --2- 172.21.15.98:0/667012046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b66a10 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.705+0000 7f3b38aba740 1 --2- 172.21.15.98:0/667012046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b67b40 0x5613b2b66090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.705+0000 7f3b38aba740 1 -- 172.21.15.98:0/667012046 >> 172.21.15.98:0/667012046 conn(0x5613b2b2f310 msgr2=0x5613b2b62c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.705+0000 7f3b38aba740 1 -- 172.21.15.98:0/667012046 shutdown_connections 2024-09-19T08:04:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.705+0000 7f3b38aba740 1 -- 172.21.15.98:0/667012046 wait complete. 2024-09-19T08:04:37.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.705+0000 7f3b38aba740 1 -- start start 2024-09-19T08:04:37.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.705+0000 7f3b38aba740 1 Processor -- start 2024-09-19T08:04:37.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b38aba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b75180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b38aba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b67b40 0x5613b2b72430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b38aba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 0x5613b2b72a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b38aba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5613b2b783c0 con 0x5613b2b665d0 2024-09-19T08:04:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b38aba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5613b2a87060 con 0x5613b2b75780 2024-09-19T08:04:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b38aba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5613b2b78540 con 0x5613b2b67b40 2024-09-19T08:04:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b38aba740 10 client.0 _ll_register_callbacks cb 0x5613b27da8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b2ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b75180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b2ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b75180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50390/0 (socket says 172.21.15.98:50390) 2024-09-19T08:04:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b2ffff640 1 -- 172.21.15.98:0/1308499186 learned_addr learned my addr 172.21.15.98:0/1308499186 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b34c7e640 1 --2- 172.21.15.98:0/1308499186 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 0x5613b2b72a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b2f7fe640 1 --2- 172.21.15.98:0/1308499186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b67b40 0x5613b2b72430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b2ffff640 1 -- 172.21.15.98:0/1308499186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b67b40 msgr2=0x5613b2b72430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b2ffff640 1 --2- 172.21.15.98:0/1308499186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b67b40 0x5613b2b72430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b2ffff640 1 -- 172.21.15.98:0/1308499186 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 msgr2=0x5613b2b72a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b2ffff640 1 --2- 172.21.15.98:0/1308499186 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 0x5613b2b72a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b2ffff640 1 -- 172.21.15.98:0/1308499186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613b283d3c0 con 0x5613b2b665d0 2024-09-19T08:04:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b2ffff640 1 --2- 172.21.15.98:0/1308499186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b75180 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3b20009900 tx=0x7f3b20031920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9715bdf2cc7ea7c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b0f7fe640 1 -- 172.21.15.98:0/1308499186 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b20040070 con 0x5613b2b665d0 2024-09-19T08:04:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b38aba740 1 -- 172.21.15.98:0/1308499186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b240a4850 con 0x5613b2b665d0 2024-09-19T08:04:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.706+0000 7f3b0f7fe640 1 -- 172.21.15.98:0/1308499186 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f3b20004450 con 0x5613b2b665d0 2024-09-19T08:04:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.707+0000 7f3b0f7fe640 1 -- 172.21.15.98:0/1308499186 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3b20035040 con 0x5613b2b665d0 2024-09-19T08:04:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.707+0000 7f3b38aba740 10 client.15906 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.707+0000 7f3b38aba740 1 -- 172.21.15.98:0/1308499186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5613b2b83dd0 con 0x5613b2b665d0 2024-09-19T08:04:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.707+0000 7f3b0f7fe640 1 -- 172.21.15.98:0/1308499186 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b240a4850 con 0x5613b2b665d0 2024-09-19T08:04:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.707+0000 7f3b0f7fe640 1 -- 172.21.15.98:0/1308499186 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3b20004cd0 con 0x5613b2b665d0 2024-09-19T08:04:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.707+0000 7f3b38aba740 10 client.15906 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.707+0000 7f3b38aba740 10 client.15906 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:04:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.707+0000 7f3b38aba740 20 client.15906 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.707+0000 7f3b38aba740 1 --2- 172.21.15.98:0/1308499186 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5613b2b85290 0x5613b2b87680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.707+0000 7f3b38aba740 4 client.15906 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:04:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.707+0000 7f3b38aba740 1 -- 172.21.15.98:0/1308499186 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613b2ad9860 con 0x5613b2b85290 2024-09-19T08:04:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.707+0000 7f3b2f7fe640 1 --2- 172.21.15.98:0/1308499186 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5613b2b85290 0x5613b2b87680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.708+0000 7f3b0f7fe640 10 client.15906 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:04:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.708+0000 7f3b2f7fe640 1 --2- 172.21.15.98:0/1308499186 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5613b2b85290 0x5613b2b87680 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f89a3ece4abda455 server_cookie=d71c15018ce2091 in_seq=0 out_seq=0 2024-09-19T08:04:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.708+0000 7f3b0f7fe640 1 -- 172.21.15.98:0/1308499186 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f3b24058e80 con 0x5613b2b85290 2024-09-19T08:04:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.708+0000 7f3b0f7fe640 10 client.15906 handle_command_reply: tid=0 2024-09-19T08:04:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.709+0000 7f3b38aba740 1 client.15906 shutdown 2024-09-19T08:04:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.709+0000 7f3b38aba740 1 -- 172.21.15.98:0/1308499186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 msgr2=0x5613b2b75180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:37.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.709+0000 7f3b38aba740 1 --2- 172.21.15.98:0/1308499186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b75180 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3b20009900 tx=0x7f3b20031920 comp rx=0 tx=0).stop 2024-09-19T08:04:37.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.709+0000 7f3b38aba740 1 -- 172.21.15.98:0/1308499186 shutdown_connections 2024-09-19T08:04:37.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.709+0000 7f3b38aba740 1 --2- 172.21.15.98:0/1308499186 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5613b2b85290 0x5613b2b87680 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.709+0000 7f3b0f7fe640 0 client.15906 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:04:37.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.709+0000 7f3b38aba740 1 --2- 172.21.15.98:0/1308499186 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 0x5613b2b72a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.709+0000 7f3b38aba740 1 --2- 172.21.15.98:0/1308499186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b67b40 0x5613b2b72430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.709+0000 7f3b38aba740 1 --2- 172.21.15.98:0/1308499186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b75180 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.709+0000 7f3b38aba740 1 -- 172.21.15.98:0/1308499186 >> 172.21.15.98:0/1308499186 conn(0x5613b2b2f310 msgr2=0x5613b2b68990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:37.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.709+0000 7f3b38aba740 1 -- 172.21.15.98:0/1308499186 shutdown_connections 2024-09-19T08:04:37.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.709+0000 7f3b38aba740 1 -- 172.21.15.98:0/1308499186 wait complete. 2024-09-19T08:04:37.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.709+0000 7f3b38aba740 20 client.15906 trim_cache size 0 max 16384 2024-09-19T08:04:37.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b38aba740 1 Processor -- start 2024-09-19T08:04:37.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b38aba740 1 -- start start 2024-09-19T08:04:37.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b38aba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b81660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b38aba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b67b40 0x5613b2b71cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b38aba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b75780 0x5613b2b61bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b38aba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301ab660 con 0x5613b2b665d0 2024-09-19T08:04:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b38aba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301ad1a0 con 0x5613b2b67b40 2024-09-19T08:04:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b38aba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301abd30 con 0x5613b2b75780 2024-09-19T08:04:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b2ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b81660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b2ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b81660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50394/0 (socket says 172.21.15.98:50394) 2024-09-19T08:04:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b2ffff640 1 -- 172.21.15.98:0/539967799 learned_addr learned my addr 172.21.15.98:0/539967799 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b2f7fe640 1 --2- 172.21.15.98:0/539967799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b67b40 0x5613b2b71cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b34c7e640 1 --2- 172.21.15.98:0/539967799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b75780 0x5613b2b61bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b2ffff640 1 -- 172.21.15.98:0/539967799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b75780 msgr2=0x5613b2b61bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b2ffff640 1 --2- 172.21.15.98:0/539967799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b75780 0x5613b2b61bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b2ffff640 1 -- 172.21.15.98:0/539967799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b67b40 msgr2=0x5613b2b71cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:37.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b2ffff640 1 --2- 172.21.15.98:0/539967799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b67b40 0x5613b2b71cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.722+0000 7f3b2ffff640 1 -- 172.21.15.98:0/539967799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613b2b713e0 con 0x5613b2b665d0 2024-09-19T08:04:37.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.723+0000 7f3b2f7fe640 1 --2- 172.21.15.98:0/539967799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b67b40 0x5613b2b71cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:37.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.723+0000 7f3b34c7e640 1 --2- 172.21.15.98:0/539967799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b75780 0x5613b2b61bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:37.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.723+0000 7f3b2ffff640 1 --2- 172.21.15.98:0/539967799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b81660 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f3b20003690 tx=0x7f3b20001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79c2a5d3cd5852e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:37.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.723+0000 7f3b15ffb640 1 -- 172.21.15.98:0/539967799 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b20051070 con 0x5613b2b665d0 2024-09-19T08:04:37.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.723+0000 7f3b15ffb640 1 -- 172.21.15.98:0/539967799 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b20003a80 con 0x5613b2b665d0 2024-09-19T08:04:37.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.723+0000 7f3b15ffb640 1 -- 172.21.15.98:0/539967799 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b20040030 con 0x5613b2b665d0 2024-09-19T08:04:37.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.723+0000 7f3b38aba740 1 -- 172.21.15.98:0/539967799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 msgr2=0x5613b2b81660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.723+0000 7f3b38aba740 1 --2- 172.21.15.98:0/539967799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b81660 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f3b20003690 tx=0x7f3b20001e70 comp rx=0 tx=0).stop 2024-09-19T08:04:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 -- 172.21.15.98:0/539967799 shutdown_connections 2024-09-19T08:04:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 --2- 172.21.15.98:0/539967799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b75780 0x5613b2b61bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 --2- 172.21.15.98:0/539967799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b67b40 0x5613b2b71cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 --2- 172.21.15.98:0/539967799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b665d0 0x5613b2b81660 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 -- 172.21.15.98:0/539967799 >> 172.21.15.98:0/539967799 conn(0x5613b2b2f310 msgr2=0x5613b2b614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 -- 172.21.15.98:0/539967799 shutdown_connections 2024-09-19T08:04:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 -- 172.21.15.98:0/539967799 wait complete. 2024-09-19T08:04:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 -- start start 2024-09-19T08:04:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 Processor -- start 2024-09-19T08:04:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b665d0 0x5613b2b7d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b67b40 0x5613b2b7dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 0x5613b2b7a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301acad0 con 0x5613b2b67b40 2024-09-19T08:04:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301ac400 con 0x5613b2b75780 2024-09-19T08:04:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301a9640 con 0x5613b2b665d0 2024-09-19T08:04:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b38aba740 10 client.0 _ll_register_callbacks cb 0x7f3b20009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.724+0000 7f3b2f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b67b40 0x5613b2b7dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b2f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b67b40 0x5613b2b7dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50396/0 (socket says 172.21.15.98:50396) 2024-09-19T08:04:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b2f7fe640 1 -- 172.21.15.98:0/2110402233 learned_addr learned my addr 172.21.15.98:0/2110402233 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b2ffff640 1 --2- 172.21.15.98:0/2110402233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b665d0 0x5613b2b7d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b34c7e640 1 --2- 172.21.15.98:0/2110402233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 0x5613b2b7a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b2f7fe640 1 -- 172.21.15.98:0/2110402233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b665d0 msgr2=0x5613b2b7d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b2f7fe640 1 --2- 172.21.15.98:0/2110402233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b665d0 0x5613b2b7d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b2f7fe640 1 -- 172.21.15.98:0/2110402233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 msgr2=0x5613b2b7a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:37.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b2f7fe640 1 --2- 172.21.15.98:0/2110402233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 0x5613b2b7a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b2f7fe640 1 -- 172.21.15.98:0/2110402233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613b2b7ade0 con 0x5613b2b67b40 2024-09-19T08:04:37.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b34c7e640 1 --2- 172.21.15.98:0/2110402233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 0x5613b2b7a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:37.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b2f7fe640 1 --2- 172.21.15.98:0/2110402233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b67b40 0x5613b2b7dd10 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f3b2409bc20 tx=0x7f3b2409bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b168ab7f58069486 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:37.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:37.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b0f7fe640 1 -- 172.21.15.98:0/2110402233 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2405f070 con 0x5613b2b67b40 2024-09-19T08:04:37.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b0f7fe640 1 -- 172.21.15.98:0/2110402233 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f3b240602f0 con 0x5613b2b67b40 2024-09-19T08:04:37.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b38aba740 1 -- 172.21.15.98:0/2110402233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b20003a80 con 0x5613b2b67b40 2024-09-19T08:04:37.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b0f7fe640 1 -- 172.21.15.98:0/2110402233 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3b24020040 con 0x5613b2b67b40 2024-09-19T08:04:37.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b38aba740 10 client.15912 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:37.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.725+0000 7f3b38aba740 1 -- 172.21.15.98:0/2110402233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5613b2bf07a0 con 0x5613b2b67b40 2024-09-19T08:04:37.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.726+0000 7f3b0f7fe640 1 -- 172.21.15.98:0/2110402233 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b20003a80 con 0x5613b2b67b40 2024-09-19T08:04:37.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.726+0000 7f3b0f7fe640 1 -- 172.21.15.98:0/2110402233 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3b240860b0 con 0x5613b2b67b40 2024-09-19T08:04:37.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.726+0000 7f3b38aba740 10 client.15912 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:37.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.726+0000 7f3b38aba740 10 client.15912 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:04:37.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.726+0000 7f3b38aba740 20 client.15912 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:37.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.726+0000 7f3b38aba740 1 --2- 172.21.15.98:0/2110402233 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5613b2b85290 0x5613b2bf2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:37.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.726+0000 7f3b38aba740 4 client.15912 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:04:37.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.726+0000 7f3b38aba740 1 -- 172.21.15.98:0/2110402233 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5613b2ad3d70 con 0x5613b2b85290 2024-09-19T08:04:37.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.726+0000 7f3b2ffff640 1 --2- 172.21.15.98:0/2110402233 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5613b2b85290 0x5613b2bf2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:37.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b2ffff640 1 --2- 172.21.15.98:0/2110402233 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5613b2b85290 0x5613b2bf2b40 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=619db30fea385d5a server_cookie=7e030eff7d90dddb in_seq=0 out_seq=0 2024-09-19T08:04:37.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b0f7fe640 10 client.15912 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:04:37.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b0f7fe640 1 -- 172.21.15.98:0/2110402233 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5613b2ad3d70 con 0x5613b2b85290 2024-09-19T08:04:37.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b0f7fe640 10 client.15912 handle_command_reply: tid=0 2024-09-19T08:04:37.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b38aba740 1 client.15912 shutdown 2024-09-19T08:04:37.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b38aba740 1 -- 172.21.15.98:0/2110402233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b67b40 msgr2=0x5613b2b7dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:37.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b38aba740 1 --2- 172.21.15.98:0/2110402233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b67b40 0x5613b2b7dd10 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f3b2409bc20 tx=0x7f3b2409bc90 comp rx=0 tx=0).stop 2024-09-19T08:04:37.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b38aba740 1 -- 172.21.15.98:0/2110402233 shutdown_connections 2024-09-19T08:04:37.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b38aba740 1 --2- 172.21.15.98:0/2110402233 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5613b2b85290 0x5613b2bf2b40 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b38aba740 1 --2- 172.21.15.98:0/2110402233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5613b2b75780 0x5613b2b7a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b0f7fe640 0 client.15912 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:04:37.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b38aba740 1 --2- 172.21.15.98:0/2110402233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613b2b67b40 0x5613b2b7dd10 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b38aba740 1 --2- 172.21.15.98:0/2110402233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b2b665d0 0x5613b2b7d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b38aba740 1 -- 172.21.15.98:0/2110402233 >> 172.21.15.98:0/2110402233 conn(0x5613b2b2f310 msgr2=0x5613b2b614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:37.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b38aba740 1 -- 172.21.15.98:0/2110402233 shutdown_connections 2024-09-19T08:04:37.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b38aba740 1 -- 172.21.15.98:0/2110402233 wait complete. 2024-09-19T08:04:37.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.727+0000 7f3b38aba740 20 client.15912 trim_cache size 0 max 16384 2024-09-19T08:04:37.735 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:04:37.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.728+0000 7f3b366e1640 1 -- 172.21.15.98:0/1522317727 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3b180a33b0 msgr2=0x7f3b180a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:37.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.728+0000 7f3b366e1640 1 --2- 172.21.15.98:0/1522317727 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3b180a33b0 0x7f3b180a5870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3b28007de0 tx=0x7f3b2800f790 comp rx=0 tx=0).stop 2024-09-19T08:04:37.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.728+0000 7f3b366e1640 1 -- 172.21.15.98:0/1522317727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b30102790 msgr2=0x7f3b30198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.728+0000 7f3b366e1640 1 --2- 172.21.15.98:0/1522317727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b30102790 0x7f3b30198910 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400d950 tx=0x7f3b2400de10 comp rx=0 tx=0).stop 2024-09-19T08:04:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.728+0000 7f3b366e1640 1 -- 172.21.15.98:0/1522317727 shutdown_connections 2024-09-19T08:04:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.728+0000 7f3b366e1640 1 --2- 172.21.15.98:0/1522317727 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3b180a33b0 0x7f3b180a5870 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.728+0000 7f3b366e1640 1 --2- 172.21.15.98:0/1522317727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3b30106f10 0x7f3b3019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.728+0000 7f3b366e1640 1 --2- 172.21.15.98:0/1522317727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b30102790 0x7f3b30198910 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.728+0000 7f3b366e1640 1 --2- 172.21.15.98:0/1522317727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b30101de0 0x7f3b301983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.728+0000 7f3b366e1640 1 -- 172.21.15.98:0/1522317727 >> 172.21.15.98:0/1522317727 conn(0x7f3b300fd8c0 msgr2=0x7f3b300febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.728+0000 7f3b366e1640 1 -- 172.21.15.98:0/1522317727 shutdown_connections 2024-09-19T08:04:37.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:37.728+0000 7f3b366e1640 1 -- 172.21.15.98:0/1522317727 wait complete. 2024-09-19T08:04:38.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:04:38.603 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:39.017 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:38 smithi098 ceph-mon[31333]: pgmap v52: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 343 KiB/s wr, 10 op/s 2024-09-19T08:04:39.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.099+0000 7f6399738640 1 -- 172.21.15.98:0/2341678333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394101df0 msgr2=0x7f63941021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:39.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.099+0000 7f6399738640 1 --2- 172.21.15.98:0/2341678333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394101df0 0x7f63941021d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6384009ca0 tx=0x7f638402a8c0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.099+0000 7f6399738640 1 -- 172.21.15.98:0/2341678333 shutdown_connections 2024-09-19T08:04:39.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.099+0000 7f6399738640 1 --2- 172.21.15.98:0/2341678333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6394106f20 0x7f639410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.099+0000 7f6399738640 1 --2- 172.21.15.98:0/2341678333 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63941027a0 0x7f63941067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.099+0000 7f6399738640 1 --2- 172.21.15.98:0/2341678333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394101df0 0x7f63941021d0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.099+0000 7f6399738640 1 -- 172.21.15.98:0/2341678333 >> 172.21.15.98:0/2341678333 conn(0x7f63940fd8b0 msgr2=0x7f63940ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:39.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.099+0000 7f6399738640 1 -- 172.21.15.98:0/2341678333 shutdown_connections 2024-09-19T08:04:39.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.099+0000 7f6399738640 1 -- 172.21.15.98:0/2341678333 wait complete. 2024-09-19T08:04:39.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.099+0000 7f6399738640 1 Processor -- start 2024-09-19T08:04:39.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.099+0000 7f6399738640 1 -- start start 2024-09-19T08:04:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6399738640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394101df0 0x7f6394198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6399738640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63941027a0 0x7f6394198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6399738640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6394106f20 0x7f639419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6399738640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6394111f80 con 0x7f6394101df0 2024-09-19T08:04:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6399738640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6394111e00 con 0x7f6394106f20 2024-09-19T08:04:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6399738640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6394112100 con 0x7f63941027a0 2024-09-19T08:04:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6392ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394101df0 0x7f6394198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6392ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394101df0 0x7f6394198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50406/0 (socket says 172.21.15.98:50406) 2024-09-19T08:04:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6392ffd640 1 -- 172.21.15.98:0/2183512729 learned_addr learned my addr 172.21.15.98:0/2183512729 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f63927fc640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63941027a0 0x7f6394198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f63937fe640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6394106f20 0x7f639419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6392ffd640 1 -- 172.21.15.98:0/2183512729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63941027a0 msgr2=0x7f6394198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6392ffd640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63941027a0 0x7f6394198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6392ffd640 1 -- 172.21.15.98:0/2183512729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6394106f20 msgr2=0x7f639419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6392ffd640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6394106f20 0x7f639419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6392ffd640 1 -- 172.21.15.98:0/2183512729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639419d3f0 con 0x7f6394101df0 2024-09-19T08:04:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f63937fe640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6394106f20 0x7f639419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6392ffd640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394101df0 0x7f6394198440 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f63840097b0 tx=0x7f6384031b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f637bfff640 1 -- 172.21.15.98:0/2183512729 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638403fc20 con 0x7f6394101df0 2024-09-19T08:04:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6399738640 1 -- 172.21.15.98:0/2183512729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f639419d680 con 0x7f6394101df0 2024-09-19T08:04:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f637bfff640 1 -- 172.21.15.98:0/2183512729 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f638403fdc0 con 0x7f6394101df0 2024-09-19T08:04:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f637bfff640 1 -- 172.21.15.98:0/2183512729 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384030de0 con 0x7f6394101df0 2024-09-19T08:04:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.100+0000 7f6399738640 1 -- 172.21.15.98:0/2183512729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6394074420 con 0x7f6394101df0 2024-09-19T08:04:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.102+0000 7f637bfff640 1 -- 172.21.15.98:0/2183512729 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f638402f790 con 0x7f6394101df0 2024-09-19T08:04:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f639bb11740 1 Processor -- start 2024-09-19T08:04:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f639bb11740 1 -- start start 2024-09-19T08:04:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f637bfff640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f637c0a33b0 0x7f637c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f639bb11740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f2eb40 0x5647e6f2d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f639bb11740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f2da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f639bb11740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f38370 0x5647e6f3a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f639bb11740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5647e6e932f0 con 0x5647e6f2d5d0 2024-09-19T08:04:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f639bb11740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5647e6e3acf0 con 0x5647e6f38370 2024-09-19T08:04:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f639bb11740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5647e6e8f2b0 con 0x5647e6f2eb40 2024-09-19T08:04:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f63937fe640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f637c0a33b0 0x7f637c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f637bfff640 1 -- 172.21.15.98:0/2183512729 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f63840f57f0 con 0x7f6394101df0 2024-09-19T08:04:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f6392ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f2eb40 0x5647e6f2d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f6392ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f2eb40 0x5647e6f2d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:47806/0 (socket says 172.21.15.98:47806) 2024-09-19T08:04:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f6392ffd640 1 -- 172.21.15.98:0/4241764182 learned_addr learned my addr 172.21.15.98:0/4241764182 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.103+0000 7f63927fc640 1 --2- 172.21.15.98:0/4241764182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f2da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.104+0000 7f63937fe640 1 --2- 172.21.15.98:0/4241764182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f38370 0x5647e6f3a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.104+0000 7f63927fc640 1 -- 172.21.15.98:0/4241764182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f2eb40 msgr2=0x5647e6f2d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.104+0000 7f63927fc640 1 --2- 172.21.15.98:0/4241764182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f2eb40 0x5647e6f2d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.104+0000 7f63927fc640 1 -- 172.21.15.98:0/4241764182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f38370 msgr2=0x5647e6f3a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.104+0000 7f63927fc640 1 --2- 172.21.15.98:0/4241764182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f38370 0x5647e6f3a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.104+0000 7f63927fc640 1 -- 172.21.15.98:0/4241764182 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647e6e1f3a0 con 0x5647e6f2d5d0 2024-09-19T08:04:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.104+0000 7f6392ffd640 1 --2- 172.21.15.98:0/4241764182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f2eb40 0x5647e6f2d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:04:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.104+0000 7f63937fe640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f637c0a33b0 0x7f637c0a5870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f638c00bb30 tx=0x7f638c00bf30 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.104+0000 7f63927fc640 1 --2- 172.21.15.98:0/4241764182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f2da10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f638800caf0 tx=0x7f638800e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e41994eb869d4aaf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.104+0000 7f63797fa640 1 -- 172.21.15.98:0/4241764182 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388010430 con 0x5647e6f2d5d0 2024-09-19T08:04:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.104+0000 7f63797fa640 1 -- 172.21.15.98:0/4241764182 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6388010ab0 con 0x5647e6f2d5d0 2024-09-19T08:04:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.104+0000 7f63797fa640 1 -- 172.21.15.98:0/4241764182 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638800fb90 con 0x5647e6f2d5d0 2024-09-19T08:04:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.105+0000 7f639bb11740 1 -- 172.21.15.98:0/4241764182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 msgr2=0x5647e6f2da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.105+0000 7f639bb11740 1 --2- 172.21.15.98:0/4241764182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f2da10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f638800caf0 tx=0x7f638800e350 comp rx=0 tx=0).stop 2024-09-19T08:04:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.105+0000 7f639bb11740 1 -- 172.21.15.98:0/4241764182 shutdown_connections 2024-09-19T08:04:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.105+0000 7f639bb11740 1 --2- 172.21.15.98:0/4241764182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f38370 0x5647e6f3a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.105+0000 7f639bb11740 1 --2- 172.21.15.98:0/4241764182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f2da10 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.105+0000 7f639bb11740 1 --2- 172.21.15.98:0/4241764182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f2eb40 0x5647e6f2d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.105+0000 7f639bb11740 1 -- 172.21.15.98:0/4241764182 >> 172.21.15.98:0/4241764182 conn(0x5647e6ef6310 msgr2=0x5647e6f29c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.105+0000 7f639bb11740 1 -- 172.21.15.98:0/4241764182 shutdown_connections 2024-09-19T08:04:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.105+0000 7f639bb11740 1 -- 172.21.15.98:0/4241764182 wait complete. 2024-09-19T08:04:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.105+0000 7f639bb11740 1 -- start start 2024-09-19T08:04:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.105+0000 7f639bb11740 1 Processor -- start 2024-09-19T08:04:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f639bb11740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f37d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f639bb11740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 0x5647e6f35030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f639bb11740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f38370 0x5647e6f35640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f639bb11740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5647e6f3b070 con 0x5647e6f2d5d0 2024-09-19T08:04:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f639bb11740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5647e6e4e060 con 0x5647e6f2eb40 2024-09-19T08:04:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f639bb11740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5647e6f3b1f0 con 0x5647e6f38370 2024-09-19T08:04:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f639bb11740 10 client.0 _ll_register_callbacks cb 0x5647e6ba18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f6392ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f37d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f6392ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f37d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50422/0 (socket says 172.21.15.98:50422) 2024-09-19T08:04:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f6392ffd640 1 -- 172.21.15.98:0/1017378910 learned_addr learned my addr 172.21.15.98:0/1017378910 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f63927fc640 1 --2- 172.21.15.98:0/1017378910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 0x5647e6f35030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f63937fe640 1 --2- 172.21.15.98:0/1017378910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f38370 0x5647e6f35640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f6392ffd640 1 -- 172.21.15.98:0/1017378910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f38370 msgr2=0x5647e6f35640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f6392ffd640 1 --2- 172.21.15.98:0/1017378910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f38370 0x5647e6f35640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f6392ffd640 1 -- 172.21.15.98:0/1017378910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 msgr2=0x5647e6f35030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f6392ffd640 1 --2- 172.21.15.98:0/1017378910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 0x5647e6f35030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f6392ffd640 1 -- 172.21.15.98:0/1017378910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647e6c043c0 con 0x5647e6f2d5d0 2024-09-19T08:04:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f63937fe640 1 --2- 172.21.15.98:0/1017378910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f38370 0x5647e6f35640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f6392ffd640 1 --2- 172.21.15.98:0/1017378910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f37d80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f63840c1910 tx=0x7f63840d49b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c33c4606c744ea21 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f6376ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f6376ff5640 1 -- 172.21.15.98:0/1017378910 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63840d8e60 con 0x5647e6f2d5d0 2024-09-19T08:04:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f6376ff5640 1 -- 172.21.15.98:0/1017378910 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f63840305c0 con 0x5647e6f2d5d0 2024-09-19T08:04:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.106+0000 7f639bb11740 1 -- 172.21.15.98:0/1017378910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6388010ab0 con 0x5647e6f2d5d0 2024-09-19T08:04:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.107+0000 7f6376ff5640 1 -- 172.21.15.98:0/1017378910 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f63840d4520 con 0x5647e6f2d5d0 2024-09-19T08:04:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.107+0000 7f639bb11740 10 client.15924 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.107+0000 7f639bb11740 1 -- 172.21.15.98:0/1017378910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5647e6f3b070 con 0x5647e6f2d5d0 2024-09-19T08:04:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.107+0000 7f6376ff5640 1 -- 172.21.15.98:0/1017378910 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6388010ab0 con 0x5647e6f2d5d0 2024-09-19T08:04:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.107+0000 7f6376ff5640 1 -- 172.21.15.98:0/1017378910 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f63840f31f0 con 0x5647e6f2d5d0 2024-09-19T08:04:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.107+0000 7f639bb11740 10 client.15924 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.107+0000 7f639bb11740 10 client.15924 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:04:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.107+0000 7f639bb11740 20 client.15924 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.107+0000 7f639bb11740 1 --2- 172.21.15.98:0/1017378910 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5647e6f47fb0 0x5647e6f4a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.107+0000 7f639bb11740 4 client.15924 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:04:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.107+0000 7f639bb11740 1 -- 172.21.15.98:0/1017378910 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647e6ea0860 con 0x5647e6f47fb0 2024-09-19T08:04:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.107+0000 7f63927fc640 1 --2- 172.21.15.98:0/1017378910 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5647e6f47fb0 0x5647e6f4a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.108+0000 7f6376ff5640 10 client.15924 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.108+0000 7f63927fc640 1 --2- 172.21.15.98:0/1017378910 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5647e6f47fb0 0x5647e6f4a3e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=489da1652728a747 server_cookie=b1dbfebcfcf51d43 in_seq=0 out_seq=0 2024-09-19T08:04:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f6376ff5640 1 -- 172.21.15.98:0/1017378910 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5647e6ea0860 con 0x5647e6f47fb0 2024-09-19T08:04:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f6376ff5640 10 client.15924 handle_command_reply: tid=0 2024-09-19T08:04:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f639bb11740 1 client.15924 shutdown 2024-09-19T08:04:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f639bb11740 1 -- 172.21.15.98:0/1017378910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 msgr2=0x5647e6f37d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f639bb11740 1 --2- 172.21.15.98:0/1017378910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f37d80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f63840c1910 tx=0x7f63840d49b0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f639bb11740 1 -- 172.21.15.98:0/1017378910 shutdown_connections 2024-09-19T08:04:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f639bb11740 1 --2- 172.21.15.98:0/1017378910 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5647e6f47fb0 0x5647e6f4a3e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f639bb11740 1 --2- 172.21.15.98:0/1017378910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f38370 0x5647e6f35640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f639bb11740 1 --2- 172.21.15.98:0/1017378910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 0x5647e6f35030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f6376ff5640 0 client.15924 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f639bb11740 1 --2- 172.21.15.98:0/1017378910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f37d80 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f639bb11740 1 -- 172.21.15.98:0/1017378910 >> 172.21.15.98:0/1017378910 conn(0x5647e6ef6310 msgr2=0x5647e6f2f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f639bb11740 1 -- 172.21.15.98:0/1017378910 shutdown_connections 2024-09-19T08:04:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f639bb11740 1 -- 172.21.15.98:0/1017378910 wait complete. 2024-09-19T08:04:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.109+0000 7f639bb11740 20 client.15924 trim_cache size 0 max 16384 2024-09-19T08:04:39.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:38 smithi162 ceph-mon[35260]: pgmap v52: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 343 KiB/s wr, 10 op/s 2024-09-19T08:04:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f639bb11740 1 Processor -- start 2024-09-19T08:04:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f639bb11740 1 -- start start 2024-09-19T08:04:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f639bb11740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f2d5d0 0x5647e6f28c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f639bb11740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 0x5647e6f29150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f639bb11740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f38370 0x5647e6f30120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f639bb11740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63941ab4c0 con 0x5647e6f38370 2024-09-19T08:04:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f639bb11740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f63941ad000 con 0x5647e6f2eb40 2024-09-19T08:04:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f639bb11740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f63941abb90 con 0x5647e6f2d5d0 2024-09-19T08:04:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f63937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f38370 0x5647e6f30120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f63937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f38370 0x5647e6f30120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50438/0 (socket says 172.21.15.98:50438) 2024-09-19T08:04:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f63937fe640 1 -- 172.21.15.98:0/1232163995 learned_addr learned my addr 172.21.15.98:0/1232163995 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f6392ffd640 1 --2- 172.21.15.98:0/1232163995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f2d5d0 0x5647e6f28c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f63927fc640 1 --2- 172.21.15.98:0/1232163995 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 0x5647e6f29150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f63937fe640 1 -- 172.21.15.98:0/1232163995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f2d5d0 msgr2=0x5647e6f28c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f63937fe640 1 --2- 172.21.15.98:0/1232163995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f2d5d0 0x5647e6f28c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f63937fe640 1 -- 172.21.15.98:0/1232163995 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 msgr2=0x5647e6f29150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f63937fe640 1 --2- 172.21.15.98:0/1232163995 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 0x5647e6f29150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f63937fe640 1 -- 172.21.15.98:0/1232163995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647e6f34000 con 0x5647e6f38370 2024-09-19T08:04:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f63927fc640 1 --2- 172.21.15.98:0/1232163995 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 0x5647e6f29150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.122+0000 7f63937fe640 1 --2- 172.21.15.98:0/1232163995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f38370 0x5647e6f30120 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f638c00b6b0 tx=0x7f638c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b6de7528b2e216 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.123+0000 7f6378ff9640 1 -- 172.21.15.98:0/1232163995 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c023070 con 0x5647e6f38370 2024-09-19T08:04:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.123+0000 7f6378ff9640 1 -- 172.21.15.98:0/1232163995 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f638c016040 con 0x5647e6f38370 2024-09-19T08:04:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.123+0000 7f6378ff9640 1 -- 172.21.15.98:0/1232163995 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c018070 con 0x5647e6f38370 2024-09-19T08:04:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.123+0000 7f639bb11740 1 -- 172.21.15.98:0/1232163995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f38370 msgr2=0x5647e6f30120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.123+0000 7f639bb11740 1 --2- 172.21.15.98:0/1232163995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f38370 0x5647e6f30120 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f638c00b6b0 tx=0x7f638c014000 comp rx=0 tx=0).stop 2024-09-19T08:04:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.123+0000 7f639bb11740 1 -- 172.21.15.98:0/1232163995 shutdown_connections 2024-09-19T08:04:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.123+0000 7f639bb11740 1 --2- 172.21.15.98:0/1232163995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f38370 0x5647e6f30120 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.123+0000 7f639bb11740 1 --2- 172.21.15.98:0/1232163995 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 0x5647e6f29150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.123+0000 7f639bb11740 1 --2- 172.21.15.98:0/1232163995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f2d5d0 0x5647e6f28c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.123+0000 7f639bb11740 1 -- 172.21.15.98:0/1232163995 >> 172.21.15.98:0/1232163995 conn(0x5647e6ef6310 msgr2=0x5647e6f28500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.123+0000 7f639bb11740 1 -- 172.21.15.98:0/1232163995 shutdown_connections 2024-09-19T08:04:39.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.123+0000 7f639bb11740 1 -- 172.21.15.98:0/1232163995 wait complete. 2024-09-19T08:04:39.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.123+0000 7f639bb11740 1 -- start start 2024-09-19T08:04:39.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f639bb11740 1 Processor -- start 2024-09-19T08:04:39.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f639bb11740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f40cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f639bb11740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 0x5647e6f31e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f639bb11740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f38370 0x5647e6f32440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f639bb11740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63941ac930 con 0x5647e6f2d5d0 2024-09-19T08:04:39.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f639bb11740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f63941ac260 con 0x5647e6f2eb40 2024-09-19T08:04:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f639bb11740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f63941a94c0 con 0x5647e6f38370 2024-09-19T08:04:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f639bb11740 10 client.0 _ll_register_callbacks cb 0x7f638402ff50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f6392ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f40cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f6392ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f40cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50450/0 (socket says 172.21.15.98:50450) 2024-09-19T08:04:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f63937fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f38370 0x5647e6f32440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f6392ffd640 1 -- 172.21.15.98:0/4043799952 learned_addr learned my addr 172.21.15.98:0/4043799952 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f63927fc640 1 --2- 172.21.15.98:0/4043799952 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 0x5647e6f31e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f6392ffd640 1 -- 172.21.15.98:0/4043799952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f38370 msgr2=0x5647e6f32440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f6392ffd640 1 --2- 172.21.15.98:0/4043799952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f38370 0x5647e6f32440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f6392ffd640 1 -- 172.21.15.98:0/4043799952 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 msgr2=0x5647e6f31e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f6392ffd640 1 --2- 172.21.15.98:0/4043799952 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 0x5647e6f31e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f6392ffd640 1 -- 172.21.15.98:0/4043799952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647e6f32c70 con 0x5647e6f2d5d0 2024-09-19T08:04:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f63937fe640 1 --2- 172.21.15.98:0/4043799952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f38370 0x5647e6f32440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:04:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f6392ffd640 1 --2- 172.21.15.98:0/4043799952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f40cb0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f638402ade0 tx=0x7f6384030880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882c5d121075746d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.124+0000 7f6376ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f6376ff5640 1 -- 172.21.15.98:0/4043799952 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63840d8e60 con 0x5647e6f2d5d0 2024-09-19T08:04:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f6376ff5640 1 -- 172.21.15.98:0/4043799952 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f6384009890 con 0x5647e6f2d5d0 2024-09-19T08:04:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f639bb11740 1 -- 172.21.15.98:0/4043799952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f638c016040 con 0x5647e6f2d5d0 2024-09-19T08:04:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f6376ff5640 1 -- 172.21.15.98:0/4043799952 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f63840c1c10 con 0x5647e6f2d5d0 2024-09-19T08:04:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f639bb11740 10 client.15930 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f639bb11740 1 -- 172.21.15.98:0/4043799952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5647e6fb2ce0 con 0x5647e6f2d5d0 2024-09-19T08:04:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f6376ff5640 1 -- 172.21.15.98:0/4043799952 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f638c016040 con 0x5647e6f2d5d0 2024-09-19T08:04:39.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f6376ff5640 1 -- 172.21.15.98:0/4043799952 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f63840ca3f0 con 0x5647e6f2d5d0 2024-09-19T08:04:39.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f639bb11740 10 client.15930 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:39.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f639bb11740 10 client.15930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:04:39.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f639bb11740 20 client.15930 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:39.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f639bb11740 1 --2- 172.21.15.98:0/4043799952 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5647e6fb32c0 0x5647e6fb56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:39.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f639bb11740 4 client.15930 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:04:39.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f639bb11740 1 -- 172.21.15.98:0/4043799952 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5647e6e9ad70 con 0x5647e6fb32c0 2024-09-19T08:04:39.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.125+0000 7f63927fc640 1 --2- 172.21.15.98:0/4043799952 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5647e6fb32c0 0x5647e6fb56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:39.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f6376ff5640 10 client.15930 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:39.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f63927fc640 1 --2- 172.21.15.98:0/4043799952 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5647e6fb32c0 0x5647e6fb56b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3819b5155f19da9 server_cookie=94ffd5597c22dcc0 in_seq=0 out_seq=0 2024-09-19T08:04:39.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f6376ff5640 1 -- 172.21.15.98:0/4043799952 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5647e6e9ad70 con 0x5647e6fb32c0 2024-09-19T08:04:39.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f6376ff5640 10 client.15930 handle_command_reply: tid=0 2024-09-19T08:04:39.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f639bb11740 1 client.15930 shutdown 2024-09-19T08:04:39.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f639bb11740 1 -- 172.21.15.98:0/4043799952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 msgr2=0x5647e6f40cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:39.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f639bb11740 1 --2- 172.21.15.98:0/4043799952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f40cb0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f638402ade0 tx=0x7f6384030880 comp rx=0 tx=0).stop 2024-09-19T08:04:39.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f639bb11740 1 -- 172.21.15.98:0/4043799952 shutdown_connections 2024-09-19T08:04:39.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f639bb11740 1 --2- 172.21.15.98:0/4043799952 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5647e6fb32c0 0x5647e6fb56b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f639bb11740 1 --2- 172.21.15.98:0/4043799952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647e6f38370 0x5647e6f32440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f639bb11740 1 --2- 172.21.15.98:0/4043799952 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e6f2eb40 0x5647e6f31e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f639bb11740 1 --2- 172.21.15.98:0/4043799952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e6f2d5d0 0x5647e6f40cb0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f6376ff5640 0 client.15930 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:39.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f639bb11740 1 -- 172.21.15.98:0/4043799952 >> 172.21.15.98:0/4043799952 conn(0x5647e6ef6310 msgr2=0x5647e6f28d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:39.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f639bb11740 1 -- 172.21.15.98:0/4043799952 shutdown_connections 2024-09-19T08:04:39.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f639bb11740 1 -- 172.21.15.98:0/4043799952 wait complete. 2024-09-19T08:04:39.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.126+0000 7f639bb11740 20 client.15930 trim_cache size 0 max 16384 2024-09-19T08:04:39.134 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:04:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.127+0000 7f6399738640 1 -- 172.21.15.98:0/2183512729 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f637c0a33b0 msgr2=0x7f637c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.127+0000 7f6399738640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f637c0a33b0 0x7f637c0a5870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f638c00bb30 tx=0x7f638c00bf30 comp rx=0 tx=0).stop 2024-09-19T08:04:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.127+0000 7f6399738640 1 -- 172.21.15.98:0/2183512729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394101df0 msgr2=0x7f6394198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.127+0000 7f6399738640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394101df0 0x7f6394198440 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f63840097b0 tx=0x7f6384031b10 comp rx=0 tx=0).stop 2024-09-19T08:04:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.127+0000 7f6399738640 1 -- 172.21.15.98:0/2183512729 shutdown_connections 2024-09-19T08:04:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.127+0000 7f6399738640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f637c0a33b0 0x7f637c0a5870 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.127+0000 7f6399738640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6394106f20 0x7f639419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.127+0000 7f6399738640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63941027a0 0x7f6394198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.127+0000 7f6399738640 1 --2- 172.21.15.98:0/2183512729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394101df0 0x7f6394198440 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:39.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.127+0000 7f6399738640 1 -- 172.21.15.98:0/2183512729 >> 172.21.15.98:0/2183512729 conn(0x7f63940fd8b0 msgr2=0x7f63940febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:39.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.127+0000 7f6399738640 1 -- 172.21.15.98:0/2183512729 shutdown_connections 2024-09-19T08:04:39.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:39.127+0000 7f6399738640 1 -- 172.21.15.98:0/2183512729 wait complete. 2024-09-19T08:04:39.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:38 smithi139 ceph-mon[34874]: pgmap v52: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 343 KiB/s wr, 10 op/s 2024-09-19T08:04:40.872 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:04:40.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:04:41.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:40 smithi162 ceph-mon[35260]: pgmap v53: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 343 KiB/s wr, 10 op/s 2024-09-19T08:04:41.133 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:41.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:40 smithi139 ceph-mon[34874]: pgmap v53: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 343 KiB/s wr, 10 op/s 2024-09-19T08:04:41.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:40 smithi098 ceph-mon[31333]: pgmap v53: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 343 KiB/s wr, 10 op/s 2024-09-19T08:04:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.592+0000 7f4140284640 1 -- 172.21.15.98:0/2140648776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413810c170 msgr2=0x7f413810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.592+0000 7f4140284640 1 --2- 172.21.15.98:0/2140648776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413810c170 0x7f413810e610 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4130009ca0 tx=0x7f413002a950 comp rx=0 tx=0).stop 2024-09-19T08:04:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 -- 172.21.15.98:0/2140648776 shutdown_connections 2024-09-19T08:04:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 --2- 172.21.15.98:0/2140648776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413810c170 0x7f413810e610 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 --2- 172.21.15.98:0/2140648776 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4138103660 0x7f413810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 --2- 172.21.15.98:0/2140648776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4138102d40 0x7f4138103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 -- 172.21.15.98:0/2140648776 >> 172.21.15.98:0/2140648776 conn(0x7f41380fc7a0 msgr2=0x7f41380febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 -- 172.21.15.98:0/2140648776 shutdown_connections 2024-09-19T08:04:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 -- 172.21.15.98:0/2140648776 wait complete. 2024-09-19T08:04:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 Processor -- start 2024-09-19T08:04:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 -- start start 2024-09-19T08:04:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4138102d40 0x7f4138198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4138103660 0x7f41381989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413810c170 0x7f413819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4138111fb0 con 0x7f413810c170 2024-09-19T08:04:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4138111e30 con 0x7f4138102d40 2024-09-19T08:04:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.593+0000 7f4140284640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4138112130 con 0x7f4138103660 2024-09-19T08:04:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f413e821640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413810c170 0x7f413819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f413e821640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413810c170 0x7f413819cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50472/0 (socket says 172.21.15.98:50472) 2024-09-19T08:04:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f413e821640 1 -- 172.21.15.98:0/1732259653 learned_addr learned my addr 172.21.15.98:0/1732259653 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f413d81f640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4138103660 0x7f41381989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f413e020640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4138102d40 0x7f4138198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f413e821640 1 -- 172.21.15.98:0/1732259653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4138103660 msgr2=0x7f41381989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f413e821640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4138103660 0x7f41381989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f413e821640 1 -- 172.21.15.98:0/1732259653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4138102d40 msgr2=0x7f4138198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f413e821640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4138102d40 0x7f4138198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f413e821640 1 -- 172.21.15.98:0/1732259653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413819d420 con 0x7f413810c170 2024-09-19T08:04:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f413e020640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4138102d40 0x7f4138198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f413e821640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413810c170 0x7f413819cd40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4130006010 tx=0x7f413002f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f412f7fe640 1 -- 172.21.15.98:0/1732259653 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4130034040 con 0x7f413810c170 2024-09-19T08:04:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f412f7fe640 1 -- 172.21.15.98:0/1732259653 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4130003e00 con 0x7f413810c170 2024-09-19T08:04:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f4140284640 1 -- 172.21.15.98:0/1732259653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f413819d6b0 con 0x7f413810c170 2024-09-19T08:04:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f412f7fe640 1 -- 172.21.15.98:0/1732259653 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41300456e0 con 0x7f413810c170 2024-09-19T08:04:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.594+0000 7f4140284640 1 -- 172.21.15.98:0/1732259653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41381a4ff0 con 0x7f413810c170 2024-09-19T08:04:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.596+0000 7f412f7fe640 1 -- 172.21.15.98:0/1732259653 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f4130037020 con 0x7f413810c170 2024-09-19T08:04:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.597+0000 7f414265d740 1 Processor -- start 2024-09-19T08:04:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.597+0000 7f414265d740 1 -- start start 2024-09-19T08:04:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.597+0000 7f412f7fe640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f41240a33b0 0x7f41240a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.597+0000 7f414265d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdade20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.597+0000 7f414265d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdae360 0x56428cdbcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.597+0000 7f414265d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 0x56428cdbf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.597+0000 7f414265d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56428cd13280 con 0x56428cdae360 2024-09-19T08:04:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.597+0000 7f414265d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56428ccbab10 con 0x56428cdbd150 2024-09-19T08:04:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.597+0000 7f414265d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56428cd0f240 con 0x56428cdada40 2024-09-19T08:04:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f412f7fe640 1 -- 172.21.15.98:0/1732259653 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f41300f0cb0 con 0x7f413810c170 2024-09-19T08:04:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413d81f640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f41240a33b0 0x7f41240a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413e020640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413d81f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdae360 0x56428cdbcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413e020640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdade20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:47864/0 (socket says 172.21.15.98:47864) 2024-09-19T08:04:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413d81f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdae360 0x56428cdbcc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50488/0 (socket says 172.21.15.98:50488) 2024-09-19T08:04:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413e020640 1 -- 172.21.15.98:0/3395033206 learned_addr learned my addr 172.21.15.98:0/3395033206 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413e821640 1 --2- 172.21.15.98:0/3395033206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 0x56428cdbf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413e020640 1 -- 172.21.15.98:0/3395033206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 msgr2=0x56428cdbf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413e020640 1 --2- 172.21.15.98:0/3395033206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 0x56428cdbf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413e020640 1 -- 172.21.15.98:0/3395033206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdae360 msgr2=0x56428cdbcc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413e020640 1 --2- 172.21.15.98:0/3395033206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdae360 0x56428cdbcc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413e020640 1 -- 172.21.15.98:0/3395033206 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56428ccc71f0 con 0x56428cdada40 2024-09-19T08:04:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413d81f640 1 --2- 172.21.15.98:0/3395033206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdae360 0x56428cdbcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:04:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413d81f640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f41240a33b0 0x7f41240a5870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4134006e10 tx=0x7f4134009930 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.598+0000 7f413e020640 1 --2- 172.21.15.98:0/3395033206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdade20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f412800cca0 tx=0x7f412800e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=176b170711bd96b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.599+0000 7f412cdea640 1 -- 172.21.15.98:0/3395033206 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128005220 con 0x56428cdada40 2024-09-19T08:04:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.599+0000 7f412cdea640 1 -- 172.21.15.98:0/3395033206 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4128004c80 con 0x56428cdada40 2024-09-19T08:04:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.599+0000 7f412cdea640 1 -- 172.21.15.98:0/3395033206 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128010720 con 0x56428cdada40 2024-09-19T08:04:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.599+0000 7f414265d740 1 -- 172.21.15.98:0/3395033206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 msgr2=0x56428cdade20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.599+0000 7f414265d740 1 --2- 172.21.15.98:0/3395033206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdade20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f412800cca0 tx=0x7f412800e720 comp rx=0 tx=0).stop 2024-09-19T08:04:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.599+0000 7f414265d740 1 -- 172.21.15.98:0/3395033206 shutdown_connections 2024-09-19T08:04:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.599+0000 7f414265d740 1 --2- 172.21.15.98:0/3395033206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 0x56428cdbf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.599+0000 7f414265d740 1 --2- 172.21.15.98:0/3395033206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdae360 0x56428cdbcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.599+0000 7f414265d740 1 --2- 172.21.15.98:0/3395033206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdade20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.599+0000 7f414265d740 1 -- 172.21.15.98:0/3395033206 >> 172.21.15.98:0/3395033206 conn(0x56428cdaa0f0 msgr2=0x56428cdaa4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.599+0000 7f414265d740 1 -- 172.21.15.98:0/3395033206 shutdown_connections 2024-09-19T08:04:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.599+0000 7f414265d740 1 -- 172.21.15.98:0/3395033206 wait complete. 2024-09-19T08:04:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.600+0000 7f414265d740 1 -- start start 2024-09-19T08:04:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.600+0000 7f414265d740 1 Processor -- start 2024-09-19T08:04:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.600+0000 7f414265d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdb9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.600+0000 7f414265d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdae360 0x56428cdba4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.600+0000 7f414265d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdbd150 0x56428cdbaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.600+0000 7f414265d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56428cdbfd20 con 0x56428cdbd150 2024-09-19T08:04:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.600+0000 7f414265d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56428cccdf10 con 0x56428cdae360 2024-09-19T08:04:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.600+0000 7f414265d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56428cdbfea0 con 0x56428cdada40 2024-09-19T08:04:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.600+0000 7f414265d740 10 client.0 _ll_register_callbacks cb 0x56428cc96a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.600+0000 7f413e821640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdbd150 0x56428cdbaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.600+0000 7f413e821640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdbd150 0x56428cdbaa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50502/0 (socket says 172.21.15.98:50502) 2024-09-19T08:04:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.600+0000 7f413e821640 1 -- 172.21.15.98:0/2610589897 learned_addr learned my addr 172.21.15.98:0/2610589897 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.600+0000 7f413e020640 1 --2- 172.21.15.98:0/2610589897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdb9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.600+0000 7f413d81f640 1 --2- 172.21.15.98:0/2610589897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdae360 0x56428cdba4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f413e821640 1 -- 172.21.15.98:0/2610589897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 msgr2=0x56428cdb9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f413e821640 1 --2- 172.21.15.98:0/2610589897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdb9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f413e821640 1 -- 172.21.15.98:0/2610589897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdae360 msgr2=0x56428cdba4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f413e821640 1 --2- 172.21.15.98:0/2610589897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdae360 0x56428cdba4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f413e821640 1 -- 172.21.15.98:0/2610589897 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56428ca846a0 con 0x56428cdbd150 2024-09-19T08:04:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f413d81f640 1 --2- 172.21.15.98:0/2610589897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdae360 0x56428cdba4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f413e821640 1 --2- 172.21.15.98:0/2610589897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdbd150 0x56428cdbaa30 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4130009430 tx=0x7f41300bcaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a3539e1503d3c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f4121ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f4121ffb640 1 -- 172.21.15.98:0/2610589897 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413002c040 con 0x56428cdbd150 2024-09-19T08:04:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f4121ffb640 1 -- 172.21.15.98:0/2610589897 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f41300c1190 con 0x56428cdbd150 2024-09-19T08:04:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f414265d740 1 -- 172.21.15.98:0/2610589897 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4128004c80 con 0x56428cdbd150 2024-09-19T08:04:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f4121ffb640 1 -- 172.21.15.98:0/2610589897 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4130004830 con 0x56428cdbd150 2024-09-19T08:04:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f414265d740 10 client.15942 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f414265d740 1 -- 172.21.15.98:0/2610589897 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56428cdcb8e0 con 0x56428cdbd150 2024-09-19T08:04:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f4121ffb640 1 -- 172.21.15.98:0/2610589897 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4128004c80 con 0x56428cdbd150 2024-09-19T08:04:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f4121ffb640 1 -- 172.21.15.98:0/2610589897 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f413004d030 con 0x56428cdbd150 2024-09-19T08:04:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f414265d740 10 client.15942 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f414265d740 10 client.15942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:04:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f414265d740 20 client.15942 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f414265d740 1 --2- 172.21.15.98:0/2610589897 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56428cdccd20 0x56428cdcf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f414265d740 4 client.15942 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:04:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.601+0000 7f414265d740 1 -- 172.21.15.98:0/2610589897 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56428cd207f0 con 0x56428cdccd20 2024-09-19T08:04:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.602+0000 7f413e020640 1 --2- 172.21.15.98:0/2610589897 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56428cdccd20 0x56428cdcf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.602+0000 7f413e020640 1 --2- 172.21.15.98:0/2610589897 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56428cdccd20 0x56428cdcf1a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=254dfb5ab02aa139 server_cookie=a59e9d1b8a413386 in_seq=0 out_seq=0 2024-09-19T08:04:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.602+0000 7f4121ffb640 10 client.15942 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f4121ffb640 1 -- 172.21.15.98:0/2610589897 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56428cd207f0 con 0x56428cdccd20 2024-09-19T08:04:41.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f4121ffb640 10 client.15942 handle_command_reply: tid=0 2024-09-19T08:04:41.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f414265d740 1 client.15942 shutdown 2024-09-19T08:04:41.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f414265d740 1 -- 172.21.15.98:0/2610589897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdbd150 msgr2=0x56428cdbaa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:41.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f414265d740 1 --2- 172.21.15.98:0/2610589897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdbd150 0x56428cdbaa30 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4130009430 tx=0x7f41300bcaf0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f414265d740 1 -- 172.21.15.98:0/2610589897 shutdown_connections 2024-09-19T08:04:41.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f414265d740 1 --2- 172.21.15.98:0/2610589897 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56428cdccd20 0x56428cdcf1a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f414265d740 1 --2- 172.21.15.98:0/2610589897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdbd150 0x56428cdbaa30 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f414265d740 1 --2- 172.21.15.98:0/2610589897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdae360 0x56428cdba4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f414265d740 1 --2- 172.21.15.98:0/2610589897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdb9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f4121ffb640 0 client.15942 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:41.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f414265d740 1 -- 172.21.15.98:0/2610589897 >> 172.21.15.98:0/2610589897 conn(0x56428cdaa0f0 msgr2=0x56428cda9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:41.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f414265d740 1 -- 172.21.15.98:0/2610589897 shutdown_connections 2024-09-19T08:04:41.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f414265d740 1 -- 172.21.15.98:0/2610589897 wait complete. 2024-09-19T08:04:41.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.603+0000 7f414265d740 20 client.15942 trim_cache size 0 max 16384 2024-09-19T08:04:41.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f414265d740 1 Processor -- start 2024-09-19T08:04:41.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f414265d740 1 -- start start 2024-09-19T08:04:41.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f414265d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdada40 0x56428cdc8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f414265d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdae360 0x56428cdbbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f414265d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 0x56428cdd4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f414265d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f41381ab4b0 con 0x56428cdada40 2024-09-19T08:04:41.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f414265d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f41381acff0 con 0x56428cdbd150 2024-09-19T08:04:41.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f414265d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f41381abb80 con 0x56428cdae360 2024-09-19T08:04:41.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f413e020640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdada40 0x56428cdc8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f413e020640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdada40 0x56428cdc8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50518/0 (socket says 172.21.15.98:50518) 2024-09-19T08:04:41.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f413e020640 1 -- 172.21.15.98:0/242959902 learned_addr learned my addr 172.21.15.98:0/242959902 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f413d81f640 1 --2- 172.21.15.98:0/242959902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdae360 0x56428cdbbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f413e821640 1 --2- 172.21.15.98:0/242959902 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 0x56428cdd4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f413e020640 1 -- 172.21.15.98:0/242959902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdae360 msgr2=0x56428cdbbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f413e020640 1 --2- 172.21.15.98:0/242959902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdae360 0x56428cdbbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f413e020640 1 -- 172.21.15.98:0/242959902 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 msgr2=0x56428cdd4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f413e020640 1 --2- 172.21.15.98:0/242959902 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 0x56428cdd4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f413e020640 1 -- 172.21.15.98:0/242959902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56428cda9b90 con 0x56428cdada40 2024-09-19T08:04:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.620+0000 7f413d81f640 1 --2- 172.21.15.98:0/242959902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdae360 0x56428cdbbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.621+0000 7f413e020640 1 --2- 172.21.15.98:0/242959902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdada40 0x56428cdc8fc0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f412800ec20 tx=0x7f412800c590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4858db2865d5d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.621+0000 7f4123fff640 1 -- 172.21.15.98:0/242959902 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128004220 con 0x56428cdada40 2024-09-19T08:04:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.621+0000 7f4123fff640 1 -- 172.21.15.98:0/242959902 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41280043c0 con 0x56428cdada40 2024-09-19T08:04:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.621+0000 7f4123fff640 1 -- 172.21.15.98:0/242959902 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128003850 con 0x56428cdada40 2024-09-19T08:04:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.621+0000 7f414265d740 1 -- 172.21.15.98:0/242959902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdada40 msgr2=0x56428cdc8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.621+0000 7f414265d740 1 --2- 172.21.15.98:0/242959902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdada40 0x56428cdc8fc0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f412800ec20 tx=0x7f412800c590 comp rx=0 tx=0).stop 2024-09-19T08:04:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.621+0000 7f414265d740 1 -- 172.21.15.98:0/242959902 shutdown_connections 2024-09-19T08:04:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.621+0000 7f414265d740 1 --2- 172.21.15.98:0/242959902 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 0x56428cdd4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.621+0000 7f414265d740 1 --2- 172.21.15.98:0/242959902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdae360 0x56428cdbbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.621+0000 7f414265d740 1 --2- 172.21.15.98:0/242959902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdada40 0x56428cdc8fc0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.621+0000 7f414265d740 1 -- 172.21.15.98:0/242959902 >> 172.21.15.98:0/242959902 conn(0x56428cdaa0f0 msgr2=0x56428cdbb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:41.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.621+0000 7f414265d740 1 -- 172.21.15.98:0/242959902 shutdown_connections 2024-09-19T08:04:41.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.622+0000 7f414265d740 1 -- 172.21.15.98:0/242959902 wait complete. 2024-09-19T08:04:41.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.622+0000 7f414265d740 1 -- start start 2024-09-19T08:04:41.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.622+0000 7f414265d740 1 Processor -- start 2024-09-19T08:04:41.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.622+0000 7f414265d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdc5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.622+0000 7f414265d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdae360 0x56428cdc0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.622+0000 7f414265d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 0x56428cdc10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.622+0000 7f414265d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f41381ac920 con 0x56428cdae360 2024-09-19T08:04:41.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.622+0000 7f414265d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f41381ac250 con 0x56428cdbd150 2024-09-19T08:04:41.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.622+0000 7f414265d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f41381a94b0 con 0x56428cdada40 2024-09-19T08:04:41.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.622+0000 7f414265d740 10 client.0 _ll_register_callbacks cb 0x7f4130009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:41.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.622+0000 7f413d81f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdae360 0x56428cdc0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.622+0000 7f413d81f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdae360 0x56428cdc0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50524/0 (socket says 172.21.15.98:50524) 2024-09-19T08:04:41.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.622+0000 7f413d81f640 1 -- 172.21.15.98:0/1275321698 learned_addr learned my addr 172.21.15.98:0/1275321698 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:41.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f413e020640 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdc5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f413e821640 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 0x56428cdc10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f413d81f640 1 -- 172.21.15.98:0/1275321698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 msgr2=0x56428cdc5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:41.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f413d81f640 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdc5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f413d81f640 1 -- 172.21.15.98:0/1275321698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 msgr2=0x56428cdc10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:41.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f413d81f640 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 0x56428cdc10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f413d81f640 1 -- 172.21.15.98:0/1275321698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56428cdc18b0 con 0x56428cdae360 2024-09-19T08:04:41.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f413e821640 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 0x56428cdc10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:41.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f413e020640 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdc5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:41.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f413d81f640 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdae360 0x56428cdc0b70 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x56428cdb9840 tx=0x7f4134009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3f3308f7631bac server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:41.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f4121ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:41.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f4121ffb640 1 -- 172.21.15.98:0/1275321698 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413402a070 con 0x56428cdae360 2024-09-19T08:04:41.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f4121ffb640 1 -- 172.21.15.98:0/1275321698 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f4134018bb0 con 0x56428cdae360 2024-09-19T08:04:41.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f414265d740 1 -- 172.21.15.98:0/1275321698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41280043c0 con 0x56428cdae360 2024-09-19T08:04:41.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f4121ffb640 1 -- 172.21.15.98:0/1275321698 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4134018db0 con 0x56428cdae360 2024-09-19T08:04:41.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f414265d740 10 client.15948 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:41.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f414265d740 1 -- 172.21.15.98:0/1275321698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56428ce37a40 con 0x56428cdae360 2024-09-19T08:04:41.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.623+0000 7f4121ffb640 1 -- 172.21.15.98:0/1275321698 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41280043c0 con 0x56428cdae360 2024-09-19T08:04:41.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.624+0000 7f4121ffb640 1 -- 172.21.15.98:0/1275321698 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4134022040 con 0x56428cdae360 2024-09-19T08:04:41.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.624+0000 7f414265d740 10 client.15948 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:41.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.624+0000 7f414265d740 10 client.15948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:04:41.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.624+0000 7f414265d740 20 client.15948 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:41.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.624+0000 7f414265d740 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56428ce38190 0x56428ce3a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:41.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.624+0000 7f414265d740 4 client.15948 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:04:41.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.624+0000 7f414265d740 1 -- 172.21.15.98:0/1275321698 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56428cd1ad00 con 0x56428ce38190 2024-09-19T08:04:41.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.624+0000 7f413e020640 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56428ce38190 0x56428ce3a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.624+0000 7f4121ffb640 10 client.15948 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.624+0000 7f413e020640 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56428ce38190 0x56428ce3a5b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2c05a1f39bb4237 server_cookie=13ad77f5f487e4f2 in_seq=0 out_seq=0 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.625+0000 7f4121ffb640 1 -- 172.21.15.98:0/1275321698 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56428cd1ad00 con 0x56428ce38190 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.625+0000 7f4121ffb640 10 client.15948 handle_command_reply: tid=0 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.625+0000 7f414265d740 1 client.15948 shutdown 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f414265d740 1 -- 172.21.15.98:0/1275321698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdae360 msgr2=0x56428cdc0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f414265d740 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdae360 0x56428cdc0b70 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x56428cdb9840 tx=0x7f4134009060 comp rx=0 tx=0).stop 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f414265d740 1 -- 172.21.15.98:0/1275321698 shutdown_connections 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f414265d740 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56428ce38190 0x56428ce3a5b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f414265d740 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428cdbd150 0x56428cdc10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f414265d740 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56428cdae360 0x56428cdc0b70 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f414265d740 1 --2- 172.21.15.98:0/1275321698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56428cdada40 0x56428cdc5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f414265d740 1 -- 172.21.15.98:0/1275321698 >> 172.21.15.98:0/1275321698 conn(0x56428cdaa0f0 msgr2=0x56428cda9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f4121ffb640 0 client.15948 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f414265d740 1 -- 172.21.15.98:0/1275321698 shutdown_connections 2024-09-19T08:04:41.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f414265d740 1 -- 172.21.15.98:0/1275321698 wait complete. 2024-09-19T08:04:41.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f414265d740 20 client.15948 trim_cache size 0 max 16384 2024-09-19T08:04:41.631 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:04:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:04:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "e65644f3-2efe-49b2-83c9-06e3cc521893", 2024-09-19T08:04:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:04:41.631 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:04:41.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f4140284640 1 -- 172.21.15.98:0/1732259653 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f41240a33b0 msgr2=0x7f41240a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:41.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f4140284640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f41240a33b0 0x7f41240a5870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4134006e10 tx=0x7f4134009930 comp rx=0 tx=0).stop 2024-09-19T08:04:41.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f4140284640 1 -- 172.21.15.98:0/1732259653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413810c170 msgr2=0x7f413819cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:41.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f4140284640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413810c170 0x7f413819cd40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4130006010 tx=0x7f413002f710 comp rx=0 tx=0).stop 2024-09-19T08:04:41.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f4140284640 1 -- 172.21.15.98:0/1732259653 shutdown_connections 2024-09-19T08:04:41.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f4140284640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f41240a33b0 0x7f41240a5870 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f4140284640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413810c170 0x7f413819cd40 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f4140284640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4138103660 0x7f41381989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f4140284640 1 --2- 172.21.15.98:0/1732259653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4138102d40 0x7f4138198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:41.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f4140284640 1 -- 172.21.15.98:0/1732259653 >> 172.21.15.98:0/1732259653 conn(0x7f41380fc7a0 msgr2=0x7f41380fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:41.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.626+0000 7f4140284640 1 -- 172.21.15.98:0/1732259653 shutdown_connections 2024-09-19T08:04:41.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:41.627+0000 7f4140284640 1 -- 172.21.15.98:0/1732259653 wait complete. 2024-09-19T08:04:42.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:04:42.532 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:42.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:42 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:04:42.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:42 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:04:42.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:42 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:04:42.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:42 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:04:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:42 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:04:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:42 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:04:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:42 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:04:42.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:42 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:04:42.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:42 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:04:42.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:42 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:04:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:42 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:04:42.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:42 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:04:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.975+0000 7f71b922b640 1 -- 172.21.15.98:0/1110376332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b410c180 msgr2=0x7f71b410e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.975+0000 7f71b922b640 1 --2- 172.21.15.98:0/1110376332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b410c180 0x7f71b410e600 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009ca0 tx=0x7f71ac02a950 comp rx=0 tx=0).stop 2024-09-19T08:04:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.975+0000 7f71b922b640 1 -- 172.21.15.98:0/1110376332 shutdown_connections 2024-09-19T08:04:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.975+0000 7f71b922b640 1 --2- 172.21.15.98:0/1110376332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b410c180 0x7f71b410e600 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.975+0000 7f71b922b640 1 --2- 172.21.15.98:0/1110376332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71b41036c0 0x7f71b410bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.975+0000 7f71b922b640 1 --2- 172.21.15.98:0/1110376332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71b4102da0 0x7f71b4103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.975+0000 7f71b922b640 1 -- 172.21.15.98:0/1110376332 >> 172.21.15.98:0/1110376332 conn(0x7f71b40fc7a0 msgr2=0x7f71b40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.975+0000 7f71b922b640 1 -- 172.21.15.98:0/1110376332 shutdown_connections 2024-09-19T08:04:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.975+0000 7f71b922b640 1 -- 172.21.15.98:0/1110376332 wait complete. 2024-09-19T08:04:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.975+0000 7f71b922b640 1 Processor -- start 2024-09-19T08:04:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.975+0000 7f71b922b640 1 -- start start 2024-09-19T08:04:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b922b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b4102da0 0x7f71b4100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b922b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71b41036c0 0x7f71b4100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b922b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71b410c180 0x7f71b4101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b922b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f71b4111f60 con 0x7f71b4102da0 2024-09-19T08:04:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b922b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f71b4111de0 con 0x7f71b41036c0 2024-09-19T08:04:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b922b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71b41120e0 con 0x7f71b410c180 2024-09-19T08:04:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b4102da0 0x7f71b4100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b4102da0 0x7f71b4100a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50542/0 (socket says 172.21.15.98:50542) 2024-09-19T08:04:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b2d9d640 1 -- 172.21.15.98:0/1261407287 learned_addr learned my addr 172.21.15.98:0/1261407287 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71abfff640 1 --2- 172.21.15.98:0/1261407287 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71b41036c0 0x7f71b4100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b359e640 1 --2- 172.21.15.98:0/1261407287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71b410c180 0x7f71b4101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b2d9d640 1 -- 172.21.15.98:0/1261407287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71b410c180 msgr2=0x7f71b4101480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b2d9d640 1 --2- 172.21.15.98:0/1261407287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71b410c180 0x7f71b4101480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b2d9d640 1 -- 172.21.15.98:0/1261407287 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71b41036c0 msgr2=0x7f71b4100f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b2d9d640 1 --2- 172.21.15.98:0/1261407287 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71b41036c0 0x7f71b4100f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b2d9d640 1 -- 172.21.15.98:0/1261407287 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b41a6880 con 0x7f71b4102da0 2024-09-19T08:04:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b2d9d640 1 --2- 172.21.15.98:0/1261407287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b4102da0 0x7f71b4100a00 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f71a400d950 tx=0x7f71a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b0d99640 1 -- 172.21.15.98:0/1261407287 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a4017070 con 0x7f71b4102da0 2024-09-19T08:04:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b0d99640 1 -- 172.21.15.98:0/1261407287 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71a4009c30 con 0x7f71b4102da0 2024-09-19T08:04:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.976+0000 7f71b922b640 1 -- 172.21.15.98:0/1261407287 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71b41a6b70 con 0x7f71b4102da0 2024-09-19T08:04:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.977+0000 7f71b0d99640 1 -- 172.21.15.98:0/1261407287 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a400f6b0 con 0x7f71b4102da0 2024-09-19T08:04:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.977+0000 7f71b922b640 1 -- 172.21.15.98:0/1261407287 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71b41a7100 con 0x7f71b4102da0 2024-09-19T08:04:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.979+0000 7f71b0d99640 1 -- 172.21.15.98:0/1261407287 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f71a40109d0 con 0x7f71b4102da0 2024-09-19T08:04:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.979+0000 7f71bb604740 1 Processor -- start 2024-09-19T08:04:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.979+0000 7f71bb604740 1 -- start start 2024-09-19T08:04:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71bb604740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca98594a0 0x55eca9859880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71b0d99640 1 --2- 172.21.15.98:0/1261407287 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f71a00a3430 0x7f71a00a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71bb604740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca9859dc0 0x55eca9864140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71bb604740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9864680 0x55eca9866a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71bb604740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eca97bf2e0 con 0x55eca9859dc0 2024-09-19T08:04:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71bb604740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55eca9766ce0 con 0x55eca9864680 2024-09-19T08:04:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71bb604740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55eca97bb2a0 con 0x55eca98594a0 2024-09-19T08:04:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71abfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca9859dc0 0x55eca9864140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71b2d9d640 1 --2- 172.21.15.98:0/1261407287 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f71a00a3430 0x7f71a00a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71abfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca9859dc0 0x55eca9864140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50544/0 (socket says 172.21.15.98:50544) 2024-09-19T08:04:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71abfff640 1 -- 172.21.15.98:0/3851271073 learned_addr learned my addr 172.21.15.98:0/3851271073 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71b359e640 1 --2- 172.21.15.98:0/3851271073 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9864680 0x55eca9866a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71b0d99640 1 -- 172.21.15.98:0/1261407287 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f71a40cf7d0 con 0x7f71b4102da0 2024-09-19T08:04:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71b359e640 1 --2- 172.21.15.98:0/3851271073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca98594a0 0x55eca9859880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71abfff640 1 -- 172.21.15.98:0/3851271073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca98594a0 msgr2=0x55eca9859880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71abfff640 1 --2- 172.21.15.98:0/3851271073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca98594a0 0x55eca9859880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71abfff640 1 -- 172.21.15.98:0/3851271073 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9864680 msgr2=0x55eca9866a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71abfff640 1 --2- 172.21.15.98:0/3851271073 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9864680 0x55eca9866a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71abfff640 1 -- 172.21.15.98:0/3851271073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eca974b440 con 0x55eca9859dc0 2024-09-19T08:04:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71b359e640 1 --2- 172.21.15.98:0/3851271073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca98594a0 0x55eca9859880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71abfff640 1 --2- 172.21.15.98:0/3851271073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca9859dc0 0x55eca9864140 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f719c00bc30 tx=0x7f719c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617f1b70b9ca2c45 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.980+0000 7f71b2d9d640 1 --2- 172.21.15.98:0/1261407287 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f71a00a3430 0x7f71a00a58f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f71a409b890 tx=0x7f71a409e040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.981+0000 7f71a97fa640 1 -- 172.21.15.98:0/3851271073 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c01c480 con 0x55eca9859dc0 2024-09-19T08:04:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.981+0000 7f71a97fa640 1 -- 172.21.15.98:0/3851271073 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f719c01cb00 con 0x55eca9859dc0 2024-09-19T08:04:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.981+0000 7f71a97fa640 1 -- 172.21.15.98:0/3851271073 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c011c00 con 0x55eca9859dc0 2024-09-19T08:04:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.981+0000 7f71bb604740 1 -- 172.21.15.98:0/3851271073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca9859dc0 msgr2=0x55eca9864140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.981+0000 7f71bb604740 1 --2- 172.21.15.98:0/3851271073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca9859dc0 0x55eca9864140 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f719c00bc30 tx=0x7f719c0107a0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.981+0000 7f71bb604740 1 -- 172.21.15.98:0/3851271073 shutdown_connections 2024-09-19T08:04:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.981+0000 7f71bb604740 1 --2- 172.21.15.98:0/3851271073 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9864680 0x55eca9866a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.981+0000 7f71bb604740 1 --2- 172.21.15.98:0/3851271073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca9859dc0 0x55eca9864140 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.981+0000 7f71bb604740 1 --2- 172.21.15.98:0/3851271073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca98594a0 0x55eca9859880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.981+0000 7f71bb604740 1 -- 172.21.15.98:0/3851271073 >> 172.21.15.98:0/3851271073 conn(0x55eca9855a30 msgr2=0x55eca9855e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.981+0000 7f71bb604740 1 -- 172.21.15.98:0/3851271073 shutdown_connections 2024-09-19T08:04:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.981+0000 7f71bb604740 1 -- 172.21.15.98:0/3851271073 wait complete. 2024-09-19T08:04:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.982+0000 7f71bb604740 1 -- start start 2024-09-19T08:04:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.982+0000 7f71bb604740 1 Processor -- start 2024-09-19T08:04:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.982+0000 7f71bb604740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca98594a0 0x55eca9863f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.982+0000 7f71bb604740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9859dc0 0x55eca98611d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.982+0000 7f71bb604740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca9864680 0x55eca9861710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.982+0000 7f71bb604740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eca9867250 con 0x55eca98594a0 2024-09-19T08:04:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.982+0000 7f71bb604740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55eca977a050 con 0x55eca9859dc0 2024-09-19T08:04:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.982+0000 7f71bb604740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55eca98673d0 con 0x55eca9864680 2024-09-19T08:04:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.982+0000 7f71bb604740 10 client.0 _ll_register_callbacks cb 0x55eca97ba160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.982+0000 7f71b359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca98594a0 0x55eca9863f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.982+0000 7f71b359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca98594a0 0x55eca9863f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50552/0 (socket says 172.21.15.98:50552) 2024-09-19T08:04:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.982+0000 7f71b359e640 1 -- 172.21.15.98:0/487846194 learned_addr learned my addr 172.21.15.98:0/487846194 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71b359e640 1 -- 172.21.15.98:0/487846194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca9864680 msgr2=0x55eca9861710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71b359e640 1 --2- 172.21.15.98:0/487846194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca9864680 0x55eca9861710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71b359e640 1 -- 172.21.15.98:0/487846194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9859dc0 msgr2=0x55eca98611d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71b359e640 1 --2- 172.21.15.98:0/487846194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9859dc0 0x55eca98611d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71b359e640 1 -- 172.21.15.98:0/487846194 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eca95303c0 con 0x55eca98594a0 2024-09-19T08:04:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71b359e640 1 --2- 172.21.15.98:0/487846194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca98594a0 0x55eca9863f20 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f71ac006010 tx=0x7f71ac009940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a0e43133480832 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f719a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f719a7fc640 1 -- 172.21.15.98:0/487846194 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac04a060 con 0x55eca98594a0 2024-09-19T08:04:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f719a7fc640 1 -- 172.21.15.98:0/487846194 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f71ac003b70 con 0x55eca98594a0 2024-09-19T08:04:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71bb604740 1 -- 172.21.15.98:0/487846194 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f719c01cb00 con 0x55eca98594a0 2024-09-19T08:04:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f719a7fc640 1 -- 172.21.15.98:0/487846194 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f71ac034040 con 0x55eca98594a0 2024-09-19T08:04:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71bb604740 10 client.15960 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71bb604740 1 -- 172.21.15.98:0/487846194 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eca9872b10 con 0x55eca98594a0 2024-09-19T08:04:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f719a7fc640 1 -- 172.21.15.98:0/487846194 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f719c01cb00 con 0x55eca98594a0 2024-09-19T08:04:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f719a7fc640 1 -- 172.21.15.98:0/487846194 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f71ac030560 con 0x55eca98594a0 2024-09-19T08:04:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71bb604740 10 client.15960 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71bb604740 10 client.15960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:04:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71bb604740 20 client.15960 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71bb604740 1 --2- 172.21.15.98:0/487846194 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55eca9873fd0 0x55eca9876460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71bb604740 4 client.15960 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:04:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.983+0000 7f71bb604740 1 -- 172.21.15.98:0/487846194 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eca97cc850 con 0x55eca9873fd0 2024-09-19T08:04:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.984+0000 7f71abfff640 1 --2- 172.21.15.98:0/487846194 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55eca9873fd0 0x55eca9876460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.984+0000 7f719a7fc640 10 client.15960 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.984+0000 7f71abfff640 1 --2- 172.21.15.98:0/487846194 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55eca9873fd0 0x55eca9876460 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=140fc4725e15b6d4 server_cookie=3ab8247db3e7c5f4 in_seq=0 out_seq=0 2024-09-19T08:04:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f719a7fc640 1 -- 172.21.15.98:0/487846194 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55eca97cc850 con 0x55eca9873fd0 2024-09-19T08:04:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f719a7fc640 10 client.15960 handle_command_reply: tid=0 2024-09-19T08:04:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f71bb604740 1 client.15960 shutdown 2024-09-19T08:04:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f71bb604740 1 -- 172.21.15.98:0/487846194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca98594a0 msgr2=0x55eca9863f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f71bb604740 1 --2- 172.21.15.98:0/487846194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca98594a0 0x55eca9863f20 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f71ac006010 tx=0x7f71ac009940 comp rx=0 tx=0).stop 2024-09-19T08:04:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f71bb604740 1 -- 172.21.15.98:0/487846194 shutdown_connections 2024-09-19T08:04:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f71bb604740 1 --2- 172.21.15.98:0/487846194 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55eca9873fd0 0x55eca9876460 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f71bb604740 1 --2- 172.21.15.98:0/487846194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca9864680 0x55eca9861710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f71bb604740 1 --2- 172.21.15.98:0/487846194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9859dc0 0x55eca98611d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f71bb604740 1 --2- 172.21.15.98:0/487846194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca98594a0 0x55eca9863f20 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f719a7fc640 0 client.15960 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f71bb604740 1 -- 172.21.15.98:0/487846194 >> 172.21.15.98:0/487846194 conn(0x55eca9855a30 msgr2=0x55eca985b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:42.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f71bb604740 1 -- 172.21.15.98:0/487846194 shutdown_connections 2024-09-19T08:04:42.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f71bb604740 1 -- 172.21.15.98:0/487846194 wait complete. 2024-09-19T08:04:42.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:42.985+0000 7f71bb604740 20 client.15960 trim_cache size 0 max 16384 2024-09-19T08:04:43.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71bb604740 1 Processor -- start 2024-09-19T08:04:43.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71bb604740 1 -- start start 2024-09-19T08:04:43.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71bb604740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca98594a0 0x55eca985faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:43.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71bb604740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9859dc0 0x55eca987bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:43.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71bb604740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca9864680 0x55eca987c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:43.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71bb604740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f71b41afa30 con 0x55eca98594a0 2024-09-19T08:04:43.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71bb604740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f71b41b1570 con 0x55eca9859dc0 2024-09-19T08:04:43.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71bb604740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71b41b0100 con 0x55eca9864680 2024-09-19T08:04:43.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71b359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca98594a0 0x55eca985faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:43.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71b359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca98594a0 0x55eca985faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50558/0 (socket says 172.21.15.98:50558) 2024-09-19T08:04:43.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71b359e640 1 -- 172.21.15.98:0/188506204 learned_addr learned my addr 172.21.15.98:0/188506204 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:43.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71b359e640 1 --2- 172.21.15.98:0/188506204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca9864680 0x55eca987c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:43.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71b359e640 1 -- 172.21.15.98:0/188506204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca9864680 msgr2=0x55eca987c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:43.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71b359e640 1 --2- 172.21.15.98:0/188506204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca9864680 0x55eca987c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71b359e640 1 -- 172.21.15.98:0/188506204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9859dc0 msgr2=0x55eca987bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:43.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.003+0000 7f71b359e640 1 --2- 172.21.15.98:0/188506204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9859dc0 0x55eca987bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.004+0000 7f71b359e640 1 -- 172.21.15.98:0/188506204 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eca9860180 con 0x55eca98594a0 2024-09-19T08:04:43.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.004+0000 7f71b359e640 1 --2- 172.21.15.98:0/188506204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca98594a0 0x55eca985faa0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009ca0 tx=0x7f71ac02f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abd915fdc31db7c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:43.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.004+0000 7f71a8ff9640 1 -- 172.21.15.98:0/188506204 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac04a040 con 0x55eca98594a0 2024-09-19T08:04:43.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.004+0000 7f71a8ff9640 1 -- 172.21.15.98:0/188506204 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71ac003e50 con 0x55eca98594a0 2024-09-19T08:04:43.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.004+0000 7f71a8ff9640 1 -- 172.21.15.98:0/188506204 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac004340 con 0x55eca98594a0 2024-09-19T08:04:43.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.005+0000 7f71bb604740 1 -- 172.21.15.98:0/188506204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca98594a0 msgr2=0x55eca985faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:43.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.005+0000 7f71bb604740 1 --2- 172.21.15.98:0/188506204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca98594a0 0x55eca985faa0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009ca0 tx=0x7f71ac02f8a0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.005+0000 7f71bb604740 1 -- 172.21.15.98:0/188506204 shutdown_connections 2024-09-19T08:04:43.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.005+0000 7f71bb604740 1 --2- 172.21.15.98:0/188506204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca9864680 0x55eca987c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.005+0000 7f71bb604740 1 --2- 172.21.15.98:0/188506204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9859dc0 0x55eca987bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.005+0000 7f71bb604740 1 --2- 172.21.15.98:0/188506204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca98594a0 0x55eca985faa0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.005+0000 7f71bb604740 1 -- 172.21.15.98:0/188506204 >> 172.21.15.98:0/188506204 conn(0x55eca9855a30 msgr2=0x55eca9854b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:43.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.005+0000 7f71bb604740 1 -- 172.21.15.98:0/188506204 shutdown_connections 2024-09-19T08:04:43.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.005+0000 7f71bb604740 1 -- 172.21.15.98:0/188506204 wait complete. 2024-09-19T08:04:43.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.005+0000 7f71bb604740 1 -- start start 2024-09-19T08:04:43.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.005+0000 7f71bb604740 1 Processor -- start 2024-09-19T08:04:43.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71bb604740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca98594a0 0x55eca986a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:43.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71bb604740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9859dc0 0x55eca986aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:43.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71bb604740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca9864680 0x55eca986af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:43.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71bb604740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f71b41b0ea0 con 0x55eca9864680 2024-09-19T08:04:43.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71bb604740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f71b41b07d0 con 0x55eca9859dc0 2024-09-19T08:04:43.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71bb604740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71b41ada10 con 0x55eca98594a0 2024-09-19T08:04:43.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71bb604740 10 client.0 _ll_register_callbacks cb 0x7f71ac009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:43.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71b359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca9864680 0x55eca986af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:43.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71b359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca9864680 0x55eca986af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50574/0 (socket says 172.21.15.98:50574) 2024-09-19T08:04:43.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71b359e640 1 -- 172.21.15.98:0/544553195 learned_addr learned my addr 172.21.15.98:0/544553195 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:43.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71abfff640 1 --2- 172.21.15.98:0/544553195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9859dc0 0x55eca986aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:43.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71b359e640 1 --2- 172.21.15.98:0/544553195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca98594a0 0x55eca986a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:43.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71b359e640 1 -- 172.21.15.98:0/544553195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca98594a0 msgr2=0x55eca986a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:43.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71b359e640 1 --2- 172.21.15.98:0/544553195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca98594a0 0x55eca986a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71b359e640 1 -- 172.21.15.98:0/544553195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9859dc0 msgr2=0x55eca986aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:43.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71b359e640 1 --2- 172.21.15.98:0/544553195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9859dc0 0x55eca986aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71b359e640 1 -- 172.21.15.98:0/544553195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eca98ddde0 con 0x55eca9864680 2024-09-19T08:04:43.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f71b359e640 1 --2- 172.21.15.98:0/544553195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca9864680 0x55eca986af60 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f71ac005e30 tx=0x7f71ac03c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d795a50f64bc129b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:43.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f719a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:43.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.006+0000 7f719a7fc640 1 -- 172.21.15.98:0/544553195 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac004120 con 0x55eca9864680 2024-09-19T08:04:43.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.007+0000 7f719a7fc640 1 -- 172.21.15.98:0/544553195 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f71ac0042c0 con 0x55eca9864680 2024-09-19T08:04:43.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.007+0000 7f71bb604740 1 -- 172.21.15.98:0/544553195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71ac003e50 con 0x55eca9864680 2024-09-19T08:04:43.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.007+0000 7f719a7fc640 1 -- 172.21.15.98:0/544553195 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f71ac0044c0 con 0x55eca9864680 2024-09-19T08:04:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.007+0000 7f71bb604740 10 client.15966 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.007+0000 7f71bb604740 1 -- 172.21.15.98:0/544553195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eca98df0d0 con 0x55eca9864680 2024-09-19T08:04:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.007+0000 7f719a7fc640 1 -- 172.21.15.98:0/544553195 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71ac003e50 con 0x55eca9864680 2024-09-19T08:04:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.007+0000 7f719a7fc640 1 -- 172.21.15.98:0/544553195 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f71ac066060 con 0x55eca9864680 2024-09-19T08:04:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.007+0000 7f71bb604740 10 client.15966 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.007+0000 7f71bb604740 10 client.15966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:04:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.007+0000 7f71bb604740 20 client.15966 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.007+0000 7f71bb604740 1 --2- 172.21.15.98:0/544553195 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55eca98df4b0 0x55eca98e18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.007+0000 7f71bb604740 4 client.15966 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:04:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.007+0000 7f71bb604740 1 -- 172.21.15.98:0/544553195 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55eca97c6d60 con 0x55eca98df4b0 2024-09-19T08:04:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.007+0000 7f71abfff640 1 --2- 172.21.15.98:0/544553195 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55eca98df4b0 0x55eca98e18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f71abfff640 1 --2- 172.21.15.98:0/544553195 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55eca98df4b0 0x55eca98e18a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cb1edd438249724 server_cookie=cb30f38848c85e39 in_seq=0 out_seq=0 2024-09-19T08:04:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f719a7fc640 10 client.15966 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f719a7fc640 1 -- 172.21.15.98:0/544553195 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55eca97c6d60 con 0x55eca98df4b0 2024-09-19T08:04:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f719a7fc640 10 client.15966 handle_command_reply: tid=0 2024-09-19T08:04:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f71bb604740 1 client.15966 shutdown 2024-09-19T08:04:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f71bb604740 1 -- 172.21.15.98:0/544553195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca9864680 msgr2=0x55eca986af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f71bb604740 1 --2- 172.21.15.98:0/544553195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca9864680 0x55eca986af60 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f71ac005e30 tx=0x7f71ac03c040 comp rx=0 tx=0).stop 2024-09-19T08:04:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f71bb604740 1 -- 172.21.15.98:0/544553195 shutdown_connections 2024-09-19T08:04:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f71bb604740 1 --2- 172.21.15.98:0/544553195 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55eca98df4b0 0x55eca98e18a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f71bb604740 1 --2- 172.21.15.98:0/544553195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eca9864680 0x55eca986af60 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f71bb604740 1 --2- 172.21.15.98:0/544553195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eca9859dc0 0x55eca986aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f71bb604740 1 --2- 172.21.15.98:0/544553195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eca98594a0 0x55eca986a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f71bb604740 1 -- 172.21.15.98:0/544553195 >> 172.21.15.98:0/544553195 conn(0x55eca9855a30 msgr2=0x55eca9862a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f719a7fc640 0 client.15966 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f71bb604740 1 -- 172.21.15.98:0/544553195 shutdown_connections 2024-09-19T08:04:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f71bb604740 1 -- 172.21.15.98:0/544553195 wait complete. 2024-09-19T08:04:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.008+0000 7f71bb604740 20 client.15966 trim_cache size 0 max 16384 2024-09-19T08:04:43.016 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:04:43.016 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-19T08:04:43.016 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-09-19T08:04:43.016 INFO:teuthology.orchestra.run.smithi098.stdout: "e65644f3-2efe-49b2-83c9-06e3cc521893": { 2024-09-19T08:04:43.016 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-09-19T08:04:43.016 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "e65644f3-2efe-49b2-83c9-06e3cc521893", 2024-09-19T08:04:43.016 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-09-19T08:04:43.016 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:43.016 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:43.017 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:04:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.009+0000 7f71b922b640 1 -- 172.21.15.98:0/1261407287 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f71a00a3430 msgr2=0x7f71a00a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.009+0000 7f71b922b640 1 --2- 172.21.15.98:0/1261407287 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f71a00a3430 0x7f71a00a58f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f71a409b890 tx=0x7f71a409e040 comp rx=0 tx=0).stop 2024-09-19T08:04:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.009+0000 7f71b922b640 1 -- 172.21.15.98:0/1261407287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b4102da0 msgr2=0x7f71b4100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.009+0000 7f71b922b640 1 --2- 172.21.15.98:0/1261407287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b4102da0 0x7f71b4100a00 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f71a400d950 tx=0x7f71a400de10 comp rx=0 tx=0).stop 2024-09-19T08:04:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.009+0000 7f71b922b640 1 -- 172.21.15.98:0/1261407287 shutdown_connections 2024-09-19T08:04:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.009+0000 7f71b922b640 1 --2- 172.21.15.98:0/1261407287 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f71a00a3430 0x7f71a00a58f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.009+0000 7f71b922b640 1 --2- 172.21.15.98:0/1261407287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71b410c180 0x7f71b4101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.009+0000 7f71b922b640 1 --2- 172.21.15.98:0/1261407287 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71b41036c0 0x7f71b4100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.009+0000 7f71b922b640 1 --2- 172.21.15.98:0/1261407287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b4102da0 0x7f71b4100a00 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.009+0000 7f71b922b640 1 -- 172.21.15.98:0/1261407287 >> 172.21.15.98:0/1261407287 conn(0x7f71b40fc7a0 msgr2=0x7f71b40fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.009+0000 7f71b922b640 1 -- 172.21.15.98:0/1261407287 shutdown_connections 2024-09-19T08:04:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:43.009+0000 7f71b922b640 1 -- 172.21.15.98:0/1261407287 wait complete. 2024-09-19T08:04:43.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:43 smithi162 ceph-mon[35260]: pgmap v54: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 343 KiB/s wr, 14 op/s 2024-09-19T08:04:43.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:04:43.917 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:43 smithi098 ceph-mon[31333]: pgmap v54: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 343 KiB/s wr, 14 op/s 2024-09-19T08:04:43.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:43 smithi139 ceph-mon[34874]: pgmap v54: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 343 KiB/s wr, 14 op/s 2024-09-19T08:04:43.988 INFO:tasks.cephfs.filesystem:scrub status for tag:e65644f3-2efe-49b2-83c9-06e3cc521893 - {'path': '/', 'tag': 'e65644f3-2efe-49b2-83c9-06e3cc521893', 'options': 'recursive,force'} 2024-09-19T08:04:44.146 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:44.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.590+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/500799112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc08100e20 msgr2=0x7fcc08101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:44.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.590+0000 7fcc0f2fd640 1 --2- 172.21.15.98:0/500799112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc08100e20 0x7fcc08101200 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcc00009900 tx=0x7fcc0002a480 comp rx=0 tx=0).stop 2024-09-19T08:04:44.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.591+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/500799112 shutdown_connections 2024-09-19T08:04:44.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.591+0000 7fcc0f2fd640 1 --2- 172.21.15.98:0/500799112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc08102ef0 0x7fcc0810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:44.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.591+0000 7fcc0f2fd640 1 --2- 172.21.15.98:0/500799112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc08102570 0x7fcc081029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:44.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.591+0000 7fcc0f2fd640 1 --2- 172.21.15.98:0/500799112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc08100e20 0x7fcc08101200 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.591+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/500799112 >> 172.21.15.98:0/500799112 conn(0x7fcc080fc7d0 msgr2=0x7fcc080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.591+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/500799112 shutdown_connections 2024-09-19T08:04:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.591+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/500799112 wait complete. 2024-09-19T08:04:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.591+0000 7fcc0f2fd640 1 Processor -- start 2024-09-19T08:04:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.591+0000 7fcc0f2fd640 1 -- start start 2024-09-19T08:04:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0f2fd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc08100e20 0x7fcc081a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0f2fd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc08102570 0x7fcc081a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0f2fd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc08102ef0 0x7fcc0819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0f2fd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc08111020 con 0x7fcc08102ef0 2024-09-19T08:04:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0f2fd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcc08110ea0 con 0x7fcc08102570 2024-09-19T08:04:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0f2fd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcc081111a0 con 0x7fcc08100e20 2024-09-19T08:04:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0d89a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc08102ef0 0x7fcc0819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0d89a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc08102ef0 0x7fcc0819a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50588/0 (socket says 172.21.15.98:50588) 2024-09-19T08:04:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0d89a640 1 -- 172.21.15.98:0/3912056599 learned_addr learned my addr 172.21.15.98:0/3912056599 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0d099640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc08100e20 0x7fcc081a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0c898640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc08102570 0x7fcc081a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0d89a640 1 -- 172.21.15.98:0/3912056599 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc08100e20 msgr2=0x7fcc081a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0d89a640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc08100e20 0x7fcc081a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0d89a640 1 -- 172.21.15.98:0/3912056599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc08102570 msgr2=0x7fcc081a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0d89a640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc08102570 0x7fcc081a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0d89a640 1 -- 172.21.15.98:0/3912056599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0819af80 con 0x7fcc08102ef0 2024-09-19T08:04:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0c898640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc08102570 0x7fcc081a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.592+0000 7fcc0d89a640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc08102ef0 0x7fcc0819a750 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009fd0 tx=0x7fcbf800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.593+0000 7fcc067fc640 1 -- 172.21.15.98:0/3912056599 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8011830 con 0x7fcc08102ef0 2024-09-19T08:04:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.593+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/3912056599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0819b270 con 0x7fcc08102ef0 2024-09-19T08:04:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.593+0000 7fcc067fc640 1 -- 172.21.15.98:0/3912056599 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbf8016070 con 0x7fcc08102ef0 2024-09-19T08:04:44.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.593+0000 7fcc067fc640 1 -- 172.21.15.98:0/3912056599 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf800f2e0 con 0x7fcc08102ef0 2024-09-19T08:04:44.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.593+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/3912056599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc081a7410 con 0x7fcc08102ef0 2024-09-19T08:04:44.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.594+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/3912056599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc08101280 con 0x7fcc08102ef0 2024-09-19T08:04:44.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.595+0000 7fcc067fc640 1 -- 172.21.15.98:0/3912056599 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fcbf80119d0 con 0x7fcc08102ef0 2024-09-19T08:04:44.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.595+0000 7fcc067fc640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcbf00a33b0 0x7fcbf00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:44.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.595+0000 7fcc067fc640 1 -- 172.21.15.98:0/3912056599 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fcbf80cf290 con 0x7fcc08102ef0 2024-09-19T08:04:44.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.595+0000 7fcc0d099640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcbf00a33b0 0x7fcbf00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:44.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.596+0000 7fcc0d099640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcbf00a33b0 0x7fcbf00a5870 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcc00000c00 tx=0x7fcc000058d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:44.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.600+0000 7fcc067fc640 1 -- 172.21.15.98:0/3912056599 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fcbf809bdb0 con 0x7fcc08102ef0 2024-09-19T08:04:44.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.803+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/3912056599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc0819bc60 con 0x7fcc08102ef0 2024-09-19T08:04:44.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.804+0000 7fcc067fc640 1 -- 172.21.15.98:0/3912056599 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11983 (secure 0 0 0) 0x7fcc0819bc60 con 0x7fcc08102ef0 2024-09-19T08:04:44.804 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:04:44.805 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"btime":"2024-09-19T08:02:59:855935+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:02:59.805764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:04:44.805 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-09-19T08:04:44.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.806+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/3912056599 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcbf00a33b0 msgr2=0x7fcbf00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:44.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.806+0000 7fcc0f2fd640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcbf00a33b0 0x7fcbf00a5870 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcc00000c00 tx=0x7fcc000058d0 comp rx=0 tx=0).stop 2024-09-19T08:04:44.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.806+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/3912056599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc08102ef0 msgr2=0x7fcc0819a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:44.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.806+0000 7fcc0f2fd640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc08102ef0 0x7fcc0819a750 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009fd0 tx=0x7fcbf800b8e0 comp rx=0 tx=0).stop 2024-09-19T08:04:44.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.806+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/3912056599 shutdown_connections 2024-09-19T08:04:44.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.806+0000 7fcc0f2fd640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcbf00a33b0 0x7fcbf00a5870 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:44.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.806+0000 7fcc0f2fd640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc08102ef0 0x7fcc0819a750 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:44.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.806+0000 7fcc0f2fd640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc08102570 0x7fcc081a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:44.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.806+0000 7fcc0f2fd640 1 --2- 172.21.15.98:0/3912056599 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc08100e20 0x7fcc081a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:44.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.806+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/3912056599 >> 172.21.15.98:0/3912056599 conn(0x7fcc080fc7d0 msgr2=0x7fcc0806a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:44.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.806+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/3912056599 shutdown_connections 2024-09-19T08:04:44.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:44.806+0000 7fcc0f2fd640 1 -- 172.21.15.98:0/3912056599 wait complete. 2024-09-19T08:04:45.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:44 smithi162 ceph-mon[35260]: pgmap v55: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 938 B/s wr, 9 op/s 2024-09-19T08:04:45.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:44 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3912056599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:45.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:44 smithi139 ceph-mon[34874]: pgmap v55: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 938 B/s wr, 9 op/s 2024-09-19T08:04:45.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:44 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3912056599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:45.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:44 smithi098 ceph-mon[31333]: pgmap v55: 97 pgs: 97 active+clean; 5.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 938 B/s wr, 9 op/s 2024-09-19T08:04:45.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:44 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3912056599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:45.480 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id 28e1fa50 --release --await' 2024-09-19T08:04:45.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id 28e1fa50 --release --await 2024-09-19T08:04:45.727 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:46.114 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:45 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:04:46.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:45 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:04:46.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:45 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:04:46.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.199+0000 7ff855ccc640 1 -- 172.21.15.98:0/3392018332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff850105e80 msgr2=0x7ff85010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:46.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.199+0000 7ff855ccc640 1 --2- 172.21.15.98:0/3392018332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff850105e80 0x7ff85010da40 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7ff848009ca0 tx=0x7ff84802a8c0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.199+0000 7ff855ccc640 1 -- 172.21.15.98:0/3392018332 shutdown_connections 2024-09-19T08:04:46.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.199+0000 7ff855ccc640 1 --2- 172.21.15.98:0/3392018332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff850105e80 0x7ff85010da40 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.199+0000 7ff855ccc640 1 --2- 172.21.15.98:0/3392018332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff850101e30 0x7ff850105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.199+0000 7ff855ccc640 1 --2- 172.21.15.98:0/3392018332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff850101480 0x7ff850101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.199+0000 7ff855ccc640 1 -- 172.21.15.98:0/3392018332 >> 172.21.15.98:0/3392018332 conn(0x7ff850077210 msgr2=0x7ff8500fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.199+0000 7ff855ccc640 1 -- 172.21.15.98:0/3392018332 shutdown_connections 2024-09-19T08:04:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.199+0000 7ff855ccc640 1 -- 172.21.15.98:0/3392018332 wait complete. 2024-09-19T08:04:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.199+0000 7ff855ccc640 1 Processor -- start 2024-09-19T08:04:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff855ccc640 1 -- start start 2024-09-19T08:04:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff855ccc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff850101480 0x7ff8501985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff855ccc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff850101e30 0x7ff850198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff855ccc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff850105e80 0x7ff85019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff855ccc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff850110fc0 con 0x7ff850101480 2024-09-19T08:04:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff855ccc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff850110e40 con 0x7ff850101e30 2024-09-19T08:04:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff855ccc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff850111140 con 0x7ff850105e80 2024-09-19T08:04:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff84f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff850101480 0x7ff8501985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff84f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff850101480 0x7ff8501985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50600/0 (socket says 172.21.15.98:50600) 2024-09-19T08:04:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff84f7fe640 1 -- 172.21.15.98:0/280535729 learned_addr learned my addr 172.21.15.98:0/280535729 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff84effd640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff850101e30 0x7ff850198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff84ffff640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff850105e80 0x7ff85019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff84f7fe640 1 -- 172.21.15.98:0/280535729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff850105e80 msgr2=0x7ff85019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff84f7fe640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff850105e80 0x7ff85019cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff84f7fe640 1 -- 172.21.15.98:0/280535729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff850101e30 msgr2=0x7ff850198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff84f7fe640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff850101e30 0x7ff850198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff84f7fe640 1 -- 172.21.15.98:0/280535729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85019d580 con 0x7ff850101480 2024-09-19T08:04:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff84ffff640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff850105e80 0x7ff85019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:04:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.200+0000 7ff84f7fe640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff850101480 0x7ff8501985d0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff84000b4a0 tx=0x7ff84000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.201+0000 7ff84cff9640 1 -- 172.21.15.98:0/280535729 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff840015020 con 0x7ff850101480 2024-09-19T08:04:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.201+0000 7ff84cff9640 1 -- 172.21.15.98:0/280535729 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff840011d10 con 0x7ff850101480 2024-09-19T08:04:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.201+0000 7ff855ccc640 1 -- 172.21.15.98:0/280535729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff85019d870 con 0x7ff850101480 2024-09-19T08:04:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.201+0000 7ff84cff9640 1 -- 172.21.15.98:0/280535729 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84000f2e0 con 0x7ff850101480 2024-09-19T08:04:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.201+0000 7ff855ccc640 1 -- 172.21.15.98:0/280535729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8501a5150 con 0x7ff850101480 2024-09-19T08:04:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.204+0000 7ff8580a5740 1 Processor -- start 2024-09-19T08:04:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.204+0000 7ff84cff9640 1 -- 172.21.15.98:0/280535729 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7ff840018020 con 0x7ff850101480 2024-09-19T08:04:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.204+0000 7ff8580a5740 1 -- start start 2024-09-19T08:04:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.204+0000 7ff8580a5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb060 0x55fee0fcb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.204+0000 7ff8580a5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb980 0x55fee0fd5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.204+0000 7ff8580a5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 0x55fee0fd8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.204+0000 7ff8580a5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fee0f30620 con 0x55fee0fcb060 2024-09-19T08:04:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.204+0000 7ff8580a5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fee0ed7e50 con 0x55fee0fd6240 2024-09-19T08:04:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.204+0000 7ff8580a5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fee0f2c5e0 con 0x55fee0fcb980 2024-09-19T08:04:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.204+0000 7ff84ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb060 0x55fee0fcb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.204+0000 7ff84ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb060 0x55fee0fcb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50612/0 (socket says 172.21.15.98:50612) 2024-09-19T08:04:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.204+0000 7ff84ffff640 1 -- 172.21.15.98:0/2798077557 learned_addr learned my addr 172.21.15.98:0/2798077557 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.204+0000 7ff84effd640 1 --2- 172.21.15.98:0/2798077557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 0x55fee0fd8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.204+0000 7ff84f7fe640 1 --2- 172.21.15.98:0/2798077557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb980 0x55fee0fd5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.205+0000 7ff84ffff640 1 -- 172.21.15.98:0/2798077557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb980 msgr2=0x55fee0fd5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:46.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.205+0000 7ff84ffff640 1 --2- 172.21.15.98:0/2798077557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb980 0x55fee0fd5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.205+0000 7ff84ffff640 1 -- 172.21.15.98:0/2798077557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 msgr2=0x55fee0fd8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:46.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.205+0000 7ff84ffff640 1 --2- 172.21.15.98:0/2798077557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 0x55fee0fd8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.205+0000 7ff84ffff640 1 -- 172.21.15.98:0/2798077557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fee0ec8d40 con 0x55fee0fcb060 2024-09-19T08:04:46.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.205+0000 7ff84effd640 1 --2- 172.21.15.98:0/2798077557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 0x55fee0fd8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:04:46.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.205+0000 7ff84cff9640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff8380a33b0 0x7ff8380a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.205+0000 7ff84ffff640 1 --2- 172.21.15.98:0/2798077557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb060 0x55fee0fcb440 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ff848009940 tx=0x7ff848009970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec003c8c574ba23 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:46.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.205+0000 7ff84f7fe640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff8380a33b0 0x7ff8380a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.205+0000 7ff84cff9640 1 -- 172.21.15.98:0/280535729 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff8400cf4b0 con 0x7ff850101480 2024-09-19T08:04:46.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.205+0000 7ff835ffb640 1 -- 172.21.15.98:0/2798077557 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff848003970 con 0x55fee0fcb060 2024-09-19T08:04:46.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.205+0000 7ff835ffb640 1 -- 172.21.15.98:0/2798077557 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff848036050 con 0x55fee0fcb060 2024-09-19T08:04:46.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.205+0000 7ff835ffb640 1 -- 172.21.15.98:0/2798077557 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8480044c0 con 0x55fee0fcb060 2024-09-19T08:04:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.206+0000 7ff8580a5740 1 -- 172.21.15.98:0/2798077557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb060 msgr2=0x55fee0fcb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.206+0000 7ff84f7fe640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff8380a33b0 0x7ff8380a5870 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff8400ccdf0 tx=0x7ff8400d5060 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.206+0000 7ff8580a5740 1 --2- 172.21.15.98:0/2798077557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb060 0x55fee0fcb440 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ff848009940 tx=0x7ff848009970 comp rx=0 tx=0).stop 2024-09-19T08:04:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.206+0000 7ff8580a5740 1 -- 172.21.15.98:0/2798077557 shutdown_connections 2024-09-19T08:04:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.206+0000 7ff8580a5740 1 --2- 172.21.15.98:0/2798077557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 0x55fee0fd8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.206+0000 7ff8580a5740 1 --2- 172.21.15.98:0/2798077557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb980 0x55fee0fd5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.206+0000 7ff8580a5740 1 --2- 172.21.15.98:0/2798077557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb060 0x55fee0fcb440 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.206+0000 7ff8580a5740 1 -- 172.21.15.98:0/2798077557 >> 172.21.15.98:0/2798077557 conn(0x55fee0fc75f0 msgr2=0x55fee0fc79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.206+0000 7ff8580a5740 1 -- 172.21.15.98:0/2798077557 shutdown_connections 2024-09-19T08:04:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.206+0000 7ff8580a5740 1 -- 172.21.15.98:0/2798077557 wait complete. 2024-09-19T08:04:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.206+0000 7ff8580a5740 1 -- start start 2024-09-19T08:04:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff8580a5740 1 Processor -- start 2024-09-19T08:04:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff8580a5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb060 0x55fee0fd31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff8580a5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb980 0x55fee0fd36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff8580a5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 0x55fee0fd3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff8580a5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fee0fd8c90 con 0x55fee0fcb980 2024-09-19T08:04:46.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff8580a5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fee0eb8f10 con 0x55fee0fd6240 2024-09-19T08:04:46.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff8580a5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fee0fd8e10 con 0x55fee0fcb060 2024-09-19T08:04:46.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff8580a5740 10 client.0 _ll_register_callbacks cb 0x55fee0edcdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:46.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff84effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb980 0x55fee0fd36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff84effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb980 0x55fee0fd36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50622/0 (socket says 172.21.15.98:50622) 2024-09-19T08:04:46.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff84effd640 1 -- 172.21.15.98:0/1014612597 learned_addr learned my addr 172.21.15.98:0/1014612597 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:46.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff84ffff640 1 --2- 172.21.15.98:0/1014612597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb060 0x55fee0fd31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff84ffff640 1 --2- 172.21.15.98:0/1014612597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 0x55fee0fd3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff84effd640 1 -- 172.21.15.98:0/1014612597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb060 msgr2=0x55fee0fd31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:46.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff84effd640 1 --2- 172.21.15.98:0/1014612597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb060 0x55fee0fd31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff84effd640 1 -- 172.21.15.98:0/1014612597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 msgr2=0x55fee0fd3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:46.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff84effd640 1 --2- 172.21.15.98:0/1014612597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 0x55fee0fd3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff84effd640 1 -- 172.21.15.98:0/1014612597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fee0c3f580 con 0x55fee0fcb980 2024-09-19T08:04:46.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff84ffff640 1 --2- 172.21.15.98:0/1014612597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 0x55fee0fd3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:46.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff84effd640 1 --2- 172.21.15.98:0/1014612597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb980 0x55fee0fd36f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ff844011570 tx=0x7ff844009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a90ca471d5d057e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:46.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.207+0000 7ff82effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:46.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.208+0000 7ff82effd640 1 -- 172.21.15.98:0/1014612597 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff844013670 con 0x55fee0fcb980 2024-09-19T08:04:46.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.208+0000 7ff82effd640 1 -- 172.21.15.98:0/1014612597 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff844013810 con 0x55fee0fcb980 2024-09-19T08:04:46.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.208+0000 7ff8580a5740 1 -- 172.21.15.98:0/1014612597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff848036050 con 0x55fee0fcb980 2024-09-19T08:04:46.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.208+0000 7ff82effd640 1 -- 172.21.15.98:0/1014612597 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff844013a10 con 0x55fee0fcb980 2024-09-19T08:04:46.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.208+0000 7ff8580a5740 10 client.15984 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:46.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.208+0000 7ff8580a5740 1 -- 172.21.15.98:0/1014612597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fee0d09960 con 0x55fee0fcb980 2024-09-19T08:04:46.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.208+0000 7ff82effd640 1 -- 172.21.15.98:0/1014612597 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff848036050 con 0x55fee0fcb980 2024-09-19T08:04:46.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.208+0000 7ff82effd640 1 -- 172.21.15.98:0/1014612597 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff844026450 con 0x55fee0fcb980 2024-09-19T08:04:46.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.208+0000 7ff8580a5740 10 client.15984 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:46.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.208+0000 7ff8580a5740 10 client.15984 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-19T08:04:46.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.208+0000 7ff8580a5740 20 client.15984 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:46.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.208+0000 7ff8580a5740 1 --2- 172.21.15.98:0/1014612597 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fee0fe5cb0 0x55fee0fe80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.208+0000 7ff8580a5740 4 client.15984 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:04:46.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.208+0000 7ff8580a5740 1 -- 172.21.15.98:0/1014612597 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fee0f3db90 con 0x55fee0fe5cb0 2024-09-19T08:04:46.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.209+0000 7ff84ffff640 1 --2- 172.21.15.98:0/1014612597 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fee0fe5cb0 0x55fee0fe80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.209+0000 7ff82effd640 10 client.15984 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:46.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.209+0000 7ff84ffff640 1 --2- 172.21.15.98:0/1014612597 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fee0fe5cb0 0x55fee0fe80d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb72c2168880ff2e server_cookie=b3747788f8d86dde in_seq=0 out_seq=0 2024-09-19T08:04:46.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff82effd640 1 -- 172.21.15.98:0/1014612597 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fee0f3db90 con 0x55fee0fe5cb0 2024-09-19T08:04:46.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff82effd640 10 client.15984 handle_command_reply: tid=0 2024-09-19T08:04:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff8580a5740 1 client.15984 shutdown 2024-09-19T08:04:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff8580a5740 1 -- 172.21.15.98:0/1014612597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb980 msgr2=0x55fee0fd36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff8580a5740 1 --2- 172.21.15.98:0/1014612597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb980 0x55fee0fd36f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ff844011570 tx=0x7ff844009f90 comp rx=0 tx=0).stop 2024-09-19T08:04:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff8580a5740 1 -- 172.21.15.98:0/1014612597 shutdown_connections 2024-09-19T08:04:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff8580a5740 1 --2- 172.21.15.98:0/1014612597 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fee0fe5cb0 0x55fee0fe80d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff8580a5740 1 --2- 172.21.15.98:0/1014612597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 0x55fee0fd3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff8580a5740 1 --2- 172.21.15.98:0/1014612597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb980 0x55fee0fd36f0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff8580a5740 1 --2- 172.21.15.98:0/1014612597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb060 0x55fee0fd31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff8580a5740 1 -- 172.21.15.98:0/1014612597 >> 172.21.15.98:0/1014612597 conn(0x55fee0fc75f0 msgr2=0x55fee0fcd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff82effd640 0 client.15984 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff8580a5740 1 -- 172.21.15.98:0/1014612597 shutdown_connections 2024-09-19T08:04:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff8580a5740 1 -- 172.21.15.98:0/1014612597 wait complete. 2024-09-19T08:04:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.210+0000 7ff8580a5740 20 client.15984 trim_cache size 0 max 16384 2024-09-19T08:04:46.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff8580a5740 1 Processor -- start 2024-09-19T08:04:46.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff8580a5740 1 -- start start 2024-09-19T08:04:46.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff8580a5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb060 0x55fee0fd2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff8580a5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb980 0x55fee0fed9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff8580a5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 0x55fee0fedee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff8580a5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8501ad870 con 0x55fee0fcb060 2024-09-19T08:04:46.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff8580a5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff8501af3b0 con 0x55fee0fd6240 2024-09-19T08:04:46.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff8580a5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff8501adf40 con 0x55fee0fcb980 2024-09-19T08:04:46.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff84ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb060 0x55fee0fd2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff84ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb060 0x55fee0fd2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50624/0 (socket says 172.21.15.98:50624) 2024-09-19T08:04:46.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff84ffff640 1 -- 172.21.15.98:0/323568122 learned_addr learned my addr 172.21.15.98:0/323568122 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:46.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff84effd640 1 --2- 172.21.15.98:0/323568122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb980 0x55fee0fed9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff84ffff640 1 -- 172.21.15.98:0/323568122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb980 msgr2=0x55fee0fed9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:46.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff84ffff640 1 --2- 172.21.15.98:0/323568122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb980 0x55fee0fed9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff84ffff640 1 -- 172.21.15.98:0/323568122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 msgr2=0x55fee0fedee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:46.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff84ffff640 1 --2- 172.21.15.98:0/323568122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 0x55fee0fedee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.227+0000 7ff84ffff640 1 -- 172.21.15.98:0/323568122 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fee0fce010 con 0x55fee0fcb060 2024-09-19T08:04:46.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.228+0000 7ff84ffff640 1 --2- 172.21.15.98:0/323568122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb060 0x55fee0fd2110 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff8480040b0 tx=0x7ff848009970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e663464f65b249e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:46.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.228+0000 7ff8357fa640 1 -- 172.21.15.98:0/323568122 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff848003970 con 0x55fee0fcb060 2024-09-19T08:04:46.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.228+0000 7ff8357fa640 1 -- 172.21.15.98:0/323568122 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff848054070 con 0x55fee0fcb060 2024-09-19T08:04:46.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.228+0000 7ff8357fa640 1 -- 172.21.15.98:0/323568122 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8480043b0 con 0x55fee0fcb060 2024-09-19T08:04:46.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.228+0000 7ff8580a5740 1 -- 172.21.15.98:0/323568122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb060 msgr2=0x55fee0fd2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:46.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.228+0000 7ff8580a5740 1 --2- 172.21.15.98:0/323568122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb060 0x55fee0fd2110 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff8480040b0 tx=0x7ff848009970 comp rx=0 tx=0).stop 2024-09-19T08:04:46.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.228+0000 7ff8580a5740 1 -- 172.21.15.98:0/323568122 shutdown_connections 2024-09-19T08:04:46.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.228+0000 7ff8580a5740 1 --2- 172.21.15.98:0/323568122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fd6240 0x55fee0fedee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.228+0000 7ff8580a5740 1 --2- 172.21.15.98:0/323568122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fcb980 0x55fee0fed9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.228+0000 7ff8580a5740 1 --2- 172.21.15.98:0/323568122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb060 0x55fee0fd2110 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.228+0000 7ff8580a5740 1 -- 172.21.15.98:0/323568122 >> 172.21.15.98:0/323568122 conn(0x55fee0fc75f0 msgr2=0x55fee0fcd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.228+0000 7ff8580a5740 1 -- 172.21.15.98:0/323568122 shutdown_connections 2024-09-19T08:04:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.228+0000 7ff8580a5740 1 -- 172.21.15.98:0/323568122 wait complete. 2024-09-19T08:04:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.229+0000 7ff8580a5740 1 -- start start 2024-09-19T08:04:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.229+0000 7ff8580a5740 1 Processor -- start 2024-09-19T08:04:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.229+0000 7ff8580a5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fcb060 0x55fee0fde060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.229+0000 7ff8580a5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb980 0x55fee0fde5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.229+0000 7ff8580a5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fd6240 0x55fee0fdb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.229+0000 7ff8580a5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8501aece0 con 0x55fee0fcb980 2024-09-19T08:04:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.229+0000 7ff8580a5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff8501ae610 con 0x55fee0fcb060 2024-09-19T08:04:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.229+0000 7ff8580a5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff8501ab850 con 0x55fee0fd6240 2024-09-19T08:04:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.229+0000 7ff8580a5740 10 client.0 _ll_register_callbacks cb 0x7ff844009e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.229+0000 7ff84effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb980 0x55fee0fde5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.229+0000 7ff84effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb980 0x55fee0fde5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50640/0 (socket says 172.21.15.98:50640) 2024-09-19T08:04:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.229+0000 7ff84effd640 1 -- 172.21.15.98:0/4165724218 learned_addr learned my addr 172.21.15.98:0/4165724218 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff84effd640 1 -- 172.21.15.98:0/4165724218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fd6240 msgr2=0x55fee0fdb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff84ffff640 1 --2- 172.21.15.98:0/4165724218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fd6240 0x55fee0fdb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff84effd640 1 --2- 172.21.15.98:0/4165724218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fd6240 0x55fee0fdb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff84effd640 1 -- 172.21.15.98:0/4165724218 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fcb060 msgr2=0x55fee0fde060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff84effd640 1 --2- 172.21.15.98:0/4165724218 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fcb060 0x55fee0fde060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff84effd640 1 -- 172.21.15.98:0/4165724218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fee0fd1110 con 0x55fee0fcb980 2024-09-19T08:04:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff84ffff640 1 --2- 172.21.15.98:0/4165724218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fd6240 0x55fee0fdb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff84effd640 1 --2- 172.21.15.98:0/4165724218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb980 0x55fee0fde5a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff84400b010 tx=0x7ff844009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb15c480b814cd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff82effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff82effd640 1 -- 172.21.15.98:0/4165724218 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff844026e60 con 0x55fee0fcb980 2024-09-19T08:04:46.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff82effd640 1 -- 172.21.15.98:0/4165724218 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff844013670 con 0x55fee0fcb980 2024-09-19T08:04:46.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff8580a5740 1 -- 172.21.15.98:0/4165724218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff848054070 con 0x55fee0fcb980 2024-09-19T08:04:46.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff82effd640 1 -- 172.21.15.98:0/4165724218 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff844026a40 con 0x55fee0fcb980 2024-09-19T08:04:46.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff8580a5740 10 client.15990 fetch_fsmap learned FSMap version 28 2024-09-19T08:04:46.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff8580a5740 1 -- 172.21.15.98:0/4165724218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fee1050ed0 con 0x55fee0fcb980 2024-09-19T08:04:46.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.230+0000 7ff82effd640 1 -- 172.21.15.98:0/4165724218 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff848054070 con 0x55fee0fcb980 2024-09-19T08:04:46.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.231+0000 7ff82effd640 1 -- 172.21.15.98:0/4165724218 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff84401d960 con 0x55fee0fcb980 2024-09-19T08:04:46.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.231+0000 7ff8580a5740 10 client.15990 fetch_fsmap finished waiting for FSMap version 28 2024-09-19T08:04:46.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.231+0000 7ff8580a5740 10 client.15990 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-19T08:04:46.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.231+0000 7ff8580a5740 20 client.15990 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:46.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.231+0000 7ff8580a5740 1 --2- 172.21.15.98:0/4165724218 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fee1051650 0x55fee1053a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:46.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.231+0000 7ff8580a5740 4 client.15990 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "28e1fa50", "release": true, "await": true}] 2024-09-19T08:04:46.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.231+0000 7ff8580a5740 1 -- 172.21.15.98:0/4165724218 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "quiesce db", "set_id": "28e1fa50", "release": true, "await": true}) -- 0x55fee0fc2950 con 0x55fee1051650 2024-09-19T08:04:46.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.231+0000 7ff84ffff640 1 --2- 172.21.15.98:0/4165724218 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fee1051650 0x55fee1053a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:46.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.231+0000 7ff84ffff640 1 --2- 172.21.15.98:0/4165724218 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fee1051650 0x55fee1053a40 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af466ce134155d04 server_cookie=fb6feefff5abea0c in_seq=0 out_seq=0 2024-09-19T08:04:46.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.231+0000 7ff82effd640 10 client.15990 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:46.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff82effd640 1 -- 172.21.15.98:0/4165724218 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55fee0fc2950 con 0x55fee1051650 2024-09-19T08:04:46.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff82effd640 10 client.15990 handle_command_reply: tid=0 2024-09-19T08:04:46.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff8580a5740 1 client.15990 shutdown 2024-09-19T08:04:46.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff8580a5740 1 -- 172.21.15.98:0/4165724218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb980 msgr2=0x55fee0fde5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:46.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff8580a5740 1 --2- 172.21.15.98:0/4165724218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb980 0x55fee0fde5a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff84400b010 tx=0x7ff844009290 comp rx=0 tx=0).stop 2024-09-19T08:04:46.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff8580a5740 1 -- 172.21.15.98:0/4165724218 shutdown_connections 2024-09-19T08:04:46.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff8580a5740 1 --2- 172.21.15.98:0/4165724218 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fee1051650 0x55fee1053a40 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff8580a5740 1 --2- 172.21.15.98:0/4165724218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fee0fd6240 0x55fee0fdb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff82effd640 0 client.15990 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:46.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff8580a5740 1 --2- 172.21.15.98:0/4165724218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fee0fcb980 0x55fee0fde5a0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff8580a5740 1 --2- 172.21.15.98:0/4165724218 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fee0fcb060 0x55fee0fde060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff8580a5740 1 -- 172.21.15.98:0/4165724218 >> 172.21.15.98:0/4165724218 conn(0x55fee0fc75f0 msgr2=0x55fee0fccab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:46.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff8580a5740 1 -- 172.21.15.98:0/4165724218 shutdown_connections 2024-09-19T08:04:46.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff8580a5740 1 -- 172.21.15.98:0/4165724218 wait complete. 2024-09-19T08:04:46.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.234+0000 7ff8580a5740 20 client.15990 trim_cache size 0 max 16384 2024-09-19T08:04:46.238 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:04:46.238 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 28, 2024-09-19T08:04:46.238 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24485, 2024-09-19T08:04:46.238 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 8, 2024-09-19T08:04:46.238 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-09-19T08:04:46.239 INFO:teuthology.orchestra.run.smithi098.stdout: "28e1fa50": { 2024-09-19T08:04:46.239 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 8, 2024-09-19T08:04:46.239 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-09-19T08:04:46.239 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-09-19T08:04:46.239 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-09-19T08:04:46.239 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-09-19T08:04:46.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:04:46.239 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 90.0, 2024-09-19T08:04:46.239 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 140.3, 2024-09-19T08:04:46.239 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-09-19T08:04:46.239 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-09-19T08:04:46.239 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-09-19T08:04:46.239 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-09-19T08:04:46.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-09-19T08:04:46.240 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-09-19T08:04:46.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:46.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:46.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:46.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:46.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:46.240 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:04:46.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.235+0000 7ff855ccc640 1 -- 172.21.15.98:0/280535729 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff8380a33b0 msgr2=0x7ff8380a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:46.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.235+0000 7ff855ccc640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff8380a33b0 0x7ff8380a5870 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff8400ccdf0 tx=0x7ff8400d5060 comp rx=0 tx=0).stop 2024-09-19T08:04:46.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.235+0000 7ff855ccc640 1 -- 172.21.15.98:0/280535729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff850101480 msgr2=0x7ff8501985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:46.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.235+0000 7ff855ccc640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff850101480 0x7ff8501985d0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff84000b4a0 tx=0x7ff84000b8a0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.235+0000 7ff855ccc640 1 -- 172.21.15.98:0/280535729 shutdown_connections 2024-09-19T08:04:46.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.235+0000 7ff855ccc640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff8380a33b0 0x7ff8380a5870 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.235+0000 7ff855ccc640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff850105e80 0x7ff85019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.235+0000 7ff855ccc640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff850101e30 0x7ff850198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.235+0000 7ff855ccc640 1 --2- 172.21.15.98:0/280535729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff850101480 0x7ff8501985d0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:46.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.235+0000 7ff855ccc640 1 -- 172.21.15.98:0/280535729 >> 172.21.15.98:0/280535729 conn(0x7ff850077210 msgr2=0x7ff850100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:46.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.235+0000 7ff855ccc640 1 -- 172.21.15.98:0/280535729 shutdown_connections 2024-09-19T08:04:46.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:46.235+0000 7ff855ccc640 1 -- 172.21.15.98:0/280535729 wait complete. 2024-09-19T08:04:46.950 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '28e1fa50', total seconds elapsed: 23.4 2024-09-19T08:04:46.950 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 20.35 2024-09-19T08:04:46.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:04:47.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:47 smithi098 ceph-mon[31333]: pgmap v56: 97 pgs: 97 active+clean; 21 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.3 MiB/s wr, 13 op/s 2024-09-19T08:04:47.282 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:47.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:47 smithi139 ceph-mon[34874]: pgmap v56: 97 pgs: 97 active+clean; 21 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.3 MiB/s wr, 13 op/s 2024-09-19T08:04:47.619 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:47 smithi162 ceph-mon[35260]: pgmap v56: 97 pgs: 97 active+clean; 21 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.3 MiB/s wr, 13 op/s 2024-09-19T08:04:47.766 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:04:47.767 INFO:tasks.workunit.client.0.smithi098.stdout: Children see throughput for 1 initial writers = 49675.75 kB/sec 2024-09-19T08:04:47.767 INFO:tasks.workunit.client.0.smithi098.stdout: Parent sees throughput for 1 initial writers = 49675.33 kB/sec 2024-09-19T08:04:47.767 INFO:tasks.workunit.client.0.smithi098.stdout: Min throughput per process = 49675.75 kB/sec 2024-09-19T08:04:47.767 INFO:tasks.workunit.client.0.smithi098.stdout: Max throughput per process = 49675.75 kB/sec 2024-09-19T08:04:47.767 INFO:tasks.workunit.client.0.smithi098.stdout: Avg throughput per process = 49675.75 kB/sec 2024-09-19T08:04:47.767 INFO:tasks.workunit.client.0.smithi098.stdout: Min xfer = 1048576.00 kB 2024-09-19T08:04:48.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:48 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/298226344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:48.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:48 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/298226344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:48.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:48 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/298226344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:04:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:04:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"btime":"2024-09-19T08:02:59:855935+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:02:59.805764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:04:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.995+0000 7f545ea5e640 1 -- 172.21.15.98:0/2110909506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5458106f10 msgr2=0x7f545810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.995+0000 7f545ea5e640 1 --2- 172.21.15.98:0/2110909506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5458106f10 0x7f545810ea90 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5450009ca0 tx=0x7f545002a8c0 comp rx=0 tx=0).stop 2024-09-19T08:04:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.995+0000 7f545ea5e640 1 -- 172.21.15.98:0/2110909506 shutdown_connections 2024-09-19T08:04:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.995+0000 7f545ea5e640 1 --2- 172.21.15.98:0/2110909506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5458106f10 0x7f545810ea90 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.995+0000 7f545ea5e640 1 --2- 172.21.15.98:0/2110909506 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5458102790 0x7f54581067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.995+0000 7f545ea5e640 1 --2- 172.21.15.98:0/2110909506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5458101de0 0x7f54581021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.995+0000 7f545ea5e640 1 -- 172.21.15.98:0/2110909506 >> 172.21.15.98:0/2110909506 conn(0x7f54580fd880 msgr2=0x7f54580ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.995+0000 7f545ea5e640 1 -- 172.21.15.98:0/2110909506 shutdown_connections 2024-09-19T08:04:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.995+0000 7f545ea5e640 1 -- 172.21.15.98:0/2110909506 wait complete. 2024-09-19T08:04:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545ea5e640 1 Processor -- start 2024-09-19T08:04:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545ea5e640 1 -- start start 2024-09-19T08:04:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545ea5e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5458101de0 0x7f54581983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545ea5e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5458102790 0x7f5458198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545ea5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5458106f10 0x7f545819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545ea5e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5458111f70 con 0x7f5458106f10 2024-09-19T08:04:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545ea5e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5458111df0 con 0x7f5458101de0 2024-09-19T08:04:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545ea5e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f54581120f0 con 0x7f5458102790 2024-09-19T08:04:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545cffb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5458106f10 0x7f545819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545cffb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5458106f10 0x7f545819cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55652/0 (socket says 172.21.15.98:55652) 2024-09-19T08:04:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f54577fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5458102790 0x7f5458198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:49.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545cffb640 1 -- 172.21.15.98:0/298226344 learned_addr learned my addr 172.21.15.98:0/298226344 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:49.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f5457fff640 1 --2- 172.21.15.98:0/298226344 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5458101de0 0x7f54581983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:49.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545cffb640 1 -- 172.21.15.98:0/298226344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5458102790 msgr2=0x7f5458198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:49.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545cffb640 1 --2- 172.21.15.98:0/298226344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5458102790 0x7f5458198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:49.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545cffb640 1 -- 172.21.15.98:0/298226344 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5458101de0 msgr2=0x7f54581983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:49.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545cffb640 1 --2- 172.21.15.98:0/298226344 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5458101de0 0x7f54581983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:49.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f545cffb640 1 -- 172.21.15.98:0/298226344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f545819d380 con 0x7f5458106f10 2024-09-19T08:04:49.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.996+0000 7f54577fe640 1 --2- 172.21.15.98:0/298226344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5458102790 0x7f5458198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:04:49.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.997+0000 7f545cffb640 1 --2- 172.21.15.98:0/298226344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5458106f10 0x7f545819cca0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5450031a20 tx=0x7f5450031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:49.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.997+0000 7f54557fa640 1 -- 172.21.15.98:0/298226344 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5450003c30 con 0x7f5458106f10 2024-09-19T08:04:49.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.997+0000 7f545ea5e640 1 -- 172.21.15.98:0/298226344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f545819d610 con 0x7f5458106f10 2024-09-19T08:04:49.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.997+0000 7f54557fa640 1 -- 172.21.15.98:0/298226344 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5450003dd0 con 0x7f5458106f10 2024-09-19T08:04:49.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.997+0000 7f54557fa640 1 -- 172.21.15.98:0/298226344 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545002fc40 con 0x7f5458106f10 2024-09-19T08:04:49.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.997+0000 7f545ea5e640 1 -- 172.21.15.98:0/298226344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54581a4f50 con 0x7f5458106f10 2024-09-19T08:04:49.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.998+0000 7f545ea5e640 1 -- 172.21.15.98:0/298226344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54581021c0 con 0x7f5458106f10 2024-09-19T08:04:49.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:47.999+0000 7f54557fa640 1 -- 172.21.15.98:0/298226344 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f5450004050 con 0x7f5458106f10 2024-09-19T08:04:49.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.000+0000 7f54557fa640 1 --2- 172.21.15.98:0/298226344 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f54440a33b0 0x7f54440a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:49.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.000+0000 7f5457fff640 1 --2- 172.21.15.98:0/298226344 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f54440a33b0 0x7f54440a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.000+0000 7f54557fa640 1 -- 172.21.15.98:0/298226344 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f54500f0f60 con 0x7f5458106f10 2024-09-19T08:04:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.000+0000 7f5457fff640 1 --2- 172.21.15.98:0/298226344 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f54440a33b0 0x7f54440a5870 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5440009b60 tx=0x7f5440005f90 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.003+0000 7f54557fa640 1 -- 172.21.15.98:0/298226344 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f54500bdb10 con 0x7f5458106f10 2024-09-19T08:04:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.214+0000 7f545ea5e640 1 -- 172.21.15.98:0/298226344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f545810a1b0 con 0x7f5458106f10 2024-09-19T08:04:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.215+0000 7f54557fa640 1 -- 172.21.15.98:0/298226344 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11983 (secure 0 0 0) 0x7f54500bd410 con 0x7f5458106f10 2024-09-19T08:04:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-09-19T08:04:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.217+0000 7f545ea5e640 1 -- 172.21.15.98:0/298226344 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f54440a33b0 msgr2=0x7f54440a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.217+0000 7f545ea5e640 1 --2- 172.21.15.98:0/298226344 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f54440a33b0 0x7f54440a5870 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5440009b60 tx=0x7f5440005f90 comp rx=0 tx=0).stop 2024-09-19T08:04:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.217+0000 7f545ea5e640 1 -- 172.21.15.98:0/298226344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5458106f10 msgr2=0x7f545819cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.217+0000 7f545ea5e640 1 --2- 172.21.15.98:0/298226344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5458106f10 0x7f545819cca0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5450031a20 tx=0x7f5450031a50 comp rx=0 tx=0).stop 2024-09-19T08:04:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.218+0000 7f545ea5e640 1 -- 172.21.15.98:0/298226344 shutdown_connections 2024-09-19T08:04:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.218+0000 7f545ea5e640 1 --2- 172.21.15.98:0/298226344 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f54440a33b0 0x7f54440a5870 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.218+0000 7f545ea5e640 1 --2- 172.21.15.98:0/298226344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5458106f10 0x7f545819cca0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:49.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.218+0000 7f545ea5e640 1 --2- 172.21.15.98:0/298226344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5458102790 0x7f5458198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:49.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.218+0000 7f545ea5e640 1 --2- 172.21.15.98:0/298226344 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5458101de0 0x7f54581983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:49.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.218+0000 7f545ea5e640 1 -- 172.21.15.98:0/298226344 >> 172.21.15.98:0/298226344 conn(0x7f54580fd880 msgr2=0x7f54580feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:49.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.218+0000 7f545ea5e640 1 -- 172.21.15.98:0/298226344 shutdown_connections 2024-09-19T08:04:49.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:48.218+0000 7f545ea5e640 1 -- 172.21.15.98:0/298226344 wait complete. 2024-09-19T08:04:49.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:49 smithi139 ceph-mon[34874]: pgmap v57: 97 pgs: 97 active+clean; 149 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 12 MiB/s wr, 12 op/s 2024-09-19T08:04:49.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:49 smithi098 ceph-mon[31333]: pgmap v57: 97 pgs: 97 active+clean; 149 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 12 MiB/s wr, 12 op/s 2024-09-19T08:04:49.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:49 smithi162 ceph-mon[35260]: pgmap v57: 97 pgs: 97 active+clean; 149 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 12 MiB/s wr, 12 op/s 2024-09-19T08:04:50.941 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 140.35 --await-for 45.0' 2024-09-19T08:04:50.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 140.35 --await-for 45.0 2024-09-19T08:04:51.191 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:04:51.571 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:51 smithi098 ceph-mon[31333]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:04:51.622 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:51 smithi098 ceph-mon[31333]: pgmap v58: 97 pgs: 97 active+clean; 149 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 12 MiB/s wr, 12 op/s 2024-09-19T08:04:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.660+0000 7f5e8c3ca640 1 -- 172.21.15.98:0/3268193217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e84101de0 msgr2=0x7f5e841021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.660+0000 7f5e8c3ca640 1 --2- 172.21.15.98:0/3268193217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e84101de0 0x7f5e841021c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f5e80009900 tx=0x7f5e8002a410 comp rx=0 tx=0).stop 2024-09-19T08:04:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.660+0000 7f5e8c3ca640 1 -- 172.21.15.98:0/3268193217 shutdown_connections 2024-09-19T08:04:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.660+0000 7f5e8c3ca640 1 --2- 172.21.15.98:0/3268193217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e84106f10 0x7f5e8410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.660+0000 7f5e8c3ca640 1 --2- 172.21.15.98:0/3268193217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e84102790 0x7f5e841067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.660+0000 7f5e8c3ca640 1 --2- 172.21.15.98:0/3268193217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e84101de0 0x7f5e841021c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.660+0000 7f5e8c3ca640 1 -- 172.21.15.98:0/3268193217 >> 172.21.15.98:0/3268193217 conn(0x7f5e840fd8c0 msgr2=0x7f5e840ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.660+0000 7f5e8c3ca640 1 -- 172.21.15.98:0/3268193217 shutdown_connections 2024-09-19T08:04:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.660+0000 7f5e8c3ca640 1 -- 172.21.15.98:0/3268193217 wait complete. 2024-09-19T08:04:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.661+0000 7f5e8c3ca640 1 Processor -- start 2024-09-19T08:04:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.661+0000 7f5e8c3ca640 1 -- start start 2024-09-19T08:04:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.661+0000 7f5e8c3ca640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e84101de0 0x7f5e84198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.661+0000 7f5e8c3ca640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e84102790 0x7f5e841989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.661+0000 7f5e8c3ca640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e84106f10 0x7f5e8419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.661+0000 7f5e8c3ca640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e84111f90 con 0x7f5e84101de0 2024-09-19T08:04:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.661+0000 7f5e8c3ca640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5e84111e10 con 0x7f5e84106f10 2024-09-19T08:04:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.661+0000 7f5e8c3ca640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e84112110 con 0x7f5e84102790 2024-09-19T08:04:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.661+0000 7f5e8a166640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e84101de0 0x7f5e84198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.661+0000 7f5e8a166640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e84101de0 0x7f5e84198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55660/0 (socket says 172.21.15.98:55660) 2024-09-19T08:04:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.661+0000 7f5e8a166640 1 -- 172.21.15.98:0/843590228 learned_addr learned my addr 172.21.15.98:0/843590228 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.662+0000 7f5e8a967640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e84106f10 0x7f5e8419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.662+0000 7f5e89965640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e84102790 0x7f5e841989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.662+0000 7f5e8a166640 1 -- 172.21.15.98:0/843590228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e84102790 msgr2=0x7f5e841989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.662+0000 7f5e8a166640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e84102790 0x7f5e841989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.662+0000 7f5e8a166640 1 -- 172.21.15.98:0/843590228 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e84106f10 msgr2=0x7f5e8419cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.662+0000 7f5e8a166640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e84106f10 0x7f5e8419cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.662+0000 7f5e8a166640 1 -- 172.21.15.98:0/843590228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8419d430 con 0x7f5e84101de0 2024-09-19T08:04:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.662+0000 7f5e89965640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e84102790 0x7f5e841989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:04:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.662+0000 7f5e8a166640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e84101de0 0x7f5e84198480 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5e80006010 tx=0x7f5e800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.662+0000 7f5e7b7fe640 1 -- 172.21.15.98:0/843590228 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8003f030 con 0x7f5e84101de0 2024-09-19T08:04:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.662+0000 7f5e7b7fe640 1 -- 172.21.15.98:0/843590228 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e80003ce0 con 0x7f5e84101de0 2024-09-19T08:04:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.662+0000 7f5e8c3ca640 1 -- 172.21.15.98:0/843590228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e8419d6c0 con 0x7f5e84101de0 2024-09-19T08:04:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.662+0000 7f5e7b7fe640 1 -- 172.21.15.98:0/843590228 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e80035070 con 0x7f5e84101de0 2024-09-19T08:04:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.662+0000 7f5e8c3ca640 1 -- 172.21.15.98:0/843590228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e841a5000 con 0x7f5e84101de0 2024-09-19T08:04:51.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.664+0000 7f5e7b7fe640 1 -- 172.21.15.98:0/843590228 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f5e80042030 con 0x7f5e84101de0 2024-09-19T08:04:51.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.665+0000 7f5e8e7a3740 1 Processor -- start 2024-09-19T08:04:51.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.665+0000 7f5e8e7a3740 1 -- start start 2024-09-19T08:04:51.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.665+0000 7f5e8e7a3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfc2850 0x555f8bfc2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.665+0000 7f5e8e7a3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc3170 0x555f8bfcd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.665+0000 7f5e8e7a3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfcda30 0x555f8bfcfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.665+0000 7f5e8e7a3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555f8bf27160 con 0x555f8bfc3170 2024-09-19T08:04:51.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.665+0000 7f5e8e7a3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555f8bece720 con 0x555f8bfc2850 2024-09-19T08:04:51.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.665+0000 7f5e8e7a3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555f8bf23120 con 0x555f8bfcda30 2024-09-19T08:04:51.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.665+0000 7f5e8a166640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc3170 0x555f8bfcd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e8a166640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc3170 0x555f8bfcd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55676/0 (socket says 172.21.15.98:55676) 2024-09-19T08:04:51.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e8a166640 1 -- 172.21.15.98:0/623708878 learned_addr learned my addr 172.21.15.98:0/623708878 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:51.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e7b7fe640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5e700a3430 0x7f5e700a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e8a967640 1 --2- 172.21.15.98:0/623708878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfc2850 0x555f8bfc2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e89965640 1 --2- 172.21.15.98:0/623708878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfcda30 0x555f8bfcfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e8a967640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5e700a3430 0x7f5e700a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e8a166640 1 -- 172.21.15.98:0/623708878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfcda30 msgr2=0x555f8bfcfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e8a166640 1 --2- 172.21.15.98:0/623708878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfcda30 0x555f8bfcfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e8a166640 1 -- 172.21.15.98:0/623708878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfc2850 msgr2=0x555f8bfc2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e8a166640 1 --2- 172.21.15.98:0/623708878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfc2850 0x555f8bfc2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e8a166640 1 -- 172.21.15.98:0/623708878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f8bc98ba0 con 0x555f8bfc3170 2024-09-19T08:04:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e7b7fe640 1 -- 172.21.15.98:0/843590228 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5e800f03b0 con 0x7f5e84101de0 2024-09-19T08:04:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e8a166640 1 --2- 172.21.15.98:0/623708878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc3170 0x555f8bfcd4f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f5e800eda40 tx=0x7f5e8002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b63fee07e54bae72 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e78dea640 1 -- 172.21.15.98:0/623708878 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e80049070 con 0x555f8bfc3170 2024-09-19T08:04:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e8a967640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5e700a3430 0x7f5e700a58f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c009fd0 tx=0x7f5e7c004a40 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e78dea640 1 -- 172.21.15.98:0/623708878 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e8003b580 con 0x555f8bfc3170 2024-09-19T08:04:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.666+0000 7f5e78dea640 1 -- 172.21.15.98:0/623708878 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e800bc720 con 0x555f8bfc3170 2024-09-19T08:04:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.667+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/623708878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc3170 msgr2=0x555f8bfcd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.667+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/623708878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc3170 0x555f8bfcd4f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f5e800eda40 tx=0x7f5e8002a410 comp rx=0 tx=0).stop 2024-09-19T08:04:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.667+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/623708878 shutdown_connections 2024-09-19T08:04:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.667+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/623708878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfcda30 0x555f8bfcfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.667+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/623708878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc3170 0x555f8bfcd4f0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.667+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/623708878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfc2850 0x555f8bfc2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.667+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/623708878 >> 172.21.15.98:0/623708878 conn(0x555f8bfbede0 msgr2=0x555f8bfbf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.667+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/623708878 shutdown_connections 2024-09-19T08:04:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.667+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/623708878 wait complete. 2024-09-19T08:04:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e8e7a3740 1 -- start start 2024-09-19T08:04:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e8e7a3740 1 Processor -- start 2024-09-19T08:04:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e8e7a3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfc2850 0x555f8bfca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e8e7a3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc3170 0x555f8bfcae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e8e7a3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfcda30 0x555f8bfcb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e8e7a3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555f8bfd47d0 con 0x555f8bfc3170 2024-09-19T08:04:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e8e7a3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555f8beaf900 con 0x555f8bfcda30 2024-09-19T08:04:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e8e7a3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555f8bfd4950 con 0x555f8bfc2850 2024-09-19T08:04:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e8e7a3740 10 client.0 _ll_register_callbacks cb 0x555f8bead780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e89965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc3170 0x555f8bfcae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e89965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc3170 0x555f8bfcae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55688/0 (socket says 172.21.15.98:55688) 2024-09-19T08:04:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e89965640 1 -- 172.21.15.98:0/2969997464 learned_addr learned my addr 172.21.15.98:0/2969997464 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e8a166640 1 --2- 172.21.15.98:0/2969997464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfc2850 0x555f8bfca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e8a967640 1 --2- 172.21.15.98:0/2969997464 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfcda30 0x555f8bfcb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e89965640 1 -- 172.21.15.98:0/2969997464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfc2850 msgr2=0x555f8bfca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e89965640 1 --2- 172.21.15.98:0/2969997464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfc2850 0x555f8bfca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e89965640 1 -- 172.21.15.98:0/2969997464 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfcda30 msgr2=0x555f8bfcb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e89965640 1 --2- 172.21.15.98:0/2969997464 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfcda30 0x555f8bfcb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.668+0000 7f5e89965640 1 -- 172.21.15.98:0/2969997464 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f8beefc50 con 0x555f8bfc3170 2024-09-19T08:04:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e89965640 1 --2- 172.21.15.98:0/2969997464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc3170 0x555f8bfcae00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009f80 tx=0x7f5e74011870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452d7bc5457ce380 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e6dffb640 1 -- 172.21.15.98:0/2969997464 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74012920 con 0x555f8bfc3170 2024-09-19T08:04:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e6dffb640 1 -- 172.21.15.98:0/2969997464 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5e74012ac0 con 0x555f8bfc3170 2024-09-19T08:04:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/2969997464 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555f8bf34890 con 0x555f8bfc3170 2024-09-19T08:04:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e6dffb640 1 -- 172.21.15.98:0/2969997464 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5e74012cc0 con 0x555f8bfc3170 2024-09-19T08:04:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e8e7a3740 10 client.16008 fetch_fsmap learned FSMap version 29 2024-09-19T08:04:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/2969997464 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555f8bfe03e0 con 0x555f8bfc3170 2024-09-19T08:04:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e6dffb640 1 -- 172.21.15.98:0/2969997464 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555f8bf34890 con 0x555f8bfc3170 2024-09-19T08:04:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e6dffb640 1 -- 172.21.15.98:0/2969997464 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f5e7400f040 con 0x555f8bfc3170 2024-09-19T08:04:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e8e7a3740 10 client.16008 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T08:04:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e8e7a3740 10 client.16008 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-19T08:04:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e8e7a3740 20 client.16008 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/2969997464 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555f8bfe1940 0x555f8bfe3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e8e7a3740 4 client.16008 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:04:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.669+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/2969997464 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f8bf2ebe0 con 0x555f8bfe1940 2024-09-19T08:04:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.670+0000 7f5e8a166640 1 --2- 172.21.15.98:0/2969997464 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555f8bfe1940 0x555f8bfe3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.670+0000 7f5e6dffb640 10 client.16008 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.670+0000 7f5e8a166640 1 --2- 172.21.15.98:0/2969997464 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555f8bfe1940 0x555f8bfe3d30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ed7efeeea351690 server_cookie=41648e7d5bfc70ce in_seq=0 out_seq=0 2024-09-19T08:04:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e6dffb640 1 -- 172.21.15.98:0/2969997464 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555f8bf2ebe0 con 0x555f8bfe1940 2024-09-19T08:04:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e6dffb640 10 client.16008 handle_command_reply: tid=0 2024-09-19T08:04:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e8e7a3740 1 client.16008 shutdown 2024-09-19T08:04:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/2969997464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc3170 msgr2=0x555f8bfcae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/2969997464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc3170 0x555f8bfcae00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009f80 tx=0x7f5e74011870 comp rx=0 tx=0).stop 2024-09-19T08:04:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/2969997464 shutdown_connections 2024-09-19T08:04:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/2969997464 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555f8bfe1940 0x555f8bfe3d30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/2969997464 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfcda30 0x555f8bfcb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/2969997464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc3170 0x555f8bfcae00 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/2969997464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfc2850 0x555f8bfca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e6dffb640 0 client.16008 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/2969997464 >> 172.21.15.98:0/2969997464 conn(0x555f8bfbede0 msgr2=0x555f8bfc4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/2969997464 shutdown_connections 2024-09-19T08:04:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/2969997464 wait complete. 2024-09-19T08:04:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.671+0000 7f5e8e7a3740 20 client.16008 trim_cache size 0 max 16384 2024-09-19T08:04:51.684 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:51 smithi139 ceph-mon[34874]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:04:51.685 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:51 smithi139 ceph-mon[34874]: pgmap v58: 97 pgs: 97 active+clean; 149 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 12 MiB/s wr, 12 op/s 2024-09-19T08:04:51.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e8e7a3740 1 Processor -- start 2024-09-19T08:04:51.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e8e7a3740 1 -- start start 2024-09-19T08:04:51.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e8e7a3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfbede0 0x555f8bfcc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e8e7a3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc2850 0x555f8bfcb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e8e7a3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfc3170 0x555f8bfe9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e8e7a3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e841ab690 con 0x555f8bfc2850 2024-09-19T08:04:51.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e8e7a3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5e841ad200 con 0x555f8bfbede0 2024-09-19T08:04:51.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e8e7a3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e841abd60 con 0x555f8bfc3170 2024-09-19T08:04:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e89965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc2850 0x555f8bfcb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e89965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc2850 0x555f8bfcb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55700/0 (socket says 172.21.15.98:55700) 2024-09-19T08:04:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e89965640 1 -- 172.21.15.98:0/2465522490 learned_addr learned my addr 172.21.15.98:0/2465522490 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e8a166640 1 --2- 172.21.15.98:0/2465522490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfbede0 0x555f8bfcc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e89965640 1 -- 172.21.15.98:0/2465522490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfc3170 msgr2=0x555f8bfe9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e89965640 1 --2- 172.21.15.98:0/2465522490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfc3170 0x555f8bfe9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e89965640 1 -- 172.21.15.98:0/2465522490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfbede0 msgr2=0x555f8bfcc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e89965640 1 --2- 172.21.15.98:0/2465522490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfbede0 0x555f8bfcc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e89965640 1 -- 172.21.15.98:0/2465522490 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f8bfbfa70 con 0x555f8bfc2850 2024-09-19T08:04:51.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e89965640 1 --2- 172.21.15.98:0/2465522490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc2850 0x555f8bfcb190 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f5e7401a540 tx=0x7f5e74005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40bfd0e8df5aa5b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:51.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.688+0000 7f5e6ffff640 1 -- 172.21.15.98:0/2465522490 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74012e60 con 0x555f8bfc2850 2024-09-19T08:04:51.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.689+0000 7f5e6ffff640 1 -- 172.21.15.98:0/2465522490 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e74031070 con 0x555f8bfc2850 2024-09-19T08:04:51.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.689+0000 7f5e6ffff640 1 -- 172.21.15.98:0/2465522490 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e740053c0 con 0x555f8bfc2850 2024-09-19T08:04:51.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.689+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/2465522490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc2850 msgr2=0x555f8bfcb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:51.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.689+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/2465522490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc2850 0x555f8bfcb190 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f5e7401a540 tx=0x7f5e74005850 comp rx=0 tx=0).stop 2024-09-19T08:04:51.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.689+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/2465522490 shutdown_connections 2024-09-19T08:04:51.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.689+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/2465522490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfc3170 0x555f8bfe9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.689+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/2465522490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc2850 0x555f8bfcb190 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.689+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/2465522490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfbede0 0x555f8bfcc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.689+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/2465522490 >> 172.21.15.98:0/2465522490 conn(0x555f8bfc62b0 msgr2=0x555f8bfbdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.689+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/2465522490 shutdown_connections 2024-09-19T08:04:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.689+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/2465522490 wait complete. 2024-09-19T08:04:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.689+0000 7f5e8e7a3740 1 -- start start 2024-09-19T08:04:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e8e7a3740 1 Processor -- start 2024-09-19T08:04:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e8e7a3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfbede0 0x555f8bfd5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e8e7a3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc2850 0x555f8bfd6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e8e7a3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfc3170 0x555f8c04b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e8e7a3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e841acb00 con 0x555f8bfc2850 2024-09-19T08:04:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e8e7a3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5e841ac430 con 0x555f8bfc3170 2024-09-19T08:04:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e8e7a3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e841a9670 con 0x555f8bfbede0 2024-09-19T08:04:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e8e7a3740 10 client.0 _ll_register_callbacks cb 0x7f5e740097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:04:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e89965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc2850 0x555f8bfd6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e89965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc2850 0x555f8bfd6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55712/0 (socket says 172.21.15.98:55712) 2024-09-19T08:04:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e89965640 1 -- 172.21.15.98:0/1985004924 learned_addr learned my addr 172.21.15.98:0/1985004924 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:04:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e8a166640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfbede0 0x555f8bfd5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e8a967640 1 --2- 172.21.15.98:0/1985004924 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfc3170 0x555f8c04b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e89965640 1 -- 172.21.15.98:0/1985004924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfbede0 msgr2=0x555f8bfd5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e89965640 1 --2- 172.21.15.98:0/1985004924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfbede0 0x555f8bfd5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e89965640 1 -- 172.21.15.98:0/1985004924 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfc3170 msgr2=0x555f8c04b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:04:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e89965640 1 --2- 172.21.15.98:0/1985004924 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfc3170 0x555f8c04b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e89965640 1 -- 172.21.15.98:0/1985004924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f8c04bb50 con 0x555f8bfc2850 2024-09-19T08:04:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e89965640 1 --2- 172.21.15.98:0/1985004924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc2850 0x555f8bfd6410 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f5e7402e040 tx=0x7f5e7401a9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9162c3be10b92f98 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:04:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.690+0000 7f5e6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:04:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.691+0000 7f5e6dffb640 1 -- 172.21.15.98:0/1985004924 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74011d70 con 0x555f8bfc2850 2024-09-19T08:04:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.691+0000 7f5e6dffb640 1 -- 172.21.15.98:0/1985004924 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5e7401aae0 con 0x555f8bfc2850 2024-09-19T08:04:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.691+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/1985004924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e74031070 con 0x555f8bfc2850 2024-09-19T08:04:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.691+0000 7f5e6dffb640 1 -- 172.21.15.98:0/1985004924 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5e74044050 con 0x555f8bfc2850 2024-09-19T08:04:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.691+0000 7f5e8e7a3740 10 client.16014 fetch_fsmap learned FSMap version 29 2024-09-19T08:04:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.691+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/1985004924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555f8c04ce30 con 0x555f8bfc2850 2024-09-19T08:04:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.691+0000 7f5e6dffb640 1 -- 172.21.15.98:0/1985004924 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e74031070 con 0x555f8bfc2850 2024-09-19T08:04:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.691+0000 7f5e6dffb640 1 -- 172.21.15.98:0/1985004924 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f5e74004950 con 0x555f8bfc2850 2024-09-19T08:04:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.691+0000 7f5e8e7a3740 10 client.16014 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T08:04:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.691+0000 7f5e8e7a3740 10 client.16014 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-19T08:04:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.691+0000 7f5e8e7a3740 20 client.16014 populate_metadata read hostname 'smithi098' 2024-09-19T08:04:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.691+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/1985004924 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555f8bfcda30 0x555f8c04f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:04:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.691+0000 7f5e8e7a3740 4 client.16014 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.35, "await_for": 45.0}] 2024-09-19T08:04:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.691+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/1985004924 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.35, "await_for": 45.0}) -- 0x555f8beaaf40 con 0x555f8bfcda30 2024-09-19T08:04:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.692+0000 7f5e8a166640 1 --2- 172.21.15.98:0/1985004924 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555f8bfcda30 0x555f8c04f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:04:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.692+0000 7f5e8a166640 1 --2- 172.21.15.98:0/1985004924 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555f8bfcda30 0x555f8c04f5d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccf0284b220d6962 server_cookie=ec2c5e90f93b4d4e in_seq=0 out_seq=0 2024-09-19T08:04:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.692+0000 7f5e6dffb640 10 client.16014 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:51.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e6dffb640 1 -- 172.21.15.98:0/1985004924 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x555f8beaaf40 con 0x555f8bfcda30 2024-09-19T08:04:51.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e6dffb640 10 client.16014 handle_command_reply: tid=0 2024-09-19T08:04:51.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8e7a3740 1 client.16014 shutdown 2024-09-19T08:04:51.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/1985004924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc2850 msgr2=0x555f8bfd6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:51.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/1985004924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc2850 0x555f8bfd6410 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f5e7402e040 tx=0x7f5e7401a9e0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/1985004924 shutdown_connections 2024-09-19T08:04:51.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/1985004924 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555f8bfcda30 0x555f8c04f5d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e6dffb640 0 client.16014 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:04:51.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/1985004924 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f8bfc3170 0x555f8c04b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/1985004924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f8bfc2850 0x555f8bfd6410 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8e7a3740 1 --2- 172.21.15.98:0/1985004924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f8bfbede0 0x555f8bfd5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/1985004924 >> 172.21.15.98:0/1985004924 conn(0x555f8bfc62b0 msgr2=0x555f8bfbd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:51.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/1985004924 shutdown_connections 2024-09-19T08:04:51.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8e7a3740 1 -- 172.21.15.98:0/1985004924 wait complete. 2024-09-19T08:04:51.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8e7a3740 20 client.16014 trim_cache size 0 max 16384 2024-09-19T08:04:51.704 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:04:51.704 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 29, 2024-09-19T08:04:51.705 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24485, 2024-09-19T08:04:51.705 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 10, 2024-09-19T08:04:51.705 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-09-19T08:04:51.705 INFO:teuthology.orchestra.run.smithi098.stdout: "8bad5758": { 2024-09-19T08:04:51.705 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 10, 2024-09-19T08:04:51.708 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-09-19T08:04:51.709 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-09-19T08:04:51.709 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-09-19T08:04:51.709 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-09-19T08:04:51.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:04:51.709 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 90.0, 2024-09-19T08:04:51.709 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 140.3, 2024-09-19T08:04:51.709 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-09-19T08:04:51.709 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-09-19T08:04:51.709 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-09-19T08:04:51.709 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-09-19T08:04:51.709 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-09-19T08:04:51.709 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-09-19T08:04:51.709 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:51.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:51.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:51.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:51.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:04:51.711 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:04:51.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8c3ca640 1 -- 172.21.15.98:0/843590228 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5e700a3430 msgr2=0x7f5e700a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:51.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8c3ca640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5e700a3430 0x7f5e700a58f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c009fd0 tx=0x7f5e7c004a40 comp rx=0 tx=0).stop 2024-09-19T08:04:51.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8c3ca640 1 -- 172.21.15.98:0/843590228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e84101de0 msgr2=0x7f5e84198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:04:51.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.702+0000 7f5e8c3ca640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e84101de0 0x7f5e84198480 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5e80006010 tx=0x7f5e800042f0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.703+0000 7f5e8c3ca640 1 -- 172.21.15.98:0/843590228 shutdown_connections 2024-09-19T08:04:51.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.703+0000 7f5e8c3ca640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5e700a3430 0x7f5e700a58f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.703+0000 7f5e8c3ca640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e84106f10 0x7f5e8419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.703+0000 7f5e8c3ca640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e84102790 0x7f5e841989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.703+0000 7f5e8c3ca640 1 --2- 172.21.15.98:0/843590228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e84101de0 0x7f5e84198480 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:04:51.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.703+0000 7f5e8c3ca640 1 -- 172.21.15.98:0/843590228 >> 172.21.15.98:0/843590228 conn(0x7f5e840fd8c0 msgr2=0x7f5e840fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:04:51.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.703+0000 7f5e8c3ca640 1 -- 172.21.15.98:0/843590228 shutdown_connections 2024-09-19T08:04:51.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:04:51.703+0000 7f5e8c3ca640 1 -- 172.21.15.98:0/843590228 wait complete. 2024-09-19T08:04:51.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:51 smithi162 ceph-mon[35260]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:04:51.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:51 smithi162 ceph-mon[35260]: pgmap v58: 97 pgs: 97 active+clean; 149 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 12 MiB/s wr, 12 op/s 2024-09-19T08:04:52.389 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '8bad5758', quiesce took 5.4 seconds. Will release after: 14.950000000000001 2024-09-19T08:04:53.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:52 smithi162 ceph-mon[35260]: pgmap v59: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 85 MiB/s wr, 36 op/s 2024-09-19T08:04:53.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:52 smithi139 ceph-mon[34874]: pgmap v59: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 85 MiB/s wr, 36 op/s 2024-09-19T08:04:53.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:52 smithi098 ceph-mon[31333]: pgmap v59: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 85 MiB/s wr, 36 op/s 2024-09-19T08:04:55.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:54 smithi162 ceph-mon[35260]: pgmap v60: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-09-19T08:04:55.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:54 smithi139 ceph-mon[34874]: pgmap v60: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-09-19T08:04:55.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:54 smithi098 ceph-mon[31333]: pgmap v60: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-09-19T08:04:57.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:56 smithi162 ceph-mon[35260]: pgmap v61: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 85 MiB/s wr, 35 op/s 2024-09-19T08:04:57.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:56 smithi139 ceph-mon[34874]: pgmap v61: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 85 MiB/s wr, 35 op/s 2024-09-19T08:04:57.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:56 smithi098 ceph-mon[31333]: pgmap v61: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 85 MiB/s wr, 35 op/s 2024-09-19T08:04:59.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:04:58 smithi162 ceph-mon[35260]: pgmap v62: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 84 MiB/s wr, 32 op/s 2024-09-19T08:04:59.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:04:58 smithi139 ceph-mon[34874]: pgmap v62: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 84 MiB/s wr, 32 op/s 2024-09-19T08:04:59.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:04:58 smithi098 ceph-mon[31333]: pgmap v62: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 84 MiB/s wr, 32 op/s 2024-09-19T08:05:01.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:00 smithi162 ceph-mon[35260]: pgmap v63: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 73 MiB/s wr, 28 op/s 2024-09-19T08:05:01.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:00 smithi139 ceph-mon[34874]: pgmap v63: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 73 MiB/s wr, 28 op/s 2024-09-19T08:05:01.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:00 smithi098 ceph-mon[31333]: pgmap v63: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 73 MiB/s wr, 28 op/s 2024-09-19T08:05:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:02 smithi162 ceph-mon[35260]: pgmap v64: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 73 MiB/s wr, 47 op/s 2024-09-19T08:05:03.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:02 smithi139 ceph-mon[34874]: pgmap v64: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 73 MiB/s wr, 47 op/s 2024-09-19T08:05:03.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:02 smithi098 ceph-mon[31333]: pgmap v64: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 73 MiB/s wr, 47 op/s 2024-09-19T08:05:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:04 smithi162 ceph-mon[35260]: pgmap v65: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-09-19T08:05:05.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:04 smithi139 ceph-mon[34874]: pgmap v65: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-09-19T08:05:05.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:04 smithi098 ceph-mon[31333]: pgmap v65: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-09-19T08:05:07.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:06 smithi162 ceph-mon[35260]: pgmap v66: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-09-19T08:05:07.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:06 smithi139 ceph-mon[34874]: pgmap v66: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-09-19T08:05:07.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:06 smithi098 ceph-mon[31333]: pgmap v66: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-09-19T08:05:07.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:05:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:05:08.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.058+0000 7f3da1844640 1 -- 172.21.15.98:0/3387303591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d9c102790 msgr2=0x7f3d9c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:08.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.058+0000 7f3da1844640 1 --2- 172.21.15.98:0/3387303591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d9c102790 0x7f3d9c1067e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c009900 tx=0x7f3d8c02a410 comp rx=0 tx=0).stop 2024-09-19T08:05:08.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.058+0000 7f3da1844640 1 -- 172.21.15.98:0/3387303591 shutdown_connections 2024-09-19T08:05:08.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.058+0000 7f3da1844640 1 --2- 172.21.15.98:0/3387303591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d9c106f10 0x7f3d9c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:08.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.058+0000 7f3da1844640 1 --2- 172.21.15.98:0/3387303591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d9c102790 0x7f3d9c1067e0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:08.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.058+0000 7f3da1844640 1 --2- 172.21.15.98:0/3387303591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d9c101de0 0x7f3d9c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:08.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.058+0000 7f3da1844640 1 -- 172.21.15.98:0/3387303591 >> 172.21.15.98:0/3387303591 conn(0x7f3d9c0fd8c0 msgr2=0x7f3d9c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:08.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.058+0000 7f3da1844640 1 -- 172.21.15.98:0/3387303591 shutdown_connections 2024-09-19T08:05:08.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.058+0000 7f3da1844640 1 -- 172.21.15.98:0/3387303591 wait complete. 2024-09-19T08:05:08.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.059+0000 7f3da1844640 1 Processor -- start 2024-09-19T08:05:08.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.059+0000 7f3da1844640 1 -- start start 2024-09-19T08:05:08.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.059+0000 7f3da1844640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d9c101de0 0x7f3d9c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:08.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.059+0000 7f3da1844640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d9c102790 0x7f3d9c198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:08.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.059+0000 7f3da1844640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d9c106f10 0x7f3d9c19cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:08.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.059+0000 7f3da1844640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c111f30 con 0x7f3d9c101de0 2024-09-19T08:05:08.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.059+0000 7f3da1844640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c111db0 con 0x7f3d9c106f10 2024-09-19T08:05:08.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.059+0000 7f3da1844640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c1120b0 con 0x7f3d9c102790 2024-09-19T08:05:08.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.059+0000 7f3d9affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d9c101de0 0x7f3d9c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:08.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.059+0000 7f3d9affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d9c101de0 0x7f3d9c198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48198/0 (socket says 172.21.15.98:48198) 2024-09-19T08:05:08.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.059+0000 7f3d9affd640 1 -- 172.21.15.98:0/4162445027 learned_addr learned my addr 172.21.15.98:0/4162445027 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:08.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.060+0000 7f3d9a7fc640 1 --2- 172.21.15.98:0/4162445027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d9c102790 0x7f3d9c198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:08.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.060+0000 7f3d9b7fe640 1 --2- 172.21.15.98:0/4162445027 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d9c106f10 0x7f3d9c19cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:08.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.060+0000 7f3d9affd640 1 -- 172.21.15.98:0/4162445027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d9c102790 msgr2=0x7f3d9c198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:08.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.060+0000 7f3d9affd640 1 --2- 172.21.15.98:0/4162445027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d9c102790 0x7f3d9c198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:08.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.060+0000 7f3d9affd640 1 -- 172.21.15.98:0/4162445027 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d9c106f10 msgr2=0x7f3d9c19cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:08.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.060+0000 7f3d9affd640 1 --2- 172.21.15.98:0/4162445027 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d9c106f10 0x7f3d9c19cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:08.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.060+0000 7f3d9affd640 1 -- 172.21.15.98:0/4162445027 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d9c19d3e0 con 0x7f3d9c101de0 2024-09-19T08:05:08.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.060+0000 7f3d9affd640 1 --2- 172.21.15.98:0/4162445027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d9c101de0 0x7f3d9c198430 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f3d9000e9f0 tx=0x7f3d9000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:08.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.060+0000 7f3da0842640 1 -- 172.21.15.98:0/4162445027 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9000cd80 con 0x7f3d9c101de0 2024-09-19T08:05:08.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.060+0000 7f3da1844640 1 -- 172.21.15.98:0/4162445027 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d9c19d6d0 con 0x7f3d9c101de0 2024-09-19T08:05:08.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.060+0000 7f3da0842640 1 -- 172.21.15.98:0/4162445027 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d90010910 con 0x7f3d9c101de0 2024-09-19T08:05:08.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.060+0000 7f3da0842640 1 -- 172.21.15.98:0/4162445027 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d900188f0 con 0x7f3d9c101de0 2024-09-19T08:05:08.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.060+0000 7f3da1844640 1 -- 172.21.15.98:0/4162445027 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d9c1a4fb0 con 0x7f3d9c101de0 2024-09-19T08:05:08.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.061+0000 7f3da1844640 1 -- 172.21.15.98:0/4162445027 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d9c1021c0 con 0x7f3d9c101de0 2024-09-19T08:05:08.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.062+0000 7f3da0842640 1 -- 172.21.15.98:0/4162445027 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f3d90010ac0 con 0x7f3d9c101de0 2024-09-19T08:05:08.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.063+0000 7f3da0842640 1 --2- 172.21.15.98:0/4162445027 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3d840a33b0 0x7f3d840a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:08.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.063+0000 7f3d9a7fc640 1 --2- 172.21.15.98:0/4162445027 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3d840a33b0 0x7f3d840a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:08.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.063+0000 7f3da0842640 1 -- 172.21.15.98:0/4162445027 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f3d900d0090 con 0x7f3d9c101de0 2024-09-19T08:05:08.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.063+0000 7f3d9a7fc640 1 --2- 172.21.15.98:0/4162445027 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3d840a33b0 0x7f3d840a5870 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c009230 tx=0x7f3d8c0058d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:08.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.067+0000 7f3da0842640 1 -- 172.21.15.98:0/4162445027 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f3d9009cb40 con 0x7f3d9c101de0 2024-09-19T08:05:08.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.267+0000 7f3da1844640 1 -- 172.21.15.98:0/4162445027 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d9c10a0f0 con 0x7f3d9c101de0 2024-09-19T08:05:08.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.268+0000 7f3da0842640 1 -- 172.21.15.98:0/4162445027 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11984 (secure 0 0 0) 0x7f3d9009c440 con 0x7f3d9c101de0 2024-09-19T08:05:08.268 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:05:08.268 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"btime":"2024-09-19T08:04:50:417827+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:04:50.397583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24461,24437,24406,24376]},"id":1}]} 2024-09-19T08:05:08.269 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-09-19T08:05:08.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.271+0000 7f3da1844640 1 -- 172.21.15.98:0/4162445027 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3d840a33b0 msgr2=0x7f3d840a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:08.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.271+0000 7f3da1844640 1 --2- 172.21.15.98:0/4162445027 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3d840a33b0 0x7f3d840a5870 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c009230 tx=0x7f3d8c0058d0 comp rx=0 tx=0).stop 2024-09-19T08:05:08.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.271+0000 7f3da1844640 1 -- 172.21.15.98:0/4162445027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d9c101de0 msgr2=0x7f3d9c198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:08.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.271+0000 7f3da1844640 1 --2- 172.21.15.98:0/4162445027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d9c101de0 0x7f3d9c198430 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f3d9000e9f0 tx=0x7f3d9000eeb0 comp rx=0 tx=0).stop 2024-09-19T08:05:08.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.271+0000 7f3da1844640 1 -- 172.21.15.98:0/4162445027 shutdown_connections 2024-09-19T08:05:08.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.271+0000 7f3da1844640 1 --2- 172.21.15.98:0/4162445027 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3d840a33b0 0x7f3d840a5870 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:08.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.271+0000 7f3da1844640 1 --2- 172.21.15.98:0/4162445027 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d9c106f10 0x7f3d9c19cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:08.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.271+0000 7f3da1844640 1 --2- 172.21.15.98:0/4162445027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d9c102790 0x7f3d9c198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:08.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.271+0000 7f3da1844640 1 --2- 172.21.15.98:0/4162445027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d9c101de0 0x7f3d9c198430 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:08.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.271+0000 7f3da1844640 1 -- 172.21.15.98:0/4162445027 >> 172.21.15.98:0/4162445027 conn(0x7f3d9c0fd8c0 msgr2=0x7f3d9c0fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:08.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.272+0000 7f3da1844640 1 -- 172.21.15.98:0/4162445027 shutdown_connections 2024-09-19T08:05:08.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:08.272+0000 7f3da1844640 1 -- 172.21.15.98:0/4162445027 wait complete. 2024-09-19T08:05:08.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:08 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/4162445027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:05:08.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:08 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/4162445027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:05:08.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:08 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/4162445027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:05:08.917 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id 8bad5758 --release --await' 2024-09-19T08:05:08.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id 8bad5758 --release --await 2024-09-19T08:05:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:05:09.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:09 smithi098 ceph-mon[31333]: pgmap v67: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-09-19T08:05:09.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:09 smithi162 ceph-mon[35260]: pgmap v67: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-09-19T08:05:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.647+0000 7f7714e41640 1 -- 172.21.15.98:0/2886936947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f771010c170 msgr2=0x7f771010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.647+0000 7f7714e41640 1 --2- 172.21.15.98:0/2886936947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f771010c170 0x7f771010e610 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7708009900 tx=0x7f770802a480 comp rx=0 tx=0).stop 2024-09-19T08:05:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.648+0000 7f7714e41640 1 -- 172.21.15.98:0/2886936947 shutdown_connections 2024-09-19T08:05:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.648+0000 7f7714e41640 1 --2- 172.21.15.98:0/2886936947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f771010c170 0x7f771010e610 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.648+0000 7f7714e41640 1 --2- 172.21.15.98:0/2886936947 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7710103660 0x7f771010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.648+0000 7f7714e41640 1 --2- 172.21.15.98:0/2886936947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7710102d40 0x7f7710103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.648+0000 7f7714e41640 1 -- 172.21.15.98:0/2886936947 >> 172.21.15.98:0/2886936947 conn(0x7f77100fc7a0 msgr2=0x7f77100febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.648+0000 7f7714e41640 1 -- 172.21.15.98:0/2886936947 shutdown_connections 2024-09-19T08:05:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.648+0000 7f7714e41640 1 -- 172.21.15.98:0/2886936947 wait complete. 2024-09-19T08:05:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.648+0000 7f7714e41640 1 Processor -- start 2024-09-19T08:05:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.648+0000 7f7714e41640 1 -- start start 2024-09-19T08:05:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f7714e41640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7710102d40 0x7f7710198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f7714e41640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7710103660 0x7f77101988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f7714e41640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f771010c170 0x7f771019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f7714e41640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7710111f30 con 0x7f7710103660 2024-09-19T08:05:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f7714e41640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7710111db0 con 0x7f771010c170 2024-09-19T08:05:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f7714e41640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f77101120b0 con 0x7f7710102d40 2024-09-19T08:05:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f770dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7710103660 0x7f77101988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f770dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7710103660 0x7f77101988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48212/0 (socket says 172.21.15.98:48212) 2024-09-19T08:05:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f770dd9b640 1 -- 172.21.15.98:0/2958310489 learned_addr learned my addr 172.21.15.98:0/2958310489 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f770e59c640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7710102d40 0x7f7710198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f770ed9d640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f771010c170 0x7f771019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f770dd9b640 1 -- 172.21.15.98:0/2958310489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7710102d40 msgr2=0x7f7710198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f770dd9b640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7710102d40 0x7f7710198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f770dd9b640 1 -- 172.21.15.98:0/2958310489 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f771010c170 msgr2=0x7f771019cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f770dd9b640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f771010c170 0x7f771019cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f770dd9b640 1 -- 172.21.15.98:0/2958310489 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f771019d320 con 0x7f7710103660 2024-09-19T08:05:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f770ed9d640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f771010c170 0x7f771019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:05:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f770dd9b640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7710103660 0x7f77101988b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f770400d740 tx=0x7f770400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f76ff7fe640 1 -- 172.21.15.98:0/2958310489 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704017070 con 0x7f7710103660 2024-09-19T08:05:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f7714e41640 1 -- 172.21.15.98:0/2958310489 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f771019d610 con 0x7f7710103660 2024-09-19T08:05:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f76ff7fe640 1 -- 172.21.15.98:0/2958310489 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f770400de50 con 0x7f7710103660 2024-09-19T08:05:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.649+0000 7f76ff7fe640 1 -- 172.21.15.98:0/2958310489 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770400f620 con 0x7f7710103660 2024-09-19T08:05:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.650+0000 7f7714e41640 1 -- 172.21.15.98:0/2958310489 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77101a4ef0 con 0x7f7710103660 2024-09-19T08:05:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.652+0000 7f76ff7fe640 1 -- 172.21.15.98:0/2958310489 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f7704009c30 con 0x7f7710103660 2024-09-19T08:05:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.652+0000 7f771721a740 1 Processor -- start 2024-09-19T08:05:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f771721a740 1 -- start start 2024-09-19T08:05:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f771721a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61060 0x5597e3f61440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f771721a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f61980 0x5597e3f6bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f771721a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f6c240 0x5597e3f6e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f771721a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5597e3ec6620 con 0x5597e3f6c240 2024-09-19T08:05:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f771721a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5597e3e6de50 con 0x5597e3f61060 2024-09-19T08:05:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f771721a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5597e3ec25e0 con 0x5597e3f61980 2024-09-19T08:05:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f76ff7fe640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f76f80a33b0 0x7f76f80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f770dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f6c240 0x5597e3f6e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f770ed9d640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f76f80a33b0 0x7f76f80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f770e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f61980 0x5597e3f6bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f770dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f6c240 0x5597e3f6e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48222/0 (socket says 172.21.15.98:48222) 2024-09-19T08:05:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f770e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f61980 0x5597e3f6bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:55390/0 (socket says 172.21.15.98:55390) 2024-09-19T08:05:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f770dd9b640 1 -- 172.21.15.98:0/626289820 learned_addr learned my addr 172.21.15.98:0/626289820 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f770ed9d640 1 --2- 172.21.15.98:0/626289820 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61060 0x5597e3f61440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f76ff7fe640 1 -- 172.21.15.98:0/2958310489 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f77040cf8e0 con 0x7f7710103660 2024-09-19T08:05:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f770dd9b640 1 -- 172.21.15.98:0/626289820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f61980 msgr2=0x5597e3f6bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f770dd9b640 1 --2- 172.21.15.98:0/626289820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f61980 0x5597e3f6bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f770dd9b640 1 -- 172.21.15.98:0/626289820 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61060 msgr2=0x5597e3f61440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f770dd9b640 1 --2- 172.21.15.98:0/626289820 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61060 0x5597e3f61440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f770dd9b640 1 -- 172.21.15.98:0/626289820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597e3e5ed40 con 0x5597e3f6c240 2024-09-19T08:05:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.653+0000 7f770e59c640 1 --2- 172.21.15.98:0/626289820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f61980 0x5597e3f6bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:05:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.654+0000 7f770dd9b640 1 --2- 172.21.15.98:0/626289820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f6c240 0x5597e3f6e630 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7704097ff0 tx=0x7f770406b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19bd77d96554cbe server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.654+0000 7f770ed9d640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f76f80a33b0 0x7f76f80a5870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7708031550 tx=0x7f7708032cb0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.654+0000 7f76fcff9640 1 -- 172.21.15.98:0/626289820 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704020070 con 0x5597e3f6c240 2024-09-19T08:05:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.654+0000 7f76fcff9640 1 -- 172.21.15.98:0/626289820 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f770409bb90 con 0x5597e3f6c240 2024-09-19T08:05:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.654+0000 7f76fcff9640 1 -- 172.21.15.98:0/626289820 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77040cd280 con 0x5597e3f6c240 2024-09-19T08:05:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.654+0000 7f771721a740 1 -- 172.21.15.98:0/626289820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f6c240 msgr2=0x5597e3f6e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.654+0000 7f771721a740 1 --2- 172.21.15.98:0/626289820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f6c240 0x5597e3f6e630 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7704097ff0 tx=0x7f770406b7b0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 -- 172.21.15.98:0/626289820 shutdown_connections 2024-09-19T08:05:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 --2- 172.21.15.98:0/626289820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f6c240 0x5597e3f6e630 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 --2- 172.21.15.98:0/626289820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f61980 0x5597e3f6bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 --2- 172.21.15.98:0/626289820 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61060 0x5597e3f61440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 -- 172.21.15.98:0/626289820 >> 172.21.15.98:0/626289820 conn(0x5597e3f5d5f0 msgr2=0x5597e3f5d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 -- 172.21.15.98:0/626289820 shutdown_connections 2024-09-19T08:05:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 -- 172.21.15.98:0/626289820 wait complete. 2024-09-19T08:05:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 -- start start 2024-09-19T08:05:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 Processor -- start 2024-09-19T08:05:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61060 0x5597e3f691a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61980 0x5597e3f696e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f6c240 0x5597e3f69c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5597e3f6edb0 con 0x5597e3f61980 2024-09-19T08:05:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5597e3e4ef10 con 0x5597e3f61060 2024-09-19T08:05:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5597e3f6ef30 con 0x5597e3f6c240 2024-09-19T08:05:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f771721a740 10 client.0 _ll_register_callbacks cb 0x5597e3e72db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f770dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61980 0x5597e3f696e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f770dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61980 0x5597e3f696e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48226/0 (socket says 172.21.15.98:48226) 2024-09-19T08:05:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.655+0000 7f770dd9b640 1 -- 172.21.15.98:0/1015682552 learned_addr learned my addr 172.21.15.98:0/1015682552 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f770ed9d640 1 --2- 172.21.15.98:0/1015682552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f6c240 0x5597e3f69c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f770e59c640 1 --2- 172.21.15.98:0/1015682552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61060 0x5597e3f691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f770dd9b640 1 -- 172.21.15.98:0/1015682552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f6c240 msgr2=0x5597e3f69c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f770dd9b640 1 --2- 172.21.15.98:0/1015682552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f6c240 0x5597e3f69c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f770dd9b640 1 -- 172.21.15.98:0/1015682552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61060 msgr2=0x5597e3f691a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f770dd9b640 1 --2- 172.21.15.98:0/1015682552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61060 0x5597e3f691a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f770dd9b640 1 -- 172.21.15.98:0/1015682552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597e3bd5580 con 0x5597e3f61980 2024-09-19T08:05:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f770e59c640 1 --2- 172.21.15.98:0/1015682552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61060 0x5597e3f691a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:05:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f770dd9b640 1 --2- 172.21.15.98:0/1015682552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61980 0x5597e3f696e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f77040a42d0 tx=0x7f770409c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a167776e1a08828c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f76f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f76f5ffb640 1 -- 172.21.15.98:0/1015682552 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704020070 con 0x5597e3f61980 2024-09-19T08:05:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f76f5ffb640 1 -- 172.21.15.98:0/1015682552 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f77040cd280 con 0x5597e3f61980 2024-09-19T08:05:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f771721a740 1 -- 172.21.15.98:0/1015682552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f770409bb90 con 0x5597e3f61980 2024-09-19T08:05:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f76f5ffb640 1 -- 172.21.15.98:0/1015682552 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f77040cd480 con 0x5597e3f61980 2024-09-19T08:05:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f771721a740 10 client.16032 fetch_fsmap learned FSMap version 29 2024-09-19T08:05:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f771721a740 1 -- 172.21.15.98:0/1015682552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5597e3c9f960 con 0x5597e3f61980 2024-09-19T08:05:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f76f5ffb640 1 -- 172.21.15.98:0/1015682552 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f770409bb90 con 0x5597e3f61980 2024-09-19T08:05:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f76f5ffb640 1 -- 172.21.15.98:0/1015682552 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f7704086070 con 0x5597e3f61980 2024-09-19T08:05:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f771721a740 10 client.16032 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T08:05:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.656+0000 7f771721a740 10 client.16032 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-19T08:05:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.657+0000 7f771721a740 20 client.16032 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.657+0000 7f771721a740 1 --2- 172.21.15.98:0/1015682552 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5597e3f7bc60 0x5597e3f7e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.657+0000 7f771721a740 4 client.16032 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:05:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.657+0000 7f771721a740 1 -- 172.21.15.98:0/1015682552 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597e3ed3b90 con 0x5597e3f7bc60 2024-09-19T08:05:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.657+0000 7f770e59c640 1 --2- 172.21.15.98:0/1015682552 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5597e3f7bc60 0x5597e3f7e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.657+0000 7f770e59c640 1 --2- 172.21.15.98:0/1015682552 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5597e3f7bc60 0x5597e3f7e050 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=462fdd5a481c589b server_cookie=ccf41fb02acb9b98 in_seq=0 out_seq=0 2024-09-19T08:05:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.657+0000 7f76f5ffb640 10 client.16032 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f76f5ffb640 1 -- 172.21.15.98:0/1015682552 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5597e3ed3b90 con 0x5597e3f7bc60 2024-09-19T08:05:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f76f5ffb640 10 client.16032 handle_command_reply: tid=0 2024-09-19T08:05:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f771721a740 1 client.16032 shutdown 2024-09-19T08:05:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f771721a740 1 -- 172.21.15.98:0/1015682552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61980 msgr2=0x5597e3f696e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f771721a740 1 --2- 172.21.15.98:0/1015682552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61980 0x5597e3f696e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f77040a42d0 tx=0x7f770409c3c0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f771721a740 1 -- 172.21.15.98:0/1015682552 shutdown_connections 2024-09-19T08:05:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f771721a740 1 --2- 172.21.15.98:0/1015682552 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5597e3f7bc60 0x5597e3f7e050 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f771721a740 1 --2- 172.21.15.98:0/1015682552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f6c240 0x5597e3f69c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f771721a740 1 --2- 172.21.15.98:0/1015682552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61980 0x5597e3f696e0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f771721a740 1 --2- 172.21.15.98:0/1015682552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61060 0x5597e3f691a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f771721a740 1 -- 172.21.15.98:0/1015682552 >> 172.21.15.98:0/1015682552 conn(0x5597e3f5d5f0 msgr2=0x5597e3f63320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f76f5ffb640 0 client.16032 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f771721a740 1 -- 172.21.15.98:0/1015682552 shutdown_connections 2024-09-19T08:05:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f771721a740 1 -- 172.21.15.98:0/1015682552 wait complete. 2024-09-19T08:05:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.658+0000 7f771721a740 20 client.16032 trim_cache size 0 max 16384 2024-09-19T08:05:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f771721a740 1 Processor -- start 2024-09-19T08:05:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f771721a740 1 -- start start 2024-09-19T08:05:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f771721a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61060 0x5597e3f80c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f771721a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f61980 0x5597e3f811d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f771721a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f6c240 0x5597e3f66400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f771721a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f77101ad7f0 con 0x5597e3f61060 2024-09-19T08:05:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f771721a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f77101af360 con 0x5597e3f6c240 2024-09-19T08:05:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f771721a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f77101adec0 con 0x5597e3f61980 2024-09-19T08:05:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f770e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61060 0x5597e3f80c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f770e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61060 0x5597e3f80c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48228/0 (socket says 172.21.15.98:48228) 2024-09-19T08:05:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f770e59c640 1 -- 172.21.15.98:0/2520731299 learned_addr learned my addr 172.21.15.98:0/2520731299 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f770ed9d640 1 --2- 172.21.15.98:0/2520731299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f6c240 0x5597e3f66400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f770e59c640 1 -- 172.21.15.98:0/2520731299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f61980 msgr2=0x5597e3f811d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f770e59c640 1 --2- 172.21.15.98:0/2520731299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f61980 0x5597e3f811d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f770e59c640 1 -- 172.21.15.98:0/2520731299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f6c240 msgr2=0x5597e3f66400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f770e59c640 1 --2- 172.21.15.98:0/2520731299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f6c240 0x5597e3f66400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f770e59c640 1 -- 172.21.15.98:0/2520731299 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597e3f5d420 con 0x5597e3f61060 2024-09-19T08:05:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.675+0000 7f770ed9d640 1 --2- 172.21.15.98:0/2520731299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f6c240 0x5597e3f66400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:05:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.676+0000 7f770e59c640 1 --2- 172.21.15.98:0/2520731299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61060 0x5597e3f80c90 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f7700004a80 tx=0x7f77000129c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0657421e7c3b9f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.676+0000 7f76f7fff640 1 -- 172.21.15.98:0/2520731299 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7700022030 con 0x5597e3f61060 2024-09-19T08:05:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.676+0000 7f76f7fff640 1 -- 172.21.15.98:0/2520731299 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f770000fa80 con 0x5597e3f61060 2024-09-19T08:05:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.676+0000 7f76f7fff640 1 -- 172.21.15.98:0/2520731299 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770001ec10 con 0x5597e3f61060 2024-09-19T08:05:09.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:09 smithi139 ceph-mon[34874]: pgmap v67: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-09-19T08:05:09.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.676+0000 7f771721a740 1 -- 172.21.15.98:0/2520731299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61060 msgr2=0x5597e3f80c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.676+0000 7f771721a740 1 --2- 172.21.15.98:0/2520731299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61060 0x5597e3f80c90 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f7700004a80 tx=0x7f77000129c0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.676+0000 7f771721a740 1 -- 172.21.15.98:0/2520731299 shutdown_connections 2024-09-19T08:05:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.676+0000 7f771721a740 1 --2- 172.21.15.98:0/2520731299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f6c240 0x5597e3f66400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.676+0000 7f771721a740 1 --2- 172.21.15.98:0/2520731299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f61980 0x5597e3f811d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.676+0000 7f771721a740 1 --2- 172.21.15.98:0/2520731299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61060 0x5597e3f80c90 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.676+0000 7f771721a740 1 -- 172.21.15.98:0/2520731299 >> 172.21.15.98:0/2520731299 conn(0x5597e3f5d5f0 msgr2=0x5597e3f67a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.676+0000 7f771721a740 1 -- 172.21.15.98:0/2520731299 shutdown_connections 2024-09-19T08:05:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.676+0000 7f771721a740 1 -- 172.21.15.98:0/2520731299 wait complete. 2024-09-19T08:05:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.677+0000 7f771721a740 1 -- start start 2024-09-19T08:05:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.677+0000 7f771721a740 1 Processor -- start 2024-09-19T08:05:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.677+0000 7f771721a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61060 0x5597e3f80cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.677+0000 7f771721a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61980 0x5597e3f811f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.677+0000 7f771721a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f6c240 0x5597e3f7bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.677+0000 7f771721a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f77101aec60 con 0x5597e3f61060 2024-09-19T08:05:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.677+0000 7f771721a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f77101ae590 con 0x5597e3f61980 2024-09-19T08:05:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.677+0000 7f771721a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f77101ab7d0 con 0x5597e3f6c240 2024-09-19T08:05:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.677+0000 7f771721a740 10 client.0 _ll_register_callbacks cb 0x7f770409b960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.677+0000 7f770e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61060 0x5597e3f80cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.677+0000 7f770e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61060 0x5597e3f80cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48244/0 (socket says 172.21.15.98:48244) 2024-09-19T08:05:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.677+0000 7f770e59c640 1 -- 172.21.15.98:0/2273674914 learned_addr learned my addr 172.21.15.98:0/2273674914 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.677+0000 7f770dd9b640 1 --2- 172.21.15.98:0/2273674914 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61980 0x5597e3f811f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f770ed9d640 1 --2- 172.21.15.98:0/2273674914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f6c240 0x5597e3f7bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f770e59c640 1 -- 172.21.15.98:0/2273674914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f6c240 msgr2=0x5597e3f7bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f770e59c640 1 --2- 172.21.15.98:0/2273674914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f6c240 0x5597e3f7bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f770e59c640 1 -- 172.21.15.98:0/2273674914 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61980 msgr2=0x5597e3f811f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f770e59c640 1 --2- 172.21.15.98:0/2273674914 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61980 0x5597e3f811f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f770e59c640 1 -- 172.21.15.98:0/2273674914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597e3f7c1a0 con 0x5597e3f61060 2024-09-19T08:05:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f770e59c640 1 --2- 172.21.15.98:0/2273674914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61060 0x5597e3f80cb0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f770001e4d0 tx=0x7f770001e500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3baebd58ab68073f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f76f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f76f5ffb640 1 -- 172.21.15.98:0/2273674914 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7700022040 con 0x5597e3f61060 2024-09-19T08:05:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f76f5ffb640 1 -- 172.21.15.98:0/2273674914 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f7700004ab0 con 0x5597e3f61060 2024-09-19T08:05:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f771721a740 1 -- 172.21.15.98:0/2273674914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f770000fa80 con 0x5597e3f61060 2024-09-19T08:05:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f76f5ffb640 1 -- 172.21.15.98:0/2273674914 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f770000fdf0 con 0x5597e3f61060 2024-09-19T08:05:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f771721a740 10 client.16038 fetch_fsmap learned FSMap version 29 2024-09-19T08:05:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f771721a740 1 -- 172.21.15.98:0/2273674914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5597e3fef940 con 0x5597e3f61060 2024-09-19T08:05:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f76f5ffb640 1 -- 172.21.15.98:0/2273674914 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f770000fa80 con 0x5597e3f61060 2024-09-19T08:05:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.678+0000 7f76f5ffb640 1 -- 172.21.15.98:0/2273674914 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f77000053f0 con 0x5597e3f61060 2024-09-19T08:05:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.679+0000 7f771721a740 10 client.16038 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T08:05:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.679+0000 7f771721a740 10 client.16038 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-19T08:05:09.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.679+0000 7f771721a740 20 client.16038 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:09.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.679+0000 7f771721a740 1 --2- 172.21.15.98:0/2273674914 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5597e3ff0000 0x5597e3ff2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:09.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.679+0000 7f771721a740 4 client.16038 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "8bad5758", "release": true, "await": true}] 2024-09-19T08:05:09.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.679+0000 7f771721a740 1 -- 172.21.15.98:0/2273674914 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "quiesce db", "set_id": "8bad5758", "release": true, "await": true}) -- 0x5597e3ece0a0 con 0x5597e3ff0000 2024-09-19T08:05:09.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.679+0000 7f770dd9b640 1 --2- 172.21.15.98:0/2273674914 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5597e3ff0000 0x5597e3ff2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:09.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.679+0000 7f770dd9b640 1 --2- 172.21.15.98:0/2273674914 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5597e3ff0000 0x5597e3ff2420 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc8fabe446811048 server_cookie=e57cadf7a313ad93 in_seq=0 out_seq=0 2024-09-19T08:05:09.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.679+0000 7f76f5ffb640 10 client.16038 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:09.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.681+0000 7f76f5ffb640 1 -- 172.21.15.98:0/2273674914 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5597e3ece0a0 con 0x5597e3ff0000 2024-09-19T08:05:09.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.681+0000 7f76f5ffb640 10 client.16038 handle_command_reply: tid=0 2024-09-19T08:05:09.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.681+0000 7f771721a740 1 client.16038 shutdown 2024-09-19T08:05:09.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f771721a740 1 -- 172.21.15.98:0/2273674914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61060 msgr2=0x5597e3f80cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:09.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f771721a740 1 --2- 172.21.15.98:0/2273674914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61060 0x5597e3f80cb0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f770001e4d0 tx=0x7f770001e500 comp rx=0 tx=0).stop 2024-09-19T08:05:09.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f771721a740 1 -- 172.21.15.98:0/2273674914 shutdown_connections 2024-09-19T08:05:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f771721a740 1 --2- 172.21.15.98:0/2273674914 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5597e3ff0000 0x5597e3ff2420 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f771721a740 1 --2- 172.21.15.98:0/2273674914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5597e3f6c240 0x5597e3f7bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f771721a740 1 --2- 172.21.15.98:0/2273674914 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5597e3f61980 0x5597e3f811f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f76f5ffb640 0 client.16038 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f771721a740 1 --2- 172.21.15.98:0/2273674914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5597e3f61060 0x5597e3f80cb0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f771721a740 1 -- 172.21.15.98:0/2273674914 >> 172.21.15.98:0/2273674914 conn(0x5597e3f5d5f0 msgr2=0x5597e3f5bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f771721a740 1 -- 172.21.15.98:0/2273674914 shutdown_connections 2024-09-19T08:05:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f771721a740 1 -- 172.21.15.98:0/2273674914 wait complete. 2024-09-19T08:05:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f771721a740 20 client.16038 trim_cache size 0 max 16384 2024-09-19T08:05:09.686 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:05:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 29, 2024-09-19T08:05:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24485, 2024-09-19T08:05:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 12, 2024-09-19T08:05:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-09-19T08:05:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "8bad5758": { 2024-09-19T08:05:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 12, 2024-09-19T08:05:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-09-19T08:05:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-09-19T08:05:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-09-19T08:05:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-09-19T08:05:09.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:05:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 90.0, 2024-09-19T08:05:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 140.3, 2024-09-19T08:05:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-09-19T08:05:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-09-19T08:05:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-09-19T08:05:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-09-19T08:05:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-09-19T08:05:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-09-19T08:05:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:05:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:05:09.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:05:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:05:09.689 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:05:09.689 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:05:09.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f7714e41640 1 -- 172.21.15.98:0/2958310489 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f76f80a33b0 msgr2=0x7f76f80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:09.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f7714e41640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f76f80a33b0 0x7f76f80a5870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7708031550 tx=0x7f7708032cb0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f7714e41640 1 -- 172.21.15.98:0/2958310489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7710103660 msgr2=0x7f77101988b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.682+0000 7f7714e41640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7710103660 0x7f77101988b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f770400d740 tx=0x7f770400dc00 comp rx=0 tx=0).stop 2024-09-19T08:05:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.683+0000 7f7714e41640 1 -- 172.21.15.98:0/2958310489 shutdown_connections 2024-09-19T08:05:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.683+0000 7f7714e41640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f76f80a33b0 0x7f76f80a5870 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.683+0000 7f7714e41640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f771010c170 0x7f771019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.683+0000 7f7714e41640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7710103660 0x7f77101988b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.683+0000 7f7714e41640 1 --2- 172.21.15.98:0/2958310489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7710102d40 0x7f7710198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.683+0000 7f7714e41640 1 -- 172.21.15.98:0/2958310489 >> 172.21.15.98:0/2958310489 conn(0x7f77100fc7a0 msgr2=0x7f77100fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.683+0000 7f7714e41640 1 -- 172.21.15.98:0/2958310489 shutdown_connections 2024-09-19T08:05:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:09.683+0000 7f7714e41640 1 -- 172.21.15.98:0/2958310489 wait complete. 2024-09-19T08:05:10.467 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '8bad5758', total seconds elapsed: 23.5 2024-09-19T08:05:11.227 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:11.227 INFO:tasks.workunit.client.0.smithi098.stdout: Children see throughput for 1 rewriters = 55639.17 kB/sec 2024-09-19T08:05:11.227 INFO:tasks.workunit.client.0.smithi098.stdout: Parent sees throughput for 1 rewriters = 55638.55 kB/sec 2024-09-19T08:05:11.228 INFO:tasks.workunit.client.0.smithi098.stdout: Min throughput per process = 55639.17 kB/sec 2024-09-19T08:05:11.228 INFO:tasks.workunit.client.0.smithi098.stdout: Max throughput per process = 55639.17 kB/sec 2024-09-19T08:05:11.228 INFO:tasks.workunit.client.0.smithi098.stdout: Avg throughput per process = 55639.17 kB/sec 2024-09-19T08:05:11.228 INFO:tasks.workunit.client.0.smithi098.stdout: Min xfer = 1048576.00 kB 2024-09-19T08:05:11.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:10 smithi162 ceph-mon[35260]: pgmap v68: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-09-19T08:05:11.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:10 smithi139 ceph-mon[34874]: pgmap v68: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-09-19T08:05:11.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:10 smithi098 ceph-mon[31333]: pgmap v68: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-09-19T08:05:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:11 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:05:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:11 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:11 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:11 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:11 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:11 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:11 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:11 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:05:12.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:11 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:05:12.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:11 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:11 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:05:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:11 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:11 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:11 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:11 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:11 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:11 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:11 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:05:12.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:11 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:05:12.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:11 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:11 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:05:12.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:11 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:11 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:11 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:11 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:11 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:11 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:12.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:11 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:05:12.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:11 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:05:12.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:11 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:05:13.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:12 smithi162 ceph-mon[35260]: pgmap v69: 97 pgs: 97 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 85 MiB/s wr, 49 op/s 2024-09-19T08:05:13.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:12 smithi139 ceph-mon[34874]: pgmap v69: 97 pgs: 97 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 85 MiB/s wr, 49 op/s 2024-09-19T08:05:13.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:12 smithi098 ceph-mon[31333]: pgmap v69: 97 pgs: 97 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 85 MiB/s wr, 49 op/s 2024-09-19T08:05:13.673 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:13.673 INFO:tasks.workunit.client.0.smithi098.stdout: Children see throughput for 1 readers = 7063310.50 kB/sec 2024-09-19T08:05:13.673 INFO:tasks.workunit.client.0.smithi098.stdout: Parent sees throughput for 1 readers = 7052527.58 kB/sec 2024-09-19T08:05:13.673 INFO:tasks.workunit.client.0.smithi098.stdout: Min throughput per process = 7063310.50 kB/sec 2024-09-19T08:05:13.673 INFO:tasks.workunit.client.0.smithi098.stdout: Max throughput per process = 7063310.50 kB/sec 2024-09-19T08:05:13.674 INFO:tasks.workunit.client.0.smithi098.stdout: Avg throughput per process = 7063310.50 kB/sec 2024-09-19T08:05:13.674 INFO:tasks.workunit.client.0.smithi098.stdout: Min xfer = 1048576.00 kB 2024-09-19T08:05:13.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:05:14.233 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:05:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.655+0000 7f19186f1640 1 -- 172.21.15.98:0/911234066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1910106a90 msgr2=0x7f1910106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.655+0000 7f19186f1640 1 --2- 172.21.15.98:0/911234066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1910106a90 0x7f1910106e70 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f190c009cd0 tx=0x7f190c02a8e0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.655+0000 7f19186f1640 1 -- 172.21.15.98:0/911234066 shutdown_connections 2024-09-19T08:05:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.655+0000 7f19186f1640 1 --2- 172.21.15.98:0/911234066 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1910101430 0x7f191010d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.655+0000 7f19186f1640 1 --2- 172.21.15.98:0/911234066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1910100a90 0x7f1910100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.655+0000 7f19186f1640 1 --2- 172.21.15.98:0/911234066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1910106a90 0x7f1910106e70 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.655+0000 7f19186f1640 1 -- 172.21.15.98:0/911234066 >> 172.21.15.98:0/911234066 conn(0x7f19100fc7d0 msgr2=0x7f19100febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.655+0000 7f19186f1640 1 -- 172.21.15.98:0/911234066 shutdown_connections 2024-09-19T08:05:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.655+0000 7f19186f1640 1 -- 172.21.15.98:0/911234066 wait complete. 2024-09-19T08:05:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f19186f1640 1 Processor -- start 2024-09-19T08:05:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f19186f1640 1 -- start start 2024-09-19T08:05:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f19186f1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1910100a90 0x7f191010e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f19186f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1910101430 0x7f191010e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f19186f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1910106a90 0x7f191010ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f19186f1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1910111fb0 con 0x7f1910106a90 2024-09-19T08:05:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f19186f1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1910111e30 con 0x7f1910100a90 2024-09-19T08:05:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f19186f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1910112130 con 0x7f1910101430 2024-09-19T08:05:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f1916c8e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1910106a90 0x7f191010ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f1916c8e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1910106a90 0x7f191010ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48254/0 (socket says 172.21.15.98:48254) 2024-09-19T08:05:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f1916c8e640 1 -- 172.21.15.98:0/816852459 learned_addr learned my addr 172.21.15.98:0/816852459 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f1915c8c640 1 --2- 172.21.15.98:0/816852459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1910101430 0x7f191010e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f191648d640 1 --2- 172.21.15.98:0/816852459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1910100a90 0x7f191010e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f1916c8e640 1 -- 172.21.15.98:0/816852459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1910101430 msgr2=0x7f191010e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f1916c8e640 1 --2- 172.21.15.98:0/816852459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1910101430 0x7f191010e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f1916c8e640 1 -- 172.21.15.98:0/816852459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1910100a90 msgr2=0x7f191010e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f1916c8e640 1 --2- 172.21.15.98:0/816852459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1910100a90 0x7f191010e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.656+0000 7f1916c8e640 1 -- 172.21.15.98:0/816852459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19101a68f0 con 0x7f1910106a90 2024-09-19T08:05:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.657+0000 7f1916c8e640 1 --2- 172.21.15.98:0/816852459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1910106a90 0x7f191010ee70 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f190800c970 tx=0x7f190800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.657+0000 7f19077fe640 1 -- 172.21.15.98:0/816852459 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908007bb0 con 0x7f1910106a90 2024-09-19T08:05:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.657+0000 7f19186f1640 1 -- 172.21.15.98:0/816852459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19101a6be0 con 0x7f1910106a90 2024-09-19T08:05:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.657+0000 7f19077fe640 1 -- 172.21.15.98:0/816852459 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1908007d50 con 0x7f1910106a90 2024-09-19T08:05:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.657+0000 7f19077fe640 1 -- 172.21.15.98:0/816852459 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908018800 con 0x7f1910106a90 2024-09-19T08:05:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.657+0000 7f19186f1640 1 -- 172.21.15.98:0/816852459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19101a7120 con 0x7f1910106a90 2024-09-19T08:05:14.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.659+0000 7f19077fe640 1 -- 172.21.15.98:0/816852459 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f190800fa50 con 0x7f1910106a90 2024-09-19T08:05:14.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.660+0000 7f191aaca740 1 Processor -- start 2024-09-19T08:05:14.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.660+0000 7f191aaca740 1 -- start start 2024-09-19T08:05:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.660+0000 7f19077fe640 1 --2- 172.21.15.98:0/816852459 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f18fc0a33b0 0x7f18fc0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.660+0000 7f191aaca740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eae4a0 0x558f89eae880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.660+0000 7f191aaca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eaedc0 0x558f89eb9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.660+0000 7f191aaca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eb9680 0x558f89ebba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.660+0000 7f191aaca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558f89e142e0 con 0x558f89eaedc0 2024-09-19T08:05:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.660+0000 7f191aaca740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558f89dbbce0 con 0x558f89eae4a0 2024-09-19T08:05:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.660+0000 7f191aaca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558f89e102a0 con 0x558f89eb9680 2024-09-19T08:05:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.660+0000 7f1915c8c640 1 --2- 172.21.15.98:0/816852459 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f18fc0a33b0 0x7f18fc0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.660+0000 7f19077fe640 1 -- 172.21.15.98:0/816852459 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f19080cfc80 con 0x7f1910106a90 2024-09-19T08:05:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.660+0000 7f1915c8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eaedc0 0x558f89eb9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.660+0000 7f1915c8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eaedc0 0x558f89eb9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48266/0 (socket says 172.21.15.98:48266) 2024-09-19T08:05:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.660+0000 7f1915c8c640 1 -- 172.21.15.98:0/3662814352 learned_addr learned my addr 172.21.15.98:0/3662814352 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.661+0000 7f1916c8e640 1 --2- 172.21.15.98:0/3662814352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eb9680 0x558f89ebba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.661+0000 7f191648d640 1 --2- 172.21.15.98:0/3662814352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eae4a0 0x558f89eae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.661+0000 7f1915c8c640 1 -- 172.21.15.98:0/3662814352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eb9680 msgr2=0x558f89ebba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.661+0000 7f1915c8c640 1 --2- 172.21.15.98:0/3662814352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eb9680 0x558f89ebba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.661+0000 7f1915c8c640 1 -- 172.21.15.98:0/3662814352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eae4a0 msgr2=0x558f89eae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.661+0000 7f1915c8c640 1 --2- 172.21.15.98:0/3662814352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eae4a0 0x558f89eae880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.661+0000 7f1915c8c640 1 -- 172.21.15.98:0/3662814352 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f89da0440 con 0x558f89eaedc0 2024-09-19T08:05:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.661+0000 7f1915c8c640 1 --2- 172.21.15.98:0/816852459 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f18fc0a33b0 0x7f18fc0a5870 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1900005d00 tx=0x7f190000bb90 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.661+0000 7f1915c8c640 1 --2- 172.21.15.98:0/3662814352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eaedc0 0x558f89eb9140 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f19000123f0 tx=0x7f190001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f916d3a4af04d82 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.661+0000 7f1904ff9640 1 -- 172.21.15.98:0/3662814352 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1900024070 con 0x558f89eaedc0 2024-09-19T08:05:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.661+0000 7f1904ff9640 1 -- 172.21.15.98:0/3662814352 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1900010e00 con 0x558f89eaedc0 2024-09-19T08:05:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.661+0000 7f1904ff9640 1 -- 172.21.15.98:0/3662814352 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1900004d60 con 0x558f89eaedc0 2024-09-19T08:05:14.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.662+0000 7f191aaca740 1 -- 172.21.15.98:0/3662814352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eaedc0 msgr2=0x558f89eb9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:14.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.662+0000 7f191aaca740 1 --2- 172.21.15.98:0/3662814352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eaedc0 0x558f89eb9140 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f19000123f0 tx=0x7f190001c000 comp rx=0 tx=0).stop 2024-09-19T08:05:14.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.662+0000 7f191aaca740 1 -- 172.21.15.98:0/3662814352 shutdown_connections 2024-09-19T08:05:14.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.662+0000 7f191aaca740 1 --2- 172.21.15.98:0/3662814352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eb9680 0x558f89ebba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.662+0000 7f191aaca740 1 --2- 172.21.15.98:0/3662814352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eaedc0 0x558f89eb9140 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.662+0000 7f191aaca740 1 --2- 172.21.15.98:0/3662814352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eae4a0 0x558f89eae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.662+0000 7f191aaca740 1 -- 172.21.15.98:0/3662814352 >> 172.21.15.98:0/3662814352 conn(0x558f89eaaa30 msgr2=0x558f89eaae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.662+0000 7f191aaca740 1 -- 172.21.15.98:0/3662814352 shutdown_connections 2024-09-19T08:05:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.662+0000 7f191aaca740 1 -- 172.21.15.98:0/3662814352 wait complete. 2024-09-19T08:05:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.662+0000 7f191aaca740 1 -- start start 2024-09-19T08:05:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.662+0000 7f191aaca740 1 Processor -- start 2024-09-19T08:05:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191aaca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eae4a0 0x558f89eb8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191aaca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eaedc0 0x558f89eb6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191aaca740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eb9680 0x558f89eb6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191aaca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558f89ebc250 con 0x558f89eae4a0 2024-09-19T08:05:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191aaca740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558f89dcf050 con 0x558f89eb9680 2024-09-19T08:05:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191aaca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558f89ebc3d0 con 0x558f89eaedc0 2024-09-19T08:05:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191aaca740 10 client.0 _ll_register_callbacks cb 0x558f89e0f160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191648d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eae4a0 0x558f89eb8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191648d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eae4a0 0x558f89eb8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48278/0 (socket says 172.21.15.98:48278) 2024-09-19T08:05:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191648d640 1 -- 172.21.15.98:0/2683220729 learned_addr learned my addr 172.21.15.98:0/2683220729 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f1915c8c640 1 --2- 172.21.15.98:0/2683220729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eaedc0 0x558f89eb6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f1916c8e640 1 --2- 172.21.15.98:0/2683220729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eb9680 0x558f89eb6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191648d640 1 -- 172.21.15.98:0/2683220729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eaedc0 msgr2=0x558f89eb6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191648d640 1 --2- 172.21.15.98:0/2683220729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eaedc0 0x558f89eb6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191648d640 1 -- 172.21.15.98:0/2683220729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eb9680 msgr2=0x558f89eb6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191648d640 1 --2- 172.21.15.98:0/2683220729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eb9680 0x558f89eb6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191648d640 1 -- 172.21.15.98:0/2683220729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f89b853c0 con 0x558f89eae4a0 2024-09-19T08:05:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f1916c8e640 1 --2- 172.21.15.98:0/2683220729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eb9680 0x558f89eb6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:05:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f191648d640 1 --2- 172.21.15.98:0/2683220729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eae4a0 0x558f89eb8f60 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f190c009cd0 tx=0x7f190c031cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd7f82bfa6df89c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.663+0000 7f18f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.664+0000 7f18f9ffb640 1 -- 172.21.15.98:0/2683220729 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c049060 con 0x558f89eae4a0 2024-09-19T08:05:14.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.664+0000 7f18f9ffb640 1 -- 172.21.15.98:0/2683220729 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f190c003b80 con 0x558f89eae4a0 2024-09-19T08:05:14.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.664+0000 7f191aaca740 1 -- 172.21.15.98:0/2683220729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1900010e00 con 0x558f89eae4a0 2024-09-19T08:05:14.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.664+0000 7f18f9ffb640 1 -- 172.21.15.98:0/2683220729 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f190c003d80 con 0x558f89eae4a0 2024-09-19T08:05:14.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.664+0000 7f191aaca740 10 client.16050 fetch_fsmap learned FSMap version 29 2024-09-19T08:05:14.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.664+0000 7f191aaca740 1 -- 172.21.15.98:0/2683220729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558f89ec7c10 con 0x558f89eae4a0 2024-09-19T08:05:14.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.664+0000 7f18f9ffb640 1 -- 172.21.15.98:0/2683220729 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1900010e00 con 0x558f89eae4a0 2024-09-19T08:05:14.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.664+0000 7f18f9ffb640 1 -- 172.21.15.98:0/2683220729 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f190c043070 con 0x558f89eae4a0 2024-09-19T08:05:14.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.664+0000 7f191aaca740 10 client.16050 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T08:05:14.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.664+0000 7f191aaca740 10 client.16050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:05:14.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.664+0000 7f191aaca740 20 client.16050 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:14.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.664+0000 7f191aaca740 1 --2- 172.21.15.98:0/2683220729 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558f89ec90d0 0x558f89ecb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.664+0000 7f191aaca740 4 client.16050 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:05:14.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.664+0000 7f191aaca740 1 -- 172.21.15.98:0/2683220729 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f89e21850 con 0x558f89ec90d0 2024-09-19T08:05:14.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.665+0000 7f1915c8c640 1 --2- 172.21.15.98:0/2683220729 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558f89ec90d0 0x558f89ecb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.665+0000 7f1915c8c640 1 --2- 172.21.15.98:0/2683220729 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558f89ec90d0 0x558f89ecb4e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb4c8108ca27b4ea server_cookie=f8d74812181fdc96 in_seq=0 out_seq=0 2024-09-19T08:05:14.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.665+0000 7f18f9ffb640 10 client.16050 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:14.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f18f9ffb640 1 -- 172.21.15.98:0/2683220729 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558f89e21850 con 0x558f89ec90d0 2024-09-19T08:05:14.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f18f9ffb640 10 client.16050 handle_command_reply: tid=0 2024-09-19T08:05:14.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f191aaca740 1 client.16050 shutdown 2024-09-19T08:05:14.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f191aaca740 1 -- 172.21.15.98:0/2683220729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eae4a0 msgr2=0x558f89eb8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:14.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f191aaca740 1 --2- 172.21.15.98:0/2683220729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eae4a0 0x558f89eb8f60 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f190c009cd0 tx=0x7f190c031cc0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f191aaca740 1 -- 172.21.15.98:0/2683220729 shutdown_connections 2024-09-19T08:05:14.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f191aaca740 1 --2- 172.21.15.98:0/2683220729 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558f89ec90d0 0x558f89ecb4e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f191aaca740 1 --2- 172.21.15.98:0/2683220729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eb9680 0x558f89eb6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f191aaca740 1 --2- 172.21.15.98:0/2683220729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eaedc0 0x558f89eb6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f191aaca740 1 --2- 172.21.15.98:0/2683220729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eae4a0 0x558f89eb8f60 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f191aaca740 1 -- 172.21.15.98:0/2683220729 >> 172.21.15.98:0/2683220729 conn(0x558f89eaaa30 msgr2=0x558f89ea9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:14.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f18f9ffb640 0 client.16050 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:14.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f191aaca740 1 -- 172.21.15.98:0/2683220729 shutdown_connections 2024-09-19T08:05:14.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f191aaca740 1 -- 172.21.15.98:0/2683220729 wait complete. 2024-09-19T08:05:14.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.666+0000 7f191aaca740 20 client.16050 trim_cache size 0 max 16384 2024-09-19T08:05:14.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f191aaca740 1 Processor -- start 2024-09-19T08:05:14.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f191aaca740 1 -- start start 2024-09-19T08:05:14.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f191aaca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eae4a0 0x558f89ed0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f191aaca740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eaedc0 0x558f89ed12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f191aaca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eb9680 0x558f89eb3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f191aaca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19101af920 con 0x558f89eb9680 2024-09-19T08:05:14.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f191aaca740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f19101b1490 con 0x558f89eaedc0 2024-09-19T08:05:14.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f191aaca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f19101afff0 con 0x558f89eae4a0 2024-09-19T08:05:14.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f1916c8e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eb9680 0x558f89eb3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f1916c8e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eb9680 0x558f89eb3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48290/0 (socket says 172.21.15.98:48290) 2024-09-19T08:05:14.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f1916c8e640 1 -- 172.21.15.98:0/1933559309 learned_addr learned my addr 172.21.15.98:0/1933559309 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:14.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f191648d640 1 --2- 172.21.15.98:0/1933559309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eae4a0 0x558f89ed0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f1915c8c640 1 --2- 172.21.15.98:0/1933559309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eaedc0 0x558f89ed12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f1916c8e640 1 -- 172.21.15.98:0/1933559309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eae4a0 msgr2=0x558f89ed0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f1916c8e640 1 --2- 172.21.15.98:0/1933559309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eae4a0 0x558f89ed0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f1916c8e640 1 -- 172.21.15.98:0/1933559309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eaedc0 msgr2=0x558f89ed12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f1916c8e640 1 --2- 172.21.15.98:0/1933559309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eaedc0 0x558f89ed12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f1916c8e640 1 -- 172.21.15.98:0/1933559309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f89eb7590 con 0x558f89eb9680 2024-09-19T08:05:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.683+0000 7f1915c8c640 1 --2- 172.21.15.98:0/1933559309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eaedc0 0x558f89ed12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:05:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.684+0000 7f1916c8e640 1 --2- 172.21.15.98:0/1933559309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eb9680 0x558f89eb3470 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f190809bec0 tx=0x7f190809ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea6f7c4cd0ed5a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.684+0000 7f18fbfff640 1 -- 172.21.15.98:0/1933559309 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190809c630 con 0x558f89eb9680 2024-09-19T08:05:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.684+0000 7f18fbfff640 1 -- 172.21.15.98:0/1933559309 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19080133b0 con 0x558f89eb9680 2024-09-19T08:05:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.684+0000 7f18fbfff640 1 -- 172.21.15.98:0/1933559309 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19080cd620 con 0x558f89eb9680 2024-09-19T08:05:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.684+0000 7f191aaca740 1 -- 172.21.15.98:0/1933559309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eb9680 msgr2=0x558f89eb3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.684+0000 7f191aaca740 1 --2- 172.21.15.98:0/1933559309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eb9680 0x558f89eb3470 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f190809bec0 tx=0x7f190809ba70 comp rx=0 tx=0).stop 2024-09-19T08:05:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.684+0000 7f191aaca740 1 -- 172.21.15.98:0/1933559309 shutdown_connections 2024-09-19T08:05:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.684+0000 7f191aaca740 1 --2- 172.21.15.98:0/1933559309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eb9680 0x558f89eb3470 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.684+0000 7f191aaca740 1 --2- 172.21.15.98:0/1933559309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eaedc0 0x558f89ed12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.684+0000 7f191aaca740 1 --2- 172.21.15.98:0/1933559309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eae4a0 0x558f89ed0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.684+0000 7f191aaca740 1 -- 172.21.15.98:0/1933559309 >> 172.21.15.98:0/1933559309 conn(0x558f89eaaa30 msgr2=0x558f89ea9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.684+0000 7f191aaca740 1 -- 172.21.15.98:0/1933559309 shutdown_connections 2024-09-19T08:05:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.684+0000 7f191aaca740 1 -- 172.21.15.98:0/1933559309 wait complete. 2024-09-19T08:05:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.685+0000 7f191aaca740 1 -- start start 2024-09-19T08:05:14.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.685+0000 7f191aaca740 1 Processor -- start 2024-09-19T08:05:14.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.685+0000 7f191aaca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eae4a0 0x558f89ec1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.685+0000 7f191aaca740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eaedc0 0x558f89ebf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.685+0000 7f191aaca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eb9680 0x558f89ebf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.685+0000 7f191aaca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19101b0d90 con 0x558f89eae4a0 2024-09-19T08:05:14.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.685+0000 7f191aaca740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f19101b06c0 con 0x558f89eaedc0 2024-09-19T08:05:14.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.685+0000 7f191aaca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f19101ad900 con 0x558f89eb9680 2024-09-19T08:05:14.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.685+0000 7f191aaca740 10 client.0 _ll_register_callbacks cb 0x7f190c0098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:14.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.685+0000 7f191648d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eae4a0 0x558f89ec1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.685+0000 7f191648d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eae4a0 0x558f89ec1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48306/0 (socket says 172.21.15.98:48306) 2024-09-19T08:05:14.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.685+0000 7f191648d640 1 -- 172.21.15.98:0/2944014452 learned_addr learned my addr 172.21.15.98:0/2944014452 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:14.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.685+0000 7f1915c8c640 1 --2- 172.21.15.98:0/2944014452 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eaedc0 0x558f89ebf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.685+0000 7f1916c8e640 1 --2- 172.21.15.98:0/2944014452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eb9680 0x558f89ebf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f191648d640 1 -- 172.21.15.98:0/2944014452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eb9680 msgr2=0x558f89ebf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:14.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f191648d640 1 --2- 172.21.15.98:0/2944014452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eb9680 0x558f89ebf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f191648d640 1 -- 172.21.15.98:0/2944014452 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eaedc0 msgr2=0x558f89ebf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:14.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f191648d640 1 --2- 172.21.15.98:0/2944014452 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eaedc0 0x558f89ebf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f191648d640 1 -- 172.21.15.98:0/2944014452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f89ebfc30 con 0x558f89eae4a0 2024-09-19T08:05:14.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f1915c8c640 1 --2- 172.21.15.98:0/2944014452 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eaedc0 0x558f89ebf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:05:14.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f191648d640 1 --2- 172.21.15.98:0/2944014452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eae4a0 0x558f89ec1cb0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f190c009ca0 tx=0x7f190c0307d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfd3db78dc8f2def server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:14.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f18f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:14.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f18f9ffb640 1 -- 172.21.15.98:0/2944014452 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c049050 con 0x558f89eae4a0 2024-09-19T08:05:14.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f18f9ffb640 1 -- 172.21.15.98:0/2944014452 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f190c041040 con 0x558f89eae4a0 2024-09-19T08:05:14.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f191aaca740 1 -- 172.21.15.98:0/2944014452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19080133b0 con 0x558f89eae4a0 2024-09-19T08:05:14.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f18f9ffb640 1 -- 172.21.15.98:0/2944014452 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f190c05b050 con 0x558f89eae4a0 2024-09-19T08:05:14.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f191aaca740 10 client.16056 fetch_fsmap learned FSMap version 29 2024-09-19T08:05:14.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f191aaca740 1 -- 172.21.15.98:0/2944014452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558f89f33bf0 con 0x558f89eae4a0 2024-09-19T08:05:14.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f18f9ffb640 1 -- 172.21.15.98:0/2944014452 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19080133b0 con 0x558f89eae4a0 2024-09-19T08:05:14.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f18f9ffb640 1 -- 172.21.15.98:0/2944014452 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f190c043040 con 0x558f89eae4a0 2024-09-19T08:05:14.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f191aaca740 10 client.16056 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T08:05:14.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.686+0000 7f191aaca740 10 client.16056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:05:14.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.687+0000 7f191aaca740 20 client.16056 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:14.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.687+0000 7f191aaca740 1 --2- 172.21.15.98:0/2944014452 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558f89f34370 0x558f89f36790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:14.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.687+0000 7f191aaca740 4 client.16056 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:05:14.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.687+0000 7f191aaca740 1 -- 172.21.15.98:0/2944014452 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558f89e1bd60 con 0x558f89f34370 2024-09-19T08:05:14.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.687+0000 7f1915c8c640 1 --2- 172.21.15.98:0/2944014452 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558f89f34370 0x558f89f36790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:14.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.687+0000 7f18f9ffb640 10 client.16056 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:14.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.687+0000 7f1915c8c640 1 --2- 172.21.15.98:0/2944014452 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558f89f34370 0x558f89f36790 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d85f4515cbe7a8cb server_cookie=a7beefc9b1e82600 in_seq=0 out_seq=0 2024-09-19T08:05:14.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.687+0000 7f18f9ffb640 1 -- 172.21.15.98:0/2944014452 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558f89e1bd60 con 0x558f89f34370 2024-09-19T08:05:14.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.687+0000 7f18f9ffb640 10 client.16056 handle_command_reply: tid=0 2024-09-19T08:05:14.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.687+0000 7f191aaca740 1 client.16056 shutdown 2024-09-19T08:05:14.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f191aaca740 1 -- 172.21.15.98:0/2944014452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eae4a0 msgr2=0x558f89ec1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:14.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f191aaca740 1 --2- 172.21.15.98:0/2944014452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eae4a0 0x558f89ec1cb0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f190c009ca0 tx=0x7f190c0307d0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f191aaca740 1 -- 172.21.15.98:0/2944014452 shutdown_connections 2024-09-19T08:05:14.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f191aaca740 1 --2- 172.21.15.98:0/2944014452 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x558f89f34370 0x558f89f36790 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f191aaca740 1 --2- 172.21.15.98:0/2944014452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f89eb9680 0x558f89ebf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f191aaca740 1 --2- 172.21.15.98:0/2944014452 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f89eaedc0 0x558f89ebf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f18f9ffb640 0 client.16056 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:14.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f191aaca740 1 --2- 172.21.15.98:0/2944014452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f89eae4a0 0x558f89ec1cb0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f191aaca740 1 -- 172.21.15.98:0/2944014452 >> 172.21.15.98:0/2944014452 conn(0x558f89eaaa30 msgr2=0x558f89eb7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:14.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f191aaca740 1 -- 172.21.15.98:0/2944014452 shutdown_connections 2024-09-19T08:05:14.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f191aaca740 1 -- 172.21.15.98:0/2944014452 wait complete. 2024-09-19T08:05:14.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f191aaca740 20 client.16056 trim_cache size 0 max 16384 2024-09-19T08:05:14.697 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:05:14.697 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:05:14.697 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:05:14.697 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:05:14.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f19186f1640 1 -- 172.21.15.98:0/816852459 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f18fc0a33b0 msgr2=0x7f18fc0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:14.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f19186f1640 1 --2- 172.21.15.98:0/816852459 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f18fc0a33b0 0x7f18fc0a5870 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1900005d00 tx=0x7f190000bb90 comp rx=0 tx=0).stop 2024-09-19T08:05:14.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f19186f1640 1 -- 172.21.15.98:0/816852459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1910106a90 msgr2=0x7f191010ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:14.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.688+0000 7f19186f1640 1 --2- 172.21.15.98:0/816852459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1910106a90 0x7f191010ee70 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f190800c970 tx=0x7f190800ce30 comp rx=0 tx=0).stop 2024-09-19T08:05:14.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.689+0000 7f19186f1640 1 -- 172.21.15.98:0/816852459 shutdown_connections 2024-09-19T08:05:14.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.689+0000 7f19186f1640 1 --2- 172.21.15.98:0/816852459 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f18fc0a33b0 0x7f18fc0a5870 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.689+0000 7f19186f1640 1 --2- 172.21.15.98:0/816852459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1910106a90 0x7f191010ee70 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.689+0000 7f19186f1640 1 --2- 172.21.15.98:0/816852459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1910101430 0x7f191010e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.689+0000 7f19186f1640 1 --2- 172.21.15.98:0/816852459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1910100a90 0x7f191010e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:14.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.689+0000 7f19186f1640 1 -- 172.21.15.98:0/816852459 >> 172.21.15.98:0/816852459 conn(0x7f19100fc7d0 msgr2=0x7f19100fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:14.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.689+0000 7f19186f1640 1 -- 172.21.15.98:0/816852459 shutdown_connections 2024-09-19T08:05:14.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:14.689+0000 7f19186f1640 1 -- 172.21.15.98:0/816852459 wait complete. 2024-09-19T08:05:14.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:14 smithi098 ceph-mon[31333]: pgmap v70: 97 pgs: 97 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 85 MiB/s wr, 29 op/s 2024-09-19T08:05:15.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:14 smithi162 ceph-mon[35260]: pgmap v70: 97 pgs: 97 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 85 MiB/s wr, 29 op/s 2024-09-19T08:05:15.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:14 smithi139 ceph-mon[34874]: pgmap v70: 97 pgs: 97 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 85 MiB/s wr, 29 op/s 2024-09-19T08:05:15.337 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:05:15.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:05:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:05:15.923 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:15.924 INFO:tasks.workunit.client.0.smithi098.stdout: Children see throughput for 1 re-readers = 7076801.00 kB/sec 2024-09-19T08:05:15.924 INFO:tasks.workunit.client.0.smithi098.stdout: Parent sees throughput for 1 re-readers = 7067305.61 kB/sec 2024-09-19T08:05:15.924 INFO:tasks.workunit.client.0.smithi098.stdout: Min throughput per process = 7076801.00 kB/sec 2024-09-19T08:05:15.924 INFO:tasks.workunit.client.0.smithi098.stdout: Max throughput per process = 7076801.00 kB/sec 2024-09-19T08:05:15.924 INFO:tasks.workunit.client.0.smithi098.stdout: Avg throughput per process = 7076801.00 kB/sec 2024-09-19T08:05:15.924 INFO:tasks.workunit.client.0.smithi098.stdout: Min xfer = 1048576.00 kB 2024-09-19T08:05:16.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.319+0000 7f675ff19640 1 -- 172.21.15.98:0/3945863368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6758101de0 msgr2=0x7f67581021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:16.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.319+0000 7f675ff19640 1 --2- 172.21.15.98:0/3945863368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6758101de0 0x7f67581021c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6754009900 tx=0x7f675402a410 comp rx=0 tx=0).stop 2024-09-19T08:05:16.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.319+0000 7f675ff19640 1 -- 172.21.15.98:0/3945863368 shutdown_connections 2024-09-19T08:05:16.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.319+0000 7f675ff19640 1 --2- 172.21.15.98:0/3945863368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6758106f10 0x7f675810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:16.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.319+0000 7f675ff19640 1 --2- 172.21.15.98:0/3945863368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6758102790 0x7f67581067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:16.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.319+0000 7f675ff19640 1 --2- 172.21.15.98:0/3945863368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6758101de0 0x7f67581021c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:16.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.319+0000 7f675ff19640 1 -- 172.21.15.98:0/3945863368 >> 172.21.15.98:0/3945863368 conn(0x7f67580fd8c0 msgr2=0x7f67580ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:16.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.320+0000 7f675ff19640 1 -- 172.21.15.98:0/3945863368 shutdown_connections 2024-09-19T08:05:16.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.320+0000 7f675ff19640 1 -- 172.21.15.98:0/3945863368 wait complete. 2024-09-19T08:05:16.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.320+0000 7f675ff19640 1 Processor -- start 2024-09-19T08:05:16.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.320+0000 7f675ff19640 1 -- start start 2024-09-19T08:05:16.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.320+0000 7f675ff19640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6758101de0 0x7f675819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:16.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.320+0000 7f675ff19640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6758102790 0x7f675819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:16.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.320+0000 7f675ff19640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6758106f10 0x7f675819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:16.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.320+0000 7f675ff19640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6758111f90 con 0x7f6758101de0 2024-09-19T08:05:16.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.320+0000 7f675ff19640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6758111e10 con 0x7f6758102790 2024-09-19T08:05:16.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.320+0000 7f675ff19640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6758112110 con 0x7f6758106f10 2024-09-19T08:05:16.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.320+0000 7f675dcb5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6758101de0 0x7f675819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:16.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.320+0000 7f675dcb5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6758101de0 0x7f675819a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57832/0 (socket says 172.21.15.98:57832) 2024-09-19T08:05:16.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f675dcb5640 1 -- 172.21.15.98:0/961976250 learned_addr learned my addr 172.21.15.98:0/961976250 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:16.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f675e4b6640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6758106f10 0x7f675819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:16.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f675dcb5640 1 -- 172.21.15.98:0/961976250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6758106f10 msgr2=0x7f675819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:16.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f675dcb5640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6758106f10 0x7f675819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:16.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f675dcb5640 1 -- 172.21.15.98:0/961976250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6758102790 msgr2=0x7f675819ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:16.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f675d4b4640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6758102790 0x7f675819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:16.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f675dcb5640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6758102790 0x7f675819ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:16.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f675dcb5640 1 -- 172.21.15.98:0/961976250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675819f6a0 con 0x7f6758101de0 2024-09-19T08:05:16.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f675e4b6640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6758106f10 0x7f675819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:05:16.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f675dcb5640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6758101de0 0x7f675819a6f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6754006010 tx=0x7f67540042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:16.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f674effd640 1 -- 172.21.15.98:0/961976250 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675403f030 con 0x7f6758101de0 2024-09-19T08:05:16.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f674effd640 1 -- 172.21.15.98:0/961976250 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6754003ce0 con 0x7f6758101de0 2024-09-19T08:05:16.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f675ff19640 1 -- 172.21.15.98:0/961976250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f675819f930 con 0x7f6758101de0 2024-09-19T08:05:16.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f674effd640 1 -- 172.21.15.98:0/961976250 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754035070 con 0x7f6758101de0 2024-09-19T08:05:16.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.321+0000 7f675ff19640 1 -- 172.21.15.98:0/961976250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67581a7270 con 0x7f6758101de0 2024-09-19T08:05:16.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.322+0000 7f675ff19640 1 -- 172.21.15.98:0/961976250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67581021c0 con 0x7f6758101de0 2024-09-19T08:05:16.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.323+0000 7f674effd640 1 -- 172.21.15.98:0/961976250 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f6754042030 con 0x7f6758101de0 2024-09-19T08:05:16.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.324+0000 7f674effd640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f67440a3430 0x7f67440a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:16.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.324+0000 7f675d4b4640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f67440a3430 0x7f67440a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:16.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.324+0000 7f674effd640 1 -- 172.21.15.98:0/961976250 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f67540f0330 con 0x7f6758101de0 2024-09-19T08:05:16.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.324+0000 7f675d4b4640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f67440a3430 0x7f67440a58f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6748005fd0 tx=0x7f6748005850 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:16.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.328+0000 7f674effd640 1 -- 172.21.15.98:0/961976250 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f67540bce00 con 0x7f6758101de0 2024-09-19T08:05:16.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.529+0000 7f675ff19640 1 -- 172.21.15.98:0/961976250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67581a7440 con 0x7f6758101de0 2024-09-19T08:05:16.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.529+0000 7f674effd640 1 -- 172.21.15.98:0/961976250 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11984 (secure 0 0 0) 0x7f67540bc700 con 0x7f6758101de0 2024-09-19T08:05:16.530 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:05:16.530 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"btime":"2024-09-19T08:04:50:417827+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:04:50.397583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24461,24437,24406,24376]},"id":1}]} 2024-09-19T08:05:16.530 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-09-19T08:05:16.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.531+0000 7f675ff19640 1 -- 172.21.15.98:0/961976250 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f67440a3430 msgr2=0x7f67440a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:16.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.531+0000 7f675ff19640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f67440a3430 0x7f67440a58f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6748005fd0 tx=0x7f6748005850 comp rx=0 tx=0).stop 2024-09-19T08:05:16.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.531+0000 7f675ff19640 1 -- 172.21.15.98:0/961976250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6758101de0 msgr2=0x7f675819a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:16.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.531+0000 7f675ff19640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6758101de0 0x7f675819a6f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6754006010 tx=0x7f67540042f0 comp rx=0 tx=0).stop 2024-09-19T08:05:16.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.532+0000 7f675ff19640 1 -- 172.21.15.98:0/961976250 shutdown_connections 2024-09-19T08:05:16.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.532+0000 7f675ff19640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f67440a3430 0x7f67440a58f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:16.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.532+0000 7f675ff19640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6758106f10 0x7f675819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:16.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.532+0000 7f675ff19640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6758102790 0x7f675819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:16.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.532+0000 7f675ff19640 1 --2- 172.21.15.98:0/961976250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6758101de0 0x7f675819a6f0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:16.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.532+0000 7f675ff19640 1 -- 172.21.15.98:0/961976250 >> 172.21.15.98:0/961976250 conn(0x7f67580fd8c0 msgr2=0x7f67580febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:16.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.532+0000 7f675ff19640 1 -- 172.21.15.98:0/961976250 shutdown_connections 2024-09-19T08:05:16.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:16.532+0000 7f675ff19640 1 -- 172.21.15.98:0/961976250 wait complete. 2024-09-19T08:05:16.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:16 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/961976250' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:05:16.895 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:16 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/961976250' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:05:16.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:16 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/961976250' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:05:17.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:05:17.479 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:05:17.841 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:17 smithi098 ceph-mon[31333]: pgmap v71: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 85 MiB/s wr, 33 op/s 2024-09-19T08:05:17.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:17 smithi162 ceph-mon[35260]: pgmap v71: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 85 MiB/s wr, 33 op/s 2024-09-19T08:05:17.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:17 smithi139 ceph-mon[34874]: pgmap v71: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 85 MiB/s wr, 33 op/s 2024-09-19T08:05:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.922+0000 7fd2b73f9640 1 -- 172.21.15.98:0/2270766496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102570 msgr2=0x7fd2b01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.922+0000 7fd2b73f9640 1 --2- 172.21.15.98:0/2270766496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102570 0x7fd2b01029b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac009900 tx=0x7fd2ac02a480 comp rx=0 tx=0).stop 2024-09-19T08:05:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.922+0000 7fd2b73f9640 1 -- 172.21.15.98:0/2270766496 shutdown_connections 2024-09-19T08:05:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.922+0000 7fd2b73f9640 1 --2- 172.21.15.98:0/2270766496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2b0102ef0 0x7fd2b010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.922+0000 7fd2b73f9640 1 --2- 172.21.15.98:0/2270766496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102570 0x7fd2b01029b0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.922+0000 7fd2b73f9640 1 --2- 172.21.15.98:0/2270766496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2b0100e20 0x7fd2b0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.922+0000 7fd2b73f9640 1 -- 172.21.15.98:0/2270766496 >> 172.21.15.98:0/2270766496 conn(0x7fd2b00fc7d0 msgr2=0x7fd2b00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.922+0000 7fd2b73f9640 1 -- 172.21.15.98:0/2270766496 shutdown_connections 2024-09-19T08:05:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.922+0000 7fd2b73f9640 1 -- 172.21.15.98:0/2270766496 wait complete. 2024-09-19T08:05:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.922+0000 7fd2b73f9640 1 Processor -- start 2024-09-19T08:05:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b73f9640 1 -- start start 2024-09-19T08:05:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b73f9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2b0100e20 0x7fd2b01984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b73f9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2b0102570 0x7fd2b0198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b73f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102ef0 0x7fd2b019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b73f9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b0110f80 con 0x7fd2b0102ef0 2024-09-19T08:05:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b73f9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b0110e00 con 0x7fd2b0100e20 2024-09-19T08:05:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b73f9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b0111100 con 0x7fd2b0102570 2024-09-19T08:05:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b5996640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102ef0 0x7fd2b019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b5996640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102ef0 0x7fd2b019cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57850/0 (socket says 172.21.15.98:57850) 2024-09-19T08:05:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b5195640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2b0100e20 0x7fd2b01984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b5996640 1 -- 172.21.15.98:0/3281587127 learned_addr learned my addr 172.21.15.98:0/3281587127 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b4994640 1 --2- 172.21.15.98:0/3281587127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2b0102570 0x7fd2b0198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b5996640 1 -- 172.21.15.98:0/3281587127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2b0102570 msgr2=0x7fd2b0198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b5996640 1 --2- 172.21.15.98:0/3281587127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2b0102570 0x7fd2b0198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b5996640 1 -- 172.21.15.98:0/3281587127 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2b0100e20 msgr2=0x7fd2b01984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b5996640 1 --2- 172.21.15.98:0/3281587127 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2b0100e20 0x7fd2b01984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b5996640 1 -- 172.21.15.98:0/3281587127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b019d4a0 con 0x7fd2b0102ef0 2024-09-19T08:05:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.923+0000 7fd2b5996640 1 --2- 172.21.15.98:0/3281587127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102ef0 0x7fd2b019cdc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd2a800cad0 tx=0x7fd2a800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.924+0000 7fd2a27fc640 1 -- 172.21.15.98:0/3281587127 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a800ae20 con 0x7fd2b0102ef0 2024-09-19T08:05:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.924+0000 7fd2a27fc640 1 -- 172.21.15.98:0/3281587127 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2a8005020 con 0x7fd2b0102ef0 2024-09-19T08:05:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.924+0000 7fd2b73f9640 1 -- 172.21.15.98:0/3281587127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2b019d790 con 0x7fd2b0102ef0 2024-09-19T08:05:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.924+0000 7fd2a27fc640 1 -- 172.21.15.98:0/3281587127 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a800f6f0 con 0x7fd2b0102ef0 2024-09-19T08:05:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.924+0000 7fd2b73f9640 1 -- 172.21.15.98:0/3281587127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2b01a5070 con 0x7fd2b0102ef0 2024-09-19T08:05:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.926+0000 7fd2a27fc640 1 -- 172.21.15.98:0/3281587127 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fd2a80051d0 con 0x7fd2b0102ef0 2024-09-19T08:05:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.926+0000 7fd2b97d2740 1 Processor -- start 2024-09-19T08:05:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.926+0000 7fd2b97d2740 1 -- start start 2024-09-19T08:05:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b97d2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a54b40 0x5578f7a53090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b97d2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a535d0 0x5578f7a53a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b97d2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a5e370 0x5578f7a60760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2a27fc640 1 --2- 172.21.15.98:0/3281587127 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd29c0a33b0 0x7fd29c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b97d2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5578f79b92f0 con 0x5578f7a5e370 2024-09-19T08:05:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b97d2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5578f7960cf0 con 0x5578f7a54b40 2024-09-19T08:05:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b97d2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5578f79b52b0 con 0x5578f7a535d0 2024-09-19T08:05:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b5996640 1 --2- 172.21.15.98:0/3281587127 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd29c0a33b0 0x7fd29c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b4994640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a5e370 0x5578f7a60760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b5195640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a535d0 0x5578f7a53a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b4994640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a5e370 0x5578f7a60760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57862/0 (socket says 172.21.15.98:57862) 2024-09-19T08:05:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b5195640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a535d0 0x5578f7a53a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:48430/0 (socket says 172.21.15.98:48430) 2024-09-19T08:05:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b4994640 1 -- 172.21.15.98:0/2208759967 learned_addr learned my addr 172.21.15.98:0/2208759967 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b4994640 1 --2- 172.21.15.98:0/2208759967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a54b40 0x5578f7a53090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2a27fc640 1 -- 172.21.15.98:0/3281587127 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fd2a80cfce0 con 0x7fd2b0102ef0 2024-09-19T08:05:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b4994640 1 -- 172.21.15.98:0/2208759967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a535d0 msgr2=0x5578f7a53a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b4994640 1 --2- 172.21.15.98:0/2208759967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a535d0 0x5578f7a53a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b4994640 1 -- 172.21.15.98:0/2208759967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a54b40 msgr2=0x5578f7a53090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b4994640 1 --2- 172.21.15.98:0/2208759967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a54b40 0x5578f7a53090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b4994640 1 -- 172.21.15.98:0/2208759967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578f79453a0 con 0x5578f7a5e370 2024-09-19T08:05:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b5996640 1 --2- 172.21.15.98:0/3281587127 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd29c0a33b0 0x7fd29c0a5870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd2a80cd510 tx=0x7fd2a80bc040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.927+0000 7fd2b4994640 1 --2- 172.21.15.98:0/2208759967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a5e370 0x5578f7a60760 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0314e0 tx=0x7fd2ac000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b6f8057bd1efd9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.928+0000 7fd29b7fe640 1 -- 172.21.15.98:0/2208759967 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac035040 con 0x5578f7a5e370 2024-09-19T08:05:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.928+0000 7fd29b7fe640 1 -- 172.21.15.98:0/2208759967 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2ac003b80 con 0x5578f7a5e370 2024-09-19T08:05:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.928+0000 7fd29b7fe640 1 -- 172.21.15.98:0/2208759967 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac031b60 con 0x5578f7a5e370 2024-09-19T08:05:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.928+0000 7fd2b97d2740 1 -- 172.21.15.98:0/2208759967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a5e370 msgr2=0x5578f7a60760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.928+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/2208759967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a5e370 0x5578f7a60760 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0314e0 tx=0x7fd2ac000bc0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.928+0000 7fd2b97d2740 1 -- 172.21.15.98:0/2208759967 shutdown_connections 2024-09-19T08:05:17.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.928+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/2208759967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a5e370 0x5578f7a60760 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.928+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/2208759967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a535d0 0x5578f7a53a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.928+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/2208759967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a54b40 0x5578f7a53090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.928+0000 7fd2b97d2740 1 -- 172.21.15.98:0/2208759967 >> 172.21.15.98:0/2208759967 conn(0x5578f7a1c310 msgr2=0x5578f7a4fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:17.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.928+0000 7fd2b97d2740 1 -- 172.21.15.98:0/2208759967 shutdown_connections 2024-09-19T08:05:17.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.928+0000 7fd2b97d2740 1 -- 172.21.15.98:0/2208759967 wait complete. 2024-09-19T08:05:17.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.929+0000 7fd2b97d2740 1 -- start start 2024-09-19T08:05:17.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.929+0000 7fd2b97d2740 1 Processor -- start 2024-09-19T08:05:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.929+0000 7fd2b97d2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a535d0 0x5578f7a5ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.929+0000 7fd2b97d2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a54b40 0x5578f7a5b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.929+0000 7fd2b97d2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a5e370 0x5578f7a5b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.929+0000 7fd2b97d2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5578f7a61010 con 0x5578f7a54b40 2024-09-19T08:05:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.929+0000 7fd2b97d2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5578f7974060 con 0x5578f7a5e370 2024-09-19T08:05:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.929+0000 7fd2b97d2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5578f7a61190 con 0x5578f7a535d0 2024-09-19T08:05:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.929+0000 7fd2b97d2740 10 client.0 _ll_register_callbacks cb 0x5578f76c78c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.929+0000 7fd2b5195640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a54b40 0x5578f7a5b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:17.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.929+0000 7fd2b5195640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a54b40 0x5578f7a5b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57864/0 (socket says 172.21.15.98:57864) 2024-09-19T08:05:17.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.929+0000 7fd2b5195640 1 -- 172.21.15.98:0/2138492980 learned_addr learned my addr 172.21.15.98:0/2138492980 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:17.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd2b4994640 1 --2- 172.21.15.98:0/2138492980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a5e370 0x5578f7a5b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:17.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd2b4994640 1 --2- 172.21.15.98:0/2138492980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a535d0 0x5578f7a5ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:17.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd2b5195640 1 -- 172.21.15.98:0/2138492980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a535d0 msgr2=0x5578f7a5ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:17.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd2b5195640 1 --2- 172.21.15.98:0/2138492980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a535d0 0x5578f7a5ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd2b5195640 1 -- 172.21.15.98:0/2138492980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a5e370 msgr2=0x5578f7a5b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:17.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd2b5195640 1 --2- 172.21.15.98:0/2138492980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a5e370 0x5578f7a5b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd2b5195640 1 -- 172.21.15.98:0/2138492980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578f772a3c0 con 0x5578f7a54b40 2024-09-19T08:05:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd2b4994640 1 --2- 172.21.15.98:0/2138492980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a5e370 0x5578f7a5b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:05:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd2b5195640 1 --2- 172.21.15.98:0/2138492980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a54b40 0x5578f7a5b060 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400ba10 tx=0x7fd2a400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1785ade83675d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd298ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd298ff9640 1 -- 172.21.15.98:0/2138492980 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4003e80 con 0x5578f7a54b40 2024-09-19T08:05:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd298ff9640 1 -- 172.21.15.98:0/2138492980 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fd2a4011440 con 0x5578f7a54b40 2024-09-19T08:05:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd2b97d2740 1 -- 172.21.15.98:0/2138492980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2ac003b80 con 0x5578f7a54b40 2024-09-19T08:05:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd298ff9640 1 -- 172.21.15.98:0/2138492980 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd2a4011640 con 0x5578f7a54b40 2024-09-19T08:05:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd2b97d2740 10 client.16074 fetch_fsmap learned FSMap version 29 2024-09-19T08:05:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd2b97d2740 1 -- 172.21.15.98:0/2138492980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5578f7a6cb60 con 0x5578f7a54b40 2024-09-19T08:05:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd298ff9640 1 -- 172.21.15.98:0/2138492980 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2ac003b80 con 0x5578f7a54b40 2024-09-19T08:05:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.930+0000 7fd298ff9640 1 -- 172.21.15.98:0/2138492980 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fd2a401bad0 con 0x5578f7a54b40 2024-09-19T08:05:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.931+0000 7fd2b97d2740 10 client.16074 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T08:05:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.931+0000 7fd2b97d2740 10 client.16074 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:05:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.931+0000 7fd2b97d2740 20 client.16074 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.931+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/2138492980 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5578f7a6e0c0 0x5578f7a70520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.931+0000 7fd2b97d2740 4 client.16074 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:05:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.931+0000 7fd2b97d2740 1 -- 172.21.15.98:0/2138492980 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578f79c6860 con 0x5578f7a6e0c0 2024-09-19T08:05:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.931+0000 7fd2b4994640 1 --2- 172.21.15.98:0/2138492980 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5578f7a6e0c0 0x5578f7a70520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.931+0000 7fd298ff9640 10 client.16074 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:05:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.931+0000 7fd2b4994640 1 --2- 172.21.15.98:0/2138492980 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5578f7a6e0c0 0x5578f7a70520 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ab3f93cc89051b76 server_cookie=62d04f4528c9284 in_seq=0 out_seq=0 2024-09-19T08:05:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd298ff9640 1 -- 172.21.15.98:0/2138492980 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5578f79c6860 con 0x5578f7a6e0c0 2024-09-19T08:05:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd298ff9640 10 client.16074 handle_command_reply: tid=0 2024-09-19T08:05:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd2b97d2740 1 client.16074 shutdown 2024-09-19T08:05:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd2b97d2740 1 -- 172.21.15.98:0/2138492980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a54b40 msgr2=0x5578f7a5b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/2138492980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a54b40 0x5578f7a5b060 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400ba10 tx=0x7fd2a400bed0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd2b97d2740 1 -- 172.21.15.98:0/2138492980 shutdown_connections 2024-09-19T08:05:17.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/2138492980 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5578f7a6e0c0 0x5578f7a70520 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/2138492980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a5e370 0x5578f7a5b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/2138492980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a54b40 0x5578f7a5b060 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd298ff9640 0 client.16074 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:05:17.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/2138492980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a535d0 0x5578f7a5ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd2b97d2740 1 -- 172.21.15.98:0/2138492980 >> 172.21.15.98:0/2138492980 conn(0x5578f7a1c310 msgr2=0x5578f7a55960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:17.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd2b97d2740 1 -- 172.21.15.98:0/2138492980 shutdown_connections 2024-09-19T08:05:17.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd2b97d2740 1 -- 172.21.15.98:0/2138492980 wait complete. 2024-09-19T08:05:17.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.932+0000 7fd2b97d2740 20 client.16074 trim_cache size 0 max 16384 2024-09-19T08:05:17.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.945+0000 7fd2b97d2740 1 Processor -- start 2024-09-19T08:05:17.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.945+0000 7fd2b97d2740 1 -- start start 2024-09-19T08:05:17.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.945+0000 7fd2b97d2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a535d0 0x5578f7a54300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.945+0000 7fd2b97d2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a54b40 0x5578f7a5a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.945+0000 7fd2b97d2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a5e370 0x5578f7a66800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.945+0000 7fd2b97d2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b01ad820 con 0x5578f7a54b40 2024-09-19T08:05:17.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.945+0000 7fd2b97d2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b01af390 con 0x5578f7a535d0 2024-09-19T08:05:17.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.945+0000 7fd2b97d2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b01adef0 con 0x5578f7a5e370 2024-09-19T08:05:17.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.946+0000 7fd2b5195640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a54b40 0x5578f7a5a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:17.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.946+0000 7fd2b5195640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a54b40 0x5578f7a5a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57880/0 (socket says 172.21.15.98:57880) 2024-09-19T08:05:17.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.946+0000 7fd2b5195640 1 -- 172.21.15.98:0/3115025081 learned_addr learned my addr 172.21.15.98:0/3115025081 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:17.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.946+0000 7fd2b4994640 1 --2- 172.21.15.98:0/3115025081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a535d0 0x5578f7a54300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:17.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.946+0000 7fd2b5195640 1 -- 172.21.15.98:0/3115025081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a5e370 msgr2=0x5578f7a66800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:17.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.946+0000 7fd2b5195640 1 --2- 172.21.15.98:0/3115025081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a5e370 0x5578f7a66800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.946+0000 7fd2b5195640 1 -- 172.21.15.98:0/3115025081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a535d0 msgr2=0x5578f7a54300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:17.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.946+0000 7fd2b5195640 1 --2- 172.21.15.98:0/3115025081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a535d0 0x5578f7a54300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.946+0000 7fd2b5195640 1 -- 172.21.15.98:0/3115025081 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578f7a50960 con 0x5578f7a54b40 2024-09-19T08:05:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.946+0000 7fd2b4994640 1 --2- 172.21.15.98:0/3115025081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a535d0 0x5578f7a54300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:05:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.946+0000 7fd2b5195640 1 --2- 172.21.15.98:0/3115025081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a54b40 0x5578f7a5a920 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd2a40104d0 tx=0x7fd2a4011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4587895474753dd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.946+0000 7fd29affd640 1 -- 172.21.15.98:0/3115025081 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4028070 con 0x5578f7a54b40 2024-09-19T08:05:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.946+0000 7fd29affd640 1 -- 172.21.15.98:0/3115025081 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2a4010e00 con 0x5578f7a54b40 2024-09-19T08:05:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.946+0000 7fd29affd640 1 -- 172.21.15.98:0/3115025081 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4000e60 con 0x5578f7a54b40 2024-09-19T08:05:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.947+0000 7fd2b97d2740 1 -- 172.21.15.98:0/3115025081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a54b40 msgr2=0x5578f7a5a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.947+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/3115025081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a54b40 0x5578f7a5a920 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd2a40104d0 tx=0x7fd2a4011440 comp rx=0 tx=0).stop 2024-09-19T08:05:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.947+0000 7fd2b97d2740 1 -- 172.21.15.98:0/3115025081 shutdown_connections 2024-09-19T08:05:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.947+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/3115025081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a5e370 0x5578f7a66800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.947+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/3115025081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a54b40 0x5578f7a5a920 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.947+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/3115025081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a535d0 0x5578f7a54300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.947+0000 7fd2b97d2740 1 -- 172.21.15.98:0/3115025081 >> 172.21.15.98:0/3115025081 conn(0x5578f7a1c310 msgr2=0x5578f7a5a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:17.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.947+0000 7fd2b97d2740 1 -- 172.21.15.98:0/3115025081 shutdown_connections 2024-09-19T08:05:17.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.947+0000 7fd2b97d2740 1 -- 172.21.15.98:0/3115025081 wait complete. 2024-09-19T08:05:17.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.947+0000 7fd2b97d2740 1 -- start start 2024-09-19T08:05:17.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.947+0000 7fd2b97d2740 1 Processor -- start 2024-09-19T08:05:17.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b97d2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a535d0 0x5578f7ad57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b97d2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a54b40 0x5578f7ad5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b97d2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a5e370 0x5578f7a75ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:17.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b97d2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b01aec90 con 0x5578f7a535d0 2024-09-19T08:05:17.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b97d2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b01ae5c0 con 0x5578f7a5e370 2024-09-19T08:05:17.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b97d2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b01ab800 con 0x5578f7a54b40 2024-09-19T08:05:17.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b97d2740 10 client.0 _ll_register_callbacks cb 0x7fd2a40042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:17.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b4994640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a535d0 0x5578f7ad57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:17.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b4994640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a535d0 0x5578f7ad57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57890/0 (socket says 172.21.15.98:57890) 2024-09-19T08:05:18.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b4994640 1 -- 172.21.15.98:0/4042595237 learned_addr learned my addr 172.21.15.98:0/4042595237 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:18.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b5195640 1 --2- 172.21.15.98:0/4042595237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a54b40 0x5578f7ad5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:18.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b4994640 1 -- 172.21.15.98:0/4042595237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a54b40 msgr2=0x5578f7ad5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:18.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b4994640 1 --2- 172.21.15.98:0/4042595237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a54b40 0x5578f7ad5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:18.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b4994640 1 -- 172.21.15.98:0/4042595237 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a5e370 msgr2=0x5578f7a75ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:18.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b4994640 1 --2- 172.21.15.98:0/4042595237 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a5e370 0x5578f7a75ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:18.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b4994640 1 -- 172.21.15.98:0/4042595237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578f7a761d0 con 0x5578f7a535d0 2024-09-19T08:05:18.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b5195640 1 --2- 172.21.15.98:0/4042595237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a54b40 0x5578f7ad5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:05:18.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd2b4994640 1 --2- 172.21.15.98:0/4042595237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a535d0 0x5578f7ad57f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x5578f7a54070 tx=0x7fd2ac005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ed1be1bfd2234e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:18.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.948+0000 7fd298ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:18.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd298ff9640 1 -- 172.21.15.98:0/4042595237 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac035040 con 0x5578f7a535d0 2024-09-19T08:05:18.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd298ff9640 1 -- 172.21.15.98:0/4042595237 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fd2ac0038a0 con 0x5578f7a535d0 2024-09-19T08:05:18.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd2b97d2740 1 -- 172.21.15.98:0/4042595237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2a4010e00 con 0x5578f7a535d0 2024-09-19T08:05:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd298ff9640 1 -- 172.21.15.98:0/4042595237 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd2ac003fc0 con 0x5578f7a535d0 2024-09-19T08:05:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd2b97d2740 10 client.16080 fetch_fsmap learned FSMap version 29 2024-09-19T08:05:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd2b97d2740 1 -- 172.21.15.98:0/4042595237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5578f7adcfb0 con 0x5578f7a535d0 2024-09-19T08:05:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd298ff9640 1 -- 172.21.15.98:0/4042595237 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2a4010e00 con 0x5578f7a535d0 2024-09-19T08:05:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd298ff9640 1 -- 172.21.15.98:0/4042595237 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fd2ac0319a0 con 0x5578f7a535d0 2024-09-19T08:05:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd2b97d2740 10 client.16080 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T08:05:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd2b97d2740 10 client.16080 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:05:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd2b97d2740 20 client.16080 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/4042595237 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5578f7add340 0x5578f7adf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd2b97d2740 4 client.16080 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:05:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd2b97d2740 1 -- 172.21.15.98:0/4042595237 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5578f79c0d70 con 0x5578f7add340 2024-09-19T08:05:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.949+0000 7fd2b5195640 1 --2- 172.21.15.98:0/4042595237 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5578f7add340 0x5578f7adf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd2b5195640 1 --2- 172.21.15.98:0/4042595237 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5578f7add340 0x5578f7adf730 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d18874f7654fdd76 server_cookie=faef99a424e56a96 in_seq=0 out_seq=0 2024-09-19T08:05:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd298ff9640 10 client.16080 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:05:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd298ff9640 1 -- 172.21.15.98:0/4042595237 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5578f79c0d70 con 0x5578f7add340 2024-09-19T08:05:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd298ff9640 10 client.16080 handle_command_reply: tid=0 2024-09-19T08:05:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd2b97d2740 1 client.16080 shutdown 2024-09-19T08:05:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd2b97d2740 1 -- 172.21.15.98:0/4042595237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a535d0 msgr2=0x5578f7ad57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/4042595237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a535d0 0x5578f7ad57f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x5578f7a54070 tx=0x7fd2ac005850 comp rx=0 tx=0).stop 2024-09-19T08:05:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd2b97d2740 1 -- 172.21.15.98:0/4042595237 shutdown_connections 2024-09-19T08:05:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/4042595237 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5578f7add340 0x5578f7adf730 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/4042595237 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5578f7a5e370 0x5578f7a75ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/4042595237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578f7a54b40 0x5578f7ad5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd298ff9640 0 client.16080 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:05:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd2b97d2740 1 --2- 172.21.15.98:0/4042595237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578f7a535d0 0x5578f7ad57f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd2b97d2740 1 -- 172.21.15.98:0/4042595237 >> 172.21.15.98:0/4042595237 conn(0x5578f7a1c310 msgr2=0x5578f7a692a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd2b97d2740 1 -- 172.21.15.98:0/4042595237 shutdown_connections 2024-09-19T08:05:18.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd2b97d2740 1 -- 172.21.15.98:0/4042595237 wait complete. 2024-09-19T08:05:18.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.950+0000 7fd2b97d2740 20 client.16080 trim_cache size 0 max 16384 2024-09-19T08:05:18.003 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:05:18.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.951+0000 7fd2b73f9640 1 -- 172.21.15.98:0/3281587127 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd29c0a33b0 msgr2=0x7fd29c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:18.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.951+0000 7fd2b73f9640 1 --2- 172.21.15.98:0/3281587127 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd29c0a33b0 0x7fd29c0a5870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd2a80cd510 tx=0x7fd2a80bc040 comp rx=0 tx=0).stop 2024-09-19T08:05:18.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.951+0000 7fd2b73f9640 1 -- 172.21.15.98:0/3281587127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102ef0 msgr2=0x7fd2b019cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:18.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.951+0000 7fd2b73f9640 1 --2- 172.21.15.98:0/3281587127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102ef0 0x7fd2b019cdc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd2a800cad0 tx=0x7fd2a800cf90 comp rx=0 tx=0).stop 2024-09-19T08:05:18.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.951+0000 7fd2b73f9640 1 -- 172.21.15.98:0/3281587127 shutdown_connections 2024-09-19T08:05:18.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.951+0000 7fd2b73f9640 1 --2- 172.21.15.98:0/3281587127 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd29c0a33b0 0x7fd29c0a5870 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:18.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.951+0000 7fd2b73f9640 1 --2- 172.21.15.98:0/3281587127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2b0102ef0 0x7fd2b019cdc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:18.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.951+0000 7fd2b73f9640 1 --2- 172.21.15.98:0/3281587127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2b0102570 0x7fd2b0198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:18.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.951+0000 7fd2b73f9640 1 --2- 172.21.15.98:0/3281587127 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2b0100e20 0x7fd2b01984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:18.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.951+0000 7fd2b73f9640 1 -- 172.21.15.98:0/3281587127 >> 172.21.15.98:0/3281587127 conn(0x7fd2b00fc7d0 msgr2=0x7fd2b0069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:18.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.951+0000 7fd2b73f9640 1 -- 172.21.15.98:0/3281587127 shutdown_connections 2024-09-19T08:05:18.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:17.951+0000 7fd2b73f9640 1 -- 172.21.15.98:0/3281587127 wait complete. 2024-09-19T08:05:18.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:05:18.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:18 smithi098 ceph-mon[31333]: pgmap v72: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-09-19T08:05:18.953 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:05:19.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:18 smithi162 ceph-mon[35260]: pgmap v72: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-09-19T08:05:19.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:18 smithi139 ceph-mon[34874]: pgmap v72: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-09-19T08:05:19.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.439+0000 7ff6e19fd640 1 -- 172.21.15.98:0/3218852625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc106a80 msgr2=0x7ff6dc106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:19.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.439+0000 7ff6e19fd640 1 --2- 172.21.15.98:0/3218852625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc106a80 0x7ff6dc106e60 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009930 tx=0x7ff6cc02a420 comp rx=0 tx=0).stop 2024-09-19T08:05:19.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.439+0000 7ff6e19fd640 1 -- 172.21.15.98:0/3218852625 shutdown_connections 2024-09-19T08:05:19.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.439+0000 7ff6e19fd640 1 --2- 172.21.15.98:0/3218852625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc101420 0x7ff6dc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.439+0000 7ff6e19fd640 1 --2- 172.21.15.98:0/3218852625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6dc100a80 0x7ff6dc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.439+0000 7ff6e19fd640 1 --2- 172.21.15.98:0/3218852625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc106a80 0x7ff6dc106e60 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.439+0000 7ff6e19fd640 1 -- 172.21.15.98:0/3218852625 >> 172.21.15.98:0/3218852625 conn(0x7ff6dc0fc7a0 msgr2=0x7ff6dc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:19.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.439+0000 7ff6e19fd640 1 -- 172.21.15.98:0/3218852625 shutdown_connections 2024-09-19T08:05:19.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.439+0000 7ff6e19fd640 1 -- 172.21.15.98:0/3218852625 wait complete. 2024-09-19T08:05:19.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.439+0000 7ff6e19fd640 1 Processor -- start 2024-09-19T08:05:19.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.439+0000 7ff6e19fd640 1 -- start start 2024-09-19T08:05:19.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6e19fd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc100a80 0x7ff6dc199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6e19fd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc101420 0x7ff6dc199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6e19fd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6dc106a80 0x7ff6dc1a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6e19fd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc110f20 con 0x7ff6dc100a80 2024-09-19T08:05:19.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6e19fd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc110da0 con 0x7ff6dc101420 2024-09-19T08:05:19.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6e19fd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc1110a0 con 0x7ff6dc106a80 2024-09-19T08:05:19.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6daffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc100a80 0x7ff6dc199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6daffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc100a80 0x7ff6dc199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57916/0 (socket says 172.21.15.98:57916) 2024-09-19T08:05:19.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6daffd640 1 -- 172.21.15.98:0/3172635730 learned_addr learned my addr 172.21.15.98:0/3172635730 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:19.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6dc106a80 0x7ff6dc1a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6d3fff640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc101420 0x7ff6dc199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6daffd640 1 -- 172.21.15.98:0/3172635730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6dc106a80 msgr2=0x7ff6dc1a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:19.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6daffd640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6dc106a80 0x7ff6dc1a44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6daffd640 1 -- 172.21.15.98:0/3172635730 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc101420 msgr2=0x7ff6dc199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:19.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6daffd640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc101420 0x7ff6dc199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6daffd640 1 -- 172.21.15.98:0/3172635730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc1a4a10 con 0x7ff6dc100a80 2024-09-19T08:05:19.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6d3fff640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc101420 0x7ff6dc199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:05:19.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6daffd640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc100a80 0x7ff6dc199450 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009a50 tx=0x7ff6cc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:19.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6e09fb640 1 -- 172.21.15.98:0/3172635730 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc03f030 con 0x7ff6dc100a80 2024-09-19T08:05:19.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6e19fd640 1 -- 172.21.15.98:0/3172635730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6dc1a4be0 con 0x7ff6dc100a80 2024-09-19T08:05:19.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6e09fb640 1 -- 172.21.15.98:0/3172635730 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6cc003ce0 con 0x7ff6dc100a80 2024-09-19T08:05:19.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.440+0000 7ff6e09fb640 1 -- 172.21.15.98:0/3172635730 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc035070 con 0x7ff6dc100a80 2024-09-19T08:05:19.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.441+0000 7ff6e19fd640 1 -- 172.21.15.98:0/3172635730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6dc1a5090 con 0x7ff6dc100a80 2024-09-19T08:05:19.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.443+0000 7ff6e3dd6740 1 Processor -- start 2024-09-19T08:05:19.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.443+0000 7ff6e09fb640 1 -- 172.21.15.98:0/3172635730 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7ff6cc042030 con 0x7ff6dc100a80 2024-09-19T08:05:19.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.443+0000 7ff6e3dd6740 1 -- start start 2024-09-19T08:05:19.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6e3dd6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d6b40 0x5604af9d5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6e3dd6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d55d0 0x5604af9d5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6e3dd6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9e0370 0x5604af9e2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6e3dd6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5604af93b2f0 con 0x5604af9d6b40 2024-09-19T08:05:19.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6e3dd6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5604af8e2cf0 con 0x5604af9e0370 2024-09-19T08:05:19.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6e3dd6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5604af9372b0 con 0x5604af9d55d0 2024-09-19T08:05:19.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d6b40 0x5604af9d5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d6b40 0x5604af9d5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57930/0 (socket says 172.21.15.98:57930) 2024-09-19T08:05:19.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6db7fe640 1 -- 172.21.15.98:0/3998859874 learned_addr learned my addr 172.21.15.98:0/3998859874 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:19.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6daffd640 1 --2- 172.21.15.98:0/3998859874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d55d0 0x5604af9d5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6d3fff640 1 --2- 172.21.15.98:0/3998859874 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9e0370 0x5604af9e2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6db7fe640 1 -- 172.21.15.98:0/3998859874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d55d0 msgr2=0x5604af9d5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:19.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/3998859874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d55d0 0x5604af9d5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6db7fe640 1 -- 172.21.15.98:0/3998859874 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9e0370 msgr2=0x5604af9e2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:19.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/3998859874 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9e0370 0x5604af9e2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.444+0000 7ff6db7fe640 1 -- 172.21.15.98:0/3998859874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604af8c73a0 con 0x5604af9d6b40 2024-09-19T08:05:19.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.445+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/3998859874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d6b40 0x5604af9d5090 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009f20 tx=0x7ff6d400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96abed4d8854604 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:19.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.445+0000 7ff6e09fb640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff6c80a33b0 0x7ff6c80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.445+0000 7ff6daffd640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff6c80a33b0 0x7ff6c80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.445+0000 7ff6d1ffb640 1 -- 172.21.15.98:0/3998859874 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d401a070 con 0x5604af9d6b40 2024-09-19T08:05:19.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.445+0000 7ff6d1ffb640 1 -- 172.21.15.98:0/3998859874 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6d40158f0 con 0x5604af9d6b40 2024-09-19T08:05:19.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.445+0000 7ff6d1ffb640 1 -- 172.21.15.98:0/3998859874 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d400a900 con 0x5604af9d6b40 2024-09-19T08:05:19.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.445+0000 7ff6e09fb640 1 -- 172.21.15.98:0/3172635730 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff6cc0f03c0 con 0x7ff6dc100a80 2024-09-19T08:05:19.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.446+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/3998859874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d6b40 msgr2=0x5604af9d5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:19.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.446+0000 7ff6daffd640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff6c80a33b0 0x7ff6c80a5870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x5604af9d2ac0 tx=0x7ff6cc0f3e70 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:19.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.446+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/3998859874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d6b40 0x5604af9d5090 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009f20 tx=0x7ff6d400c8b0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.446+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/3998859874 shutdown_connections 2024-09-19T08:05:19.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.446+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/3998859874 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9e0370 0x5604af9e2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.446+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/3998859874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d55d0 0x5604af9d5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.446+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/3998859874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d6b40 0x5604af9d5090 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.446+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/3998859874 >> 172.21.15.98:0/3998859874 conn(0x5604af99e310 msgr2=0x5604af9d1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:19.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.446+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/3998859874 shutdown_connections 2024-09-19T08:05:19.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.446+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/3998859874 wait complete. 2024-09-19T08:05:19.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.446+0000 7ff6e3dd6740 1 -- start start 2024-09-19T08:05:19.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.446+0000 7ff6e3dd6740 1 Processor -- start 2024-09-19T08:05:19.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6e3dd6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d55d0 0x5604af9dfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6e3dd6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 0x5604af9dd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6e3dd6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9e0370 0x5604af9dd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6e3dd6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5604af9e3070 con 0x5604af9d55d0 2024-09-19T08:05:19.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6e3dd6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5604af8f6060 con 0x5604af9e0370 2024-09-19T08:05:19.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6e3dd6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5604af9e31f0 con 0x5604af9d6b40 2024-09-19T08:05:19.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6e3dd6740 10 client.0 _ll_register_callbacks cb 0x5604af6498c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:19.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d55d0 0x5604af9dfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d55d0 0x5604af9dfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57934/0 (socket says 172.21.15.98:57934) 2024-09-19T08:05:19.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6db7fe640 1 -- 172.21.15.98:0/2166629506 learned_addr learned my addr 172.21.15.98:0/2166629506 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:19.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6d3fff640 1 --2- 172.21.15.98:0/2166629506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 0x5604af9dd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/2166629506 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9e0370 0x5604af9dd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6db7fe640 1 -- 172.21.15.98:0/2166629506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 msgr2=0x5604af9dd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:19.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/2166629506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 0x5604af9dd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6db7fe640 1 -- 172.21.15.98:0/2166629506 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9e0370 msgr2=0x5604af9dd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:19.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/2166629506 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9e0370 0x5604af9dd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6db7fe640 1 -- 172.21.15.98:0/2166629506 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604af6ac3c0 con 0x5604af9d55d0 2024-09-19T08:05:19.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/2166629506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d55d0 0x5604af9dfd70 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400a4e0 tx=0x7ff6d400a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711d73edc4d80912 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:19.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6baffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:19.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6baffd640 1 -- 172.21.15.98:0/2166629506 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d401a050 con 0x5604af9d55d0 2024-09-19T08:05:19.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6baffd640 1 -- 172.21.15.98:0/2166629506 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff6d400a850 con 0x5604af9d55d0 2024-09-19T08:05:19.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.447+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/2166629506 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6d40158f0 con 0x5604af9d55d0 2024-09-19T08:05:19.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.448+0000 7ff6baffd640 1 -- 172.21.15.98:0/2166629506 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff6d400aa70 con 0x5604af9d55d0 2024-09-19T08:05:19.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.448+0000 7ff6e3dd6740 10 client.16092 fetch_fsmap learned FSMap version 29 2024-09-19T08:05:19.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.448+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/2166629506 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5604af9eeb40 con 0x5604af9d55d0 2024-09-19T08:05:19.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.448+0000 7ff6baffd640 1 -- 172.21.15.98:0/2166629506 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6d40158f0 con 0x5604af9d55d0 2024-09-19T08:05:19.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.448+0000 7ff6baffd640 1 -- 172.21.15.98:0/2166629506 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7ff6d4022cc0 con 0x5604af9d55d0 2024-09-19T08:05:19.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.448+0000 7ff6e3dd6740 10 client.16092 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T08:05:19.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.448+0000 7ff6e3dd6740 10 client.16092 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:05:19.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.448+0000 7ff6e3dd6740 20 client.16092 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:19.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.448+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/2166629506 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604af9efef0 0x5604af9f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.448+0000 7ff6e3dd6740 4 client.16092 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:05:19.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.448+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/2166629506 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604af948860 con 0x5604af9efef0 2024-09-19T08:05:19.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.448+0000 7ff6d3fff640 1 --2- 172.21.15.98:0/2166629506 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604af9efef0 0x5604af9f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.449+0000 7ff6baffd640 10 client.16092 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:05:19.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.449+0000 7ff6d3fff640 1 --2- 172.21.15.98:0/2166629506 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604af9efef0 0x5604af9f22e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ec0248f164006e88 server_cookie=3dc6fb1c2e04ee68 in_seq=0 out_seq=0 2024-09-19T08:05:19.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6baffd640 1 -- 172.21.15.98:0/2166629506 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5604af948860 con 0x5604af9efef0 2024-09-19T08:05:19.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6baffd640 10 client.16092 handle_command_reply: tid=0 2024-09-19T08:05:19.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6e3dd6740 1 client.16092 shutdown 2024-09-19T08:05:19.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/2166629506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d55d0 msgr2=0x5604af9dfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:19.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/2166629506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d55d0 0x5604af9dfd70 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400a4e0 tx=0x7ff6d400a540 comp rx=0 tx=0).stop 2024-09-19T08:05:19.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/2166629506 shutdown_connections 2024-09-19T08:05:19.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/2166629506 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9e0370 0x5604af9dd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/2166629506 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604af9efef0 0x5604af9f22e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/2166629506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 0x5604af9dd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/2166629506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d55d0 0x5604af9dfd70 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/2166629506 >> 172.21.15.98:0/2166629506 conn(0x5604af99e310 msgr2=0x5604af9d7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:19.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6baffd640 0 client.16092 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:05:19.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/2166629506 shutdown_connections 2024-09-19T08:05:19.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/2166629506 wait complete. 2024-09-19T08:05:19.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.450+0000 7ff6e3dd6740 20 client.16092 trim_cache size 0 max 16384 2024-09-19T08:05:19.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.462+0000 7ff6e3dd6740 1 Processor -- start 2024-09-19T08:05:19.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.462+0000 7ff6e3dd6740 1 -- start start 2024-09-19T08:05:19.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6e3dd6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d55d0 0x5604af9ec340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6e3dd6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 0x5604af9e8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6e3dd6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9e0370 0x5604af9e8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6e3dd6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc1ad950 con 0x5604af9d55d0 2024-09-19T08:05:19.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6e3dd6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc1af4c0 con 0x5604af9e0370 2024-09-19T08:05:19.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6e3dd6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc1ae020 con 0x5604af9d6b40 2024-09-19T08:05:19.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d55d0 0x5604af9ec340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d55d0 0x5604af9ec340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57936/0 (socket says 172.21.15.98:57936) 2024-09-19T08:05:19.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6db7fe640 1 -- 172.21.15.98:0/3632639781 learned_addr learned my addr 172.21.15.98:0/3632639781 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:19.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6d3fff640 1 --2- 172.21.15.98:0/3632639781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 0x5604af9e8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6db7fe640 1 -- 172.21.15.98:0/3632639781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 msgr2=0x5604af9e8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:19.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/3632639781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 0x5604af9e8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6db7fe640 1 -- 172.21.15.98:0/3632639781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9e0370 msgr2=0x5604af9e8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:19.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/3632639781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9e0370 0x5604af9e8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6db7fe640 1 -- 172.21.15.98:0/3632639781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604af9dd980 con 0x5604af9d55d0 2024-09-19T08:05:19.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/3632639781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d55d0 0x5604af9ec340 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4004b90 tx=0x7ff6d4009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5224e0f69dc8859c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:19.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6d17fa640 1 -- 172.21.15.98:0/3632639781 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d401a040 con 0x5604af9d55d0 2024-09-19T08:05:19.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6d17fa640 1 -- 172.21.15.98:0/3632639781 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6d401b040 con 0x5604af9d55d0 2024-09-19T08:05:19.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.463+0000 7ff6d17fa640 1 -- 172.21.15.98:0/3632639781 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d402b420 con 0x5604af9d55d0 2024-09-19T08:05:19.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.464+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/3632639781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d55d0 msgr2=0x5604af9ec340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:19.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.464+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/3632639781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d55d0 0x5604af9ec340 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4004b90 tx=0x7ff6d4009a50 comp rx=0 tx=0).stop 2024-09-19T08:05:19.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.464+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/3632639781 shutdown_connections 2024-09-19T08:05:19.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.464+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/3632639781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9e0370 0x5604af9e8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.464+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/3632639781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 0x5604af9e8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.464+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/3632639781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9d55d0 0x5604af9ec340 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.464+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/3632639781 >> 172.21.15.98:0/3632639781 conn(0x5604af99e310 msgr2=0x5604af9d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:19.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.464+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/3632639781 shutdown_connections 2024-09-19T08:05:19.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.464+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/3632639781 wait complete. 2024-09-19T08:05:19.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.464+0000 7ff6e3dd6740 1 -- start start 2024-09-19T08:05:19.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6e3dd6740 1 Processor -- start 2024-09-19T08:05:19.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6e3dd6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9d55d0 0x5604afa52840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6e3dd6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 0x5604afa52d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6e3dd6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9e0370 0x5604afa5d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6e3dd6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc1aedc0 con 0x5604af9e0370 2024-09-19T08:05:19.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6e3dd6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc1ae6f0 con 0x5604af9d55d0 2024-09-19T08:05:19.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6e3dd6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc1ab930 con 0x5604af9d6b40 2024-09-19T08:05:19.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6e3dd6740 10 client.0 _ll_register_callbacks cb 0x7ff6d4018070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:19.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9e0370 0x5604afa5d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9e0370 0x5604afa5d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57948/0 (socket says 172.21.15.98:57948) 2024-09-19T08:05:19.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6db7fe640 1 -- 172.21.15.98:0/110197667 learned_addr learned my addr 172.21.15.98:0/110197667 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:19.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/110197667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9d55d0 0x5604afa52840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6d3fff640 1 --2- 172.21.15.98:0/110197667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 0x5604afa52d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6db7fe640 1 -- 172.21.15.98:0/110197667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 msgr2=0x5604afa52d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:19.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/110197667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 0x5604afa52d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6db7fe640 1 -- 172.21.15.98:0/110197667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9d55d0 msgr2=0x5604afa52840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:19.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/110197667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9d55d0 0x5604afa52840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6db7fe640 1 -- 172.21.15.98:0/110197667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604afa53610 con 0x5604af9e0370 2024-09-19T08:05:19.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6db7fe640 1 --2- 172.21.15.98:0/110197667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9e0370 0x5604afa5d8c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff6d402aa10 tx=0x7ff6d4015720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78545e19f4b50b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:19.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6baffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:19.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.465+0000 7ff6baffd640 1 -- 172.21.15.98:0/110197667 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4009870 con 0x5604af9e0370 2024-09-19T08:05:19.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.466+0000 7ff6baffd640 1 -- 172.21.15.98:0/110197667 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff6d402aca0 con 0x5604af9e0370 2024-09-19T08:05:19.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.466+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/110197667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6d401b040 con 0x5604af9e0370 2024-09-19T08:05:19.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.466+0000 7ff6baffd640 1 -- 172.21.15.98:0/110197667 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff6d4034450 con 0x5604af9e0370 2024-09-19T08:05:19.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.466+0000 7ff6e3dd6740 10 client.16098 fetch_fsmap learned FSMap version 29 2024-09-19T08:05:19.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.466+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/110197667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5604afa5eed0 con 0x5604af9e0370 2024-09-19T08:05:19.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.466+0000 7ff6baffd640 1 -- 172.21.15.98:0/110197667 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6d401b040 con 0x5604af9e0370 2024-09-19T08:05:19.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.466+0000 7ff6baffd640 1 -- 172.21.15.98:0/110197667 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7ff6d4032020 con 0x5604af9e0370 2024-09-19T08:05:19.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.466+0000 7ff6e3dd6740 10 client.16098 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T08:05:19.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.466+0000 7ff6e3dd6740 10 client.16098 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:05:19.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.466+0000 7ff6e3dd6740 20 client.16098 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:19.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.466+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/110197667 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604afa5f500 0x5604afa61920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:19.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.466+0000 7ff6e3dd6740 4 client.16098 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:05:19.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.466+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/110197667 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5604af942d70 con 0x5604afa5f500 2024-09-19T08:05:19.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.466+0000 7ff6d3fff640 1 --2- 172.21.15.98:0/110197667 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604afa5f500 0x5604afa61920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:19.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6d3fff640 1 --2- 172.21.15.98:0/110197667 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604afa5f500 0x5604afa61920 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1627a6e4b1df0de server_cookie=f27103e5e9afec3 in_seq=0 out_seq=0 2024-09-19T08:05:19.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6baffd640 10 client.16098 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:05:19.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6baffd640 1 -- 172.21.15.98:0/110197667 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5604af942d70 con 0x5604afa5f500 2024-09-19T08:05:19.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6baffd640 10 client.16098 handle_command_reply: tid=0 2024-09-19T08:05:19.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6e3dd6740 1 client.16098 shutdown 2024-09-19T08:05:19.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/110197667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9e0370 msgr2=0x5604afa5d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:19.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/110197667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9e0370 0x5604afa5d8c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff6d402aa10 tx=0x7ff6d4015720 comp rx=0 tx=0).stop 2024-09-19T08:05:19.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/110197667 shutdown_connections 2024-09-19T08:05:19.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/110197667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604af9e0370 0x5604afa5d8c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/110197667 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5604afa5f500 0x5604afa61920 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6baffd640 0 client.16098 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:05:19.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/110197667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604af9d6b40 0x5604afa52d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6e3dd6740 1 --2- 172.21.15.98:0/110197667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604af9d55d0 0x5604afa52840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/110197667 >> 172.21.15.98:0/110197667 conn(0x5604af99e310 msgr2=0x5604af9db7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:19.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.467+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/110197667 shutdown_connections 2024-09-19T08:05:19.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.468+0000 7ff6e3dd6740 1 -- 172.21.15.98:0/110197667 wait complete. 2024-09-19T08:05:19.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.468+0000 7ff6e3dd6740 20 client.16098 trim_cache size 0 max 16384 2024-09-19T08:05:19.474 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:05:19.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.468+0000 7ff6e19fd640 1 -- 172.21.15.98:0/3172635730 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff6c80a33b0 msgr2=0x7ff6c80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:19.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.468+0000 7ff6e19fd640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff6c80a33b0 0x7ff6c80a5870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x5604af9d2ac0 tx=0x7ff6cc0f3e70 comp rx=0 tx=0).stop 2024-09-19T08:05:19.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.468+0000 7ff6e19fd640 1 -- 172.21.15.98:0/3172635730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc100a80 msgr2=0x7ff6dc199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:19.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.468+0000 7ff6e19fd640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc100a80 0x7ff6dc199450 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009a50 tx=0x7ff6cc004270 comp rx=0 tx=0).stop 2024-09-19T08:05:19.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.468+0000 7ff6e19fd640 1 -- 172.21.15.98:0/3172635730 shutdown_connections 2024-09-19T08:05:19.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.468+0000 7ff6e19fd640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff6c80a33b0 0x7ff6c80a5870 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.468+0000 7ff6e19fd640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6dc106a80 0x7ff6dc1a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.468+0000 7ff6e19fd640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc101420 0x7ff6dc199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.468+0000 7ff6e19fd640 1 --2- 172.21.15.98:0/3172635730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc100a80 0x7ff6dc199450 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:19.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.468+0000 7ff6e19fd640 1 -- 172.21.15.98:0/3172635730 >> 172.21.15.98:0/3172635730 conn(0x7ff6dc0fc7a0 msgr2=0x7ff6dc0fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:19.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.468+0000 7ff6e19fd640 1 -- 172.21.15.98:0/3172635730 shutdown_connections 2024-09-19T08:05:19.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:19.468+0000 7ff6e19fd640 1 -- 172.21.15.98:0/3172635730 wait complete. 2024-09-19T08:05:20.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:05:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:05:21.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.017+0000 7f3e3ff9e640 1 -- 172.21.15.98:0/2496241293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e38100e20 msgr2=0x7f3e38101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:21.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.017+0000 7f3e3ff9e640 1 --2- 172.21.15.98:0/2496241293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e38100e20 0x7f3e38101200 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009ca0 tx=0x7f3e2802a950 comp rx=0 tx=0).stop 2024-09-19T08:05:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.017+0000 7f3e3ff9e640 1 -- 172.21.15.98:0/2496241293 shutdown_connections 2024-09-19T08:05:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.017+0000 7f3e3ff9e640 1 --2- 172.21.15.98:0/2496241293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e38102ef0 0x7f3e3810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.017+0000 7f3e3ff9e640 1 --2- 172.21.15.98:0/2496241293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e38102570 0x7f3e381029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.017+0000 7f3e3ff9e640 1 --2- 172.21.15.98:0/2496241293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e38100e20 0x7f3e38101200 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.017+0000 7f3e3ff9e640 1 -- 172.21.15.98:0/2496241293 >> 172.21.15.98:0/2496241293 conn(0x7f3e380fc7d0 msgr2=0x7f3e380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.017+0000 7f3e3ff9e640 1 -- 172.21.15.98:0/2496241293 shutdown_connections 2024-09-19T08:05:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.017+0000 7f3e3ff9e640 1 -- 172.21.15.98:0/2496241293 wait complete. 2024-09-19T08:05:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.017+0000 7f3e3ff9e640 1 Processor -- start 2024-09-19T08:05:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.017+0000 7f3e3ff9e640 1 -- start start 2024-09-19T08:05:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3ff9e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e38100e20 0x7f3e3819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3ff9e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e38102570 0x7f3e3819abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3ff9e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e38102ef0 0x7f3e3819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3ff9e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e38110f80 con 0x7f3e38102570 2024-09-19T08:05:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3ff9e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3e38110e00 con 0x7f3e38100e20 2024-09-19T08:05:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3ff9e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3e38111100 con 0x7f3e38102ef0 2024-09-19T08:05:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3d539640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e38102570 0x7f3e3819abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3d539640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e38102570 0x7f3e3819abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57964/0 (socket says 172.21.15.98:57964) 2024-09-19T08:05:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3d539640 1 -- 172.21.15.98:0/2495704131 learned_addr learned my addr 172.21.15.98:0/2495704131 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3e53b640 1 --2- 172.21.15.98:0/2495704131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e38102ef0 0x7f3e3819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3dd3a640 1 --2- 172.21.15.98:0/2495704131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e38100e20 0x7f3e3819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3d539640 1 -- 172.21.15.98:0/2495704131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e38102ef0 msgr2=0x7f3e3819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3d539640 1 --2- 172.21.15.98:0/2495704131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e38102ef0 0x7f3e3819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3d539640 1 -- 172.21.15.98:0/2495704131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e38100e20 msgr2=0x7f3e3819a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3d539640 1 --2- 172.21.15.98:0/2495704131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e38100e20 0x7f3e3819a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3d539640 1 -- 172.21.15.98:0/2495704131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e3819f640 con 0x7f3e38102570 2024-09-19T08:05:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3d539640 1 --2- 172.21.15.98:0/2495704131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e38102570 0x7f3e3819abd0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00c890 tx=0x7f3e2c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e36ffd640 1 -- 172.21.15.98:0/2495704131 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c004220 con 0x7f3e38102570 2024-09-19T08:05:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3ff9e640 1 -- 172.21.15.98:0/2495704131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e3819f930 con 0x7f3e38102570 2024-09-19T08:05:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e36ffd640 1 -- 172.21.15.98:0/2495704131 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e2c0043c0 con 0x7f3e38102570 2024-09-19T08:05:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e36ffd640 1 -- 172.21.15.98:0/2495704131 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c004a00 con 0x7f3e38102570 2024-09-19T08:05:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.018+0000 7f3e3ff9e640 1 -- 172.21.15.98:0/2495704131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e381a7210 con 0x7f3e38102570 2024-09-19T08:05:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.021+0000 7f3e36ffd640 1 -- 172.21.15.98:0/2495704131 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f3e2c0079b0 con 0x7f3e38102570 2024-09-19T08:05:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.021+0000 7f3e42377740 1 Processor -- start 2024-09-19T08:05:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.021+0000 7f3e42377740 1 -- start start 2024-09-19T08:05:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.021+0000 7f3e42377740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d357b40 0x563b7d356090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.021+0000 7f3e42377740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d3565d0 0x563b7d356a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.021+0000 7f3e42377740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d361370 0x563b7d363760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.021+0000 7f3e42377740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563b7d2bc2f0 con 0x563b7d361370 2024-09-19T08:05:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.021+0000 7f3e42377740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563b7d263cf0 con 0x563b7d3565d0 2024-09-19T08:05:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.021+0000 7f3e42377740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563b7d2b82b0 con 0x563b7d357b40 2024-09-19T08:05:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.021+0000 7f3e3d539640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d361370 0x563b7d363760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e3d539640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d361370 0x563b7d363760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57980/0 (socket says 172.21.15.98:57980) 2024-09-19T08:05:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e3d539640 1 -- 172.21.15.98:0/358469579 learned_addr learned my addr 172.21.15.98:0/358469579 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e3e53b640 1 --2- 172.21.15.98:0/358469579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d357b40 0x563b7d356090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e36ffd640 1 --2- 172.21.15.98:0/2495704131 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3e200a33b0 0x7f3e200a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e3dd3a640 1 --2- 172.21.15.98:0/358469579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d3565d0 0x563b7d356a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e3e53b640 1 --2- 172.21.15.98:0/2495704131 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3e200a33b0 0x7f3e200a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e3d539640 1 -- 172.21.15.98:0/358469579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d357b40 msgr2=0x563b7d356090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e3d539640 1 --2- 172.21.15.98:0/358469579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d357b40 0x563b7d356090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e3d539640 1 -- 172.21.15.98:0/358469579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d3565d0 msgr2=0x563b7d356a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e3d539640 1 --2- 172.21.15.98:0/358469579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d3565d0 0x563b7d356a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e3d539640 1 -- 172.21.15.98:0/358469579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b7d2483a0 con 0x563b7d361370 2024-09-19T08:05:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e36ffd640 1 -- 172.21.15.98:0/2495704131 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f3e2c0cf9f0 con 0x7f3e38102570 2024-09-19T08:05:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e3d539640 1 --2- 172.21.15.98:0/358469579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d361370 0x563b7d363760 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0cd0e0 tx=0x7f3e2c005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1059c1a228083140 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e3e53b640 1 --2- 172.21.15.98:0/2495704131 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3e200a33b0 0x7f3e200a5870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3e30005d00 tx=0x7f3e30004730 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e1ffff640 1 -- 172.21.15.98:0/358469579 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c01d020 con 0x563b7d361370 2024-09-19T08:05:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e1ffff640 1 -- 172.21.15.98:0/358469579 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e2c09c400 con 0x563b7d361370 2024-09-19T08:05:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.022+0000 7f3e1ffff640 1 -- 172.21.15.98:0/358469579 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c09bd00 con 0x563b7d361370 2024-09-19T08:05:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.023+0000 7f3e42377740 1 -- 172.21.15.98:0/358469579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d361370 msgr2=0x563b7d363760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.023+0000 7f3e42377740 1 --2- 172.21.15.98:0/358469579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d361370 0x563b7d363760 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0cd0e0 tx=0x7f3e2c005300 comp rx=0 tx=0).stop 2024-09-19T08:05:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.023+0000 7f3e42377740 1 -- 172.21.15.98:0/358469579 shutdown_connections 2024-09-19T08:05:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.023+0000 7f3e42377740 1 --2- 172.21.15.98:0/358469579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d361370 0x563b7d363760 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.023+0000 7f3e42377740 1 --2- 172.21.15.98:0/358469579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d3565d0 0x563b7d356a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.023+0000 7f3e42377740 1 --2- 172.21.15.98:0/358469579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d357b40 0x563b7d356090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.023+0000 7f3e42377740 1 -- 172.21.15.98:0/358469579 >> 172.21.15.98:0/358469579 conn(0x563b7d31f310 msgr2=0x563b7d352c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.023+0000 7f3e42377740 1 -- 172.21.15.98:0/358469579 shutdown_connections 2024-09-19T08:05:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.023+0000 7f3e42377740 1 -- 172.21.15.98:0/358469579 wait complete. 2024-09-19T08:05:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e42377740 1 -- start start 2024-09-19T08:05:21.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e42377740 1 Processor -- start 2024-09-19T08:05:21.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e42377740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d3565d0 0x563b7d360cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e42377740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d35dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e42377740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d361370 0x563b7d35e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e42377740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563b7d364010 con 0x563b7d357b40 2024-09-19T08:05:21.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e42377740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563b7d277060 con 0x563b7d361370 2024-09-19T08:05:21.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e42377740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563b7d364190 con 0x563b7d3565d0 2024-09-19T08:05:21.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e42377740 10 client.0 _ll_register_callbacks cb 0x563b7cfca8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:21.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e3d539640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d35dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e3d539640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d35dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57990/0 (socket says 172.21.15.98:57990) 2024-09-19T08:05:21.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e3d539640 1 -- 172.21.15.98:0/2488801282 learned_addr learned my addr 172.21.15.98:0/2488801282 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:21.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e3dd3a640 1 --2- 172.21.15.98:0/2488801282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d3565d0 0x563b7d360cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e3d539640 1 -- 172.21.15.98:0/2488801282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d3565d0 msgr2=0x563b7d360cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e3d539640 1 --2- 172.21.15.98:0/2488801282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d3565d0 0x563b7d360cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e3d539640 1 -- 172.21.15.98:0/2488801282 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d361370 msgr2=0x563b7d35e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e3d539640 1 --2- 172.21.15.98:0/2488801282 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d361370 0x563b7d35e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e3d539640 1 -- 172.21.15.98:0/2488801282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b7d02d3c0 con 0x563b7d357b40 2024-09-19T08:05:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.024+0000 7f3e3dd3a640 1 --2- 172.21.15.98:0/2488801282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d3565d0 0x563b7d360cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:05:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e3d539640 1 --2- 172.21.15.98:0/2488801282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d35dfa0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c049630 tx=0x7f3e2c0985a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f952e5a7fd7f2c71 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e1d7fa640 1 -- 172.21.15.98:0/2488801282 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c01d070 con 0x563b7d357b40 2024-09-19T08:05:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e1d7fa640 1 -- 172.21.15.98:0/2488801282 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f3e2c097a50 con 0x563b7d357b40 2024-09-19T08:05:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e42377740 1 -- 172.21.15.98:0/2488801282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e2c09c400 con 0x563b7d357b40 2024-09-19T08:05:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e1d7fa640 1 -- 172.21.15.98:0/2488801282 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3e2c09b840 con 0x563b7d357b40 2024-09-19T08:05:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e42377740 10 client.16110 fetch_fsmap learned FSMap version 29 2024-09-19T08:05:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e42377740 1 -- 172.21.15.98:0/2488801282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563b7d364010 con 0x563b7d357b40 2024-09-19T08:05:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e1d7fa640 1 -- 172.21.15.98:0/2488801282 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e2c09c400 con 0x563b7d357b40 2024-09-19T08:05:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e1d7fa640 1 -- 172.21.15.98:0/2488801282 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f3e2c049750 con 0x563b7d357b40 2024-09-19T08:05:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e42377740 10 client.16110 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T08:05:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e42377740 10 client.16110 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:05:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e42377740 20 client.16110 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e42377740 1 --2- 172.21.15.98:0/2488801282 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x563b7d370bc0 0x563b7d372fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e42377740 4 client.16110 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:05:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.025+0000 7f3e42377740 1 -- 172.21.15.98:0/2488801282 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b7d2c9860 con 0x563b7d370bc0 2024-09-19T08:05:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.026+0000 7f3e3dd3a640 1 --2- 172.21.15.98:0/2488801282 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x563b7d370bc0 0x563b7d372fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.026+0000 7f3e3dd3a640 1 --2- 172.21.15.98:0/2488801282 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x563b7d370bc0 0x563b7d372fb0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4c5e80b46f2ee69c server_cookie=6438fc35a189f703 in_seq=0 out_seq=0 2024-09-19T08:05:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.026+0000 7f3e1d7fa640 10 client.16110 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:05:21.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e1d7fa640 1 -- 172.21.15.98:0/2488801282 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563b7d2c9860 con 0x563b7d370bc0 2024-09-19T08:05:21.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e1d7fa640 10 client.16110 handle_command_reply: tid=0 2024-09-19T08:05:21.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e42377740 1 client.16110 shutdown 2024-09-19T08:05:21.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e42377740 1 -- 172.21.15.98:0/2488801282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 msgr2=0x563b7d35dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:21.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e42377740 1 --2- 172.21.15.98:0/2488801282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d35dfa0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c049630 tx=0x7f3e2c0985a0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e42377740 1 -- 172.21.15.98:0/2488801282 shutdown_connections 2024-09-19T08:05:21.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e42377740 1 --2- 172.21.15.98:0/2488801282 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x563b7d370bc0 0x563b7d372fb0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e42377740 1 --2- 172.21.15.98:0/2488801282 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d361370 0x563b7d35e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e42377740 1 --2- 172.21.15.98:0/2488801282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d35dfa0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e42377740 1 --2- 172.21.15.98:0/2488801282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d3565d0 0x563b7d360cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e1d7fa640 0 client.16110 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:05:21.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e42377740 1 -- 172.21.15.98:0/2488801282 >> 172.21.15.98:0/2488801282 conn(0x563b7d31f310 msgr2=0x563b7d358960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:21.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e42377740 1 -- 172.21.15.98:0/2488801282 shutdown_connections 2024-09-19T08:05:21.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e42377740 1 -- 172.21.15.98:0/2488801282 wait complete. 2024-09-19T08:05:21.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.027+0000 7f3e42377740 20 client.16110 trim_cache size 0 max 16384 2024-09-19T08:05:21.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.040+0000 7f3e42377740 1 Processor -- start 2024-09-19T08:05:21.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.040+0000 7f3e42377740 1 -- start start 2024-09-19T08:05:21.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.040+0000 7f3e42377740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d3565d0 0x563b7d36d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.040+0000 7f3e42377740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d359120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.040+0000 7f3e42377740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d361370 0x563b7d359660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.040+0000 7f3e42377740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e381af970 con 0x563b7d357b40 2024-09-19T08:05:21.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.040+0000 7f3e42377740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3e381b14e0 con 0x563b7d361370 2024-09-19T08:05:21.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.040+0000 7f3e42377740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3e381b0040 con 0x563b7d3565d0 2024-09-19T08:05:21.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.041+0000 7f3e3d539640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d359120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.041+0000 7f3e3d539640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d359120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58000/0 (socket says 172.21.15.98:58000) 2024-09-19T08:05:21.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.041+0000 7f3e3d539640 1 -- 172.21.15.98:0/592242702 learned_addr learned my addr 172.21.15.98:0/592242702 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:21.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.041+0000 7f3e3dd3a640 1 --2- 172.21.15.98:0/592242702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d3565d0 0x563b7d36d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:20 smithi098 ceph-mon[31333]: pgmap v73: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 MiB/s wr, 29 op/s 2024-09-19T08:05:21.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.041+0000 7f3e3d539640 1 -- 172.21.15.98:0/592242702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d3565d0 msgr2=0x563b7d36d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:21.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.041+0000 7f3e3d539640 1 --2- 172.21.15.98:0/592242702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d3565d0 0x563b7d36d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.041+0000 7f3e3d539640 1 -- 172.21.15.98:0/592242702 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d361370 msgr2=0x563b7d359660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.041+0000 7f3e3d539640 1 --2- 172.21.15.98:0/592242702 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d361370 0x563b7d359660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.041+0000 7f3e3d539640 1 -- 172.21.15.98:0/592242702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b7d373540 con 0x563b7d357b40 2024-09-19T08:05:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.041+0000 7f3e3d539640 1 --2- 172.21.15.98:0/592242702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d359120 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c09ba00 tx=0x7f3e2c0054a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c84fb51ab47270 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.041+0000 7f3e1f7fe640 1 -- 172.21.15.98:0/592242702 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c01d070 con 0x563b7d357b40 2024-09-19T08:05:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.041+0000 7f3e1f7fe640 1 -- 172.21.15.98:0/592242702 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e2c09c400 con 0x563b7d357b40 2024-09-19T08:05:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.041+0000 7f3e1f7fe640 1 -- 172.21.15.98:0/592242702 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c0d31d0 con 0x563b7d357b40 2024-09-19T08:05:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.042+0000 7f3e42377740 1 -- 172.21.15.98:0/592242702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 msgr2=0x563b7d359120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.042+0000 7f3e42377740 1 --2- 172.21.15.98:0/592242702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d359120 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c09ba00 tx=0x7f3e2c0054a0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.042+0000 7f3e42377740 1 -- 172.21.15.98:0/592242702 shutdown_connections 2024-09-19T08:05:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.042+0000 7f3e42377740 1 --2- 172.21.15.98:0/592242702 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d361370 0x563b7d359660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.042+0000 7f3e42377740 1 --2- 172.21.15.98:0/592242702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d359120 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.042+0000 7f3e42377740 1 --2- 172.21.15.98:0/592242702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d3565d0 0x563b7d36d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.042+0000 7f3e42377740 1 -- 172.21.15.98:0/592242702 >> 172.21.15.98:0/592242702 conn(0x563b7d31f310 msgr2=0x563b7d360020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.042+0000 7f3e42377740 1 -- 172.21.15.98:0/592242702 shutdown_connections 2024-09-19T08:05:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.042+0000 7f3e42377740 1 -- 172.21.15.98:0/592242702 wait complete. 2024-09-19T08:05:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.043+0000 7f3e42377740 1 -- start start 2024-09-19T08:05:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.043+0000 7f3e42377740 1 Processor -- start 2024-09-19T08:05:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.043+0000 7f3e42377740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d3565d0 0x563b7d369600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.043+0000 7f3e42377740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d369b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.043+0000 7f3e42377740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d361370 0x563b7d35a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.043+0000 7f3e42377740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e381b0de0 con 0x563b7d357b40 2024-09-19T08:05:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.043+0000 7f3e42377740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3e381b0710 con 0x563b7d3565d0 2024-09-19T08:05:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.043+0000 7f3e42377740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3e381ad950 con 0x563b7d361370 2024-09-19T08:05:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.043+0000 7f3e42377740 10 client.0 _ll_register_callbacks cb 0x7f3e30004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:21.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.043+0000 7f3e3d539640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d369b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e3d539640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d369b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58002/0 (socket says 172.21.15.98:58002) 2024-09-19T08:05:21.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e3d539640 1 -- 172.21.15.98:0/228307219 learned_addr learned my addr 172.21.15.98:0/228307219 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:21.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e3e53b640 1 --2- 172.21.15.98:0/228307219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d361370 0x563b7d35a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e3dd3a640 1 --2- 172.21.15.98:0/228307219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d3565d0 0x563b7d369600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e3d539640 1 -- 172.21.15.98:0/228307219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d361370 msgr2=0x563b7d35a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:21.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e3d539640 1 --2- 172.21.15.98:0/228307219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d361370 0x563b7d35a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e3d539640 1 -- 172.21.15.98:0/228307219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d3565d0 msgr2=0x563b7d369600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:21.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e3d539640 1 --2- 172.21.15.98:0/228307219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d3565d0 0x563b7d369600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e3d539640 1 -- 172.21.15.98:0/228307219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b7d35afd0 con 0x563b7d357b40 2024-09-19T08:05:21.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e3d539640 1 --2- 172.21.15.98:0/228307219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d369b40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c049230 tx=0x7f3e2c007eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3387a7f55029e59f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:21.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e1d7fa640 1 -- 172.21.15.98:0/228307219 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c01d050 con 0x563b7d357b40 2024-09-19T08:05:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e1d7fa640 1 -- 172.21.15.98:0/228307219 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f3e2c09b150 con 0x563b7d357b40 2024-09-19T08:05:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e42377740 1 -- 172.21.15.98:0/228307219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e2c09c400 con 0x563b7d357b40 2024-09-19T08:05:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e1d7fa640 1 -- 172.21.15.98:0/228307219 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3e2c0a00b0 con 0x563b7d357b40 2024-09-19T08:05:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e42377740 10 client.16116 fetch_fsmap learned FSMap version 29 2024-09-19T08:05:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.044+0000 7f3e42377740 1 -- 172.21.15.98:0/228307219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563b7d35afd0 con 0x563b7d357b40 2024-09-19T08:05:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.045+0000 7f3e1d7fa640 1 -- 172.21.15.98:0/228307219 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e2c09c400 con 0x563b7d357b40 2024-09-19T08:05:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.045+0000 7f3e1d7fa640 1 -- 172.21.15.98:0/228307219 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f3e2c086110 con 0x563b7d357b40 2024-09-19T08:05:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.045+0000 7f3e42377740 10 client.16116 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T08:05:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.045+0000 7f3e42377740 10 client.16116 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:05:21.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.045+0000 7f3e42377740 20 client.16116 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.045+0000 7f3e42377740 1 --2- 172.21.15.98:0/228307219 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x563b7d3db4a0 0x563b7d3dd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.045+0000 7f3e42377740 4 client.16116 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:05:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.045+0000 7f3e42377740 1 -- 172.21.15.98:0/228307219 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563b7d2c3d70 con 0x563b7d3db4a0 2024-09-19T08:05:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.045+0000 7f3e3dd3a640 1 --2- 172.21.15.98:0/228307219 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x563b7d3db4a0 0x563b7d3dd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e1d7fa640 10 client.16116 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:05:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e3dd3a640 1 --2- 172.21.15.98:0/228307219 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x563b7d3db4a0 0x563b7d3dd8c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ae6fbaad18d53d20 server_cookie=d585f96ac75f4571 in_seq=0 out_seq=0 2024-09-19T08:05:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e1d7fa640 1 -- 172.21.15.98:0/228307219 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563b7d2c3d70 con 0x563b7d3db4a0 2024-09-19T08:05:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e1d7fa640 10 client.16116 handle_command_reply: tid=0 2024-09-19T08:05:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e42377740 1 client.16116 shutdown 2024-09-19T08:05:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e42377740 1 -- 172.21.15.98:0/228307219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 msgr2=0x563b7d369b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e42377740 1 --2- 172.21.15.98:0/228307219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d369b40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c049230 tx=0x7f3e2c007eb0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e42377740 1 -- 172.21.15.98:0/228307219 shutdown_connections 2024-09-19T08:05:21.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e42377740 1 --2- 172.21.15.98:0/228307219 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x563b7d3db4a0 0x563b7d3dd8c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e42377740 1 --2- 172.21.15.98:0/228307219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b7d361370 0x563b7d35a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e1d7fa640 0 client.16116 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:05:21.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e42377740 1 --2- 172.21.15.98:0/228307219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b7d357b40 0x563b7d369b40 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e42377740 1 --2- 172.21.15.98:0/228307219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b7d3565d0 0x563b7d369600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e42377740 1 -- 172.21.15.98:0/228307219 >> 172.21.15.98:0/228307219 conn(0x563b7d31f310 msgr2=0x563b7d35c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:21.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e42377740 1 -- 172.21.15.98:0/228307219 shutdown_connections 2024-09-19T08:05:21.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e42377740 1 -- 172.21.15.98:0/228307219 wait complete. 2024-09-19T08:05:21.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.046+0000 7f3e42377740 20 client.16116 trim_cache size 0 max 16384 2024-09-19T08:05:21.057 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:05:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.047+0000 7f3e3ff9e640 1 -- 172.21.15.98:0/2495704131 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3e200a33b0 msgr2=0x7f3e200a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.047+0000 7f3e3ff9e640 1 --2- 172.21.15.98:0/2495704131 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3e200a33b0 0x7f3e200a5870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3e30005d00 tx=0x7f3e30004730 comp rx=0 tx=0).stop 2024-09-19T08:05:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.047+0000 7f3e3ff9e640 1 -- 172.21.15.98:0/2495704131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e38102570 msgr2=0x7f3e3819abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.047+0000 7f3e3ff9e640 1 --2- 172.21.15.98:0/2495704131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e38102570 0x7f3e3819abd0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00c890 tx=0x7f3e2c00cd50 comp rx=0 tx=0).stop 2024-09-19T08:05:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.047+0000 7f3e3ff9e640 1 -- 172.21.15.98:0/2495704131 shutdown_connections 2024-09-19T08:05:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.047+0000 7f3e3ff9e640 1 --2- 172.21.15.98:0/2495704131 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3e200a33b0 0x7f3e200a5870 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.047+0000 7f3e3ff9e640 1 --2- 172.21.15.98:0/2495704131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e38102ef0 0x7f3e3819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.047+0000 7f3e3ff9e640 1 --2- 172.21.15.98:0/2495704131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e38102570 0x7f3e3819abd0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.047+0000 7f3e3ff9e640 1 --2- 172.21.15.98:0/2495704131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e38100e20 0x7f3e3819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.047+0000 7f3e3ff9e640 1 -- 172.21.15.98:0/2495704131 >> 172.21.15.98:0/2495704131 conn(0x7f3e380fc7d0 msgr2=0x7f3e3806a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.047+0000 7f3e3ff9e640 1 -- 172.21.15.98:0/2495704131 shutdown_connections 2024-09-19T08:05:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:21.047+0000 7f3e3ff9e640 1 -- 172.21.15.98:0/2495704131 wait complete. 2024-09-19T08:05:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:20 smithi162 ceph-mon[35260]: pgmap v73: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 MiB/s wr, 29 op/s 2024-09-19T08:05:21.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:20 smithi139 ceph-mon[34874]: pgmap v73: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 MiB/s wr, 29 op/s 2024-09-19T08:05:21.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:05:21.993 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:05:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.466+0000 7f62937cb640 1 -- 172.21.15.98:0/392634708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c106a80 msgr2=0x7f628c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.466+0000 7f62937cb640 1 --2- 172.21.15.98:0/392634708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c106a80 0x7f628c106e60 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6288009930 tx=0x7f628802a420 comp rx=0 tx=0).stop 2024-09-19T08:05:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.466+0000 7f62937cb640 1 -- 172.21.15.98:0/392634708 shutdown_connections 2024-09-19T08:05:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.466+0000 7f62937cb640 1 --2- 172.21.15.98:0/392634708 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f628c101420 0x7f628c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.466+0000 7f62937cb640 1 --2- 172.21.15.98:0/392634708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c100a80 0x7f628c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.466+0000 7f62937cb640 1 --2- 172.21.15.98:0/392634708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c106a80 0x7f628c106e60 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.466+0000 7f62937cb640 1 -- 172.21.15.98:0/392634708 >> 172.21.15.98:0/392634708 conn(0x7f628c0fc7a0 msgr2=0x7f628c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.466+0000 7f62937cb640 1 -- 172.21.15.98:0/392634708 shutdown_connections 2024-09-19T08:05:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.466+0000 7f62937cb640 1 -- 172.21.15.98:0/392634708 wait complete. 2024-09-19T08:05:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f62937cb640 1 Processor -- start 2024-09-19T08:05:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f62937cb640 1 -- start start 2024-09-19T08:05:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f62937cb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c100a80 0x7f628c197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f62937cb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c101420 0x7f628c1975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f62937cb640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f628c106a80 0x7f628c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f62937cb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f628c110e40 con 0x7f628c101420 2024-09-19T08:05:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f62937cb640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f628c110cc0 con 0x7f628c106a80 2024-09-19T08:05:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f62937cb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f628c110fc0 con 0x7f628c100a80 2024-09-19T08:05:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f6290d66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c101420 0x7f628c1975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f6290d66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c101420 0x7f628c1975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58016/0 (socket says 172.21.15.98:58016) 2024-09-19T08:05:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f6290d66640 1 -- 172.21.15.98:0/3290121361 learned_addr learned my addr 172.21.15.98:0/3290121361 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f6291567640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c100a80 0x7f628c197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f6290d66640 1 -- 172.21.15.98:0/3290121361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c100a80 msgr2=0x7f628c197080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f6290d66640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c100a80 0x7f628c197080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f6291d68640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f628c106a80 0x7f628c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f6290d66640 1 -- 172.21.15.98:0/3290121361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f628c106a80 msgr2=0x7f628c197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f6290d66640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f628c106a80 0x7f628c197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f6290d66640 1 -- 172.21.15.98:0/3290121361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628c1a68f0 con 0x7f628c101420 2024-09-19T08:05:22.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.467+0000 7f6291567640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c100a80 0x7f628c197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:05:22.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.468+0000 7f6290d66640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c101420 0x7f628c1975c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f627c00c920 tx=0x7f627c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:22.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.468+0000 7f62827fc640 1 -- 172.21.15.98:0/3290121361 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c007c70 con 0x7f628c101420 2024-09-19T08:05:22.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.468+0000 7f62827fc640 1 -- 172.21.15.98:0/3290121361 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f627c007e10 con 0x7f628c101420 2024-09-19T08:05:22.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.468+0000 7f62827fc640 1 -- 172.21.15.98:0/3290121361 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c018890 con 0x7f628c101420 2024-09-19T08:05:22.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.468+0000 7f62937cb640 1 -- 172.21.15.98:0/3290121361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f628c1a6be0 con 0x7f628c101420 2024-09-19T08:05:22.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.468+0000 7f62937cb640 1 -- 172.21.15.98:0/3290121361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f628c1a70d0 con 0x7f628c101420 2024-09-19T08:05:22.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.470+0000 7f62827fc640 1 -- 172.21.15.98:0/3290121361 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f627c018a70 con 0x7f628c101420 2024-09-19T08:05:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6295ba4740 1 Processor -- start 2024-09-19T08:05:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6295ba4740 1 -- start start 2024-09-19T08:05:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6295ba4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0edb40 0x55c60c0ec090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6295ba4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0ec5d0 0x55c60c0eca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6295ba4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0f7370 0x55c60c0f9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6295ba4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c60c0522f0 con 0x55c60c0edb40 2024-09-19T08:05:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6295ba4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c60bff9cf0 con 0x55c60c0f7370 2024-09-19T08:05:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6295ba4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c60c04e2b0 con 0x55c60c0ec5d0 2024-09-19T08:05:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f62827fc640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f62780a33b0 0x7f62780a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6291d68640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0edb40 0x55c60c0ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6291d68640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0edb40 0x55c60c0ec090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58026/0 (socket says 172.21.15.98:58026) 2024-09-19T08:05:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6291d68640 1 -- 172.21.15.98:0/1219228655 learned_addr learned my addr 172.21.15.98:0/1219228655 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6291567640 1 --2- 172.21.15.98:0/1219228655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0ec5d0 0x55c60c0eca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6290d66640 1 --2- 172.21.15.98:0/1219228655 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0f7370 0x55c60c0f9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f62827fc640 1 -- 172.21.15.98:0/3290121361 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f627c0cfef0 con 0x7f628c101420 2024-09-19T08:05:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6291d68640 1 -- 172.21.15.98:0/1219228655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0ec5d0 msgr2=0x55c60c0eca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6291d68640 1 --2- 172.21.15.98:0/1219228655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0ec5d0 0x55c60c0eca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6291d68640 1 -- 172.21.15.98:0/1219228655 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0f7370 msgr2=0x55c60c0f9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6291d68640 1 --2- 172.21.15.98:0/1219228655 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0f7370 0x55c60c0f9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.471+0000 7f6291d68640 1 -- 172.21.15.98:0/1219228655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c60bfde3a0 con 0x55c60c0edb40 2024-09-19T08:05:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.472+0000 7f6291d68640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f62780a33b0 0x7f62780a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.472+0000 7f6291d68640 1 --2- 172.21.15.98:0/1219228655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0edb40 0x55c60c0ec090 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f6284008da0 tx=0x7f628400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77854855c0c2c99 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.472+0000 7f62777fe640 1 -- 172.21.15.98:0/1219228655 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6284011e00 con 0x55c60c0edb40 2024-09-19T08:05:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.472+0000 7f62777fe640 1 -- 172.21.15.98:0/1219228655 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f628400fde0 con 0x55c60c0edb40 2024-09-19T08:05:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.472+0000 7f62777fe640 1 -- 172.21.15.98:0/1219228655 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6284019be0 con 0x55c60c0edb40 2024-09-19T08:05:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.473+0000 7f6295ba4740 1 -- 172.21.15.98:0/1219228655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0edb40 msgr2=0x55c60c0ec090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.473+0000 7f6291d68640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f62780a33b0 0x7f62780a5870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f628401e000 tx=0x7f628401f000 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.473+0000 7f6295ba4740 1 --2- 172.21.15.98:0/1219228655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0edb40 0x55c60c0ec090 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f6284008da0 tx=0x7f628400f730 comp rx=0 tx=0).stop 2024-09-19T08:05:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.473+0000 7f6295ba4740 1 -- 172.21.15.98:0/1219228655 shutdown_connections 2024-09-19T08:05:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.473+0000 7f6295ba4740 1 --2- 172.21.15.98:0/1219228655 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0f7370 0x55c60c0f9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.473+0000 7f6295ba4740 1 --2- 172.21.15.98:0/1219228655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0ec5d0 0x55c60c0eca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.473+0000 7f6295ba4740 1 --2- 172.21.15.98:0/1219228655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0edb40 0x55c60c0ec090 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.473+0000 7f6295ba4740 1 -- 172.21.15.98:0/1219228655 >> 172.21.15.98:0/1219228655 conn(0x55c60c0b5310 msgr2=0x55c60c0e8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.473+0000 7f6295ba4740 1 -- 172.21.15.98:0/1219228655 shutdown_connections 2024-09-19T08:05:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.473+0000 7f6295ba4740 1 -- 172.21.15.98:0/1219228655 wait complete. 2024-09-19T08:05:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.473+0000 7f6295ba4740 1 -- start start 2024-09-19T08:05:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.473+0000 7f6295ba4740 1 Processor -- start 2024-09-19T08:05:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6295ba4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0ec5d0 0x55c60c0f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6295ba4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 0x55c60c0f4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6295ba4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0f7370 0x55c60c0f45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6295ba4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c60c0fa070 con 0x55c60c0ec5d0 2024-09-19T08:05:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6295ba4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c60c00d060 con 0x55c60c0edb40 2024-09-19T08:05:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6295ba4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c60c0fa1f0 con 0x55c60c0f7370 2024-09-19T08:05:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6295ba4740 10 client.0 _ll_register_callbacks cb 0x55c60bd608c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6291567640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0ec5d0 0x55c60c0f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6291567640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0ec5d0 0x55c60c0f6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58036/0 (socket says 172.21.15.98:58036) 2024-09-19T08:05:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6291567640 1 -- 172.21.15.98:0/4155567020 learned_addr learned my addr 172.21.15.98:0/4155567020 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6290d66640 1 --2- 172.21.15.98:0/4155567020 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 0x55c60c0f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6291d68640 1 --2- 172.21.15.98:0/4155567020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0f7370 0x55c60c0f45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6291567640 1 -- 172.21.15.98:0/4155567020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0f7370 msgr2=0x55c60c0f45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:22.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6291567640 1 --2- 172.21.15.98:0/4155567020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0f7370 0x55c60c0f45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6291567640 1 -- 172.21.15.98:0/4155567020 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 msgr2=0x55c60c0f4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:22.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6291567640 1 --2- 172.21.15.98:0/4155567020 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 0x55c60c0f4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6291567640 1 -- 172.21.15.98:0/4155567020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c60bdc33c0 con 0x55c60c0ec5d0 2024-09-19T08:05:22.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6291567640 1 --2- 172.21.15.98:0/4155567020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0ec5d0 0x55c60c0f6d00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f6288031550 tx=0x7f6288004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe4ecb41d53b62 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:22.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6274ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:22.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6274ff9640 1 -- 172.21.15.98:0/4155567020 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6288035040 con 0x55c60c0ec5d0 2024-09-19T08:05:22.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6274ff9640 1 -- 172.21.15.98:0/4155567020 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f6288005020 con 0x55c60c0ec5d0 2024-09-19T08:05:22.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.474+0000 7f6295ba4740 1 -- 172.21.15.98:0/4155567020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f628400fde0 con 0x55c60c0ec5d0 2024-09-19T08:05:22.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.475+0000 7f6274ff9640 1 -- 172.21.15.98:0/4155567020 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6288005220 con 0x55c60c0ec5d0 2024-09-19T08:05:22.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.475+0000 7f6295ba4740 10 client.16128 fetch_fsmap learned FSMap version 30 2024-09-19T08:05:22.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.475+0000 7f6295ba4740 1 -- 172.21.15.98:0/4155567020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c60c105ae0 con 0x55c60c0ec5d0 2024-09-19T08:05:22.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.475+0000 7f6274ff9640 1 -- 172.21.15.98:0/4155567020 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f628400fde0 con 0x55c60c0ec5d0 2024-09-19T08:05:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.475+0000 7f6274ff9640 1 -- 172.21.15.98:0/4155567020 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f6288003730 con 0x55c60c0ec5d0 2024-09-19T08:05:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.475+0000 7f6295ba4740 10 client.16128 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:05:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.475+0000 7f6295ba4740 10 client.16128 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:05:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.475+0000 7f6295ba4740 20 client.16128 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.475+0000 7f6295ba4740 1 --2- 172.21.15.98:0/4155567020 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55c60c106fa0 0x55c60c1093b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.475+0000 7f6295ba4740 4 client.16128 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:05:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.475+0000 7f6295ba4740 1 -- 172.21.15.98:0/4155567020 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c60c05f860 con 0x55c60c106fa0 2024-09-19T08:05:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.475+0000 7f6290d66640 1 --2- 172.21.15.98:0/4155567020 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55c60c106fa0 0x55c60c1093b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.476+0000 7f6290d66640 1 --2- 172.21.15.98:0/4155567020 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55c60c106fa0 0x55c60c1093b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2aa7ff5acc3bceb6 server_cookie=78522e79d58ba0ee in_seq=0 out_seq=0 2024-09-19T08:05:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.476+0000 7f6274ff9640 10 client.16128 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:05:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.476+0000 7f6274ff9640 1 -- 172.21.15.98:0/4155567020 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c60c05f860 con 0x55c60c106fa0 2024-09-19T08:05:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.476+0000 7f6274ff9640 10 client.16128 handle_command_reply: tid=0 2024-09-19T08:05:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.476+0000 7f6295ba4740 1 client.16128 shutdown 2024-09-19T08:05:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.477+0000 7f6295ba4740 1 -- 172.21.15.98:0/4155567020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0ec5d0 msgr2=0x55c60c0f6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.477+0000 7f6295ba4740 1 --2- 172.21.15.98:0/4155567020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0ec5d0 0x55c60c0f6d00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f6288031550 tx=0x7f6288004270 comp rx=0 tx=0).stop 2024-09-19T08:05:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.477+0000 7f6295ba4740 1 -- 172.21.15.98:0/4155567020 shutdown_connections 2024-09-19T08:05:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.477+0000 7f6295ba4740 1 --2- 172.21.15.98:0/4155567020 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55c60c106fa0 0x55c60c1093b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.477+0000 7f6295ba4740 1 --2- 172.21.15.98:0/4155567020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0f7370 0x55c60c0f45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.477+0000 7f6295ba4740 1 --2- 172.21.15.98:0/4155567020 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 0x55c60c0f4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.477+0000 7f6295ba4740 1 --2- 172.21.15.98:0/4155567020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0ec5d0 0x55c60c0f6d00 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.477+0000 7f6295ba4740 1 -- 172.21.15.98:0/4155567020 >> 172.21.15.98:0/4155567020 conn(0x55c60c0b5310 msgr2=0x55c60c0ee960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.477+0000 7f6274ff9640 0 client.16128 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:05:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.477+0000 7f6295ba4740 1 -- 172.21.15.98:0/4155567020 shutdown_connections 2024-09-19T08:05:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.477+0000 7f6295ba4740 1 -- 172.21.15.98:0/4155567020 wait complete. 2024-09-19T08:05:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.477+0000 7f6295ba4740 20 client.16128 trim_cache size 0 max 16384 2024-09-19T08:05:22.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6295ba4740 1 Processor -- start 2024-09-19T08:05:22.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6295ba4740 1 -- start start 2024-09-19T08:05:22.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6295ba4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0ec5d0 0x55c60c103310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6295ba4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 0x55c60c0f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6295ba4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0f7370 0x55c60c0ef120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6295ba4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f628c1af9e0 con 0x55c60c0ec5d0 2024-09-19T08:05:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6295ba4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f628c1b1580 con 0x55c60c0edb40 2024-09-19T08:05:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6295ba4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f628c1b00b0 con 0x55c60c0f7370 2024-09-19T08:05:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6291567640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0ec5d0 0x55c60c103310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6291567640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0ec5d0 0x55c60c103310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58040/0 (socket says 172.21.15.98:58040) 2024-09-19T08:05:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6291567640 1 -- 172.21.15.98:0/2207039383 learned_addr learned my addr 172.21.15.98:0/2207039383 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6290d66640 1 --2- 172.21.15.98:0/2207039383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 0x55c60c0f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6291567640 1 -- 172.21.15.98:0/2207039383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0f7370 msgr2=0x55c60c0ef120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6291d68640 1 --2- 172.21.15.98:0/2207039383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0f7370 0x55c60c0ef120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6291567640 1 --2- 172.21.15.98:0/2207039383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0f7370 0x55c60c0ef120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6291567640 1 -- 172.21.15.98:0/2207039383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 msgr2=0x55c60c0f38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6291567640 1 --2- 172.21.15.98:0/2207039383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 0x55c60c0f38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6291567640 1 -- 172.21.15.98:0/2207039383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c60c0e9960 con 0x55c60c0ec5d0 2024-09-19T08:05:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.490+0000 7f6290d66640 1 --2- 172.21.15.98:0/2207039383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 0x55c60c0f38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:05:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.491+0000 7f6291567640 1 --2- 172.21.15.98:0/2207039383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0ec5d0 0x55c60c103310 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f6288003ad0 tx=0x7f6288031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6ed58d03534fd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:22.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.491+0000 7f6276ffd640 1 -- 172.21.15.98:0/2207039383 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6288035070 con 0x55c60c0ec5d0 2024-09-19T08:05:22.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.491+0000 7f6276ffd640 1 -- 172.21.15.98:0/2207039383 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62880042e0 con 0x55c60c0ec5d0 2024-09-19T08:05:22.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.491+0000 7f6276ffd640 1 -- 172.21.15.98:0/2207039383 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6288052040 con 0x55c60c0ec5d0 2024-09-19T08:05:22.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.491+0000 7f6295ba4740 1 -- 172.21.15.98:0/2207039383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0ec5d0 msgr2=0x55c60c103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:22.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.491+0000 7f6295ba4740 1 --2- 172.21.15.98:0/2207039383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0ec5d0 0x55c60c103310 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f6288003ad0 tx=0x7f6288031400 comp rx=0 tx=0).stop 2024-09-19T08:05:22.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.491+0000 7f6295ba4740 1 -- 172.21.15.98:0/2207039383 shutdown_connections 2024-09-19T08:05:22.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.491+0000 7f6295ba4740 1 --2- 172.21.15.98:0/2207039383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0f7370 0x55c60c0ef120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.491+0000 7f6295ba4740 1 --2- 172.21.15.98:0/2207039383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 0x55c60c0f38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.491+0000 7f6295ba4740 1 --2- 172.21.15.98:0/2207039383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0ec5d0 0x55c60c103310 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.491+0000 7f6295ba4740 1 -- 172.21.15.98:0/2207039383 >> 172.21.15.98:0/2207039383 conn(0x55c60c0b5310 msgr2=0x55c60c0e8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:22.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.492+0000 7f6295ba4740 1 -- 172.21.15.98:0/2207039383 shutdown_connections 2024-09-19T08:05:22.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.492+0000 7f6295ba4740 1 -- 172.21.15.98:0/2207039383 wait complete. 2024-09-19T08:05:22.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.492+0000 7f6295ba4740 1 -- start start 2024-09-19T08:05:22.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.492+0000 7f6295ba4740 1 Processor -- start 2024-09-19T08:05:22.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.492+0000 7f6295ba4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0ec5d0 0x55c60c0fabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.492+0000 7f6295ba4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 0x55c60c0fb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.492+0000 7f6295ba4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0f7370 0x55c60c16fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.492+0000 7f6295ba4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f628c1b0e80 con 0x55c60c0f7370 2024-09-19T08:05:22.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.492+0000 7f6295ba4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f628c1b07b0 con 0x55c60c0edb40 2024-09-19T08:05:22.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.492+0000 7f6295ba4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f628c1ad9c0 con 0x55c60c0ec5d0 2024-09-19T08:05:22.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.492+0000 7f6295ba4740 10 client.0 _ll_register_callbacks cb 0x7f6288031a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:22.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.492+0000 7f6291d68640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0f7370 0x55c60c16fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6291d68640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0f7370 0x55c60c16fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58046/0 (socket says 172.21.15.98:58046) 2024-09-19T08:05:22.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6291d68640 1 -- 172.21.15.98:0/1598082283 learned_addr learned my addr 172.21.15.98:0/1598082283 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:22.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6290d66640 1 --2- 172.21.15.98:0/1598082283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 0x55c60c0fb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6291567640 1 --2- 172.21.15.98:0/1598082283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0ec5d0 0x55c60c0fabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6291d68640 1 -- 172.21.15.98:0/1598082283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0ec5d0 msgr2=0x55c60c0fabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6291d68640 1 --2- 172.21.15.98:0/1598082283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0ec5d0 0x55c60c0fabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6291d68640 1 -- 172.21.15.98:0/1598082283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 msgr2=0x55c60c0fb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6291d68640 1 --2- 172.21.15.98:0/1598082283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 0x55c60c0fb100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6291d68640 1 -- 172.21.15.98:0/1598082283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c60c170330 con 0x55c60c0f7370 2024-09-19T08:05:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6291567640 1 --2- 172.21.15.98:0/1598082283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0ec5d0 0x55c60c0fabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:05:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6291d68640 1 --2- 172.21.15.98:0/1598082283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0f7370 0x55c60c16fc40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6284008fd0 tx=0x7f62840084f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aad882542d54ad4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6274ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6274ff9640 1 -- 172.21.15.98:0/1598082283 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6284019410 con 0x55c60c0f7370 2024-09-19T08:05:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6274ff9640 1 -- 172.21.15.98:0/1598082283 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f6284020040 con 0x55c60c0f7370 2024-09-19T08:05:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6295ba4740 1 -- 172.21.15.98:0/1598082283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62880042e0 con 0x55c60c0f7370 2024-09-19T08:05:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6274ff9640 1 -- 172.21.15.98:0/1598082283 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f62840195b0 con 0x55c60c0f7370 2024-09-19T08:05:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6295ba4740 10 client.16134 fetch_fsmap learned FSMap version 30 2024-09-19T08:05:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6295ba4740 1 -- 172.21.15.98:0/1598082283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c60c171610 con 0x55c60c0f7370 2024-09-19T08:05:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.493+0000 7f6274ff9640 1 -- 172.21.15.98:0/1598082283 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62880042e0 con 0x55c60c0f7370 2024-09-19T08:05:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.494+0000 7f6274ff9640 1 -- 172.21.15.98:0/1598082283 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f6284024480 con 0x55c60c0f7370 2024-09-19T08:05:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.494+0000 7f6295ba4740 10 client.16134 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:05:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.494+0000 7f6295ba4740 10 client.16134 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:05:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.494+0000 7f6295ba4740 20 client.16134 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.494+0000 7f6295ba4740 1 --2- 172.21.15.98:0/1598082283 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55c60c171a50 0x55c60c173e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.494+0000 7f6295ba4740 4 client.16134 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:05:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.494+0000 7f6295ba4740 1 -- 172.21.15.98:0/1598082283 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c60c059d70 con 0x55c60c171a50 2024-09-19T08:05:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.494+0000 7f6291567640 1 --2- 172.21.15.98:0/1598082283 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55c60c171a50 0x55c60c173e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.494+0000 7f6291567640 1 --2- 172.21.15.98:0/1598082283 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55c60c171a50 0x55c60c173e70 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9572160c1444e84e server_cookie=7aced85bd88364cf in_seq=0 out_seq=0 2024-09-19T08:05:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.494+0000 7f6274ff9640 10 client.16134 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:05:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6274ff9640 1 -- 172.21.15.98:0/1598082283 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c60c059d70 con 0x55c60c171a50 2024-09-19T08:05:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6274ff9640 10 client.16134 handle_command_reply: tid=0 2024-09-19T08:05:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6295ba4740 1 client.16134 shutdown 2024-09-19T08:05:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6295ba4740 1 -- 172.21.15.98:0/1598082283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0f7370 msgr2=0x55c60c16fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6295ba4740 1 --2- 172.21.15.98:0/1598082283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0f7370 0x55c60c16fc40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6284008fd0 tx=0x7f62840084f0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6295ba4740 1 -- 172.21.15.98:0/1598082283 shutdown_connections 2024-09-19T08:05:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6295ba4740 1 --2- 172.21.15.98:0/1598082283 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55c60c171a50 0x55c60c173e70 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6295ba4740 1 --2- 172.21.15.98:0/1598082283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c60c0f7370 0x55c60c16fc40 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6295ba4740 1 --2- 172.21.15.98:0/1598082283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c60c0edb40 0x55c60c0fb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6274ff9640 0 client.16134 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:05:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6295ba4740 1 --2- 172.21.15.98:0/1598082283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60c0ec5d0 0x55c60c0fabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6295ba4740 1 -- 172.21.15.98:0/1598082283 >> 172.21.15.98:0/1598082283 conn(0x55c60c0b5310 msgr2=0x55c60c0f24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6295ba4740 1 -- 172.21.15.98:0/1598082283 shutdown_connections 2024-09-19T08:05:22.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6295ba4740 1 -- 172.21.15.98:0/1598082283 wait complete. 2024-09-19T08:05:22.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f6295ba4740 20 client.16134 trim_cache size 0 max 16384 2024-09-19T08:05:22.508 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:05:22.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f62937cb640 1 -- 172.21.15.98:0/3290121361 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f62780a33b0 msgr2=0x7f62780a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:22.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f62937cb640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f62780a33b0 0x7f62780a5870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f628401e000 tx=0x7f628401f000 comp rx=0 tx=0).stop 2024-09-19T08:05:22.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f62937cb640 1 -- 172.21.15.98:0/3290121361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c101420 msgr2=0x7f628c1975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:22.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.495+0000 7f62937cb640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c101420 0x7f628c1975c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f627c00c920 tx=0x7f627c00cde0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.496+0000 7f62937cb640 1 -- 172.21.15.98:0/3290121361 shutdown_connections 2024-09-19T08:05:22.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.496+0000 7f62937cb640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f62780a33b0 0x7f62780a5870 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.496+0000 7f62937cb640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f628c106a80 0x7f628c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.496+0000 7f62937cb640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c101420 0x7f628c1975c0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.496+0000 7f62937cb640 1 --2- 172.21.15.98:0/3290121361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c100a80 0x7f628c197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:22.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.496+0000 7f62937cb640 1 -- 172.21.15.98:0/3290121361 >> 172.21.15.98:0/3290121361 conn(0x7f628c0fc7a0 msgr2=0x7f628c0fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:22.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.496+0000 7f62937cb640 1 -- 172.21.15.98:0/3290121361 shutdown_connections 2024-09-19T08:05:22.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:22.496+0000 7f62937cb640 1 -- 172.21.15.98:0/3290121361 wait complete. 2024-09-19T08:05:22.816 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:22.816 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:22.816 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:22.817 INFO:tasks.workunit.client.0.smithi098.stdout:iozone test complete. 2024-09-19T08:05:22.817 INFO:tasks.workunit.client.0.smithi098.stderr:+ iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-09-19T08:05:22.883 INFO:tasks.workunit.client.0.smithi098.stdout: Iozone: Performance Test of File I/O 2024-09-19T08:05:22.944 INFO:tasks.workunit.client.0.smithi098.stdout: Version $Revision: 3.494 $ 2024-09-19T08:05:22.944 INFO:tasks.workunit.client.0.smithi098.stdout: Compiled for 64 bit mode. 2024-09-19T08:05:22.944 INFO:tasks.workunit.client.0.smithi098.stdout: Build: linux-AMD64 2024-09-19T08:05:22.944 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:22.944 INFO:tasks.workunit.client.0.smithi098.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-09-19T08:05:22.944 INFO:tasks.workunit.client.0.smithi098.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-09-19T08:05:22.945 INFO:tasks.workunit.client.0.smithi098.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-09-19T08:05:22.945 INFO:tasks.workunit.client.0.smithi098.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-09-19T08:05:22.945 INFO:tasks.workunit.client.0.smithi098.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-09-19T08:05:22.945 INFO:tasks.workunit.client.0.smithi098.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-09-19T08:05:22.945 INFO:tasks.workunit.client.0.smithi098.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-09-19T08:05:22.945 INFO:tasks.workunit.client.0.smithi098.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-09-19T08:05:22.945 INFO:tasks.workunit.client.0.smithi098.stdout: Alexey Skidanov, Sudhir Kumar. 2024-09-19T08:05:22.945 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:22.945 INFO:tasks.workunit.client.0.smithi098.stdout: Run began: Thu Sep 19 08:05:22 2024 2024-09-19T08:05:22.945 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:22.945 INFO:tasks.workunit.client.0.smithi098.stdout: Include close in write timing 2024-09-19T08:05:22.945 INFO:tasks.workunit.client.0.smithi098.stdout: Include fsync in write timing 2024-09-19T08:05:22.945 INFO:tasks.workunit.client.0.smithi098.stdout: File size set to 1048576 kB 2024-09-19T08:05:22.946 INFO:tasks.workunit.client.0.smithi098.stdout: Record Size 1024 kB 2024-09-19T08:05:22.946 INFO:tasks.workunit.client.0.smithi098.stdout: Command line used: iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-09-19T08:05:22.946 INFO:tasks.workunit.client.0.smithi098.stdout: Output is in kBytes/sec 2024-09-19T08:05:22.946 INFO:tasks.workunit.client.0.smithi098.stdout: Time Resolution = 0.000001 seconds. 2024-09-19T08:05:22.946 INFO:tasks.workunit.client.0.smithi098.stdout: Processor cache size set to 1024 kBytes. 2024-09-19T08:05:22.946 INFO:tasks.workunit.client.0.smithi098.stdout: Processor cache line size set to 32 bytes. 2024-09-19T08:05:22.946 INFO:tasks.workunit.client.0.smithi098.stdout: File stride size set to 17 * record size. 2024-09-19T08:05:22.946 INFO:tasks.workunit.client.0.smithi098.stdout: Throughput test with 1 process 2024-09-19T08:05:22.946 INFO:tasks.workunit.client.0.smithi098.stdout: Each process writes a 1048576 kByte file in 1024 kByte records 2024-09-19T08:05:23.175 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:22 smithi098 ceph-mon[31333]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:05:23.176 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:22 smithi098 ceph-mon[31333]: pgmap v74: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 MiB/s wr, 34 op/s 2024-09-19T08:05:23.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:22 smithi139 ceph-mon[34874]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:05:23.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:22 smithi139 ceph-mon[34874]: pgmap v74: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 MiB/s wr, 34 op/s 2024-09-19T08:05:23.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:22 smithi162 ceph-mon[35260]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:05:23.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:22 smithi162 ceph-mon[35260]: pgmap v74: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 MiB/s wr, 34 op/s 2024-09-19T08:05:23.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:05:24.045 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:05:24.510 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:24.510 INFO:tasks.workunit.client.0.smithi098.stdout: Children see throughput for 1 initial writers = 657776.94 kB/sec 2024-09-19T08:05:24.510 INFO:tasks.workunit.client.0.smithi098.stdout: Parent sees throughput for 1 initial writers = 657686.65 kB/sec 2024-09-19T08:05:24.510 INFO:tasks.workunit.client.0.smithi098.stdout: Min throughput per process = 657776.94 kB/sec 2024-09-19T08:05:24.510 INFO:tasks.workunit.client.0.smithi098.stdout: Max throughput per process = 657776.94 kB/sec 2024-09-19T08:05:24.510 INFO:tasks.workunit.client.0.smithi098.stdout: Avg throughput per process = 657776.94 kB/sec 2024-09-19T08:05:24.510 INFO:tasks.workunit.client.0.smithi098.stdout: Min xfer = 1048576.00 kB 2024-09-19T08:05:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.578+0000 7fb1cdea2640 1 -- 172.21.15.98:0/1958469129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8102d90 msgr2=0x7fb1c8103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.578+0000 7fb1cdea2640 1 --2- 172.21.15.98:0/1958469129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8102d90 0x7fb1c8103170 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8009930 tx=0x7fb1b802a420 comp rx=0 tx=0).stop 2024-09-19T08:05:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.578+0000 7fb1cdea2640 1 -- 172.21.15.98:0/1958469129 shutdown_connections 2024-09-19T08:05:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.578+0000 7fb1cdea2640 1 --2- 172.21.15.98:0/1958469129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1c8108290 0x7fb1c810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.578+0000 7fb1cdea2640 1 --2- 172.21.15.98:0/1958469129 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c8103740 0x7fb1c8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.578+0000 7fb1cdea2640 1 --2- 172.21.15.98:0/1958469129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8102d90 0x7fb1c8103170 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.578+0000 7fb1cdea2640 1 -- 172.21.15.98:0/1958469129 >> 172.21.15.98:0/1958469129 conn(0x7fb1c80fc7d0 msgr2=0x7fb1c80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.578+0000 7fb1cdea2640 1 -- 172.21.15.98:0/1958469129 shutdown_connections 2024-09-19T08:05:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.578+0000 7fb1cdea2640 1 -- 172.21.15.98:0/1958469129 wait complete. 2024-09-19T08:05:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.578+0000 7fb1cdea2640 1 Processor -- start 2024-09-19T08:05:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1cdea2640 1 -- start start 2024-09-19T08:05:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1cdea2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8102d90 0x7fb1c81009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1cdea2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c8103740 0x7fb1c8100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1cdea2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1c8108290 0x7fb1c8101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1cdea2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c8112090 con 0x7fb1c8102d90 2024-09-19T08:05:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1cdea2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c8111f10 con 0x7fb1c8103740 2024-09-19T08:05:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1cdea2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c8112210 con 0x7fb1c8108290 2024-09-19T08:05:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8102d90 0x7fb1c81009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8102d90 0x7fb1c81009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58064/0 (socket says 172.21.15.98:58064) 2024-09-19T08:05:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1c77fe640 1 -- 172.21.15.98:0/1392850568 learned_addr learned my addr 172.21.15.98:0/1392850568 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1c7fff640 1 --2- 172.21.15.98:0/1392850568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1c8108290 0x7fb1c8101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1c6ffd640 1 --2- 172.21.15.98:0/1392850568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c8103740 0x7fb1c8100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1c77fe640 1 -- 172.21.15.98:0/1392850568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1c8108290 msgr2=0x7fb1c8101470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/1392850568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1c8108290 0x7fb1c8101470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1c77fe640 1 -- 172.21.15.98:0/1392850568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c8103740 msgr2=0x7fb1c8100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/1392850568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c8103740 0x7fb1c8100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.579+0000 7fb1c77fe640 1 -- 172.21.15.98:0/1392850568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c81a6a90 con 0x7fb1c8102d90 2024-09-19T08:05:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.580+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/1392850568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8102d90 0x7fb1c81009f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8009230 tx=0x7fb1b8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.580+0000 7fb1c4ff9640 1 -- 172.21.15.98:0/1392850568 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b803f030 con 0x7fb1c8102d90 2024-09-19T08:05:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.580+0000 7fb1c4ff9640 1 -- 172.21.15.98:0/1392850568 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b8003ad0 con 0x7fb1c8102d90 2024-09-19T08:05:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.580+0000 7fb1cdea2640 1 -- 172.21.15.98:0/1392850568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c81a6d20 con 0x7fb1c8102d90 2024-09-19T08:05:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.580+0000 7fb1c4ff9640 1 -- 172.21.15.98:0/1392850568 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8035070 con 0x7fb1c8102d90 2024-09-19T08:05:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.580+0000 7fb1cdea2640 1 -- 172.21.15.98:0/1392850568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c81a71b0 con 0x7fb1c8102d90 2024-09-19T08:05:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.582+0000 7fb1c4ff9640 1 -- 172.21.15.98:0/1392850568 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fb1b8042030 con 0x7fb1c8102d90 2024-09-19T08:05:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.582+0000 7fb1d027b740 1 Processor -- start 2024-09-19T08:05:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.582+0000 7fb1d027b740 1 -- start start 2024-09-19T08:05:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.582+0000 7fb1d027b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e4bb40 0x563390e4a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.582+0000 7fb1d027b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e4aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.582+0000 7fb1d027b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e55370 0x563390e57760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.582+0000 7fb1d027b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563390db02f0 con 0x563390e4a5d0 2024-09-19T08:05:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.582+0000 7fb1d027b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563390d57cf0 con 0x563390e55370 2024-09-19T08:05:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.582+0000 7fb1d027b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563390dac2b0 con 0x563390e4bb40 2024-09-19T08:05:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e4aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e4aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58072/0 (socket says 172.21.15.98:58072) 2024-09-19T08:05:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c77fe640 1 -- 172.21.15.98:0/2264295317 learned_addr learned my addr 172.21.15.98:0/2264295317 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c4ff9640 1 --2- 172.21.15.98:0/1392850568 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb1b00a33b0 0x7fb1b00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c6ffd640 1 --2- 172.21.15.98:0/2264295317 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e55370 0x563390e57760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c7fff640 1 --2- 172.21.15.98:0/2264295317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e4bb40 0x563390e4a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c77fe640 1 -- 172.21.15.98:0/2264295317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e4bb40 msgr2=0x563390e4a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/2264295317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e4bb40 0x563390e4a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c77fe640 1 -- 172.21.15.98:0/2264295317 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e55370 msgr2=0x563390e57760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c7fff640 1 --2- 172.21.15.98:0/1392850568 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb1b00a33b0 0x7fb1b00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/2264295317 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e55370 0x563390e57760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c77fe640 1 -- 172.21.15.98:0/2264295317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563390d3c3a0 con 0x563390e4a5d0 2024-09-19T08:05:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c4ff9640 1 -- 172.21.15.98:0/1392850568 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fb1b80f00a0 con 0x7fb1c8102d90 2024-09-19T08:05:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/2264295317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e4aa10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb1b80ed830 tx=0x7fb1b8047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4f94283dd50f11 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1c7fff640 1 --2- 172.21.15.98:0/1392850568 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb1b00a33b0 0x7fb1b00a5870 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0005fd0 tx=0x7fb1c0004ab0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1adffb640 1 -- 172.21.15.98:0/2264295317 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8049050 con 0x563390e4a5d0 2024-09-19T08:05:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1adffb640 1 -- 172.21.15.98:0/2264295317 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b80c0800 con 0x563390e4a5d0 2024-09-19T08:05:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.583+0000 7fb1adffb640 1 -- 172.21.15.98:0/2264295317 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b803b510 con 0x563390e4a5d0 2024-09-19T08:05:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.584+0000 7fb1d027b740 1 -- 172.21.15.98:0/2264295317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 msgr2=0x563390e4aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.584+0000 7fb1d027b740 1 --2- 172.21.15.98:0/2264295317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e4aa10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb1b80ed830 tx=0x7fb1b8047000 comp rx=0 tx=0).stop 2024-09-19T08:05:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.584+0000 7fb1d027b740 1 -- 172.21.15.98:0/2264295317 shutdown_connections 2024-09-19T08:05:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.584+0000 7fb1d027b740 1 --2- 172.21.15.98:0/2264295317 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e55370 0x563390e57760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.584+0000 7fb1d027b740 1 --2- 172.21.15.98:0/2264295317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e4aa10 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.584+0000 7fb1d027b740 1 --2- 172.21.15.98:0/2264295317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e4bb40 0x563390e4a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.584+0000 7fb1d027b740 1 -- 172.21.15.98:0/2264295317 >> 172.21.15.98:0/2264295317 conn(0x563390e13310 msgr2=0x563390e46c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.584+0000 7fb1d027b740 1 -- 172.21.15.98:0/2264295317 shutdown_connections 2024-09-19T08:05:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.584+0000 7fb1d027b740 1 -- 172.21.15.98:0/2264295317 wait complete. 2024-09-19T08:05:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.584+0000 7fb1d027b740 1 -- start start 2024-09-19T08:05:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1d027b740 1 Processor -- start 2024-09-19T08:05:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1d027b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e54d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1d027b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e4bb40 0x563390e51fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1d027b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e55370 0x563390e52570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1d027b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563390db02f0 con 0x563390e4a5d0 2024-09-19T08:05:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1d027b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563390d6b060 con 0x563390e55370 2024-09-19T08:05:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1d027b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563390e57e30 con 0x563390e4bb40 2024-09-19T08:05:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1d027b740 10 client.0 _ll_register_callbacks cb 0x563390abe8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e54d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e54d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58088/0 (socket says 172.21.15.98:58088) 2024-09-19T08:05:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1c77fe640 1 -- 172.21.15.98:0/1131291262 learned_addr learned my addr 172.21.15.98:0/1131291262 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1c6ffd640 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e4bb40 0x563390e51fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1c7fff640 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e55370 0x563390e52570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1c77fe640 1 -- 172.21.15.98:0/1131291262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e4bb40 msgr2=0x563390e51fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e4bb40 0x563390e51fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1c77fe640 1 -- 172.21.15.98:0/1131291262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e55370 msgr2=0x563390e52570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e55370 0x563390e52570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1c77fe640 1 -- 172.21.15.98:0/1131291262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563390b213c0 con 0x563390e4a5d0 2024-09-19T08:05:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1c6ffd640 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e4bb40 0x563390e51fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:05:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1c7fff640 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e55370 0x563390e52570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:05:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e54d20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb1b803d040 tx=0x7fb1b80ed900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40f184e25cd1588a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.585+0000 7fb1a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/1131291262 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b80d5060 con 0x563390e4a5d0 2024-09-19T08:05:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/1131291262 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fb1b8003c80 con 0x563390e4a5d0 2024-09-19T08:05:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1d027b740 1 -- 172.21.15.98:0/1131291262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1b80c0800 con 0x563390e4a5d0 2024-09-19T08:05:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/1131291262 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb1b8049040 con 0x563390e4a5d0 2024-09-19T08:05:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1d027b740 10 client.16146 fetch_fsmap learned FSMap version 30 2024-09-19T08:05:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1d027b740 1 -- 172.21.15.98:0/1131291262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563390e63730 con 0x563390e4a5d0 2024-09-19T08:05:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/1131291262 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b80c0800 con 0x563390e4a5d0 2024-09-19T08:05:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/1131291262 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fb1b80d5200 con 0x563390e4a5d0 2024-09-19T08:05:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1d027b740 10 client.16146 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:05:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1d027b740 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:05:24.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1d027b740 20 client.16146 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:24.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563390e64900 0x563390e66cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1d027b740 4 client.16146 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:05:24.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1d027b740 1 -- 172.21.15.98:0/1131291262 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563390dbd860 con 0x563390e64900 2024-09-19T08:05:24.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.586+0000 7fb1c6ffd640 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563390e64900 0x563390e66cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.587+0000 7fb1c6ffd640 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563390e64900 0x563390e66cf0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=177a74f3f33f0444 server_cookie=749dc088ef218277 in_seq=0 out_seq=0 2024-09-19T08:05:24.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.587+0000 7fb1a6ffd640 10 client.16146 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:24.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.587+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/1131291262 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563390dbd860 con 0x563390e64900 2024-09-19T08:05:24.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.587+0000 7fb1a6ffd640 10 client.16146 handle_command_reply: tid=0 2024-09-19T08:05:24.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.587+0000 7fb1d027b740 1 client.16146 shutdown 2024-09-19T08:05:24.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.588+0000 7fb1d027b740 1 -- 172.21.15.98:0/1131291262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 msgr2=0x563390e54d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:24.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.588+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e54d20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb1b803d040 tx=0x7fb1b80ed900 comp rx=0 tx=0).stop 2024-09-19T08:05:24.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.588+0000 7fb1d027b740 1 -- 172.21.15.98:0/1131291262 shutdown_connections 2024-09-19T08:05:24.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.588+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563390e64900 0x563390e66cf0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.588+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e55370 0x563390e52570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.588+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e4bb40 0x563390e51fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.588+0000 7fb1a6ffd640 0 client.16146 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:24.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.588+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1131291262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e54d20 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.588+0000 7fb1d027b740 1 -- 172.21.15.98:0/1131291262 >> 172.21.15.98:0/1131291262 conn(0x563390e13310 msgr2=0x563390e4c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:24.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.588+0000 7fb1d027b740 1 -- 172.21.15.98:0/1131291262 shutdown_connections 2024-09-19T08:05:24.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.588+0000 7fb1d027b740 1 -- 172.21.15.98:0/1131291262 wait complete. 2024-09-19T08:05:24.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.588+0000 7fb1d027b740 20 client.16146 trim_cache size 0 max 16384 2024-09-19T08:05:24.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.600+0000 7fb1d027b740 1 Processor -- start 2024-09-19T08:05:24.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.600+0000 7fb1d027b740 1 -- start start 2024-09-19T08:05:24.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.600+0000 7fb1d027b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e4d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.600+0000 7fb1d027b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e4bb40 0x563390e4d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.600+0000 7fb1d027b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e55370 0x563390e6c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1d027b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c81afb10 con 0x563390e4a5d0 2024-09-19T08:05:24.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1d027b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c81b16b0 con 0x563390e4bb40 2024-09-19T08:05:24.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1d027b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c81b01e0 con 0x563390e55370 2024-09-19T08:05:24.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e4d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e4d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58102/0 (socket says 172.21.15.98:58102) 2024-09-19T08:05:24.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1c77fe640 1 -- 172.21.15.98:0/1081960082 learned_addr learned my addr 172.21.15.98:0/1081960082 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:24.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1c7fff640 1 --2- 172.21.15.98:0/1081960082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e55370 0x563390e6c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1c77fe640 1 -- 172.21.15.98:0/1081960082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e55370 msgr2=0x563390e6c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:24.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1c6ffd640 1 --2- 172.21.15.98:0/1081960082 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e4bb40 0x563390e4d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/1081960082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e55370 0x563390e6c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1c77fe640 1 -- 172.21.15.98:0/1081960082 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e4bb40 msgr2=0x563390e4d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:24.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/1081960082 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e4bb40 0x563390e4d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1c77fe640 1 -- 172.21.15.98:0/1081960082 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563390e50520 con 0x563390e4a5d0 2024-09-19T08:05:24.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1c6ffd640 1 --2- 172.21.15.98:0/1081960082 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e4bb40 0x563390e4d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:05:24.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/1081960082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e4d120 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb1b80bc490 tx=0x7fb1b80baa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38220c97a675c11 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:24.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1ad7fa640 1 -- 172.21.15.98:0/1081960082 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b80d5040 con 0x563390e4a5d0 2024-09-19T08:05:24.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1ad7fa640 1 -- 172.21.15.98:0/1081960082 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b80c0800 con 0x563390e4a5d0 2024-09-19T08:05:24.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.601+0000 7fb1ad7fa640 1 -- 172.21.15.98:0/1081960082 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b80b8d10 con 0x563390e4a5d0 2024-09-19T08:05:24.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.602+0000 7fb1d027b740 1 -- 172.21.15.98:0/1081960082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 msgr2=0x563390e4d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:24.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.602+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1081960082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e4d120 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb1b80bc490 tx=0x7fb1b80baa10 comp rx=0 tx=0).stop 2024-09-19T08:05:24.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.602+0000 7fb1d027b740 1 -- 172.21.15.98:0/1081960082 shutdown_connections 2024-09-19T08:05:24.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.602+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1081960082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e55370 0x563390e6c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.602+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1081960082 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e4bb40 0x563390e4d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.602+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1081960082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4a5d0 0x563390e4d120 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.602+0000 7fb1d027b740 1 -- 172.21.15.98:0/1081960082 >> 172.21.15.98:0/1081960082 conn(0x563390e13310 msgr2=0x563390e45b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:24.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.602+0000 7fb1d027b740 1 -- 172.21.15.98:0/1081960082 shutdown_connections 2024-09-19T08:05:24.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.602+0000 7fb1d027b740 1 -- 172.21.15.98:0/1081960082 wait complete. 2024-09-19T08:05:24.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.602+0000 7fb1d027b740 1 -- start start 2024-09-19T08:05:24.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.602+0000 7fb1d027b740 1 Processor -- start 2024-09-19T08:05:24.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1d027b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e4a5d0 0x563390e4f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1d027b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4bb40 0x563390e4f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1d027b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e55370 0x563390e4fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1d027b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c81b0fb0 con 0x563390e4bb40 2024-09-19T08:05:24.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1d027b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c81b08e0 con 0x563390e4a5d0 2024-09-19T08:05:24.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1d027b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c81adaf0 con 0x563390e55370 2024-09-19T08:05:24.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1d027b740 10 client.0 _ll_register_callbacks cb 0x7fb1b80b8040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:24.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1c6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4bb40 0x563390e4f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1c6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4bb40 0x563390e4f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58114/0 (socket says 172.21.15.98:58114) 2024-09-19T08:05:24.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1c6ffd640 1 -- 172.21.15.98:0/1180496900 learned_addr learned my addr 172.21.15.98:0/1180496900 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:24.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/1180496900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e4a5d0 0x563390e4f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1c7fff640 1 --2- 172.21.15.98:0/1180496900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e55370 0x563390e4fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1c6ffd640 1 -- 172.21.15.98:0/1180496900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e55370 msgr2=0x563390e4fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:24.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1c6ffd640 1 --2- 172.21.15.98:0/1180496900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e55370 0x563390e4fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1c6ffd640 1 -- 172.21.15.98:0/1180496900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e4a5d0 msgr2=0x563390e4f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:24.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1c6ffd640 1 --2- 172.21.15.98:0/1180496900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e4a5d0 0x563390e4f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1c6ffd640 1 -- 172.21.15.98:0/1180496900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563390ece740 con 0x563390e4bb40 2024-09-19T08:05:24.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1c6ffd640 1 --2- 172.21.15.98:0/1180496900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4bb40 0x563390e4f6c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc005d00 tx=0x7fb1bc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60229f75019ca1ad server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:24.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:24.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/1180496900 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc005220 con 0x563390e4bb40 2024-09-19T08:05:24.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/1180496900 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fb1bc0053c0 con 0x563390e4bb40 2024-09-19T08:05:24.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1d027b740 1 -- 172.21.15.98:0/1180496900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1b80c0800 con 0x563390e4bb40 2024-09-19T08:05:24.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/1180496900 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb1bc02d050 con 0x563390e4bb40 2024-09-19T08:05:24.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1d027b740 10 client.16152 fetch_fsmap learned FSMap version 30 2024-09-19T08:05:24.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.603+0000 7fb1d027b740 1 -- 172.21.15.98:0/1180496900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563390ecf990 con 0x563390e4bb40 2024-09-19T08:05:24.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.604+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/1180496900 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b80c0800 con 0x563390e4bb40 2024-09-19T08:05:24.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.604+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/1180496900 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fb1bc004ad0 con 0x563390e4bb40 2024-09-19T08:05:24.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.604+0000 7fb1d027b740 10 client.16152 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:05:24.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.604+0000 7fb1d027b740 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:05:24.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.604+0000 7fb1d027b740 20 client.16152 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:24.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.604+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1180496900 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563390ed00c0 0x563390ed24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:24.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.604+0000 7fb1d027b740 4 client.16152 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:05:24.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.604+0000 7fb1d027b740 1 -- 172.21.15.98:0/1180496900 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563390db7d70 con 0x563390ed00c0 2024-09-19T08:05:24.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.604+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/1180496900 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563390ed00c0 0x563390ed24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:24.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.604+0000 7fb1a6ffd640 10 client.16152 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:24.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.604+0000 7fb1c77fe640 1 --2- 172.21.15.98:0/1180496900 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563390ed00c0 0x563390ed24e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3b7b39d74075028 server_cookie=22ec17aae85bbde7 in_seq=0 out_seq=0 2024-09-19T08:05:24.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/1180496900 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563390db7d70 con 0x563390ed00c0 2024-09-19T08:05:24.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1a6ffd640 10 client.16152 handle_command_reply: tid=0 2024-09-19T08:05:24.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1d027b740 1 client.16152 shutdown 2024-09-19T08:05:24.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1d027b740 1 -- 172.21.15.98:0/1180496900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4bb40 msgr2=0x563390e4f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:24.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1180496900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4bb40 0x563390e4f6c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc005d00 tx=0x7fb1bc011810 comp rx=0 tx=0).stop 2024-09-19T08:05:24.994 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:05:24.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1d027b740 1 -- 172.21.15.98:0/1180496900 shutdown_connections 2024-09-19T08:05:24.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1180496900 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563390ed00c0 0x563390ed24e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1a6ffd640 0 client.16152 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:24.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1180496900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563390e55370 0x563390e4fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1180496900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563390e4bb40 0x563390e4f6c0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1d027b740 1 --2- 172.21.15.98:0/1180496900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563390e4a5d0 0x563390e4f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1d027b740 1 -- 172.21.15.98:0/1180496900 >> 172.21.15.98:0/1180496900 conn(0x563390e13310 msgr2=0x563390e45310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:24.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1d027b740 1 -- 172.21.15.98:0/1180496900 shutdown_connections 2024-09-19T08:05:24.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1d027b740 1 -- 172.21.15.98:0/1180496900 wait complete. 2024-09-19T08:05:24.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.605+0000 7fb1d027b740 20 client.16152 trim_cache size 0 max 16384 2024-09-19T08:05:24.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.606+0000 7fb1cdea2640 1 -- 172.21.15.98:0/1392850568 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb1b00a33b0 msgr2=0x7fb1b00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:24.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.606+0000 7fb1cdea2640 1 --2- 172.21.15.98:0/1392850568 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb1b00a33b0 0x7fb1b00a5870 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0005fd0 tx=0x7fb1c0004ab0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.606+0000 7fb1cdea2640 1 -- 172.21.15.98:0/1392850568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8102d90 msgr2=0x7fb1c81009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:24.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.606+0000 7fb1cdea2640 1 --2- 172.21.15.98:0/1392850568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8102d90 0x7fb1c81009f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8009230 tx=0x7fb1b8004270 comp rx=0 tx=0).stop 2024-09-19T08:05:24.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.606+0000 7fb1cdea2640 1 -- 172.21.15.98:0/1392850568 shutdown_connections 2024-09-19T08:05:24.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.606+0000 7fb1cdea2640 1 --2- 172.21.15.98:0/1392850568 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb1b00a33b0 0x7fb1b00a5870 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.606+0000 7fb1cdea2640 1 --2- 172.21.15.98:0/1392850568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1c8108290 0x7fb1c8101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.606+0000 7fb1cdea2640 1 --2- 172.21.15.98:0/1392850568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c8103740 0x7fb1c8100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.606+0000 7fb1cdea2640 1 --2- 172.21.15.98:0/1392850568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8102d90 0x7fb1c81009f0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:24.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.606+0000 7fb1cdea2640 1 -- 172.21.15.98:0/1392850568 >> 172.21.15.98:0/1392850568 conn(0x7fb1c80fc7d0 msgr2=0x7fb1c8106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:24.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.606+0000 7fb1cdea2640 1 -- 172.21.15.98:0/1392850568 shutdown_connections 2024-09-19T08:05:24.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:24.606+0000 7fb1cdea2640 1 -- 172.21.15.98:0/1392850568 wait complete. 2024-09-19T08:05:25.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:24 smithi162 ceph-mon[35260]: pgmap v75: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-09-19T08:05:25.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:24 smithi139 ceph-mon[34874]: pgmap v75: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-09-19T08:05:25.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:24 smithi098 ceph-mon[31333]: pgmap v75: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-09-19T08:05:26.695 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:05:26.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:05:26.973 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:05:27.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:26 smithi098 ceph-mon[31333]: pgmap v76: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 62 MiB/s wr, 46 op/s 2024-09-19T08:05:27.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:26 smithi162 ceph-mon[35260]: pgmap v76: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 62 MiB/s wr, 46 op/s 2024-09-19T08:05:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:26 smithi139 ceph-mon[34874]: pgmap v76: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 62 MiB/s wr, 46 op/s 2024-09-19T08:05:27.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.602+0000 7fb381b43640 1 -- 172.21.15.98:0/1549401010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb37c1027a0 msgr2=0x7fb37c1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:27.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.602+0000 7fb381b43640 1 --2- 172.21.15.98:0/1549401010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb37c1027a0 0x7fb37c1067f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb370009900 tx=0x7fb37002a440 comp rx=0 tx=0).stop 2024-09-19T08:05:27.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.603+0000 7fb381b43640 1 -- 172.21.15.98:0/1549401010 shutdown_connections 2024-09-19T08:05:27.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.603+0000 7fb381b43640 1 --2- 172.21.15.98:0/1549401010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb37c106f20 0x7fb37c10eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.603+0000 7fb381b43640 1 --2- 172.21.15.98:0/1549401010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb37c1027a0 0x7fb37c1067f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.603+0000 7fb381b43640 1 --2- 172.21.15.98:0/1549401010 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb37c101df0 0x7fb37c1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.603+0000 7fb381b43640 1 -- 172.21.15.98:0/1549401010 >> 172.21.15.98:0/1549401010 conn(0x7fb37c0fd8f0 msgr2=0x7fb37c0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:27.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.603+0000 7fb381b43640 1 -- 172.21.15.98:0/1549401010 shutdown_connections 2024-09-19T08:05:27.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.603+0000 7fb381b43640 1 -- 172.21.15.98:0/1549401010 wait complete. 2024-09-19T08:05:27.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.604+0000 7fb381b43640 1 Processor -- start 2024-09-19T08:05:27.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.604+0000 7fb381b43640 1 -- start start 2024-09-19T08:05:27.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.604+0000 7fb381b43640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb37c101df0 0x7fb37c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.604+0000 7fb381b43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb37c1027a0 0x7fb37c074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.604+0000 7fb381b43640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb37c106f20 0x7fb37c074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.604+0000 7fb381b43640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c1120b0 con 0x7fb37c1027a0 2024-09-19T08:05:27.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.605+0000 7fb381b43640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c111f30 con 0x7fb37c106f20 2024-09-19T08:05:27.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.605+0000 7fb381b43640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c112230 con 0x7fb37c101df0 2024-09-19T08:05:27.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.605+0000 7fb37affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb37c1027a0 0x7fb37c074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:27.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.605+0000 7fb37affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb37c1027a0 0x7fb37c074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60698/0 (socket says 172.21.15.98:60698) 2024-09-19T08:05:27.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.605+0000 7fb37affd640 1 -- 172.21.15.98:0/77910854 learned_addr learned my addr 172.21.15.98:0/77910854 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.605+0000 7fb37affd640 1 -- 172.21.15.98:0/77910854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb37c101df0 msgr2=0x7fb37c0744f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T08:05:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.605+0000 7fb37affd640 1 --2- 172.21.15.98:0/77910854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb37c101df0 0x7fb37c0744f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.605+0000 7fb37affd640 1 -- 172.21.15.98:0/77910854 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb37c106f20 msgr2=0x7fb37c074f70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T08:05:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.605+0000 7fb37affd640 1 --2- 172.21.15.98:0/77910854 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb37c106f20 0x7fb37c074f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.605+0000 7fb37affd640 1 -- 172.21.15.98:0/77910854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37c1a6ab0 con 0x7fb37c1027a0 2024-09-19T08:05:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.606+0000 7fb37affd640 1 --2- 172.21.15.98:0/77910854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb37c1027a0 0x7fb37c074a30 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb37002aa30 tx=0x7fb370003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.606+0000 7fb378ff9640 1 -- 172.21.15.98:0/77910854 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37003f070 con 0x7fb37c1027a0 2024-09-19T08:05:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.606+0000 7fb378ff9640 1 -- 172.21.15.98:0/77910854 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb37003b460 con 0x7fb37c1027a0 2024-09-19T08:05:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.606+0000 7fb381b43640 1 -- 172.21.15.98:0/77910854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb37c1a6d40 con 0x7fb37c1027a0 2024-09-19T08:05:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.606+0000 7fb378ff9640 1 -- 172.21.15.98:0/77910854 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb370035030 con 0x7fb37c1027a0 2024-09-19T08:05:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.606+0000 7fb381b43640 1 -- 172.21.15.98:0/77910854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb37c1a71c0 con 0x7fb37c1027a0 2024-09-19T08:05:27.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.609+0000 7fb378ff9640 1 -- 172.21.15.98:0/77910854 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fb370042070 con 0x7fb37c1027a0 2024-09-19T08:05:27.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.609+0000 7fb383f1c740 1 Processor -- start 2024-09-19T08:05:27.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.609+0000 7fb383f1c740 1 -- start start 2024-09-19T08:05:27.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb383f1c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b6fab60 0x56124b6faf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb383f1c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fb480 0x56124b705800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb383f1c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b705d40 0x56124b708130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb383f1c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56124b660280 con 0x56124b6fb480 2024-09-19T08:05:27.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb383f1c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56124b607b10 con 0x56124b6fab60 2024-09-19T08:05:27.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb383f1c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56124b65c240 con 0x56124b705d40 2024-09-19T08:05:27.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb37b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fb480 0x56124b705800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:27.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb37b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fb480 0x56124b705800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60700/0 (socket says 172.21.15.98:60700) 2024-09-19T08:05:27.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb37b7fe640 1 -- 172.21.15.98:0/3279080568 learned_addr learned my addr 172.21.15.98:0/3279080568 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:27.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb378ff9640 1 --2- 172.21.15.98:0/77910854 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb3640a33b0 0x7fb3640a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb37b7fe640 1 -- 172.21.15.98:0/3279080568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b705d40 msgr2=0x56124b708130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T08:05:27.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb37b7fe640 1 --2- 172.21.15.98:0/3279080568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b705d40 0x56124b708130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb37b7fe640 1 -- 172.21.15.98:0/3279080568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b6fab60 msgr2=0x56124b6faf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:27.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb37bfff640 1 --2- 172.21.15.98:0/3279080568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b6fab60 0x56124b6faf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:27.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.610+0000 7fb37b7fe640 1 --2- 172.21.15.98:0/3279080568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b6fab60 0x56124b6faf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.611+0000 7fb37b7fe640 1 -- 172.21.15.98:0/3279080568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56124b6141f0 con 0x56124b6fb480 2024-09-19T08:05:27.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.611+0000 7fb37bfff640 1 --2- 172.21.15.98:0/77910854 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb3640a33b0 0x7fb3640a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:27.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.611+0000 7fb378ff9640 1 -- 172.21.15.98:0/77910854 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fb3700f07f0 con 0x7fb37c1027a0 2024-09-19T08:05:27.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.611+0000 7fb37b7fe640 1 --2- 172.21.15.98:0/3279080568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fb480 0x56124b705800 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb36c009770 tx=0x7fb36c009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8996cc0ff03d2c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:27.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.611+0000 7fb37bfff640 1 --2- 172.21.15.98:0/77910854 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb3640a33b0 0x7fb3640a5870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb374009d60 tx=0x7fb374004a60 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:27.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.611+0000 7fb361ffb640 1 -- 172.21.15.98:0/3279080568 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c015020 con 0x56124b6fb480 2024-09-19T08:05:27.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.611+0000 7fb361ffb640 1 -- 172.21.15.98:0/3279080568 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb36c011420 con 0x56124b6fb480 2024-09-19T08:05:27.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.611+0000 7fb361ffb640 1 -- 172.21.15.98:0/3279080568 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c0105e0 con 0x56124b6fb480 2024-09-19T08:05:27.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.613+0000 7fb383f1c740 1 -- 172.21.15.98:0/3279080568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fb480 msgr2=0x56124b705800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:27.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.613+0000 7fb383f1c740 1 --2- 172.21.15.98:0/3279080568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fb480 0x56124b705800 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb36c009770 tx=0x7fb36c009c30 comp rx=0 tx=0).stop 2024-09-19T08:05:27.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.613+0000 7fb383f1c740 1 -- 172.21.15.98:0/3279080568 shutdown_connections 2024-09-19T08:05:27.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.613+0000 7fb383f1c740 1 --2- 172.21.15.98:0/3279080568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b705d40 0x56124b708130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.613+0000 7fb383f1c740 1 --2- 172.21.15.98:0/3279080568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fb480 0x56124b705800 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.613+0000 7fb383f1c740 1 --2- 172.21.15.98:0/3279080568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b6fab60 0x56124b6faf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.613+0000 7fb383f1c740 1 -- 172.21.15.98:0/3279080568 >> 172.21.15.98:0/3279080568 conn(0x56124b6f70f0 msgr2=0x56124b6f74d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.613+0000 7fb383f1c740 1 -- 172.21.15.98:0/3279080568 shutdown_connections 2024-09-19T08:05:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.613+0000 7fb383f1c740 1 -- 172.21.15.98:0/3279080568 wait complete. 2024-09-19T08:05:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.613+0000 7fb383f1c740 1 -- start start 2024-09-19T08:05:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.614+0000 7fb383f1c740 1 Processor -- start 2024-09-19T08:05:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.614+0000 7fb383f1c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fab60 0x56124b702be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.614+0000 7fb383f1c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b6fb480 0x56124b703120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.614+0000 7fb383f1c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b705d40 0x56124b703660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.614+0000 7fb383f1c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56124b708910 con 0x56124b6fab60 2024-09-19T08:05:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.614+0000 7fb383f1c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56124b61af10 con 0x56124b6fb480 2024-09-19T08:05:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.614+0000 7fb383f1c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56124b708a90 con 0x56124b705d40 2024-09-19T08:05:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.614+0000 7fb383f1c740 10 client.0 _ll_register_callbacks cb 0x56124b5e3a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.614+0000 7fb37b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fab60 0x56124b702be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.614+0000 7fb37b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fab60 0x56124b702be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60708/0 (socket says 172.21.15.98:60708) 2024-09-19T08:05:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.614+0000 7fb37b7fe640 1 -- 172.21.15.98:0/2020805524 learned_addr learned my addr 172.21.15.98:0/2020805524 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.615+0000 7fb37affd640 1 --2- 172.21.15.98:0/2020805524 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b6fb480 0x56124b703120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.615+0000 7fb37b7fe640 1 -- 172.21.15.98:0/2020805524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b705d40 msgr2=0x56124b703660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T08:05:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.615+0000 7fb37b7fe640 1 --2- 172.21.15.98:0/2020805524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b705d40 0x56124b703660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.615+0000 7fb37b7fe640 1 -- 172.21.15.98:0/2020805524 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b6fb480 msgr2=0x56124b703120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.615+0000 7fb37b7fe640 1 --2- 172.21.15.98:0/2020805524 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b6fb480 0x56124b703120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.615+0000 7fb37b7fe640 1 -- 172.21.15.98:0/2020805524 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56124b3d16a0 con 0x56124b6fab60 2024-09-19T08:05:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.615+0000 7fb37b7fe640 1 --2- 172.21.15.98:0/2020805524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fab60 0x56124b702be0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb36c011800 tx=0x7fb36c011830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd1151b123ef4ba server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.615+0000 7fb35affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.615+0000 7fb35affd640 1 -- 172.21.15.98:0/2020805524 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c00bce0 con 0x56124b6fab60 2024-09-19T08:05:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.615+0000 7fb35affd640 1 -- 172.21.15.98:0/2020805524 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fb36c004040 con 0x56124b6fab60 2024-09-19T08:05:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.615+0000 7fb383f1c740 1 -- 172.21.15.98:0/2020805524 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb36c011420 con 0x56124b6fab60 2024-09-19T08:05:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.616+0000 7fb35affd640 1 -- 172.21.15.98:0/2020805524 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb36c00be80 con 0x56124b6fab60 2024-09-19T08:05:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.616+0000 7fb383f1c740 10 client.16164 fetch_fsmap learned FSMap version 30 2024-09-19T08:05:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.616+0000 7fb383f1c740 1 -- 172.21.15.98:0/2020805524 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56124b714540 con 0x56124b6fab60 2024-09-19T08:05:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.616+0000 7fb35affd640 1 -- 172.21.15.98:0/2020805524 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb36c011420 con 0x56124b6fab60 2024-09-19T08:05:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.616+0000 7fb35affd640 1 -- 172.21.15.98:0/2020805524 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fb36c01a420 con 0x56124b6fab60 2024-09-19T08:05:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.617+0000 7fb383f1c740 10 client.16164 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:05:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.617+0000 7fb383f1c740 10 client.16164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:05:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.617+0000 7fb383f1c740 20 client.16164 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.617+0000 7fb383f1c740 1 --2- 172.21.15.98:0/2020805524 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56124b715c60 0x56124b718050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.617+0000 7fb383f1c740 4 client.16164 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:05:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.617+0000 7fb383f1c740 1 -- 172.21.15.98:0/2020805524 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56124b66d7f0 con 0x56124b715c60 2024-09-19T08:05:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.617+0000 7fb37affd640 1 --2- 172.21.15.98:0/2020805524 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56124b715c60 0x56124b718050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.620+0000 7fb35affd640 10 client.16164 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.620+0000 7fb37affd640 1 --2- 172.21.15.98:0/2020805524 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56124b715c60 0x56124b718050 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89f051c2595082c2 server_cookie=9002f8ce262fb4cf in_seq=0 out_seq=0 2024-09-19T08:05:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.621+0000 7fb35affd640 1 -- 172.21.15.98:0/2020805524 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56124b66d7f0 con 0x56124b715c60 2024-09-19T08:05:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.621+0000 7fb35affd640 10 client.16164 handle_command_reply: tid=0 2024-09-19T08:05:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.621+0000 7fb383f1c740 1 client.16164 shutdown 2024-09-19T08:05:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.622+0000 7fb383f1c740 1 -- 172.21.15.98:0/2020805524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fab60 msgr2=0x56124b702be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.622+0000 7fb383f1c740 1 --2- 172.21.15.98:0/2020805524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fab60 0x56124b702be0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb36c011800 tx=0x7fb36c011830 comp rx=0 tx=0).stop 2024-09-19T08:05:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.622+0000 7fb383f1c740 1 -- 172.21.15.98:0/2020805524 shutdown_connections 2024-09-19T08:05:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.622+0000 7fb383f1c740 1 --2- 172.21.15.98:0/2020805524 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56124b715c60 0x56124b718050 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.622+0000 7fb383f1c740 1 --2- 172.21.15.98:0/2020805524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b705d40 0x56124b703660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.622+0000 7fb383f1c740 1 --2- 172.21.15.98:0/2020805524 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b6fb480 0x56124b703120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.622+0000 7fb383f1c740 1 --2- 172.21.15.98:0/2020805524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fab60 0x56124b702be0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.622+0000 7fb383f1c740 1 -- 172.21.15.98:0/2020805524 >> 172.21.15.98:0/2020805524 conn(0x56124b6f70f0 msgr2=0x56124b6fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:27.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.622+0000 7fb35affd640 0 client.16164 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.622+0000 7fb383f1c740 1 -- 172.21.15.98:0/2020805524 shutdown_connections 2024-09-19T08:05:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.622+0000 7fb383f1c740 1 -- 172.21.15.98:0/2020805524 wait complete. 2024-09-19T08:05:27.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.622+0000 7fb383f1c740 20 client.16164 trim_cache size 0 max 16384 2024-09-19T08:05:27.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.645+0000 7fb383f1c740 1 Processor -- start 2024-09-19T08:05:27.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.645+0000 7fb383f1c740 1 -- start start 2024-09-19T08:05:27.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.645+0000 7fb383f1c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fb480 0x56124b7024c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.645+0000 7fb383f1c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b705d40 0x56124b6fccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.645+0000 7fb383f1c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b71ab00 0x56124b71af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.645+0000 7fb383f1c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c1afac0 con 0x56124b6fb480 2024-09-19T08:05:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.645+0000 7fb383f1c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c1b1660 con 0x56124b71ab00 2024-09-19T08:05:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.646+0000 7fb383f1c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c1b0190 con 0x56124b705d40 2024-09-19T08:05:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.646+0000 7fb37b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fb480 0x56124b7024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.646+0000 7fb37b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fb480 0x56124b7024c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60712/0 (socket says 172.21.15.98:60712) 2024-09-19T08:05:27.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.646+0000 7fb37b7fe640 1 -- 172.21.15.98:0/1120752432 learned_addr learned my addr 172.21.15.98:0/1120752432 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:27.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.646+0000 7fb37b7fe640 1 -- 172.21.15.98:0/1120752432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b705d40 msgr2=0x56124b6fccc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T08:05:27.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.646+0000 7fb37b7fe640 1 --2- 172.21.15.98:0/1120752432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b705d40 0x56124b6fccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.646+0000 7fb37b7fe640 1 -- 172.21.15.98:0/1120752432 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b71ab00 msgr2=0x56124b71af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:27.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.646+0000 7fb37b7fe640 1 --2- 172.21.15.98:0/1120752432 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b71ab00 0x56124b71af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.646+0000 7fb37b7fe640 1 -- 172.21.15.98:0/1120752432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56124b6f6f20 con 0x56124b6fb480 2024-09-19T08:05:27.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.646+0000 7fb37b7fe640 1 --2- 172.21.15.98:0/1120752432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fb480 0x56124b7024c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb36c011800 tx=0x7fb36c018590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1a23bc09d84a51 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:27.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.646+0000 7fb3617fa640 1 -- 172.21.15.98:0/1120752432 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c02f020 con 0x56124b6fb480 2024-09-19T08:05:27.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.647+0000 7fb3617fa640 1 -- 172.21.15.98:0/1120752432 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb36c0297b0 con 0x56124b6fb480 2024-09-19T08:05:27.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.647+0000 7fb3617fa640 1 -- 172.21.15.98:0/1120752432 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c015040 con 0x56124b6fb480 2024-09-19T08:05:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.647+0000 7fb383f1c740 1 -- 172.21.15.98:0/1120752432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fb480 msgr2=0x56124b7024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.647+0000 7fb383f1c740 1 --2- 172.21.15.98:0/1120752432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fb480 0x56124b7024c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb36c011800 tx=0x7fb36c018590 comp rx=0 tx=0).stop 2024-09-19T08:05:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.647+0000 7fb383f1c740 1 -- 172.21.15.98:0/1120752432 shutdown_connections 2024-09-19T08:05:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.647+0000 7fb383f1c740 1 --2- 172.21.15.98:0/1120752432 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b71ab00 0x56124b71af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.647+0000 7fb383f1c740 1 --2- 172.21.15.98:0/1120752432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b705d40 0x56124b6fccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.647+0000 7fb383f1c740 1 --2- 172.21.15.98:0/1120752432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b6fb480 0x56124b7024c0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.647+0000 7fb383f1c740 1 -- 172.21.15.98:0/1120752432 >> 172.21.15.98:0/1120752432 conn(0x56124b6f70f0 msgr2=0x56124b6fc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.648+0000 7fb383f1c740 1 -- 172.21.15.98:0/1120752432 shutdown_connections 2024-09-19T08:05:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.648+0000 7fb383f1c740 1 -- 172.21.15.98:0/1120752432 wait complete. 2024-09-19T08:05:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.648+0000 7fb383f1c740 1 -- start start 2024-09-19T08:05:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.648+0000 7fb383f1c740 1 Processor -- start 2024-09-19T08:05:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.648+0000 7fb383f1c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b6fb480 0x56124b717950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb383f1c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b705d40 0x56124b717e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb383f1c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b71ab00 0x56124b7183d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb383f1c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c1b0f60 con 0x56124b705d40 2024-09-19T08:05:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb383f1c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c1b0890 con 0x56124b71ab00 2024-09-19T08:05:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb383f1c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c1adaa0 con 0x56124b6fb480 2024-09-19T08:05:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb383f1c740 10 client.0 _ll_register_callbacks cb 0x7fb36c0095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb37affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b705d40 0x56124b717e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb37affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b705d40 0x56124b717e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60722/0 (socket says 172.21.15.98:60722) 2024-09-19T08:05:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb37affd640 1 -- 172.21.15.98:0/4184576202 learned_addr learned my addr 172.21.15.98:0/4184576202 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb37affd640 1 -- 172.21.15.98:0/4184576202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b6fb480 msgr2=0x56124b717950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T08:05:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb37affd640 1 --2- 172.21.15.98:0/4184576202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b6fb480 0x56124b717950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb37affd640 1 -- 172.21.15.98:0/4184576202 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b71ab00 msgr2=0x56124b7183d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T08:05:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb37affd640 1 --2- 172.21.15.98:0/4184576202 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b71ab00 0x56124b7183d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb37affd640 1 -- 172.21.15.98:0/4184576202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56124b77f5a0 con 0x56124b705d40 2024-09-19T08:05:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb37affd640 1 --2- 172.21.15.98:0/4184576202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b705d40 0x56124b717e90 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fb3700b8f00 tx=0x7fb370009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3a73977e0e5a09 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.649+0000 7fb35affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.650+0000 7fb35affd640 1 -- 172.21.15.98:0/4184576202 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3700d5060 con 0x56124b705d40 2024-09-19T08:05:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.650+0000 7fb35affd640 1 -- 172.21.15.98:0/4184576202 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fb3700b9110 con 0x56124b705d40 2024-09-19T08:05:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.650+0000 7fb383f1c740 1 -- 172.21.15.98:0/4184576202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56124b667d00 con 0x56124b705d40 2024-09-19T08:05:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.650+0000 7fb35affd640 1 -- 172.21.15.98:0/4184576202 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb3700da050 con 0x56124b705d40 2024-09-19T08:05:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.650+0000 7fb383f1c740 10 client.16170 fetch_fsmap learned FSMap version 30 2024-09-19T08:05:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.650+0000 7fb383f1c740 1 -- 172.21.15.98:0/4184576202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56124b7808b0 con 0x56124b705d40 2024-09-19T08:05:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.650+0000 7fb35affd640 1 -- 172.21.15.98:0/4184576202 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56124b667d00 con 0x56124b705d40 2024-09-19T08:05:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.650+0000 7fb35affd640 1 -- 172.21.15.98:0/4184576202 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fb37003b610 con 0x56124b705d40 2024-09-19T08:05:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.651+0000 7fb383f1c740 10 client.16170 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:05:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.651+0000 7fb383f1c740 10 client.16170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:05:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.651+0000 7fb383f1c740 20 client.16170 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.651+0000 7fb383f1c740 1 --2- 172.21.15.98:0/4184576202 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56124b781090 0x56124b7834b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:27.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.651+0000 7fb383f1c740 4 client.16170 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:05:27.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.651+0000 7fb383f1c740 1 -- 172.21.15.98:0/4184576202 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56124b5e4110 con 0x56124b781090 2024-09-19T08:05:27.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.654+0000 7fb37b7fe640 1 --2- 172.21.15.98:0/4184576202 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56124b781090 0x56124b7834b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:27.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.657+0000 7fb37b7fe640 1 --2- 172.21.15.98:0/4184576202 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56124b781090 0x56124b7834b0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5718fdd8f75bc6d server_cookie=ec450ce421922a65 in_seq=0 out_seq=0 2024-09-19T08:05:27.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.657+0000 7fb35affd640 10 client.16170 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:27.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.658+0000 7fb35affd640 1 -- 172.21.15.98:0/4184576202 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56124b5e4110 con 0x56124b781090 2024-09-19T08:05:27.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.658+0000 7fb35affd640 10 client.16170 handle_command_reply: tid=0 2024-09-19T08:05:27.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.658+0000 7fb383f1c740 1 client.16170 shutdown 2024-09-19T08:05:27.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.659+0000 7fb383f1c740 1 -- 172.21.15.98:0/4184576202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b705d40 msgr2=0x56124b717e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:27.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.659+0000 7fb383f1c740 1 --2- 172.21.15.98:0/4184576202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b705d40 0x56124b717e90 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fb3700b8f00 tx=0x7fb370009260 comp rx=0 tx=0).stop 2024-09-19T08:05:27.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.659+0000 7fb383f1c740 1 -- 172.21.15.98:0/4184576202 shutdown_connections 2024-09-19T08:05:27.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.659+0000 7fb383f1c740 1 --2- 172.21.15.98:0/4184576202 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56124b781090 0x56124b7834b0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.659+0000 7fb383f1c740 1 --2- 172.21.15.98:0/4184576202 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56124b71ab00 0x56124b7183d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.659+0000 7fb383f1c740 1 --2- 172.21.15.98:0/4184576202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56124b705d40 0x56124b717e90 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.659+0000 7fb383f1c740 1 --2- 172.21.15.98:0/4184576202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56124b6fb480 0x56124b717950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.659+0000 7fb383f1c740 1 -- 172.21.15.98:0/4184576202 >> 172.21.15.98:0/4184576202 conn(0x56124b6f70f0 msgr2=0x56124b6fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.659+0000 7fb35affd640 0 client.16170 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.659+0000 7fb383f1c740 1 -- 172.21.15.98:0/4184576202 shutdown_connections 2024-09-19T08:05:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.659+0000 7fb383f1c740 1 -- 172.21.15.98:0/4184576202 wait complete. 2024-09-19T08:05:27.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.659+0000 7fb383f1c740 20 client.16170 trim_cache size 0 max 16384 2024-09-19T08:05:27.661 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:05:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:05:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "0a5fe8b1-8ea5-4af2-99ef-c85e1cdb11e6", 2024-09-19T08:05:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:05:27.661 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:05:27.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.660+0000 7fb381b43640 1 -- 172.21.15.98:0/77910854 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb3640a33b0 msgr2=0x7fb3640a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:27.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.660+0000 7fb381b43640 1 --2- 172.21.15.98:0/77910854 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb3640a33b0 0x7fb3640a5870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb374009d60 tx=0x7fb374004a60 comp rx=0 tx=0).stop 2024-09-19T08:05:27.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.660+0000 7fb381b43640 1 -- 172.21.15.98:0/77910854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb37c1027a0 msgr2=0x7fb37c074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:27.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.660+0000 7fb381b43640 1 --2- 172.21.15.98:0/77910854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb37c1027a0 0x7fb37c074a30 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb37002aa30 tx=0x7fb370003d50 comp rx=0 tx=0).stop 2024-09-19T08:05:27.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.660+0000 7fb381b43640 1 -- 172.21.15.98:0/77910854 shutdown_connections 2024-09-19T08:05:27.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.660+0000 7fb381b43640 1 --2- 172.21.15.98:0/77910854 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb3640a33b0 0x7fb3640a5870 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.660+0000 7fb381b43640 1 --2- 172.21.15.98:0/77910854 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb37c106f20 0x7fb37c074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.660+0000 7fb381b43640 1 --2- 172.21.15.98:0/77910854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb37c1027a0 0x7fb37c074a30 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.660+0000 7fb381b43640 1 --2- 172.21.15.98:0/77910854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb37c101df0 0x7fb37c0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:27.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.660+0000 7fb381b43640 1 -- 172.21.15.98:0/77910854 >> 172.21.15.98:0/77910854 conn(0x7fb37c0fd8f0 msgr2=0x7fb37c0fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:27.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.661+0000 7fb381b43640 1 -- 172.21.15.98:0/77910854 shutdown_connections 2024-09-19T08:05:27.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:27.661+0000 7fb381b43640 1 -- 172.21.15.98:0/77910854 wait complete. 2024-09-19T08:05:28.296 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:28.296 INFO:tasks.workunit.client.0.smithi098.stdout: Children see throughput for 1 rewriters = 727478.56 kB/sec 2024-09-19T08:05:28.297 INFO:tasks.workunit.client.0.smithi098.stdout: Parent sees throughput for 1 rewriters = 727368.01 kB/sec 2024-09-19T08:05:28.297 INFO:tasks.workunit.client.0.smithi098.stdout: Min throughput per process = 727478.56 kB/sec 2024-09-19T08:05:28.297 INFO:tasks.workunit.client.0.smithi098.stdout: Max throughput per process = 727478.56 kB/sec 2024-09-19T08:05:28.297 INFO:tasks.workunit.client.0.smithi098.stdout: Avg throughput per process = 727478.56 kB/sec 2024-09-19T08:05:28.297 INFO:tasks.workunit.client.0.smithi098.stdout: Min xfer = 1048576.00 kB 2024-09-19T08:05:28.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:05:28.840 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:05:28.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:28 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:05:28.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:28 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:05:28.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:28 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:05:28.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:28 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:05:28.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:28 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:05:28.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:28 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:05:28.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:28 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:05:28.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:28 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:05:28.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:28 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:05:28.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:28 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:05:28.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:28 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:05:28.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:28 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:05:29.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.318+0000 7f6eb22ef640 1 -- 172.21.15.98:0/2325243995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eac106f50 msgr2=0x7f6eac10eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:29.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.318+0000 7f6eb22ef640 1 --2- 172.21.15.98:0/2325243995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eac106f50 0x7f6eac10eaf0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009cd0 tx=0x7f6ea402a8e0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.318+0000 7f6eb22ef640 1 -- 172.21.15.98:0/2325243995 shutdown_connections 2024-09-19T08:05:29.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.318+0000 7f6eb22ef640 1 --2- 172.21.15.98:0/2325243995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eac106f50 0x7f6eac10eaf0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.318+0000 7f6eb22ef640 1 --2- 172.21.15.98:0/2325243995 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6eac100ca0 0x7f6eac106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.318+0000 7f6eb22ef640 1 --2- 172.21.15.98:0/2325243995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6eac1002f0 0x7f6eac1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.318+0000 7f6eb22ef640 1 -- 172.21.15.98:0/2325243995 >> 172.21.15.98:0/2325243995 conn(0x7f6eac077210 msgr2=0x7f6eac0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:29.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.318+0000 7f6eb22ef640 1 -- 172.21.15.98:0/2325243995 shutdown_connections 2024-09-19T08:05:29.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.318+0000 7f6eb22ef640 1 -- 172.21.15.98:0/2325243995 wait complete. 2024-09-19T08:05:29.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.319+0000 7f6eb22ef640 1 Processor -- start 2024-09-19T08:05:29.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.319+0000 7f6eb22ef640 1 -- start start 2024-09-19T08:05:29.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.319+0000 7f6eb22ef640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eac1002f0 0x7f6eac198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.319+0000 7f6eb22ef640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6eac100ca0 0x7f6eac1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.319+0000 7f6eb22ef640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6eac106f50 0x7f6eac19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.319+0000 7f6eb22ef640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac111fa0 con 0x7f6eac1002f0 2024-09-19T08:05:29.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.319+0000 7f6eb22ef640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac111e20 con 0x7f6eac106f50 2024-09-19T08:05:29.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.319+0000 7f6eb22ef640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac112120 con 0x7f6eac100ca0 2024-09-19T08:05:29.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6eabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eac1002f0 0x7f6eac198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6eabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eac1002f0 0x7f6eac198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60744/0 (socket says 172.21.15.98:60744) 2024-09-19T08:05:29.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6eabfff640 1 -- 172.21.15.98:0/1004996605 learned_addr learned my addr 172.21.15.98:0/1004996605 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:29.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6eab7fe640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6eac100ca0 0x7f6eac1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6eb088c640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6eac106f50 0x7f6eac19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6eabfff640 1 -- 172.21.15.98:0/1004996605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6eac100ca0 msgr2=0x7f6eac1989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:29.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6eabfff640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6eac100ca0 0x7f6eac1989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6eabfff640 1 -- 172.21.15.98:0/1004996605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6eac106f50 msgr2=0x7f6eac19cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:29.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6eabfff640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6eac106f50 0x7f6eac19cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6eabfff640 1 -- 172.21.15.98:0/1004996605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eac19d410 con 0x7f6eac1002f0 2024-09-19T08:05:29.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6eb088c640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6eac106f50 0x7f6eac19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:05:29.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6eabfff640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eac1002f0 0x7f6eac198460 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00d950 tx=0x7f6e9c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:29.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6ea97fa640 1 -- 172.21.15.98:0/1004996605 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c017070 con 0x7f6eac1002f0 2024-09-19T08:05:29.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6eb22ef640 1 -- 172.21.15.98:0/1004996605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eac19d700 con 0x7f6eac1002f0 2024-09-19T08:05:29.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6ea97fa640 1 -- 172.21.15.98:0/1004996605 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c009c30 con 0x7f6eac1002f0 2024-09-19T08:05:29.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6ea97fa640 1 -- 172.21.15.98:0/1004996605 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c00f6b0 con 0x7f6eac1002f0 2024-09-19T08:05:29.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.320+0000 7f6eb22ef640 1 -- 172.21.15.98:0/1004996605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eac1a4fe0 con 0x7f6eac1002f0 2024-09-19T08:05:29.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.322+0000 7f6ea97fa640 1 -- 172.21.15.98:0/1004996605 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f6e9c00f850 con 0x7f6eac1002f0 2024-09-19T08:05:29.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.323+0000 7f6eb46c8740 1 Processor -- start 2024-09-19T08:05:29.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.323+0000 7f6eb46c8740 1 -- start start 2024-09-19T08:05:29.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.323+0000 7f6eb46c8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf4374a0 0x55fddf437880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.323+0000 7f6eb46c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf437dc0 0x55fddf442140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.323+0000 7f6eb46c8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf442680 0x55fddf444a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.323+0000 7f6eb46c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fddf39d2e0 con 0x55fddf437dc0 2024-09-19T08:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.323+0000 7f6eb46c8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fddf344ce0 con 0x55fddf442680 2024-09-19T08:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.323+0000 7f6eb46c8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fddf3992a0 con 0x55fddf4374a0 2024-09-19T08:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.323+0000 7f6eabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf437dc0 0x55fddf442140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.323+0000 7f6eabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf437dc0 0x55fddf442140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60758/0 (socket says 172.21.15.98:60758) 2024-09-19T08:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.323+0000 7f6eabfff640 1 -- 172.21.15.98:0/3052895016 learned_addr learned my addr 172.21.15.98:0/3052895016 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.323+0000 7f6ea97fa640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6e940a33b0 0x7f6e940a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.324+0000 7f6eab7fe640 1 --2- 172.21.15.98:0/3052895016 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf442680 0x55fddf444a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.324+0000 7f6eb088c640 1 --2- 172.21.15.98:0/3052895016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf4374a0 0x55fddf437880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.324+0000 7f6eb088c640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6e940a33b0 0x7f6e940a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.324+0000 7f6ea97fa640 1 -- 172.21.15.98:0/1004996605 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f6e9c0cf980 con 0x7f6eac1002f0 2024-09-19T08:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.324+0000 7f6eabfff640 1 -- 172.21.15.98:0/3052895016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf4374a0 msgr2=0x55fddf437880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.324+0000 7f6eabfff640 1 --2- 172.21.15.98:0/3052895016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf4374a0 0x55fddf437880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.324+0000 7f6eabfff640 1 -- 172.21.15.98:0/3052895016 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf442680 msgr2=0x55fddf444a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.324+0000 7f6eabfff640 1 --2- 172.21.15.98:0/3052895016 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf442680 0x55fddf444a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.324+0000 7f6eabfff640 1 -- 172.21.15.98:0/3052895016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fddf329440 con 0x55fddf437dc0 2024-09-19T08:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.324+0000 7f6eabfff640 1 --2- 172.21.15.98:0/3052895016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf437dc0 0x55fddf442140 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c09baf0 tx=0x7f6e9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3570e66e388de18f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.324+0000 7f6eb088c640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6e940a33b0 0x7f6e940a5870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4033040 tx=0x7f6ea40095c0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.324+0000 7f6e927fc640 1 -- 172.21.15.98:0/3052895016 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c020070 con 0x55fddf437dc0 2024-09-19T08:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.324+0000 7f6e927fc640 1 -- 172.21.15.98:0/3052895016 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c0a48a0 con 0x55fddf437dc0 2024-09-19T08:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.324+0000 7f6e927fc640 1 -- 172.21.15.98:0/3052895016 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c09b240 con 0x55fddf437dc0 2024-09-19T08:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.325+0000 7f6eb46c8740 1 -- 172.21.15.98:0/3052895016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf437dc0 msgr2=0x55fddf442140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.325+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/3052895016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf437dc0 0x55fddf442140 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c09baf0 tx=0x7f6e9c005880 comp rx=0 tx=0).stop 2024-09-19T08:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.325+0000 7f6eb46c8740 1 -- 172.21.15.98:0/3052895016 shutdown_connections 2024-09-19T08:05:29.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.325+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/3052895016 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf442680 0x55fddf444a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.325+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/3052895016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf437dc0 0x55fddf442140 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.325+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/3052895016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf4374a0 0x55fddf437880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.325+0000 7f6eb46c8740 1 -- 172.21.15.98:0/3052895016 >> 172.21.15.98:0/3052895016 conn(0x55fddf433a30 msgr2=0x55fddf433e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:29.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.325+0000 7f6eb46c8740 1 -- 172.21.15.98:0/3052895016 shutdown_connections 2024-09-19T08:05:29.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.325+0000 7f6eb46c8740 1 -- 172.21.15.98:0/3052895016 wait complete. 2024-09-19T08:05:29.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.325+0000 7f6eb46c8740 1 -- start start 2024-09-19T08:05:29.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb46c8740 1 Processor -- start 2024-09-19T08:05:29.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb46c8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf4374a0 0x55fddf441f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb46c8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf437dc0 0x55fddf43f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb46c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf442680 0x55fddf43f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb46c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fddf445250 con 0x55fddf442680 2024-09-19T08:05:29.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb46c8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fddf358050 con 0x55fddf437dc0 2024-09-19T08:05:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb46c8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fddf4453d0 con 0x55fddf4374a0 2024-09-19T08:05:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb088c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf442680 0x55fddf43f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb46c8740 10 client.0 _ll_register_callbacks cb 0x55fddf398160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb088c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf442680 0x55fddf43f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60768/0 (socket says 172.21.15.98:60768) 2024-09-19T08:05:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb088c640 1 -- 172.21.15.98:0/1318161132 learned_addr learned my addr 172.21.15.98:0/1318161132 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eab7fe640 1 --2- 172.21.15.98:0/1318161132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf437dc0 0x55fddf43f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eabfff640 1 --2- 172.21.15.98:0/1318161132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf4374a0 0x55fddf441f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb088c640 1 -- 172.21.15.98:0/1318161132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf4374a0 msgr2=0x55fddf441f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb088c640 1 --2- 172.21.15.98:0/1318161132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf4374a0 0x55fddf441f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb088c640 1 -- 172.21.15.98:0/1318161132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf437dc0 msgr2=0x55fddf43f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb088c640 1 --2- 172.21.15.98:0/1318161132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf437dc0 0x55fddf43f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.326+0000 7f6eb088c640 1 -- 172.21.15.98:0/1318161132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fddf10e3c0 con 0x55fddf442680 2024-09-19T08:05:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.327+0000 7f6eb088c640 1 --2- 172.21.15.98:0/1318161132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf442680 0x55fddf43f750 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4035000 tx=0x7f6ea402a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44770a7d34c84bf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.327+0000 7f6e8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.327+0000 7f6e8b7fe640 1 -- 172.21.15.98:0/1318161132 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4050030 con 0x55fddf442680 2024-09-19T08:05:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.327+0000 7f6e8b7fe640 1 -- 172.21.15.98:0/1318161132 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f6ea4046070 con 0x55fddf442680 2024-09-19T08:05:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.327+0000 7f6eb46c8740 1 -- 172.21.15.98:0/1318161132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e9c0a48a0 con 0x55fddf442680 2024-09-19T08:05:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.327+0000 7f6e8b7fe640 1 -- 172.21.15.98:0/1318161132 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6ea4003e30 con 0x55fddf442680 2024-09-19T08:05:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.327+0000 7f6eb46c8740 10 client.16182 fetch_fsmap learned FSMap version 30 2024-09-19T08:05:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.327+0000 7f6eb46c8740 1 -- 172.21.15.98:0/1318161132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fddf450c10 con 0x55fddf442680 2024-09-19T08:05:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.327+0000 7f6e8b7fe640 1 -- 172.21.15.98:0/1318161132 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c0a48a0 con 0x55fddf442680 2024-09-19T08:05:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.327+0000 7f6e8b7fe640 1 -- 172.21.15.98:0/1318161132 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f6ea4049050 con 0x55fddf442680 2024-09-19T08:05:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.327+0000 7f6eb46c8740 10 client.16182 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:05:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.327+0000 7f6eb46c8740 10 client.16182 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:05:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.327+0000 7f6eb46c8740 20 client.16182 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.328+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/1318161132 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fddf4520e0 0x55fddf454560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.328+0000 7f6eb46c8740 4 client.16182 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:05:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.328+0000 7f6eb46c8740 1 -- 172.21.15.98:0/1318161132 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fddf3aa850 con 0x55fddf4520e0 2024-09-19T08:05:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.328+0000 7f6eabfff640 1 --2- 172.21.15.98:0/1318161132 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fddf4520e0 0x55fddf454560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.328+0000 7f6e8b7fe640 10 client.16182 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.328+0000 7f6eabfff640 1 --2- 172.21.15.98:0/1318161132 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fddf4520e0 0x55fddf454560 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=199c1aea48e62335 server_cookie=68a62760c62f76fb in_seq=0 out_seq=0 2024-09-19T08:05:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6e8b7fe640 1 -- 172.21.15.98:0/1318161132 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fddf3aa850 con 0x55fddf4520e0 2024-09-19T08:05:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6e8b7fe640 10 client.16182 handle_command_reply: tid=0 2024-09-19T08:05:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6eb46c8740 1 client.16182 shutdown 2024-09-19T08:05:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6eb46c8740 1 -- 172.21.15.98:0/1318161132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf442680 msgr2=0x55fddf43f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/1318161132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf442680 0x55fddf43f750 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4035000 tx=0x7f6ea402a8e0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6eb46c8740 1 -- 172.21.15.98:0/1318161132 shutdown_connections 2024-09-19T08:05:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/1318161132 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fddf4520e0 0x55fddf454560 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/1318161132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf442680 0x55fddf43f750 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/1318161132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf437dc0 0x55fddf43f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/1318161132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf4374a0 0x55fddf441f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6eb46c8740 1 -- 172.21.15.98:0/1318161132 >> 172.21.15.98:0/1318161132 conn(0x55fddf433a30 msgr2=0x55fddf439710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6e8b7fe640 0 client.16182 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6eb46c8740 1 -- 172.21.15.98:0/1318161132 shutdown_connections 2024-09-19T08:05:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6eb46c8740 1 -- 172.21.15.98:0/1318161132 wait complete. 2024-09-19T08:05:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.329+0000 7f6eb46c8740 20 client.16182 trim_cache size 0 max 16384 2024-09-19T08:05:29.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.346+0000 7f6eb46c8740 1 Processor -- start 2024-09-19T08:05:29.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.346+0000 7f6eb46c8740 1 -- start start 2024-09-19T08:05:29.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.346+0000 7f6eb46c8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf4374a0 0x55fddf44e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.346+0000 7f6eb46c8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf437dc0 0x55fddf43a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.346+0000 7f6eb46c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf442680 0x55fddf44aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.346+0000 7f6eb46c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac1ad8d0 con 0x55fddf442680 2024-09-19T08:05:29.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.346+0000 7f6eb46c8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac1af470 con 0x55fddf4374a0 2024-09-19T08:05:29.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.346+0000 7f6eb46c8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac1adfa0 con 0x55fddf437dc0 2024-09-19T08:05:29.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.346+0000 7f6eb088c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf442680 0x55fddf44aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.346+0000 7f6eb088c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf442680 0x55fddf44aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60772/0 (socket says 172.21.15.98:60772) 2024-09-19T08:05:29.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.346+0000 7f6eb088c640 1 -- 172.21.15.98:0/3399496171 learned_addr learned my addr 172.21.15.98:0/3399496171 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:29.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.347+0000 7f6eab7fe640 1 --2- 172.21.15.98:0/3399496171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf437dc0 0x55fddf43a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.347+0000 7f6eabfff640 1 --2- 172.21.15.98:0/3399496171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf4374a0 0x55fddf44e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.347+0000 7f6eb088c640 1 -- 172.21.15.98:0/3399496171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf437dc0 msgr2=0x55fddf43a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:29.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.347+0000 7f6eb088c640 1 --2- 172.21.15.98:0/3399496171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf437dc0 0x55fddf43a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.347+0000 7f6eb088c640 1 -- 172.21.15.98:0/3399496171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf4374a0 msgr2=0x55fddf44e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:29.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.347+0000 7f6eb088c640 1 --2- 172.21.15.98:0/3399496171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf4374a0 0x55fddf44e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.347+0000 7f6eb088c640 1 -- 172.21.15.98:0/3399496171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fddf437bf0 con 0x55fddf442680 2024-09-19T08:05:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.347+0000 7f6eb088c640 1 --2- 172.21.15.98:0/3399496171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf442680 0x55fddf44aa00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4006010 tx=0x7f6ea4031e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76528ac470e06bea server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.347+0000 7f6e91ffb640 1 -- 172.21.15.98:0/3399496171 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4050070 con 0x55fddf442680 2024-09-19T08:05:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.347+0000 7f6e91ffb640 1 -- 172.21.15.98:0/3399496171 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea4063020 con 0x55fddf442680 2024-09-19T08:05:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.347+0000 7f6e91ffb640 1 -- 172.21.15.98:0/3399496171 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea40570f0 con 0x55fddf442680 2024-09-19T08:05:29.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.348+0000 7f6eb46c8740 1 -- 172.21.15.98:0/3399496171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf442680 msgr2=0x55fddf44aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:29.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.348+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/3399496171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf442680 0x55fddf44aa00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4006010 tx=0x7f6ea4031e40 comp rx=0 tx=0).stop 2024-09-19T08:05:29.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.348+0000 7f6eb46c8740 1 -- 172.21.15.98:0/3399496171 shutdown_connections 2024-09-19T08:05:29.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.348+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/3399496171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf442680 0x55fddf44aa00 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.348+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/3399496171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf437dc0 0x55fddf43a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.348+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/3399496171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf4374a0 0x55fddf44e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.348+0000 7f6eb46c8740 1 -- 172.21.15.98:0/3399496171 >> 172.21.15.98:0/3399496171 conn(0x55fddf433a30 msgr2=0x55fddf432b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.348+0000 7f6eb46c8740 1 -- 172.21.15.98:0/3399496171 shutdown_connections 2024-09-19T08:05:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.348+0000 7f6eb46c8740 1 -- 172.21.15.98:0/3399496171 wait complete. 2024-09-19T08:05:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.348+0000 7f6eb46c8740 1 -- start start 2024-09-19T08:05:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.348+0000 7f6eb46c8740 1 Processor -- start 2024-09-19T08:05:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eb46c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf4374a0 0x55fddf4b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eb46c8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf437dc0 0x55fddf4b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eb46c8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf442680 0x55fddf4b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eb46c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac1aed70 con 0x55fddf4374a0 2024-09-19T08:05:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eb46c8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac1ae6a0 con 0x55fddf442680 2024-09-19T08:05:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eb46c8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac1ab8b0 con 0x55fddf437dc0 2024-09-19T08:05:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eb46c8740 10 client.0 _ll_register_callbacks cb 0x7f6ea4031720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:05:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf4374a0 0x55fddf4b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf4374a0 0x55fddf4b4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60786/0 (socket says 172.21.15.98:60786) 2024-09-19T08:05:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eabfff640 1 -- 172.21.15.98:0/2695429006 learned_addr learned my addr 172.21.15.98:0/2695429006 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:05:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eab7fe640 1 --2- 172.21.15.98:0/2695429006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf437dc0 0x55fddf4b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eabfff640 1 -- 172.21.15.98:0/2695429006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf437dc0 msgr2=0x55fddf4b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eabfff640 1 --2- 172.21.15.98:0/2695429006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf437dc0 0x55fddf4b5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eabfff640 1 -- 172.21.15.98:0/2695429006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf442680 msgr2=0x55fddf4b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:05:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eb088c640 1 --2- 172.21.15.98:0/2695429006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf442680 0x55fddf4b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eabfff640 1 --2- 172.21.15.98:0/2695429006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf442680 0x55fddf4b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eabfff640 1 -- 172.21.15.98:0/2695429006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fddf4c00e0 con 0x55fddf4374a0 2024-09-19T08:05:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eabfff640 1 --2- 172.21.15.98:0/2695429006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf4374a0 0x55fddf4b4dc0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c098250 tx=0x7f6e9c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7dbb68d13f0bc22 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:05:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6e8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:05:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6e8b7fe640 1 -- 172.21.15.98:0/2695429006 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c020040 con 0x55fddf4374a0 2024-09-19T08:05:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6e8b7fe640 1 -- 172.21.15.98:0/2695429006 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f6e9c0984e0 con 0x55fddf4374a0 2024-09-19T08:05:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eb46c8740 1 -- 172.21.15.98:0/2695429006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ea4063020 con 0x55fddf4374a0 2024-09-19T08:05:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6e8b7fe640 1 -- 172.21.15.98:0/2695429006 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6e9c0a48a0 con 0x55fddf4374a0 2024-09-19T08:05:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.349+0000 7f6eb46c8740 10 client.16188 fetch_fsmap learned FSMap version 30 2024-09-19T08:05:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.350+0000 7f6eb46c8740 1 -- 172.21.15.98:0/2695429006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fddf4c1450 con 0x55fddf4374a0 2024-09-19T08:05:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.350+0000 7f6e8b7fe640 1 -- 172.21.15.98:0/2695429006 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea4063020 con 0x55fddf4374a0 2024-09-19T08:05:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.350+0000 7f6e8b7fe640 1 -- 172.21.15.98:0/2695429006 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f6e9c097b40 con 0x55fddf4374a0 2024-09-19T08:05:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.350+0000 7f6eb46c8740 10 client.16188 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:05:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.350+0000 7f6eb46c8740 10 client.16188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:05:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.350+0000 7f6eb46c8740 20 client.16188 populate_metadata read hostname 'smithi098' 2024-09-19T08:05:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.350+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/2695429006 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fddf4c17b0 0x55fddf4c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:05:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.350+0000 7f6eb46c8740 4 client.16188 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:05:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.350+0000 7f6eb46c8740 1 -- 172.21.15.98:0/2695429006 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fddf3a4d60 con 0x55fddf4c17b0 2024-09-19T08:05:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.350+0000 7f6eab7fe640 1 --2- 172.21.15.98:0/2695429006 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fddf4c17b0 0x55fddf4c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:05:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6e8b7fe640 10 client.16188 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6eab7fe640 1 --2- 172.21.15.98:0/2695429006 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fddf4c17b0 0x55fddf4c3ba0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb6e6f2610d0569d server_cookie=6e25af4f9f9c14b5 in_seq=0 out_seq=0 2024-09-19T08:05:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6e8b7fe640 1 -- 172.21.15.98:0/2695429006 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fddf3a4d60 con 0x55fddf4c17b0 2024-09-19T08:05:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6e8b7fe640 10 client.16188 handle_command_reply: tid=0 2024-09-19T08:05:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6eb46c8740 1 client.16188 shutdown 2024-09-19T08:05:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6eb46c8740 1 -- 172.21.15.98:0/2695429006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf4374a0 msgr2=0x55fddf4b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/2695429006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf4374a0 0x55fddf4b4dc0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c098250 tx=0x7f6e9c000bc0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6eb46c8740 1 -- 172.21.15.98:0/2695429006 shutdown_connections 2024-09-19T08:05:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/2695429006 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55fddf4c17b0 0x55fddf4c3ba0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/2695429006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fddf442680 0x55fddf4b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/2695429006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fddf437dc0 0x55fddf4b5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6eb46c8740 1 --2- 172.21.15.98:0/2695429006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fddf4374a0 0x55fddf4b4dc0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6eb46c8740 1 -- 172.21.15.98:0/2695429006 >> 172.21.15.98:0/2695429006 conn(0x55fddf433a30 msgr2=0x55fddf432b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6e8b7fe640 0 client.16188 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:05:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6eb46c8740 1 -- 172.21.15.98:0/2695429006 shutdown_connections 2024-09-19T08:05:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.351+0000 7f6eb46c8740 1 -- 172.21.15.98:0/2695429006 wait complete. 2024-09-19T08:05:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.352+0000 7f6eb46c8740 20 client.16188 trim_cache size 0 max 16384 2024-09-19T08:05:29.358 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:05:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-19T08:05:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-09-19T08:05:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "0a5fe8b1-8ea5-4af2-99ef-c85e1cdb11e6": { 2024-09-19T08:05:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-09-19T08:05:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "0a5fe8b1-8ea5-4af2-99ef-c85e1cdb11e6", 2024-09-19T08:05:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-09-19T08:05:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:05:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:05:29.359 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:05:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.352+0000 7f6eb22ef640 1 -- 172.21.15.98:0/1004996605 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6e940a33b0 msgr2=0x7f6e940a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.352+0000 7f6eb22ef640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6e940a33b0 0x7f6e940a5870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4033040 tx=0x7f6ea40095c0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.352+0000 7f6eb22ef640 1 -- 172.21.15.98:0/1004996605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eac1002f0 msgr2=0x7f6eac198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:05:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.352+0000 7f6eb22ef640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eac1002f0 0x7f6eac198460 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00d950 tx=0x7f6e9c00de10 comp rx=0 tx=0).stop 2024-09-19T08:05:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.352+0000 7f6eb22ef640 1 -- 172.21.15.98:0/1004996605 shutdown_connections 2024-09-19T08:05:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.352+0000 7f6eb22ef640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6e940a33b0 0x7f6e940a5870 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.352+0000 7f6eb22ef640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6eac106f50 0x7f6eac19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.352+0000 7f6eb22ef640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6eac100ca0 0x7f6eac1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.352+0000 7f6eb22ef640 1 --2- 172.21.15.98:0/1004996605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eac1002f0 0x7f6eac198460 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:05:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.352+0000 7f6eb22ef640 1 -- 172.21.15.98:0/1004996605 >> 172.21.15.98:0/1004996605 conn(0x7f6eac077210 msgr2=0x7f6eac105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:05:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.352+0000 7f6eb22ef640 1 -- 172.21.15.98:0/1004996605 shutdown_connections 2024-09-19T08:05:29.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:05:29.352+0000 7f6eb22ef640 1 -- 172.21.15.98:0/1004996605 wait complete. 2024-09-19T08:05:29.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:29 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:05:29.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:29 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:05:29.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:29 smithi098 ceph-mon[31333]: pgmap v77: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 88 MiB/s wr, 56 op/s 2024-09-19T08:05:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:29 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:05:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:29 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:05:29.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:29 smithi162 ceph-mon[35260]: pgmap v77: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 88 MiB/s wr, 56 op/s 2024-09-19T08:05:29.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:29 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:05:29.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:29 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:05:29.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:29 smithi139 ceph-mon[34874]: pgmap v77: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 88 MiB/s wr, 56 op/s 2024-09-19T08:05:30.149 INFO:tasks.cephfs.filesystem:scrub status for tag:0a5fe8b1-8ea5-4af2-99ef-c85e1cdb11e6 - {'path': '/', 'tag': '0a5fe8b1-8ea5-4af2-99ef-c85e1cdb11e6', 'options': 'recursive,force'} 2024-09-19T08:05:30.731 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:30.731 INFO:tasks.workunit.client.0.smithi098.stdout: Children see throughput for 1 readers = 7233453.00 kB/sec 2024-09-19T08:05:30.731 INFO:tasks.workunit.client.0.smithi098.stdout: Parent sees throughput for 1 readers = 7222346.52 kB/sec 2024-09-19T08:05:30.731 INFO:tasks.workunit.client.0.smithi098.stdout: Min throughput per process = 7233453.00 kB/sec 2024-09-19T08:05:30.732 INFO:tasks.workunit.client.0.smithi098.stdout: Max throughput per process = 7233453.00 kB/sec 2024-09-19T08:05:30.732 INFO:tasks.workunit.client.0.smithi098.stdout: Avg throughput per process = 7233453.00 kB/sec 2024-09-19T08:05:30.732 INFO:tasks.workunit.client.0.smithi098.stdout: Min xfer = 1048576.00 kB 2024-09-19T08:05:31.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:30 smithi162 ceph-mon[35260]: pgmap v78: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 88 MiB/s wr, 55 op/s 2024-09-19T08:05:31.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:30 smithi139 ceph-mon[34874]: pgmap v78: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 88 MiB/s wr, 55 op/s 2024-09-19T08:05:31.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:30 smithi098 ceph-mon[31333]: pgmap v78: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 88 MiB/s wr, 55 op/s 2024-09-19T08:05:32.979 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:32.980 INFO:tasks.workunit.client.0.smithi098.stdout: Children see throughput for 1 re-readers = 6913354.00 kB/sec 2024-09-19T08:05:32.980 INFO:tasks.workunit.client.0.smithi098.stdout: Parent sees throughput for 1 re-readers = 6899342.31 kB/sec 2024-09-19T08:05:32.980 INFO:tasks.workunit.client.0.smithi098.stdout: Min throughput per process = 6913354.00 kB/sec 2024-09-19T08:05:32.980 INFO:tasks.workunit.client.0.smithi098.stdout: Max throughput per process = 6913354.00 kB/sec 2024-09-19T08:05:32.980 INFO:tasks.workunit.client.0.smithi098.stdout: Avg throughput per process = 6913354.00 kB/sec 2024-09-19T08:05:32.980 INFO:tasks.workunit.client.0.smithi098.stdout: Min xfer = 1048576.00 kB 2024-09-19T08:05:33.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:32 smithi162 ceph-mon[35260]: pgmap v79: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 171 MiB/s wr, 81 op/s 2024-09-19T08:05:33.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:32 smithi139 ceph-mon[34874]: pgmap v79: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 171 MiB/s wr, 81 op/s 2024-09-19T08:05:33.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:32 smithi098 ceph-mon[31333]: pgmap v79: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 171 MiB/s wr, 81 op/s 2024-09-19T08:05:35.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:34 smithi162 ceph-mon[35260]: pgmap v80: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 171 MiB/s wr, 76 op/s 2024-09-19T08:05:35.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:34 smithi139 ceph-mon[34874]: pgmap v80: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 171 MiB/s wr, 76 op/s 2024-09-19T08:05:35.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:34 smithi098 ceph-mon[31333]: pgmap v80: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 171 MiB/s wr, 76 op/s 2024-09-19T08:05:36.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:35 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:05:36.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:35 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:05:36.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:35 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:05:36.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:35 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:05:36.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:35 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:05:36.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:35 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:05:37.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:36 smithi139 ceph-mon[34874]: pgmap v81: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 171 MiB/s wr, 80 op/s 2024-09-19T08:05:37.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:36 smithi098 ceph-mon[31333]: pgmap v81: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 171 MiB/s wr, 80 op/s 2024-09-19T08:05:37.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:36 smithi162 ceph-mon[35260]: pgmap v81: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 171 MiB/s wr, 80 op/s 2024-09-19T08:05:39.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:38 smithi162 ceph-mon[35260]: pgmap v82: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 109 MiB/s wr, 43 op/s 2024-09-19T08:05:39.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:38 smithi139 ceph-mon[34874]: pgmap v82: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 109 MiB/s wr, 43 op/s 2024-09-19T08:05:39.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:38 smithi098 ceph-mon[31333]: pgmap v82: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 109 MiB/s wr, 43 op/s 2024-09-19T08:05:39.313 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:39.314 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:39.315 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:39.315 INFO:tasks.workunit.client.0.smithi098.stdout:iozone test complete. 2024-09-19T08:05:39.315 INFO:tasks.workunit.client.0.smithi098.stderr:+ iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-09-19T08:05:39.381 INFO:tasks.workunit.client.0.smithi098.stdout: Iozone: Performance Test of File I/O 2024-09-19T08:05:39.381 INFO:tasks.workunit.client.0.smithi098.stdout: Version $Revision: 3.494 $ 2024-09-19T08:05:39.381 INFO:tasks.workunit.client.0.smithi098.stdout: Compiled for 64 bit mode. 2024-09-19T08:05:39.381 INFO:tasks.workunit.client.0.smithi098.stdout: Build: linux-AMD64 2024-09-19T08:05:39.381 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:39.382 INFO:tasks.workunit.client.0.smithi098.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-09-19T08:05:39.382 INFO:tasks.workunit.client.0.smithi098.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-09-19T08:05:39.382 INFO:tasks.workunit.client.0.smithi098.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-09-19T08:05:39.382 INFO:tasks.workunit.client.0.smithi098.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-09-19T08:05:39.382 INFO:tasks.workunit.client.0.smithi098.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-09-19T08:05:39.382 INFO:tasks.workunit.client.0.smithi098.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-09-19T08:05:39.382 INFO:tasks.workunit.client.0.smithi098.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-09-19T08:05:39.382 INFO:tasks.workunit.client.0.smithi098.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-09-19T08:05:39.382 INFO:tasks.workunit.client.0.smithi098.stdout: Alexey Skidanov, Sudhir Kumar. 2024-09-19T08:05:39.382 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:39.382 INFO:tasks.workunit.client.0.smithi098.stdout: Run began: Thu Sep 19 08:05:39 2024 2024-09-19T08:05:39.382 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:39.382 INFO:tasks.workunit.client.0.smithi098.stdout: Include close in write timing 2024-09-19T08:05:39.383 INFO:tasks.workunit.client.0.smithi098.stdout: Include fsync in write timing 2024-09-19T08:05:39.383 INFO:tasks.workunit.client.0.smithi098.stdout: File size set to 10485760 kB 2024-09-19T08:05:39.383 INFO:tasks.workunit.client.0.smithi098.stdout: Record Size 1024 kB 2024-09-19T08:05:39.383 INFO:tasks.workunit.client.0.smithi098.stdout: Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-09-19T08:05:39.383 INFO:tasks.workunit.client.0.smithi098.stdout: Output is in kBytes/sec 2024-09-19T08:05:39.383 INFO:tasks.workunit.client.0.smithi098.stdout: Time Resolution = 0.000001 seconds. 2024-09-19T08:05:39.383 INFO:tasks.workunit.client.0.smithi098.stdout: Processor cache size set to 1024 kBytes. 2024-09-19T08:05:39.383 INFO:tasks.workunit.client.0.smithi098.stdout: Processor cache line size set to 32 bytes. 2024-09-19T08:05:39.383 INFO:tasks.workunit.client.0.smithi098.stdout: File stride size set to 17 * record size. 2024-09-19T08:05:39.383 INFO:tasks.workunit.client.0.smithi098.stdout: Throughput test with 1 process 2024-09-19T08:05:39.383 INFO:tasks.workunit.client.0.smithi098.stdout: Each process writes a 10485760 kByte file in 1024 kByte records 2024-09-19T08:05:42.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:41 smithi139 ceph-mon[34874]: pgmap v83: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 83 MiB/s wr, 30 op/s 2024-09-19T08:05:42.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:41 smithi098 ceph-mon[31333]: pgmap v83: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 83 MiB/s wr, 30 op/s 2024-09-19T08:05:42.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:41 smithi162 ceph-mon[35260]: pgmap v83: 97 pgs: 97 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 83 MiB/s wr, 30 op/s 2024-09-19T08:05:43.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:42 smithi162 ceph-mon[35260]: pgmap v84: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 202 MiB/s wr, 167 op/s 2024-09-19T08:05:43.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:42 smithi139 ceph-mon[34874]: pgmap v84: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 202 MiB/s wr, 167 op/s 2024-09-19T08:05:43.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:42 smithi098 ceph-mon[31333]: pgmap v84: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 202 MiB/s wr, 167 op/s 2024-09-19T08:05:45.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:44 smithi162 ceph-mon[35260]: pgmap v85: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 119 MiB/s wr, 141 op/s 2024-09-19T08:05:45.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:44 smithi139 ceph-mon[34874]: pgmap v85: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 119 MiB/s wr, 141 op/s 2024-09-19T08:05:45.442 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:44 smithi098 ceph-mon[31333]: pgmap v85: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 119 MiB/s wr, 141 op/s 2024-09-19T08:05:47.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:46 smithi162 ceph-mon[35260]: pgmap v86: 97 pgs: 97 active+clean; 5.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 343 MiB/s wr, 312 op/s 2024-09-19T08:05:47.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:46 smithi139 ceph-mon[34874]: pgmap v86: 97 pgs: 97 active+clean; 5.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 343 MiB/s wr, 312 op/s 2024-09-19T08:05:47.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:46 smithi098 ceph-mon[31333]: pgmap v86: 97 pgs: 97 active+clean; 5.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 343 MiB/s wr, 312 op/s 2024-09-19T08:05:49.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:49 smithi098 ceph-mon[31333]: pgmap v87: 97 pgs: 97 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 482 MiB/s wr, 344 op/s 2024-09-19T08:05:49.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:49 smithi162 ceph-mon[35260]: pgmap v87: 97 pgs: 97 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 482 MiB/s wr, 344 op/s 2024-09-19T08:05:49.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:49 smithi139 ceph-mon[34874]: pgmap v87: 97 pgs: 97 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 482 MiB/s wr, 344 op/s 2024-09-19T08:05:51.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:51 smithi098 ceph-mon[31333]: pgmap v88: 97 pgs: 97 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 482 MiB/s wr, 343 op/s 2024-09-19T08:05:51.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:51 smithi162 ceph-mon[35260]: pgmap v88: 97 pgs: 97 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 482 MiB/s wr, 343 op/s 2024-09-19T08:05:51.898 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:05:51.898 INFO:tasks.workunit.client.0.smithi098.stdout: Children see throughput for 1 initial writers = 839822.06 kB/sec 2024-09-19T08:05:51.898 INFO:tasks.workunit.client.0.smithi098.stdout: Parent sees throughput for 1 initial writers = 839807.31 kB/sec 2024-09-19T08:05:51.898 INFO:tasks.workunit.client.0.smithi098.stdout: Min throughput per process = 839822.06 kB/sec 2024-09-19T08:05:51.899 INFO:tasks.workunit.client.0.smithi098.stdout: Max throughput per process = 839822.06 kB/sec 2024-09-19T08:05:51.899 INFO:tasks.workunit.client.0.smithi098.stdout: Avg throughput per process = 839822.06 kB/sec 2024-09-19T08:05:51.899 INFO:tasks.workunit.client.0.smithi098.stdout: Min xfer = 10485760.00 kB 2024-09-19T08:05:51.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:51 smithi139 ceph-mon[34874]: pgmap v88: 97 pgs: 97 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 482 MiB/s wr, 343 op/s 2024-09-19T08:05:53.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:52 smithi162 ceph-mon[35260]: pgmap v89: 97 pgs: 97 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 850 MiB/s wr, 446 op/s 2024-09-19T08:05:53.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:52 smithi139 ceph-mon[34874]: pgmap v89: 97 pgs: 97 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 850 MiB/s wr, 446 op/s 2024-09-19T08:05:53.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:52 smithi098 ceph-mon[31333]: pgmap v89: 97 pgs: 97 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 850 MiB/s wr, 446 op/s 2024-09-19T08:05:55.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:54 smithi162 ceph-mon[35260]: pgmap v90: 97 pgs: 97 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 731 MiB/s wr, 309 op/s 2024-09-19T08:05:55.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:54 smithi139 ceph-mon[34874]: pgmap v90: 97 pgs: 97 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 731 MiB/s wr, 309 op/s 2024-09-19T08:05:55.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:54 smithi098 ceph-mon[31333]: pgmap v90: 97 pgs: 97 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 731 MiB/s wr, 309 op/s 2024-09-19T08:05:57.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:57 smithi162 ceph-mon[35260]: pgmap v91: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 810 MiB/s wr, 331 op/s 2024-09-19T08:05:57.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:56 smithi139 ceph-mon[34874]: pgmap v91: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 810 MiB/s wr, 331 op/s 2024-09-19T08:05:57.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:56 smithi098 ceph-mon[31333]: pgmap v91: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 810 MiB/s wr, 331 op/s 2024-09-19T08:05:59.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:05:58 smithi162 ceph-mon[35260]: pgmap v92: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 671 MiB/s wr, 230 op/s 2024-09-19T08:05:59.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:05:58 smithi139 ceph-mon[34874]: pgmap v92: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 671 MiB/s wr, 230 op/s 2024-09-19T08:05:59.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:05:58 smithi098 ceph-mon[31333]: pgmap v92: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 671 MiB/s wr, 230 op/s 2024-09-19T08:06:00.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:06:00.642 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:01.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:01 smithi162 ceph-mon[35260]: pgmap v93: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 532 MiB/s wr, 195 op/s 2024-09-19T08:06:01.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:01 smithi139 ceph-mon[34874]: pgmap v93: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 532 MiB/s wr, 195 op/s 2024-09-19T08:06:01.442 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:01 smithi098 ceph-mon[31333]: pgmap v93: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 532 MiB/s wr, 195 op/s 2024-09-19T08:06:01.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.571+0000 7f24e846e640 1 -- 172.21.15.98:0/3109812542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e0108b50 msgr2=0x7f24e0115070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:01.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.571+0000 7f24e846e640 1 --2- 172.21.15.98:0/3109812542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e0108b50 0x7f24e0115070 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f24d8009cd0 tx=0x7f24d802a910 comp rx=0 tx=0).stop 2024-09-19T08:06:01.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.571+0000 7f24e846e640 1 -- 172.21.15.98:0/3109812542 shutdown_connections 2024-09-19T08:06:01.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.571+0000 7f24e846e640 1 --2- 172.21.15.98:0/3109812542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e0108b50 0x7f24e0115070 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.571+0000 7f24e846e640 1 --2- 172.21.15.98:0/3109812542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24e01081f0 0x7f24e0108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.571+0000 7f24e846e640 1 --2- 172.21.15.98:0/3109812542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e010e230 0x7f24e010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.571+0000 7f24e846e640 1 -- 172.21.15.98:0/3109812542 >> 172.21.15.98:0/3109812542 conn(0x7f24e0077bb0 msgr2=0x7f24e0079fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.571+0000 7f24e846e640 1 -- 172.21.15.98:0/3109812542 shutdown_connections 2024-09-19T08:06:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.571+0000 7f24e846e640 1 -- 172.21.15.98:0/3109812542 wait complete. 2024-09-19T08:06:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.572+0000 7f24e846e640 1 Processor -- start 2024-09-19T08:06:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.572+0000 7f24e846e640 1 -- start start 2024-09-19T08:06:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e846e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e01081f0 0x7f24e01a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e846e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24e0108b50 0x7f24e01a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e846e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e010e230 0x7f24e019ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e846e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f24e0118650 con 0x7f24e010e230 2024-09-19T08:06:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e846e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f24e01184d0 con 0x7f24e0108b50 2024-09-19T08:06:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e846e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f24e01187d0 con 0x7f24e01081f0 2024-09-19T08:06:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e6a0b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e010e230 0x7f24e019ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e6a0b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e010e230 0x7f24e019ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52670/0 (socket says 172.21.15.98:52670) 2024-09-19T08:06:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e6a0b640 1 -- 172.21.15.98:0/3215318461 learned_addr learned my addr 172.21.15.98:0/3215318461 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e6a0b640 1 -- 172.21.15.98:0/3215318461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e01081f0 msgr2=0x7f24e01a4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e6a0b640 1 --2- 172.21.15.98:0/3215318461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e01081f0 0x7f24e01a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e6a0b640 1 -- 172.21.15.98:0/3215318461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24e0108b50 msgr2=0x7f24e01a4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e5a09640 1 --2- 172.21.15.98:0/3215318461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24e0108b50 0x7f24e01a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e6a0b640 1 --2- 172.21.15.98:0/3215318461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24e0108b50 0x7f24e01a4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e6a0b640 1 -- 172.21.15.98:0/3215318461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e019f170 con 0x7f24e010e230 2024-09-19T08:06:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.573+0000 7f24e5a09640 1 --2- 172.21.15.98:0/3215318461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24e0108b50 0x7f24e01a4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.574+0000 7f24e6a0b640 1 --2- 172.21.15.98:0/3215318461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e010e230 0x7f24e019ea40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f24d8009ca0 tx=0x7f24d8003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.574+0000 7f24df7fe640 1 -- 172.21.15.98:0/3215318461 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d8031760 con 0x7f24e010e230 2024-09-19T08:06:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.574+0000 7f24df7fe640 1 -- 172.21.15.98:0/3215318461 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d8031900 con 0x7f24e010e230 2024-09-19T08:06:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.574+0000 7f24e846e640 1 -- 172.21.15.98:0/3215318461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24e019f400 con 0x7f24e010e230 2024-09-19T08:06:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.574+0000 7f24df7fe640 1 -- 172.21.15.98:0/3215318461 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d802fd10 con 0x7f24e010e230 2024-09-19T08:06:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.574+0000 7f24e846e640 1 -- 172.21.15.98:0/3215318461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24e01ab750 con 0x7f24e010e230 2024-09-19T08:06:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.577+0000 7f24ea847740 1 Processor -- start 2024-09-19T08:06:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.577+0000 7f24df7fe640 1 -- 172.21.15.98:0/3215318461 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f24d8004040 con 0x7f24e010e230 2024-09-19T08:06:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.577+0000 7f24ea847740 1 -- start start 2024-09-19T08:06:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24ea847740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9670f4a0 0x555d9670f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24ea847740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9670fdc0 0x555d9671a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24ea847740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9671a680 0x555d9671ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24ea847740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555d966752e0 con 0x555d9670f4a0 2024-09-19T08:06:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24ea847740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555d9661cce0 con 0x555d9670fdc0 2024-09-19T08:06:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24ea847740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555d966712a0 con 0x555d9671a680 2024-09-19T08:06:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24e5a09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9670f4a0 0x555d9670f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24e5a09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9670f4a0 0x555d9670f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52684/0 (socket says 172.21.15.98:52684) 2024-09-19T08:06:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24e5a09640 1 -- 172.21.15.98:0/327171740 learned_addr learned my addr 172.21.15.98:0/327171740 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24e5a09640 1 -- 172.21.15.98:0/327171740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9671a680 msgr2=0x555d9671ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24e5a09640 1 --2- 172.21.15.98:0/327171740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9671a680 0x555d9671ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24e5a09640 1 -- 172.21.15.98:0/327171740 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9670fdc0 msgr2=0x555d9671a140 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T08:06:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24df7fe640 1 --2- 172.21.15.98:0/3215318461 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f24c80a3430 0x7f24c80a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24e5a09640 1 --2- 172.21.15.98:0/327171740 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9670fdc0 0x555d9671a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24e5a09640 1 -- 172.21.15.98:0/327171740 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d96601440 con 0x555d9670f4a0 2024-09-19T08:06:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.578+0000 7f24e6a0b640 1 --2- 172.21.15.98:0/3215318461 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f24c80a3430 0x7f24c80a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.579+0000 7f24e5a09640 1 --2- 172.21.15.98:0/327171740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9670f4a0 0x555d9670f880 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f24d400aa70 tx=0x7f24d400af30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7376a08b147347dc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.579+0000 7f24dcdea640 1 -- 172.21.15.98:0/327171740 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d400f040 con 0x555d9670f4a0 2024-09-19T08:06:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.579+0000 7f24df7fe640 1 -- 172.21.15.98:0/3215318461 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f24d80f10c0 con 0x7f24e010e230 2024-09-19T08:06:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.579+0000 7f24dcdea640 1 -- 172.21.15.98:0/327171740 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d4018070 con 0x555d9670f4a0 2024-09-19T08:06:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.579+0000 7f24dcdea640 1 -- 172.21.15.98:0/327171740 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d40123c0 con 0x555d9670f4a0 2024-09-19T08:06:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.580+0000 7f24ea847740 1 -- 172.21.15.98:0/327171740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9670f4a0 msgr2=0x555d9670f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.580+0000 7f24e6a0b640 1 --2- 172.21.15.98:0/3215318461 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f24c80a3430 0x7f24c80a58f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f24d80b9940 tx=0x7f24d80c1040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.580+0000 7f24ea847740 1 --2- 172.21.15.98:0/327171740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9670f4a0 0x555d9670f880 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f24d400aa70 tx=0x7f24d400af30 comp rx=0 tx=0).stop 2024-09-19T08:06:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.580+0000 7f24ea847740 1 -- 172.21.15.98:0/327171740 shutdown_connections 2024-09-19T08:06:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.580+0000 7f24ea847740 1 --2- 172.21.15.98:0/327171740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9671a680 0x555d9671ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.580+0000 7f24ea847740 1 --2- 172.21.15.98:0/327171740 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9670fdc0 0x555d9671a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.580+0000 7f24ea847740 1 --2- 172.21.15.98:0/327171740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9670f4a0 0x555d9670f880 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.580+0000 7f24ea847740 1 -- 172.21.15.98:0/327171740 >> 172.21.15.98:0/327171740 conn(0x555d9670ba30 msgr2=0x555d9670be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.580+0000 7f24ea847740 1 -- 172.21.15.98:0/327171740 shutdown_connections 2024-09-19T08:06:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.580+0000 7f24ea847740 1 -- 172.21.15.98:0/327171740 wait complete. 2024-09-19T08:06:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.580+0000 7f24ea847740 1 -- start start 2024-09-19T08:06:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.581+0000 7f24ea847740 1 Processor -- start 2024-09-19T08:06:01.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.581+0000 7f24ea847740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9670f4a0 0x555d9671a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.581+0000 7f24ea847740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9670fdc0 0x555d967172d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.581+0000 7f24ea847740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9671a680 0x555d96717870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.581+0000 7f24ea847740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555d9671d250 con 0x555d9671a680 2024-09-19T08:06:01.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.581+0000 7f24ea847740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555d96630050 con 0x555d9670fdc0 2024-09-19T08:06:01.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.581+0000 7f24ea847740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555d9671d3d0 con 0x555d9670f4a0 2024-09-19T08:06:01.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.581+0000 7f24ea847740 10 client.0 _ll_register_callbacks cb 0x555d96670160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:01.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.581+0000 7f24e5a09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9671a680 0x555d96717870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:01.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.581+0000 7f24e5a09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9671a680 0x555d96717870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52692/0 (socket says 172.21.15.98:52692) 2024-09-19T08:06:01.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.581+0000 7f24e5a09640 1 -- 172.21.15.98:0/1173310483 learned_addr learned my addr 172.21.15.98:0/1173310483 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:01.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.582+0000 7f24e5a09640 1 -- 172.21.15.98:0/1173310483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9670f4a0 msgr2=0x555d9671a020 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T08:06:01.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.582+0000 7f24e5a09640 1 --2- 172.21.15.98:0/1173310483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9670f4a0 0x555d9671a020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.582+0000 7f24e5a09640 1 -- 172.21.15.98:0/1173310483 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9670fdc0 msgr2=0x555d967172d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T08:06:01.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.582+0000 7f24e5a09640 1 --2- 172.21.15.98:0/1173310483 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9670fdc0 0x555d967172d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.582+0000 7f24e5a09640 1 -- 172.21.15.98:0/1173310483 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d963e63c0 con 0x555d9671a680 2024-09-19T08:06:01.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.582+0000 7f24e5a09640 1 --2- 172.21.15.98:0/1173310483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9671a680 0x555d96717870 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f24d4004810 tx=0x7f24d4014fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14e14fc3552ce7f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:01.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.582+0000 7f24c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:01.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.582+0000 7f24c5ffb640 1 -- 172.21.15.98:0/1173310483 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d400ece0 con 0x555d9671a680 2024-09-19T08:06:01.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.582+0000 7f24c5ffb640 1 -- 172.21.15.98:0/1173310483 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f24d4028480 con 0x555d9671a680 2024-09-19T08:06:01.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.582+0000 7f24ea847740 1 -- 172.21.15.98:0/1173310483 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24d4018070 con 0x555d9671a680 2024-09-19T08:06:01.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.582+0000 7f24c5ffb640 1 -- 172.21.15.98:0/1173310483 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f24d400a710 con 0x555d9671a680 2024-09-19T08:06:01.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.582+0000 7f24ea847740 10 client.16200 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:01.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.582+0000 7f24ea847740 1 -- 172.21.15.98:0/1173310483 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555d96728c60 con 0x555d9671a680 2024-09-19T08:06:01.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.583+0000 7f24c5ffb640 1 -- 172.21.15.98:0/1173310483 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d4018070 con 0x555d9671a680 2024-09-19T08:06:01.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.583+0000 7f24c5ffb640 1 -- 172.21.15.98:0/1173310483 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f24d4031740 con 0x555d9671a680 2024-09-19T08:06:01.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.583+0000 7f24ea847740 10 client.16200 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:01.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.583+0000 7f24ea847740 10 client.16200 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:06:01.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.583+0000 7f24ea847740 20 client.16200 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:01.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.583+0000 7f24ea847740 1 --2- 172.21.15.98:0/1173310483 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555d9672a010 0x555d9672c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.583+0000 7f24ea847740 4 client.16200 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:01.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.583+0000 7f24ea847740 1 -- 172.21.15.98:0/1173310483 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d96682850 con 0x555d9672a010 2024-09-19T08:06:01.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.585+0000 7f24e620a640 1 --2- 172.21.15.98:0/1173310483 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555d9672a010 0x555d9672c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.589+0000 7f24c5ffb640 10 client.16200 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.589+0000 7f24e620a640 1 --2- 172.21.15.98:0/1173310483 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555d9672a010 0x555d9672c400 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c87540bff20a559c server_cookie=70d7731b5e301471 in_seq=0 out_seq=0 2024-09-19T08:06:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.591+0000 7f24c5ffb640 1 -- 172.21.15.98:0/1173310483 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555d96682850 con 0x555d9672a010 2024-09-19T08:06:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.591+0000 7f24c5ffb640 10 client.16200 handle_command_reply: tid=0 2024-09-19T08:06:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.591+0000 7f24ea847740 1 client.16200 shutdown 2024-09-19T08:06:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.591+0000 7f24ea847740 1 -- 172.21.15.98:0/1173310483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9671a680 msgr2=0x555d96717870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.591+0000 7f24ea847740 1 --2- 172.21.15.98:0/1173310483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9671a680 0x555d96717870 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f24d4004810 tx=0x7f24d4014fa0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.591+0000 7f24ea847740 1 -- 172.21.15.98:0/1173310483 shutdown_connections 2024-09-19T08:06:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.591+0000 7f24ea847740 1 --2- 172.21.15.98:0/1173310483 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555d9672a010 0x555d9672c400 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.591+0000 7f24ea847740 1 --2- 172.21.15.98:0/1173310483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9671a680 0x555d96717870 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.591+0000 7f24ea847740 1 --2- 172.21.15.98:0/1173310483 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9670fdc0 0x555d967172d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.592+0000 7f24ea847740 1 --2- 172.21.15.98:0/1173310483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9670f4a0 0x555d9671a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.592+0000 7f24ea847740 1 -- 172.21.15.98:0/1173310483 >> 172.21.15.98:0/1173310483 conn(0x555d9670ba30 msgr2=0x555d96711760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.592+0000 7f24c5ffb640 0 client.16200 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.592+0000 7f24ea847740 1 -- 172.21.15.98:0/1173310483 shutdown_connections 2024-09-19T08:06:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.592+0000 7f24ea847740 1 -- 172.21.15.98:0/1173310483 wait complete. 2024-09-19T08:06:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.592+0000 7f24ea847740 20 client.16200 trim_cache size 0 max 16384 2024-09-19T08:06:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.621+0000 7f24ea847740 1 Processor -- start 2024-09-19T08:06:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.622+0000 7f24ea847740 1 -- start start 2024-09-19T08:06:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.622+0000 7f24ea847740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9670f4a0 0x555d9670aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.622+0000 7f24ea847740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9670fdc0 0x555d96731a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.622+0000 7f24ea847740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9671a680 0x555d96731fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.622+0000 7f24ea847740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f24e01b4010 con 0x555d9671a680 2024-09-19T08:06:01.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.622+0000 7f24ea847740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f24e01b5bb0 con 0x555d9670fdc0 2024-09-19T08:06:01.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.622+0000 7f24ea847740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f24e01b46e0 con 0x555d9670f4a0 2024-09-19T08:06:01.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.622+0000 7f24e5a09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9671a680 0x555d96731fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:01.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.622+0000 7f24e5a09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9671a680 0x555d96731fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52696/0 (socket says 172.21.15.98:52696) 2024-09-19T08:06:01.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.622+0000 7f24e5a09640 1 -- 172.21.15.98:0/420706265 learned_addr learned my addr 172.21.15.98:0/420706265 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:01.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.623+0000 7f24e5a09640 1 -- 172.21.15.98:0/420706265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9670f4a0 msgr2=0x555d9670aa80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T08:06:01.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.623+0000 7f24e5a09640 1 --2- 172.21.15.98:0/420706265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9670f4a0 0x555d9670aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.623+0000 7f24e5a09640 1 -- 172.21.15.98:0/420706265 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9670fdc0 msgr2=0x555d96731a80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T08:06:01.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.623+0000 7f24e5a09640 1 --2- 172.21.15.98:0/420706265 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9670fdc0 0x555d96731a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.623+0000 7f24e5a09640 1 -- 172.21.15.98:0/420706265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d9670fbb0 con 0x555d9671a680 2024-09-19T08:06:01.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.623+0000 7f24e5a09640 1 --2- 172.21.15.98:0/420706265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9671a680 0x555d96731fc0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f24d4006d70 tx=0x7f24d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc2b0decec99fb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:01.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.623+0000 7f24c7fff640 1 -- 172.21.15.98:0/420706265 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d400ece0 con 0x555d9671a680 2024-09-19T08:06:01.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.623+0000 7f24c7fff640 1 -- 172.21.15.98:0/420706265 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d4018040 con 0x555d9671a680 2024-09-19T08:06:01.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.623+0000 7f24c7fff640 1 -- 172.21.15.98:0/420706265 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d401daa0 con 0x555d9671a680 2024-09-19T08:06:01.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.624+0000 7f24ea847740 1 -- 172.21.15.98:0/420706265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9671a680 msgr2=0x555d96731fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:01.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.624+0000 7f24ea847740 1 --2- 172.21.15.98:0/420706265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9671a680 0x555d96731fc0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f24d4006d70 tx=0x7f24d40046c0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.625+0000 7f24ea847740 1 -- 172.21.15.98:0/420706265 shutdown_connections 2024-09-19T08:06:01.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.625+0000 7f24ea847740 1 --2- 172.21.15.98:0/420706265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9671a680 0x555d96731fc0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.625+0000 7f24ea847740 1 --2- 172.21.15.98:0/420706265 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9670fdc0 0x555d96731a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.625+0000 7f24ea847740 1 --2- 172.21.15.98:0/420706265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9670f4a0 0x555d9670aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.625+0000 7f24ea847740 1 -- 172.21.15.98:0/420706265 >> 172.21.15.98:0/420706265 conn(0x555d9670ba30 msgr2=0x555d9670a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:01.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.625+0000 7f24ea847740 1 -- 172.21.15.98:0/420706265 shutdown_connections 2024-09-19T08:06:01.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.625+0000 7f24ea847740 1 -- 172.21.15.98:0/420706265 wait complete. 2024-09-19T08:06:01.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.625+0000 7f24ea847740 1 -- start start 2024-09-19T08:06:01.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.625+0000 7f24ea847740 1 Processor -- start 2024-09-19T08:06:01.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.626+0000 7f24ea847740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9670f4a0 0x555d96722a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.626+0000 7f24ea847740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9670fdc0 0x555d9671fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.627+0000 7f24ea847740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9671a680 0x555d96720320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.627+0000 7f24ea847740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f24e01b54b0 con 0x555d9670f4a0 2024-09-19T08:06:01.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.627+0000 7f24ea847740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f24e01b4de0 con 0x555d9671a680 2024-09-19T08:06:01.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.627+0000 7f24ea847740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f24e01b1ff0 con 0x555d9670fdc0 2024-09-19T08:06:01.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.627+0000 7f24ea847740 10 client.0 _ll_register_callbacks cb 0x7f24d400a970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:01.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.627+0000 7f24e5a09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9670f4a0 0x555d96722a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:01.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.627+0000 7f24e5a09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9670f4a0 0x555d96722a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52698/0 (socket says 172.21.15.98:52698) 2024-09-19T08:06:01.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.627+0000 7f24e5a09640 1 -- 172.21.15.98:0/2869724327 learned_addr learned my addr 172.21.15.98:0/2869724327 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:01.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.627+0000 7f24e5a09640 1 -- 172.21.15.98:0/2869724327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9670fdc0 msgr2=0x555d9671fde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T08:06:01.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.628+0000 7f24e5a09640 1 --2- 172.21.15.98:0/2869724327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9670fdc0 0x555d9671fde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.628+0000 7f24e5a09640 1 -- 172.21.15.98:0/2869724327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9671a680 msgr2=0x555d96720320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:01.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.628+0000 7f24e5a09640 1 --2- 172.21.15.98:0/2869724327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9671a680 0x555d96720320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.628+0000 7f24e5a09640 1 -- 172.21.15.98:0/2869724327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d96720860 con 0x555d9670f4a0 2024-09-19T08:06:01.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.628+0000 7f24e5a09640 1 --2- 172.21.15.98:0/2869724327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9670f4a0 0x555d96722a90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f24d400af70 tx=0x7f24d400a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e9cbb316d33f03 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:01.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.628+0000 7f24c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:01.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.629+0000 7f24c5ffb640 1 -- 172.21.15.98:0/2869724327 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d4013950 con 0x555d9670f4a0 2024-09-19T08:06:01.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.629+0000 7f24c5ffb640 1 -- 172.21.15.98:0/2869724327 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f24d4031450 con 0x555d9670f4a0 2024-09-19T08:06:01.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.629+0000 7f24ea847740 1 -- 172.21.15.98:0/2869724327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24d4018040 con 0x555d9670f4a0 2024-09-19T08:06:01.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.629+0000 7f24c5ffb640 1 -- 172.21.15.98:0/2869724327 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f24d4031650 con 0x555d9670f4a0 2024-09-19T08:06:01.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.629+0000 7f24ea847740 10 client.16206 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:01.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.629+0000 7f24ea847740 1 -- 172.21.15.98:0/2869724327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555d96794780 con 0x555d9670f4a0 2024-09-19T08:06:01.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.629+0000 7f24c5ffb640 1 -- 172.21.15.98:0/2869724327 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d4018040 con 0x555d9670f4a0 2024-09-19T08:06:01.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.630+0000 7f24c5ffb640 1 -- 172.21.15.98:0/2869724327 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f24d403a570 con 0x555d9670f4a0 2024-09-19T08:06:01.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.630+0000 7f24ea847740 10 client.16206 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:01.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.630+0000 7f24ea847740 10 client.16206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:06:01.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.630+0000 7f24ea847740 20 client.16206 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:01.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.630+0000 7f24ea847740 1 --2- 172.21.15.98:0/2869724327 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555d96794dd0 0x555d967971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:01.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.630+0000 7f24ea847740 4 client.16206 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:06:01.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.630+0000 7f24ea847740 1 -- 172.21.15.98:0/2869724327 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555d9667cd60 con 0x555d96794dd0 2024-09-19T08:06:01.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.631+0000 7f24e620a640 1 --2- 172.21.15.98:0/2869724327 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555d96794dd0 0x555d967971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:01.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.634+0000 7f24e620a640 1 --2- 172.21.15.98:0/2869724327 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555d96794dd0 0x555d967971f0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7732d9e27a33a57c server_cookie=d8f30a4b1488c9c in_seq=0 out_seq=0 2024-09-19T08:06:01.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.634+0000 7f24c5ffb640 10 client.16206 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:01.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.634+0000 7f24c5ffb640 1 -- 172.21.15.98:0/2869724327 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555d9667cd60 con 0x555d96794dd0 2024-09-19T08:06:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.634+0000 7f24c5ffb640 10 client.16206 handle_command_reply: tid=0 2024-09-19T08:06:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.634+0000 7f24ea847740 1 client.16206 shutdown 2024-09-19T08:06:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.635+0000 7f24ea847740 1 -- 172.21.15.98:0/2869724327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9670f4a0 msgr2=0x555d96722a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.635+0000 7f24ea847740 1 --2- 172.21.15.98:0/2869724327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9670f4a0 0x555d96722a90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f24d400af70 tx=0x7f24d400a250 comp rx=0 tx=0).stop 2024-09-19T08:06:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.635+0000 7f24ea847740 1 -- 172.21.15.98:0/2869724327 shutdown_connections 2024-09-19T08:06:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.635+0000 7f24ea847740 1 --2- 172.21.15.98:0/2869724327 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555d96794dd0 0x555d967971f0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.635+0000 7f24c5ffb640 0 client.16206 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.635+0000 7f24ea847740 1 --2- 172.21.15.98:0/2869724327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555d9671a680 0x555d96720320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.635+0000 7f24ea847740 1 --2- 172.21.15.98:0/2869724327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555d9670fdc0 0x555d9671fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.635+0000 7f24ea847740 1 --2- 172.21.15.98:0/2869724327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d9670f4a0 0x555d96722a90 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.635+0000 7f24ea847740 1 -- 172.21.15.98:0/2869724327 >> 172.21.15.98:0/2869724327 conn(0x555d9670ba30 msgr2=0x555d96719350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.635+0000 7f24ea847740 1 -- 172.21.15.98:0/2869724327 shutdown_connections 2024-09-19T08:06:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.635+0000 7f24ea847740 1 -- 172.21.15.98:0/2869724327 wait complete. 2024-09-19T08:06:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.635+0000 7f24ea847740 20 client.16206 trim_cache size 0 max 16384 2024-09-19T08:06:01.637 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:06:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:06:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:06:01.637 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:06:01.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.637+0000 7f24e846e640 1 -- 172.21.15.98:0/3215318461 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f24c80a3430 msgr2=0x7f24c80a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:01.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.637+0000 7f24e846e640 1 --2- 172.21.15.98:0/3215318461 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f24c80a3430 0x7f24c80a58f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f24d80b9940 tx=0x7f24d80c1040 comp rx=0 tx=0).stop 2024-09-19T08:06:01.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.637+0000 7f24e846e640 1 -- 172.21.15.98:0/3215318461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e010e230 msgr2=0x7f24e019ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:01.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.637+0000 7f24e846e640 1 --2- 172.21.15.98:0/3215318461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e010e230 0x7f24e019ea40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f24d8009ca0 tx=0x7f24d8003f90 comp rx=0 tx=0).stop 2024-09-19T08:06:01.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.637+0000 7f24e846e640 1 -- 172.21.15.98:0/3215318461 shutdown_connections 2024-09-19T08:06:01.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.637+0000 7f24e846e640 1 --2- 172.21.15.98:0/3215318461 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f24c80a3430 0x7f24c80a58f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.637+0000 7f24e846e640 1 --2- 172.21.15.98:0/3215318461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e010e230 0x7f24e019ea40 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.637+0000 7f24e846e640 1 --2- 172.21.15.98:0/3215318461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24e0108b50 0x7f24e01a4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.637+0000 7f24e846e640 1 --2- 172.21.15.98:0/3215318461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e01081f0 0x7f24e01a4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.637+0000 7f24e846e640 1 -- 172.21.15.98:0/3215318461 >> 172.21.15.98:0/3215318461 conn(0x7f24e0077bb0 msgr2=0x7f24e00795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.637+0000 7f24e846e640 1 -- 172.21.15.98:0/3215318461 shutdown_connections 2024-09-19T08:06:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:01.638+0000 7f24e846e640 1 -- 172.21.15.98:0/3215318461 wait complete. 2024-09-19T08:06:02.517 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:06:02.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:06:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.547+0000 7f878bd4d640 1 -- 172.21.15.98:0/158661982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784100a70 msgr2=0x7f8784100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.547+0000 7f878bd4d640 1 --2- 172.21.15.98:0/158661982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784100a70 0x7f8784100ed0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8774009930 tx=0x7f877402a480 comp rx=0 tx=0).stop 2024-09-19T08:06:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.547+0000 7f878bd4d640 1 -- 172.21.15.98:0/158661982 shutdown_connections 2024-09-19T08:06:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.547+0000 7f878bd4d640 1 --2- 172.21.15.98:0/158661982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8784101410 0x7f878410d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.547+0000 7f878bd4d640 1 --2- 172.21.15.98:0/158661982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784100a70 0x7f8784100ed0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.547+0000 7f878bd4d640 1 --2- 172.21.15.98:0/158661982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8784106a70 0x7f8784106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.547+0000 7f878bd4d640 1 -- 172.21.15.98:0/158661982 >> 172.21.15.98:0/158661982 conn(0x7f87840fc7d0 msgr2=0x7f87840febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.548+0000 7f878bd4d640 1 -- 172.21.15.98:0/158661982 shutdown_connections 2024-09-19T08:06:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.548+0000 7f878bd4d640 1 -- 172.21.15.98:0/158661982 wait complete. 2024-09-19T08:06:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.549+0000 7f878bd4d640 1 Processor -- start 2024-09-19T08:06:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.549+0000 7f878bd4d640 1 -- start start 2024-09-19T08:06:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.549+0000 7f878bd4d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8784100a70 0x7f87841994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.549+0000 7f878bd4d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784101410 0x7f8784199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.549+0000 7f878bd4d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8784106a70 0x7f87841a4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.549+0000 7f878bd4d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8784110f50 con 0x7f8784101410 2024-09-19T08:06:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.549+0000 7f878bd4d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8784110dd0 con 0x7f8784100a70 2024-09-19T08:06:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.549+0000 7f878bd4d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f87841110d0 con 0x7f8784106a70 2024-09-19T08:06:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.549+0000 7f87892e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784101410 0x7f8784199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.549+0000 7f87892e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784101410 0x7f8784199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52710/0 (socket says 172.21.15.98:52710) 2024-09-19T08:06:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.549+0000 7f87892e8640 1 -- 172.21.15.98:0/1365533088 learned_addr learned my addr 172.21.15.98:0/1365533088 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.550+0000 7f878a2ea640 1 --2- 172.21.15.98:0/1365533088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8784106a70 0x7f87841a4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.550+0000 7f8789ae9640 1 --2- 172.21.15.98:0/1365533088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8784100a70 0x7f87841994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.550+0000 7f87892e8640 1 -- 172.21.15.98:0/1365533088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8784106a70 msgr2=0x7f87841a4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.550+0000 7f87892e8640 1 --2- 172.21.15.98:0/1365533088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8784106a70 0x7f87841a4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.550+0000 7f87892e8640 1 -- 172.21.15.98:0/1365533088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8784100a70 msgr2=0x7f87841994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.550+0000 7f87892e8640 1 --2- 172.21.15.98:0/1365533088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8784100a70 0x7f87841994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.550+0000 7f87892e8640 1 -- 172.21.15.98:0/1365533088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87841a4aa0 con 0x7f8784101410 2024-09-19T08:06:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.550+0000 7f87892e8640 1 --2- 172.21.15.98:0/1365533088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784101410 0x7f8784199a20 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f8774005ab0 tx=0x7f87740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.550+0000 7f877affd640 1 -- 172.21.15.98:0/1365533088 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877403f030 con 0x7f8784101410 2024-09-19T08:06:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.550+0000 7f878bd4d640 1 -- 172.21.15.98:0/1365533088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87841a4c70 con 0x7f8784101410 2024-09-19T08:06:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.550+0000 7f877affd640 1 -- 172.21.15.98:0/1365533088 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8774003ce0 con 0x7f8784101410 2024-09-19T08:06:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.550+0000 7f877affd640 1 -- 172.21.15.98:0/1365533088 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774035070 con 0x7f8784101410 2024-09-19T08:06:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.551+0000 7f878bd4d640 1 -- 172.21.15.98:0/1365533088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87841a5120 con 0x7f8784101410 2024-09-19T08:06:03.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.552+0000 7f878bd4d640 1 -- 172.21.15.98:0/1365533088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8784100f50 con 0x7f8784101410 2024-09-19T08:06:03.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.554+0000 7f877affd640 1 -- 172.21.15.98:0/1365533088 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f8774042030 con 0x7f8784101410 2024-09-19T08:06:03.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.555+0000 7f877affd640 1 --2- 172.21.15.98:0/1365533088 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f87700a33b0 0x7f87700a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.556+0000 7f8789ae9640 1 --2- 172.21.15.98:0/1365533088 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f87700a33b0 0x7f87700a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.556+0000 7f877affd640 1 -- 172.21.15.98:0/1365533088 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f87740eff30 con 0x7f8784101410 2024-09-19T08:06:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.556+0000 7f8789ae9640 1 --2- 172.21.15.98:0/1365533088 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f87700a33b0 0x7f87700a5870 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f878000a7d0 tx=0x7f8780008040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:03.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.562+0000 7f877affd640 1 -- 172.21.15.98:0/1365533088 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f87740bc940 con 0x7f8784101410 2024-09-19T08:06:03.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:03 smithi098 ceph-mon[31333]: pgmap v94: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 880 MiB/s wr, 366 op/s 2024-09-19T08:06:03.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:03 smithi162 ceph-mon[35260]: pgmap v94: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 880 MiB/s wr, 366 op/s 2024-09-19T08:06:03.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.887+0000 7f878bd4d640 1 -- 172.21.15.98:0/1365533088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8784108fe0 con 0x7f8784101410 2024-09-19T08:06:03.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.888+0000 7f877affd640 1 -- 172.21.15.98:0/1365533088 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11985 (secure 0 0 0) 0x7f87740bc240 con 0x7f8784101410 2024-09-19T08:06:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:06:03.890 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":30,"btime":"2024-09-19T08:05:21:864704+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:05:21.551452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:06:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 30 2024-09-19T08:06:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.892+0000 7f878bd4d640 1 -- 172.21.15.98:0/1365533088 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f87700a33b0 msgr2=0x7f87700a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.892+0000 7f878bd4d640 1 --2- 172.21.15.98:0/1365533088 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f87700a33b0 0x7f87700a5870 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f878000a7d0 tx=0x7f8780008040 comp rx=0 tx=0).stop 2024-09-19T08:06:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.892+0000 7f878bd4d640 1 -- 172.21.15.98:0/1365533088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784101410 msgr2=0x7f8784199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.892+0000 7f878bd4d640 1 --2- 172.21.15.98:0/1365533088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784101410 0x7f8784199a20 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f8774005ab0 tx=0x7f87740042f0 comp rx=0 tx=0).stop 2024-09-19T08:06:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.893+0000 7f878bd4d640 1 -- 172.21.15.98:0/1365533088 shutdown_connections 2024-09-19T08:06:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.893+0000 7f878bd4d640 1 --2- 172.21.15.98:0/1365533088 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f87700a33b0 0x7f87700a5870 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.893+0000 7f878bd4d640 1 --2- 172.21.15.98:0/1365533088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8784106a70 0x7f87841a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.893+0000 7f878bd4d640 1 --2- 172.21.15.98:0/1365533088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784101410 0x7f8784199a20 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.893+0000 7f878bd4d640 1 --2- 172.21.15.98:0/1365533088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8784100a70 0x7f87841994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.893+0000 7f878bd4d640 1 -- 172.21.15.98:0/1365533088 >> 172.21.15.98:0/1365533088 conn(0x7f87840fc7d0 msgr2=0x7f87840fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.893+0000 7f878bd4d640 1 -- 172.21.15.98:0/1365533088 shutdown_connections 2024-09-19T08:06:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:03.893+0000 7f878bd4d640 1 -- 172.21.15.98:0/1365533088 wait complete. 2024-09-19T08:06:04.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:03 smithi139 ceph-mon[34874]: pgmap v94: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 880 MiB/s wr, 366 op/s 2024-09-19T08:06:04.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:06:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:05.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:04 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1365533088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:06:05.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:04 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1365533088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:06:05.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:04 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1365533088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:06:05.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.783+0000 7f8936ca6640 1 -- 172.21.15.98:0/1541322879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8930109830 msgr2=0x7f8930109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:05.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.783+0000 7f8936ca6640 1 --2- 172.21.15.98:0/1541322879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8930109830 0x7f8930109c90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8928009ca0 tx=0x7f892802a8f0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.784+0000 7f8936ca6640 1 -- 172.21.15.98:0/1541322879 shutdown_connections 2024-09-19T08:06:05.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.784+0000 7f8936ca6640 1 --2- 172.21.15.98:0/1541322879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f893011b140 0x7f893011d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.784+0000 7f8936ca6640 1 --2- 172.21.15.98:0/1541322879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8930109830 0x7f8930109c90 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.784+0000 7f8936ca6640 1 --2- 172.21.15.98:0/1541322879 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8930109450 0x7f8930110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.784+0000 7f8936ca6640 1 -- 172.21.15.98:0/1541322879 >> 172.21.15.98:0/1541322879 conn(0x7f893006e140 msgr2=0x7f893006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:05.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.787+0000 7f8936ca6640 1 -- 172.21.15.98:0/1541322879 shutdown_connections 2024-09-19T08:06:05.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.789+0000 7f8936ca6640 1 -- 172.21.15.98:0/1541322879 wait complete. 2024-09-19T08:06:05.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.790+0000 7f8936ca6640 1 Processor -- start 2024-09-19T08:06:05.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.791+0000 7f8936ca6640 1 -- start start 2024-09-19T08:06:05.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.791+0000 7f8936ca6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8930109450 0x7f893011a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.791+0000 7f8936ca6640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8930109830 0x7f8930114960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.791+0000 7f8936ca6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f893011b140 0x7f8930114ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.791+0000 7f8936ca6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8930111370 con 0x7f8930109450 2024-09-19T08:06:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.791+0000 7f8936ca6640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f89301111f0 con 0x7f8930109830 2024-09-19T08:06:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.791+0000 7f8936ca6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f89301114f0 con 0x7f893011b140 2024-09-19T08:06:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.792+0000 7f8934a42640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8930109450 0x7f893011a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.792+0000 7f8934a42640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8930109450 0x7f893011a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52736/0 (socket says 172.21.15.98:52736) 2024-09-19T08:06:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.792+0000 7f8934a42640 1 -- 172.21.15.98:0/2275772785 learned_addr learned my addr 172.21.15.98:0/2275772785 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.792+0000 7f892ffff640 1 --2- 172.21.15.98:0/2275772785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8930109830 0x7f8930114960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.792+0000 7f8935243640 1 --2- 172.21.15.98:0/2275772785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f893011b140 0x7f8930114ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.793+0000 7f8934a42640 1 -- 172.21.15.98:0/2275772785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f893011b140 msgr2=0x7f8930114ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.793+0000 7f8934a42640 1 --2- 172.21.15.98:0/2275772785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f893011b140 0x7f8930114ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.793+0000 7f8934a42640 1 -- 172.21.15.98:0/2275772785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8930109830 msgr2=0x7f8930114960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.793+0000 7f8934a42640 1 --2- 172.21.15.98:0/2275772785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8930109830 0x7f8930114960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.793+0000 7f8934a42640 1 -- 172.21.15.98:0/2275772785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8930115620 con 0x7f8930109450 2024-09-19T08:06:05.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.793+0000 7f8934a42640 1 --2- 172.21.15.98:0/2275772785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8930109450 0x7f893011a8e0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f892000b450 tx=0x7f892000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:05.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.794+0000 7f892dffb640 1 -- 172.21.15.98:0/2275772785 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920015020 con 0x7f8930109450 2024-09-19T08:06:05.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.794+0000 7f8936ca6640 1 -- 172.21.15.98:0/2275772785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89301bc1d0 con 0x7f8930109450 2024-09-19T08:06:05.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.794+0000 7f8936ca6640 1 -- 172.21.15.98:0/2275772785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89301bc680 con 0x7f8930109450 2024-09-19T08:06:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.799+0000 7f893907f740 1 Processor -- start 2024-09-19T08:06:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.799+0000 7f893907f740 1 -- start start 2024-09-19T08:06:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.799+0000 7f893907f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfae0b40 0x5640dfadf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.799+0000 7f893907f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfadf5d0 0x5640dfadfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.799+0000 7f893907f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfaea370 0x5640dfaec760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.800+0000 7f893907f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5640dfa452f0 con 0x5640dfadf5d0 2024-09-19T08:06:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.800+0000 7f893907f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5640df9eccf0 con 0x5640dfae0b40 2024-09-19T08:06:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.800+0000 7f893907f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5640dfa412b0 con 0x5640dfaea370 2024-09-19T08:06:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.800+0000 7f892dffb640 1 -- 172.21.15.98:0/2275772785 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8920011d70 con 0x7f8930109450 2024-09-19T08:06:05.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.800+0000 7f892dffb640 1 -- 172.21.15.98:0/2275772785 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892000f340 con 0x7f8930109450 2024-09-19T08:06:05.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.801+0000 7f892ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfaea370 0x5640dfaec760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.801+0000 7f8935243640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfae0b40 0x5640dfadf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.802+0000 7f8935243640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfae0b40 0x5640dfadf090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:33502/0 (socket says 172.21.15.98:33502) 2024-09-19T08:06:05.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.802+0000 7f8935243640 1 -- 172.21.15.98:0/2299421511 learned_addr learned my addr 172.21.15.98:0/2299421511 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:05.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.802+0000 7f8934a42640 1 --2- 172.21.15.98:0/2299421511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfadf5d0 0x5640dfadfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.803+0000 7f8934a42640 1 -- 172.21.15.98:0/2299421511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfaea370 msgr2=0x5640dfaec760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:05.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.803+0000 7f8934a42640 1 --2- 172.21.15.98:0/2299421511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfaea370 0x5640dfaec760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.803+0000 7f8934a42640 1 -- 172.21.15.98:0/2299421511 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfae0b40 msgr2=0x5640dfadf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:05.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.803+0000 7f8934a42640 1 --2- 172.21.15.98:0/2299421511 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfae0b40 0x5640dfadf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.803+0000 7f8934a42640 1 -- 172.21.15.98:0/2299421511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640df9d13a0 con 0x5640dfadf5d0 2024-09-19T08:06:05.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.806+0000 7f892dffb640 1 -- 172.21.15.98:0/2275772785 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f8920018020 con 0x7f8930109450 2024-09-19T08:06:05.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.806+0000 7f8934a42640 1 --2- 172.21.15.98:0/2299421511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfadf5d0 0x5640dfadfa10 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f89301bc850 tx=0x7f8920011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d0955918b91038 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:05.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.807+0000 7f892dffb640 1 --2- 172.21.15.98:0/2275772785 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f89180c5550 0x7f89180c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.808+0000 7f8935243640 1 --2- 172.21.15.98:0/2275772785 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f89180c5550 0x7f89180c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.811+0000 7f8935243640 1 --2- 172.21.15.98:0/2275772785 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f89180c5550 0x7f89180c7a10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8924009b10 tx=0x7f8924009060 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.811+0000 7f8916ffd640 1 -- 172.21.15.98:0/2299421511 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89200d3030 con 0x5640dfadf5d0 2024-09-19T08:06:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.812+0000 7f892dffb640 1 -- 172.21.15.98:0/2275772785 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f8920011a80 con 0x7f8930109450 2024-09-19T08:06:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.812+0000 7f8916ffd640 1 -- 172.21.15.98:0/2299421511 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f892009cfa0 con 0x5640dfadf5d0 2024-09-19T08:06:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.812+0000 7f8916ffd640 1 -- 172.21.15.98:0/2299421511 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892009b2a0 con 0x5640dfadf5d0 2024-09-19T08:06:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.813+0000 7f893907f740 1 -- 172.21.15.98:0/2299421511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfadf5d0 msgr2=0x5640dfadfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.813+0000 7f893907f740 1 --2- 172.21.15.98:0/2299421511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfadf5d0 0x5640dfadfa10 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f89301bc850 tx=0x7f8920011890 comp rx=0 tx=0).stop 2024-09-19T08:06:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.814+0000 7f893907f740 1 -- 172.21.15.98:0/2299421511 shutdown_connections 2024-09-19T08:06:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.814+0000 7f893907f740 1 --2- 172.21.15.98:0/2299421511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfaea370 0x5640dfaec760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.814+0000 7f893907f740 1 --2- 172.21.15.98:0/2299421511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfadf5d0 0x5640dfadfa10 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.814+0000 7f893907f740 1 --2- 172.21.15.98:0/2299421511 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfae0b40 0x5640dfadf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.814+0000 7f893907f740 1 -- 172.21.15.98:0/2299421511 >> 172.21.15.98:0/2299421511 conn(0x5640dfaa8310 msgr2=0x5640dfadbc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.814+0000 7f893907f740 1 -- 172.21.15.98:0/2299421511 shutdown_connections 2024-09-19T08:06:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.816+0000 7f893907f740 1 -- 172.21.15.98:0/2299421511 wait complete. 2024-09-19T08:06:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.817+0000 7f893907f740 1 -- start start 2024-09-19T08:06:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.817+0000 7f893907f740 1 Processor -- start 2024-09-19T08:06:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.818+0000 7f893907f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfadf5d0 0x5640dfae9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.818+0000 7f893907f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfae0b40 0x5640dfae6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.818+0000 7f893907f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfaea370 0x5640dfae7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.819+0000 7f893907f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5640dfaed070 con 0x5640dfaea370 2024-09-19T08:06:05.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.819+0000 7f893907f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5640dfa00060 con 0x5640dfae0b40 2024-09-19T08:06:05.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.819+0000 7f893907f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5640dfaed1f0 con 0x5640dfadf5d0 2024-09-19T08:06:05.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.819+0000 7f893907f740 10 client.0 _ll_register_callbacks cb 0x5640df7538c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:05.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.819+0000 7f8935243640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfaea370 0x5640dfae7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.819+0000 7f8935243640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfaea370 0x5640dfae7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52766/0 (socket says 172.21.15.98:52766) 2024-09-19T08:06:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.819+0000 7f8935243640 1 -- 172.21.15.98:0/4005019327 learned_addr learned my addr 172.21.15.98:0/4005019327 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.820+0000 7f892ffff640 1 --2- 172.21.15.98:0/4005019327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfae0b40 0x5640dfae6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.820+0000 7f8934a42640 1 --2- 172.21.15.98:0/4005019327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfadf5d0 0x5640dfae9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.820+0000 7f8935243640 1 -- 172.21.15.98:0/4005019327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfadf5d0 msgr2=0x5640dfae9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.820+0000 7f8935243640 1 --2- 172.21.15.98:0/4005019327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfadf5d0 0x5640dfae9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.820+0000 7f8935243640 1 -- 172.21.15.98:0/4005019327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfae0b40 msgr2=0x5640dfae6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.820+0000 7f8935243640 1 --2- 172.21.15.98:0/4005019327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfae0b40 0x5640dfae6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.820+0000 7f8935243640 1 -- 172.21.15.98:0/4005019327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640df7b63c0 con 0x5640dfaea370 2024-09-19T08:06:05.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.821+0000 7f8935243640 1 --2- 172.21.15.98:0/4005019327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfaea370 0x5640dfae7480 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f89240149c0 tx=0x7f89240047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f56a8d4e7743b8e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:05.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.821+0000 7f890ffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.821+0000 7f890ffff640 1 -- 172.21.15.98:0/4005019327 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924025070 con 0x5640dfaea370 2024-09-19T08:06:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.821+0000 7f893907f740 1 -- 172.21.15.98:0/4005019327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f892009cfa0 con 0x5640dfaea370 2024-09-19T08:06:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.822+0000 7f890ffff640 1 -- 172.21.15.98:0/4005019327 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f8924021600 con 0x5640dfaea370 2024-09-19T08:06:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.822+0000 7f890ffff640 1 -- 172.21.15.98:0/4005019327 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8924021800 con 0x5640dfaea370 2024-09-19T08:06:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.823+0000 7f893907f740 10 client.16224 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.823+0000 7f893907f740 1 -- 172.21.15.98:0/4005019327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5640dfaf8a80 con 0x5640dfaea370 2024-09-19T08:06:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.823+0000 7f890ffff640 1 -- 172.21.15.98:0/4005019327 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f892009cfa0 con 0x5640dfaea370 2024-09-19T08:06:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.824+0000 7f890ffff640 1 -- 172.21.15.98:0/4005019327 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f8924032020 con 0x5640dfaea370 2024-09-19T08:06:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.824+0000 7f893907f740 10 client.16224 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.824+0000 7f893907f740 10 client.16224 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:06:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.824+0000 7f893907f740 20 client.16224 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.824+0000 7f893907f740 1 --2- 172.21.15.98:0/4005019327 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5640dfaf9f90 0x5640dfafc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.824+0000 7f893907f740 4 client.16224 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.824+0000 7f893907f740 1 -- 172.21.15.98:0/4005019327 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640dfa52860 con 0x5640dfaf9f90 2024-09-19T08:06:05.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.824+0000 7f8934a42640 1 --2- 172.21.15.98:0/4005019327 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5640dfaf9f90 0x5640dfafc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.826+0000 7f8934a42640 1 --2- 172.21.15.98:0/4005019327 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5640dfaf9f90 0x5640dfafc3d0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c6452578322c988d server_cookie=f97fb51378972432 in_seq=0 out_seq=0 2024-09-19T08:06:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.826+0000 7f890ffff640 10 client.16224 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:06:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.827+0000 7f890ffff640 1 -- 172.21.15.98:0/4005019327 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5640dfa52860 con 0x5640dfaf9f90 2024-09-19T08:06:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.827+0000 7f890ffff640 10 client.16224 handle_command_reply: tid=0 2024-09-19T08:06:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.827+0000 7f893907f740 1 client.16224 shutdown 2024-09-19T08:06:05.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.827+0000 7f893907f740 1 -- 172.21.15.98:0/4005019327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfaea370 msgr2=0x5640dfae7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:05.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.827+0000 7f893907f740 1 --2- 172.21.15.98:0/4005019327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfaea370 0x5640dfae7480 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f89240149c0 tx=0x7f89240047a0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.829+0000 7f893907f740 1 -- 172.21.15.98:0/4005019327 shutdown_connections 2024-09-19T08:06:05.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.829+0000 7f893907f740 1 --2- 172.21.15.98:0/4005019327 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5640dfaf9f90 0x5640dfafc3d0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.829+0000 7f893907f740 1 --2- 172.21.15.98:0/4005019327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfaea370 0x5640dfae7480 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.829+0000 7f893907f740 1 --2- 172.21.15.98:0/4005019327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfae0b40 0x5640dfae6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.829+0000 7f893907f740 1 --2- 172.21.15.98:0/4005019327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfadf5d0 0x5640dfae9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.829+0000 7f893907f740 1 -- 172.21.15.98:0/4005019327 >> 172.21.15.98:0/4005019327 conn(0x5640dfaa8310 msgr2=0x5640dfae1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.829+0000 7f890ffff640 0 client.16224 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:06:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.830+0000 7f893907f740 1 -- 172.21.15.98:0/4005019327 shutdown_connections 2024-09-19T08:06:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.830+0000 7f893907f740 1 -- 172.21.15.98:0/4005019327 wait complete. 2024-09-19T08:06:05.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.830+0000 7f893907f740 20 client.16224 trim_cache size 0 max 16384 2024-09-19T08:06:05.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:05 smithi098 ceph-mon[31333]: pgmap v95: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 512 MiB/s wr, 263 op/s 2024-09-19T08:06:05.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.850+0000 7f893907f740 1 Processor -- start 2024-09-19T08:06:05.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.850+0000 7f893907f740 1 -- start start 2024-09-19T08:06:05.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.850+0000 7f893907f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfadf5d0 0x5640dfaf6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.850+0000 7f893907f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfae0b40 0x5640dfae2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.851+0000 7f893907f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfaea370 0x5640dfae2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.851+0000 7f893907f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f891c00e7c0 con 0x5640dfaea370 2024-09-19T08:06:05.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.851+0000 7f893907f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f891c010360 con 0x5640dfadf5d0 2024-09-19T08:06:05.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.851+0000 7f893907f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f891c00ee90 con 0x5640dfae0b40 2024-09-19T08:06:05.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.851+0000 7f8935243640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfaea370 0x5640dfae2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.851+0000 7f8935243640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfaea370 0x5640dfae2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52780/0 (socket says 172.21.15.98:52780) 2024-09-19T08:06:05.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.851+0000 7f8935243640 1 -- 172.21.15.98:0/4193496232 learned_addr learned my addr 172.21.15.98:0/4193496232 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:05.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.851+0000 7f892ffff640 1 --2- 172.21.15.98:0/4193496232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfae0b40 0x5640dfae2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.851+0000 7f8934a42640 1 --2- 172.21.15.98:0/4193496232 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfadf5d0 0x5640dfaf6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.852+0000 7f8935243640 1 -- 172.21.15.98:0/4193496232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfae0b40 msgr2=0x5640dfae2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:05.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.852+0000 7f8935243640 1 --2- 172.21.15.98:0/4193496232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfae0b40 0x5640dfae2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.852+0000 7f8935243640 1 -- 172.21.15.98:0/4193496232 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfadf5d0 msgr2=0x5640dfaf6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:05.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.852+0000 7f8935243640 1 --2- 172.21.15.98:0/4193496232 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfadf5d0 0x5640dfaf6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.852+0000 7f8935243640 1 -- 172.21.15.98:0/4193496232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640dfadbed0 con 0x5640dfaea370 2024-09-19T08:06:05.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.852+0000 7f8935243640 1 --2- 172.21.15.98:0/4193496232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfaea370 0x5640dfae2660 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8924009610 tx=0x7f8924021790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72c37b9d3503a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:05.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.853+0000 7f89167fc640 1 -- 172.21.15.98:0/4193496232 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892403a070 con 0x5640dfaea370 2024-09-19T08:06:05.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.853+0000 7f89167fc640 1 -- 172.21.15.98:0/4193496232 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f892401b070 con 0x5640dfaea370 2024-09-19T08:06:05.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.853+0000 7f89167fc640 1 -- 172.21.15.98:0/4193496232 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924025040 con 0x5640dfaea370 2024-09-19T08:06:05.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.855+0000 7f893907f740 1 -- 172.21.15.98:0/4193496232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfaea370 msgr2=0x5640dfae2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:05.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.855+0000 7f893907f740 1 --2- 172.21.15.98:0/4193496232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfaea370 0x5640dfae2660 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8924009610 tx=0x7f8924021790 comp rx=0 tx=0).stop 2024-09-19T08:06:05.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.855+0000 7f893907f740 1 -- 172.21.15.98:0/4193496232 shutdown_connections 2024-09-19T08:06:05.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.855+0000 7f893907f740 1 --2- 172.21.15.98:0/4193496232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfaea370 0x5640dfae2660 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.855+0000 7f893907f740 1 --2- 172.21.15.98:0/4193496232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfae0b40 0x5640dfae2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.855+0000 7f893907f740 1 --2- 172.21.15.98:0/4193496232 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfadf5d0 0x5640dfaf6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.855+0000 7f893907f740 1 -- 172.21.15.98:0/4193496232 >> 172.21.15.98:0/4193496232 conn(0x5640dfaa8310 msgr2=0x5640dfae8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:05.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.856+0000 7f893907f740 1 -- 172.21.15.98:0/4193496232 shutdown_connections 2024-09-19T08:06:05.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.856+0000 7f893907f740 1 -- 172.21.15.98:0/4193496232 wait complete. 2024-09-19T08:06:05.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.856+0000 7f893907f740 1 -- start start 2024-09-19T08:06:05.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.857+0000 7f893907f740 1 Processor -- start 2024-09-19T08:06:05.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.858+0000 7f893907f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfadf5d0 0x5640dfaf2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.858+0000 7f893907f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfae0b40 0x5640dfaee050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.858+0000 7f893907f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfaea370 0x5640dfaee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.858+0000 7f893907f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f891c00fc60 con 0x5640dfadf5d0 2024-09-19T08:06:05.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.858+0000 7f893907f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f891c00f590 con 0x5640dfaea370 2024-09-19T08:06:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.859+0000 7f892ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfae0b40 0x5640dfaee050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.859+0000 7f892ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfae0b40 0x5640dfaee050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:37926/0 (socket says 172.21.15.98:37926) 2024-09-19T08:06:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.859+0000 7f892ffff640 1 -- 172.21.15.98:0/413088377 learned_addr learned my addr 172.21.15.98:0/413088377 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.860+0000 7f8934a42640 1 --2- 172.21.15.98:0/413088377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfadf5d0 0x5640dfaf2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.860+0000 7f8935243640 1 --2- 172.21.15.98:0/413088377 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfaea370 0x5640dfaee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.860+0000 7f893907f740 1 -- 172.21.15.98:0/413088377 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f891c00c7a0 con 0x5640dfae0b40 2024-09-19T08:06:05.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.860+0000 7f893907f740 10 client.0 _ll_register_callbacks cb 0x7f8924009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:05.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.861+0000 7f892ffff640 1 -- 172.21.15.98:0/413088377 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfaea370 msgr2=0x5640dfaee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:05.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.861+0000 7f892ffff640 1 --2- 172.21.15.98:0/413088377 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfaea370 0x5640dfaee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.861+0000 7f892ffff640 1 -- 172.21.15.98:0/413088377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfadf5d0 msgr2=0x5640dfaf2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:05.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.861+0000 7f892ffff640 1 --2- 172.21.15.98:0/413088377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfadf5d0 0x5640dfaf2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.861+0000 7f892ffff640 1 -- 172.21.15.98:0/413088377 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640dfb63c70 con 0x5640dfae0b40 2024-09-19T08:06:05.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.862+0000 7f892ffff640 1 --2- 172.21.15.98:0/413088377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfae0b40 0x5640dfaee050 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8928009ca0 tx=0x7f8928003ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b29b618d23ee2bf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:05.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.863+0000 7f890ffff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-09-19T08:06:05.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.863+0000 7f890ffff640 1 -- 172.21.15.98:0/413088377 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8928031890 con 0x5640dfae0b40 2024-09-19T08:06:05.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.863+0000 7f890ffff640 1 -- 172.21.15.98:0/413088377 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f8928004040 con 0x5640dfae0b40 2024-09-19T08:06:05.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.863+0000 7f893907f740 1 -- 172.21.15.98:0/413088377 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f892401b070 con 0x5640dfae0b40 2024-09-19T08:06:05.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.864+0000 7f890ffff640 1 -- 172.21.15.98:0/413088377 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8928004240 con 0x5640dfae0b40 2024-09-19T08:06:05.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.864+0000 7f893907f740 10 client.25468 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:05.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.864+0000 7f893907f740 1 -- 172.21.15.98:0/413088377 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5640dfb64dd0 con 0x5640dfae0b40 2024-09-19T08:06:05.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.865+0000 7f890ffff640 1 -- 172.21.15.98:0/413088377 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f892401b070 con 0x5640dfae0b40 2024-09-19T08:06:05.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.865+0000 7f890ffff640 1 -- 172.21.15.98:0/413088377 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f892804f030 con 0x5640dfae0b40 2024-09-19T08:06:05.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.865+0000 7f893907f740 10 client.25468 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:05.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.865+0000 7f893907f740 10 client.25468 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:06:05.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.865+0000 7f893907f740 20 client.25468 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:05.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.865+0000 7f893907f740 1 --2- 172.21.15.98:0/413088377 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5640dfb65600 0x5640dfb67a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:05.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.865+0000 7f893907f740 4 client.25468 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:06:05.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.865+0000 7f893907f740 1 -- 172.21.15.98:0/413088377 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5640dfa4cd70 con 0x5640dfb65600 2024-09-19T08:06:05.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.867+0000 7f8934a42640 1 --2- 172.21.15.98:0/413088377 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5640dfb65600 0x5640dfb67a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:05.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.869+0000 7f890ffff640 10 client.25468 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:06:05.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.870+0000 7f8934a42640 1 --2- 172.21.15.98:0/413088377 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5640dfb65600 0x5640dfb67a20 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=47f821aa46f5098f server_cookie=9c521b7fb3d0dee4 in_seq=0 out_seq=0 2024-09-19T08:06:05.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.870+0000 7f890ffff640 1 -- 172.21.15.98:0/413088377 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5640dfa4cd70 con 0x5640dfb65600 2024-09-19T08:06:05.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.870+0000 7f890ffff640 10 client.25468 handle_command_reply: tid=0 2024-09-19T08:06:05.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.870+0000 7f893907f740 1 client.25468 shutdown 2024-09-19T08:06:05.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.874+0000 7f893907f740 1 -- 172.21.15.98:0/413088377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfae0b40 msgr2=0x5640dfaee050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:05.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.874+0000 7f893907f740 1 --2- 172.21.15.98:0/413088377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfae0b40 0x5640dfaee050 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8928009ca0 tx=0x7f8928003ce0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.874+0000 7f893907f740 1 -- 172.21.15.98:0/413088377 shutdown_connections 2024-09-19T08:06:05.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.874+0000 7f893907f740 1 --2- 172.21.15.98:0/413088377 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5640dfb65600 0x5640dfb67a20 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.874+0000 7f893907f740 1 --2- 172.21.15.98:0/413088377 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5640dfaea370 0x5640dfaee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.874+0000 7f893907f740 1 --2- 172.21.15.98:0/413088377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640dfae0b40 0x5640dfaee050 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.874+0000 7f893907f740 1 --2- 172.21.15.98:0/413088377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5640dfadf5d0 0x5640dfaf2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.874+0000 7f893907f740 1 -- 172.21.15.98:0/413088377 >> 172.21.15.98:0/413088377 conn(0x5640dfaa8310 msgr2=0x5640dfada4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:05.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.875+0000 7f890ffff640 0 client.25468 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:06:05.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.875+0000 7f893907f740 1 -- 172.21.15.98:0/413088377 shutdown_connections 2024-09-19T08:06:05.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.875+0000 7f893907f740 1 -- 172.21.15.98:0/413088377 wait complete. 2024-09-19T08:06:05.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.878+0000 7f893907f740 20 client.25468 trim_cache size 0 max 16384 2024-09-19T08:06:05.879 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:06:05.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.880+0000 7f8936ca6640 1 -- 172.21.15.98:0/2275772785 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f89180c5550 msgr2=0x7f89180c7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:05.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.880+0000 7f8936ca6640 1 --2- 172.21.15.98:0/2275772785 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f89180c5550 0x7f89180c7a10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8924009b10 tx=0x7f8924009060 comp rx=0 tx=0).stop 2024-09-19T08:06:05.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.880+0000 7f8936ca6640 1 -- 172.21.15.98:0/2275772785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8930109450 msgr2=0x7f893011a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:05.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.880+0000 7f8936ca6640 1 --2- 172.21.15.98:0/2275772785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8930109450 0x7f893011a8e0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f892000b450 tx=0x7f892000b910 comp rx=0 tx=0).stop 2024-09-19T08:06:05.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.880+0000 7f8936ca6640 1 -- 172.21.15.98:0/2275772785 shutdown_connections 2024-09-19T08:06:05.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.880+0000 7f8936ca6640 1 --2- 172.21.15.98:0/2275772785 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f89180c5550 0x7f89180c7a10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.880+0000 7f8936ca6640 1 --2- 172.21.15.98:0/2275772785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f893011b140 0x7f8930114ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.880+0000 7f8936ca6640 1 --2- 172.21.15.98:0/2275772785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8930109830 0x7f8930114960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.880+0000 7f8936ca6640 1 --2- 172.21.15.98:0/2275772785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8930109450 0x7f893011a8e0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:05.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.880+0000 7f8936ca6640 1 -- 172.21.15.98:0/2275772785 >> 172.21.15.98:0/2275772785 conn(0x7f893006e140 msgr2=0x7f89300722b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:05.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.880+0000 7f8936ca6640 1 -- 172.21.15.98:0/2275772785 shutdown_connections 2024-09-19T08:06:05.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:05.880+0000 7f8936ca6640 1 -- 172.21.15.98:0/2275772785 wait complete. 2024-09-19T08:06:06.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:05 smithi162 ceph-mon[35260]: pgmap v95: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 512 MiB/s wr, 263 op/s 2024-09-19T08:06:06.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:05 smithi139 ceph-mon[34874]: pgmap v95: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 512 MiB/s wr, 263 op/s 2024-09-19T08:06:06.782 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:06:06.782 INFO:tasks.workunit.client.0.smithi098.stdout: Children see throughput for 1 rewriters = 823313.75 kB/sec 2024-09-19T08:06:06.782 INFO:tasks.workunit.client.0.smithi098.stdout: Parent sees throughput for 1 rewriters = 823298.10 kB/sec 2024-09-19T08:06:06.783 INFO:tasks.workunit.client.0.smithi098.stdout: Min throughput per process = 823313.75 kB/sec 2024-09-19T08:06:06.783 INFO:tasks.workunit.client.0.smithi098.stdout: Max throughput per process = 823313.75 kB/sec 2024-09-19T08:06:06.783 INFO:tasks.workunit.client.0.smithi098.stdout: Avg throughput per process = 823313.75 kB/sec 2024-09-19T08:06:06.783 INFO:tasks.workunit.client.0.smithi098.stdout: Min xfer = 10485760.00 kB 2024-09-19T08:06:07.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:06 smithi139 ceph-mon[34874]: pgmap v96: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 694 MiB/s wr, 311 op/s 2024-09-19T08:06:07.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:06 smithi098 ceph-mon[31333]: pgmap v96: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 694 MiB/s wr, 311 op/s 2024-09-19T08:06:07.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:06:07.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:06 smithi162 ceph-mon[35260]: pgmap v96: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 694 MiB/s wr, 311 op/s 2024-09-19T08:06:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:08.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.016+0000 7f2a62f8c640 1 -- 172.21.15.98:0/3665554745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a5c106f20 msgr2=0x7f2a5c10eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:08.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.016+0000 7f2a62f8c640 1 --2- 172.21.15.98:0/3665554745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a5c106f20 0x7f2a5c10eaa0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f2a54009900 tx=0x7f2a5402a410 comp rx=0 tx=0).stop 2024-09-19T08:06:08.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.016+0000 7f2a62f8c640 1 -- 172.21.15.98:0/3665554745 shutdown_connections 2024-09-19T08:06:08.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.016+0000 7f2a62f8c640 1 --2- 172.21.15.98:0/3665554745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a5c106f20 0x7f2a5c10eaa0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.016+0000 7f2a62f8c640 1 --2- 172.21.15.98:0/3665554745 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a5c1027a0 0x7f2a5c1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.016+0000 7f2a62f8c640 1 --2- 172.21.15.98:0/3665554745 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a5c101df0 0x7f2a5c1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.016+0000 7f2a62f8c640 1 -- 172.21.15.98:0/3665554745 >> 172.21.15.98:0/3665554745 conn(0x7f2a5c0fd8f0 msgr2=0x7f2a5c0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:08.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.016+0000 7f2a62f8c640 1 -- 172.21.15.98:0/3665554745 shutdown_connections 2024-09-19T08:06:08.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.016+0000 7f2a62f8c640 1 -- 172.21.15.98:0/3665554745 wait complete. 2024-09-19T08:06:08.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.016+0000 7f2a62f8c640 1 Processor -- start 2024-09-19T08:06:08.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.016+0000 7f2a62f8c640 1 -- start start 2024-09-19T08:06:08.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a62f8c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a5c101df0 0x7f2a5c198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a62f8c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a5c1027a0 0x7f2a5c198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a62f8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a5c106f20 0x7f2a5c19cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a62f8c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c111f80 con 0x7f2a5c106f20 2024-09-19T08:06:08.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a62f8c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c111e00 con 0x7f2a5c101df0 2024-09-19T08:06:08.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a62f8c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c112100 con 0x7f2a5c1027a0 2024-09-19T08:06:08.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a61529640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a5c106f20 0x7f2a5c19cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a61529640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a5c106f20 0x7f2a5c19cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43800/0 (socket says 172.21.15.98:43800) 2024-09-19T08:06:08.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a61529640 1 -- 172.21.15.98:0/1209472764 learned_addr learned my addr 172.21.15.98:0/1209472764 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:08.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a60d28640 1 --2- 172.21.15.98:0/1209472764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a5c101df0 0x7f2a5c198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a5bfff640 1 --2- 172.21.15.98:0/1209472764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a5c1027a0 0x7f2a5c198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a61529640 1 -- 172.21.15.98:0/1209472764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a5c1027a0 msgr2=0x7f2a5c198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:08.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a61529640 1 --2- 172.21.15.98:0/1209472764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a5c1027a0 0x7f2a5c198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a61529640 1 -- 172.21.15.98:0/1209472764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a5c101df0 msgr2=0x7f2a5c198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:08.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a61529640 1 --2- 172.21.15.98:0/1209472764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a5c101df0 0x7f2a5c198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a61529640 1 -- 172.21.15.98:0/1209472764 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5c19d2f0 con 0x7f2a5c106f20 2024-09-19T08:06:08.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a61529640 1 --2- 172.21.15.98:0/1209472764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a5c106f20 0x7f2a5c19cc10 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2a54005ab0 tx=0x7f2a540042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:08.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a59ffb640 1 -- 172.21.15.98:0/1209472764 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5403f030 con 0x7f2a5c106f20 2024-09-19T08:06:08.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a59ffb640 1 -- 172.21.15.98:0/1209472764 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a54003ce0 con 0x7f2a5c106f20 2024-09-19T08:06:08.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a62f8c640 1 -- 172.21.15.98:0/1209472764 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a5c19d580 con 0x7f2a5c106f20 2024-09-19T08:06:08.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.017+0000 7f2a59ffb640 1 -- 172.21.15.98:0/1209472764 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a54035070 con 0x7f2a5c106f20 2024-09-19T08:06:08.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.018+0000 7f2a62f8c640 1 -- 172.21.15.98:0/1209472764 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a5c1a4ec0 con 0x7f2a5c106f20 2024-09-19T08:06:08.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.020+0000 7f2a59ffb640 1 -- 172.21.15.98:0/1209472764 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f2a54042030 con 0x7f2a5c106f20 2024-09-19T08:06:08.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.020+0000 7f2a65365740 1 Processor -- start 2024-09-19T08:06:08.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.020+0000 7f2a65365740 1 -- start start 2024-09-19T08:06:08.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.020+0000 7f2a59ffb640 1 --2- 172.21.15.98:0/1209472764 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2a480a33b0 0x7f2a480a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.020+0000 7f2a65365740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fca7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.020+0000 7f2a65365740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 0x55f05fca7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.020+0000 7f2a65365740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.020+0000 7f2a65365740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f05fc0d2f0 con 0x55f05fca8b40 2024-09-19T08:06:08.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a65365740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f05fbb4cf0 con 0x55f05fcb6780 2024-09-19T08:06:08.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a65365740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f05fc092b0 con 0x55f05fca75d0 2024-09-19T08:06:08.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a5bfff640 1 --2- 172.21.15.98:0/1209472764 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2a480a33b0 0x7f2a480a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a59ffb640 1 -- 172.21.15.98:0/1209472764 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f2a540f0420 con 0x7f2a5c106f20 2024-09-19T08:06:08.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a60d28640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fca7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a60d28640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fca7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43806/0 (socket says 172.21.15.98:43806) 2024-09-19T08:06:08.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a60d28640 1 -- 172.21.15.98:0/4197657283 learned_addr learned my addr 172.21.15.98:0/4197657283 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:08.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a61529640 1 --2- 172.21.15.98:0/4197657283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a5bfff640 1 --2- 172.21.15.98:0/4197657283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 0x55f05fca7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a60d28640 1 -- 172.21.15.98:0/4197657283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 msgr2=0x55f05fca7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:08.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a60d28640 1 --2- 172.21.15.98:0/4197657283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 0x55f05fca7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a60d28640 1 -- 172.21.15.98:0/4197657283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 msgr2=0x55f05fcb8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:08.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a60d28640 1 --2- 172.21.15.98:0/4197657283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a60d28640 1 -- 172.21.15.98:0/4197657283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f05fb993a0 con 0x55f05fca8b40 2024-09-19T08:06:08.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a61529640 1 --2- 172.21.15.98:0/4197657283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:08.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a5bfff640 1 --2- 172.21.15.98:0/1209472764 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2a480a33b0 0x7f2a480a5870 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2a440058a0 tx=0x7f2a4400b6f0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:08.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a60d28640 1 --2- 172.21.15.98:0/4197657283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fca7090 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c007f90 tx=0x7f2a4c00f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826887fdafd05916 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:08.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a52ffd640 1 -- 172.21.15.98:0/4197657283 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c004d20 con 0x55f05fca8b40 2024-09-19T08:06:08.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a52ffd640 1 -- 172.21.15.98:0/4197657283 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a4c010be0 con 0x55f05fca8b40 2024-09-19T08:06:08.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.021+0000 7f2a52ffd640 1 -- 172.21.15.98:0/4197657283 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c005780 con 0x55f05fca8b40 2024-09-19T08:06:08.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.022+0000 7f2a65365740 1 -- 172.21.15.98:0/4197657283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 msgr2=0x55f05fca7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:08.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.022+0000 7f2a65365740 1 --2- 172.21.15.98:0/4197657283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fca7090 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c007f90 tx=0x7f2a4c00f850 comp rx=0 tx=0).stop 2024-09-19T08:06:08.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.022+0000 7f2a65365740 1 -- 172.21.15.98:0/4197657283 shutdown_connections 2024-09-19T08:06:08.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.022+0000 7f2a65365740 1 --2- 172.21.15.98:0/4197657283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.022+0000 7f2a65365740 1 --2- 172.21.15.98:0/4197657283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 0x55f05fca7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.022+0000 7f2a65365740 1 --2- 172.21.15.98:0/4197657283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fca7090 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.022+0000 7f2a65365740 1 -- 172.21.15.98:0/4197657283 >> 172.21.15.98:0/4197657283 conn(0x55f05fc70310 msgr2=0x55f05fca3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:08.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.022+0000 7f2a65365740 1 -- 172.21.15.98:0/4197657283 shutdown_connections 2024-09-19T08:06:08.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.022+0000 7f2a65365740 1 -- 172.21.15.98:0/4197657283 wait complete. 2024-09-19T08:06:08.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.022+0000 7f2a65365740 1 -- start start 2024-09-19T08:06:08.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.022+0000 7f2a65365740 1 Processor -- start 2024-09-19T08:06:08.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a65365740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 0x55f05fcb60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a65365740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fcb33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a65365740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a65365740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f05fcb9480 con 0x55f05fca8b40 2024-09-19T08:06:08.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a65365740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f05fbc8060 con 0x55f05fcb6780 2024-09-19T08:06:08.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a65365740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f05fcb9600 con 0x55f05fca75d0 2024-09-19T08:06:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a65365740 10 client.0 _ll_register_callbacks cb 0x55f05f91b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a5bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fcb33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a5bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fcb33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43810/0 (socket says 172.21.15.98:43810) 2024-09-19T08:06:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a5bfff640 1 -- 172.21.15.98:0/3261440982 learned_addr learned my addr 172.21.15.98:0/3261440982 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a60d28640 1 --2- 172.21.15.98:0/3261440982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 0x55f05fcb60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a61529640 1 --2- 172.21.15.98:0/3261440982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a5bfff640 1 -- 172.21.15.98:0/3261440982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 msgr2=0x55f05fcb60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a5bfff640 1 --2- 172.21.15.98:0/3261440982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 0x55f05fcb60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a5bfff640 1 -- 172.21.15.98:0/3261440982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 msgr2=0x55f05fcb3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a5bfff640 1 --2- 172.21.15.98:0/3261440982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a5bfff640 1 -- 172.21.15.98:0/3261440982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f05f97e3c0 con 0x55f05fca8b40 2024-09-19T08:06:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a61529640 1 --2- 172.21.15.98:0/3261440982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a5bfff640 1 --2- 172.21.15.98:0/3261440982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fcb33a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f2a4401e660 tx=0x7f2a44004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d04f77fce2addf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.023+0000 7f2a3bfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.024+0000 7f2a3bfff640 1 -- 172.21.15.98:0/3261440982 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4402b060 con 0x55f05fca8b40 2024-09-19T08:06:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.024+0000 7f2a3bfff640 1 -- 172.21.15.98:0/3261440982 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f2a4401c780 con 0x55f05fca8b40 2024-09-19T08:06:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.024+0000 7f2a65365740 1 -- 172.21.15.98:0/3261440982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a4c010be0 con 0x55f05fca8b40 2024-09-19T08:06:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.024+0000 7f2a3bfff640 1 -- 172.21.15.98:0/3261440982 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2a4401c980 con 0x55f05fca8b40 2024-09-19T08:06:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.024+0000 7f2a65365740 10 client.16242 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.024+0000 7f2a65365740 1 -- 172.21.15.98:0/3261440982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f05fcc4ef0 con 0x55f05fca8b40 2024-09-19T08:06:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.024+0000 7f2a3bfff640 1 -- 172.21.15.98:0/3261440982 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a4c010be0 con 0x55f05fca8b40 2024-09-19T08:06:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.024+0000 7f2a3bfff640 1 -- 172.21.15.98:0/3261440982 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f2a4400f3c0 con 0x55f05fca8b40 2024-09-19T08:06:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.024+0000 7f2a65365740 10 client.16242 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.024+0000 7f2a65365740 10 client.16242 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:06:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.024+0000 7f2a65365740 20 client.16242 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:08.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.024+0000 7f2a65365740 1 --2- 172.21.15.98:0/3261440982 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f05fcc6400 0x55f05fcc8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.024+0000 7f2a65365740 4 client.16242 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:08.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.024+0000 7f2a65365740 1 -- 172.21.15.98:0/3261440982 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f05fc1a860 con 0x55f05fcc6400 2024-09-19T08:06:08.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.025+0000 7f2a60d28640 1 --2- 172.21.15.98:0/3261440982 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f05fcc6400 0x55f05fcc8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.025+0000 7f2a3bfff640 10 client.16242 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:06:08.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.025+0000 7f2a60d28640 1 --2- 172.21.15.98:0/3261440982 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f05fcc6400 0x55f05fcc8810 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=13c7c664f246a67 server_cookie=a0acd4f5e2f1e177 in_seq=0 out_seq=0 2024-09-19T08:06:08.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a3bfff640 1 -- 172.21.15.98:0/3261440982 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f05fc1a860 con 0x55f05fcc6400 2024-09-19T08:06:08.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a3bfff640 10 client.16242 handle_command_reply: tid=0 2024-09-19T08:06:08.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a65365740 1 client.16242 shutdown 2024-09-19T08:06:08.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a65365740 1 -- 172.21.15.98:0/3261440982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 msgr2=0x55f05fcb33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:08.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a65365740 1 --2- 172.21.15.98:0/3261440982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fcb33a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f2a4401e660 tx=0x7f2a44004730 comp rx=0 tx=0).stop 2024-09-19T08:06:08.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a65365740 1 -- 172.21.15.98:0/3261440982 shutdown_connections 2024-09-19T08:06:08.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a65365740 1 --2- 172.21.15.98:0/3261440982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a65365740 1 --2- 172.21.15.98:0/3261440982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fcb33a0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a65365740 1 --2- 172.21.15.98:0/3261440982 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f05fcc6400 0x55f05fcc8810 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a65365740 1 --2- 172.21.15.98:0/3261440982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 0x55f05fcb60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a3bfff640 0 client.16242 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:06:08.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a65365740 1 -- 172.21.15.98:0/3261440982 >> 172.21.15.98:0/3261440982 conn(0x55f05fc70310 msgr2=0x55f05fca9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:08.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a65365740 1 -- 172.21.15.98:0/3261440982 shutdown_connections 2024-09-19T08:06:08.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a65365740 1 -- 172.21.15.98:0/3261440982 wait complete. 2024-09-19T08:06:08.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.026+0000 7f2a65365740 20 client.16242 trim_cache size 0 max 16384 2024-09-19T08:06:08.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.039+0000 7f2a65365740 1 Processor -- start 2024-09-19T08:06:08.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.039+0000 7f2a65365740 1 -- start start 2024-09-19T08:06:08.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.039+0000 7f2a65365740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 0x55f05fca3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.039+0000 7f2a65365740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fca8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.039+0000 7f2a65365740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcc2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.039+0000 7f2a65365740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c1ab550 con 0x55f05fca8b40 2024-09-19T08:06:08.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.039+0000 7f2a65365740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c1ad0f0 con 0x55f05fcb6780 2024-09-19T08:06:08.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.039+0000 7f2a65365740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c1abc20 con 0x55f05fca75d0 2024-09-19T08:06:08.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.040+0000 7f2a5bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fca8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.040+0000 7f2a5bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fca8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43818/0 (socket says 172.21.15.98:43818) 2024-09-19T08:06:08.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.040+0000 7f2a5bfff640 1 -- 172.21.15.98:0/3249110380 learned_addr learned my addr 172.21.15.98:0/3249110380 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:08.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.040+0000 7f2a60d28640 1 --2- 172.21.15.98:0/3249110380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 0x55f05fca3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.040+0000 7f2a5bfff640 1 -- 172.21.15.98:0/3249110380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 msgr2=0x55f05fca3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:08.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.040+0000 7f2a5bfff640 1 --2- 172.21.15.98:0/3249110380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 0x55f05fca3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.040+0000 7f2a5bfff640 1 -- 172.21.15.98:0/3249110380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 msgr2=0x55f05fcc2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:08.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.040+0000 7f2a5bfff640 1 --2- 172.21.15.98:0/3249110380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcc2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.040+0000 7f2a5bfff640 1 -- 172.21.15.98:0/3249110380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f05fca4d40 con 0x55f05fca8b40 2024-09-19T08:06:08.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.040+0000 7f2a5bfff640 1 --2- 172.21.15.98:0/3249110380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fca8300 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400f810 tx=0x7f2a44012df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa593c0f678a06e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:08.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.040+0000 7f2a527fc640 1 -- 172.21.15.98:0/3249110380 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4400fdd0 con 0x55f05fca8b40 2024-09-19T08:06:08.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.040+0000 7f2a527fc640 1 -- 172.21.15.98:0/3249110380 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a4400f3c0 con 0x55f05fca8b40 2024-09-19T08:06:08.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.040+0000 7f2a527fc640 1 -- 172.21.15.98:0/3249110380 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a440308f0 con 0x55f05fca8b40 2024-09-19T08:06:08.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.041+0000 7f2a65365740 1 -- 172.21.15.98:0/3249110380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 msgr2=0x55f05fca8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:08.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.041+0000 7f2a65365740 1 --2- 172.21.15.98:0/3249110380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fca8300 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400f810 tx=0x7f2a44012df0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.041+0000 7f2a65365740 1 -- 172.21.15.98:0/3249110380 shutdown_connections 2024-09-19T08:06:08.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.041+0000 7f2a65365740 1 --2- 172.21.15.98:0/3249110380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcc2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.041+0000 7f2a65365740 1 --2- 172.21.15.98:0/3249110380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca8b40 0x55f05fca8300 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.041+0000 7f2a65365740 1 --2- 172.21.15.98:0/3249110380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca75d0 0x55f05fca3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.041+0000 7f2a65365740 1 -- 172.21.15.98:0/3249110380 >> 172.21.15.98:0/3249110380 conn(0x55f05fc70310 msgr2=0x55f05fca3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:08.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.041+0000 7f2a65365740 1 -- 172.21.15.98:0/3249110380 shutdown_connections 2024-09-19T08:06:08.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.041+0000 7f2a65365740 1 -- 172.21.15.98:0/3249110380 wait complete. 2024-09-19T08:06:08.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.041+0000 7f2a65365740 1 -- start start 2024-09-19T08:06:08.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.041+0000 7f2a65365740 1 Processor -- start 2024-09-19T08:06:08.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a65365740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca75d0 0x55f05fcbed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a65365740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca8b40 0x55f05fcafed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a65365740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a65365740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c1ac9f0 con 0x55f05fca75d0 2024-09-19T08:06:08.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a65365740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c1ac320 con 0x55f05fcb6780 2024-09-19T08:06:08.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a65365740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c1a9550 con 0x55f05fca8b40 2024-09-19T08:06:08.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a65365740 10 client.0 _ll_register_callbacks cb 0x7f2a4400b570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:08.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a60d28640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca75d0 0x55f05fcbed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a60d28640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca75d0 0x55f05fcbed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43820/0 (socket says 172.21.15.98:43820) 2024-09-19T08:06:08.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a60d28640 1 -- 172.21.15.98:0/2658347657 learned_addr learned my addr 172.21.15.98:0/2658347657 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:08.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a61529640 1 --2- 172.21.15.98:0/2658347657 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a5bfff640 1 --2- 172.21.15.98:0/2658347657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca8b40 0x55f05fcafed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a60d28640 1 -- 172.21.15.98:0/2658347657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca8b40 msgr2=0x55f05fcafed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:08.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a60d28640 1 --2- 172.21.15.98:0/2658347657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca8b40 0x55f05fcafed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a60d28640 1 -- 172.21.15.98:0/2658347657 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 msgr2=0x55f05fcb0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:08.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a60d28640 1 --2- 172.21.15.98:0/2658347657 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a60d28640 1 -- 172.21.15.98:0/2658347657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f05fca4960 con 0x55f05fca75d0 2024-09-19T08:06:08.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a61529640 1 --2- 172.21.15.98:0/2658347657 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:08.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a60d28640 1 --2- 172.21.15.98:0/2658347657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca75d0 0x55f05fcbed80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c007b60 tx=0x7f2a4c005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9dcc53672bf5537 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:08.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a3bfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:08.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a3bfff640 1 -- 172.21.15.98:0/2658347657 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c01d040 con 0x55f05fca75d0 2024-09-19T08:06:08.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a3bfff640 1 -- 172.21.15.98:0/2658347657 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f2a4c005520 con 0x55f05fca75d0 2024-09-19T08:06:08.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.042+0000 7f2a65365740 1 -- 172.21.15.98:0/2658347657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a4400f3c0 con 0x55f05fca75d0 2024-09-19T08:06:08.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.043+0000 7f2a3bfff640 1 -- 172.21.15.98:0/2658347657 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2a4c039050 con 0x55f05fca75d0 2024-09-19T08:06:08.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.043+0000 7f2a65365740 10 client.16248 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:08.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.043+0000 7f2a65365740 1 -- 172.21.15.98:0/2658347657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f05fd30d30 con 0x55f05fca75d0 2024-09-19T08:06:08.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.043+0000 7f2a3bfff640 1 -- 172.21.15.98:0/2658347657 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a4400f3c0 con 0x55f05fca75d0 2024-09-19T08:06:08.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.043+0000 7f2a3bfff640 1 -- 172.21.15.98:0/2658347657 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f2a4c004ba0 con 0x55f05fca75d0 2024-09-19T08:06:08.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.043+0000 7f2a65365740 10 client.16248 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:08.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.043+0000 7f2a65365740 10 client.16248 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:06:08.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.043+0000 7f2a65365740 20 client.16248 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:08.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.043+0000 7f2a65365740 1 --2- 172.21.15.98:0/2658347657 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f05fd313a0 0x55f05fd337c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:08.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.043+0000 7f2a65365740 4 client.16248 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:06:08.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.043+0000 7f2a65365740 1 -- 172.21.15.98:0/2658347657 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f05fc14d70 con 0x55f05fd313a0 2024-09-19T08:06:08.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.043+0000 7f2a5bfff640 1 --2- 172.21.15.98:0/2658347657 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f05fd313a0 0x55f05fd337c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:08.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a3bfff640 10 client.16248 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:06:08.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a5bfff640 1 --2- 172.21.15.98:0/2658347657 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f05fd313a0 0x55f05fd337c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9ca8e52eb61bb8d3 server_cookie=995ad945d1740915 in_seq=0 out_seq=0 2024-09-19T08:06:08.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a3bfff640 1 -- 172.21.15.98:0/2658347657 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f05fc14d70 con 0x55f05fd313a0 2024-09-19T08:06:08.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a3bfff640 10 client.16248 handle_command_reply: tid=0 2024-09-19T08:06:08.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a65365740 1 client.16248 shutdown 2024-09-19T08:06:08.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a65365740 1 -- 172.21.15.98:0/2658347657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca75d0 msgr2=0x55f05fcbed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:08.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a65365740 1 --2- 172.21.15.98:0/2658347657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca75d0 0x55f05fcbed80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c007b60 tx=0x7f2a4c005bf0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a65365740 1 -- 172.21.15.98:0/2658347657 shutdown_connections 2024-09-19T08:06:08.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a65365740 1 --2- 172.21.15.98:0/2658347657 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f05fcb6780 0x55f05fcb0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a65365740 1 --2- 172.21.15.98:0/2658347657 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f05fd313a0 0x55f05fd337c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a65365740 1 --2- 172.21.15.98:0/2658347657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05fca8b40 0x55f05fcafed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a3bfff640 0 client.16248 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:06:08.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a65365740 1 --2- 172.21.15.98:0/2658347657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f05fca75d0 0x55f05fcbed80 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a65365740 1 -- 172.21.15.98:0/2658347657 >> 172.21.15.98:0/2658347657 conn(0x55f05fc70310 msgr2=0x55f05fca3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:08.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a65365740 1 -- 172.21.15.98:0/2658347657 shutdown_connections 2024-09-19T08:06:08.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a65365740 1 -- 172.21.15.98:0/2658347657 wait complete. 2024-09-19T08:06:08.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.044+0000 7f2a65365740 20 client.16248 trim_cache size 0 max 16384 2024-09-19T08:06:08.057 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:06:08.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.045+0000 7f2a62f8c640 1 -- 172.21.15.98:0/1209472764 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2a480a33b0 msgr2=0x7f2a480a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:08.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.045+0000 7f2a62f8c640 1 --2- 172.21.15.98:0/1209472764 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2a480a33b0 0x7f2a480a5870 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2a440058a0 tx=0x7f2a4400b6f0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.045+0000 7f2a62f8c640 1 -- 172.21.15.98:0/1209472764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a5c106f20 msgr2=0x7f2a5c19cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:08.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.045+0000 7f2a62f8c640 1 --2- 172.21.15.98:0/1209472764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a5c106f20 0x7f2a5c19cc10 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2a54005ab0 tx=0x7f2a540042f0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.045+0000 7f2a62f8c640 1 -- 172.21.15.98:0/1209472764 shutdown_connections 2024-09-19T08:06:08.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.045+0000 7f2a62f8c640 1 --2- 172.21.15.98:0/1209472764 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2a480a33b0 0x7f2a480a5870 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.045+0000 7f2a62f8c640 1 --2- 172.21.15.98:0/1209472764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a5c106f20 0x7f2a5c19cc10 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.045+0000 7f2a62f8c640 1 --2- 172.21.15.98:0/1209472764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a5c1027a0 0x7f2a5c198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.045+0000 7f2a62f8c640 1 --2- 172.21.15.98:0/1209472764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a5c101df0 0x7f2a5c198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:08.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.045+0000 7f2a62f8c640 1 -- 172.21.15.98:0/1209472764 >> 172.21.15.98:0/1209472764 conn(0x7f2a5c0fd8f0 msgr2=0x7f2a5c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:08.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.045+0000 7f2a62f8c640 1 -- 172.21.15.98:0/1209472764 shutdown_connections 2024-09-19T08:06:08.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:08.045+0000 7f2a62f8c640 1 -- 172.21.15.98:0/1209472764 wait complete. 2024-09-19T08:06:08.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:06:09.090 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:09.172 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:08 smithi162 ceph-mon[35260]: pgmap v97: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 778 MiB/s wr, 331 op/s 2024-09-19T08:06:09.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:08 smithi139 ceph-mon[34874]: pgmap v97: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 778 MiB/s wr, 331 op/s 2024-09-19T08:06:09.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:08 smithi098 ceph-mon[31333]: pgmap v97: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 778 MiB/s wr, 331 op/s 2024-09-19T08:06:09.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.601+0000 7f5635c27640 1 -- 172.21.15.98:0/585447456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56301027a0 msgr2=0x7f56301067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:09.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.601+0000 7f5635c27640 1 --2- 172.21.15.98:0/585447456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56301027a0 0x7f56301067f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5624009900 tx=0x7f562402a410 comp rx=0 tx=0).stop 2024-09-19T08:06:09.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.601+0000 7f5635c27640 1 -- 172.21.15.98:0/585447456 shutdown_connections 2024-09-19T08:06:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.601+0000 7f5635c27640 1 --2- 172.21.15.98:0/585447456 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5630106f20 0x7f563010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.601+0000 7f5635c27640 1 --2- 172.21.15.98:0/585447456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56301027a0 0x7f56301067f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.601+0000 7f5635c27640 1 --2- 172.21.15.98:0/585447456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5630101df0 0x7f56301021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.601+0000 7f5635c27640 1 -- 172.21.15.98:0/585447456 >> 172.21.15.98:0/585447456 conn(0x7f56300fd8f0 msgr2=0x7f56300ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.601+0000 7f5635c27640 1 -- 172.21.15.98:0/585447456 shutdown_connections 2024-09-19T08:06:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.601+0000 7f5635c27640 1 -- 172.21.15.98:0/585447456 wait complete. 2024-09-19T08:06:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.602+0000 7f5635c27640 1 Processor -- start 2024-09-19T08:06:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.602+0000 7f5635c27640 1 -- start start 2024-09-19T08:06:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.602+0000 7f5635c27640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5630101df0 0x7f5630198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.602+0000 7f5635c27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56301027a0 0x7f5630198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.602+0000 7f5635c27640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5630106f20 0x7f563019ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.602+0000 7f5635c27640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5630111f80 con 0x7f5630101df0 2024-09-19T08:06:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.602+0000 7f5635c27640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5630111e00 con 0x7f5630106f20 2024-09-19T08:06:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.602+0000 7f5635c27640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5630112100 con 0x7f56301027a0 2024-09-19T08:06:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5630101df0 0x7f5630198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5630101df0 0x7f5630198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43836/0 (socket says 172.21.15.98:43836) 2024-09-19T08:06:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562f7fe640 1 -- 172.21.15.98:0/2282966925 learned_addr learned my addr 172.21.15.98:0/2282966925 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562effd640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56301027a0 0x7f5630198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562ffff640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5630106f20 0x7f563019ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562f7fe640 1 -- 172.21.15.98:0/2282966925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56301027a0 msgr2=0x7f5630198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562f7fe640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56301027a0 0x7f5630198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562f7fe640 1 -- 172.21.15.98:0/2282966925 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5630106f20 msgr2=0x7f563019ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562f7fe640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5630106f20 0x7f563019ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562f7fe640 1 -- 172.21.15.98:0/2282966925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f563019d3d0 con 0x7f5630101df0 2024-09-19T08:06:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562ffff640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5630106f20 0x7f563019ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562f7fe640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5630101df0 0x7f5630198420 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f562000d950 tx=0x7f562000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562cff9640 1 -- 172.21.15.98:0/2282966925 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5620017030 con 0x7f5630101df0 2024-09-19T08:06:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f5635c27640 1 -- 172.21.15.98:0/2282966925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f563019d6c0 con 0x7f5630101df0 2024-09-19T08:06:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562cff9640 1 -- 172.21.15.98:0/2282966925 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5620004020 con 0x7f5630101df0 2024-09-19T08:06:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f562cff9640 1 -- 172.21.15.98:0/2282966925 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56200047a0 con 0x7f5630101df0 2024-09-19T08:06:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.603+0000 7f5635c27640 1 -- 172.21.15.98:0/2282966925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56301a4fa0 con 0x7f5630101df0 2024-09-19T08:06:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.606+0000 7f562cff9640 1 -- 172.21.15.98:0/2282966925 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f56200041d0 con 0x7f5630101df0 2024-09-19T08:06:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.606+0000 7f5638000740 1 Processor -- start 2024-09-19T08:06:09.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.606+0000 7f5638000740 1 -- start start 2024-09-19T08:06:09.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f5638000740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232788b40 0x563232787090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f5638000740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632327875d0 0x563232787a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f5638000740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563232792370 0x563232794760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f5638000740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5632326ed2f0 con 0x5632327875d0 2024-09-19T08:06:09.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f5638000740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563232694cf0 con 0x563232792370 2024-09-19T08:06:09.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f5638000740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5632326e92b0 con 0x563232788b40 2024-09-19T08:06:09.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f562cff9640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f56180a33b0 0x7f56180a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f562f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632327875d0 0x563232787a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f562f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632327875d0 0x563232787a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43838/0 (socket says 172.21.15.98:43838) 2024-09-19T08:06:09.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f562f7fe640 1 -- 172.21.15.98:0/3685363207 learned_addr learned my addr 172.21.15.98:0/3685363207 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:09.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f562effd640 1 --2- 172.21.15.98:0/3685363207 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563232792370 0x563232794760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f562ffff640 1 --2- 172.21.15.98:0/3685363207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232788b40 0x563232787090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f562ffff640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f56180a33b0 0x7f56180a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f562effd640 1 -- 172.21.15.98:0/3685363207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232788b40 msgr2=0x563232787090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:09.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f562effd640 1 --2- 172.21.15.98:0/3685363207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232788b40 0x563232787090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f562effd640 1 -- 172.21.15.98:0/3685363207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632327875d0 msgr2=0x563232787a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:09.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f562effd640 1 --2- 172.21.15.98:0/3685363207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632327875d0 0x563232787a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f562effd640 1 -- 172.21.15.98:0/3685363207 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632326793a0 con 0x563232792370 2024-09-19T08:06:09.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.607+0000 7f562cff9640 1 -- 172.21.15.98:0/2282966925 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f56200cf6d0 con 0x7f5630101df0 2024-09-19T08:06:09.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.608+0000 7f562ffff640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f56180a33b0 0x7f56180a5870 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5628007310 tx=0x7f56280072a0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:09.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.608+0000 7f562effd640 1 --2- 172.21.15.98:0/3685363207 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563232792370 0x563232794760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5624005940 tx=0x7f5624031630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76a5f4a5382b2a4a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:09.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.608+0000 7f5615ffb640 1 -- 172.21.15.98:0/3685363207 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624040070 con 0x563232792370 2024-09-19T08:06:09.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.608+0000 7f5615ffb640 1 -- 172.21.15.98:0/3685363207 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5624005020 con 0x563232792370 2024-09-19T08:06:09.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.608+0000 7f5615ffb640 1 -- 172.21.15.98:0/3685363207 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624035040 con 0x563232792370 2024-09-19T08:06:09.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.609+0000 7f5638000740 1 -- 172.21.15.98:0/3685363207 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563232792370 msgr2=0x563232794760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:09.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.609+0000 7f5638000740 1 --2- 172.21.15.98:0/3685363207 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563232792370 0x563232794760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5624005940 tx=0x7f5624031630 comp rx=0 tx=0).stop 2024-09-19T08:06:09.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.609+0000 7f5638000740 1 -- 172.21.15.98:0/3685363207 shutdown_connections 2024-09-19T08:06:09.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.609+0000 7f5638000740 1 --2- 172.21.15.98:0/3685363207 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563232792370 0x563232794760 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.609+0000 7f5638000740 1 --2- 172.21.15.98:0/3685363207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632327875d0 0x563232787a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.609+0000 7f5638000740 1 --2- 172.21.15.98:0/3685363207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232788b40 0x563232787090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.609+0000 7f5638000740 1 -- 172.21.15.98:0/3685363207 >> 172.21.15.98:0/3685363207 conn(0x563232750310 msgr2=0x563232783c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:09.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.609+0000 7f5638000740 1 -- 172.21.15.98:0/3685363207 shutdown_connections 2024-09-19T08:06:09.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.609+0000 7f5638000740 1 -- 172.21.15.98:0/3685363207 wait complete. 2024-09-19T08:06:09.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.609+0000 7f5638000740 1 -- start start 2024-09-19T08:06:09.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.609+0000 7f5638000740 1 Processor -- start 2024-09-19T08:06:09.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f5638000740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5632327875d0 0x563232791c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f5638000740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x56323278ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f5638000740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232792370 0x56323278f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f5638000740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563232795010 con 0x563232788b40 2024-09-19T08:06:09.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f5638000740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5632326a8060 con 0x5632327875d0 2024-09-19T08:06:09.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f5638000740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563232795190 con 0x563232792370 2024-09-19T08:06:09.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f5638000740 10 client.0 _ll_register_callbacks cb 0x5632323fb8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:09.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f562effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x56323278ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f562effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x56323278ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43846/0 (socket says 172.21.15.98:43846) 2024-09-19T08:06:09.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f562effd640 1 -- 172.21.15.98:0/2880357069 learned_addr learned my addr 172.21.15.98:0/2880357069 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:09.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f562ffff640 1 --2- 172.21.15.98:0/2880357069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232792370 0x56323278f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f562f7fe640 1 --2- 172.21.15.98:0/2880357069 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5632327875d0 0x563232791c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f562effd640 1 -- 172.21.15.98:0/2880357069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232792370 msgr2=0x56323278f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:09.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f562effd640 1 --2- 172.21.15.98:0/2880357069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232792370 0x56323278f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f562effd640 1 -- 172.21.15.98:0/2880357069 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5632327875d0 msgr2=0x563232791c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:09.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f562effd640 1 --2- 172.21.15.98:0/2880357069 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5632327875d0 0x563232791c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f562effd640 1 -- 172.21.15.98:0/2880357069 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56323245e3c0 con 0x563232788b40 2024-09-19T08:06:09.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f562effd640 1 --2- 172.21.15.98:0/2880357069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x56323278ef80 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f562402a930 tx=0x7f5624005400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e8db54e58daadd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:09.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.610+0000 7f560effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:09.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.611+0000 7f560effd640 1 -- 172.21.15.98:0/2880357069 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624040030 con 0x563232788b40 2024-09-19T08:06:09.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.611+0000 7f560effd640 1 -- 172.21.15.98:0/2880357069 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5624004030 con 0x563232788b40 2024-09-19T08:06:09.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.611+0000 7f5638000740 1 -- 172.21.15.98:0/2880357069 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5624005020 con 0x563232788b40 2024-09-19T08:06:09.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.611+0000 7f560effd640 1 -- 172.21.15.98:0/2880357069 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5624035070 con 0x563232788b40 2024-09-19T08:06:09.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.611+0000 7f5638000740 10 client.16260 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:09.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.611+0000 7f5638000740 1 -- 172.21.15.98:0/2880357069 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5632327a0a20 con 0x563232788b40 2024-09-19T08:06:09.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.611+0000 7f560effd640 1 -- 172.21.15.98:0/2880357069 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5624005020 con 0x563232788b40 2024-09-19T08:06:09.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.611+0000 7f560effd640 1 -- 172.21.15.98:0/2880357069 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f5624003790 con 0x563232788b40 2024-09-19T08:06:09.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.611+0000 7f5638000740 10 client.16260 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:09.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.611+0000 7f5638000740 10 client.16260 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:06:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.611+0000 7f5638000740 20 client.16260 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.611+0000 7f5638000740 1 --2- 172.21.15.98:0/2880357069 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5632327a1d70 0x5632327a4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.611+0000 7f5638000740 4 client.16260 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.611+0000 7f5638000740 1 -- 172.21.15.98:0/2880357069 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632326fa860 con 0x5632327a1d70 2024-09-19T08:06:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.612+0000 7f562f7fe640 1 --2- 172.21.15.98:0/2880357069 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5632327a1d70 0x5632327a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.612+0000 7f560effd640 10 client.16260 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:06:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.612+0000 7f562f7fe640 1 --2- 172.21.15.98:0/2880357069 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5632327a1d70 0x5632327a4160 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=68c4022c15c6d86d server_cookie=174bf3f299d0a1 in_seq=0 out_seq=0 2024-09-19T08:06:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f560effd640 1 -- 172.21.15.98:0/2880357069 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5632326fa860 con 0x5632327a1d70 2024-09-19T08:06:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f560effd640 10 client.16260 handle_command_reply: tid=0 2024-09-19T08:06:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f5638000740 1 client.16260 shutdown 2024-09-19T08:06:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f5638000740 1 -- 172.21.15.98:0/2880357069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 msgr2=0x56323278ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f5638000740 1 --2- 172.21.15.98:0/2880357069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x56323278ef80 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f562402a930 tx=0x7f5624005400 comp rx=0 tx=0).stop 2024-09-19T08:06:09.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f5638000740 1 -- 172.21.15.98:0/2880357069 shutdown_connections 2024-09-19T08:06:09.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f5638000740 1 --2- 172.21.15.98:0/2880357069 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5632327a1d70 0x5632327a4160 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f5638000740 1 --2- 172.21.15.98:0/2880357069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232792370 0x56323278f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f5638000740 1 --2- 172.21.15.98:0/2880357069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x56323278ef80 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f5638000740 1 --2- 172.21.15.98:0/2880357069 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5632327875d0 0x563232791c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f5638000740 1 -- 172.21.15.98:0/2880357069 >> 172.21.15.98:0/2880357069 conn(0x563232750310 msgr2=0x563232789960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:09.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f560effd640 0 client.16260 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:06:09.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f5638000740 1 -- 172.21.15.98:0/2880357069 shutdown_connections 2024-09-19T08:06:09.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f5638000740 1 -- 172.21.15.98:0/2880357069 wait complete. 2024-09-19T08:06:09.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.613+0000 7f5638000740 20 client.16260 trim_cache size 0 max 16384 2024-09-19T08:06:09.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.626+0000 7f5638000740 1 Processor -- start 2024-09-19T08:06:09.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.626+0000 7f5638000740 1 -- start start 2024-09-19T08:06:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f5638000740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5632327875d0 0x56323279e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f5638000740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x563232790ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f5638000740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232792370 0x5632327a9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f5638000740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56301ad760 con 0x563232788b40 2024-09-19T08:06:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f5638000740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f56301af300 con 0x5632327875d0 2024-09-19T08:06:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f5638000740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f56301ade30 con 0x563232792370 2024-09-19T08:06:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f562effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x563232790ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f562f7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5632327875d0 0x56323279e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f562effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x563232790ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43852/0 (socket says 172.21.15.98:43852) 2024-09-19T08:06:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f562effd640 1 -- 172.21.15.98:0/3489348739 learned_addr learned my addr 172.21.15.98:0/3489348739 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f562ffff640 1 --2- 172.21.15.98:0/3489348739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232792370 0x5632327a9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f562f7fe640 1 -- 172.21.15.98:0/3489348739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232792370 msgr2=0x5632327a9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f562f7fe640 1 --2- 172.21.15.98:0/3489348739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232792370 0x5632327a9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f562f7fe640 1 -- 172.21.15.98:0/3489348739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 msgr2=0x563232790ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f562f7fe640 1 --2- 172.21.15.98:0/3489348739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x563232790ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f562f7fe640 1 -- 172.21.15.98:0/3489348739 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563232784730 con 0x5632327875d0 2024-09-19T08:06:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f562effd640 1 --2- 172.21.15.98:0/3489348739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x563232790ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:06:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.627+0000 7f562f7fe640 1 --2- 172.21.15.98:0/3489348739 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5632327875d0 0x56323279e2b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f56200ccf00 tx=0x7f56200d1bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d04ca0c16c89599 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.628+0000 7f56157fa640 1 -- 172.21.15.98:0/3489348739 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562009b9e0 con 0x5632327875d0 2024-09-19T08:06:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.628+0000 7f56157fa640 1 -- 172.21.15.98:0/3489348739 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f562009fd90 con 0x5632327875d0 2024-09-19T08:06:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.628+0000 7f56157fa640 1 -- 172.21.15.98:0/3489348739 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56200ba120 con 0x5632327875d0 2024-09-19T08:06:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.629+0000 7f5638000740 1 -- 172.21.15.98:0/3489348739 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5632327875d0 msgr2=0x56323279e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.629+0000 7f5638000740 1 --2- 172.21.15.98:0/3489348739 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5632327875d0 0x56323279e2b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f56200ccf00 tx=0x7f56200d1bb0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.629+0000 7f5638000740 1 -- 172.21.15.98:0/3489348739 shutdown_connections 2024-09-19T08:06:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.629+0000 7f5638000740 1 --2- 172.21.15.98:0/3489348739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563232792370 0x5632327a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.629+0000 7f5638000740 1 --2- 172.21.15.98:0/3489348739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x563232790ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.629+0000 7f5638000740 1 --2- 172.21.15.98:0/3489348739 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5632327875d0 0x56323279e2b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.629+0000 7f5638000740 1 -- 172.21.15.98:0/3489348739 >> 172.21.15.98:0/3489348739 conn(0x563232750310 msgr2=0x5632327907e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.629+0000 7f5638000740 1 -- 172.21.15.98:0/3489348739 shutdown_connections 2024-09-19T08:06:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.629+0000 7f5638000740 1 -- 172.21.15.98:0/3489348739 wait complete. 2024-09-19T08:06:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.629+0000 7f5638000740 1 -- start start 2024-09-19T08:06:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.629+0000 7f5638000740 1 Processor -- start 2024-09-19T08:06:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f5638000740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632327875d0 0x56323279a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f5638000740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x56323279ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f5638000740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563232792370 0x563232806260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f5638000740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56301aec00 con 0x563232788b40 2024-09-19T08:06:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f5638000740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f56301ae530 con 0x563232792370 2024-09-19T08:06:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f5638000740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f56301ab740 con 0x5632327875d0 2024-09-19T08:06:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f5638000740 10 client.0 _ll_register_callbacks cb 0x7f562009b7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f562f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632327875d0 0x56323279a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f562ffff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563232792370 0x563232806260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f562ffff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563232792370 0x563232806260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:47870/0 (socket says 172.21.15.98:47870) 2024-09-19T08:06:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f562f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632327875d0 0x56323279a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:36164/0 (socket says 172.21.15.98:36164) 2024-09-19T08:06:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f562ffff640 1 -- 172.21.15.98:0/3970371000 learned_addr learned my addr 172.21.15.98:0/3970371000 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f562effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x56323279ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f562f7fe640 1 -- 172.21.15.98:0/3970371000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563232792370 msgr2=0x563232806260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f562f7fe640 1 --2- 172.21.15.98:0/3970371000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563232792370 0x563232806260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f562f7fe640 1 -- 172.21.15.98:0/3970371000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 msgr2=0x56323279ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f562f7fe640 1 --2- 172.21.15.98:0/3970371000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x56323279ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f562f7fe640 1 -- 172.21.15.98:0/3970371000 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563232806a60 con 0x5632327875d0 2024-09-19T08:06:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.630+0000 7f562ffff640 1 --2- 172.21.15.98:0/3970371000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563232792370 0x563232806260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:06:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f562f7fe640 1 --2- 172.21.15.98:0/3970371000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632327875d0 0x56323279a750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5620099f30 tx=0x7f5620097d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b885fd0fe756c5f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f560effd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-09-19T08:06:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f560effd640 1 -- 172.21.15.98:0/3970371000 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562009b9e0 con 0x5632327875d0 2024-09-19T08:06:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f560effd640 1 -- 172.21.15.98:0/3970371000 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f56200cd0d0 con 0x5632327875d0 2024-09-19T08:06:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f5638000740 1 -- 172.21.15.98:0/3970371000 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f562009fd90 con 0x5632327875d0 2024-09-19T08:06:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f560effd640 1 -- 172.21.15.98:0/3970371000 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f56200cd2d0 con 0x5632327875d0 2024-09-19T08:06:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f5638000740 10 client.25495 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f5638000740 1 -- 172.21.15.98:0/3970371000 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563232815ee0 con 0x5632327875d0 2024-09-19T08:06:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f560effd640 1 -- 172.21.15.98:0/3970371000 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f562009fd90 con 0x5632327875d0 2024-09-19T08:06:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f560effd640 1 -- 172.21.15.98:0/3970371000 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f5620084ce0 con 0x5632327875d0 2024-09-19T08:06:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f5638000740 10 client.25495 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f5638000740 10 client.25495 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:06:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f5638000740 20 client.25495 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f5638000740 1 --2- 172.21.15.98:0/3970371000 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x563232816720 0x563232818b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:09.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f5638000740 4 client.25495 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:06:09.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.631+0000 7f5638000740 1 -- 172.21.15.98:0/3970371000 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5632326f4d70 con 0x563232816720 2024-09-19T08:06:09.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.632+0000 7f562effd640 1 --2- 172.21.15.98:0/3970371000 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x563232816720 0x563232818b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:09.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.632+0000 7f562effd640 1 --2- 172.21.15.98:0/3970371000 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x563232816720 0x563232818b40 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=541508d7119beffe server_cookie=8644ec3494cd79f2 in_seq=0 out_seq=0 2024-09-19T08:06:09.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.632+0000 7f560effd640 10 client.25495 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:06:09.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f560effd640 1 -- 172.21.15.98:0/3970371000 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5632326f4d70 con 0x563232816720 2024-09-19T08:06:09.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f560effd640 10 client.25495 handle_command_reply: tid=0 2024-09-19T08:06:09.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f5638000740 1 client.25495 shutdown 2024-09-19T08:06:09.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f5638000740 1 -- 172.21.15.98:0/3970371000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632327875d0 msgr2=0x56323279a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:09.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f5638000740 1 --2- 172.21.15.98:0/3970371000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632327875d0 0x56323279a750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5620099f30 tx=0x7f5620097d50 comp rx=0 tx=0).stop 2024-09-19T08:06:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f5638000740 1 -- 172.21.15.98:0/3970371000 shutdown_connections 2024-09-19T08:06:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f5638000740 1 --2- 172.21.15.98:0/3970371000 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x563232816720 0x563232818b40 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f5638000740 1 --2- 172.21.15.98:0/3970371000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563232792370 0x563232806260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f5638000740 1 --2- 172.21.15.98:0/3970371000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563232788b40 0x56323279ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f5638000740 1 --2- 172.21.15.98:0/3970371000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632327875d0 0x56323279a750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f560effd640 0 client.25495 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:06:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f5638000740 1 -- 172.21.15.98:0/3970371000 >> 172.21.15.98:0/3970371000 conn(0x563232750310 msgr2=0x563232791000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f5638000740 1 -- 172.21.15.98:0/3970371000 shutdown_connections 2024-09-19T08:06:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f5638000740 1 -- 172.21.15.98:0/3970371000 wait complete. 2024-09-19T08:06:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.633+0000 7f5638000740 20 client.25495 trim_cache size 0 max 16384 2024-09-19T08:06:09.650 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:06:09.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.634+0000 7f5635c27640 1 -- 172.21.15.98:0/2282966925 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f56180a33b0 msgr2=0x7f56180a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:09.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.634+0000 7f5635c27640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f56180a33b0 0x7f56180a5870 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5628007310 tx=0x7f56280072a0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.634+0000 7f5635c27640 1 -- 172.21.15.98:0/2282966925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5630101df0 msgr2=0x7f5630198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:09.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.634+0000 7f5635c27640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5630101df0 0x7f5630198420 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f562000d950 tx=0x7f562000de10 comp rx=0 tx=0).stop 2024-09-19T08:06:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.634+0000 7f5635c27640 1 -- 172.21.15.98:0/2282966925 shutdown_connections 2024-09-19T08:06:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.634+0000 7f5635c27640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f56180a33b0 0x7f56180a5870 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.634+0000 7f5635c27640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5630106f20 0x7f563019ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.634+0000 7f5635c27640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56301027a0 0x7f5630198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.634+0000 7f5635c27640 1 --2- 172.21.15.98:0/2282966925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5630101df0 0x7f5630198420 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.634+0000 7f5635c27640 1 -- 172.21.15.98:0/2282966925 >> 172.21.15.98:0/2282966925 conn(0x7f56300fd8f0 msgr2=0x7f56300feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.634+0000 7f5635c27640 1 -- 172.21.15.98:0/2282966925 shutdown_connections 2024-09-19T08:06:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:09.634+0000 7f5635c27640 1 -- 172.21.15.98:0/2282966925 wait complete. 2024-09-19T08:06:10.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:06:10.447 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:06:10.447 INFO:tasks.workunit.client.0.smithi098.stdout: Children see throughput for 1 readers = 7836989.50 kB/sec 2024-09-19T08:06:10.447 INFO:tasks.workunit.client.0.smithi098.stdout: Parent sees throughput for 1 readers = 7835603.48 kB/sec 2024-09-19T08:06:10.447 INFO:tasks.workunit.client.0.smithi098.stdout: Min throughput per process = 7836989.50 kB/sec 2024-09-19T08:06:10.447 INFO:tasks.workunit.client.0.smithi098.stdout: Max throughput per process = 7836989.50 kB/sec 2024-09-19T08:06:10.447 INFO:tasks.workunit.client.0.smithi098.stdout: Avg throughput per process = 7836989.50 kB/sec 2024-09-19T08:06:10.448 INFO:tasks.workunit.client.0.smithi098.stdout: Min xfer = 10485760.00 kB 2024-09-19T08:06:10.618 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:11.022 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:10 smithi098 ceph-mon[31333]: pgmap v98: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 693 MiB/s wr, 261 op/s 2024-09-19T08:06:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.104+0000 7f21bd603640 1 -- 172.21.15.98:0/2801621240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b8100a70 msgr2=0x7f21b8100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.104+0000 7f21bd603640 1 --2- 172.21.15.98:0/2801621240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b8100a70 0x7f21b8100ed0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f21ac009930 tx=0x7f21ac02a420 comp rx=0 tx=0).stop 2024-09-19T08:06:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.104+0000 7f21bd603640 1 -- 172.21.15.98:0/2801621240 shutdown_connections 2024-09-19T08:06:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.104+0000 7f21bd603640 1 --2- 172.21.15.98:0/2801621240 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21b8101410 0x7f21b810d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.104+0000 7f21bd603640 1 --2- 172.21.15.98:0/2801621240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b8100a70 0x7f21b8100ed0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.104+0000 7f21bd603640 1 --2- 172.21.15.98:0/2801621240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21b8106a70 0x7f21b8106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.104+0000 7f21bd603640 1 -- 172.21.15.98:0/2801621240 >> 172.21.15.98:0/2801621240 conn(0x7f21b80fc790 msgr2=0x7f21b80febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.104+0000 7f21bd603640 1 -- 172.21.15.98:0/2801621240 shutdown_connections 2024-09-19T08:06:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.104+0000 7f21bd603640 1 -- 172.21.15.98:0/2801621240 wait complete. 2024-09-19T08:06:11.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21bd603640 1 Processor -- start 2024-09-19T08:06:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21bd603640 1 -- start start 2024-09-19T08:06:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21bd603640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21b8100a70 0x7f21b81992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21bd603640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b8101410 0x7f21b8199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21bd603640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21b8106a70 0x7f21b81a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21bd603640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21b8110e70 con 0x7f21b8101410 2024-09-19T08:06:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21bd603640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f21b8110cf0 con 0x7f21b8100a70 2024-09-19T08:06:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21bd603640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f21b8110ff0 con 0x7f21b8106a70 2024-09-19T08:06:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b8101410 0x7f21b8199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b8101410 0x7f21b8199820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43876/0 (socket says 172.21.15.98:43876) 2024-09-19T08:06:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21b67fc640 1 -- 172.21.15.98:0/874004305 learned_addr learned my addr 172.21.15.98:0/874004305 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21b77fe640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21b8106a70 0x7f21b81a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21b6ffd640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21b8100a70 0x7f21b81992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21b67fc640 1 -- 172.21.15.98:0/874004305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21b8106a70 msgr2=0x7f21b81a4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21b67fc640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21b8106a70 0x7f21b81a4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21b67fc640 1 -- 172.21.15.98:0/874004305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21b8100a70 msgr2=0x7f21b81992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21b67fc640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21b8100a70 0x7f21b81992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21b67fc640 1 -- 172.21.15.98:0/874004305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b81a48a0 con 0x7f21b8101410 2024-09-19T08:06:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.105+0000 7f21b6ffd640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21b8100a70 0x7f21b81992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.106+0000 7f21b67fc640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b8101410 0x7f21b8199820 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f21ac009230 tx=0x7f21ac004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.106+0000 7f219ffff640 1 -- 172.21.15.98:0/874004305 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac03f030 con 0x7f21b8101410 2024-09-19T08:06:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.106+0000 7f219ffff640 1 -- 172.21.15.98:0/874004305 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21ac003ad0 con 0x7f21b8101410 2024-09-19T08:06:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.106+0000 7f21bd603640 1 -- 172.21.15.98:0/874004305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b81a4a70 con 0x7f21b8101410 2024-09-19T08:06:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.106+0000 7f219ffff640 1 -- 172.21.15.98:0/874004305 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac035070 con 0x7f21b8101410 2024-09-19T08:06:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.106+0000 7f21bd603640 1 -- 172.21.15.98:0/874004305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b81a4ec0 con 0x7f21b8101410 2024-09-19T08:06:11.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.109+0000 7f219ffff640 1 -- 172.21.15.98:0/874004305 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f21ac042030 con 0x7f21b8101410 2024-09-19T08:06:11.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.109+0000 7f21bf9dc740 1 Processor -- start 2024-09-19T08:06:11.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.109+0000 7f21bf9dc740 1 -- start start 2024-09-19T08:06:11.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.109+0000 7f21bf9dc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1eb40 0x55fe30d1d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.109+0000 7f21bf9dc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1d5d0 0x55fe30d1da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.109+0000 7f21bf9dc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 0x55fe30d2a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.109+0000 7f21bf9dc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fe30c832f0 con 0x55fe30d1d5d0 2024-09-19T08:06:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.109+0000 7f21bf9dc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fe30c2acf0 con 0x55fe30d1eb40 2024-09-19T08:06:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.109+0000 7f21bf9dc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fe30c7f2b0 con 0x55fe30d28370 2024-09-19T08:06:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.109+0000 7f21b6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1d5d0 0x55fe30d1da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.109+0000 7f21b6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1d5d0 0x55fe30d1da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43892/0 (socket says 172.21.15.98:43892) 2024-09-19T08:06:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.109+0000 7f21b6ffd640 1 -- 172.21.15.98:0/20276211 learned_addr learned my addr 172.21.15.98:0/20276211 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.109+0000 7f219ffff640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f21a00a33b0 0x7f21a00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f21b77fe640 1 --2- 172.21.15.98:0/20276211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1eb40 0x55fe30d1d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f21b67fc640 1 --2- 172.21.15.98:0/20276211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 0x55fe30d2a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f21b77fe640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f21a00a33b0 0x7f21a00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f21b6ffd640 1 -- 172.21.15.98:0/20276211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 msgr2=0x55fe30d2a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f21b6ffd640 1 --2- 172.21.15.98:0/20276211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 0x55fe30d2a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f21b6ffd640 1 -- 172.21.15.98:0/20276211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1eb40 msgr2=0x55fe30d1d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f21b6ffd640 1 --2- 172.21.15.98:0/20276211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1eb40 0x55fe30d1d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f21b6ffd640 1 -- 172.21.15.98:0/20276211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe30c0f3a0 con 0x55fe30d1d5d0 2024-09-19T08:06:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f21b77fe640 1 --2- 172.21.15.98:0/20276211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1eb40 0x55fe30d1d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f219ffff640 1 -- 172.21.15.98:0/874004305 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f21ac0f0210 con 0x7f21b8101410 2024-09-19T08:06:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f21b6ffd640 1 --2- 172.21.15.98:0/20276211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1d5d0 0x55fe30d1da10 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f21a800baf0 tx=0x7f21a800d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4acd0ae08c42cd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f21b77fe640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f21a00a33b0 0x7f21a00a5870 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f21b0005fd0 tx=0x7f21b0009600 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f219d7fa640 1 -- 172.21.15.98:0/20276211 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a8017420 con 0x55fe30d1d5d0 2024-09-19T08:06:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f219d7fa640 1 -- 172.21.15.98:0/20276211 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21a8017aa0 con 0x55fe30d1d5d0 2024-09-19T08:06:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.110+0000 7f219d7fa640 1 -- 172.21.15.98:0/20276211 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a800fb00 con 0x55fe30d1d5d0 2024-09-19T08:06:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.111+0000 7f21bf9dc740 1 -- 172.21.15.98:0/20276211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1d5d0 msgr2=0x55fe30d1da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.111+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/20276211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1d5d0 0x55fe30d1da10 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f21a800baf0 tx=0x7f21a800d340 comp rx=0 tx=0).stop 2024-09-19T08:06:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.111+0000 7f21bf9dc740 1 -- 172.21.15.98:0/20276211 shutdown_connections 2024-09-19T08:06:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.111+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/20276211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 0x55fe30d2a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.111+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/20276211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1d5d0 0x55fe30d1da10 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.111+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/20276211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1eb40 0x55fe30d1d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.111+0000 7f21bf9dc740 1 -- 172.21.15.98:0/20276211 >> 172.21.15.98:0/20276211 conn(0x55fe30ce6310 msgr2=0x55fe30d19c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.111+0000 7f21bf9dc740 1 -- 172.21.15.98:0/20276211 shutdown_connections 2024-09-19T08:06:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.111+0000 7f21bf9dc740 1 -- 172.21.15.98:0/20276211 wait complete. 2024-09-19T08:06:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.111+0000 7f21bf9dc740 1 -- start start 2024-09-19T08:06:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.111+0000 7f21bf9dc740 1 Processor -- start 2024-09-19T08:06:11.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21bf9dc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d1d5d0 0x55fe30d27cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21bf9dc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1eb40 0x55fe30d24f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21bf9dc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d28370 0x55fe30d254d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21bf9dc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fe30d2b070 con 0x55fe30d1eb40 2024-09-19T08:06:11.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21bf9dc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fe30c3e060 con 0x55fe30d28370 2024-09-19T08:06:11.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21bf9dc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fe30d2b1f0 con 0x55fe30d1d5d0 2024-09-19T08:06:11.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21bf9dc740 10 client.0 _ll_register_callbacks cb 0x55fe309918c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1eb40 0x55fe30d24f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1eb40 0x55fe30d24f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43900/0 (socket says 172.21.15.98:43900) 2024-09-19T08:06:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21b67fc640 1 -- 172.21.15.98:0/1271275667 learned_addr learned my addr 172.21.15.98:0/1271275667 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21b6ffd640 1 --2- 172.21.15.98:0/1271275667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d1d5d0 0x55fe30d27cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21b77fe640 1 --2- 172.21.15.98:0/1271275667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d28370 0x55fe30d254d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21b67fc640 1 -- 172.21.15.98:0/1271275667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d1d5d0 msgr2=0x55fe30d27cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21b67fc640 1 --2- 172.21.15.98:0/1271275667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d1d5d0 0x55fe30d27cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21b67fc640 1 -- 172.21.15.98:0/1271275667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d28370 msgr2=0x55fe30d254d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21b67fc640 1 --2- 172.21.15.98:0/1271275667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d28370 0x55fe30d254d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21b67fc640 1 -- 172.21.15.98:0/1271275667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe309f43c0 con 0x55fe30d1eb40 2024-09-19T08:06:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21b67fc640 1 --2- 172.21.15.98:0/1271275667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1eb40 0x55fe30d24f70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f21ac0edf90 tx=0x7f21ac08b210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea774766d01a508 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f219aff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f219aff5640 1 -- 172.21.15.98:0/1271275667 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac049040 con 0x55fe30d1eb40 2024-09-19T08:06:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f219aff5640 1 -- 172.21.15.98:0/1271275667 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f21ac0edc10 con 0x55fe30d1eb40 2024-09-19T08:06:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.112+0000 7f21bf9dc740 1 -- 172.21.15.98:0/1271275667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21a8017aa0 con 0x55fe30d1eb40 2024-09-19T08:06:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.113+0000 7f219aff5640 1 -- 172.21.15.98:0/1271275667 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f21ac0bbe80 con 0x55fe30d1eb40 2024-09-19T08:06:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.113+0000 7f21bf9dc740 10 client.16278 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.113+0000 7f21bf9dc740 1 -- 172.21.15.98:0/1271275667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fe30d36a80 con 0x55fe30d1eb40 2024-09-19T08:06:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.113+0000 7f219aff5640 1 -- 172.21.15.98:0/1271275667 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21a8017aa0 con 0x55fe30d1eb40 2024-09-19T08:06:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.113+0000 7f219aff5640 1 -- 172.21.15.98:0/1271275667 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f21ac0b8b90 con 0x55fe30d1eb40 2024-09-19T08:06:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.113+0000 7f21bf9dc740 10 client.16278 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.113+0000 7f21bf9dc740 10 client.16278 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:06:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.113+0000 7f21bf9dc740 20 client.16278 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.113+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/1271275667 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55fe30d37dd0 0x55fe30d3a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.113+0000 7f21bf9dc740 4 client.16278 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.113+0000 7f21bf9dc740 1 -- 172.21.15.98:0/1271275667 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe30c90860 con 0x55fe30d37dd0 2024-09-19T08:06:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.113+0000 7f21b6ffd640 1 --2- 172.21.15.98:0/1271275667 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55fe30d37dd0 0x55fe30d3a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.114+0000 7f21b6ffd640 1 --2- 172.21.15.98:0/1271275667 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55fe30d37dd0 0x55fe30d3a1e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d312c1421c540c13 server_cookie=7dbd3944ad02a62b in_seq=0 out_seq=0 2024-09-19T08:06:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.114+0000 7f219aff5640 10 client.16278 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:06:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.114+0000 7f219aff5640 1 -- 172.21.15.98:0/1271275667 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fe30c90860 con 0x55fe30d37dd0 2024-09-19T08:06:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.114+0000 7f219aff5640 10 client.16278 handle_command_reply: tid=0 2024-09-19T08:06:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.114+0000 7f21bf9dc740 1 client.16278 shutdown 2024-09-19T08:06:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.115+0000 7f21bf9dc740 1 -- 172.21.15.98:0/1271275667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1eb40 msgr2=0x55fe30d24f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.115+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/1271275667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1eb40 0x55fe30d24f70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f21ac0edf90 tx=0x7f21ac08b210 comp rx=0 tx=0).stop 2024-09-19T08:06:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.115+0000 7f21bf9dc740 1 -- 172.21.15.98:0/1271275667 shutdown_connections 2024-09-19T08:06:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.115+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/1271275667 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55fe30d37dd0 0x55fe30d3a1e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.115+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/1271275667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d28370 0x55fe30d254d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.115+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/1271275667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1eb40 0x55fe30d24f70 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.115+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/1271275667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d1d5d0 0x55fe30d27cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.115+0000 7f21bf9dc740 1 -- 172.21.15.98:0/1271275667 >> 172.21.15.98:0/1271275667 conn(0x55fe30ce6310 msgr2=0x55fe30d1f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.115+0000 7f219aff5640 0 client.16278 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:06:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.115+0000 7f21bf9dc740 1 -- 172.21.15.98:0/1271275667 shutdown_connections 2024-09-19T08:06:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.115+0000 7f21bf9dc740 1 -- 172.21.15.98:0/1271275667 wait complete. 2024-09-19T08:06:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.115+0000 7f21bf9dc740 20 client.16278 trim_cache size 0 max 16384 2024-09-19T08:06:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.127+0000 7f21bf9dc740 1 Processor -- start 2024-09-19T08:06:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.127+0000 7f21bf9dc740 1 -- start start 2024-09-19T08:06:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21bf9dc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1d5d0 0x55fe30d19e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21bf9dc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1eb40 0x55fe30d34310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21bf9dc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 0x55fe30d24830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21bf9dc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81ad820 con 0x55fe30d1eb40 2024-09-19T08:06:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21bf9dc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81af3c0 con 0x55fe30d1d5d0 2024-09-19T08:06:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21bf9dc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81adef0 con 0x55fe30d28370 2024-09-19T08:06:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1eb40 0x55fe30d34310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1eb40 0x55fe30d34310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43912/0 (socket says 172.21.15.98:43912) 2024-09-19T08:06:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21b67fc640 1 -- 172.21.15.98:0/968579821 learned_addr learned my addr 172.21.15.98:0/968579821 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21b6ffd640 1 --2- 172.21.15.98:0/968579821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1d5d0 0x55fe30d19e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21b77fe640 1 --2- 172.21.15.98:0/968579821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 0x55fe30d24830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21b67fc640 1 -- 172.21.15.98:0/968579821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 msgr2=0x55fe30d24830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21b67fc640 1 --2- 172.21.15.98:0/968579821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 0x55fe30d24830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21b67fc640 1 -- 172.21.15.98:0/968579821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1d5d0 msgr2=0x55fe30d19e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21b67fc640 1 --2- 172.21.15.98:0/968579821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1d5d0 0x55fe30d19e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21b67fc640 1 -- 172.21.15.98:0/968579821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe30d23f20 con 0x55fe30d1eb40 2024-09-19T08:06:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f21b67fc640 1 --2- 172.21.15.98:0/968579821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1eb40 0x55fe30d34310 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f21ac0eda40 tx=0x7f21ac005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860d72e3a1aff54f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f219cff9640 1 -- 172.21.15.98:0/968579821 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac049050 con 0x55fe30d1eb40 2024-09-19T08:06:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f219cff9640 1 -- 172.21.15.98:0/968579821 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21ac08b040 con 0x55fe30d1eb40 2024-09-19T08:06:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.128+0000 7f219cff9640 1 -- 172.21.15.98:0/968579821 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac0b8b90 con 0x55fe30d1eb40 2024-09-19T08:06:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.129+0000 7f21bf9dc740 1 -- 172.21.15.98:0/968579821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1eb40 msgr2=0x55fe30d34310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.129+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/968579821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1eb40 0x55fe30d34310 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f21ac0eda40 tx=0x7f21ac005e50 comp rx=0 tx=0).stop 2024-09-19T08:06:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.129+0000 7f21bf9dc740 1 -- 172.21.15.98:0/968579821 shutdown_connections 2024-09-19T08:06:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.129+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/968579821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 0x55fe30d24830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.129+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/968579821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1eb40 0x55fe30d34310 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.129+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/968579821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1d5d0 0x55fe30d19e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.129+0000 7f21bf9dc740 1 -- 172.21.15.98:0/968579821 >> 172.21.15.98:0/968579821 conn(0x55fe30ce6310 msgr2=0x55fe30d19700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.129+0000 7f21bf9dc740 1 -- 172.21.15.98:0/968579821 shutdown_connections 2024-09-19T08:06:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.129+0000 7f21bf9dc740 1 -- 172.21.15.98:0/968579821 wait complete. 2024-09-19T08:06:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.129+0000 7f21bf9dc740 1 -- start start 2024-09-19T08:06:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.129+0000 7f21bf9dc740 1 Processor -- start 2024-09-19T08:06:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21bf9dc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1d5d0 0x55fe30d30c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21bf9dc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1eb40 0x55fe30d2bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21bf9dc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 0x55fe30d2c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21bf9dc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81aecc0 con 0x55fe30d1d5d0 2024-09-19T08:06:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21bf9dc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81ae5f0 con 0x55fe30d1eb40 2024-09-19T08:06:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21bf9dc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81ab800 con 0x55fe30d28370 2024-09-19T08:06:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21bf9dc740 10 client.0 _ll_register_callbacks cb 0x7f21ac0b8830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21b6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1d5d0 0x55fe30d30c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21b6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1d5d0 0x55fe30d30c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43924/0 (socket says 172.21.15.98:43924) 2024-09-19T08:06:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21b6ffd640 1 -- 172.21.15.98:0/2592231734 learned_addr learned my addr 172.21.15.98:0/2592231734 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21b67fc640 1 --2- 172.21.15.98:0/2592231734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1eb40 0x55fe30d2bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21b77fe640 1 --2- 172.21.15.98:0/2592231734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 0x55fe30d2c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21b6ffd640 1 -- 172.21.15.98:0/2592231734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 msgr2=0x55fe30d2c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21b6ffd640 1 --2- 172.21.15.98:0/2592231734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 0x55fe30d2c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21b6ffd640 1 -- 172.21.15.98:0/2592231734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1eb40 msgr2=0x55fe30d2bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21b6ffd640 1 --2- 172.21.15.98:0/2592231734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1eb40 0x55fe30d2bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21b6ffd640 1 -- 172.21.15.98:0/2592231734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe30d2cc80 con 0x55fe30d1d5d0 2024-09-19T08:06:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21b6ffd640 1 --2- 172.21.15.98:0/2592231734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1d5d0 0x55fe30d30c10 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f21a800bac0 tx=0x7f21a800f780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f2b3bf782315a71 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f219aff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f219aff5640 1 -- 172.21.15.98:0/2592231734 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a800fa00 con 0x55fe30d1d5d0 2024-09-19T08:06:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f219aff5640 1 -- 172.21.15.98:0/2592231734 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f21a800fba0 con 0x55fe30d1d5d0 2024-09-19T08:06:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.130+0000 7f21bf9dc740 1 -- 172.21.15.98:0/2592231734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21ac08b040 con 0x55fe30d1d5d0 2024-09-19T08:06:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.131+0000 7f219aff5640 1 -- 172.21.15.98:0/2592231734 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f21a8039050 con 0x55fe30d1d5d0 2024-09-19T08:06:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.131+0000 7f21bf9dc740 10 client.16284 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.131+0000 7f21bf9dc740 1 -- 172.21.15.98:0/2592231734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fe30da2ab0 con 0x55fe30d1d5d0 2024-09-19T08:06:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.131+0000 7f219aff5640 1 -- 172.21.15.98:0/2592231734 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21ac08b040 con 0x55fe30d1d5d0 2024-09-19T08:06:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.131+0000 7f219aff5640 1 -- 172.21.15.98:0/2592231734 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f21a8022410 con 0x55fe30d1d5d0 2024-09-19T08:06:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.131+0000 7f21bf9dc740 10 client.16284 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.131+0000 7f21bf9dc740 10 client.16284 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:06:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.131+0000 7f21bf9dc740 20 client.16284 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.131+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/2592231734 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55fe30da2f00 0x55fe30da52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.131+0000 7f21bf9dc740 4 client.16284 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:06:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.131+0000 7f21bf9dc740 1 -- 172.21.15.98:0/2592231734 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe30c8ad70 con 0x55fe30da2f00 2024-09-19T08:06:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.131+0000 7f21b67fc640 1 --2- 172.21.15.98:0/2592231734 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55fe30da2f00 0x55fe30da52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f21b67fc640 1 --2- 172.21.15.98:0/2592231734 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55fe30da2f00 0x55fe30da52f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=411e995a7b52ab68 server_cookie=d90070de73e5eae5 in_seq=0 out_seq=0 2024-09-19T08:06:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f219aff5640 10 client.16284 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:06:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f219aff5640 1 -- 172.21.15.98:0/2592231734 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe30c8ad70 con 0x55fe30da2f00 2024-09-19T08:06:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f219aff5640 10 client.16284 handle_command_reply: tid=0 2024-09-19T08:06:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f21bf9dc740 1 client.16284 shutdown 2024-09-19T08:06:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f21bf9dc740 1 -- 172.21.15.98:0/2592231734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1d5d0 msgr2=0x55fe30d30c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/2592231734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1d5d0 0x55fe30d30c10 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f21a800bac0 tx=0x7f21a800f780 comp rx=0 tx=0).stop 2024-09-19T08:06:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f21bf9dc740 1 -- 172.21.15.98:0/2592231734 shutdown_connections 2024-09-19T08:06:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/2592231734 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55fe30da2f00 0x55fe30da52f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/2592231734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe30d28370 0x55fe30d2c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/2592231734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe30d1eb40 0x55fe30d2bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f219aff5640 0 client.16284 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:06:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f21bf9dc740 1 --2- 172.21.15.98:0/2592231734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fe30d1d5d0 0x55fe30d30c10 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f21bf9dc740 1 -- 172.21.15.98:0/2592231734 >> 172.21.15.98:0/2592231734 conn(0x55fe30ce6310 msgr2=0x55fe30d267d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f21bf9dc740 1 -- 172.21.15.98:0/2592231734 shutdown_connections 2024-09-19T08:06:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f21bf9dc740 1 -- 172.21.15.98:0/2592231734 wait complete. 2024-09-19T08:06:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.132+0000 7f21bf9dc740 20 client.16284 trim_cache size 0 max 16384 2024-09-19T08:06:11.140 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:06:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.133+0000 7f21bd603640 1 -- 172.21.15.98:0/874004305 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f21a00a33b0 msgr2=0x7f21a00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.133+0000 7f21bd603640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f21a00a33b0 0x7f21a00a5870 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f21b0005fd0 tx=0x7f21b0009600 comp rx=0 tx=0).stop 2024-09-19T08:06:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.133+0000 7f21bd603640 1 -- 172.21.15.98:0/874004305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b8101410 msgr2=0x7f21b8199820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.133+0000 7f21bd603640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b8101410 0x7f21b8199820 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f21ac009230 tx=0x7f21ac004270 comp rx=0 tx=0).stop 2024-09-19T08:06:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.133+0000 7f21bd603640 1 -- 172.21.15.98:0/874004305 shutdown_connections 2024-09-19T08:06:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.133+0000 7f21bd603640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f21a00a33b0 0x7f21a00a5870 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.133+0000 7f21bd603640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21b8106a70 0x7f21b81a4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.133+0000 7f21bd603640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b8101410 0x7f21b8199820 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.133+0000 7f21bd603640 1 --2- 172.21.15.98:0/874004305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21b8100a70 0x7f21b81992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.133+0000 7f21bd603640 1 -- 172.21.15.98:0/874004305 >> 172.21.15.98:0/874004305 conn(0x7f21b80fc790 msgr2=0x7f21b80fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.133+0000 7f21bd603640 1 -- 172.21.15.98:0/874004305 shutdown_connections 2024-09-19T08:06:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:11.133+0000 7f21bd603640 1 -- 172.21.15.98:0/874004305 wait complete. 2024-09-19T08:06:11.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:10 smithi139 ceph-mon[34874]: pgmap v98: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 693 MiB/s wr, 261 op/s 2024-09-19T08:06:11.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:10 smithi162 ceph-mon[35260]: pgmap v98: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 693 MiB/s wr, 261 op/s 2024-09-19T08:06:11.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:06:12.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:11 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:06:12.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:11 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:06:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:12.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:11 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:06:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.747+0000 7f53ad2d3640 1 -- 172.21.15.98:0/1350820768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8106a80 msgr2=0x7f53a8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:12.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.747+0000 7f53ad2d3640 1 --2- 172.21.15.98:0/1350820768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8106a80 0x7f53a8106e60 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f5398009930 tx=0x7f539802a420 comp rx=0 tx=0).stop 2024-09-19T08:06:12.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.747+0000 7f53ad2d3640 1 -- 172.21.15.98:0/1350820768 shutdown_connections 2024-09-19T08:06:12.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.747+0000 7f53ad2d3640 1 --2- 172.21.15.98:0/1350820768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53a8101420 0x7f53a810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.747+0000 7f53ad2d3640 1 --2- 172.21.15.98:0/1350820768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53a8100a80 0x7f53a8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.747+0000 7f53ad2d3640 1 --2- 172.21.15.98:0/1350820768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8106a80 0x7f53a8106e60 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.747+0000 7f53ad2d3640 1 -- 172.21.15.98:0/1350820768 >> 172.21.15.98:0/1350820768 conn(0x7f53a80fc7a0 msgr2=0x7f53a80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:12.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.747+0000 7f53ad2d3640 1 -- 172.21.15.98:0/1350820768 shutdown_connections 2024-09-19T08:06:12.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.747+0000 7f53ad2d3640 1 -- 172.21.15.98:0/1350820768 wait complete. 2024-09-19T08:06:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.748+0000 7f53ad2d3640 1 Processor -- start 2024-09-19T08:06:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.748+0000 7f53ad2d3640 1 -- start start 2024-09-19T08:06:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.748+0000 7f53ad2d3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53a8100a80 0x7f53a8198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.748+0000 7f53ad2d3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53a8101420 0x7f53a8198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.748+0000 7f53ad2d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8106a80 0x7f53a819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.748+0000 7f53ad2d3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53a8110e80 con 0x7f53a8106a80 2024-09-19T08:06:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.748+0000 7f53ad2d3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53a8110d00 con 0x7f53a8101420 2024-09-19T08:06:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.748+0000 7f53ad2d3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f53a8111000 con 0x7f53a8100a80 2024-09-19T08:06:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f53a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8106a80 0x7f53a819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f53a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8106a80 0x7f53a819cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43950/0 (socket says 172.21.15.98:43950) 2024-09-19T08:06:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f53a77fe640 1 -- 172.21.15.98:0/260214008 learned_addr learned my addr 172.21.15.98:0/260214008 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:12.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f53a6ffd640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53a8100a80 0x7f53a8198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f53a67fc640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53a8101420 0x7f53a8198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f53a77fe640 1 -- 172.21.15.98:0/260214008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53a8100a80 msgr2=0x7f53a8198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:12.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f53a77fe640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53a8100a80 0x7f53a8198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f53a77fe640 1 -- 172.21.15.98:0/260214008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53a8101420 msgr2=0x7f53a8198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:12.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f53a77fe640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53a8101420 0x7f53a8198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f53a77fe640 1 -- 172.21.15.98:0/260214008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a819d3c0 con 0x7f53a8106a80 2024-09-19T08:06:12.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f53a67fc640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53a8101420 0x7f53a8198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:12.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f53a77fe640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8106a80 0x7f53a819cce0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f53a000eb10 tx=0x7f53a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:12.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f539ffff640 1 -- 172.21.15.98:0/260214008 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0004b20 con 0x7f53a8106a80 2024-09-19T08:06:12.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f53ad2d3640 1 -- 172.21.15.98:0/260214008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a819d6b0 con 0x7f53a8106a80 2024-09-19T08:06:12.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f539ffff640 1 -- 172.21.15.98:0/260214008 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a0004cc0 con 0x7f53a8106a80 2024-09-19T08:06:12.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f539ffff640 1 -- 172.21.15.98:0/260214008 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0010760 con 0x7f53a8106a80 2024-09-19T08:06:12.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.749+0000 7f53ad2d3640 1 -- 172.21.15.98:0/260214008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a81a4f90 con 0x7f53a8106a80 2024-09-19T08:06:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.752+0000 7f53af6ac740 1 Processor -- start 2024-09-19T08:06:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.752+0000 7f539ffff640 1 -- 172.21.15.98:0/260214008 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f53a0005020 con 0x7f53a8106a80 2024-09-19T08:06:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.752+0000 7f53af6ac740 1 -- start start 2024-09-19T08:06:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53af6ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aacb40 0x5637e2aab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53af6ac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aab5d0 0x5637e2aaba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53af6ac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 0x5637e2ab8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53af6ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5637e2a112f0 con 0x5637e2aacb40 2024-09-19T08:06:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53af6ac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5637e29b8cf0 con 0x5637e2ab6370 2024-09-19T08:06:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53af6ac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5637e2a0d2b0 con 0x5637e2aab5d0 2024-09-19T08:06:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53a67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aacb40 0x5637e2aab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53a6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aab5d0 0x5637e2aaba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53a6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aab5d0 0x5637e2aaba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:36224/0 (socket says 172.21.15.98:36224) 2024-09-19T08:06:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53a67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aacb40 0x5637e2aab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43954/0 (socket says 172.21.15.98:43954) 2024-09-19T08:06:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53a6ffd640 1 -- 172.21.15.98:0/4033559145 learned_addr learned my addr 172.21.15.98:0/4033559145 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53a67fc640 1 --2- 172.21.15.98:0/4033559145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 0x5637e2ab8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f539ffff640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f53940a33b0 0x7f53940a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53a6ffd640 1 -- 172.21.15.98:0/4033559145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 msgr2=0x5637e2ab8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53a6ffd640 1 --2- 172.21.15.98:0/4033559145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 0x5637e2ab8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53a6ffd640 1 -- 172.21.15.98:0/4033559145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aacb40 msgr2=0x5637e2aab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53a6ffd640 1 --2- 172.21.15.98:0/4033559145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aacb40 0x5637e2aab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53a6ffd640 1 -- 172.21.15.98:0/4033559145 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637e299d3a0 con 0x5637e2aab5d0 2024-09-19T08:06:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53a67fc640 1 --2- 172.21.15.98:0/4033559145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aacb40 0x5637e2aab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:06:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.753+0000 7f53a77fe640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f53940a33b0 0x7f53940a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.754+0000 7f53a6ffd640 1 --2- 172.21.15.98:0/4033559145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aab5d0 0x5637e2aaba10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f539802a940 tx=0x7f5398005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7587f5788c2032c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.754+0000 7f539ffff640 1 -- 172.21.15.98:0/260214008 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f53a00d03e0 con 0x7f53a8106a80 2024-09-19T08:06:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.754+0000 7f539d7fa640 1 -- 172.21.15.98:0/4033559145 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398040070 con 0x5637e2aab5d0 2024-09-19T08:06:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.754+0000 7f539d7fa640 1 -- 172.21.15.98:0/4033559145 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53980318f0 con 0x5637e2aab5d0 2024-09-19T08:06:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.754+0000 7f539d7fa640 1 -- 172.21.15.98:0/4033559145 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398035040 con 0x5637e2aab5d0 2024-09-19T08:06:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 -- 172.21.15.98:0/4033559145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aab5d0 msgr2=0x5637e2aaba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53a77fe640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f53940a33b0 0x7f53940a5870 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f53a009c680 tx=0x7f53a000efd0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 --2- 172.21.15.98:0/4033559145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aab5d0 0x5637e2aaba10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f539802a940 tx=0x7f5398005c00 comp rx=0 tx=0).stop 2024-09-19T08:06:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 -- 172.21.15.98:0/4033559145 shutdown_connections 2024-09-19T08:06:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 --2- 172.21.15.98:0/4033559145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 0x5637e2ab8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 --2- 172.21.15.98:0/4033559145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aab5d0 0x5637e2aaba10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 --2- 172.21.15.98:0/4033559145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aacb40 0x5637e2aab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 -- 172.21.15.98:0/4033559145 >> 172.21.15.98:0/4033559145 conn(0x5637e2a74310 msgr2=0x5637e2aa7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 -- 172.21.15.98:0/4033559145 shutdown_connections 2024-09-19T08:06:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 -- 172.21.15.98:0/4033559145 wait complete. 2024-09-19T08:06:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 -- start start 2024-09-19T08:06:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 Processor -- start 2024-09-19T08:06:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aab5d0 0x5637e2ab5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aacb40 0x5637e2ab2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 0x5637e2ab34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5637e2ab9010 con 0x5637e2aab5d0 2024-09-19T08:06:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5637e29cc060 con 0x5637e2ab6370 2024-09-19T08:06:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.755+0000 7f53af6ac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5637e2ab9190 con 0x5637e2aacb40 2024-09-19T08:06:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f53af6ac740 10 client.0 _ll_register_callbacks cb 0x5637e271f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f53a67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aab5d0 0x5637e2ab5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f53a6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aacb40 0x5637e2ab2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f53a67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aab5d0 0x5637e2ab5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43962/0 (socket says 172.21.15.98:43962) 2024-09-19T08:06:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f53a6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aacb40 0x5637e2ab2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:36232/0 (socket says 172.21.15.98:36232) 2024-09-19T08:06:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f53a67fc640 1 -- 172.21.15.98:0/4060613765 learned_addr learned my addr 172.21.15.98:0/4060613765 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f53a67fc640 1 --2- 172.21.15.98:0/4060613765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 0x5637e2ab34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f53a6ffd640 1 -- 172.21.15.98:0/4060613765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 msgr2=0x5637e2ab34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f53a6ffd640 1 --2- 172.21.15.98:0/4060613765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 0x5637e2ab34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f53a6ffd640 1 -- 172.21.15.98:0/4060613765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aab5d0 msgr2=0x5637e2ab5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f53a6ffd640 1 --2- 172.21.15.98:0/4060613765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aab5d0 0x5637e2ab5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f53a6ffd640 1 -- 172.21.15.98:0/4060613765 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637e27823c0 con 0x5637e2aacb40 2024-09-19T08:06:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f53a6ffd640 1 --2- 172.21.15.98:0/4060613765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aacb40 0x5637e2ab2f90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5398006010 tx=0x7f53980052d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22ea2958ead8e431 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f538e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-09-19T08:06:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f538e7fc640 1 -- 172.21.15.98:0/4060613765 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398040030 con 0x5637e2aacb40 2024-09-19T08:06:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f53af6ac740 1 -- 172.21.15.98:0/4060613765 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53980318f0 con 0x5637e2aacb40 2024-09-19T08:06:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.756+0000 7f538e7fc640 1 -- 172.21.15.98:0/4060613765 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5398004260 con 0x5637e2aacb40 2024-09-19T08:06:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.757+0000 7f538e7fc640 1 -- 172.21.15.98:0/4060613765 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5398056050 con 0x5637e2aacb40 2024-09-19T08:06:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.757+0000 7f53af6ac740 10 client.25522 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.757+0000 7f53af6ac740 1 -- 172.21.15.98:0/4060613765 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5637e2ab9060 con 0x5637e2aacb40 2024-09-19T08:06:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.757+0000 7f538e7fc640 1 -- 172.21.15.98:0/4060613765 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53980318f0 con 0x5637e2aacb40 2024-09-19T08:06:12.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.757+0000 7f538e7fc640 1 -- 172.21.15.98:0/4060613765 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f53980036f0 con 0x5637e2aacb40 2024-09-19T08:06:12.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.757+0000 7f53af6ac740 10 client.25522 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:12.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.757+0000 7f53af6ac740 10 client.25522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:06:12.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.757+0000 7f53af6ac740 20 client.25522 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:12.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.757+0000 7f53af6ac740 1 --2- 172.21.15.98:0/4060613765 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5637e2ac9fb0 0x5637e2acc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.757+0000 7f53af6ac740 4 client.25522 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:12.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.757+0000 7f53af6ac740 1 -- 172.21.15.98:0/4060613765 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637e2a1e860 con 0x5637e2ac9fb0 2024-09-19T08:06:12.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.757+0000 7f53a67fc640 1 --2- 172.21.15.98:0/4060613765 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5637e2ac9fb0 0x5637e2acc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.758+0000 7f53a67fc640 1 --2- 172.21.15.98:0/4060613765 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5637e2ac9fb0 0x5637e2acc3a0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd1035641204fdbd server_cookie=e340f158ec49c852 in_seq=0 out_seq=0 2024-09-19T08:06:12.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.758+0000 7f538e7fc640 10 client.25522 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:12.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.758+0000 7f538e7fc640 1 -- 172.21.15.98:0/4060613765 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5637e2a1e860 con 0x5637e2ac9fb0 2024-09-19T08:06:12.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.758+0000 7f538e7fc640 10 client.25522 handle_command_reply: tid=0 2024-09-19T08:06:12.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.758+0000 7f53af6ac740 1 client.25522 shutdown 2024-09-19T08:06:12.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.759+0000 7f53af6ac740 1 -- 172.21.15.98:0/4060613765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aacb40 msgr2=0x5637e2ab2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:12.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.759+0000 7f53af6ac740 1 --2- 172.21.15.98:0/4060613765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aacb40 0x5637e2ab2f90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5398006010 tx=0x7f53980052d0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.759+0000 7f53af6ac740 1 -- 172.21.15.98:0/4060613765 shutdown_connections 2024-09-19T08:06:12.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.759+0000 7f53af6ac740 1 --2- 172.21.15.98:0/4060613765 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5637e2ac9fb0 0x5637e2acc3a0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.759+0000 7f53af6ac740 1 --2- 172.21.15.98:0/4060613765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 0x5637e2ab34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.759+0000 7f53af6ac740 1 --2- 172.21.15.98:0/4060613765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aacb40 0x5637e2ab2f90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.759+0000 7f538e7fc640 0 client.25522 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:12.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.759+0000 7f53af6ac740 1 --2- 172.21.15.98:0/4060613765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aab5d0 0x5637e2ab5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.759+0000 7f53af6ac740 1 -- 172.21.15.98:0/4060613765 >> 172.21.15.98:0/4060613765 conn(0x5637e2a74310 msgr2=0x5637e2aad960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:12.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.759+0000 7f53af6ac740 1 -- 172.21.15.98:0/4060613765 shutdown_connections 2024-09-19T08:06:12.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.759+0000 7f53af6ac740 1 -- 172.21.15.98:0/4060613765 wait complete. 2024-09-19T08:06:12.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.759+0000 7f53af6ac740 20 client.25522 trim_cache size 0 max 16384 2024-09-19T08:06:12.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.772+0000 7f53af6ac740 1 Processor -- start 2024-09-19T08:06:12.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.772+0000 7f53af6ac740 1 -- start start 2024-09-19T08:06:12.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53af6ac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aab5d0 0x5637e2ac6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53af6ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aacb40 0x5637e2ab4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53af6ac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 0x5637e2ab7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53af6ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53a81ab5e0 con 0x5637e2aacb40 2024-09-19T08:06:12.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53af6ac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53a81ad180 con 0x5637e2ab6370 2024-09-19T08:06:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53af6ac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f53a81abcb0 con 0x5637e2aab5d0 2024-09-19T08:06:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53a6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aacb40 0x5637e2ab4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53a67fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aab5d0 0x5637e2ac6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53a67fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aab5d0 0x5637e2ac6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:36246/0 (socket says 172.21.15.98:36246) 2024-09-19T08:06:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53a6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aacb40 0x5637e2ab4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43970/0 (socket says 172.21.15.98:43970) 2024-09-19T08:06:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53a67fc640 1 -- 172.21.15.98:0/408001824 learned_addr learned my addr 172.21.15.98:0/408001824 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53a67fc640 1 --2- 172.21.15.98:0/408001824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 0x5637e2ab7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53a67fc640 1 -- 172.21.15.98:0/408001824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 msgr2=0x5637e2ab7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53a67fc640 1 --2- 172.21.15.98:0/408001824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 0x5637e2ab7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53a67fc640 1 -- 172.21.15.98:0/408001824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aacb40 msgr2=0x5637e2ab4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53a67fc640 1 --2- 172.21.15.98:0/408001824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aacb40 0x5637e2ab4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53a67fc640 1 -- 172.21.15.98:0/408001824 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637e2aa8960 con 0x5637e2aab5d0 2024-09-19T08:06:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.773+0000 7f53a6ffd640 1 --2- 172.21.15.98:0/408001824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aacb40 0x5637e2ab4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:06:12.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.774+0000 7f53a67fc640 1 --2- 172.21.15.98:0/408001824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aab5d0 0x5637e2ac6740 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5390005fd0 tx=0x7f5390015040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a903bdd6453c1c6a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:12.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.774+0000 7f539cff9640 1 -- 172.21.15.98:0/408001824 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539001c040 con 0x5637e2aab5d0 2024-09-19T08:06:12.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.774+0000 7f539cff9640 1 -- 172.21.15.98:0/408001824 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5390022c10 con 0x5637e2aab5d0 2024-09-19T08:06:12.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.774+0000 7f539cff9640 1 -- 172.21.15.98:0/408001824 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5390010430 con 0x5637e2aab5d0 2024-09-19T08:06:12.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 -- 172.21.15.98:0/408001824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aab5d0 msgr2=0x5637e2ac6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:12.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 --2- 172.21.15.98:0/408001824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aab5d0 0x5637e2ac6740 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5390005fd0 tx=0x7f5390015040 comp rx=0 tx=0).stop 2024-09-19T08:06:12.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 -- 172.21.15.98:0/408001824 shutdown_connections 2024-09-19T08:06:12.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 --2- 172.21.15.98:0/408001824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2ab6370 0x5637e2ab7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 --2- 172.21.15.98:0/408001824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2aacb40 0x5637e2ab4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 --2- 172.21.15.98:0/408001824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aab5d0 0x5637e2ac6740 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 -- 172.21.15.98:0/408001824 >> 172.21.15.98:0/408001824 conn(0x5637e2a74310 msgr2=0x5637e2ab47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:12.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 -- 172.21.15.98:0/408001824 shutdown_connections 2024-09-19T08:06:12.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 -- 172.21.15.98:0/408001824 wait complete. 2024-09-19T08:06:12.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 -- start start 2024-09-19T08:06:12.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 Processor -- start 2024-09-19T08:06:12.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2aab5d0 0x5637e2ac3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aacb40 0x5637e2aafd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2ab6370 0x5637e2ab0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53a81aca80 con 0x5637e2ab6370 2024-09-19T08:06:12.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53a81ac3b0 con 0x5637e2aab5d0 2024-09-19T08:06:12.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.775+0000 7f53af6ac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f53a81a95c0 con 0x5637e2aacb40 2024-09-19T08:06:12.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f53af6ac740 10 client.0 _ll_register_callbacks cb 0x7f539000b4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:12.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f53a67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2ab6370 0x5637e2ab0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f53a67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2ab6370 0x5637e2ab0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43982/0 (socket says 172.21.15.98:43982) 2024-09-19T08:06:12.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f53a67fc640 1 -- 172.21.15.98:0/371224247 learned_addr learned my addr 172.21.15.98:0/371224247 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:12.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f53a6ffd640 1 --2- 172.21.15.98:0/371224247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aacb40 0x5637e2aafd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f53a67fc640 1 --2- 172.21.15.98:0/371224247 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2aab5d0 0x5637e2ac3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f53a6ffd640 1 -- 172.21.15.98:0/371224247 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2aab5d0 msgr2=0x5637e2ac3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f53a6ffd640 1 --2- 172.21.15.98:0/371224247 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2aab5d0 0x5637e2ac3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f53a6ffd640 1 -- 172.21.15.98:0/371224247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2ab6370 msgr2=0x5637e2ab0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f53a6ffd640 1 --2- 172.21.15.98:0/371224247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2ab6370 0x5637e2ab0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f53a6ffd640 1 -- 172.21.15.98:0/371224247 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637e2ab0aa0 con 0x5637e2aacb40 2024-09-19T08:06:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f53a6ffd640 1 --2- 172.21.15.98:0/371224247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aacb40 0x5637e2aafd50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5398003ac0 tx=0x7f5398003ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98c6a231ce6e290 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f538e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-09-19T08:06:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f538e7fc640 1 -- 172.21.15.98:0/371224247 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539805e030 con 0x5637e2aacb40 2024-09-19T08:06:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f538e7fc640 1 -- 172.21.15.98:0/371224247 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5398004470 con 0x5637e2aacb40 2024-09-19T08:06:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.776+0000 7f53af6ac740 1 -- 172.21.15.98:0/371224247 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5390022c10 con 0x5637e2aacb40 2024-09-19T08:06:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.777+0000 7f538e7fc640 1 -- 172.21.15.98:0/371224247 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5398068050 con 0x5637e2aacb40 2024-09-19T08:06:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.777+0000 7f53af6ac740 10 client.25528 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.777+0000 7f53af6ac740 1 -- 172.21.15.98:0/371224247 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5637e2b34bc0 con 0x5637e2aacb40 2024-09-19T08:06:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.777+0000 7f538e7fc640 1 -- 172.21.15.98:0/371224247 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5390022c10 con 0x5637e2aacb40 2024-09-19T08:06:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.777+0000 7f538e7fc640 1 -- 172.21.15.98:0/371224247 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f53980053f0 con 0x5637e2aacb40 2024-09-19T08:06:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.777+0000 7f53af6ac740 10 client.25528 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.777+0000 7f53af6ac740 10 client.25528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:06:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.777+0000 7f53af6ac740 20 client.25528 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.777+0000 7f53af6ac740 1 --2- 172.21.15.98:0/371224247 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5637e2b35360 0x5637e2b37780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.777+0000 7f53af6ac740 4 client.25528 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:06:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.777+0000 7f53af6ac740 1 -- 172.21.15.98:0/371224247 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5637e2a18d70 con 0x5637e2b35360 2024-09-19T08:06:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.777+0000 7f53a67fc640 1 --2- 172.21.15.98:0/371224247 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5637e2b35360 0x5637e2b37780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f538e7fc640 10 client.25528 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f53a67fc640 1 --2- 172.21.15.98:0/371224247 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5637e2b35360 0x5637e2b37780 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5093551d2f25caa1 server_cookie=fa208f67f6cf7ec4 in_seq=0 out_seq=0 2024-09-19T08:06:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f538e7fc640 1 -- 172.21.15.98:0/371224247 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5637e2a18d70 con 0x5637e2b35360 2024-09-19T08:06:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f538e7fc640 10 client.25528 handle_command_reply: tid=0 2024-09-19T08:06:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f53af6ac740 1 client.25528 shutdown 2024-09-19T08:06:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f53af6ac740 1 -- 172.21.15.98:0/371224247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aacb40 msgr2=0x5637e2aafd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f53af6ac740 1 --2- 172.21.15.98:0/371224247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aacb40 0x5637e2aafd50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5398003ac0 tx=0x7f5398003ec0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f53af6ac740 1 -- 172.21.15.98:0/371224247 shutdown_connections 2024-09-19T08:06:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f53af6ac740 1 --2- 172.21.15.98:0/371224247 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5637e2b35360 0x5637e2b37780 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f53af6ac740 1 --2- 172.21.15.98:0/371224247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637e2ab6370 0x5637e2ab0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f53af6ac740 1 --2- 172.21.15.98:0/371224247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637e2aacb40 0x5637e2aafd50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f538e7fc640 0 client.25528 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f53af6ac740 1 --2- 172.21.15.98:0/371224247 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5637e2aab5d0 0x5637e2ac3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f53af6ac740 1 -- 172.21.15.98:0/371224247 >> 172.21.15.98:0/371224247 conn(0x5637e2a74310 msgr2=0x5637e2ab1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f53af6ac740 1 -- 172.21.15.98:0/371224247 shutdown_connections 2024-09-19T08:06:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f53af6ac740 1 -- 172.21.15.98:0/371224247 wait complete. 2024-09-19T08:06:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.778+0000 7f53af6ac740 20 client.25528 trim_cache size 0 max 16384 2024-09-19T08:06:12.785 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:06:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.779+0000 7f53ad2d3640 1 -- 172.21.15.98:0/260214008 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f53940a33b0 msgr2=0x7f53940a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.779+0000 7f53ad2d3640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f53940a33b0 0x7f53940a5870 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f53a009c680 tx=0x7f53a000efd0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.779+0000 7f53ad2d3640 1 -- 172.21.15.98:0/260214008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8106a80 msgr2=0x7f53a819cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.779+0000 7f53ad2d3640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8106a80 0x7f53a819cce0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f53a000eb10 tx=0x7f53a000c2d0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.779+0000 7f53ad2d3640 1 -- 172.21.15.98:0/260214008 shutdown_connections 2024-09-19T08:06:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.779+0000 7f53ad2d3640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f53940a33b0 0x7f53940a5870 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.779+0000 7f53ad2d3640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8106a80 0x7f53a819cce0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.779+0000 7f53ad2d3640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53a8101420 0x7f53a8198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.779+0000 7f53ad2d3640 1 --2- 172.21.15.98:0/260214008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53a8100a80 0x7f53a8198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.779+0000 7f53ad2d3640 1 -- 172.21.15.98:0/260214008 >> 172.21.15.98:0/260214008 conn(0x7f53a80fc7a0 msgr2=0x7f53a806aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.779+0000 7f53ad2d3640 1 -- 172.21.15.98:0/260214008 shutdown_connections 2024-09-19T08:06:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:12.779+0000 7f53ad2d3640 1 -- 172.21.15.98:0/260214008 wait complete. 2024-09-19T08:06:12.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:12 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:06:12.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:12 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:06:12.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:12 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:06:12.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:12 smithi098 ceph-mon[31333]: pgmap v99: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 693 MiB/s wr, 266 op/s 2024-09-19T08:06:13.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:12 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:06:13.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:12 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:06:13.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:12 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:06:13.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:12 smithi139 ceph-mon[34874]: pgmap v99: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 693 MiB/s wr, 266 op/s 2024-09-19T08:06:13.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:12 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:06:13.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:12 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:06:13.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:12 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:06:13.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:12 smithi162 ceph-mon[35260]: pgmap v99: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 693 MiB/s wr, 266 op/s 2024-09-19T08:06:13.890 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:06:13.891 INFO:tasks.workunit.client.0.smithi098.stdout: Children see throughput for 1 re-readers = 7846990.00 kB/sec 2024-09-19T08:06:13.891 INFO:tasks.workunit.client.0.smithi098.stdout: Parent sees throughput for 1 re-readers = 7845216.95 kB/sec 2024-09-19T08:06:13.891 INFO:tasks.workunit.client.0.smithi098.stdout: Min throughput per process = 7846990.00 kB/sec 2024-09-19T08:06:13.891 INFO:tasks.workunit.client.0.smithi098.stdout: Max throughput per process = 7846990.00 kB/sec 2024-09-19T08:06:13.891 INFO:tasks.workunit.client.0.smithi098.stdout: Avg throughput per process = 7846990.00 kB/sec 2024-09-19T08:06:13.891 INFO:tasks.workunit.client.0.smithi098.stdout: Min xfer = 10485760.00 kB 2024-09-19T08:06:14.627 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:06:14.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:06:14.892 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:14.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:14 smithi098 ceph-mon[31333]: pgmap v100: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 345 MiB/s wr, 94 op/s 2024-09-19T08:06:15.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:14 smithi139 ceph-mon[34874]: pgmap v100: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 345 MiB/s wr, 94 op/s 2024-09-19T08:06:15.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.316+0000 7f5e3b3ba640 1 -- 172.21.15.98:0/1698472644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e34101420 msgr2=0x7f5e3410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:15.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.316+0000 7f5e3b3ba640 1 --2- 172.21.15.98:0/1698472644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e34101420 0x7f5e3410d950 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009cd0 tx=0x7f5e2c02a8e0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.316+0000 7f5e3b3ba640 1 -- 172.21.15.98:0/1698472644 shutdown_connections 2024-09-19T08:06:15.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.316+0000 7f5e3b3ba640 1 --2- 172.21.15.98:0/1698472644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e34101420 0x7f5e3410d950 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.316+0000 7f5e3b3ba640 1 --2- 172.21.15.98:0/1698472644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e34100a80 0x7f5e34100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.316+0000 7f5e3b3ba640 1 --2- 172.21.15.98:0/1698472644 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e34106a80 0x7f5e34106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.316+0000 7f5e3b3ba640 1 -- 172.21.15.98:0/1698472644 >> 172.21.15.98:0/1698472644 conn(0x7f5e340fc7a0 msgr2=0x7f5e340febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:15.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.316+0000 7f5e3b3ba640 1 -- 172.21.15.98:0/1698472644 shutdown_connections 2024-09-19T08:06:15.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.316+0000 7f5e3b3ba640 1 -- 172.21.15.98:0/1698472644 wait complete. 2024-09-19T08:06:15.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e3b3ba640 1 Processor -- start 2024-09-19T08:06:15.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e3b3ba640 1 -- start start 2024-09-19T08:06:15.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e3b3ba640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e34100a80 0x7f5e34198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e3b3ba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e34101420 0x7f5e34198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e3b3ba640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e34106a80 0x7f5e3419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e3b3ba640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e34110e80 con 0x7f5e34101420 2024-09-19T08:06:15.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e3b3ba640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5e34110d00 con 0x7f5e34106a80 2024-09-19T08:06:15.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e3b3ba640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e34111000 con 0x7f5e34100a80 2024-09-19T08:06:15.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e38955640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e34101420 0x7f5e34198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e38955640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e34101420 0x7f5e34198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44002/0 (socket says 172.21.15.98:44002) 2024-09-19T08:06:15.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e38955640 1 -- 172.21.15.98:0/1783690165 learned_addr learned my addr 172.21.15.98:0/1783690165 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:15.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e39156640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e34100a80 0x7f5e34198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e39957640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e34106a80 0x7f5e3419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e38955640 1 -- 172.21.15.98:0/1783690165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e34100a80 msgr2=0x7f5e34198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:15.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e38955640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e34100a80 0x7f5e34198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e38955640 1 -- 172.21.15.98:0/1783690165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e34106a80 msgr2=0x7f5e3419ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:15.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e38955640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e34106a80 0x7f5e3419ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e38955640 1 -- 172.21.15.98:0/1783690165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e3419d3d0 con 0x7f5e34101420 2024-09-19T08:06:15.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.317+0000 7f5e39957640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e34106a80 0x7f5e3419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:15.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.318+0000 7f5e38955640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e34101420 0x7f5e34198960 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000d770 tx=0x7f5e3000db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:15.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.318+0000 7f5e267fc640 1 -- 172.21.15.98:0/1783690165 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30017070 con 0x7f5e34101420 2024-09-19T08:06:15.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.318+0000 7f5e3b3ba640 1 -- 172.21.15.98:0/1783690165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e3419d6c0 con 0x7f5e34101420 2024-09-19T08:06:15.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.318+0000 7f5e267fc640 1 -- 172.21.15.98:0/1783690165 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e30009820 con 0x7f5e34101420 2024-09-19T08:06:15.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.318+0000 7f5e267fc640 1 -- 172.21.15.98:0/1783690165 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30004cd0 con 0x7f5e34101420 2024-09-19T08:06:15.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.318+0000 7f5e3b3ba640 1 -- 172.21.15.98:0/1783690165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e341a4fa0 con 0x7f5e34101420 2024-09-19T08:06:15.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.320+0000 7f5e267fc640 1 -- 172.21.15.98:0/1783690165 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f5e300099d0 con 0x7f5e34101420 2024-09-19T08:06:15.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.320+0000 7f5e3d793740 1 Processor -- start 2024-09-19T08:06:15.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.320+0000 7f5e3d793740 1 -- start start 2024-09-19T08:06:15.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e3d793740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504cd5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e3d793740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 0x564504ce0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e3d793740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504ce3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e3d793740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564504c3b280 con 0x564504cd5b60 2024-09-19T08:06:15.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e3d793740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564504be2b10 con 0x564504cd6480 2024-09-19T08:06:15.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e3d793740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564504c37240 con 0x564504ce0d40 2024-09-19T08:06:15.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e267fc640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5e200a33b0 0x7f5e200a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e39957640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504cd5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e39957640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504cd5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44006/0 (socket says 172.21.15.98:44006) 2024-09-19T08:06:15.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e39957640 1 -- 172.21.15.98:0/2477782472 learned_addr learned my addr 172.21.15.98:0/2477782472 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:15.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e39957640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5e200a33b0 0x7f5e200a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e38955640 1 --2- 172.21.15.98:0/2477782472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504ce3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e39156640 1 --2- 172.21.15.98:0/2477782472 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 0x564504ce0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e267fc640 1 -- 172.21.15.98:0/1783690165 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5e300cf730 con 0x7f5e34101420 2024-09-19T08:06:15.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e39957640 1 -- 172.21.15.98:0/2477782472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 msgr2=0x564504ce3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:15.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e39957640 1 --2- 172.21.15.98:0/2477782472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504ce3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e39957640 1 -- 172.21.15.98:0/2477782472 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 msgr2=0x564504ce0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:15.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e39957640 1 --2- 172.21.15.98:0/2477782472 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 0x564504ce0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.321+0000 7f5e39957640 1 -- 172.21.15.98:0/2477782472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564504bef1f0 con 0x564504cd5b60 2024-09-19T08:06:15.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.322+0000 7f5e39957640 1 --2- 172.21.15.98:0/2477782472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504cd5f40 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c031d00 tx=0x7f5e2c031d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8226ee548b3872f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:15.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.322+0000 7f5e39957640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5e200a33b0 0x7f5e200a5870 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c031da0 tx=0x7f5e2c048000 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:15.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.322+0000 7f5e1f7fe640 1 -- 172.21.15.98:0/2477782472 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c04e070 con 0x564504cd5b60 2024-09-19T08:06:15.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.322+0000 7f5e1f7fe640 1 -- 172.21.15.98:0/2477782472 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e2c036040 con 0x564504cd5b60 2024-09-19T08:06:15.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.322+0000 7f5e1f7fe640 1 -- 172.21.15.98:0/2477782472 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c0307a0 con 0x564504cd5b60 2024-09-19T08:06:15.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.322+0000 7f5e3d793740 1 -- 172.21.15.98:0/2477782472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 msgr2=0x564504cd5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:15.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.322+0000 7f5e3d793740 1 --2- 172.21.15.98:0/2477782472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504cd5f40 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c031d00 tx=0x7f5e2c031d30 comp rx=0 tx=0).stop 2024-09-19T08:06:15.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.322+0000 7f5e3d793740 1 -- 172.21.15.98:0/2477782472 shutdown_connections 2024-09-19T08:06:15.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.322+0000 7f5e3d793740 1 --2- 172.21.15.98:0/2477782472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504ce3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.322+0000 7f5e3d793740 1 --2- 172.21.15.98:0/2477782472 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 0x564504ce0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.322+0000 7f5e3d793740 1 --2- 172.21.15.98:0/2477782472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504cd5f40 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.322+0000 7f5e3d793740 1 -- 172.21.15.98:0/2477782472 >> 172.21.15.98:0/2477782472 conn(0x564504cd20f0 msgr2=0x564504cd24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:15.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.323+0000 7f5e3d793740 1 -- 172.21.15.98:0/2477782472 shutdown_connections 2024-09-19T08:06:15.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.323+0000 7f5e3d793740 1 -- 172.21.15.98:0/2477782472 wait complete. 2024-09-19T08:06:15.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.323+0000 7f5e3d793740 1 -- start start 2024-09-19T08:06:15.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.323+0000 7f5e3d793740 1 Processor -- start 2024-09-19T08:06:15.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.323+0000 7f5e3d793740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504cddc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.323+0000 7f5e3d793740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 0x564504cde190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.323+0000 7f5e3d793740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504cde6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.323+0000 7f5e3d793740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564504ce3910 con 0x564504cd5b60 2024-09-19T08:06:15.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.323+0000 7f5e3d793740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564504bf5f10 con 0x564504cd6480 2024-09-19T08:06:15.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.323+0000 7f5e3d793740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564504ce3a90 con 0x564504ce0d40 2024-09-19T08:06:15.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.323+0000 7f5e3d793740 10 client.0 _ll_register_callbacks cb 0x564504bbea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:15.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e39156640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504cddc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e39156640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504cddc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44016/0 (socket says 172.21.15.98:44016) 2024-09-19T08:06:15.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e39957640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504cde6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e38955640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 0x564504cde190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e39156640 1 -- 172.21.15.98:0/37585324 learned_addr learned my addr 172.21.15.98:0/37585324 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:15.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e39156640 1 -- 172.21.15.98:0/37585324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 msgr2=0x564504cde6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:15.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e39156640 1 --2- 172.21.15.98:0/37585324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504cde6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e39156640 1 -- 172.21.15.98:0/37585324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 msgr2=0x564504cde190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:15.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e39156640 1 --2- 172.21.15.98:0/37585324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 0x564504cde190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e39156640 1 -- 172.21.15.98:0/37585324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645049ac6a0 con 0x564504cd5b60 2024-09-19T08:06:15.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e39156640 1 --2- 172.21.15.98:0/37585324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504cddc50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5e28010a80 tx=0x7f5e28010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a8f2d22d1b73cc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:15.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:15.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e1cff9640 1 -- 172.21.15.98:0/37585324 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2801e060 con 0x564504cd5b60 2024-09-19T08:06:15.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e1cff9640 1 -- 172.21.15.98:0/37585324 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5e2800fc50 con 0x564504cd5b60 2024-09-19T08:06:15.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e3d793740 1 -- 172.21.15.98:0/37585324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e2c036040 con 0x564504cd5b60 2024-09-19T08:06:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e1cff9640 1 -- 172.21.15.98:0/37585324 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5e2800fe50 con 0x564504cd5b60 2024-09-19T08:06:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e3d793740 10 client.16314 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.324+0000 7f5e3d793740 1 -- 172.21.15.98:0/37585324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564504cef5a0 con 0x564504cd5b60 2024-09-19T08:06:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.325+0000 7f5e1cff9640 1 -- 172.21.15.98:0/37585324 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e2c036040 con 0x564504cd5b60 2024-09-19T08:06:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.325+0000 7f5e1cff9640 1 -- 172.21.15.98:0/37585324 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f5e28018640 con 0x564504cd5b60 2024-09-19T08:06:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.325+0000 7f5e3d793740 10 client.16314 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.325+0000 7f5e3d793740 10 client.16314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:06:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.325+0000 7f5e3d793740 20 client.16314 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.325+0000 7f5e3d793740 1 --2- 172.21.15.98:0/37585324 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x564504cf0a60 0x564504cf2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.325+0000 7f5e3d793740 4 client.16314 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.325+0000 7f5e3d793740 1 -- 172.21.15.98:0/37585324 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564504c487f0 con 0x564504cf0a60 2024-09-19T08:06:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.325+0000 7f5e38955640 1 --2- 172.21.15.98:0/37585324 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x564504cf0a60 0x564504cf2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.326+0000 7f5e38955640 1 --2- 172.21.15.98:0/37585324 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x564504cf0a60 0x564504cf2e90 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46cc685010786658 server_cookie=c4efa47e8e75a290 in_seq=0 out_seq=0 2024-09-19T08:06:15.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.326+0000 7f5e1cff9640 10 client.16314 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:15.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.326+0000 7f5e1cff9640 1 -- 172.21.15.98:0/37585324 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564504c487f0 con 0x564504cf0a60 2024-09-19T08:06:15.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.326+0000 7f5e1cff9640 10 client.16314 handle_command_reply: tid=0 2024-09-19T08:06:15.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.326+0000 7f5e3d793740 1 client.16314 shutdown 2024-09-19T08:06:15.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.327+0000 7f5e3d793740 1 -- 172.21.15.98:0/37585324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 msgr2=0x564504cddc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:15.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.327+0000 7f5e3d793740 1 --2- 172.21.15.98:0/37585324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504cddc50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5e28010a80 tx=0x7f5e28010f40 comp rx=0 tx=0).stop 2024-09-19T08:06:15.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.327+0000 7f5e3d793740 1 -- 172.21.15.98:0/37585324 shutdown_connections 2024-09-19T08:06:15.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.327+0000 7f5e3d793740 1 --2- 172.21.15.98:0/37585324 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x564504cf0a60 0x564504cf2e90 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.327+0000 7f5e3d793740 1 --2- 172.21.15.98:0/37585324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504cde6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.327+0000 7f5e3d793740 1 --2- 172.21.15.98:0/37585324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 0x564504cde190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.327+0000 7f5e3d793740 1 --2- 172.21.15.98:0/37585324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504cddc50 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.327+0000 7f5e3d793740 1 -- 172.21.15.98:0/37585324 >> 172.21.15.98:0/37585324 conn(0x564504cd20f0 msgr2=0x564504cd1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:15.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.327+0000 7f5e1cff9640 0 client.16314 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:15.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.327+0000 7f5e3d793740 1 -- 172.21.15.98:0/37585324 shutdown_connections 2024-09-19T08:06:15.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.327+0000 7f5e3d793740 1 -- 172.21.15.98:0/37585324 wait complete. 2024-09-19T08:06:15.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.327+0000 7f5e3d793740 20 client.16314 trim_cache size 0 max 16384 2024-09-19T08:06:15.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e3d793740 1 Processor -- start 2024-09-19T08:06:15.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e3d793740 1 -- start start 2024-09-19T08:06:15.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e3d793740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd5b60 0x564504cd85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e3d793740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd6480 0x564504cdfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e3d793740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504ce90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e3d793740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e341ad7b0 con 0x564504cd6480 2024-09-19T08:06:15.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e3d793740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5e341af350 con 0x564504cd5b60 2024-09-19T08:06:15.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e3d793740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e341ade80 con 0x564504ce0d40 2024-09-19T08:06:15.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e38955640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd6480 0x564504cdfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e38955640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd6480 0x564504cdfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44020/0 (socket says 172.21.15.98:44020) 2024-09-19T08:06:15.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e38955640 1 -- 172.21.15.98:0/729315669 learned_addr learned my addr 172.21.15.98:0/729315669 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:15.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e39957640 1 --2- 172.21.15.98:0/729315669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504ce90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e39156640 1 --2- 172.21.15.98:0/729315669 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd5b60 0x564504cd85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e38955640 1 -- 172.21.15.98:0/729315669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 msgr2=0x564504ce90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:15.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e38955640 1 --2- 172.21.15.98:0/729315669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504ce90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e38955640 1 -- 172.21.15.98:0/729315669 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd5b60 msgr2=0x564504cd85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:15.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e38955640 1 --2- 172.21.15.98:0/729315669 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd5b60 0x564504cd85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e38955640 1 -- 172.21.15.98:0/729315669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564504cdc1a0 con 0x564504cd6480 2024-09-19T08:06:15.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.344+0000 7f5e39957640 1 --2- 172.21.15.98:0/729315669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504ce90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:15.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.345+0000 7f5e38955640 1 --2- 172.21.15.98:0/729315669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd6480 0x564504cdfab0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000dfd0 tx=0x7f5e3009f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be468a93196c4d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:15.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.345+0000 7f5e1effd640 1 -- 172.21.15.98:0/729315669 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e300b3020 con 0x564504cd6480 2024-09-19T08:06:15.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.345+0000 7f5e1effd640 1 -- 172.21.15.98:0/729315669 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e3009c140 con 0x564504cd6480 2024-09-19T08:06:15.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.345+0000 7f5e1effd640 1 -- 172.21.15.98:0/729315669 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30020040 con 0x564504cd6480 2024-09-19T08:06:15.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.345+0000 7f5e3d793740 1 -- 172.21.15.98:0/729315669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd6480 msgr2=0x564504cdfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:15.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.345+0000 7f5e3d793740 1 --2- 172.21.15.98:0/729315669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd6480 0x564504cdfab0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000dfd0 tx=0x7f5e3009f040 comp rx=0 tx=0).stop 2024-09-19T08:06:15.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.345+0000 7f5e3d793740 1 -- 172.21.15.98:0/729315669 shutdown_connections 2024-09-19T08:06:15.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.345+0000 7f5e3d793740 1 --2- 172.21.15.98:0/729315669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504ce90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.345+0000 7f5e3d793740 1 --2- 172.21.15.98:0/729315669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd6480 0x564504cdfab0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.345+0000 7f5e3d793740 1 --2- 172.21.15.98:0/729315669 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd5b60 0x564504cd85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.345+0000 7f5e3d793740 1 -- 172.21.15.98:0/729315669 >> 172.21.15.98:0/729315669 conn(0x564504cd20f0 msgr2=0x564504cdf3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:15.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.345+0000 7f5e3d793740 1 -- 172.21.15.98:0/729315669 shutdown_connections 2024-09-19T08:06:15.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.345+0000 7f5e3d793740 1 -- 172.21.15.98:0/729315669 wait complete. 2024-09-19T08:06:15.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.346+0000 7f5e3d793740 1 -- start start 2024-09-19T08:06:15.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.346+0000 7f5e3d793740 1 Processor -- start 2024-09-19T08:06:15.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.346+0000 7f5e3d793740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504d538d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.346+0000 7f5e3d793740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 0x564504d53e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.346+0000 7f5e3d793740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504d58740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.346+0000 7f5e3d793740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e341aec50 con 0x564504cd5b60 2024-09-19T08:06:15.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.346+0000 7f5e3d793740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5e341ae580 con 0x564504cd6480 2024-09-19T08:06:15.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.346+0000 7f5e3d793740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e341ab790 con 0x564504ce0d40 2024-09-19T08:06:15.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.346+0000 7f5e3d793740 10 client.0 _ll_register_callbacks cb 0x7f5e2800a740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:15.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.346+0000 7f5e39156640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504d538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.346+0000 7f5e39156640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504d538d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44024/0 (socket says 172.21.15.98:44024) 2024-09-19T08:06:15.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.346+0000 7f5e39156640 1 -- 172.21.15.98:0/1123132396 learned_addr learned my addr 172.21.15.98:0/1123132396 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:15.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.346+0000 7f5e38955640 1 --2- 172.21.15.98:0/1123132396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 0x564504d53e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e39156640 1 -- 172.21.15.98:0/1123132396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 msgr2=0x564504d58740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e39156640 1 --2- 172.21.15.98:0/1123132396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504d58740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e39156640 1 -- 172.21.15.98:0/1123132396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 msgr2=0x564504d53e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e39156640 1 --2- 172.21.15.98:0/1123132396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 0x564504d53e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e39156640 1 -- 172.21.15.98:0/1123132396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564504d58d80 con 0x564504cd5b60 2024-09-19T08:06:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e38955640 1 --2- 172.21.15.98:0/1123132396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 0x564504d53e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e39156640 1 --2- 172.21.15.98:0/1123132396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504d538d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5e280109f0 tx=0x7f5e2800f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd8fb42294e162e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e1cff9640 1 -- 172.21.15.98:0/1123132396 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e28031060 con 0x564504cd5b60 2024-09-19T08:06:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e1cff9640 1 -- 172.21.15.98:0/1123132396 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5e2800fc50 con 0x564504cd5b60 2024-09-19T08:06:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e3d793740 1 -- 172.21.15.98:0/1123132396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e3009c140 con 0x564504cd5b60 2024-09-19T08:06:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e1cff9640 1 -- 172.21.15.98:0/1123132396 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5e28036480 con 0x564504cd5b60 2024-09-19T08:06:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e3d793740 10 client.16320 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e3d793740 1 -- 172.21.15.98:0/1123132396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564504d60170 con 0x564504cd5b60 2024-09-19T08:06:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e1cff9640 1 -- 172.21.15.98:0/1123132396 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e3009c140 con 0x564504cd5b60 2024-09-19T08:06:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e1cff9640 1 -- 172.21.15.98:0/1123132396 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f5e2802b570 con 0x564504cd5b60 2024-09-19T08:06:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e3d793740 10 client.16320 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e3d793740 10 client.16320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:06:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e3d793740 20 client.16320 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.347+0000 7f5e3d793740 1 --2- 172.21.15.98:0/1123132396 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x564504d60950 0x564504d62d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.348+0000 7f5e3d793740 4 client.16320 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:06:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.348+0000 7f5e3d793740 1 -- 172.21.15.98:0/1123132396 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564504c42d00 con 0x564504d60950 2024-09-19T08:06:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.348+0000 7f5e38955640 1 --2- 172.21.15.98:0/1123132396 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x564504d60950 0x564504d62d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.348+0000 7f5e38955640 1 --2- 172.21.15.98:0/1123132396 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x564504d60950 0x564504d62d90 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb816021479a703e server_cookie=3079a166c0c111cc in_seq=0 out_seq=0 2024-09-19T08:06:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.348+0000 7f5e1cff9640 10 client.16320 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.349+0000 7f5e1cff9640 1 -- 172.21.15.98:0/1123132396 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564504c42d00 con 0x564504d60950 2024-09-19T08:06:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.349+0000 7f5e1cff9640 10 client.16320 handle_command_reply: tid=0 2024-09-19T08:06:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.349+0000 7f5e3d793740 1 client.16320 shutdown 2024-09-19T08:06:15.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.349+0000 7f5e3d793740 1 -- 172.21.15.98:0/1123132396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 msgr2=0x564504d538d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:15.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.349+0000 7f5e3d793740 1 --2- 172.21.15.98:0/1123132396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504d538d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5e280109f0 tx=0x7f5e2800f870 comp rx=0 tx=0).stop 2024-09-19T08:06:15.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.349+0000 7f5e3d793740 1 -- 172.21.15.98:0/1123132396 shutdown_connections 2024-09-19T08:06:15.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.349+0000 7f5e3d793740 1 --2- 172.21.15.98:0/1123132396 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x564504d60950 0x564504d62d90 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.349+0000 7f5e3d793740 1 --2- 172.21.15.98:0/1123132396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564504ce0d40 0x564504d58740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.349+0000 7f5e1cff9640 0 client.16320 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:15.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.349+0000 7f5e3d793740 1 --2- 172.21.15.98:0/1123132396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564504cd6480 0x564504d53e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.349+0000 7f5e3d793740 1 --2- 172.21.15.98:0/1123132396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564504cd5b60 0x564504d538d0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.349+0000 7f5e3d793740 1 -- 172.21.15.98:0/1123132396 >> 172.21.15.98:0/1123132396 conn(0x564504cd20f0 msgr2=0x564504cd1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:15.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3d793740 1 -- 172.21.15.98:0/1123132396 shutdown_connections 2024-09-19T08:06:15.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3d793740 1 -- 172.21.15.98:0/1123132396 wait complete. 2024-09-19T08:06:15.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3d793740 20 client.16320 trim_cache size 0 max 16384 2024-09-19T08:06:15.355 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:06:15.355 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:06:15.355 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "ca063f4e-89f5-422f-ac32-0bdb6e01c935", 2024-09-19T08:06:15.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:06:15.355 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:06:15.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3b3ba640 1 -- 172.21.15.98:0/1783690165 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5e200a33b0 msgr2=0x7f5e200a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:15.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3b3ba640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5e200a33b0 0x7f5e200a5870 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c031da0 tx=0x7f5e2c048000 comp rx=0 tx=0).stop 2024-09-19T08:06:15.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3b3ba640 1 -- 172.21.15.98:0/1783690165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e34101420 msgr2=0x7f5e34198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:15.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3b3ba640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e34101420 0x7f5e34198960 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000d770 tx=0x7f5e3000db70 comp rx=0 tx=0).stop 2024-09-19T08:06:15.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3b3ba640 1 -- 172.21.15.98:0/1783690165 shutdown_connections 2024-09-19T08:06:15.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3b3ba640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5e200a33b0 0x7f5e200a5870 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3b3ba640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e34106a80 0x7f5e3419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3b3ba640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e34101420 0x7f5e34198960 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3b3ba640 1 --2- 172.21.15.98:0/1783690165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e34100a80 0x7f5e34198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:15.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3b3ba640 1 -- 172.21.15.98:0/1783690165 >> 172.21.15.98:0/1783690165 conn(0x7f5e340fc7a0 msgr2=0x7f5e3406a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:15.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3b3ba640 1 -- 172.21.15.98:0/1783690165 shutdown_connections 2024-09-19T08:06:15.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:15.350+0000 7f5e3b3ba640 1 -- 172.21.15.98:0/1783690165 wait complete. 2024-09-19T08:06:15.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:14 smithi162 ceph-mon[35260]: pgmap v100: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 345 MiB/s wr, 94 op/s 2024-09-19T08:06:16.103 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:06:16.126 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:15 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:06:16.126 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:15 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:06:16.126 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:15 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:06:16.126 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:15 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:06:16.126 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:15 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:06:16.127 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:15 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:06:16.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:15 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:06:16.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:15 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:06:16.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:15 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:06:16.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:15 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:06:16.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:15 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:06:16.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:15 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:06:16.349 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:15 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:06:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:15 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:06:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:15 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:06:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:15 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:06:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:15 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:06:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:15 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:06:16.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.808+0000 7f4c8a15a640 1 -- 172.21.15.98:0/12500267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c84106a80 msgr2=0x7f4c84106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:16.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.808+0000 7f4c8a15a640 1 --2- 172.21.15.98:0/12500267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c84106a80 0x7f4c84106e60 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009930 tx=0x7f4c7402a420 comp rx=0 tx=0).stop 2024-09-19T08:06:16.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.808+0000 7f4c8a15a640 1 -- 172.21.15.98:0/12500267 shutdown_connections 2024-09-19T08:06:16.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.808+0000 7f4c8a15a640 1 --2- 172.21.15.98:0/12500267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c84101420 0x7f4c8410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.808+0000 7f4c8a15a640 1 --2- 172.21.15.98:0/12500267 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c84100a80 0x7f4c84100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.808+0000 7f4c8a15a640 1 --2- 172.21.15.98:0/12500267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c84106a80 0x7f4c84106e60 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.808+0000 7f4c8a15a640 1 -- 172.21.15.98:0/12500267 >> 172.21.15.98:0/12500267 conn(0x7f4c840fc7a0 msgr2=0x7f4c840febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:16.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.808+0000 7f4c8a15a640 1 -- 172.21.15.98:0/12500267 shutdown_connections 2024-09-19T08:06:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.808+0000 7f4c8a15a640 1 -- 172.21.15.98:0/12500267 wait complete. 2024-09-19T08:06:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.809+0000 7f4c8a15a640 1 Processor -- start 2024-09-19T08:06:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.809+0000 7f4c8a15a640 1 -- start start 2024-09-19T08:06:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.809+0000 7f4c8a15a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c84100a80 0x7f4c84198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.809+0000 7f4c8a15a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c84101420 0x7f4c84198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.809+0000 7f4c8a15a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c84106a80 0x7f4c8419ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.809+0000 7f4c8a15a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4c84110f10 con 0x7f4c84100a80 2024-09-19T08:06:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.809+0000 7f4c8a15a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4c84110d90 con 0x7f4c84101420 2024-09-19T08:06:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.809+0000 7f4c8a15a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4c84111090 con 0x7f4c84106a80 2024-09-19T08:06:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.809+0000 7f4c837fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c84100a80 0x7f4c84198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.809+0000 7f4c837fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c84100a80 0x7f4c84198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40754/0 (socket says 172.21.15.98:40754) 2024-09-19T08:06:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.809+0000 7f4c837fe640 1 -- 172.21.15.98:0/1638204252 learned_addr learned my addr 172.21.15.98:0/1638204252 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.809+0000 7f4c83fff640 1 --2- 172.21.15.98:0/1638204252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c84106a80 0x7f4c8419ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.810+0000 7f4c82ffd640 1 --2- 172.21.15.98:0/1638204252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c84101420 0x7f4c84198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.810+0000 7f4c837fe640 1 -- 172.21.15.98:0/1638204252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c84106a80 msgr2=0x7f4c8419ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.810+0000 7f4c837fe640 1 --2- 172.21.15.98:0/1638204252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c84106a80 0x7f4c8419ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.810+0000 7f4c837fe640 1 -- 172.21.15.98:0/1638204252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c84101420 msgr2=0x7f4c84198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.810+0000 7f4c837fe640 1 --2- 172.21.15.98:0/1638204252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c84101420 0x7f4c84198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.810+0000 7f4c837fe640 1 -- 172.21.15.98:0/1638204252 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c8419d4f0 con 0x7f4c84100a80 2024-09-19T08:06:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.810+0000 7f4c837fe640 1 --2- 172.21.15.98:0/1638204252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c84100a80 0x7f4c84198540 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4c7402a940 tx=0x7f4c74003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.810+0000 7f4c80ff9640 1 -- 172.21.15.98:0/1638204252 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7403f070 con 0x7f4c84100a80 2024-09-19T08:06:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.810+0000 7f4c8a15a640 1 -- 172.21.15.98:0/1638204252 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c8419d780 con 0x7f4c84100a80 2024-09-19T08:06:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.810+0000 7f4c80ff9640 1 -- 172.21.15.98:0/1638204252 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c74031c90 con 0x7f4c84100a80 2024-09-19T08:06:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.810+0000 7f4c80ff9640 1 -- 172.21.15.98:0/1638204252 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74035030 con 0x7f4c84100a80 2024-09-19T08:06:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.810+0000 7f4c8a15a640 1 -- 172.21.15.98:0/1638204252 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c841a50c0 con 0x7f4c84100a80 2024-09-19T08:06:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.812+0000 7f4c80ff9640 1 -- 172.21.15.98:0/1638204252 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f4c74042070 con 0x7f4c84100a80 2024-09-19T08:06:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c8c533740 1 Processor -- start 2024-09-19T08:06:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c8c533740 1 -- start start 2024-09-19T08:06:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c8c533740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f1a4a0 0x561ac1f1a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c8c533740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f25140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c8c533740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f25680 0x561ac1f27a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c8c533740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561ac1e802e0 con 0x561ac1f25680 2024-09-19T08:06:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c8c533740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561ac1e27ce0 con 0x561ac1f1adc0 2024-09-19T08:06:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c8c533740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561ac1e7c2a0 con 0x561ac1f1a4a0 2024-09-19T08:06:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c80ff9640 1 --2- 172.21.15.98:0/1638204252 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4c700a33b0 0x7f4c700a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c82ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f25680 0x561ac1f27a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c83fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f1a4a0 0x561ac1f1a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c82ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f25680 0x561ac1f27a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40770/0 (socket says 172.21.15.98:40770) 2024-09-19T08:06:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c82ffd640 1 -- 172.21.15.98:0/376915162 learned_addr learned my addr 172.21.15.98:0/376915162 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c837fe640 1 --2- 172.21.15.98:0/376915162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f25140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c83fff640 1 --2- 172.21.15.98:0/1638204252 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4c700a33b0 0x7f4c700a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c80ff9640 1 -- 172.21.15.98:0/1638204252 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f4c740f0410 con 0x7f4c84100a80 2024-09-19T08:06:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c82ffd640 1 -- 172.21.15.98:0/376915162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f1a4a0 msgr2=0x561ac1f1a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c82ffd640 1 --2- 172.21.15.98:0/376915162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f1a4a0 0x561ac1f1a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c82ffd640 1 -- 172.21.15.98:0/376915162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 msgr2=0x561ac1f25140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c82ffd640 1 --2- 172.21.15.98:0/376915162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f25140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.813+0000 7f4c82ffd640 1 -- 172.21.15.98:0/376915162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ac1e0c440 con 0x561ac1f25680 2024-09-19T08:06:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.814+0000 7f4c82ffd640 1 --2- 172.21.15.98:0/376915162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f25680 0x561ac1f27a70 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00b740 tx=0x7f4c6c00bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a4ec2ea888554e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.814+0000 7f4c83fff640 1 --2- 172.21.15.98:0/1638204252 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4c700a33b0 0x7f4c700a5870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c009900 tx=0x7f4c7c009650 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.814+0000 7f4c79ffb640 1 -- 172.21.15.98:0/376915162 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c003dd0 con 0x561ac1f25680 2024-09-19T08:06:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.814+0000 7f4c79ffb640 1 -- 172.21.15.98:0/376915162 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c6c015070 con 0x561ac1f25680 2024-09-19T08:06:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.814+0000 7f4c79ffb640 1 -- 172.21.15.98:0/376915162 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c004e00 con 0x561ac1f25680 2024-09-19T08:06:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.814+0000 7f4c8c533740 1 -- 172.21.15.98:0/376915162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f25680 msgr2=0x561ac1f27a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.814+0000 7f4c8c533740 1 --2- 172.21.15.98:0/376915162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f25680 0x561ac1f27a70 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00b740 tx=0x7f4c6c00bc00 comp rx=0 tx=0).stop 2024-09-19T08:06:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.814+0000 7f4c8c533740 1 -- 172.21.15.98:0/376915162 shutdown_connections 2024-09-19T08:06:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.814+0000 7f4c8c533740 1 --2- 172.21.15.98:0/376915162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f25680 0x561ac1f27a70 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c8c533740 1 --2- 172.21.15.98:0/376915162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f25140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c8c533740 1 --2- 172.21.15.98:0/376915162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f1a4a0 0x561ac1f1a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c8c533740 1 -- 172.21.15.98:0/376915162 >> 172.21.15.98:0/376915162 conn(0x561ac1f16a30 msgr2=0x561ac1f16e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c8c533740 1 -- 172.21.15.98:0/376915162 shutdown_connections 2024-09-19T08:06:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c8c533740 1 -- 172.21.15.98:0/376915162 wait complete. 2024-09-19T08:06:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c8c533740 1 -- start start 2024-09-19T08:06:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c8c533740 1 Processor -- start 2024-09-19T08:06:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c8c533740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f1a4a0 0x561ac1f24f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c8c533740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f22220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c8c533740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f25680 0x561ac1f22760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c8c533740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561ac1f28250 con 0x561ac1f1a4a0 2024-09-19T08:06:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c8c533740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561ac1e3b050 con 0x561ac1f1adc0 2024-09-19T08:06:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c8c533740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561ac1f283d0 con 0x561ac1f25680 2024-09-19T08:06:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c8c533740 10 client.0 _ll_register_callbacks cb 0x561ac1e7b160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c837fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f1a4a0 0x561ac1f24f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c837fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f1a4a0 0x561ac1f24f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40774/0 (socket says 172.21.15.98:40774) 2024-09-19T08:06:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.815+0000 7f4c837fe640 1 -- 172.21.15.98:0/4009673941 learned_addr learned my addr 172.21.15.98:0/4009673941 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c82ffd640 1 --2- 172.21.15.98:0/4009673941 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f22220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c837fe640 1 -- 172.21.15.98:0/4009673941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f25680 msgr2=0x561ac1f22760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c837fe640 1 --2- 172.21.15.98:0/4009673941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f25680 0x561ac1f22760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c837fe640 1 -- 172.21.15.98:0/4009673941 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 msgr2=0x561ac1f22220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c837fe640 1 --2- 172.21.15.98:0/4009673941 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f22220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c837fe640 1 -- 172.21.15.98:0/4009673941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ac1bf13c0 con 0x561ac1f1a4a0 2024-09-19T08:06:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c837fe640 1 --2- 172.21.15.98:0/4009673941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f1a4a0 0x561ac1f24f70 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4c74000c00 tx=0x7f4c74049e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ee7111f0cc3fc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c62ffd640 1 -- 172.21.15.98:0/4009673941 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74049040 con 0x561ac1f1a4a0 2024-09-19T08:06:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c62ffd640 1 -- 172.21.15.98:0/4009673941 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f4c7403b5f0 con 0x561ac1f1a4a0 2024-09-19T08:06:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c8c533740 1 -- 172.21.15.98:0/4009673941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c6c015070 con 0x561ac1f1a4a0 2024-09-19T08:06:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c62ffd640 1 -- 172.21.15.98:0/4009673941 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4c740bc130 con 0x561ac1f1a4a0 2024-09-19T08:06:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c8c533740 10 client.16332 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c8c533740 1 -- 172.21.15.98:0/4009673941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561ac1f33bc0 con 0x561ac1f1a4a0 2024-09-19T08:06:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c62ffd640 1 -- 172.21.15.98:0/4009673941 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c6c015070 con 0x561ac1f1a4a0 2024-09-19T08:06:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c62ffd640 1 -- 172.21.15.98:0/4009673941 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f4c74087230 con 0x561ac1f1a4a0 2024-09-19T08:06:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c8c533740 10 client.16332 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.816+0000 7f4c8c533740 10 client.16332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:06:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.817+0000 7f4c8c533740 20 client.16332 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.817+0000 7f4c8c533740 1 --2- 172.21.15.98:0/4009673941 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561ac1f35130 0x561ac1f375f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.817+0000 7f4c8c533740 4 client.16332 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.817+0000 7f4c8c533740 1 -- 172.21.15.98:0/4009673941 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ac1e8d850 con 0x561ac1f35130 2024-09-19T08:06:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.817+0000 7f4c82ffd640 1 --2- 172.21.15.98:0/4009673941 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561ac1f35130 0x561ac1f375f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.817+0000 7f4c62ffd640 10 client.16332 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.817+0000 7f4c82ffd640 1 --2- 172.21.15.98:0/4009673941 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561ac1f35130 0x561ac1f375f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6efd2d40a358d03c server_cookie=9584798c02802b61 in_seq=0 out_seq=0 2024-09-19T08:06:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c62ffd640 1 -- 172.21.15.98:0/4009673941 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561ac1e8d850 con 0x561ac1f35130 2024-09-19T08:06:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c62ffd640 10 client.16332 handle_command_reply: tid=0 2024-09-19T08:06:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c8c533740 1 client.16332 shutdown 2024-09-19T08:06:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c8c533740 1 -- 172.21.15.98:0/4009673941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f1a4a0 msgr2=0x561ac1f24f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c8c533740 1 --2- 172.21.15.98:0/4009673941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f1a4a0 0x561ac1f24f70 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4c74000c00 tx=0x7f4c74049e10 comp rx=0 tx=0).stop 2024-09-19T08:06:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c8c533740 1 -- 172.21.15.98:0/4009673941 shutdown_connections 2024-09-19T08:06:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c8c533740 1 --2- 172.21.15.98:0/4009673941 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561ac1f35130 0x561ac1f375f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c8c533740 1 --2- 172.21.15.98:0/4009673941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f25680 0x561ac1f22760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c8c533740 1 --2- 172.21.15.98:0/4009673941 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f22220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c62ffd640 0 client.16332 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c8c533740 1 --2- 172.21.15.98:0/4009673941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f1a4a0 0x561ac1f24f70 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c8c533740 1 -- 172.21.15.98:0/4009673941 >> 172.21.15.98:0/4009673941 conn(0x561ac1f16a30 msgr2=0x561ac1f1c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c8c533740 1 -- 172.21.15.98:0/4009673941 shutdown_connections 2024-09-19T08:06:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c8c533740 1 -- 172.21.15.98:0/4009673941 wait complete. 2024-09-19T08:06:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.818+0000 7f4c8c533740 20 client.16332 trim_cache size 0 max 16384 2024-09-19T08:06:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c8c533740 1 Processor -- start 2024-09-19T08:06:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c8c533740 1 -- start start 2024-09-19T08:06:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c8c533740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f1a4a0 0x561ac1f314b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c8c533740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f3cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c8c533740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f25680 0x561ac1f3d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c8c533740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841ad760 con 0x561ac1f1a4a0 2024-09-19T08:06:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c8c533740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841af300 con 0x561ac1f1adc0 2024-09-19T08:06:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c8c533740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841ade30 con 0x561ac1f25680 2024-09-19T08:06:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c837fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f1a4a0 0x561ac1f314b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c837fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f1a4a0 0x561ac1f314b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40788/0 (socket says 172.21.15.98:40788) 2024-09-19T08:06:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c837fe640 1 -- 172.21.15.98:0/4287770004 learned_addr learned my addr 172.21.15.98:0/4287770004 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c83fff640 1 --2- 172.21.15.98:0/4287770004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f25680 0x561ac1f3d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c82ffd640 1 --2- 172.21.15.98:0/4287770004 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f3cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c837fe640 1 -- 172.21.15.98:0/4287770004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f25680 msgr2=0x561ac1f3d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c837fe640 1 --2- 172.21.15.98:0/4287770004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f25680 0x561ac1f3d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c837fe640 1 -- 172.21.15.98:0/4287770004 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 msgr2=0x561ac1f3cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c837fe640 1 --2- 172.21.15.98:0/4287770004 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f3cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c837fe640 1 -- 172.21.15.98:0/4287770004 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ac1f20770 con 0x561ac1f1a4a0 2024-09-19T08:06:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c83fff640 1 --2- 172.21.15.98:0/4287770004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f25680 0x561ac1f3d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c82ffd640 1 --2- 172.21.15.98:0/4287770004 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f3cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c837fe640 1 --2- 172.21.15.98:0/4287770004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f1a4a0 0x561ac1f314b0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f4c740b8c30 tx=0x7f4c7403b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62fd199921445f02 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.836+0000 7f4c797fa640 1 -- 172.21.15.98:0/4287770004 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74049070 con 0x561ac1f1a4a0 2024-09-19T08:06:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.837+0000 7f4c797fa640 1 -- 172.21.15.98:0/4287770004 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c740b6070 con 0x561ac1f1a4a0 2024-09-19T08:06:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.837+0000 7f4c797fa640 1 -- 172.21.15.98:0/4287770004 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c740b8a70 con 0x561ac1f1a4a0 2024-09-19T08:06:16.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.837+0000 7f4c8c533740 1 -- 172.21.15.98:0/4287770004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f1a4a0 msgr2=0x561ac1f314b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:16.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.837+0000 7f4c8c533740 1 --2- 172.21.15.98:0/4287770004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f1a4a0 0x561ac1f314b0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f4c740b8c30 tx=0x7f4c7403b990 comp rx=0 tx=0).stop 2024-09-19T08:06:16.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.837+0000 7f4c8c533740 1 -- 172.21.15.98:0/4287770004 shutdown_connections 2024-09-19T08:06:16.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.837+0000 7f4c8c533740 1 --2- 172.21.15.98:0/4287770004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f25680 0x561ac1f3d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.837+0000 7f4c8c533740 1 --2- 172.21.15.98:0/4287770004 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f3cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.837+0000 7f4c8c533740 1 --2- 172.21.15.98:0/4287770004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f1a4a0 0x561ac1f314b0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.837+0000 7f4c8c533740 1 -- 172.21.15.98:0/4287770004 >> 172.21.15.98:0/4287770004 conn(0x561ac1f16a30 msgr2=0x561ac1f20af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.837+0000 7f4c8c533740 1 -- 172.21.15.98:0/4287770004 shutdown_connections 2024-09-19T08:06:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.837+0000 7f4c8c533740 1 -- 172.21.15.98:0/4287770004 wait complete. 2024-09-19T08:06:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c8c533740 1 -- start start 2024-09-19T08:06:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c8c533740 1 Processor -- start 2024-09-19T08:06:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c8c533740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f1a4a0 0x561ac1f29360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c8c533740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f298a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c8c533740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f25680 0x561ac1f29de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c8c533740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841aec00 con 0x561ac1f25680 2024-09-19T08:06:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c8c533740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841ae530 con 0x561ac1f1adc0 2024-09-19T08:06:16.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c8c533740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841ab740 con 0x561ac1f1a4a0 2024-09-19T08:06:16.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c8c533740 10 client.0 _ll_register_callbacks cb 0x7f4c74031490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:16.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c83fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f25680 0x561ac1f29de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c83fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f25680 0x561ac1f29de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40796/0 (socket says 172.21.15.98:40796) 2024-09-19T08:06:16.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c83fff640 1 -- 172.21.15.98:0/126616433 learned_addr learned my addr 172.21.15.98:0/126616433 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:16.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c83fff640 1 -- 172.21.15.98:0/126616433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f1a4a0 msgr2=0x561ac1f29360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:16.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c83fff640 1 --2- 172.21.15.98:0/126616433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f1a4a0 0x561ac1f29360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c83fff640 1 -- 172.21.15.98:0/126616433 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 msgr2=0x561ac1f298a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:16.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.838+0000 7f4c83fff640 1 --2- 172.21.15.98:0/126616433 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f298a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c83fff640 1 -- 172.21.15.98:0/126616433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ac1f9eed0 con 0x561ac1f25680 2024-09-19T08:06:16.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c83fff640 1 --2- 172.21.15.98:0/126616433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f25680 0x561ac1f29de0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c017e20 tx=0x7f4c7c004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f4f78c517d31d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:16.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:16.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c62ffd640 1 -- 172.21.15.98:0/126616433 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c02a060 con 0x561ac1f25680 2024-09-19T08:06:16.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c62ffd640 1 -- 172.21.15.98:0/126616433 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f4c7c019790 con 0x561ac1f25680 2024-09-19T08:06:16.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c8c533740 1 -- 172.21.15.98:0/126616433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c740b6070 con 0x561ac1f25680 2024-09-19T08:06:16.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c62ffd640 1 -- 172.21.15.98:0/126616433 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4c7c014040 con 0x561ac1f25680 2024-09-19T08:06:16.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c8c533740 10 client.16338 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:16.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c8c533740 1 -- 172.21.15.98:0/126616433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561ac1fa01c0 con 0x561ac1f25680 2024-09-19T08:06:16.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c62ffd640 1 -- 172.21.15.98:0/126616433 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c740b6070 con 0x561ac1f25680 2024-09-19T08:06:16.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c62ffd640 1 -- 172.21.15.98:0/126616433 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f4c7c025a40 con 0x561ac1f25680 2024-09-19T08:06:16.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c8c533740 10 client.16338 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:16.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c8c533740 10 client.16338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:06:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c8c533740 20 client.16338 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c8c533740 1 --2- 172.21.15.98:0/126616433 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561ac1fa0970 0x561ac1fa2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c8c533740 4 client.16338 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:06:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.839+0000 7f4c8c533740 1 -- 172.21.15.98:0/126616433 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561ac1e87d60 con 0x561ac1fa0970 2024-09-19T08:06:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.840+0000 7f4c837fe640 1 --2- 172.21.15.98:0/126616433 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561ac1fa0970 0x561ac1fa2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.840+0000 7f4c837fe640 1 --2- 172.21.15.98:0/126616433 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561ac1fa0970 0x561ac1fa2e10 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b5591db3061e6d4 server_cookie=e585dd1658d7e7ed in_seq=0 out_seq=0 2024-09-19T08:06:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.840+0000 7f4c62ffd640 10 client.16338 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.840+0000 7f4c62ffd640 1 -- 172.21.15.98:0/126616433 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561ac1e87d60 con 0x561ac1fa0970 2024-09-19T08:06:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.840+0000 7f4c62ffd640 10 client.16338 handle_command_reply: tid=0 2024-09-19T08:06:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.840+0000 7f4c8c533740 1 client.16338 shutdown 2024-09-19T08:06:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.840+0000 7f4c8c533740 1 -- 172.21.15.98:0/126616433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f25680 msgr2=0x561ac1f29de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.840+0000 7f4c8c533740 1 --2- 172.21.15.98:0/126616433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f25680 0x561ac1f29de0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c017e20 tx=0x7f4c7c004b50 comp rx=0 tx=0).stop 2024-09-19T08:06:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.841+0000 7f4c8c533740 1 -- 172.21.15.98:0/126616433 shutdown_connections 2024-09-19T08:06:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.841+0000 7f4c8c533740 1 --2- 172.21.15.98:0/126616433 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561ac1fa0970 0x561ac1fa2e10 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.841+0000 7f4c8c533740 1 --2- 172.21.15.98:0/126616433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561ac1f25680 0x561ac1f29de0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.841+0000 7f4c8c533740 1 --2- 172.21.15.98:0/126616433 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561ac1f1adc0 0x561ac1f298a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.841+0000 7f4c62ffd640 0 client.16338 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.841+0000 7f4c8c533740 1 --2- 172.21.15.98:0/126616433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ac1f1a4a0 0x561ac1f29360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.841+0000 7f4c8c533740 1 -- 172.21.15.98:0/126616433 >> 172.21.15.98:0/126616433 conn(0x561ac1f16a30 msgr2=0x561ac1f20ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.841+0000 7f4c8c533740 1 -- 172.21.15.98:0/126616433 shutdown_connections 2024-09-19T08:06:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.841+0000 7f4c8c533740 1 -- 172.21.15.98:0/126616433 wait complete. 2024-09-19T08:06:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.841+0000 7f4c8c533740 20 client.16338 trim_cache size 0 max 16384 2024-09-19T08:06:16.848 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:06:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-19T08:06:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-09-19T08:06:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: "ca063f4e-89f5-422f-ac32-0bdb6e01c935": { 2024-09-19T08:06:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-09-19T08:06:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "ca063f4e-89f5-422f-ac32-0bdb6e01c935", 2024-09-19T08:06:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-09-19T08:06:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:06:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:06:16.848 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:06:16.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.841+0000 7f4c8a15a640 1 -- 172.21.15.98:0/1638204252 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4c700a33b0 msgr2=0x7f4c700a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:16.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.841+0000 7f4c8a15a640 1 --2- 172.21.15.98:0/1638204252 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4c700a33b0 0x7f4c700a5870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c009900 tx=0x7f4c7c009650 comp rx=0 tx=0).stop 2024-09-19T08:06:16.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.841+0000 7f4c8a15a640 1 -- 172.21.15.98:0/1638204252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c84100a80 msgr2=0x7f4c84198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:16.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.841+0000 7f4c8a15a640 1 --2- 172.21.15.98:0/1638204252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c84100a80 0x7f4c84198540 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4c7402a940 tx=0x7f4c74003b90 comp rx=0 tx=0).stop 2024-09-19T08:06:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.842+0000 7f4c8a15a640 1 -- 172.21.15.98:0/1638204252 shutdown_connections 2024-09-19T08:06:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.842+0000 7f4c8a15a640 1 --2- 172.21.15.98:0/1638204252 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4c700a33b0 0x7f4c700a5870 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.842+0000 7f4c8a15a640 1 --2- 172.21.15.98:0/1638204252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c84106a80 0x7f4c8419ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.842+0000 7f4c8a15a640 1 --2- 172.21.15.98:0/1638204252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c84101420 0x7f4c84198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.842+0000 7f4c8a15a640 1 --2- 172.21.15.98:0/1638204252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c84100a80 0x7f4c84198540 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.842+0000 7f4c8a15a640 1 -- 172.21.15.98:0/1638204252 >> 172.21.15.98:0/1638204252 conn(0x7f4c840fc7a0 msgr2=0x7f4c8406aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.842+0000 7f4c8a15a640 1 -- 172.21.15.98:0/1638204252 shutdown_connections 2024-09-19T08:06:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:16.842+0000 7f4c8a15a640 1 -- 172.21.15.98:0/1638204252 wait complete. 2024-09-19T08:06:17.130 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:16 smithi098 ceph-mon[31333]: pgmap v101: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 345 MiB/s wr, 97 op/s 2024-09-19T08:06:17.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:16 smithi139 ceph-mon[34874]: pgmap v101: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 345 MiB/s wr, 97 op/s 2024-09-19T08:06:17.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:16 smithi162 ceph-mon[35260]: pgmap v101: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 345 MiB/s wr, 97 op/s 2024-09-19T08:06:17.562 INFO:tasks.cephfs.filesystem:scrub status for tag:ca063f4e-89f5-422f-ac32-0bdb6e01c935 - {'path': '/', 'tag': 'ca063f4e-89f5-422f-ac32-0bdb6e01c935', 'options': 'recursive,force'} 2024-09-19T08:06:19.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:18 smithi139 ceph-mon[34874]: pgmap v102: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 163 MiB/s wr, 51 op/s 2024-09-19T08:06:19.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:18 smithi098 ceph-mon[31333]: pgmap v102: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 163 MiB/s wr, 51 op/s 2024-09-19T08:06:19.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:18 smithi162 ceph-mon[35260]: pgmap v102: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 163 MiB/s wr, 51 op/s 2024-09-19T08:06:20.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:20 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:06:20.903 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:06:20.903 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:06:20.904 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-19T08:06:20.904 INFO:tasks.workunit.client.0.smithi098.stdout:iozone test complete. 2024-09-19T08:06:20.905 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-19T08:06:20.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-19T08:06:20.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:20 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:06:20.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:20 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:06:20.943 INFO:tasks.workunit:Stopping ['suites/iozone.sh'] on client.0... 2024-09-19T08:06:20.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-19T08:06:21.278 DEBUG:teuthology.parallel:result is None 2024-09-19T08:06:21.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-19T08:06:21.316 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-19T08:06:21.316 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-09-19T08:06:21.326 INFO:tasks.quiescer:joining Quiescers 2024-09-19T08:06:21.327 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-09-19T08:06:21.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:06:21.602 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:21.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:21 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:06:21.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:21 smithi098 ceph-mon[31333]: pgmap v103: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-09-19T08:06:21.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:21 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:06:21.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:21 smithi162 ceph-mon[35260]: pgmap v103: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-09-19T08:06:21.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:21 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:06:21.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:21 smithi139 ceph-mon[34874]: pgmap v103: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-09-19T08:06:22.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.072+0000 7fb327b52640 1 -- 172.21.15.98:0/3243997953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb320100a80 msgr2=0x7fb320100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:22.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.072+0000 7fb327b52640 1 --2- 172.21.15.98:0/3243997953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb320100a80 0x7fb320100ee0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb310009930 tx=0x7fb31002a420 comp rx=0 tx=0).stop 2024-09-19T08:06:22.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.072+0000 7fb327b52640 1 -- 172.21.15.98:0/3243997953 shutdown_connections 2024-09-19T08:06:22.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.072+0000 7fb327b52640 1 --2- 172.21.15.98:0/3243997953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb320101420 0x7fb32010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:22.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.072+0000 7fb327b52640 1 --2- 172.21.15.98:0/3243997953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb320100a80 0x7fb320100ee0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:22.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.072+0000 7fb327b52640 1 --2- 172.21.15.98:0/3243997953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb320106a80 0x7fb320106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:22.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.072+0000 7fb327b52640 1 -- 172.21.15.98:0/3243997953 >> 172.21.15.98:0/3243997953 conn(0x7fb3200fc7a0 msgr2=0x7fb3200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:22.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.072+0000 7fb327b52640 1 -- 172.21.15.98:0/3243997953 shutdown_connections 2024-09-19T08:06:22.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.072+0000 7fb327b52640 1 -- 172.21.15.98:0/3243997953 wait complete. 2024-09-19T08:06:22.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.073+0000 7fb327b52640 1 Processor -- start 2024-09-19T08:06:22.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.073+0000 7fb327b52640 1 -- start start 2024-09-19T08:06:22.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.073+0000 7fb327b52640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb320100a80 0x7fb320198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:22.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.073+0000 7fb327b52640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb320101420 0x7fb320198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.073+0000 7fb327b52640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb320106a80 0x7fb32019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.073+0000 7fb327b52640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb32006ad60 con 0x7fb320101420 2024-09-19T08:06:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.073+0000 7fb327b52640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb32006abe0 con 0x7fb320100a80 2024-09-19T08:06:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.073+0000 7fb327b52640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb32006aee0 con 0x7fb320106a80 2024-09-19T08:06:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.073+0000 7fb3250ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb320101420 0x7fb320198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.073+0000 7fb3250ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb320101420 0x7fb320198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40814/0 (socket says 172.21.15.98:40814) 2024-09-19T08:06:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.073+0000 7fb3250ed640 1 -- 172.21.15.98:0/4124328332 learned_addr learned my addr 172.21.15.98:0/4124328332 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.074+0000 7fb3258ee640 1 --2- 172.21.15.98:0/4124328332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb320100a80 0x7fb320198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.074+0000 7fb3260ef640 1 --2- 172.21.15.98:0/4124328332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb320106a80 0x7fb32019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.074+0000 7fb3250ed640 1 -- 172.21.15.98:0/4124328332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb320106a80 msgr2=0x7fb32019cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.074+0000 7fb3250ed640 1 --2- 172.21.15.98:0/4124328332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb320106a80 0x7fb32019cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.074+0000 7fb3250ed640 1 -- 172.21.15.98:0/4124328332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb320100a80 msgr2=0x7fb320198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.074+0000 7fb3250ed640 1 --2- 172.21.15.98:0/4124328332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb320100a80 0x7fb320198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.074+0000 7fb3250ed640 1 -- 172.21.15.98:0/4124328332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb32019d4d0 con 0x7fb320101420 2024-09-19T08:06:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.074+0000 7fb3250ed640 1 --2- 172.21.15.98:0/4124328332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb320101420 0x7fb320198a60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb310009230 tx=0x7fb310004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.074+0000 7fb316ffd640 1 -- 172.21.15.98:0/4124328332 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31003f030 con 0x7fb320101420 2024-09-19T08:06:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.074+0000 7fb316ffd640 1 -- 172.21.15.98:0/4124328332 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb310003ad0 con 0x7fb320101420 2024-09-19T08:06:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.074+0000 7fb316ffd640 1 -- 172.21.15.98:0/4124328332 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb310035070 con 0x7fb320101420 2024-09-19T08:06:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.074+0000 7fb327b52640 1 -- 172.21.15.98:0/4124328332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb32019d760 con 0x7fb320101420 2024-09-19T08:06:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.074+0000 7fb327b52640 1 -- 172.21.15.98:0/4124328332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3201a50a0 con 0x7fb320101420 2024-09-19T08:06:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.075+0000 7fb327b52640 1 -- 172.21.15.98:0/4124328332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3201021c0 con 0x7fb320101420 2024-09-19T08:06:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.076+0000 7fb316ffd640 1 -- 172.21.15.98:0/4124328332 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fb310042030 con 0x7fb320101420 2024-09-19T08:06:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.077+0000 7fb316ffd640 1 --2- 172.21.15.98:0/4124328332 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb30c0a33b0 0x7fb30c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.077+0000 7fb3258ee640 1 --2- 172.21.15.98:0/4124328332 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb30c0a33b0 0x7fb30c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.077+0000 7fb316ffd640 1 -- 172.21.15.98:0/4124328332 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fb3100f0570 con 0x7fb320101420 2024-09-19T08:06:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.078+0000 7fb3258ee640 1 --2- 172.21.15.98:0/4124328332 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb30c0a33b0 0x7fb30c0a5870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb31c005fd0 tx=0x7fb31c00b390 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:22.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.080+0000 7fb316ffd640 1 -- 172.21.15.98:0/4124328332 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb3100bc930 con 0x7fb320101420 2024-09-19T08:06:22.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.279+0000 7fb327b52640 1 -- 172.21.15.98:0/4124328332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3201a5270 con 0x7fb320101420 2024-09-19T08:06:22.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.280+0000 7fb316ffd640 1 -- 172.21.15.98:0/4124328332 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11985 (secure 0 0 0) 0x7fb3100c0c30 con 0x7fb320101420 2024-09-19T08:06:22.280 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:06:22.280 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":30,"btime":"2024-09-19T08:05:21:864704+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:05:21.551452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:06:22.280 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 30 2024-09-19T08:06:22.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.283+0000 7fb327b52640 1 -- 172.21.15.98:0/4124328332 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb30c0a33b0 msgr2=0x7fb30c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:22.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.283+0000 7fb327b52640 1 --2- 172.21.15.98:0/4124328332 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb30c0a33b0 0x7fb30c0a5870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb31c005fd0 tx=0x7fb31c00b390 comp rx=0 tx=0).stop 2024-09-19T08:06:22.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.283+0000 7fb327b52640 1 -- 172.21.15.98:0/4124328332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb320101420 msgr2=0x7fb320198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:22.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.283+0000 7fb327b52640 1 --2- 172.21.15.98:0/4124328332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb320101420 0x7fb320198a60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb310009230 tx=0x7fb310004270 comp rx=0 tx=0).stop 2024-09-19T08:06:22.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.283+0000 7fb327b52640 1 -- 172.21.15.98:0/4124328332 shutdown_connections 2024-09-19T08:06:22.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.283+0000 7fb327b52640 1 --2- 172.21.15.98:0/4124328332 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb30c0a33b0 0x7fb30c0a5870 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:22.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.283+0000 7fb327b52640 1 --2- 172.21.15.98:0/4124328332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb320106a80 0x7fb32019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:22.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.283+0000 7fb327b52640 1 --2- 172.21.15.98:0/4124328332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb320101420 0x7fb320198a60 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:22.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.283+0000 7fb327b52640 1 --2- 172.21.15.98:0/4124328332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb320100a80 0x7fb320198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:22.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.283+0000 7fb327b52640 1 -- 172.21.15.98:0/4124328332 >> 172.21.15.98:0/4124328332 conn(0x7fb3200fc7a0 msgr2=0x7fb3200fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:22.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.283+0000 7fb327b52640 1 -- 172.21.15.98:0/4124328332 shutdown_connections 2024-09-19T08:06:22.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:22.283+0000 7fb327b52640 1 -- 172.21.15.98:0/4124328332 wait complete. 2024-09-19T08:06:22.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:22 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/4124328332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:06:22.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:22 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/4124328332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:06:22.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:22 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/4124328332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:06:23.003 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --cancel --all' 2024-09-19T08:06:23.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --cancel --all 2024-09-19T08:06:23.249 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:23.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:23 smithi098 ceph-mon[31333]: pgmap v104: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-09-19T08:06:23.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.751+0000 7f6c00e90640 1 -- 172.21.15.98:0/814481993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bfc1055b0 msgr2=0x7f6bfc10d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:23.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.751+0000 7f6c00e90640 1 --2- 172.21.15.98:0/814481993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bfc1055b0 0x7f6bfc10d130 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4009ca0 tx=0x7f6bf402a8c0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.751+0000 7f6c00e90640 1 -- 172.21.15.98:0/814481993 shutdown_connections 2024-09-19T08:06:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.751+0000 7f6c00e90640 1 --2- 172.21.15.98:0/814481993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bfc1055b0 0x7f6bfc10d130 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.751+0000 7f6c00e90640 1 --2- 172.21.15.98:0/814481993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc100e30 0x7f6bfc104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.751+0000 7f6c00e90640 1 --2- 172.21.15.98:0/814481993 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6bfc100480 0x7f6bfc100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.751+0000 7f6c00e90640 1 -- 172.21.15.98:0/814481993 >> 172.21.15.98:0/814481993 conn(0x7f6bfc077210 msgr2=0x7f6bfc0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.751+0000 7f6c00e90640 1 -- 172.21.15.98:0/814481993 shutdown_connections 2024-09-19T08:06:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.751+0000 7f6c00e90640 1 -- 172.21.15.98:0/814481993 wait complete. 2024-09-19T08:06:23.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.752+0000 7f6c00e90640 1 Processor -- start 2024-09-19T08:06:23.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.752+0000 7f6c00e90640 1 -- start start 2024-09-19T08:06:23.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.752+0000 7f6c00e90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bfc100480 0x7f6bfc1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.752+0000 7f6c00e90640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6bfc100e30 0x7f6bfc198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.752+0000 7f6c00e90640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc1055b0 0x7f6bfc19cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.752+0000 7f6c00e90640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc110e90 con 0x7f6bfc100480 2024-09-19T08:06:23.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.752+0000 7f6c00e90640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc110d10 con 0x7f6bfc100e30 2024-09-19T08:06:23.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.752+0000 7f6c00e90640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc111010 con 0x7f6bfc1055b0 2024-09-19T08:06:23.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.752+0000 7f6bfa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bfc100480 0x7f6bfc1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.752+0000 7f6bfa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bfc100480 0x7f6bfc1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40846/0 (socket says 172.21.15.98:40846) 2024-09-19T08:06:23.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.752+0000 7f6bfa59c640 1 -- 172.21.15.98:0/2644856235 learned_addr learned my addr 172.21.15.98:0/2644856235 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:23.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.753+0000 7f6bf9d9b640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6bfc100e30 0x7f6bfc198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.753+0000 7f6bfa59c640 1 -- 172.21.15.98:0/2644856235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc1055b0 msgr2=0x7f6bfc19cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.753+0000 7f6bfad9d640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc1055b0 0x7f6bfc19cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.753+0000 7f6bfa59c640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc1055b0 0x7f6bfc19cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.753+0000 7f6bfa59c640 1 -- 172.21.15.98:0/2644856235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6bfc100e30 msgr2=0x7f6bfc198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.753+0000 7f6bfa59c640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6bfc100e30 0x7f6bfc198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.753+0000 7f6bfa59c640 1 -- 172.21.15.98:0/2644856235 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bfc19d380 con 0x7f6bfc100480 2024-09-19T08:06:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.753+0000 7f6bf9d9b640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6bfc100e30 0x7f6bfc198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.753+0000 7f6bfa59c640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bfc100480 0x7f6bfc1983d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00d950 tx=0x7f6bec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.753+0000 7f6beb7fe640 1 -- 172.21.15.98:0/2644856235 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec017070 con 0x7f6bfc100480 2024-09-19T08:06:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.753+0000 7f6c00e90640 1 -- 172.21.15.98:0/2644856235 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bfc19d670 con 0x7f6bfc100480 2024-09-19T08:06:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.753+0000 7f6beb7fe640 1 -- 172.21.15.98:0/2644856235 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bec009c30 con 0x7f6bfc100480 2024-09-19T08:06:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.753+0000 7f6beb7fe640 1 -- 172.21.15.98:0/2644856235 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec00f6b0 con 0x7f6bfc100480 2024-09-19T08:06:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.753+0000 7f6c00e90640 1 -- 172.21.15.98:0/2644856235 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bfc1a4f50 con 0x7f6bfc100480 2024-09-19T08:06:23.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.756+0000 7f6beb7fe640 1 -- 172.21.15.98:0/2644856235 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f6bec0109d0 con 0x7f6bfc100480 2024-09-19T08:06:23.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.756+0000 7f6c03269740 1 Processor -- start 2024-09-19T08:06:23.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.756+0000 7f6c03269740 1 -- start start 2024-09-19T08:06:23.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.756+0000 7f6c03269740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba9499a00 0x555ba9499de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.756+0000 7f6c03269740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba949a320 0x555ba94a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6c03269740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba94a4be0 0x555ba94a6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6c03269740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555ba93fe200 con 0x555ba9499a00 2024-09-19T08:06:23.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6c03269740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555ba93a4dc0 con 0x555ba94a4be0 2024-09-19T08:06:23.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6c03269740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555ba93fa1c0 con 0x555ba949a320 2024-09-19T08:06:23.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6bfad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba9499a00 0x555ba9499de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6bfad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba9499a00 0x555ba9499de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40862/0 (socket says 172.21.15.98:40862) 2024-09-19T08:06:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6bfad9d640 1 -- 172.21.15.98:0/2116401913 learned_addr learned my addr 172.21.15.98:0/2116401913 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6bf9d9b640 1 --2- 172.21.15.98:0/2116401913 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba94a4be0 0x555ba94a6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6bfa59c640 1 --2- 172.21.15.98:0/2116401913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba949a320 0x555ba94a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6bfad9d640 1 -- 172.21.15.98:0/2116401913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba949a320 msgr2=0x555ba94a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6bfad9d640 1 --2- 172.21.15.98:0/2116401913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba949a320 0x555ba94a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6bfad9d640 1 -- 172.21.15.98:0/2116401913 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba94a4be0 msgr2=0x555ba94a6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6bfad9d640 1 --2- 172.21.15.98:0/2116401913 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba94a4be0 0x555ba94a6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6bfad9d640 1 -- 172.21.15.98:0/2116401913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ba916f620 con 0x555ba9499a00 2024-09-19T08:06:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6beb7fe640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6be40a33b0 0x7f6be40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6bfad9d640 1 --2- 172.21.15.98:0/2116401913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba9499a00 0x555ba9499de0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4005cc0 tx=0x7f6bf4005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdce59e3f5be43f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6bfa59c640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6be40a33b0 0x7f6be40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.757+0000 7f6be8ff9640 1 -- 172.21.15.98:0/2116401913 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4031db0 con 0x555ba9499a00 2024-09-19T08:06:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.758+0000 7f6beb7fe640 1 -- 172.21.15.98:0/2644856235 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f6bec0cfa30 con 0x7f6bfc100480 2024-09-19T08:06:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.758+0000 7f6be8ff9640 1 -- 172.21.15.98:0/2116401913 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bf4036070 con 0x555ba9499a00 2024-09-19T08:06:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.758+0000 7f6be8ff9640 1 -- 172.21.15.98:0/2116401913 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf40042a0 con 0x555ba9499a00 2024-09-19T08:06:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.758+0000 7f6c03269740 1 -- 172.21.15.98:0/2116401913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba9499a00 msgr2=0x555ba9499de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.758+0000 7f6bfa59c640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6be40a33b0 0x7f6be40a5870 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6bec0cd260 tx=0x7f6bec086040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.758+0000 7f6c03269740 1 --2- 172.21.15.98:0/2116401913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba9499a00 0x555ba9499de0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4005cc0 tx=0x7f6bf4005f70 comp rx=0 tx=0).stop 2024-09-19T08:06:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.758+0000 7f6c03269740 1 -- 172.21.15.98:0/2116401913 shutdown_connections 2024-09-19T08:06:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.758+0000 7f6c03269740 1 --2- 172.21.15.98:0/2116401913 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba94a4be0 0x555ba94a6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.758+0000 7f6c03269740 1 --2- 172.21.15.98:0/2116401913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba949a320 0x555ba94a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.758+0000 7f6c03269740 1 --2- 172.21.15.98:0/2116401913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba9499a00 0x555ba9499de0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.758+0000 7f6c03269740 1 -- 172.21.15.98:0/2116401913 >> 172.21.15.98:0/2116401913 conn(0x555ba9495f90 msgr2=0x555ba9496370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6c03269740 1 -- 172.21.15.98:0/2116401913 shutdown_connections 2024-09-19T08:06:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6c03269740 1 -- 172.21.15.98:0/2116401913 wait complete. 2024-09-19T08:06:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6c03269740 1 -- start start 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6c03269740 1 Processor -- start 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6c03269740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba9499a00 0x555ba94a44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6c03269740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6c03269740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba94a4be0 0x555ba94a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6c03269740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555ba94a77b0 con 0x555ba949a320 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6c03269740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555ba9398c50 con 0x555ba94a4be0 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6c03269740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555ba94a7930 con 0x555ba9499a00 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6c03269740 10 client.0 _ll_register_callbacks cb 0x555ba93aaba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6bf9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6bf9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40868/0 (socket says 172.21.15.98:40868) 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6bf9d9b640 1 -- 172.21.15.98:0/1692378544 learned_addr learned my addr 172.21.15.98:0/1692378544 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.759+0000 7f6bfad9d640 1 --2- 172.21.15.98:0/1692378544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba9499a00 0x555ba94a44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6bf9d9b640 1 -- 172.21.15.98:0/1692378544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba9499a00 msgr2=0x555ba94a44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6bf9d9b640 1 --2- 172.21.15.98:0/1692378544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba9499a00 0x555ba94a44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6bf9d9b640 1 -- 172.21.15.98:0/1692378544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba94a4be0 msgr2=0x555ba94a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6bf9d9b640 1 --2- 172.21.15.98:0/1692378544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba94a4be0 0x555ba94a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6bf9d9b640 1 -- 172.21.15.98:0/1692378544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ba94a24c0 con 0x555ba949a320 2024-09-19T08:06:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6bfad9d640 1 --2- 172.21.15.98:0/1692378544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba9499a00 0x555ba94a44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6bf9d9b640 1 --2- 172.21.15.98:0/1692378544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a1750 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000bf80 tx=0x7f6bf00128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87516007c1abef03 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6be1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6be1ffb640 1 -- 172.21.15.98:0/1692378544 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf0005220 con 0x555ba949a320 2024-09-19T08:06:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6c03269740 1 -- 172.21.15.98:0/1692378544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bf4036070 con 0x555ba949a320 2024-09-19T08:06:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6be1ffb640 1 -- 172.21.15.98:0/1692378544 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f6bf00053c0 con 0x555ba949a320 2024-09-19T08:06:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6be1ffb640 1 -- 172.21.15.98:0/1692378544 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6bf00055c0 con 0x555ba949a320 2024-09-19T08:06:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6c03269740 10 client.16356 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6c03269740 1 -- 172.21.15.98:0/1692378544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555ba94b3320 con 0x555ba949a320 2024-09-19T08:06:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6be1ffb640 1 -- 172.21.15.98:0/1692378544 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bf4036070 con 0x555ba949a320 2024-09-19T08:06:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6be1ffb640 1 -- 172.21.15.98:0/1692378544 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f6bf0019030 con 0x555ba949a320 2024-09-19T08:06:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6c03269740 10 client.16356 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.760+0000 7f6c03269740 10 client.16356 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-19T08:06:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.761+0000 7f6c03269740 20 client.16356 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.761+0000 7f6c03269740 1 --2- 172.21.15.98:0/1692378544 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555ba94b4cd0 0x555ba94b7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.761+0000 7f6c03269740 4 client.16356 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.761+0000 7f6c03269740 1 -- 172.21.15.98:0/1692378544 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ba940b770 con 0x555ba94b4cd0 2024-09-19T08:06:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.761+0000 7f6bfad9d640 1 --2- 172.21.15.98:0/1692378544 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555ba94b4cd0 0x555ba94b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.761+0000 7f6be1ffb640 10 client.16356 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.761+0000 7f6bfad9d640 1 --2- 172.21.15.98:0/1692378544 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555ba94b4cd0 0x555ba94b7110 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ab13686c621e997 server_cookie=4b1d277621a01dc1 in_seq=0 out_seq=0 2024-09-19T08:06:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.762+0000 7f6be1ffb640 1 -- 172.21.15.98:0/1692378544 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555ba940b770 con 0x555ba94b4cd0 2024-09-19T08:06:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.762+0000 7f6be1ffb640 10 client.16356 handle_command_reply: tid=0 2024-09-19T08:06:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.762+0000 7f6c03269740 1 client.16356 shutdown 2024-09-19T08:06:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.762+0000 7f6c03269740 1 -- 172.21.15.98:0/1692378544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 msgr2=0x555ba94a1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.762+0000 7f6c03269740 1 --2- 172.21.15.98:0/1692378544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a1750 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000bf80 tx=0x7f6bf00128e0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.762+0000 7f6c03269740 1 -- 172.21.15.98:0/1692378544 shutdown_connections 2024-09-19T08:06:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.762+0000 7f6c03269740 1 --2- 172.21.15.98:0/1692378544 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555ba94b4cd0 0x555ba94b7110 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.762+0000 7f6c03269740 1 --2- 172.21.15.98:0/1692378544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba94a4be0 0x555ba94a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.762+0000 7f6c03269740 1 --2- 172.21.15.98:0/1692378544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a1750 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.762+0000 7f6be1ffb640 0 client.16356 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.762+0000 7f6c03269740 1 --2- 172.21.15.98:0/1692378544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba9499a00 0x555ba94a44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.762+0000 7f6c03269740 1 -- 172.21.15.98:0/1692378544 >> 172.21.15.98:0/1692378544 conn(0x555ba9495f90 msgr2=0x555ba94951d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.762+0000 7f6c03269740 1 -- 172.21.15.98:0/1692378544 shutdown_connections 2024-09-19T08:06:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.762+0000 7f6c03269740 1 -- 172.21.15.98:0/1692378544 wait complete. 2024-09-19T08:06:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.763+0000 7f6c03269740 20 client.16356 trim_cache size 0 max 16384 2024-09-19T08:06:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.779+0000 7f6c03269740 1 Processor -- start 2024-09-19T08:06:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.779+0000 7f6c03269740 1 -- start start 2024-09-19T08:06:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.779+0000 7f6c03269740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba9499a00 0x555ba949ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.779+0000 7f6c03269740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.779+0000 7f6c03269740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba94a4be0 0x555ba94b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.779+0000 7f6c03269740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc1ad7b0 con 0x555ba949a320 2024-09-19T08:06:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.779+0000 7f6c03269740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc1af350 con 0x555ba9499a00 2024-09-19T08:06:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.779+0000 7f6c03269740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc1ade80 con 0x555ba94a4be0 2024-09-19T08:06:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.780+0000 7f6bf9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.780+0000 7f6bf9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40872/0 (socket says 172.21.15.98:40872) 2024-09-19T08:06:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.780+0000 7f6bf9d9b640 1 -- 172.21.15.98:0/3754371039 learned_addr learned my addr 172.21.15.98:0/3754371039 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.780+0000 7f6bfad9d640 1 --2- 172.21.15.98:0/3754371039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba9499a00 0x555ba949ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.780+0000 7f6bf9d9b640 1 -- 172.21.15.98:0/3754371039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba94a4be0 msgr2=0x555ba94b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.780+0000 7f6bf9d9b640 1 --2- 172.21.15.98:0/3754371039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba94a4be0 0x555ba94b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.780+0000 7f6bf9d9b640 1 -- 172.21.15.98:0/3754371039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba9499a00 msgr2=0x555ba949ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.780+0000 7f6bf9d9b640 1 --2- 172.21.15.98:0/3754371039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba9499a00 0x555ba949ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.780+0000 7f6bf9d9b640 1 -- 172.21.15.98:0/3754371039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ba949a150 con 0x555ba949a320 2024-09-19T08:06:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.780+0000 7f6bfad9d640 1 --2- 172.21.15.98:0/3754371039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba9499a00 0x555ba949ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.780+0000 7f6bf9d9b640 1 --2- 172.21.15.98:0/3754371039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a0020 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0000c00 tx=0x7f6bf000b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b811b668458c396 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.780+0000 7f6be3fff640 1 -- 172.21.15.98:0/3754371039 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf00109c0 con 0x555ba949a320 2024-09-19T08:06:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.780+0000 7f6be3fff640 1 -- 172.21.15.98:0/3754371039 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bf0030070 con 0x555ba949a320 2024-09-19T08:06:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.780+0000 7f6be3fff640 1 -- 172.21.15.98:0/3754371039 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf002b660 con 0x555ba949a320 2024-09-19T08:06:23.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 -- 172.21.15.98:0/3754371039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 msgr2=0x555ba94a0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:23.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 --2- 172.21.15.98:0/3754371039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a0020 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0000c00 tx=0x7f6bf000b7f0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 -- 172.21.15.98:0/3754371039 shutdown_connections 2024-09-19T08:06:23.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 --2- 172.21.15.98:0/3754371039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba94a4be0 0x555ba94b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 --2- 172.21.15.98:0/3754371039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a0020 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 --2- 172.21.15.98:0/3754371039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba9499a00 0x555ba949ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 -- 172.21.15.98:0/3754371039 >> 172.21.15.98:0/3754371039 conn(0x555ba9495f90 msgr2=0x555ba94a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 -- 172.21.15.98:0/3754371039 shutdown_connections 2024-09-19T08:06:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 -- 172.21.15.98:0/3754371039 wait complete. 2024-09-19T08:06:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 -- start start 2024-09-19T08:06:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 Processor -- start 2024-09-19T08:06:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba9499a00 0x555ba94ad010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba94a4be0 0x555ba94a88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc1aec50 con 0x555ba949a320 2024-09-19T08:06:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc1ae580 con 0x555ba9499a00 2024-09-19T08:06:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.781+0000 7f6c03269740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc1ab790 con 0x555ba94a4be0 2024-09-19T08:06:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6c03269740 10 client.0 _ll_register_callbacks cb 0x7f6bf000bb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6bf9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6bf9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40876/0 (socket says 172.21.15.98:40876) 2024-09-19T08:06:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6bf9d9b640 1 -- 172.21.15.98:0/2784171085 learned_addr learned my addr 172.21.15.98:0/2784171085 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6bfad9d640 1 --2- 172.21.15.98:0/2784171085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba94a4be0 0x555ba94a88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6bfad9d640 1 --2- 172.21.15.98:0/2784171085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba9499a00 0x555ba94ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6bf9d9b640 1 -- 172.21.15.98:0/2784171085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba94a4be0 msgr2=0x555ba94a88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6bf9d9b640 1 --2- 172.21.15.98:0/2784171085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba94a4be0 0x555ba94a88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6bf9d9b640 1 -- 172.21.15.98:0/2784171085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba9499a00 msgr2=0x555ba94ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6bf9d9b640 1 --2- 172.21.15.98:0/2784171085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba9499a00 0x555ba94ad010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6bf9d9b640 1 -- 172.21.15.98:0/2784171085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ba94a8e10 con 0x555ba949a320 2024-09-19T08:06:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6bf9d9b640 1 --2- 172.21.15.98:0/2784171085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a8390 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0010f40 tx=0x7f6bf0013ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a8d8b5c1b58371 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6be1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6be1ffb640 1 -- 172.21.15.98:0/2784171085 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf003a070 con 0x555ba949a320 2024-09-19T08:06:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6be1ffb640 1 -- 172.21.15.98:0/2784171085 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f6bf0023500 con 0x555ba949a320 2024-09-19T08:06:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6c03269740 1 -- 172.21.15.98:0/2784171085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bf0030070 con 0x555ba949a320 2024-09-19T08:06:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.782+0000 7f6be1ffb640 1 -- 172.21.15.98:0/2784171085 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6bf0023700 con 0x555ba949a320 2024-09-19T08:06:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.783+0000 7f6c03269740 10 client.16362 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.783+0000 7f6c03269740 1 -- 172.21.15.98:0/2784171085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555ba951f690 con 0x555ba949a320 2024-09-19T08:06:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.783+0000 7f6be1ffb640 1 -- 172.21.15.98:0/2784171085 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bf0030070 con 0x555ba949a320 2024-09-19T08:06:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.783+0000 7f6be1ffb640 1 -- 172.21.15.98:0/2784171085 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f6bf0019040 con 0x555ba949a320 2024-09-19T08:06:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.783+0000 7f6c03269740 10 client.16362 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.783+0000 7f6c03269740 10 client.16362 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-19T08:06:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.783+0000 7f6c03269740 20 client.16362 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.783+0000 7f6c03269740 1 --2- 172.21.15.98:0/2784171085 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555ba94b4cd0 0x555ba9521e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.783+0000 7f6c03269740 4 client.16362 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-09-19T08:06:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.783+0000 7f6c03269740 1 -- 172.21.15.98:0/2784171085 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x555ba9405c80 con 0x555ba94b4cd0 2024-09-19T08:06:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.783+0000 7f6bfad9d640 1 --2- 172.21.15.98:0/2784171085 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555ba94b4cd0 0x555ba9521e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6bfad9d640 1 --2- 172.21.15.98:0/2784171085 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555ba94b4cd0 0x555ba9521e10 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be797367352c22fd server_cookie=71c4bb6c0ec82e29 in_seq=0 out_seq=0 2024-09-19T08:06:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6be1ffb640 10 client.16362 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6be1ffb640 1 -- 172.21.15.98:0/2784171085 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x555ba9405c80 con 0x555ba94b4cd0 2024-09-19T08:06:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6be1ffb640 10 client.16362 handle_command_reply: tid=0 2024-09-19T08:06:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6c03269740 1 client.16362 shutdown 2024-09-19T08:06:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6c03269740 1 -- 172.21.15.98:0/2784171085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 msgr2=0x555ba94a8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6c03269740 1 --2- 172.21.15.98:0/2784171085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a8390 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0010f40 tx=0x7f6bf0013ec0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6c03269740 1 -- 172.21.15.98:0/2784171085 shutdown_connections 2024-09-19T08:06:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6c03269740 1 --2- 172.21.15.98:0/2784171085 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555ba94b4cd0 0x555ba9521e10 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6c03269740 1 --2- 172.21.15.98:0/2784171085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ba94a4be0 0x555ba94a88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6c03269740 1 --2- 172.21.15.98:0/2784171085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555ba949a320 0x555ba94a8390 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6be1ffb640 0 client.16362 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6c03269740 1 --2- 172.21.15.98:0/2784171085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555ba9499a00 0x555ba94ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6c03269740 1 -- 172.21.15.98:0/2784171085 >> 172.21.15.98:0/2784171085 conn(0x555ba9495f90 msgr2=0x555ba94a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6c03269740 1 -- 172.21.15.98:0/2784171085 shutdown_connections 2024-09-19T08:06:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6c03269740 1 -- 172.21.15.98:0/2784171085 wait complete. 2024-09-19T08:06:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.784+0000 7f6c03269740 20 client.16362 trim_cache size 0 max 16384 2024-09-19T08:06:23.792 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:06:23.792 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 30, 2024-09-19T08:06:23.792 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24485, 2024-09-19T08:06:23.792 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 12, 2024-09-19T08:06:23.792 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": {} 2024-09-19T08:06:23.792 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:06:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.785+0000 7f6c00e90640 1 -- 172.21.15.98:0/2644856235 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6be40a33b0 msgr2=0x7f6be40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.785+0000 7f6c00e90640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6be40a33b0 0x7f6be40a5870 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6bec0cd260 tx=0x7f6bec086040 comp rx=0 tx=0).stop 2024-09-19T08:06:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.785+0000 7f6c00e90640 1 -- 172.21.15.98:0/2644856235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bfc100480 msgr2=0x7f6bfc1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.785+0000 7f6c00e90640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bfc100480 0x7f6bfc1983d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00d950 tx=0x7f6bec00de10 comp rx=0 tx=0).stop 2024-09-19T08:06:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.785+0000 7f6c00e90640 1 -- 172.21.15.98:0/2644856235 shutdown_connections 2024-09-19T08:06:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.785+0000 7f6c00e90640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6be40a33b0 0x7f6be40a5870 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.785+0000 7f6c00e90640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc1055b0 0x7f6bfc19cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.785+0000 7f6c00e90640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6bfc100e30 0x7f6bfc198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.785+0000 7f6c00e90640 1 --2- 172.21.15.98:0/2644856235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bfc100480 0x7f6bfc1983d0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.785+0000 7f6c00e90640 1 -- 172.21.15.98:0/2644856235 >> 172.21.15.98:0/2644856235 conn(0x7f6bfc077210 msgr2=0x7f6bfc10db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.785+0000 7f6c00e90640 1 -- 172.21.15.98:0/2644856235 shutdown_connections 2024-09-19T08:06:23.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:23.785+0000 7f6c00e90640 1 -- 172.21.15.98:0/2644856235 wait complete. 2024-09-19T08:06:23.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:23 smithi162 ceph-mon[35260]: pgmap v104: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-09-19T08:06:23.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:23 smithi139 ceph-mon[34874]: pgmap v104: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-09-19T08:06:24.695 INFO:tasks.quiescer:done joining Quiescers 2024-09-19T08:06:24.695 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-09-19T08:06:24.705 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-09-19T08:06:24.705 INFO:teuthology.task.exec:Executing custom commands... 2024-09-19T08:06:24.705 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-09-19T08:06:24.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-09-19T08:06:24.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-09-19T08:06:24.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:24 smithi098 ceph-mon[31333]: pgmap v105: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-09-19T08:06:25.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:24 smithi139 ceph-mon[34874]: pgmap v105: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-09-19T08:06:25.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:24 smithi162 ceph-mon[35260]: pgmap v105: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-09-19T08:06:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:26 smithi139 ceph-mon[34874]: pgmap v106: 97 pgs: 97 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.7 KiB/s wr, 124 op/s 2024-09-19T08:06:27.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:26 smithi098 ceph-mon[31333]: pgmap v106: 97 pgs: 97 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.7 KiB/s wr, 124 op/s 2024-09-19T08:06:27.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:26 smithi162 ceph-mon[35260]: pgmap v106: 97 pgs: 97 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.7 KiB/s wr, 124 op/s 2024-09-19T08:06:29.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:28 smithi139 ceph-mon[34874]: pgmap v107: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 KiB/s wr, 226 op/s 2024-09-19T08:06:29.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:28 smithi098 ceph-mon[31333]: pgmap v107: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 KiB/s wr, 226 op/s 2024-09-19T08:06:29.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:28 smithi162 ceph-mon[35260]: pgmap v107: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 KiB/s wr, 226 op/s 2024-09-19T08:06:31.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:30 smithi139 ceph-mon[34874]: pgmap v108: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 224 op/s 2024-09-19T08:06:31.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:30 smithi098 ceph-mon[31333]: pgmap v108: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 224 op/s 2024-09-19T08:06:31.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:30 smithi162 ceph-mon[35260]: pgmap v108: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 224 op/s 2024-09-19T08:06:33.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:32 smithi139 ceph-mon[34874]: pgmap v109: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.9 KiB/s wr, 229 op/s 2024-09-19T08:06:33.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:32 smithi098 ceph-mon[31333]: pgmap v109: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.9 KiB/s wr, 229 op/s 2024-09-19T08:06:33.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:32 smithi162 ceph-mon[35260]: pgmap v109: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.9 KiB/s wr, 229 op/s 2024-09-19T08:06:35.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:34 smithi139 ceph-mon[34874]: pgmap v110: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.6 KiB/s wr, 223 op/s 2024-09-19T08:06:35.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:34 smithi098 ceph-mon[31333]: pgmap v110: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.6 KiB/s wr, 223 op/s 2024-09-19T08:06:35.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:34 smithi162 ceph-mon[35260]: pgmap v110: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.6 KiB/s wr, 223 op/s 2024-09-19T08:06:37.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:36 smithi139 ceph-mon[34874]: pgmap v111: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.6 KiB/s wr, 226 op/s 2024-09-19T08:06:37.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:36 smithi098 ceph-mon[31333]: pgmap v111: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.6 KiB/s wr, 226 op/s 2024-09-19T08:06:37.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:36 smithi162 ceph-mon[35260]: pgmap v111: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.6 KiB/s wr, 226 op/s 2024-09-19T08:06:40.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:39 smithi139 ceph-mon[34874]: pgmap v112: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.3 KiB/s wr, 113 op/s 2024-09-19T08:06:40.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:39 smithi098 ceph-mon[31333]: pgmap v112: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.3 KiB/s wr, 113 op/s 2024-09-19T08:06:40.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:39 smithi162 ceph-mon[35260]: pgmap v112: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.3 KiB/s wr, 113 op/s 2024-09-19T08:06:42.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:41 smithi139 ceph-mon[34874]: pgmap v113: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-09-19T08:06:42.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:41 smithi098 ceph-mon[31333]: pgmap v113: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-09-19T08:06:42.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:41 smithi162 ceph-mon[35260]: pgmap v113: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-09-19T08:06:44.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:43 smithi139 ceph-mon[34874]: pgmap v114: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-09-19T08:06:44.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:43 smithi098 ceph-mon[31333]: pgmap v114: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-09-19T08:06:44.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:43 smithi162 ceph-mon[35260]: pgmap v114: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-09-19T08:06:46.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:45 smithi139 ceph-mon[34874]: pgmap v115: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:06:46.185 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:45 smithi098 ceph-mon[31333]: pgmap v115: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:06:46.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:45 smithi162 ceph-mon[35260]: pgmap v115: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:06:47.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:06:47.827 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:48.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:47 smithi139 ceph-mon[34874]: pgmap v116: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-19T08:06:48.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:47 smithi098 ceph-mon[31333]: pgmap v116: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-19T08:06:48.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.291+0000 7fa7a5638640 1 -- 172.21.15.98:0/4187303165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0102570 msgr2=0x7fa7a01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.291+0000 7fa7a5638640 1 --2- 172.21.15.98:0/4187303165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0102570 0x7fa7a01029b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa788009900 tx=0x7fa78802a480 comp rx=0 tx=0).stop 2024-09-19T08:06:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.291+0000 7fa7a5638640 1 -- 172.21.15.98:0/4187303165 shutdown_connections 2024-09-19T08:06:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.291+0000 7fa7a5638640 1 --2- 172.21.15.98:0/4187303165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7a0102ef0 0x7fa7a010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.291+0000 7fa7a5638640 1 --2- 172.21.15.98:0/4187303165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0102570 0x7fa7a01029b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.291+0000 7fa7a5638640 1 --2- 172.21.15.98:0/4187303165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7a0100e20 0x7fa7a0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.291+0000 7fa7a5638640 1 -- 172.21.15.98:0/4187303165 >> 172.21.15.98:0/4187303165 conn(0x7fa7a00fc7d0 msgr2=0x7fa7a00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.291+0000 7fa7a5638640 1 -- 172.21.15.98:0/4187303165 shutdown_connections 2024-09-19T08:06:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.291+0000 7fa7a5638640 1 -- 172.21.15.98:0/4187303165 wait complete. 2024-09-19T08:06:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.291+0000 7fa7a5638640 1 Processor -- start 2024-09-19T08:06:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa7a5638640 1 -- start start 2024-09-19T08:06:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa7a5638640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0100e20 0x7fa7a0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa7a5638640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7a0102570 0x7fa7a0198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa7a5638640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7a0102ef0 0x7fa7a019ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa7a5638640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a0110f50 con 0x7fa7a0100e20 2024-09-19T08:06:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa7a5638640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a0110dd0 con 0x7fa7a0102ef0 2024-09-19T08:06:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa7a5638640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01110d0 con 0x7fa7a0102570 2024-09-19T08:06:48.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa79effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0100e20 0x7fa7a0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa79effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0100e20 0x7fa7a0198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33962/0 (socket says 172.21.15.98:33962) 2024-09-19T08:06:48.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa79effd640 1 -- 172.21.15.98:0/2256669262 learned_addr learned my addr 172.21.15.98:0/2256669262 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7a0102570 0x7fa7a0198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa79f7fe640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7a0102ef0 0x7fa7a019ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa79effd640 1 -- 172.21.15.98:0/2256669262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7a0102570 msgr2=0x7fa7a0198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa79effd640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7a0102570 0x7fa7a0198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa79effd640 1 -- 172.21.15.98:0/2256669262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7a0102ef0 msgr2=0x7fa7a019ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa79effd640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7a0102ef0 0x7fa7a019ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa79effd640 1 -- 172.21.15.98:0/2256669262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a019d500 con 0x7fa7a0100e20 2024-09-19T08:06:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7a0102570 0x7fa7a0198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.292+0000 7fa79f7fe640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7a0102ef0 0x7fa7a019ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.293+0000 7fa79effd640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0100e20 0x7fa7a0198550 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fa79000c890 tx=0x7fa79000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.293+0000 7fa797fff640 1 -- 172.21.15.98:0/2256669262 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790004220 con 0x7fa7a0100e20 2024-09-19T08:06:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.293+0000 7fa7a5638640 1 -- 172.21.15.98:0/2256669262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a019d7f0 con 0x7fa7a0100e20 2024-09-19T08:06:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.293+0000 7fa797fff640 1 -- 172.21.15.98:0/2256669262 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7900043c0 con 0x7fa7a0100e20 2024-09-19T08:06:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.293+0000 7fa797fff640 1 -- 172.21.15.98:0/2256669262 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900049e0 con 0x7fa7a0100e20 2024-09-19T08:06:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.293+0000 7fa7a5638640 1 -- 172.21.15.98:0/2256669262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a01a50d0 con 0x7fa7a0100e20 2024-09-19T08:06:48.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.295+0000 7fa797fff640 1 -- 172.21.15.98:0/2256669262 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fa790004c40 con 0x7fa7a0100e20 2024-09-19T08:06:48.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.296+0000 7fa797fff640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa78c0a3430 0x7fa78c0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.296+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa78c0a3430 0x7fa78c0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.296+0000 7fa797fff640 1 -- 172.21.15.98:0/2256669262 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fa7900cfdc0 con 0x7fa7a0100e20 2024-09-19T08:06:48.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.296+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa78c0a3430 0x7fa78c0a58f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0100050 tx=0x7fa788005a20 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:48.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.296+0000 7fa7a7a11740 1 Processor -- start 2024-09-19T08:06:48.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa7a7a11740 1 -- start start 2024-09-19T08:06:48.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa7a7a11740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af18880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa7a7a11740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 0x56347af23140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa7a7a11740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af25a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa7a7a11740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56347ae7e2e0 con 0x56347af184a0 2024-09-19T08:06:48.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa7a7a11740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56347ae25ce0 con 0x56347af23680 2024-09-19T08:06:48.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa7a7a11740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56347ae7a2a0 con 0x56347af18dc0 2024-09-19T08:06:48.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa79effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af18880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa79effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af18880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33966/0 (socket says 172.21.15.98:33966) 2024-09-19T08:06:48.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa79effd640 1 -- 172.21.15.98:0/585988407 learned_addr learned my addr 172.21.15.98:0/585988407 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:48.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/585988407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 0x56347af23140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa79effd640 1 -- 172.21.15.98:0/585988407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 msgr2=0x56347af23140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:48.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa79effd640 1 --2- 172.21.15.98:0/585988407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 0x56347af23140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa79f7fe640 1 --2- 172.21.15.98:0/585988407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af25a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa79effd640 1 -- 172.21.15.98:0/585988407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 msgr2=0x56347af25a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:48.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.297+0000 7fa79effd640 1 --2- 172.21.15.98:0/585988407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af25a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.298+0000 7fa79effd640 1 -- 172.21.15.98:0/585988407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56347ae0a440 con 0x56347af184a0 2024-09-19T08:06:48.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.298+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/585988407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 0x56347af23140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:48.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.298+0000 7fa79f7fe640 1 --2- 172.21.15.98:0/585988407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af25a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:48.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.298+0000 7fa79effd640 1 --2- 172.21.15.98:0/585988407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af18880 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fa7900a4810 tx=0x7fa79009ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11645de0630d57e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:48.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.298+0000 7fa7957fa640 1 -- 172.21.15.98:0/585988407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79001d040 con 0x56347af184a0 2024-09-19T08:06:48.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.298+0000 7fa7957fa640 1 -- 172.21.15.98:0/585988407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7900a0480 con 0x56347af184a0 2024-09-19T08:06:48.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.298+0000 7fa7957fa640 1 -- 172.21.15.98:0/585988407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790098740 con 0x56347af184a0 2024-09-19T08:06:48.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.299+0000 7fa7a7a11740 1 -- 172.21.15.98:0/585988407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 msgr2=0x56347af18880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:48.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.299+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/585988407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af18880 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fa7900a4810 tx=0x7fa79009ba30 comp rx=0 tx=0).stop 2024-09-19T08:06:48.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.299+0000 7fa7a7a11740 1 -- 172.21.15.98:0/585988407 shutdown_connections 2024-09-19T08:06:48.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.299+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/585988407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af25a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.299+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/585988407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 0x56347af23140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.299+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/585988407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af18880 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.299+0000 7fa7a7a11740 1 -- 172.21.15.98:0/585988407 >> 172.21.15.98:0/585988407 conn(0x56347af14a30 msgr2=0x56347af14e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:48.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.299+0000 7fa7a7a11740 1 -- 172.21.15.98:0/585988407 shutdown_connections 2024-09-19T08:06:48.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.299+0000 7fa7a7a11740 1 -- 172.21.15.98:0/585988407 wait complete. 2024-09-19T08:06:48.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.299+0000 7fa7a7a11740 1 -- start start 2024-09-19T08:06:48.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.299+0000 7fa7a7a11740 1 Processor -- start 2024-09-19T08:06:48.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa7a7a11740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af22fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa7a7a11740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 0x56347af20260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa7a7a11740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af207a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa7a7a11740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56347af26250 con 0x56347af184a0 2024-09-19T08:06:48.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa7a7a11740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56347ae39050 con 0x56347af23680 2024-09-19T08:06:48.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa7a7a11740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56347af263d0 con 0x56347af18dc0 2024-09-19T08:06:48.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa79effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af22fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa79effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af22fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33982/0 (socket says 172.21.15.98:33982) 2024-09-19T08:06:48.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa79effd640 1 -- 172.21.15.98:0/2782629123 learned_addr learned my addr 172.21.15.98:0/2782629123 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:48.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa7a7a11740 10 client.0 _ll_register_callbacks cb 0x56347ae79160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:48.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa79f7fe640 1 --2- 172.21.15.98:0/2782629123 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af207a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/2782629123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 0x56347af20260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa79effd640 1 -- 172.21.15.98:0/2782629123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 msgr2=0x56347af20260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:48.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa79effd640 1 --2- 172.21.15.98:0/2782629123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 0x56347af20260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa79effd640 1 -- 172.21.15.98:0/2782629123 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 msgr2=0x56347af207a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:48.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa79effd640 1 --2- 172.21.15.98:0/2782629123 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af207a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa79effd640 1 -- 172.21.15.98:0/2782629123 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56347abef3c0 con 0x56347af184a0 2024-09-19T08:06:48.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.300+0000 7fa79f7fe640 1 --2- 172.21.15.98:0/2782629123 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af207a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:48.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa79effd640 1 --2- 172.21.15.98:0/2782629123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af22fb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa790098530 tx=0x7fa7900a03b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616fd4bce431a02d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:48.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7867fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:48.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7867fc640 1 -- 172.21.15.98:0/2782629123 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79001d070 con 0x56347af184a0 2024-09-19T08:06:48.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7867fc640 1 -- 172.21.15.98:0/2782629123 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fa7900a48b0 con 0x56347af184a0 2024-09-19T08:06:48.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7a7a11740 1 -- 172.21.15.98:0/2782629123 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7900a0480 con 0x56347af184a0 2024-09-19T08:06:48.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7867fc640 1 -- 172.21.15.98:0/2782629123 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa7900a4ab0 con 0x56347af184a0 2024-09-19T08:06:48.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7a7a11740 10 client.16374 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:48.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7a7a11740 1 -- 172.21.15.98:0/2782629123 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56347af31b70 con 0x56347af184a0 2024-09-19T08:06:48.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7867fc640 1 -- 172.21.15.98:0/2782629123 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7900a0480 con 0x56347af184a0 2024-09-19T08:06:48.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7867fc640 1 -- 172.21.15.98:0/2782629123 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fa79009b640 con 0x56347af184a0 2024-09-19T08:06:48.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7a7a11740 10 client.16374 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:48.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7a7a11740 10 client.16374 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:06:48.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7a7a11740 20 client.16374 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:48.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/2782629123 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56347af32ec0 0x56347af352b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7a7a11740 4 client.16374 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:48.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.301+0000 7fa7a7a11740 1 -- 172.21.15.98:0/2782629123 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56347ae8b850 con 0x56347af32ec0 2024-09-19T08:06:48.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.302+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/2782629123 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56347af32ec0 0x56347af352b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.302+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/2782629123 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56347af32ec0 0x56347af352b0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2726fb5bc3e52668 server_cookie=5fa56c204c274b9a in_seq=0 out_seq=0 2024-09-19T08:06:48.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.302+0000 7fa7867fc640 10 client.16374 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:48.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.303+0000 7fa7867fc640 1 -- 172.21.15.98:0/2782629123 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56347ae8b850 con 0x56347af32ec0 2024-09-19T08:06:48.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.303+0000 7fa7867fc640 10 client.16374 handle_command_reply: tid=0 2024-09-19T08:06:48.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.303+0000 7fa7a7a11740 1 client.16374 shutdown 2024-09-19T08:06:48.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.303+0000 7fa7a7a11740 1 -- 172.21.15.98:0/2782629123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 msgr2=0x56347af22fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:48.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.303+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/2782629123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af22fb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa790098530 tx=0x7fa7900a03b0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.303+0000 7fa7a7a11740 1 -- 172.21.15.98:0/2782629123 shutdown_connections 2024-09-19T08:06:48.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.303+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/2782629123 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56347af32ec0 0x56347af352b0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.303+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/2782629123 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af207a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.303+0000 7fa7867fc640 0 client.16374 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:48.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.303+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/2782629123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 0x56347af20260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.303+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/2782629123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af22fb0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.303+0000 7fa7a7a11740 1 -- 172.21.15.98:0/2782629123 >> 172.21.15.98:0/2782629123 conn(0x56347af14a30 msgr2=0x56347af13b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:48.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.303+0000 7fa7a7a11740 1 -- 172.21.15.98:0/2782629123 shutdown_connections 2024-09-19T08:06:48.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.303+0000 7fa7a7a11740 1 -- 172.21.15.98:0/2782629123 wait complete. 2024-09-19T08:06:48.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.304+0000 7fa7a7a11740 20 client.16374 trim_cache size 0 max 16384 2024-09-19T08:06:48.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.337+0000 7fa7a7a11740 1 Processor -- start 2024-09-19T08:06:48.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.337+0000 7fa7a7a11740 1 -- start start 2024-09-19T08:06:48.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa7a7a11740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af184a0 0x56347af382d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa7a7a11740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af18dc0 0x56347af38810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa7a7a11740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af23680 0x56347af1d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa7a7a11740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01ad880 con 0x56347af18dc0 2024-09-19T08:06:48.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa7a7a11740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01af420 con 0x56347af184a0 2024-09-19T08:06:48.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa7a7a11740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01adf50 con 0x56347af23680 2024-09-19T08:06:48.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa79e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af18dc0 0x56347af38810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa79e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af18dc0 0x56347af38810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33998/0 (socket says 172.21.15.98:33998) 2024-09-19T08:06:48.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa79e7fc640 1 -- 172.21.15.98:0/3952955050 learned_addr learned my addr 172.21.15.98:0/3952955050 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:48.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa79f7fe640 1 --2- 172.21.15.98:0/3952955050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af23680 0x56347af1d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa79effd640 1 --2- 172.21.15.98:0/3952955050 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af184a0 0x56347af382d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa79e7fc640 1 -- 172.21.15.98:0/3952955050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af23680 msgr2=0x56347af1d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:48.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/3952955050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af23680 0x56347af1d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa79e7fc640 1 -- 172.21.15.98:0/3952955050 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af184a0 msgr2=0x56347af382d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:48.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/3952955050 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af184a0 0x56347af382d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.338+0000 7fa79e7fc640 1 -- 172.21.15.98:0/3952955050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56347af15ae0 con 0x56347af18dc0 2024-09-19T08:06:48.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.339+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/3952955050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af18dc0 0x56347af38810 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa78802a9a0 tx=0x7fa78803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133ec4ff2ef3c41a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:48.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.339+0000 7fa794ff9640 1 -- 172.21.15.98:0/3952955050 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa788035050 con 0x56347af18dc0 2024-09-19T08:06:48.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.339+0000 7fa794ff9640 1 -- 172.21.15.98:0/3952955050 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa788056070 con 0x56347af18dc0 2024-09-19T08:06:48.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.339+0000 7fa794ff9640 1 -- 172.21.15.98:0/3952955050 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa788032550 con 0x56347af18dc0 2024-09-19T08:06:48.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.339+0000 7fa7a7a11740 1 -- 172.21.15.98:0/3952955050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af18dc0 msgr2=0x56347af38810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:48.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.339+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/3952955050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af18dc0 0x56347af38810 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa78802a9a0 tx=0x7fa78803e000 comp rx=0 tx=0).stop 2024-09-19T08:06:48.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 -- 172.21.15.98:0/3952955050 shutdown_connections 2024-09-19T08:06:48.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/3952955050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af23680 0x56347af1d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/3952955050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af18dc0 0x56347af38810 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/3952955050 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af184a0 0x56347af382d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 -- 172.21.15.98:0/3952955050 >> 172.21.15.98:0/3952955050 conn(0x56347af14a30 msgr2=0x56347af1a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:48.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 -- 172.21.15.98:0/3952955050 shutdown_connections 2024-09-19T08:06:48.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 -- 172.21.15.98:0/3952955050 wait complete. 2024-09-19T08:06:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 -- start start 2024-09-19T08:06:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 Processor -- start 2024-09-19T08:06:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af36250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 0x56347af36790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af36cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01aed20 con 0x56347af184a0 2024-09-19T08:06:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01ae650 con 0x56347af23680 2024-09-19T08:06:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01ab860 con 0x56347af18dc0 2024-09-19T08:06:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa7a7a11740 10 client.0 _ll_register_callbacks cb 0x7fa79009bf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa79effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af36250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.340+0000 7fa79effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af36250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34014/0 (socket says 172.21.15.98:34014) 2024-09-19T08:06:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa79effd640 1 -- 172.21.15.98:0/169022975 learned_addr learned my addr 172.21.15.98:0/169022975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 0x56347af36790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa79f7fe640 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af36cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa79effd640 1 -- 172.21.15.98:0/169022975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 msgr2=0x56347af36790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:48.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa79effd640 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 0x56347af36790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa79effd640 1 -- 172.21.15.98:0/169022975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 msgr2=0x56347af36cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:48.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa79effd640 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af36cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa79effd640 1 -- 172.21.15.98:0/169022975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56347afa0e10 con 0x56347af184a0 2024-09-19T08:06:48.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa79f7fe640 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af36cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:48.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 0x56347af36790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:48.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa79effd640 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af36250 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fa79009bdd0 tx=0x7fa790098950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5058898807f9b253 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:48.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa7867fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:48.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa7867fc640 1 -- 172.21.15.98:0/169022975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79001d040 con 0x56347af184a0 2024-09-19T08:06:48.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa7867fc640 1 -- 172.21.15.98:0/169022975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fa7900a1700 con 0x56347af184a0 2024-09-19T08:06:48.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa7a7a11740 1 -- 172.21.15.98:0/169022975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa788056070 con 0x56347af184a0 2024-09-19T08:06:48.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa7867fc640 1 -- 172.21.15.98:0/169022975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa79009a490 con 0x56347af184a0 2024-09-19T08:06:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa7a7a11740 10 client.16380 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.341+0000 7fa7a7a11740 1 -- 172.21.15.98:0/169022975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56347afa20e0 con 0x56347af184a0 2024-09-19T08:06:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.342+0000 7fa7867fc640 1 -- 172.21.15.98:0/169022975 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa788056070 con 0x56347af184a0 2024-09-19T08:06:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.342+0000 7fa7867fc640 1 -- 172.21.15.98:0/169022975 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fa7900b01a0 con 0x56347af184a0 2024-09-19T08:06:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.342+0000 7fa7a7a11740 10 client.16380 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.342+0000 7fa7a7a11740 10 client.16380 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:06:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.342+0000 7fa7a7a11740 20 client.16380 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.342+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56347af32ec0 0x56347afa45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.342+0000 7fa7a7a11740 4 client.16380 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:06:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.342+0000 7fa7a7a11740 1 -- 172.21.15.98:0/169022975 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56347ae85d60 con 0x56347af32ec0 2024-09-19T08:06:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.342+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56347af32ec0 0x56347afa45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa79e7fc640 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56347af32ec0 0x56347afa45c0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41db00831a927904 server_cookie=bccea7fc53b59359 in_seq=0 out_seq=0 2024-09-19T08:06:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7867fc640 10 client.16380 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7867fc640 1 -- 172.21.15.98:0/169022975 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56347ae85d60 con 0x56347af32ec0 2024-09-19T08:06:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7867fc640 10 client.16380 handle_command_reply: tid=0 2024-09-19T08:06:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7a7a11740 1 client.16380 shutdown 2024-09-19T08:06:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7a7a11740 1 -- 172.21.15.98:0/169022975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 msgr2=0x56347af36250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af36250 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fa79009bdd0 tx=0x7fa790098950 comp rx=0 tx=0).stop 2024-09-19T08:06:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7a7a11740 1 -- 172.21.15.98:0/169022975 shutdown_connections 2024-09-19T08:06:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56347af32ec0 0x56347afa45c0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56347af23680 0x56347af36cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56347af18dc0 0x56347af36790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7a7a11740 1 --2- 172.21.15.98:0/169022975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56347af184a0 0x56347af36250 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7a7a11740 1 -- 172.21.15.98:0/169022975 >> 172.21.15.98:0/169022975 conn(0x56347af14a30 msgr2=0x56347af21ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7867fc640 0 client.16380 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7a7a11740 1 -- 172.21.15.98:0/169022975 shutdown_connections 2024-09-19T08:06:48.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7a7a11740 1 -- 172.21.15.98:0/169022975 wait complete. 2024-09-19T08:06:48.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.343+0000 7fa7a7a11740 20 client.16380 trim_cache size 0 max 16384 2024-09-19T08:06:48.350 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:06:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:06:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:06:48.350 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:06:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.344+0000 7fa7a5638640 1 -- 172.21.15.98:0/2256669262 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa78c0a3430 msgr2=0x7fa78c0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.344+0000 7fa7a5638640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa78c0a3430 0x7fa78c0a58f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0100050 tx=0x7fa788005a20 comp rx=0 tx=0).stop 2024-09-19T08:06:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.344+0000 7fa7a5638640 1 -- 172.21.15.98:0/2256669262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0100e20 msgr2=0x7fa7a0198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.344+0000 7fa7a5638640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0100e20 0x7fa7a0198550 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fa79000c890 tx=0x7fa79000cd50 comp rx=0 tx=0).stop 2024-09-19T08:06:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.344+0000 7fa7a5638640 1 -- 172.21.15.98:0/2256669262 shutdown_connections 2024-09-19T08:06:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.344+0000 7fa7a5638640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa78c0a3430 0x7fa78c0a58f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.344+0000 7fa7a5638640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7a0102ef0 0x7fa7a019ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.344+0000 7fa7a5638640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7a0102570 0x7fa7a0198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.344+0000 7fa7a5638640 1 --2- 172.21.15.98:0/2256669262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0100e20 0x7fa7a0198550 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.344+0000 7fa7a5638640 1 -- 172.21.15.98:0/2256669262 >> 172.21.15.98:0/2256669262 conn(0x7fa7a00fc7d0 msgr2=0x7fa7a006a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.344+0000 7fa7a5638640 1 -- 172.21.15.98:0/2256669262 shutdown_connections 2024-09-19T08:06:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:48.344+0000 7fa7a5638640 1 -- 172.21.15.98:0/2256669262 wait complete. 2024-09-19T08:06:48.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:47 smithi162 ceph-mon[35260]: pgmap v116: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-19T08:06:49.205 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:06:49.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:06:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:49.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.903+0000 7f8e4bd18640 1 -- 172.21.15.98:0/720629072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e44100460 msgr2=0x7f8e44100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:49.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.903+0000 7f8e4bd18640 1 --2- 172.21.15.98:0/720629072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e44100460 0x7f8e44100840 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8e40009900 tx=0x7f8e4002a410 comp rx=0 tx=0).stop 2024-09-19T08:06:49.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 -- 172.21.15.98:0/720629072 shutdown_connections 2024-09-19T08:06:49.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 --2- 172.21.15.98:0/720629072 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e44105590 0x7f8e4410d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 --2- 172.21.15.98:0/720629072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e44100e10 0x7f8e44104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 --2- 172.21.15.98:0/720629072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e44100460 0x7f8e44100840 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 -- 172.21.15.98:0/720629072 >> 172.21.15.98:0/720629072 conn(0x7f8e440fbee0 msgr2=0x7f8e440fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 -- 172.21.15.98:0/720629072 shutdown_connections 2024-09-19T08:06:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 -- 172.21.15.98:0/720629072 wait complete. 2024-09-19T08:06:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 Processor -- start 2024-09-19T08:06:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 -- start start 2024-09-19T08:06:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e44100460 0x7f8e441992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e44100e10 0x7f8e441997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e44105590 0x7f8e441a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e44110e70 con 0x7f8e44100e10 2024-09-19T08:06:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8e44110cf0 con 0x7f8e44100460 2024-09-19T08:06:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.904+0000 7f8e4bd18640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8e44110ff0 con 0x7f8e44105590 2024-09-19T08:06:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e492b3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e44100e10 0x7f8e441997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e492b3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e44100e10 0x7f8e441997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34026/0 (socket says 172.21.15.98:34026) 2024-09-19T08:06:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e492b3640 1 -- 172.21.15.98:0/3479905252 learned_addr learned my addr 172.21.15.98:0/3479905252 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e4a2b5640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e44105590 0x7f8e441a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e49ab4640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e44100460 0x7f8e441992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e492b3640 1 -- 172.21.15.98:0/3479905252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e44105590 msgr2=0x7f8e441a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e492b3640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e44105590 0x7f8e441a4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e492b3640 1 -- 172.21.15.98:0/3479905252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e44100460 msgr2=0x7f8e441992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e492b3640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e44100460 0x7f8e441992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e492b3640 1 -- 172.21.15.98:0/3479905252 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e441a4870 con 0x7f8e44100e10 2024-09-19T08:06:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e49ab4640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e44100460 0x7f8e441992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e492b3640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e44100e10 0x7f8e441997f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400d950 tx=0x7f8e3400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:49.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e3affd640 1 -- 172.21.15.98:0/3479905252 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34017070 con 0x7f8e44100e10 2024-09-19T08:06:49.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e3affd640 1 -- 172.21.15.98:0/3479905252 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e34009c30 con 0x7f8e44100e10 2024-09-19T08:06:49.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e4bd18640 1 -- 172.21.15.98:0/3479905252 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e441a4a40 con 0x7f8e44100e10 2024-09-19T08:06:49.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e3affd640 1 -- 172.21.15.98:0/3479905252 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3400f6b0 con 0x7f8e44100e10 2024-09-19T08:06:49.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.905+0000 7f8e4bd18640 1 -- 172.21.15.98:0/3479905252 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e441a4f80 con 0x7f8e44100e10 2024-09-19T08:06:49.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.906+0000 7f8e4bd18640 1 -- 172.21.15.98:0/3479905252 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e44100840 con 0x7f8e44100e10 2024-09-19T08:06:49.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.907+0000 7f8e3affd640 1 -- 172.21.15.98:0/3479905252 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f8e340109d0 con 0x7f8e44100e10 2024-09-19T08:06:49.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.908+0000 7f8e3affd640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8e300a3430 0x7f8e300a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:49.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.908+0000 7f8e49ab4640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8e300a3430 0x7f8e300a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:49.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.908+0000 7f8e3affd640 1 -- 172.21.15.98:0/3479905252 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f8e340cf8c0 con 0x7f8e44100e10 2024-09-19T08:06:49.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.908+0000 7f8e49ab4640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8e300a3430 0x7f8e300a58f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8e40005ab0 tx=0x7f8e40005a20 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:49.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:49.912+0000 7f8e3affd640 1 -- 172.21.15.98:0/3479905252 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f8e34009de0 con 0x7f8e44100e10 2024-09-19T08:06:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:50.114+0000 7f8e4bd18640 1 -- 172.21.15.98:0/3479905252 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e44108770 con 0x7f8e44100e10 2024-09-19T08:06:50.115 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:49 smithi098 ceph-mon[31333]: pgmap v117: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:06:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:50.114+0000 7f8e3affd640 1 -- 172.21.15.98:0/3479905252 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11985 (secure 0 0 0) 0x7f8e3409c350 con 0x7f8e44100e10 2024-09-19T08:06:50.116 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:06:50.116 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":30,"btime":"2024-09-19T08:05:21:864704+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:05:21.551452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:06:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 30 2024-09-19T08:06:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:50.116+0000 7f8e4bd18640 1 -- 172.21.15.98:0/3479905252 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8e300a3430 msgr2=0x7f8e300a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:50.116+0000 7f8e4bd18640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8e300a3430 0x7f8e300a58f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8e40005ab0 tx=0x7f8e40005a20 comp rx=0 tx=0).stop 2024-09-19T08:06:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:50.116+0000 7f8e4bd18640 1 -- 172.21.15.98:0/3479905252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e44100e10 msgr2=0x7f8e441997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:50.116+0000 7f8e4bd18640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e44100e10 0x7f8e441997f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400d950 tx=0x7f8e3400de10 comp rx=0 tx=0).stop 2024-09-19T08:06:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:50.116+0000 7f8e4bd18640 1 -- 172.21.15.98:0/3479905252 shutdown_connections 2024-09-19T08:06:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:50.116+0000 7f8e4bd18640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8e300a3430 0x7f8e300a58f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:50.116+0000 7f8e4bd18640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e44105590 0x7f8e441a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:50.116+0000 7f8e4bd18640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e44100e10 0x7f8e441997f0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:50.116+0000 7f8e4bd18640 1 --2- 172.21.15.98:0/3479905252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e44100460 0x7f8e441992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:50.116+0000 7f8e4bd18640 1 -- 172.21.15.98:0/3479905252 >> 172.21.15.98:0/3479905252 conn(0x7f8e440fbee0 msgr2=0x7f8e440fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:50.117+0000 7f8e4bd18640 1 -- 172.21.15.98:0/3479905252 shutdown_connections 2024-09-19T08:06:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:50.117+0000 7f8e4bd18640 1 -- 172.21.15.98:0/3479905252 wait complete. 2024-09-19T08:06:50.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:49 smithi139 ceph-mon[34874]: pgmap v117: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:06:50.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:49 smithi162 ceph-mon[35260]: pgmap v117: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:06:50.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:06:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:51.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:50 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3479905252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:06:51.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:50 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3479905252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:06:51.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:50 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3479905252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:06:51.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.519+0000 7f1920fe1640 1 -- 172.21.15.98:0/3398459006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191c101420 msgr2=0x7f191c10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:51.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.519+0000 7f1920fe1640 1 --2- 172.21.15.98:0/3398459006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191c101420 0x7f191c10d950 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f1914009cd0 tx=0x7f191402a8e0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.519+0000 7f1920fe1640 1 -- 172.21.15.98:0/3398459006 shutdown_connections 2024-09-19T08:06:51.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.519+0000 7f1920fe1640 1 --2- 172.21.15.98:0/3398459006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191c101420 0x7f191c10d950 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.519+0000 7f1920fe1640 1 --2- 172.21.15.98:0/3398459006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f191c100a80 0x7f191c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.519+0000 7f1920fe1640 1 --2- 172.21.15.98:0/3398459006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f191c106a80 0x7f191c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.519+0000 7f1920fe1640 1 -- 172.21.15.98:0/3398459006 >> 172.21.15.98:0/3398459006 conn(0x7f191c0fc7a0 msgr2=0x7f191c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:51.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.519+0000 7f1920fe1640 1 -- 172.21.15.98:0/3398459006 shutdown_connections 2024-09-19T08:06:51.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.519+0000 7f1920fe1640 1 -- 172.21.15.98:0/3398459006 wait complete. 2024-09-19T08:06:51.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.520+0000 7f1920fe1640 1 Processor -- start 2024-09-19T08:06:51.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.520+0000 7f1920fe1640 1 -- start start 2024-09-19T08:06:51.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.520+0000 7f1920fe1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f191c100a80 0x7f191c1991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.520+0000 7f1920fe1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f191c101420 0x7f191c199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.520+0000 7f1920fe1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191c106a80 0x7f191c1a4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f1920fe1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f191c110e80 con 0x7f191c106a80 2024-09-19T08:06:51.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f1920fe1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f191c110d00 con 0x7f191c101420 2024-09-19T08:06:51.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f1920fe1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f191c111000 con 0x7f191c100a80 2024-09-19T08:06:51.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f191ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191c106a80 0x7f191c1a4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f191ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191c106a80 0x7f191c1a4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34038/0 (socket says 172.21.15.98:34038) 2024-09-19T08:06:51.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f191ad9d640 1 -- 172.21.15.98:0/1372648230 learned_addr learned my addr 172.21.15.98:0/1372648230 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:51.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f1919d9b640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f191c101420 0x7f191c199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f191a59c640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f191c100a80 0x7f191c1991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f191ad9d640 1 -- 172.21.15.98:0/1372648230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f191c100a80 msgr2=0x7f191c1991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:51.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f191ad9d640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f191c100a80 0x7f191c1991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f191ad9d640 1 -- 172.21.15.98:0/1372648230 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f191c101420 msgr2=0x7f191c199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:51.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f191ad9d640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f191c101420 0x7f191c199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f191ad9d640 1 -- 172.21.15.98:0/1372648230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f191c1a47a0 con 0x7f191c106a80 2024-09-19T08:06:51.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f191a59c640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f191c100a80 0x7f191c1991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:51.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f191ad9d640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191c106a80 0x7f191c1a4260 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1914006010 tx=0x7f1914003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:51.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f190b7fe640 1 -- 172.21.15.98:0/1372648230 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1914004450 con 0x7f191c106a80 2024-09-19T08:06:51.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f190b7fe640 1 -- 172.21.15.98:0/1372648230 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1914003e10 con 0x7f191c106a80 2024-09-19T08:06:51.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f1920fe1640 1 -- 172.21.15.98:0/1372648230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f191c1a4970 con 0x7f191c106a80 2024-09-19T08:06:51.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f190b7fe640 1 -- 172.21.15.98:0/1372648230 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1914046480 con 0x7f191c106a80 2024-09-19T08:06:51.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.521+0000 7f1920fe1640 1 -- 172.21.15.98:0/1372648230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f191c1a4e20 con 0x7f191c106a80 2024-09-19T08:06:51.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.524+0000 7f190b7fe640 1 -- 172.21.15.98:0/1372648230 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f19140045f0 con 0x7f191c106a80 2024-09-19T08:06:51.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.524+0000 7f19233ba740 1 Processor -- start 2024-09-19T08:06:51.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.524+0000 7f19233ba740 1 -- start start 2024-09-19T08:06:51.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.524+0000 7f19233ba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564992716b40 0x564992715090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.524+0000 7f19233ba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649927155d0 0x564992715a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.524+0000 7f19233ba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564992724780 0x564992726b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.524+0000 7f19233ba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56499267b2f0 con 0x564992724780 2024-09-19T08:06:51.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.524+0000 7f19233ba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564992622cf0 con 0x564992716b40 2024-09-19T08:06:51.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.524+0000 7f19233ba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5649926772b0 con 0x5649927155d0 2024-09-19T08:06:51.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f1919d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564992724780 0x564992726b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f190b7fe640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f19040a33b0 0x7f19040a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f1919d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564992724780 0x564992726b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34046/0 (socket says 172.21.15.98:34046) 2024-09-19T08:06:51.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f1919d9b640 1 -- 172.21.15.98:0/2303404515 learned_addr learned my addr 172.21.15.98:0/2303404515 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:51.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f191a59c640 1 --2- 172.21.15.98:0/2303404515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649927155d0 0x564992715a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f191ad9d640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f19040a33b0 0x7f19040a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f1919d9b640 1 --2- 172.21.15.98:0/2303404515 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564992716b40 0x564992715090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f190b7fe640 1 -- 172.21.15.98:0/1372648230 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f191402e030 con 0x7f191c106a80 2024-09-19T08:06:51.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f1919d9b640 1 -- 172.21.15.98:0/2303404515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649927155d0 msgr2=0x564992715a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:51.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f1919d9b640 1 --2- 172.21.15.98:0/2303404515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649927155d0 0x564992715a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f1919d9b640 1 -- 172.21.15.98:0/2303404515 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564992716b40 msgr2=0x564992715090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:51.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f1919d9b640 1 --2- 172.21.15.98:0/2303404515 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564992716b40 0x564992715090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f1919d9b640 1 -- 172.21.15.98:0/2303404515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649926073a0 con 0x564992724780 2024-09-19T08:06:51.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f1919d9b640 1 --2- 172.21.15.98:0/2303404515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564992724780 0x564992726b70 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f19100115e0 tx=0x7f191000a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5223f66f258163f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:51.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f191ad9d640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f19040a33b0 0x7f19040a5870 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f19140bd6b0 tx=0x7f191402a8e0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:51.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f1908ff9640 1 -- 172.21.15.98:0/2303404515 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910005220 con 0x564992724780 2024-09-19T08:06:51.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f1908ff9640 1 -- 172.21.15.98:0/2303404515 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1910004c80 con 0x564992724780 2024-09-19T08:06:51.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.525+0000 7f1908ff9640 1 -- 172.21.15.98:0/2303404515 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191000fc40 con 0x564992724780 2024-09-19T08:06:51.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.526+0000 7f19233ba740 1 -- 172.21.15.98:0/2303404515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564992724780 msgr2=0x564992726b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:51.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.526+0000 7f19233ba740 1 --2- 172.21.15.98:0/2303404515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564992724780 0x564992726b70 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f19100115e0 tx=0x7f191000a810 comp rx=0 tx=0).stop 2024-09-19T08:06:51.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.526+0000 7f19233ba740 1 -- 172.21.15.98:0/2303404515 shutdown_connections 2024-09-19T08:06:51.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.526+0000 7f19233ba740 1 --2- 172.21.15.98:0/2303404515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564992724780 0x564992726b70 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.526+0000 7f19233ba740 1 --2- 172.21.15.98:0/2303404515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649927155d0 0x564992715a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.526+0000 7f19233ba740 1 --2- 172.21.15.98:0/2303404515 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564992716b40 0x564992715090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.526+0000 7f19233ba740 1 -- 172.21.15.98:0/2303404515 >> 172.21.15.98:0/2303404515 conn(0x5649926de310 msgr2=0x564992711c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:51.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.526+0000 7f19233ba740 1 -- 172.21.15.98:0/2303404515 shutdown_connections 2024-09-19T08:06:51.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.526+0000 7f19233ba740 1 -- 172.21.15.98:0/2303404515 wait complete. 2024-09-19T08:06:51.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.526+0000 7f19233ba740 1 -- start start 2024-09-19T08:06:51.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f19233ba740 1 Processor -- start 2024-09-19T08:06:51.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f19233ba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5649927155d0 0x564992724140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f19233ba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564992716b40 0x5649927213f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f19233ba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564992724780 0x5649927219e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f19233ba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564992727480 con 0x564992716b40 2024-09-19T08:06:51.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f19233ba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564992636060 con 0x5649927155d0 2024-09-19T08:06:51.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f19233ba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564992727600 con 0x564992724780 2024-09-19T08:06:51.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f19233ba740 10 client.0 _ll_register_callbacks cb 0x5649923898c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:51.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f191a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564992716b40 0x5649927213f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f191a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564992716b40 0x5649927213f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34062/0 (socket says 172.21.15.98:34062) 2024-09-19T08:06:51.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f191a59c640 1 -- 172.21.15.98:0/2461050682 learned_addr learned my addr 172.21.15.98:0/2461050682 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:51.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f1919d9b640 1 --2- 172.21.15.98:0/2461050682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564992724780 0x5649927219e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f1919d9b640 1 --2- 172.21.15.98:0/2461050682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5649927155d0 0x564992724140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f191a59c640 1 -- 172.21.15.98:0/2461050682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564992724780 msgr2=0x5649927219e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:51.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f191a59c640 1 --2- 172.21.15.98:0/2461050682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564992724780 0x5649927219e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f191a59c640 1 -- 172.21.15.98:0/2461050682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5649927155d0 msgr2=0x564992724140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:51.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f191a59c640 1 --2- 172.21.15.98:0/2461050682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5649927155d0 0x564992724140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f191a59c640 1 -- 172.21.15.98:0/2461050682 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649923ec3c0 con 0x564992716b40 2024-09-19T08:06:51.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f1919d9b640 1 --2- 172.21.15.98:0/2461050682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564992724780 0x5649927219e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:51.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f191a59c640 1 --2- 172.21.15.98:0/2461050682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564992716b40 0x5649927213f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f190c011830 tx=0x7f190c011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f6ea29f135862d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:51.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.527+0000 7f1901ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:51.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.528+0000 7f1901ffb640 1 -- 172.21.15.98:0/2461050682 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c00fd50 con 0x564992716b40 2024-09-19T08:06:51.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.528+0000 7f1901ffb640 1 -- 172.21.15.98:0/2461050682 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f190c004020 con 0x564992716b40 2024-09-19T08:06:51.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.528+0000 7f19233ba740 1 -- 172.21.15.98:0/2461050682 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1910004c80 con 0x564992716b40 2024-09-19T08:06:51.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.528+0000 7f1901ffb640 1 -- 172.21.15.98:0/2461050682 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f190c004220 con 0x564992716b40 2024-09-19T08:06:51.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.528+0000 7f19233ba740 10 client.16398 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:51.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.528+0000 7f19233ba740 1 -- 172.21.15.98:0/2461050682 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564992732fc0 con 0x564992716b40 2024-09-19T08:06:51.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.528+0000 7f1901ffb640 1 -- 172.21.15.98:0/2461050682 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1910004c80 con 0x564992716b40 2024-09-19T08:06:51.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.528+0000 7f1901ffb640 1 -- 172.21.15.98:0/2461050682 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f190c018030 con 0x564992716b40 2024-09-19T08:06:51.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.528+0000 7f19233ba740 10 client.16398 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:51.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.528+0000 7f19233ba740 10 client.16398 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:06:51.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.528+0000 7f19233ba740 20 client.16398 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.528+0000 7f19233ba740 1 --2- 172.21.15.98:0/2461050682 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x564992734520 0x564992736940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.528+0000 7f19233ba740 4 client.16398 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.528+0000 7f19233ba740 1 -- 172.21.15.98:0/2461050682 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564992688860 con 0x564992734520 2024-09-19T08:06:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.529+0000 7f1919d9b640 1 --2- 172.21.15.98:0/2461050682 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x564992734520 0x564992736940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.529+0000 7f1901ffb640 10 client.16398 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:06:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.529+0000 7f1919d9b640 1 --2- 172.21.15.98:0/2461050682 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x564992734520 0x564992736940 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=52e7a8f1826bc5f2 server_cookie=5d341192960cb3b2 in_seq=0 out_seq=0 2024-09-19T08:06:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f1901ffb640 1 -- 172.21.15.98:0/2461050682 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564992688860 con 0x564992734520 2024-09-19T08:06:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f1901ffb640 10 client.16398 handle_command_reply: tid=0 2024-09-19T08:06:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f19233ba740 1 client.16398 shutdown 2024-09-19T08:06:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f19233ba740 1 -- 172.21.15.98:0/2461050682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564992716b40 msgr2=0x5649927213f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f19233ba740 1 --2- 172.21.15.98:0/2461050682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564992716b40 0x5649927213f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f190c011830 tx=0x7f190c011cf0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f19233ba740 1 -- 172.21.15.98:0/2461050682 shutdown_connections 2024-09-19T08:06:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f19233ba740 1 --2- 172.21.15.98:0/2461050682 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x564992734520 0x564992736940 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f1901ffb640 0 client.16398 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:06:51.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f19233ba740 1 --2- 172.21.15.98:0/2461050682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564992724780 0x5649927219e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f19233ba740 1 --2- 172.21.15.98:0/2461050682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564992716b40 0x5649927213f0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f19233ba740 1 --2- 172.21.15.98:0/2461050682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5649927155d0 0x564992724140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f19233ba740 1 -- 172.21.15.98:0/2461050682 >> 172.21.15.98:0/2461050682 conn(0x5649926de310 msgr2=0x564992717960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:51.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f19233ba740 1 -- 172.21.15.98:0/2461050682 shutdown_connections 2024-09-19T08:06:51.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f19233ba740 1 -- 172.21.15.98:0/2461050682 wait complete. 2024-09-19T08:06:51.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.530+0000 7f19233ba740 20 client.16398 trim_cache size 0 max 16384 2024-09-19T08:06:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.543+0000 7f19233ba740 1 Processor -- start 2024-09-19T08:06:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.543+0000 7f19233ba740 1 -- start start 2024-09-19T08:06:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.543+0000 7f19233ba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649927155d0 0x5649927213c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.543+0000 7f19233ba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564992716b40 0x564992730750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.543+0000 7f19233ba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564992724780 0x56499273c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.543+0000 7f19233ba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f191c1ab4a0 con 0x5649927155d0 2024-09-19T08:06:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.543+0000 7f19233ba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f191c1ad040 con 0x564992724780 2024-09-19T08:06:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.543+0000 7f19233ba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f191c1abb70 con 0x564992716b40 2024-09-19T08:06:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.544+0000 7f1919d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649927155d0 0x5649927213c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.544+0000 7f1919d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649927155d0 0x5649927213c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34070/0 (socket says 172.21.15.98:34070) 2024-09-19T08:06:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.544+0000 7f1919d9b640 1 -- 172.21.15.98:0/2575386339 learned_addr learned my addr 172.21.15.98:0/2575386339 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.544+0000 7f1919d9b640 1 -- 172.21.15.98:0/2575386339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564992716b40 msgr2=0x564992730750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.544+0000 7f1919d9b640 1 --2- 172.21.15.98:0/2575386339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564992716b40 0x564992730750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.544+0000 7f1919d9b640 1 -- 172.21.15.98:0/2575386339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564992724780 msgr2=0x56499273c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.544+0000 7f1919d9b640 1 --2- 172.21.15.98:0/2575386339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564992724780 0x56499273c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.544+0000 7f1919d9b640 1 -- 172.21.15.98:0/2575386339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564992721d50 con 0x5649927155d0 2024-09-19T08:06:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.544+0000 7f1919d9b640 1 --2- 172.21.15.98:0/2575386339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649927155d0 0x5649927213c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x564992716070 tx=0x7f191000a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b776bc1ecd4ce7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.544+0000 7f1903fff640 1 -- 172.21.15.98:0/2575386339 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910021040 con 0x5649927155d0 2024-09-19T08:06:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.544+0000 7f1903fff640 1 -- 172.21.15.98:0/2575386339 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f191000a2d0 con 0x5649927155d0 2024-09-19T08:06:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.544+0000 7f1903fff640 1 -- 172.21.15.98:0/2575386339 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191000fac0 con 0x5649927155d0 2024-09-19T08:06:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.545+0000 7f19233ba740 1 -- 172.21.15.98:0/2575386339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649927155d0 msgr2=0x5649927213c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.545+0000 7f19233ba740 1 --2- 172.21.15.98:0/2575386339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649927155d0 0x5649927213c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x564992716070 tx=0x7f191000a510 comp rx=0 tx=0).stop 2024-09-19T08:06:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.545+0000 7f19233ba740 1 -- 172.21.15.98:0/2575386339 shutdown_connections 2024-09-19T08:06:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.545+0000 7f19233ba740 1 --2- 172.21.15.98:0/2575386339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564992724780 0x56499273c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.545+0000 7f19233ba740 1 --2- 172.21.15.98:0/2575386339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564992716b40 0x564992730750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.545+0000 7f19233ba740 1 --2- 172.21.15.98:0/2575386339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649927155d0 0x5649927213c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.545+0000 7f19233ba740 1 -- 172.21.15.98:0/2575386339 >> 172.21.15.98:0/2575386339 conn(0x5649926de310 msgr2=0x564992720cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.545+0000 7f19233ba740 1 -- 172.21.15.98:0/2575386339 shutdown_connections 2024-09-19T08:06:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.545+0000 7f19233ba740 1 -- 172.21.15.98:0/2575386339 wait complete. 2024-09-19T08:06:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.545+0000 7f19233ba740 1 -- start start 2024-09-19T08:06:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.545+0000 7f19233ba740 1 Processor -- start 2024-09-19T08:06:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f19233ba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649927155d0 0x56499272cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f19233ba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564992716b40 0x56499271e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f19233ba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564992724780 0x56499271e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f19233ba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f191c1ac940 con 0x5649927155d0 2024-09-19T08:06:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f19233ba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f191c1ac270 con 0x564992716b40 2024-09-19T08:06:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f19233ba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f191c1a9480 con 0x564992724780 2024-09-19T08:06:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f19233ba740 10 client.0 _ll_register_callbacks cb 0x7f190c00c7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f1919d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649927155d0 0x56499272cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f1919d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649927155d0 0x56499272cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34082/0 (socket says 172.21.15.98:34082) 2024-09-19T08:06:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f1919d9b640 1 -- 172.21.15.98:0/3080266318 learned_addr learned my addr 172.21.15.98:0/3080266318 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f191a59c640 1 --2- 172.21.15.98:0/3080266318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564992716b40 0x56499271e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f1919d9b640 1 -- 172.21.15.98:0/3080266318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564992724780 msgr2=0x56499271e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f1919d9b640 1 --2- 172.21.15.98:0/3080266318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564992724780 0x56499271e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f1919d9b640 1 -- 172.21.15.98:0/3080266318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564992716b40 msgr2=0x56499271e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f1919d9b640 1 --2- 172.21.15.98:0/3080266318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564992716b40 0x56499271e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f1919d9b640 1 -- 172.21.15.98:0/3080266318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56499271ec50 con 0x5649927155d0 2024-09-19T08:06:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f1919d9b640 1 --2- 172.21.15.98:0/3080266318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649927155d0 0x56499272cee0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f191000a5c0 tx=0x7f1910004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a861575c7bdf2f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f1901ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f1901ffb640 1 -- 172.21.15.98:0/3080266318 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191003a070 con 0x5649927155d0 2024-09-19T08:06:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f1901ffb640 1 -- 172.21.15.98:0/3080266318 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f1910005190 con 0x5649927155d0 2024-09-19T08:06:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.546+0000 7f19233ba740 1 -- 172.21.15.98:0/3080266318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f191000a2d0 con 0x5649927155d0 2024-09-19T08:06:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.547+0000 7f1901ffb640 1 -- 172.21.15.98:0/3080266318 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1910005390 con 0x5649927155d0 2024-09-19T08:06:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.547+0000 7f19233ba740 10 client.16404 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.547+0000 7f19233ba740 1 -- 172.21.15.98:0/3080266318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56499279f0a0 con 0x5649927155d0 2024-09-19T08:06:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.547+0000 7f1901ffb640 1 -- 172.21.15.98:0/3080266318 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f191000a2d0 con 0x5649927155d0 2024-09-19T08:06:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.547+0000 7f1901ffb640 1 -- 172.21.15.98:0/3080266318 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f1910027040 con 0x5649927155d0 2024-09-19T08:06:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.547+0000 7f19233ba740 10 client.16404 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.547+0000 7f19233ba740 10 client.16404 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:06:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.547+0000 7f19233ba740 20 client.16404 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.547+0000 7f19233ba740 1 --2- 172.21.15.98:0/3080266318 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x564992734520 0x5649927a1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.547+0000 7f19233ba740 4 client.16404 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:06:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.547+0000 7f19233ba740 1 -- 172.21.15.98:0/3080266318 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564992682d70 con 0x564992734520 2024-09-19T08:06:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.547+0000 7f191a59c640 1 --2- 172.21.15.98:0/3080266318 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x564992734520 0x5649927a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f191a59c640 1 --2- 172.21.15.98:0/3080266318 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x564992734520 0x5649927a1530 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=19600b72fb9f83ec server_cookie=74e2cec6d9615aaa in_seq=0 out_seq=0 2024-09-19T08:06:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f1901ffb640 10 client.16404 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:06:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f1901ffb640 1 -- 172.21.15.98:0/3080266318 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564992682d70 con 0x564992734520 2024-09-19T08:06:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f1901ffb640 10 client.16404 handle_command_reply: tid=0 2024-09-19T08:06:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f19233ba740 1 client.16404 shutdown 2024-09-19T08:06:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f19233ba740 1 -- 172.21.15.98:0/3080266318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649927155d0 msgr2=0x56499272cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f19233ba740 1 --2- 172.21.15.98:0/3080266318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649927155d0 0x56499272cee0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f191000a5c0 tx=0x7f1910004f70 comp rx=0 tx=0).stop 2024-09-19T08:06:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f19233ba740 1 -- 172.21.15.98:0/3080266318 shutdown_connections 2024-09-19T08:06:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f19233ba740 1 --2- 172.21.15.98:0/3080266318 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x564992734520 0x5649927a1530 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f19233ba740 1 --2- 172.21.15.98:0/3080266318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564992724780 0x56499271e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f19233ba740 1 --2- 172.21.15.98:0/3080266318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564992716b40 0x56499271e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f19233ba740 1 --2- 172.21.15.98:0/3080266318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649927155d0 0x56499272cee0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f1901ffb640 0 client.16404 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:06:51.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f19233ba740 1 -- 172.21.15.98:0/3080266318 >> 172.21.15.98:0/3080266318 conn(0x5649926de310 msgr2=0x564992722c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:51.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f19233ba740 1 -- 172.21.15.98:0/3080266318 shutdown_connections 2024-09-19T08:06:51.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f19233ba740 1 -- 172.21.15.98:0/3080266318 wait complete. 2024-09-19T08:06:51.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.548+0000 7f19233ba740 20 client.16404 trim_cache size 0 max 16384 2024-09-19T08:06:51.556 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:06:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.549+0000 7f1920fe1640 1 -- 172.21.15.98:0/1372648230 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f19040a33b0 msgr2=0x7f19040a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.549+0000 7f1920fe1640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f19040a33b0 0x7f19040a5870 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f19140bd6b0 tx=0x7f191402a8e0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.549+0000 7f1920fe1640 1 -- 172.21.15.98:0/1372648230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191c106a80 msgr2=0x7f191c1a4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.549+0000 7f1920fe1640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191c106a80 0x7f191c1a4260 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1914006010 tx=0x7f1914003c20 comp rx=0 tx=0).stop 2024-09-19T08:06:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.549+0000 7f1920fe1640 1 -- 172.21.15.98:0/1372648230 shutdown_connections 2024-09-19T08:06:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.549+0000 7f1920fe1640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f19040a33b0 0x7f19040a5870 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.549+0000 7f1920fe1640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191c106a80 0x7f191c1a4260 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.549+0000 7f1920fe1640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f191c101420 0x7f191c199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.549+0000 7f1920fe1640 1 --2- 172.21.15.98:0/1372648230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f191c100a80 0x7f191c1991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.549+0000 7f1920fe1640 1 -- 172.21.15.98:0/1372648230 >> 172.21.15.98:0/1372648230 conn(0x7f191c0fc7a0 msgr2=0x7f191c0fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.549+0000 7f1920fe1640 1 -- 172.21.15.98:0/1372648230 shutdown_connections 2024-09-19T08:06:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:51.549+0000 7f1920fe1640 1 -- 172.21.15.98:0/1372648230 wait complete. 2024-09-19T08:06:52.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:51 smithi139 ceph-mon[34874]: pgmap v118: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:06:52.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:51 smithi098 ceph-mon[31333]: pgmap v118: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:06:52.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:06:52.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:51 smithi162 ceph-mon[35260]: pgmap v118: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:06:52.600 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.072+0000 7ff7f150e640 1 -- 172.21.15.98:0/3959392361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec074f40 msgr2=0x7ff7ec10de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.072+0000 7ff7f150e640 1 --2- 172.21.15.98:0/3959392361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec074f40 0x7ff7ec10de70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4009900 tx=0x7ff7d402a480 comp rx=0 tx=0).stop 2024-09-19T08:06:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.072+0000 7ff7f150e640 1 -- 172.21.15.98:0/3959392361 shutdown_connections 2024-09-19T08:06:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.072+0000 7ff7f150e640 1 --2- 172.21.15.98:0/3959392361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7ec10e3b0 0x7ff7ec110800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.072+0000 7ff7f150e640 1 --2- 172.21.15.98:0/3959392361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec074f40 0x7ff7ec10de70 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.072+0000 7ff7f150e640 1 --2- 172.21.15.98:0/3959392361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7ec074620 0x7ff7ec074a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.072+0000 7ff7f150e640 1 -- 172.21.15.98:0/3959392361 >> 172.21.15.98:0/3959392361 conn(0x7ff7ec0fd8b0 msgr2=0x7ff7ec0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.072+0000 7ff7f150e640 1 -- 172.21.15.98:0/3959392361 shutdown_connections 2024-09-19T08:06:53.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.072+0000 7ff7f150e640 1 -- 172.21.15.98:0/3959392361 wait complete. 2024-09-19T08:06:53.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.073+0000 7ff7f150e640 1 Processor -- start 2024-09-19T08:06:53.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.073+0000 7ff7f150e640 1 -- start start 2024-09-19T08:06:53.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.073+0000 7ff7f150e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec074620 0x7ff7ec1a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.073+0000 7ff7f150e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7ec074f40 0x7ff7ec1a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.073+0000 7ff7f150e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7ec10e3b0 0x7ff7ec19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.073+0000 7ff7f150e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec114220 con 0x7ff7ec074620 2024-09-19T08:06:53.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.073+0000 7ff7f150e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec1140a0 con 0x7ff7ec10e3b0 2024-09-19T08:06:53.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.073+0000 7ff7f150e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec1143a0 con 0x7ff7ec074f40 2024-09-19T08:06:53.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.073+0000 7ff7eaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec074620 0x7ff7ec1a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.073+0000 7ff7eaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec074620 0x7ff7ec1a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34098/0 (socket says 172.21.15.98:34098) 2024-09-19T08:06:53.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.073+0000 7ff7eaffd640 1 -- 172.21.15.98:0/3355860770 learned_addr learned my addr 172.21.15.98:0/3355860770 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:53.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.074+0000 7ff7ea7fc640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7ec074f40 0x7ff7ec1a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.074+0000 7ff7eb7fe640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7ec10e3b0 0x7ff7ec19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.074+0000 7ff7eaffd640 1 -- 172.21.15.98:0/3355860770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7ec074f40 msgr2=0x7ff7ec1a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:53.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.074+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7ec074f40 0x7ff7ec1a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.074+0000 7ff7eaffd640 1 -- 172.21.15.98:0/3355860770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7ec10e3b0 msgr2=0x7ff7ec19a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:53.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.074+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7ec10e3b0 0x7ff7ec19a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.074+0000 7ff7eaffd640 1 -- 172.21.15.98:0/3355860770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ec19aee0 con 0x7ff7ec074620 2024-09-19T08:06:53.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.074+0000 7ff7eb7fe640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7ec10e3b0 0x7ff7ec19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:53.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.074+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec074620 0x7ff7ec1a05a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc00ba50 tx=0x7ff7dc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:53.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.074+0000 7ff7e3fff640 1 -- 172.21.15.98:0/3355860770 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc019070 con 0x7ff7ec074620 2024-09-19T08:06:53.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.074+0000 7ff7f150e640 1 -- 172.21.15.98:0/3355860770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7ec19b1d0 con 0x7ff7ec074620 2024-09-19T08:06:53.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.074+0000 7ff7e3fff640 1 -- 172.21.15.98:0/3355860770 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7dc00cbe0 con 0x7ff7ec074620 2024-09-19T08:06:53.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.074+0000 7ff7e3fff640 1 -- 172.21.15.98:0/3355860770 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc0076e0 con 0x7ff7ec074620 2024-09-19T08:06:53.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.074+0000 7ff7f150e640 1 -- 172.21.15.98:0/3355860770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7ec1a7520 con 0x7ff7ec074620 2024-09-19T08:06:53.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.076+0000 7ff7e3fff640 1 -- 172.21.15.98:0/3355860770 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7ff7dc01c070 con 0x7ff7ec074620 2024-09-19T08:06:53.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.077+0000 7ff7f38e7740 1 Processor -- start 2024-09-19T08:06:53.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.077+0000 7ff7f38e7740 1 -- start start 2024-09-19T08:06:53.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.077+0000 7ff7f38e7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b15b40 0x55f716b14090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.077+0000 7ff7f38e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b145d0 0x55f716b14a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.077+0000 7ff7f38e7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b1f370 0x55f716b21760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.077+0000 7ff7f38e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f716a7a2f0 con 0x55f716b145d0 2024-09-19T08:06:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.077+0000 7ff7f38e7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f716a21cf0 con 0x55f716b1f370 2024-09-19T08:06:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.077+0000 7ff7f38e7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f716a762b0 con 0x55f716b15b40 2024-09-19T08:06:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.077+0000 7ff7e3fff640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff7d80a33b0 0x7ff7d80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.077+0000 7ff7eaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b145d0 0x55f716b14a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.077+0000 7ff7eaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b145d0 0x55f716b14a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34108/0 (socket says 172.21.15.98:34108) 2024-09-19T08:06:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.077+0000 7ff7eaffd640 1 -- 172.21.15.98:0/3581210980 learned_addr learned my addr 172.21.15.98:0/3581210980 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.077+0000 7ff7eb7fe640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff7d80a33b0 0x7ff7d80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.078+0000 7ff7eb7fe640 1 --2- 172.21.15.98:0/3581210980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b15b40 0x55f716b14090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.078+0000 7ff7ea7fc640 1 --2- 172.21.15.98:0/3581210980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b1f370 0x55f716b21760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.078+0000 7ff7e3fff640 1 -- 172.21.15.98:0/3355860770 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff7dc0cf2f0 con 0x7ff7ec074620 2024-09-19T08:06:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.078+0000 7ff7eaffd640 1 -- 172.21.15.98:0/3581210980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b15b40 msgr2=0x55f716b14090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.078+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/3581210980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b15b40 0x55f716b14090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.078+0000 7ff7eaffd640 1 -- 172.21.15.98:0/3581210980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b1f370 msgr2=0x55f716b21760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.078+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/3581210980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b1f370 0x55f716b21760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.078+0000 7ff7eaffd640 1 -- 172.21.15.98:0/3581210980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f716a063a0 con 0x55f716b145d0 2024-09-19T08:06:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.078+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/3581210980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b145d0 0x55f716b14a10 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc09bd10 tx=0x7ff7dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96374c2037f30886 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.078+0000 7ff7eb7fe640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff7d80a33b0 0x7ff7d80a5870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4015600 tx=0x7ff7e4006d50 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.078+0000 7ff7e17fa640 1 -- 172.21.15.98:0/3581210980 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc023040 con 0x55f716b145d0 2024-09-19T08:06:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.078+0000 7ff7e17fa640 1 -- 172.21.15.98:0/3581210980 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7dc09bdb0 con 0x55f716b145d0 2024-09-19T08:06:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.078+0000 7ff7e17fa640 1 -- 172.21.15.98:0/3581210980 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc0a3e90 con 0x55f716b145d0 2024-09-19T08:06:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.079+0000 7ff7f38e7740 1 -- 172.21.15.98:0/3581210980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b145d0 msgr2=0x55f716b14a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.079+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/3581210980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b145d0 0x55f716b14a10 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc09bd10 tx=0x7ff7dc004a40 comp rx=0 tx=0).stop 2024-09-19T08:06:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.079+0000 7ff7f38e7740 1 -- 172.21.15.98:0/3581210980 shutdown_connections 2024-09-19T08:06:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.079+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/3581210980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b1f370 0x55f716b21760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.079+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/3581210980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b145d0 0x55f716b14a10 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.079+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/3581210980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b15b40 0x55f716b14090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.079+0000 7ff7f38e7740 1 -- 172.21.15.98:0/3581210980 >> 172.21.15.98:0/3581210980 conn(0x55f716add310 msgr2=0x55f716b10c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.079+0000 7ff7f38e7740 1 -- 172.21.15.98:0/3581210980 shutdown_connections 2024-09-19T08:06:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.079+0000 7ff7f38e7740 1 -- 172.21.15.98:0/3581210980 wait complete. 2024-09-19T08:06:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.079+0000 7ff7f38e7740 1 -- start start 2024-09-19T08:06:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.079+0000 7ff7f38e7740 1 Processor -- start 2024-09-19T08:06:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7f38e7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b145d0 0x55f716b1ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7f38e7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b15b40 0x55f716b1c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7f38e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b1f370 0x55f716b1c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7f38e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f716b22070 con 0x55f716b1f370 2024-09-19T08:06:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7f38e7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f716a35060 con 0x55f716b15b40 2024-09-19T08:06:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7f38e7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f716b221f0 con 0x55f716b145d0 2024-09-19T08:06:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7f38e7740 10 client.0 _ll_register_callbacks cb 0x55f7167888c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7eb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b1f370 0x55f716b1c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7eb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b1f370 0x55f716b1c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34110/0 (socket says 172.21.15.98:34110) 2024-09-19T08:06:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7eb7fe640 1 -- 172.21.15.98:0/122859396 learned_addr learned my addr 172.21.15.98:0/122859396 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b145d0 0x55f716b1ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7ea7fc640 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b15b40 0x55f716b1c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7eb7fe640 1 -- 172.21.15.98:0/122859396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b145d0 msgr2=0x55f716b1ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7eb7fe640 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b145d0 0x55f716b1ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7eb7fe640 1 -- 172.21.15.98:0/122859396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b15b40 msgr2=0x55f716b1c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7eb7fe640 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b15b40 0x55f716b1c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7eb7fe640 1 -- 172.21.15.98:0/122859396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7167eb3c0 con 0x55f716b1f370 2024-09-19T08:06:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7ea7fc640 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b15b40 0x55f716b1c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b145d0 0x55f716b1ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7eb7fe640 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b1f370 0x55f716b1c840 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4017d80 tx=0x7ff7e40047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a441776bfb0b7ee5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7d27fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7d27fc640 1 -- 172.21.15.98:0/122859396 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e400f040 con 0x55f716b1f370 2024-09-19T08:06:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7d27fc640 1 -- 172.21.15.98:0/122859396 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff7e40213f0 con 0x55f716b1f370 2024-09-19T08:06:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.080+0000 7ff7f38e7740 1 -- 172.21.15.98:0/122859396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7dc09bdb0 con 0x55f716b1f370 2024-09-19T08:06:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.081+0000 7ff7d27fc640 1 -- 172.21.15.98:0/122859396 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff7e402f050 con 0x55f716b1f370 2024-09-19T08:06:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.081+0000 7ff7f38e7740 10 client.16416 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.081+0000 7ff7f38e7740 1 -- 172.21.15.98:0/122859396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f716b2dbc0 con 0x55f716b1f370 2024-09-19T08:06:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.081+0000 7ff7d27fc640 1 -- 172.21.15.98:0/122859396 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7dc09bdb0 con 0x55f716b1f370 2024-09-19T08:06:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.081+0000 7ff7d27fc640 1 -- 172.21.15.98:0/122859396 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7ff7e402a4d0 con 0x55f716b1f370 2024-09-19T08:06:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.081+0000 7ff7f38e7740 10 client.16416 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.081+0000 7ff7f38e7740 10 client.16416 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:06:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.081+0000 7ff7f38e7740 20 client.16416 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.081+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f716b2f080 0x55f716b31520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.081+0000 7ff7f38e7740 4 client.16416 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.081+0000 7ff7f38e7740 1 -- 172.21.15.98:0/122859396 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f716a87860 con 0x55f716b2f080 2024-09-19T08:06:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.081+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f716b2f080 0x55f716b31520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.082+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f716b2f080 0x55f716b31520 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=db44c18110e863b1 server_cookie=1f6a433d75204a27 in_seq=0 out_seq=0 2024-09-19T08:06:53.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.082+0000 7ff7d27fc640 10 client.16416 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:06:53.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.082+0000 7ff7d27fc640 1 -- 172.21.15.98:0/122859396 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f716a87860 con 0x55f716b2f080 2024-09-19T08:06:53.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.082+0000 7ff7d27fc640 10 client.16416 handle_command_reply: tid=0 2024-09-19T08:06:53.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.082+0000 7ff7f38e7740 1 client.16416 shutdown 2024-09-19T08:06:53.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.083+0000 7ff7f38e7740 1 -- 172.21.15.98:0/122859396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b1f370 msgr2=0x55f716b1c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:53.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.083+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b1f370 0x55f716b1c840 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4017d80 tx=0x7ff7e40047d0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.083+0000 7ff7f38e7740 1 -- 172.21.15.98:0/122859396 shutdown_connections 2024-09-19T08:06:53.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.083+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b1f370 0x55f716b1c840 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.083+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b15b40 0x55f716b1c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.083+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f716b2f080 0x55f716b31520 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.083+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/122859396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b145d0 0x55f716b1ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.083+0000 7ff7f38e7740 1 -- 172.21.15.98:0/122859396 >> 172.21.15.98:0/122859396 conn(0x55f716add310 msgr2=0x55f716b16990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:53.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.083+0000 7ff7d27fc640 0 client.16416 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:06:53.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.083+0000 7ff7f38e7740 1 -- 172.21.15.98:0/122859396 shutdown_connections 2024-09-19T08:06:53.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.083+0000 7ff7f38e7740 1 -- 172.21.15.98:0/122859396 wait complete. 2024-09-19T08:06:53.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.083+0000 7ff7f38e7740 20 client.16416 trim_cache size 0 max 16384 2024-09-19T08:06:53.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.095+0000 7ff7f38e7740 1 Processor -- start 2024-09-19T08:06:53.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.095+0000 7ff7f38e7740 1 -- start start 2024-09-19T08:06:53.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.095+0000 7ff7f38e7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b145d0 0x55f716b10e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.095+0000 7ff7f38e7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b15b40 0x55f716b36e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.095+0000 7ff7f38e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b1f370 0x55f716b37380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.095+0000 7ff7f38e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec1afc60 con 0x55f716b1f370 2024-09-19T08:06:53.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.095+0000 7ff7f38e7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec1b1800 con 0x55f716b145d0 2024-09-19T08:06:53.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.095+0000 7ff7f38e7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec1b0330 con 0x55f716b15b40 2024-09-19T08:06:53.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7eb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b1f370 0x55f716b37380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7eb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b1f370 0x55f716b37380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34114/0 (socket says 172.21.15.98:34114) 2024-09-19T08:06:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7eb7fe640 1 -- 172.21.15.98:0/4221169552 learned_addr learned my addr 172.21.15.98:0/4221169552 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7ea7fc640 1 --2- 172.21.15.98:0/4221169552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b15b40 0x55f716b36e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/4221169552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b145d0 0x55f716b10e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7eb7fe640 1 -- 172.21.15.98:0/4221169552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b15b40 msgr2=0x55f716b36e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7eb7fe640 1 --2- 172.21.15.98:0/4221169552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b15b40 0x55f716b36e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7eb7fe640 1 -- 172.21.15.98:0/4221169552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b145d0 msgr2=0x55f716b10e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7eb7fe640 1 --2- 172.21.15.98:0/4221169552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b145d0 0x55f716b10e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7eb7fe640 1 -- 172.21.15.98:0/4221169552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f716b11960 con 0x55f716b1f370 2024-09-19T08:06:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/4221169552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b145d0 0x55f716b10e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7eb7fe640 1 --2- 172.21.15.98:0/4221169552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b1f370 0x55f716b37380 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4009040 tx=0x7ff7e4006cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c371cd6c3b06fe2f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7e0ff9640 1 -- 172.21.15.98:0/4221169552 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4039070 con 0x55f716b1f370 2024-09-19T08:06:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7e0ff9640 1 -- 172.21.15.98:0/4221169552 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7e4026070 con 0x55f716b1f370 2024-09-19T08:06:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.096+0000 7ff7e0ff9640 1 -- 172.21.15.98:0/4221169552 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4035420 con 0x55f716b1f370 2024-09-19T08:06:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.097+0000 7ff7f38e7740 1 -- 172.21.15.98:0/4221169552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b1f370 msgr2=0x55f716b37380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.097+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/4221169552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b1f370 0x55f716b37380 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4009040 tx=0x7ff7e4006cb0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.097+0000 7ff7f38e7740 1 -- 172.21.15.98:0/4221169552 shutdown_connections 2024-09-19T08:06:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.097+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/4221169552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b1f370 0x55f716b37380 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.097+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/4221169552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b15b40 0x55f716b36e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.097+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/4221169552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b145d0 0x55f716b10e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.097+0000 7ff7f38e7740 1 -- 172.21.15.98:0/4221169552 >> 172.21.15.98:0/4221169552 conn(0x55f716add310 msgr2=0x55f716b10700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.097+0000 7ff7f38e7740 1 -- 172.21.15.98:0/4221169552 shutdown_connections 2024-09-19T08:06:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.097+0000 7ff7f38e7740 1 -- 172.21.15.98:0/4221169552 wait complete. 2024-09-19T08:06:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.097+0000 7ff7f38e7740 1 -- start start 2024-09-19T08:06:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.097+0000 7ff7f38e7740 1 Processor -- start 2024-09-19T08:06:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7f38e7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b145d0 0x55f716b27660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7f38e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b15b40 0x55f716b27ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7f38e7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b1f370 0x55f716b188e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7f38e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec1b1100 con 0x55f716b15b40 2024-09-19T08:06:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7f38e7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec1b0a30 con 0x55f716b1f370 2024-09-19T08:06:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7f38e7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec1adc40 con 0x55f716b145d0 2024-09-19T08:06:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7f38e7740 10 client.0 _ll_register_callbacks cb 0x7ff7e4017a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7ea7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b15b40 0x55f716b27ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7ea7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b15b40 0x55f716b27ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34122/0 (socket says 172.21.15.98:34122) 2024-09-19T08:06:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7ea7fc640 1 -- 172.21.15.98:0/14955886 learned_addr learned my addr 172.21.15.98:0/14955886 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/14955886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b145d0 0x55f716b27660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7eb7fe640 1 --2- 172.21.15.98:0/14955886 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b1f370 0x55f716b188e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7ea7fc640 1 -- 172.21.15.98:0/14955886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b145d0 msgr2=0x55f716b27660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7ea7fc640 1 --2- 172.21.15.98:0/14955886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b145d0 0x55f716b27660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7ea7fc640 1 -- 172.21.15.98:0/14955886 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b1f370 msgr2=0x55f716b188e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7ea7fc640 1 --2- 172.21.15.98:0/14955886 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b1f370 0x55f716b188e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7ea7fc640 1 -- 172.21.15.98:0/14955886 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f716b18fa0 con 0x55f716b15b40 2024-09-19T08:06:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7ea7fc640 1 --2- 172.21.15.98:0/14955886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b15b40 0x55f716b27ba0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4005ab0 tx=0x7ff7d4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f2f3c1c2f7729d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7d27fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7d27fc640 1 -- 172.21.15.98:0/14955886 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d4035040 con 0x55f716b15b40 2024-09-19T08:06:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7d27fc640 1 -- 172.21.15.98:0/14955886 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff7d4005220 con 0x55f716b15b40 2024-09-19T08:06:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.098+0000 7ff7f38e7740 1 -- 172.21.15.98:0/14955886 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7e4026070 con 0x55f716b15b40 2024-09-19T08:06:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.099+0000 7ff7d27fc640 1 -- 172.21.15.98:0/14955886 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff7d4005420 con 0x55f716b15b40 2024-09-19T08:06:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.099+0000 7ff7f38e7740 10 client.16422 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.099+0000 7ff7f38e7740 1 -- 172.21.15.98:0/14955886 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f716b99740 con 0x55f716b15b40 2024-09-19T08:06:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.099+0000 7ff7d27fc640 1 -- 172.21.15.98:0/14955886 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7e4026070 con 0x55f716b15b40 2024-09-19T08:06:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.099+0000 7ff7d27fc640 1 -- 172.21.15.98:0/14955886 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7ff7d4043070 con 0x55f716b15b40 2024-09-19T08:06:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.099+0000 7ff7f38e7740 10 client.16422 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:53.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.099+0000 7ff7f38e7740 10 client.16422 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:06:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.099+0000 7ff7f38e7740 20 client.16422 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.099+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/14955886 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f716b99ef0 0x55f716b9c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.099+0000 7ff7f38e7740 4 client.16422 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:06:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.099+0000 7ff7f38e7740 1 -- 172.21.15.98:0/14955886 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f716a81d70 con 0x55f716b99ef0 2024-09-19T08:06:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/14955886 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f716b99ef0 0x55f716b9c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7eaffd640 1 --2- 172.21.15.98:0/14955886 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f716b99ef0 0x55f716b9c310 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7baa1502c7ec62f6 server_cookie=2f4f6e53526b5277 in_seq=0 out_seq=0 2024-09-19T08:06:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7d27fc640 10 client.16422 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:06:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7d27fc640 1 -- 172.21.15.98:0/14955886 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f716a81d70 con 0x55f716b99ef0 2024-09-19T08:06:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7d27fc640 10 client.16422 handle_command_reply: tid=0 2024-09-19T08:06:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7f38e7740 1 client.16422 shutdown 2024-09-19T08:06:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7f38e7740 1 -- 172.21.15.98:0/14955886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b15b40 msgr2=0x55f716b27ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/14955886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b15b40 0x55f716b27ba0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4005ab0 tx=0x7ff7d4004270 comp rx=0 tx=0).stop 2024-09-19T08:06:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7f38e7740 1 -- 172.21.15.98:0/14955886 shutdown_connections 2024-09-19T08:06:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/14955886 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f716b1f370 0x55f716b188e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/14955886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f716b15b40 0x55f716b27ba0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/14955886 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f716b99ef0 0x55f716b9c310 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7f38e7740 1 --2- 172.21.15.98:0/14955886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f716b145d0 0x55f716b27660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7d27fc640 0 client.16422 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:06:53.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7f38e7740 1 -- 172.21.15.98:0/14955886 >> 172.21.15.98:0/14955886 conn(0x55f716add310 msgr2=0x55f716b1a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:53.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.100+0000 7ff7f38e7740 1 -- 172.21.15.98:0/14955886 shutdown_connections 2024-09-19T08:06:53.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.101+0000 7ff7f38e7740 1 -- 172.21.15.98:0/14955886 wait complete. 2024-09-19T08:06:53.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.101+0000 7ff7f38e7740 20 client.16422 trim_cache size 0 max 16384 2024-09-19T08:06:53.121 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:06:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.101+0000 7ff7f150e640 1 -- 172.21.15.98:0/3355860770 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff7d80a33b0 msgr2=0x7ff7d80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.101+0000 7ff7f150e640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff7d80a33b0 0x7ff7d80a5870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4015600 tx=0x7ff7e4006d50 comp rx=0 tx=0).stop 2024-09-19T08:06:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.101+0000 7ff7f150e640 1 -- 172.21.15.98:0/3355860770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec074620 msgr2=0x7ff7ec1a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.101+0000 7ff7f150e640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec074620 0x7ff7ec1a05a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc00ba50 tx=0x7ff7dc00bf10 comp rx=0 tx=0).stop 2024-09-19T08:06:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.101+0000 7ff7f150e640 1 -- 172.21.15.98:0/3355860770 shutdown_connections 2024-09-19T08:06:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.101+0000 7ff7f150e640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff7d80a33b0 0x7ff7d80a5870 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.101+0000 7ff7f150e640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7ec10e3b0 0x7ff7ec19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.101+0000 7ff7f150e640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7ec074f40 0x7ff7ec1a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.101+0000 7ff7f150e640 1 --2- 172.21.15.98:0/3355860770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec074620 0x7ff7ec1a05a0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.101+0000 7ff7f150e640 1 -- 172.21.15.98:0/3355860770 >> 172.21.15.98:0/3355860770 conn(0x7ff7ec0fd8b0 msgr2=0x7ff7ec0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.101+0000 7ff7f150e640 1 -- 172.21.15.98:0/3355860770 shutdown_connections 2024-09-19T08:06:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:53.101+0000 7ff7f150e640 1 -- 172.21.15.98:0/3355860770 wait complete. 2024-09-19T08:06:53.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:06:54.105 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:54.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:53 smithi139 ceph-mon[34874]: pgmap v119: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:06:54.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:53 smithi098 ceph-mon[31333]: pgmap v119: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:06:54.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:53 smithi162 ceph-mon[35260]: pgmap v119: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:06:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.563+0000 7fb4cea58640 1 -- 172.21.15.98:0/1400490355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8100cc0 msgr2=0x7fb4c8106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.563+0000 7fb4cea58640 1 --2- 172.21.15.98:0/1400490355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8100cc0 0x7fb4c8106800 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc009900 tx=0x7fb4bc02a410 comp rx=0 tx=0).stop 2024-09-19T08:06:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.563+0000 7fb4cea58640 1 -- 172.21.15.98:0/1400490355 shutdown_connections 2024-09-19T08:06:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.563+0000 7fb4cea58640 1 --2- 172.21.15.98:0/1400490355 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4c8106f30 0x7fb4c810eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.563+0000 7fb4cea58640 1 --2- 172.21.15.98:0/1400490355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8100cc0 0x7fb4c8106800 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.563+0000 7fb4cea58640 1 --2- 172.21.15.98:0/1400490355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4c8100310 0x7fb4c81006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.563+0000 7fb4cea58640 1 -- 172.21.15.98:0/1400490355 >> 172.21.15.98:0/1400490355 conn(0x7fb4c8077210 msgr2=0x7fb4c80fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.563+0000 7fb4cea58640 1 -- 172.21.15.98:0/1400490355 shutdown_connections 2024-09-19T08:06:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.563+0000 7fb4cea58640 1 -- 172.21.15.98:0/1400490355 wait complete. 2024-09-19T08:06:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.564+0000 7fb4cea58640 1 Processor -- start 2024-09-19T08:06:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.564+0000 7fb4cea58640 1 -- start start 2024-09-19T08:06:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.564+0000 7fb4cea58640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4c8100310 0x7fb4c81984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.564+0000 7fb4cea58640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4c8100cc0 0x7fb4c8198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.564+0000 7fb4cea58640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8106f30 0x7fb4c819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.564+0000 7fb4cea58640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c8111f90 con 0x7fb4c8106f30 2024-09-19T08:06:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.564+0000 7fb4cea58640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c8111e10 con 0x7fb4c8100cc0 2024-09-19T08:06:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.564+0000 7fb4cea58640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c8112110 con 0x7fb4c8100310 2024-09-19T08:06:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.564+0000 7fb4ccff5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8106f30 0x7fb4c819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.564+0000 7fb4ccff5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8106f30 0x7fb4c819cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34140/0 (socket says 172.21.15.98:34140) 2024-09-19T08:06:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.564+0000 7fb4ccff5640 1 -- 172.21.15.98:0/3503778945 learned_addr learned my addr 172.21.15.98:0/3503778945 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.564+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/3503778945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4c8100cc0 0x7fb4c8198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.565+0000 7fb4c7fff640 1 --2- 172.21.15.98:0/3503778945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4c8100310 0x7fb4c81984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.565+0000 7fb4ccff5640 1 -- 172.21.15.98:0/3503778945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4c8100310 msgr2=0x7fb4c81984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.565+0000 7fb4ccff5640 1 --2- 172.21.15.98:0/3503778945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4c8100310 0x7fb4c81984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.565+0000 7fb4ccff5640 1 -- 172.21.15.98:0/3503778945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4c8100cc0 msgr2=0x7fb4c8198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.565+0000 7fb4ccff5640 1 --2- 172.21.15.98:0/3503778945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4c8100cc0 0x7fb4c8198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.565+0000 7fb4ccff5640 1 -- 172.21.15.98:0/3503778945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c819d480 con 0x7fb4c8106f30 2024-09-19T08:06:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.565+0000 7fb4ccff5640 1 --2- 172.21.15.98:0/3503778945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8106f30 0x7fb4c819cda0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000eaa0 tx=0x7fb4c000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.565+0000 7fb4c57fa640 1 -- 172.21.15.98:0/3503778945 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c000cdb0 con 0x7fb4c8106f30 2024-09-19T08:06:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.565+0000 7fb4cea58640 1 -- 172.21.15.98:0/3503778945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c819d770 con 0x7fb4c8106f30 2024-09-19T08:06:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.565+0000 7fb4c57fa640 1 -- 172.21.15.98:0/3503778945 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4c0005020 con 0x7fb4c8106f30 2024-09-19T08:06:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.565+0000 7fb4c57fa640 1 -- 172.21.15.98:0/3503778945 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c00116b0 con 0x7fb4c8106f30 2024-09-19T08:06:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.565+0000 7fb4cea58640 1 -- 172.21.15.98:0/3503778945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c81a5050 con 0x7fb4c8106f30 2024-09-19T08:06:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.568+0000 7fb4d0e31740 1 Processor -- start 2024-09-19T08:06:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.568+0000 7fb4c57fa640 1 -- 172.21.15.98:0/3503778945 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fb4c0005560 con 0x7fb4c8106f30 2024-09-19T08:06:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.568+0000 7fb4d0e31740 1 -- start start 2024-09-19T08:06:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.568+0000 7fb4d0e31740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605adb40 0x55a7605ac090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.568+0000 7fb4d0e31740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605ac5d0 0x55a7605aca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.568+0000 7fb4d0e31740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605b7370 0x55a7605b9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.568+0000 7fb4d0e31740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a7605122f0 con 0x55a7605b7370 2024-09-19T08:06:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.568+0000 7fb4d0e31740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a7604b9cf0 con 0x55a7605ac5d0 2024-09-19T08:06:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.568+0000 7fb4d0e31740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a76050e2b0 con 0x55a7605adb40 2024-09-19T08:06:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605b7370 0x55a7605b9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605b7370 0x55a7605b9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34152/0 (socket says 172.21.15.98:34152) 2024-09-19T08:06:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4c77fe640 1 -- 172.21.15.98:0/3981408725 learned_addr learned my addr 172.21.15.98:0/3981408725 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4c7fff640 1 --2- 172.21.15.98:0/3981408725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605ac5d0 0x55a7605aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/3981408725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605adb40 0x55a7605ac090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4c57fa640 1 --2- 172.21.15.98:0/3503778945 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb4b00a33b0 0x7fb4b00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4c77fe640 1 -- 172.21.15.98:0/3981408725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605adb40 msgr2=0x55a7605ac090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/3981408725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605adb40 0x55a7605ac090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4c77fe640 1 -- 172.21.15.98:0/3981408725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605ac5d0 msgr2=0x55a7605aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/3981408725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605ac5d0 0x55a7605aca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4c77fe640 1 -- 172.21.15.98:0/3981408725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a76049e3a0 con 0x55a7605b7370 2024-09-19T08:06:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4ccff5640 1 --2- 172.21.15.98:0/3503778945 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb4b00a33b0 0x7fb4b00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4c57fa640 1 -- 172.21.15.98:0/3503778945 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fb4c00cfee0 con 0x7fb4c8106f30 2024-09-19T08:06:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/3981408725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605b7370 0x55a7605b9760 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc02a930 tx=0x7fb4bc005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78aea15a3f3b5d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4ae7fc640 1 -- 172.21.15.98:0/3981408725 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc035070 con 0x55a7605b7370 2024-09-19T08:06:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4ae7fc640 1 -- 172.21.15.98:0/3981408725 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4bc004030 con 0x55a7605b7370 2024-09-19T08:06:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.569+0000 7fb4ae7fc640 1 -- 172.21.15.98:0/3981408725 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc005600 con 0x55a7605b7370 2024-09-19T08:06:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.570+0000 7fb4d0e31740 1 -- 172.21.15.98:0/3981408725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605b7370 msgr2=0x55a7605b9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.570+0000 7fb4ccff5640 1 --2- 172.21.15.98:0/3503778945 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb4b00a33b0 0x7fb4b00a5870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000efd0 tx=0x7fb4c0086ae0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.570+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/3981408725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605b7370 0x55a7605b9760 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc02a930 tx=0x7fb4bc005c00 comp rx=0 tx=0).stop 2024-09-19T08:06:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.570+0000 7fb4d0e31740 1 -- 172.21.15.98:0/3981408725 shutdown_connections 2024-09-19T08:06:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.570+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/3981408725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605b7370 0x55a7605b9760 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.570+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/3981408725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605ac5d0 0x55a7605aca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.570+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/3981408725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605adb40 0x55a7605ac090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.570+0000 7fb4d0e31740 1 -- 172.21.15.98:0/3981408725 >> 172.21.15.98:0/3981408725 conn(0x55a760575310 msgr2=0x55a7605a8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.570+0000 7fb4d0e31740 1 -- 172.21.15.98:0/3981408725 shutdown_connections 2024-09-19T08:06:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.570+0000 7fb4d0e31740 1 -- 172.21.15.98:0/3981408725 wait complete. 2024-09-19T08:06:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.570+0000 7fb4d0e31740 1 -- start start 2024-09-19T08:06:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4d0e31740 1 Processor -- start 2024-09-19T08:06:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4d0e31740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605ac5d0 0x55a7605b6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4d0e31740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605adb40 0x55a7605b4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4d0e31740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605b7370 0x55a7605b4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4d0e31740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a7605ba010 con 0x55a7605b7370 2024-09-19T08:06:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4d0e31740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a7604cd060 con 0x55a7605adb40 2024-09-19T08:06:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4d0e31740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a7605ba190 con 0x55a7605ac5d0 2024-09-19T08:06:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4d0e31740 10 client.0 _ll_register_callbacks cb 0x55a7602208c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605b7370 0x55a7605b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605b7370 0x55a7605b4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34168/0 (socket says 172.21.15.98:34168) 2024-09-19T08:06:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4c77fe640 1 -- 172.21.15.98:0/331795217 learned_addr learned my addr 172.21.15.98:0/331795217 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/331795217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605ac5d0 0x55a7605b6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4c7fff640 1 --2- 172.21.15.98:0/331795217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605adb40 0x55a7605b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4c77fe640 1 -- 172.21.15.98:0/331795217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605ac5d0 msgr2=0x55a7605b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/331795217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605ac5d0 0x55a7605b6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4c77fe640 1 -- 172.21.15.98:0/331795217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605adb40 msgr2=0x55a7605b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/331795217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605adb40 0x55a7605b4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4c77fe640 1 -- 172.21.15.98:0/331795217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7602833c0 con 0x55a7605b7370 2024-09-19T08:06:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.571+0000 7fb4c7fff640 1 --2- 172.21.15.98:0/331795217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605adb40 0x55a7605b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/331795217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605b7370 0x55a7605b4610 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc000c00 tx=0x7fb4bc02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acad3fbb11b3db5e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4a77fe640 1 -- 172.21.15.98:0/331795217 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc035070 con 0x55a7605b7370 2024-09-19T08:06:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4a77fe640 1 -- 172.21.15.98:0/331795217 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fb4bc031630 con 0x55a7605b7370 2024-09-19T08:06:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4d0e31740 1 -- 172.21.15.98:0/331795217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4bc004030 con 0x55a7605b7370 2024-09-19T08:06:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4a77fe640 1 -- 172.21.15.98:0/331795217 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb4bc031830 con 0x55a7605b7370 2024-09-19T08:06:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4d0e31740 10 client.16434 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4d0e31740 1 -- 172.21.15.98:0/331795217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a7605c5ab0 con 0x55a7605b7370 2024-09-19T08:06:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4a77fe640 1 -- 172.21.15.98:0/331795217 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4bc004030 con 0x55a7605b7370 2024-09-19T08:06:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4a77fe640 1 -- 172.21.15.98:0/331795217 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fb4bc003710 con 0x55a7605b7370 2024-09-19T08:06:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4d0e31740 10 client.16434 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4d0e31740 10 client.16434 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:06:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4d0e31740 20 client.16434 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/331795217 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a7605c6e00 0x55a7605c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4d0e31740 4 client.16434 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.572+0000 7fb4d0e31740 1 -- 172.21.15.98:0/331795217 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a76051f860 con 0x55a7605c6e00 2024-09-19T08:06:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.573+0000 7fb4c7fff640 1 --2- 172.21.15.98:0/331795217 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a7605c6e00 0x55a7605c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.573+0000 7fb4c7fff640 1 --2- 172.21.15.98:0/331795217 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a7605c6e00 0x55a7605c91f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6c99b5590c3cae5e server_cookie=609e3879e4a42af8 in_seq=0 out_seq=0 2024-09-19T08:06:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.573+0000 7fb4a77fe640 10 client.16434 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:06:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4a77fe640 1 -- 172.21.15.98:0/331795217 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a76051f860 con 0x55a7605c6e00 2024-09-19T08:06:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4a77fe640 10 client.16434 handle_command_reply: tid=0 2024-09-19T08:06:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4d0e31740 1 client.16434 shutdown 2024-09-19T08:06:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4d0e31740 1 -- 172.21.15.98:0/331795217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605b7370 msgr2=0x55a7605b4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/331795217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605b7370 0x55a7605b4610 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc000c00 tx=0x7fb4bc02a410 comp rx=0 tx=0).stop 2024-09-19T08:06:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4d0e31740 1 -- 172.21.15.98:0/331795217 shutdown_connections 2024-09-19T08:06:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/331795217 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a7605c6e00 0x55a7605c91f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/331795217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605b7370 0x55a7605b4610 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/331795217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605adb40 0x55a7605b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/331795217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605ac5d0 0x55a7605b6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4a77fe640 0 client.16434 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:06:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4d0e31740 1 -- 172.21.15.98:0/331795217 >> 172.21.15.98:0/331795217 conn(0x55a760575310 msgr2=0x55a7605ae960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4d0e31740 1 -- 172.21.15.98:0/331795217 shutdown_connections 2024-09-19T08:06:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4d0e31740 1 -- 172.21.15.98:0/331795217 wait complete. 2024-09-19T08:06:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.574+0000 7fb4d0e31740 20 client.16434 trim_cache size 0 max 16384 2024-09-19T08:06:54.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.587+0000 7fb4d0e31740 1 Processor -- start 2024-09-19T08:06:54.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.587+0000 7fb4d0e31740 1 -- start start 2024-09-19T08:06:54.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.587+0000 7fb4d0e31740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605ac5d0 0x55a7605a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.587+0000 7fb4d0e31740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605adb40 0x55a7605cc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.587+0000 7fb4d0e31740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605b7370 0x55a7605cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.587+0000 7fb4d0e31740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c81ad750 con 0x55a7605ac5d0 2024-09-19T08:06:54.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.587+0000 7fb4d0e31740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c81af2f0 con 0x55a7605adb40 2024-09-19T08:06:54.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.587+0000 7fb4d0e31740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c81ade20 con 0x55a7605b7370 2024-09-19T08:06:54.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.587+0000 7fb4c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605ac5d0 0x55a7605a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.587+0000 7fb4c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605ac5d0 0x55a7605a8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34178/0 (socket says 172.21.15.98:34178) 2024-09-19T08:06:54.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.587+0000 7fb4c77fe640 1 -- 172.21.15.98:0/714779895 learned_addr learned my addr 172.21.15.98:0/714779895 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:54.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.588+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/714779895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605b7370 0x55a7605cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.588+0000 7fb4c77fe640 1 -- 172.21.15.98:0/714779895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605b7370 msgr2=0x55a7605cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:54.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.588+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/714779895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605b7370 0x55a7605cc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.588+0000 7fb4c77fe640 1 -- 172.21.15.98:0/714779895 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605adb40 msgr2=0x55a7605cc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:54.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.588+0000 7fb4c7fff640 1 --2- 172.21.15.98:0/714779895 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605adb40 0x55a7605cc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.588+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/714779895 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605adb40 0x55a7605cc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.588+0000 7fb4c77fe640 1 -- 172.21.15.98:0/714779895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7605ac2d0 con 0x55a7605ac5d0 2024-09-19T08:06:54.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.588+0000 7fb4c7fff640 1 --2- 172.21.15.98:0/714779895 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605adb40 0x55a7605cc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:54.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.588+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/714779895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605ac5d0 0x55a7605a8df0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc02a930 tx=0x7fb4bc005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77eb986a391a518a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:54.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.588+0000 7fb4adffb640 1 -- 172.21.15.98:0/714779895 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc035070 con 0x55a7605ac5d0 2024-09-19T08:06:54.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.588+0000 7fb4adffb640 1 -- 172.21.15.98:0/714779895 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4bc0315b0 con 0x55a7605ac5d0 2024-09-19T08:06:54.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.588+0000 7fb4adffb640 1 -- 172.21.15.98:0/714779895 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc0554c0 con 0x55a7605ac5d0 2024-09-19T08:06:54.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.589+0000 7fb4d0e31740 1 -- 172.21.15.98:0/714779895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605ac5d0 msgr2=0x55a7605a8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:54.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.589+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/714779895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605ac5d0 0x55a7605a8df0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc02a930 tx=0x7fb4bc005a20 comp rx=0 tx=0).stop 2024-09-19T08:06:54.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.589+0000 7fb4d0e31740 1 -- 172.21.15.98:0/714779895 shutdown_connections 2024-09-19T08:06:54.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.589+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/714779895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605b7370 0x55a7605cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.589+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/714779895 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605adb40 0x55a7605cc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.589+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/714779895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605ac5d0 0x55a7605a8df0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.589+0000 7fb4d0e31740 1 -- 172.21.15.98:0/714779895 >> 172.21.15.98:0/714779895 conn(0x55a760575310 msgr2=0x55a7605a86e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:54.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.589+0000 7fb4d0e31740 1 -- 172.21.15.98:0/714779895 shutdown_connections 2024-09-19T08:06:54.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.589+0000 7fb4d0e31740 1 -- 172.21.15.98:0/714779895 wait complete. 2024-09-19T08:06:54.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.589+0000 7fb4d0e31740 1 -- start start 2024-09-19T08:06:54.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.589+0000 7fb4d0e31740 1 Processor -- start 2024-09-19T08:06:54.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4d0e31740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605ac5d0 0x55a7605cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4d0e31740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605adb40 0x55a7605c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4d0e31740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605b7370 0x55a7605ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4d0e31740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c81aebf0 con 0x55a7605adb40 2024-09-19T08:06:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4d0e31740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c81ae520 con 0x55a7605ac5d0 2024-09-19T08:06:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4d0e31740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c81ab730 con 0x55a7605b7370 2024-09-19T08:06:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4d0e31740 10 client.0 _ll_register_callbacks cb 0x7fb4b800b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4c7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605adb40 0x55a7605c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4c7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605adb40 0x55a7605c9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34184/0 (socket says 172.21.15.98:34184) 2024-09-19T08:06:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4c7fff640 1 -- 172.21.15.98:0/2800219459 learned_addr learned my addr 172.21.15.98:0/2800219459 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/2800219459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605ac5d0 0x55a7605cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/2800219459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605b7370 0x55a7605ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4c7fff640 1 -- 172.21.15.98:0/2800219459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605b7370 msgr2=0x55a7605ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4c7fff640 1 --2- 172.21.15.98:0/2800219459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605b7370 0x55a7605ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4c7fff640 1 -- 172.21.15.98:0/2800219459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605ac5d0 msgr2=0x55a7605cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4c7fff640 1 --2- 172.21.15.98:0/2800219459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605ac5d0 0x55a7605cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4c7fff640 1 -- 172.21.15.98:0/2800219459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7605caad0 con 0x55a7605adb40 2024-09-19T08:06:54.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/2800219459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605ac5d0 0x55a7605cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:54.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4c7fff640 1 --2- 172.21.15.98:0/2800219459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605adb40 0x55a7605c9d40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb4b800b370 tx=0x7fb4b8012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16720afb589637e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:54.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:54.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4a77fe640 1 -- 172.21.15.98:0/2800219459 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8005460 con 0x55a7605adb40 2024-09-19T08:06:54.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4a77fe640 1 -- 172.21.15.98:0/2800219459 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fb4b8004730 con 0x55a7605adb40 2024-09-19T08:06:54.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.590+0000 7fb4d0e31740 1 -- 172.21.15.98:0/2800219459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4bc0315b0 con 0x55a7605adb40 2024-09-19T08:06:54.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.591+0000 7fb4a77fe640 1 -- 172.21.15.98:0/2800219459 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb4b802b050 con 0x55a7605adb40 2024-09-19T08:06:54.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.591+0000 7fb4d0e31740 10 client.16440 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:54.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.591+0000 7fb4d0e31740 1 -- 172.21.15.98:0/2800219459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a7606317d0 con 0x55a7605adb40 2024-09-19T08:06:54.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.591+0000 7fb4a77fe640 1 -- 172.21.15.98:0/2800219459 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4bc0315b0 con 0x55a7605adb40 2024-09-19T08:06:54.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.591+0000 7fb4a77fe640 1 -- 172.21.15.98:0/2800219459 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fb4b801b8c0 con 0x55a7605adb40 2024-09-19T08:06:54.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.591+0000 7fb4d0e31740 10 client.16440 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:54.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.591+0000 7fb4d0e31740 10 client.16440 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:06:54.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.591+0000 7fb4d0e31740 20 client.16440 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:54.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.591+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/2800219459 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a760631f60 0x55a760634380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:54.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.591+0000 7fb4d0e31740 4 client.16440 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:06:54.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.591+0000 7fb4d0e31740 1 -- 172.21.15.98:0/2800219459 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a760519d70 con 0x55a760631f60 2024-09-19T08:06:54.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.591+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/2800219459 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a760631f60 0x55a760634380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:54.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4a77fe640 10 client.16440 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:06:54.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4c77fe640 1 --2- 172.21.15.98:0/2800219459 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a760631f60 0x55a760634380 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=61a5377845261d17 server_cookie=5db68023fa54dd21 in_seq=0 out_seq=0 2024-09-19T08:06:54.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4a77fe640 1 -- 172.21.15.98:0/2800219459 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a760519d70 con 0x55a760631f60 2024-09-19T08:06:54.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4a77fe640 10 client.16440 handle_command_reply: tid=0 2024-09-19T08:06:54.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4d0e31740 1 client.16440 shutdown 2024-09-19T08:06:54.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4d0e31740 1 -- 172.21.15.98:0/2800219459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605adb40 msgr2=0x55a7605c9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:54.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/2800219459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605adb40 0x55a7605c9d40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb4b800b370 tx=0x7fb4b8012820 comp rx=0 tx=0).stop 2024-09-19T08:06:54.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4d0e31740 1 -- 172.21.15.98:0/2800219459 shutdown_connections 2024-09-19T08:06:54.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/2800219459 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a760631f60 0x55a760634380 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/2800219459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a7605b7370 0x55a7605ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/2800219459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7605adb40 0x55a7605c9d40 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4d0e31740 1 --2- 172.21.15.98:0/2800219459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7605ac5d0 0x55a7605cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4a77fe640 0 client.16440 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:06:54.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4d0e31740 1 -- 172.21.15.98:0/2800219459 >> 172.21.15.98:0/2800219459 conn(0x55a760575310 msgr2=0x55a7605b5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:54.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4d0e31740 1 -- 172.21.15.98:0/2800219459 shutdown_connections 2024-09-19T08:06:54.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4d0e31740 1 -- 172.21.15.98:0/2800219459 wait complete. 2024-09-19T08:06:54.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.592+0000 7fb4d0e31740 20 client.16440 trim_cache size 0 max 16384 2024-09-19T08:06:54.623 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:06:54.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.593+0000 7fb4cea58640 1 -- 172.21.15.98:0/3503778945 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb4b00a33b0 msgr2=0x7fb4b00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:54.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.593+0000 7fb4cea58640 1 --2- 172.21.15.98:0/3503778945 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb4b00a33b0 0x7fb4b00a5870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000efd0 tx=0x7fb4c0086ae0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.593+0000 7fb4cea58640 1 -- 172.21.15.98:0/3503778945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8106f30 msgr2=0x7fb4c819cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:54.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.593+0000 7fb4cea58640 1 --2- 172.21.15.98:0/3503778945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8106f30 0x7fb4c819cda0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000eaa0 tx=0x7fb4c000ef60 comp rx=0 tx=0).stop 2024-09-19T08:06:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.593+0000 7fb4cea58640 1 -- 172.21.15.98:0/3503778945 shutdown_connections 2024-09-19T08:06:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.593+0000 7fb4cea58640 1 --2- 172.21.15.98:0/3503778945 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb4b00a33b0 0x7fb4b00a5870 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.593+0000 7fb4cea58640 1 --2- 172.21.15.98:0/3503778945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8106f30 0x7fb4c819cda0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.593+0000 7fb4cea58640 1 --2- 172.21.15.98:0/3503778945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4c8100cc0 0x7fb4c8198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.593+0000 7fb4cea58640 1 --2- 172.21.15.98:0/3503778945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4c8100310 0x7fb4c81984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.593+0000 7fb4cea58640 1 -- 172.21.15.98:0/3503778945 >> 172.21.15.98:0/3503778945 conn(0x7fb4c8077210 msgr2=0x7fb4c8105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.593+0000 7fb4cea58640 1 -- 172.21.15.98:0/3503778945 shutdown_connections 2024-09-19T08:06:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:54.593+0000 7fb4cea58640 1 -- 172.21.15.98:0/3503778945 wait complete. 2024-09-19T08:06:55.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:06:55.654 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:56.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:55 smithi098 ceph-mon[31333]: pgmap v120: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:06:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.090+0000 7f4910b12640 1 -- 172.21.15.98:0/2558491306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f490c101de0 msgr2=0x7f490c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.090+0000 7f4910b12640 1 --2- 172.21.15.98:0/2558491306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f490c101de0 0x7f490c1021c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f4904009900 tx=0x7f490402a3f0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.090+0000 7f4910b12640 1 -- 172.21.15.98:0/2558491306 shutdown_connections 2024-09-19T08:06:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.090+0000 7f4910b12640 1 --2- 172.21.15.98:0/2558491306 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f490c106f10 0x7f490c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.090+0000 7f4910b12640 1 --2- 172.21.15.98:0/2558491306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f490c102790 0x7f490c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.090+0000 7f4910b12640 1 --2- 172.21.15.98:0/2558491306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f490c101de0 0x7f490c1021c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.090+0000 7f4910b12640 1 -- 172.21.15.98:0/2558491306 >> 172.21.15.98:0/2558491306 conn(0x7f490c0fd8c0 msgr2=0x7f490c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.090+0000 7f4910b12640 1 -- 172.21.15.98:0/2558491306 shutdown_connections 2024-09-19T08:06:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.090+0000 7f4910b12640 1 -- 172.21.15.98:0/2558491306 wait complete. 2024-09-19T08:06:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.091+0000 7f4910b12640 1 Processor -- start 2024-09-19T08:06:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.091+0000 7f4910b12640 1 -- start start 2024-09-19T08:06:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.091+0000 7f4910b12640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f490c101de0 0x7f490c19a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.091+0000 7f4910b12640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f490c102790 0x7f490c19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.091+0000 7f4910b12640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f490c106f10 0x7f490c19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.091+0000 7f4910b12640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f490c111f30 con 0x7f490c106f10 2024-09-19T08:06:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.091+0000 7f4910b12640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f490c111db0 con 0x7f490c102790 2024-09-19T08:06:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.091+0000 7f4910b12640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f490c1120b0 con 0x7f490c101de0 2024-09-19T08:06:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.091+0000 7f490ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f490c106f10 0x7f490c19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.091+0000 7f490ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f490c106f10 0x7f490c19ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34198/0 (socket says 172.21.15.98:34198) 2024-09-19T08:06:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.091+0000 7f490ad9d640 1 -- 172.21.15.98:0/2200841834 learned_addr learned my addr 172.21.15.98:0/2200841834 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.091+0000 7f490a59c640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f490c101de0 0x7f490c19a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.092+0000 7f4909d9b640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f490c102790 0x7f490c19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.092+0000 7f490ad9d640 1 -- 172.21.15.98:0/2200841834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f490c101de0 msgr2=0x7f490c19a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.092+0000 7f490ad9d640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f490c101de0 0x7f490c19a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.092+0000 7f490ad9d640 1 -- 172.21.15.98:0/2200841834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f490c102790 msgr2=0x7f490c19aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.092+0000 7f490ad9d640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f490c102790 0x7f490c19aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.092+0000 7f490ad9d640 1 -- 172.21.15.98:0/2200841834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f490c19f4e0 con 0x7f490c106f10 2024-09-19T08:06:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.092+0000 7f4909d9b640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f490c102790 0x7f490c19aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.092+0000 7f490ad9d640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f490c106f10 0x7f490c19ee00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00c970 tx=0x7f48fc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.092+0000 7f48fb7fe640 1 -- 172.21.15.98:0/2200841834 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48fc007bb0 con 0x7f490c106f10 2024-09-19T08:06:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.092+0000 7f48fb7fe640 1 -- 172.21.15.98:0/2200841834 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48fc007d50 con 0x7f490c106f10 2024-09-19T08:06:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.092+0000 7f4910b12640 1 -- 172.21.15.98:0/2200841834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f490c19f7d0 con 0x7f490c106f10 2024-09-19T08:06:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.092+0000 7f48fb7fe640 1 -- 172.21.15.98:0/2200841834 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48fc018800 con 0x7f490c106f10 2024-09-19T08:06:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.092+0000 7f4910b12640 1 -- 172.21.15.98:0/2200841834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f490c1a70b0 con 0x7f490c106f10 2024-09-19T08:06:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.094+0000 7f4912eeb740 1 Processor -- start 2024-09-19T08:06:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.095+0000 7f48fb7fe640 1 -- 172.21.15.98:0/2200841834 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f48fc00fa50 con 0x7f490c106f10 2024-09-19T08:06:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.095+0000 7f4912eeb740 1 -- start start 2024-09-19T08:06:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.095+0000 7f4912eeb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd333b40 0x5610fd332090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.095+0000 7f4912eeb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd3325d0 0x5610fd332a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.095+0000 7f4912eeb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd33d370 0x5610fd33f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.095+0000 7f4912eeb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5610fd2982f0 con 0x5610fd3325d0 2024-09-19T08:06:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.095+0000 7f4912eeb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5610fd23fcf0 con 0x5610fd333b40 2024-09-19T08:06:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.095+0000 7f4912eeb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5610fd2942b0 con 0x5610fd33d370 2024-09-19T08:06:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.095+0000 7f490a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd3325d0 0x5610fd332a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.095+0000 7f490a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd3325d0 0x5610fd332a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34206/0 (socket says 172.21.15.98:34206) 2024-09-19T08:06:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.095+0000 7f490a59c640 1 -- 172.21.15.98:0/3378600387 learned_addr learned my addr 172.21.15.98:0/3378600387 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.095+0000 7f4909d9b640 1 --2- 172.21.15.98:0/3378600387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd333b40 0x5610fd332090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.095+0000 7f48fb7fe640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f48f40a33b0 0x7f48f40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.095+0000 7f4909d9b640 1 --2- 172.21.15.98:0/3378600387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd33d370 0x5610fd33f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.096+0000 7f490a59c640 1 -- 172.21.15.98:0/3378600387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd33d370 msgr2=0x5610fd33f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.096+0000 7f490a59c640 1 --2- 172.21.15.98:0/3378600387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd33d370 0x5610fd33f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.096+0000 7f490a59c640 1 -- 172.21.15.98:0/3378600387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd333b40 msgr2=0x5610fd332090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.096+0000 7f490a59c640 1 --2- 172.21.15.98:0/3378600387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd333b40 0x5610fd332090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.096+0000 7f490a59c640 1 -- 172.21.15.98:0/3378600387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610fd2243a0 con 0x5610fd3325d0 2024-09-19T08:06:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.096+0000 7f4909d9b640 1 --2- 172.21.15.98:0/3378600387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd33d370 0x5610fd33f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:56.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.096+0000 7f490ad9d640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f48f40a33b0 0x7f48f40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.096+0000 7f490a59c640 1 --2- 172.21.15.98:0/3378600387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd3325d0 0x5610fd332a10 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f4904009900 tx=0x7f49040316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f97c698cca4c11 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:56.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.096+0000 7f48fb7fe640 1 -- 172.21.15.98:0/2200841834 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f48fc0cfcf0 con 0x7f490c106f10 2024-09-19T08:06:56.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.096+0000 7f48f8ff9640 1 -- 172.21.15.98:0/3378600387 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4904040070 con 0x5610fd3325d0 2024-09-19T08:06:56.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.096+0000 7f48f8ff9640 1 -- 172.21.15.98:0/3378600387 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4904005020 con 0x5610fd3325d0 2024-09-19T08:06:56.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.096+0000 7f48f8ff9640 1 -- 172.21.15.98:0/3378600387 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4904035040 con 0x5610fd3325d0 2024-09-19T08:06:56.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.097+0000 7f4912eeb740 1 -- 172.21.15.98:0/3378600387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd3325d0 msgr2=0x5610fd332a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:56.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.097+0000 7f490ad9d640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f48f40a33b0 0x7f48f40a5870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f48fc09bd00 tx=0x7f48fc0868f0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:56.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.097+0000 7f4912eeb740 1 --2- 172.21.15.98:0/3378600387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd3325d0 0x5610fd332a10 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f4904009900 tx=0x7f49040316e0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.097+0000 7f4912eeb740 1 -- 172.21.15.98:0/3378600387 shutdown_connections 2024-09-19T08:06:56.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.097+0000 7f4912eeb740 1 --2- 172.21.15.98:0/3378600387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd33d370 0x5610fd33f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.097+0000 7f4912eeb740 1 --2- 172.21.15.98:0/3378600387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd3325d0 0x5610fd332a10 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.097+0000 7f4912eeb740 1 --2- 172.21.15.98:0/3378600387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd333b40 0x5610fd332090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.097+0000 7f4912eeb740 1 -- 172.21.15.98:0/3378600387 >> 172.21.15.98:0/3378600387 conn(0x5610fd2fb310 msgr2=0x5610fd32ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:56.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.097+0000 7f4912eeb740 1 -- 172.21.15.98:0/3378600387 shutdown_connections 2024-09-19T08:06:56.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.097+0000 7f4912eeb740 1 -- 172.21.15.98:0/3378600387 wait complete. 2024-09-19T08:06:56.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.097+0000 7f4912eeb740 1 -- start start 2024-09-19T08:06:56.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.097+0000 7f4912eeb740 1 Processor -- start 2024-09-19T08:06:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f4912eeb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd3325d0 0x5610fd33cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f4912eeb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd333b40 0x5610fd339f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f4912eeb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd33d370 0x5610fd33a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f4912eeb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5610fd340010 con 0x5610fd333b40 2024-09-19T08:06:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f4912eeb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5610fd253060 con 0x5610fd3325d0 2024-09-19T08:06:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f4912eeb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5610fd340190 con 0x5610fd33d370 2024-09-19T08:06:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f4912eeb740 10 client.0 _ll_register_callbacks cb 0x5610fcfa68c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f490a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd333b40 0x5610fd339f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f490a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd333b40 0x5610fd339f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34208/0 (socket says 172.21.15.98:34208) 2024-09-19T08:06:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f490a59c640 1 -- 172.21.15.98:0/3121940912 learned_addr learned my addr 172.21.15.98:0/3121940912 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f4909d9b640 1 --2- 172.21.15.98:0/3121940912 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd3325d0 0x5610fd33cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f4909d9b640 1 --2- 172.21.15.98:0/3121940912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd33d370 0x5610fd33a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f490a59c640 1 -- 172.21.15.98:0/3121940912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd33d370 msgr2=0x5610fd33a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f490a59c640 1 --2- 172.21.15.98:0/3121940912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd33d370 0x5610fd33a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f490a59c640 1 -- 172.21.15.98:0/3121940912 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd3325d0 msgr2=0x5610fd33cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f490a59c640 1 --2- 172.21.15.98:0/3121940912 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd3325d0 0x5610fd33cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f490a59c640 1 -- 172.21.15.98:0/3121940912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610fd0093c0 con 0x5610fd333b40 2024-09-19T08:06:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f4909d9b640 1 --2- 172.21.15.98:0/3121940912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd33d370 0x5610fd33a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f490a59c640 1 --2- 172.21.15.98:0/3121940912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd333b40 0x5610fd339f90 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4904009520 tx=0x7f4904009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7414cbda32557bfa server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f48f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f48f1ffb640 1 -- 172.21.15.98:0/3121940912 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490404c020 con 0x5610fd333b40 2024-09-19T08:06:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f48f1ffb640 1 -- 172.21.15.98:0/3121940912 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f49040048f0 con 0x5610fd333b40 2024-09-19T08:06:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.098+0000 7f4912eeb740 1 -- 172.21.15.98:0/3121940912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4904005020 con 0x5610fd333b40 2024-09-19T08:06:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.099+0000 7f48f1ffb640 1 -- 172.21.15.98:0/3121940912 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4904057050 con 0x5610fd333b40 2024-09-19T08:06:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.099+0000 7f4912eeb740 10 client.16452 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.099+0000 7f4912eeb740 1 -- 172.21.15.98:0/3121940912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5610fd34b990 con 0x5610fd333b40 2024-09-19T08:06:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.099+0000 7f48f1ffb640 1 -- 172.21.15.98:0/3121940912 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4904005020 con 0x5610fd333b40 2024-09-19T08:06:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.099+0000 7f48f1ffb640 1 -- 172.21.15.98:0/3121940912 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f4904044400 con 0x5610fd333b40 2024-09-19T08:06:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.099+0000 7f4912eeb740 10 client.16452 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.099+0000 7f4912eeb740 10 client.16452 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:06:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.099+0000 7f4912eeb740 20 client.16452 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.099+0000 7f4912eeb740 1 --2- 172.21.15.98:0/3121940912 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5610fd34cce0 0x5610fd34f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.099+0000 7f4912eeb740 4 client.16452 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.099+0000 7f4912eeb740 1 -- 172.21.15.98:0/3121940912 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610fd2a5860 con 0x5610fd34cce0 2024-09-19T08:06:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.099+0000 7f4909d9b640 1 --2- 172.21.15.98:0/3121940912 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5610fd34cce0 0x5610fd34f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.100+0000 7f48f1ffb640 10 client.16452 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:06:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.100+0000 7f4909d9b640 1 --2- 172.21.15.98:0/3121940912 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5610fd34cce0 0x5610fd34f0d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b5d8d31e84b86a09 server_cookie=63d0c32a2e3896c5 in_seq=0 out_seq=0 2024-09-19T08:06:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f48f1ffb640 1 -- 172.21.15.98:0/3121940912 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5610fd2a5860 con 0x5610fd34cce0 2024-09-19T08:06:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f48f1ffb640 10 client.16452 handle_command_reply: tid=0 2024-09-19T08:06:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f4912eeb740 1 client.16452 shutdown 2024-09-19T08:06:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f4912eeb740 1 -- 172.21.15.98:0/3121940912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd333b40 msgr2=0x5610fd339f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f4912eeb740 1 --2- 172.21.15.98:0/3121940912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd333b40 0x5610fd339f90 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4904009520 tx=0x7f4904009550 comp rx=0 tx=0).stop 2024-09-19T08:06:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f4912eeb740 1 -- 172.21.15.98:0/3121940912 shutdown_connections 2024-09-19T08:06:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f4912eeb740 1 --2- 172.21.15.98:0/3121940912 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5610fd34cce0 0x5610fd34f0d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f4912eeb740 1 --2- 172.21.15.98:0/3121940912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd33d370 0x5610fd33a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f48f1ffb640 0 client.16452 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:06:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f4912eeb740 1 --2- 172.21.15.98:0/3121940912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd333b40 0x5610fd339f90 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f4912eeb740 1 --2- 172.21.15.98:0/3121940912 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd3325d0 0x5610fd33cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f4912eeb740 1 -- 172.21.15.98:0/3121940912 >> 172.21.15.98:0/3121940912 conn(0x5610fd2fb310 msgr2=0x5610fd334930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f4912eeb740 1 -- 172.21.15.98:0/3121940912 shutdown_connections 2024-09-19T08:06:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f4912eeb740 1 -- 172.21.15.98:0/3121940912 wait complete. 2024-09-19T08:06:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.101+0000 7f4912eeb740 20 client.16452 trim_cache size 0 max 16384 2024-09-19T08:06:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.117+0000 7f4912eeb740 1 Processor -- start 2024-09-19T08:06:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4912eeb740 1 -- start start 2024-09-19T08:06:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4912eeb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd3325d0 0x5610fd33c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4912eeb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd333b40 0x5610fd3492b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4912eeb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd33d370 0x5610fd354700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4912eeb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f490c1af8b0 con 0x5610fd3325d0 2024-09-19T08:06:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4912eeb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f490c1b1450 con 0x5610fd33d370 2024-09-19T08:06:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4912eeb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f490c1aff80 con 0x5610fd333b40 2024-09-19T08:06:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4909d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd3325d0 0x5610fd33c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4909d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd3325d0 0x5610fd33c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34214/0 (socket says 172.21.15.98:34214) 2024-09-19T08:06:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4909d9b640 1 -- 172.21.15.98:0/126185150 learned_addr learned my addr 172.21.15.98:0/126185150 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f490a59c640 1 --2- 172.21.15.98:0/126185150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd333b40 0x5610fd3492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4909d9b640 1 -- 172.21.15.98:0/126185150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd333b40 msgr2=0x5610fd3492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4909d9b640 1 --2- 172.21.15.98:0/126185150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd333b40 0x5610fd3492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4909d9b640 1 -- 172.21.15.98:0/126185150 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd33d370 msgr2=0x5610fd354700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4909d9b640 1 --2- 172.21.15.98:0/126185150 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd33d370 0x5610fd354700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4909d9b640 1 -- 172.21.15.98:0/126185150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610fd33a860 con 0x5610fd3325d0 2024-09-19T08:06:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.118+0000 7f4909d9b640 1 --2- 172.21.15.98:0/126185150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd3325d0 0x5610fd33c2c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f4900007730 tx=0x7f4900007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d07f1e7f51b258 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.119+0000 7f48f3fff640 1 -- 172.21.15.98:0/126185150 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4900004b20 con 0x5610fd3325d0 2024-09-19T08:06:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.119+0000 7f48f3fff640 1 -- 172.21.15.98:0/126185150 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4900004cc0 con 0x5610fd3325d0 2024-09-19T08:06:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.119+0000 7f48f3fff640 1 -- 172.21.15.98:0/126185150 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49000130d0 con 0x5610fd3325d0 2024-09-19T08:06:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.119+0000 7f4912eeb740 1 -- 172.21.15.98:0/126185150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd3325d0 msgr2=0x5610fd33c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.119+0000 7f4912eeb740 1 --2- 172.21.15.98:0/126185150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd3325d0 0x5610fd33c2c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f4900007730 tx=0x7f4900007e20 comp rx=0 tx=0).stop 2024-09-19T08:06:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.119+0000 7f4912eeb740 1 -- 172.21.15.98:0/126185150 shutdown_connections 2024-09-19T08:06:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.119+0000 7f4912eeb740 1 --2- 172.21.15.98:0/126185150 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd33d370 0x5610fd354700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.119+0000 7f4912eeb740 1 --2- 172.21.15.98:0/126185150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd333b40 0x5610fd3492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.119+0000 7f4912eeb740 1 --2- 172.21.15.98:0/126185150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd3325d0 0x5610fd33c2c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.119+0000 7f4912eeb740 1 -- 172.21.15.98:0/126185150 >> 172.21.15.98:0/126185150 conn(0x5610fd2fb310 msgr2=0x5610fd338f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.119+0000 7f4912eeb740 1 -- 172.21.15.98:0/126185150 shutdown_connections 2024-09-19T08:06:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f4912eeb740 1 -- 172.21.15.98:0/126185150 wait complete. 2024-09-19T08:06:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f4912eeb740 1 -- start start 2024-09-19T08:06:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f4912eeb740 1 Processor -- start 2024-09-19T08:06:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f4912eeb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd3325d0 0x5610fd345d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f4912eeb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd333b40 0x5610fd336e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f4912eeb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd33d370 0x5610fd337560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f4912eeb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f490c1b0d50 con 0x5610fd33d370 2024-09-19T08:06:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f4912eeb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f490c1b0680 con 0x5610fd3325d0 2024-09-19T08:06:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f4912eeb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f490c1ad890 con 0x5610fd333b40 2024-09-19T08:06:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f4912eeb740 10 client.0 _ll_register_callbacks cb 0x7f49040315b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f4909d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd33d370 0x5610fd337560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f4909d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd33d370 0x5610fd337560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34220/0 (socket says 172.21.15.98:34220) 2024-09-19T08:06:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f490a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd333b40 0x5610fd336e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f4909d9b640 1 -- 172.21.15.98:0/2089576857 learned_addr learned my addr 172.21.15.98:0/2089576857 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:56.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.120+0000 7f4909d9b640 1 --2- 172.21.15.98:0/2089576857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd3325d0 0x5610fd345d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f4909d9b640 1 -- 172.21.15.98:0/2089576857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd333b40 msgr2=0x5610fd336e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:56.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f4909d9b640 1 --2- 172.21.15.98:0/2089576857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd333b40 0x5610fd336e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f4909d9b640 1 -- 172.21.15.98:0/2089576857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd3325d0 msgr2=0x5610fd345d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f4909d9b640 1 --2- 172.21.15.98:0/2089576857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd3325d0 0x5610fd345d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f4909d9b640 1 -- 172.21.15.98:0/2089576857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610fd3b66d0 con 0x5610fd33d370 2024-09-19T08:06:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f490a59c640 1 --2- 172.21.15.98:0/2089576857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd333b40 0x5610fd336e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:06:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f4909d9b640 1 --2- 172.21.15.98:0/2089576857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd33d370 0x5610fd337560 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f4900007c10 tx=0x7f4900010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc55ad34c872ff0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f48f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f48f1ffb640 1 -- 172.21.15.98:0/2089576857 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4900011a30 con 0x5610fd33d370 2024-09-19T08:06:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f48f1ffb640 1 -- 172.21.15.98:0/2089576857 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f4900011bd0 con 0x5610fd33d370 2024-09-19T08:06:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f4912eeb740 1 -- 172.21.15.98:0/2089576857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4900004cc0 con 0x5610fd33d370 2024-09-19T08:06:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f48f1ffb640 1 -- 172.21.15.98:0/2089576857 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f49000130d0 con 0x5610fd33d370 2024-09-19T08:06:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f4912eeb740 10 client.16458 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f4912eeb740 1 -- 172.21.15.98:0/2089576857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5610fd3b7830 con 0x5610fd33d370 2024-09-19T08:06:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.121+0000 7f48f1ffb640 1 -- 172.21.15.98:0/2089576857 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4900004cc0 con 0x5610fd33d370 2024-09-19T08:06:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.122+0000 7f48f1ffb640 1 -- 172.21.15.98:0/2089576857 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f4900026070 con 0x5610fd33d370 2024-09-19T08:06:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.122+0000 7f4912eeb740 10 client.16458 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.122+0000 7f4912eeb740 10 client.16458 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:06:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.122+0000 7f4912eeb740 20 client.16458 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.122+0000 7f4912eeb740 1 --2- 172.21.15.98:0/2089576857 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5610fd3b7e40 0x5610fd3ba260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.122+0000 7f4912eeb740 4 client.16458 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:06:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.122+0000 7f4912eeb740 1 -- 172.21.15.98:0/2089576857 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5610fd29fd70 con 0x5610fd3b7e40 2024-09-19T08:06:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.122+0000 7f490a59c640 1 --2- 172.21.15.98:0/2089576857 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5610fd3b7e40 0x5610fd3ba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f48f1ffb640 10 client.16458 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:06:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f490a59c640 1 --2- 172.21.15.98:0/2089576857 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5610fd3b7e40 0x5610fd3ba260 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=32723aa189064e3a server_cookie=1f52a45a57fe3371 in_seq=0 out_seq=0 2024-09-19T08:06:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f48f1ffb640 1 -- 172.21.15.98:0/2089576857 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5610fd29fd70 con 0x5610fd3b7e40 2024-09-19T08:06:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f48f1ffb640 10 client.16458 handle_command_reply: tid=0 2024-09-19T08:06:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f4912eeb740 1 client.16458 shutdown 2024-09-19T08:06:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f4912eeb740 1 -- 172.21.15.98:0/2089576857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd33d370 msgr2=0x5610fd337560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f4912eeb740 1 --2- 172.21.15.98:0/2089576857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd33d370 0x5610fd337560 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f4900007c10 tx=0x7f4900010ce0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f4912eeb740 1 -- 172.21.15.98:0/2089576857 shutdown_connections 2024-09-19T08:06:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f4912eeb740 1 --2- 172.21.15.98:0/2089576857 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5610fd3b7e40 0x5610fd3ba260 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f4912eeb740 1 --2- 172.21.15.98:0/2089576857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610fd33d370 0x5610fd337560 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f4912eeb740 1 --2- 172.21.15.98:0/2089576857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610fd333b40 0x5610fd336e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f4912eeb740 1 --2- 172.21.15.98:0/2089576857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5610fd3325d0 0x5610fd345d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f4912eeb740 1 -- 172.21.15.98:0/2089576857 >> 172.21.15.98:0/2089576857 conn(0x5610fd2fb310 msgr2=0x5610fd334110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f48f1ffb640 0 client.16458 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:06:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f4912eeb740 1 -- 172.21.15.98:0/2089576857 shutdown_connections 2024-09-19T08:06:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f4912eeb740 1 -- 172.21.15.98:0/2089576857 wait complete. 2024-09-19T08:06:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.123+0000 7f4912eeb740 20 client.16458 trim_cache size 0 max 16384 2024-09-19T08:06:56.143 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:06:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.124+0000 7f4910b12640 1 -- 172.21.15.98:0/2200841834 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f48f40a33b0 msgr2=0x7f48f40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.124+0000 7f4910b12640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f48f40a33b0 0x7f48f40a5870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f48fc09bd00 tx=0x7f48fc0868f0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.124+0000 7f4910b12640 1 -- 172.21.15.98:0/2200841834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f490c106f10 msgr2=0x7f490c19ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.124+0000 7f4910b12640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f490c106f10 0x7f490c19ee00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00c970 tx=0x7f48fc00ce30 comp rx=0 tx=0).stop 2024-09-19T08:06:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.124+0000 7f4910b12640 1 -- 172.21.15.98:0/2200841834 shutdown_connections 2024-09-19T08:06:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.124+0000 7f4910b12640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f48f40a33b0 0x7f48f40a5870 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.124+0000 7f4910b12640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f490c106f10 0x7f490c19ee00 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.124+0000 7f4910b12640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f490c102790 0x7f490c19aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.124+0000 7f4910b12640 1 --2- 172.21.15.98:0/2200841834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f490c101de0 0x7f490c19a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.124+0000 7f4910b12640 1 -- 172.21.15.98:0/2200841834 >> 172.21.15.98:0/2200841834 conn(0x7f490c0fd8c0 msgr2=0x7f490c0ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.124+0000 7f4910b12640 1 -- 172.21.15.98:0/2200841834 shutdown_connections 2024-09-19T08:06:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:56.124+0000 7f4910b12640 1 -- 172.21.15.98:0/2200841834 wait complete. 2024-09-19T08:06:56.188 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:55 smithi139 ceph-mon[34874]: pgmap v120: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:06:56.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:55 smithi162 ceph-mon[35260]: pgmap v120: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:06:56.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:06:56.925 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.415+0000 7f61fc287640 1 -- 172.21.15.98:0/2695280049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f4100a80 msgr2=0x7f61f4100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.415+0000 7f61fc287640 1 --2- 172.21.15.98:0/2695280049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f4100a80 0x7f61f4100ee0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f61e4009930 tx=0x7f61e402a420 comp rx=0 tx=0).stop 2024-09-19T08:06:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.415+0000 7f61fc287640 1 -- 172.21.15.98:0/2695280049 shutdown_connections 2024-09-19T08:06:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.415+0000 7f61fc287640 1 --2- 172.21.15.98:0/2695280049 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61f4101420 0x7f61f410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.415+0000 7f61fc287640 1 --2- 172.21.15.98:0/2695280049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f4100a80 0x7f61f4100ee0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.415+0000 7f61fc287640 1 --2- 172.21.15.98:0/2695280049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61f4106a80 0x7f61f4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.415+0000 7f61fc287640 1 -- 172.21.15.98:0/2695280049 >> 172.21.15.98:0/2695280049 conn(0x7f61f40fc7a0 msgr2=0x7f61f40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.415+0000 7f61fc287640 1 -- 172.21.15.98:0/2695280049 shutdown_connections 2024-09-19T08:06:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.415+0000 7f61fc287640 1 -- 172.21.15.98:0/2695280049 wait complete. 2024-09-19T08:06:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.416+0000 7f61fc287640 1 Processor -- start 2024-09-19T08:06:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.416+0000 7f61fc287640 1 -- start start 2024-09-19T08:06:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.416+0000 7f61fc287640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61f4100a80 0x7f61f41a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.416+0000 7f61fc287640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f4101420 0x7f61f41a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.416+0000 7f61fc287640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61f4106a80 0x7f61f419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.416+0000 7f61fc287640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61f4110f20 con 0x7f61f4101420 2024-09-19T08:06:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.416+0000 7f61fc287640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f61f4110da0 con 0x7f61f4100a80 2024-09-19T08:06:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.416+0000 7f61fc287640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f61f41110a0 con 0x7f61f4106a80 2024-09-19T08:06:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.416+0000 7f61f9822640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f4101420 0x7f61f41a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.416+0000 7f61f9822640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f4101420 0x7f61f41a09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40294/0 (socket says 172.21.15.98:40294) 2024-09-19T08:06:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.416+0000 7f61f9822640 1 -- 172.21.15.98:0/1645740268 learned_addr learned my addr 172.21.15.98:0/1645740268 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.416+0000 7f61fa023640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61f4100a80 0x7f61f41a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.416+0000 7f61fa824640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61f4106a80 0x7f61f419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.417+0000 7f61f9822640 1 -- 172.21.15.98:0/1645740268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61f4106a80 msgr2=0x7f61f419a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.417+0000 7f61f9822640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61f4106a80 0x7f61f419a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.417+0000 7f61f9822640 1 -- 172.21.15.98:0/1645740268 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61f4100a80 msgr2=0x7f61f41a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.417+0000 7f61f9822640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61f4100a80 0x7f61f41a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.417+0000 7f61f9822640 1 -- 172.21.15.98:0/1645740268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f419ad50 con 0x7f61f4101420 2024-09-19T08:06:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.417+0000 7f61fa023640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61f4100a80 0x7f61f41a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.417+0000 7f61fa824640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61f4106a80 0x7f61f419a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.417+0000 7f61f9822640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f4101420 0x7f61f41a09a0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f61e4009a50 tx=0x7f61e4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.417+0000 7f61eb7fe640 1 -- 172.21.15.98:0/1645740268 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e403f030 con 0x7f61f4101420 2024-09-19T08:06:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.417+0000 7f61eb7fe640 1 -- 172.21.15.98:0/1645740268 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61e4003ce0 con 0x7f61f4101420 2024-09-19T08:06:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.417+0000 7f61fc287640 1 -- 172.21.15.98:0/1645740268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f419afe0 con 0x7f61f4101420 2024-09-19T08:06:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.417+0000 7f61eb7fe640 1 -- 172.21.15.98:0/1645740268 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e4035070 con 0x7f61f4101420 2024-09-19T08:06:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.417+0000 7f61fc287640 1 -- 172.21.15.98:0/1645740268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f41a7330 con 0x7f61f4101420 2024-09-19T08:06:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.419+0000 7f61eb7fe640 1 -- 172.21.15.98:0/1645740268 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f61e4042030 con 0x7f61f4101420 2024-09-19T08:06:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.419+0000 7f61fe660740 1 Processor -- start 2024-09-19T08:06:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fe660740 1 -- start start 2024-09-19T08:06:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fe660740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aaeb40 0x555632aad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fe660740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632aad5d0 0x555632aada10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fe660740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632ab8370 0x555632aba760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fe660740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555632a132f0 con 0x555632aaeb40 2024-09-19T08:06:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fe660740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5556329bacf0 con 0x555632aad5d0 2024-09-19T08:06:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fe660740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555632a0f2b0 con 0x555632ab8370 2024-09-19T08:06:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fa824640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aaeb40 0x555632aad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fa824640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aaeb40 0x555632aad090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40296/0 (socket says 172.21.15.98:40296) 2024-09-19T08:06:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fa824640 1 -- 172.21.15.98:0/4269162415 learned_addr learned my addr 172.21.15.98:0/4269162415 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61eb7fe640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f61e00a33b0 0x7f61e00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61f9822640 1 --2- 172.21.15.98:0/4269162415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632ab8370 0x555632aba760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fa023640 1 --2- 172.21.15.98:0/4269162415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632aad5d0 0x555632aada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fa824640 1 -- 172.21.15.98:0/4269162415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632ab8370 msgr2=0x555632aba760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fa824640 1 --2- 172.21.15.98:0/4269162415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632ab8370 0x555632aba760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fa824640 1 -- 172.21.15.98:0/4269162415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632aad5d0 msgr2=0x555632aada10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61eb7fe640 1 -- 172.21.15.98:0/1645740268 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f61e40effe0 con 0x7f61f4101420 2024-09-19T08:06:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fa824640 1 --2- 172.21.15.98:0/4269162415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632aad5d0 0x555632aada10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fa824640 1 -- 172.21.15.98:0/4269162415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55563299f3a0 con 0x555632aaeb40 2024-09-19T08:06:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.420+0000 7f61fa023640 1 --2- 172.21.15.98:0/4269162415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632aad5d0 0x555632aada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.421+0000 7f61fa824640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f61e00a33b0 0x7f61e00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.421+0000 7f61fa824640 1 --2- 172.21.15.98:0/4269162415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aaeb40 0x555632aad090 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00c370 tx=0x7f61ec00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ac7046754ede97 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.421+0000 7f61e8dea640 1 -- 172.21.15.98:0/4269162415 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec016070 con 0x555632aaeb40 2024-09-19T08:06:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.421+0000 7f61e8dea640 1 -- 172.21.15.98:0/4269162415 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61ec011d00 con 0x555632aaeb40 2024-09-19T08:06:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.421+0000 7f61e8dea640 1 -- 172.21.15.98:0/4269162415 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec00fba0 con 0x555632aaeb40 2024-09-19T08:06:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.421+0000 7f61fa824640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f61e00a33b0 0x7f61e00a5870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f61ec01d000 tx=0x7f61ec01e000 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.422+0000 7f61fe660740 1 -- 172.21.15.98:0/4269162415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aaeb40 msgr2=0x555632aad090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.422+0000 7f61fe660740 1 --2- 172.21.15.98:0/4269162415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aaeb40 0x555632aad090 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00c370 tx=0x7f61ec00c830 comp rx=0 tx=0).stop 2024-09-19T08:06:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.422+0000 7f61fe660740 1 -- 172.21.15.98:0/4269162415 shutdown_connections 2024-09-19T08:06:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.422+0000 7f61fe660740 1 --2- 172.21.15.98:0/4269162415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632ab8370 0x555632aba760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.422+0000 7f61fe660740 1 --2- 172.21.15.98:0/4269162415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632aad5d0 0x555632aada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.422+0000 7f61fe660740 1 --2- 172.21.15.98:0/4269162415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aaeb40 0x555632aad090 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.422+0000 7f61fe660740 1 -- 172.21.15.98:0/4269162415 >> 172.21.15.98:0/4269162415 conn(0x555632a76310 msgr2=0x555632aa9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.422+0000 7f61fe660740 1 -- 172.21.15.98:0/4269162415 shutdown_connections 2024-09-19T08:06:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.422+0000 7f61fe660740 1 -- 172.21.15.98:0/4269162415 wait complete. 2024-09-19T08:06:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.422+0000 7f61fe660740 1 -- start start 2024-09-19T08:06:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.422+0000 7f61fe660740 1 Processor -- start 2024-09-19T08:06:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.422+0000 7f61fe660740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632aad5d0 0x555632ab7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.422+0000 7f61fe660740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aaeb40 0x555632ab4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61fe660740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632ab8370 0x555632ab55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61fe660740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555632abb070 con 0x555632aaeb40 2024-09-19T08:06:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61fe660740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5556329ce060 con 0x555632ab8370 2024-09-19T08:06:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61fe660740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555632abb1f0 con 0x555632aad5d0 2024-09-19T08:06:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61fe660740 10 client.0 _ll_register_callbacks cb 0x5556327218c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61f9822640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aaeb40 0x555632ab4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61f9822640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aaeb40 0x555632ab4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40306/0 (socket says 172.21.15.98:40306) 2024-09-19T08:06:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61f9822640 1 -- 172.21.15.98:0/2146014779 learned_addr learned my addr 172.21.15.98:0/2146014779 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61fa023640 1 --2- 172.21.15.98:0/2146014779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632aad5d0 0x555632ab7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61fa824640 1 --2- 172.21.15.98:0/2146014779 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632ab8370 0x555632ab55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61f9822640 1 -- 172.21.15.98:0/2146014779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632aad5d0 msgr2=0x555632ab7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61f9822640 1 --2- 172.21.15.98:0/2146014779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632aad5d0 0x555632ab7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61f9822640 1 -- 172.21.15.98:0/2146014779 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632ab8370 msgr2=0x555632ab55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61f9822640 1 --2- 172.21.15.98:0/2146014779 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632ab8370 0x555632ab55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61f9822640 1 -- 172.21.15.98:0/2146014779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556327843c0 con 0x555632aaeb40 2024-09-19T08:06:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61f9822640 1 --2- 172.21.15.98:0/2146014779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aaeb40 0x555632ab4ff0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f61e40b8750 tx=0x7f61e402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c15eecff7b6ba5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61ddffb640 1 -- 172.21.15.98:0/2146014779 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e4049040 con 0x555632aaeb40 2024-09-19T08:06:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61ddffb640 1 -- 172.21.15.98:0/2146014779 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f61e40b8960 con 0x555632aaeb40 2024-09-19T08:06:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61fe660740 1 -- 172.21.15.98:0/2146014779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61ec011d00 con 0x555632aaeb40 2024-09-19T08:06:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61ddffb640 1 -- 172.21.15.98:0/2146014779 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f61e40bc9f0 con 0x555632aaeb40 2024-09-19T08:06:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61fe660740 10 client.16470 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.423+0000 7f61fe660740 1 -- 172.21.15.98:0/2146014779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555632ac6ae0 con 0x555632aaeb40 2024-09-19T08:06:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.424+0000 7f61ddffb640 1 -- 172.21.15.98:0/2146014779 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61ec011d00 con 0x555632aaeb40 2024-09-19T08:06:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.424+0000 7f61ddffb640 1 -- 172.21.15.98:0/2146014779 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f61e403b460 con 0x555632aaeb40 2024-09-19T08:06:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.424+0000 7f61fe660740 10 client.16470 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.424+0000 7f61fe660740 10 client.16470 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:06:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.424+0000 7f61fe660740 20 client.16470 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.424+0000 7f61fe660740 1 --2- 172.21.15.98:0/2146014779 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555632ac7e30 0x555632aca240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.424+0000 7f61fe660740 4 client.16470 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.424+0000 7f61fe660740 1 -- 172.21.15.98:0/2146014779 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555632a20860 con 0x555632ac7e30 2024-09-19T08:06:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.424+0000 7f61fa023640 1 --2- 172.21.15.98:0/2146014779 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555632ac7e30 0x555632aca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.425+0000 7f61fa023640 1 --2- 172.21.15.98:0/2146014779 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555632ac7e30 0x555632aca240 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45378634ebbb37eb server_cookie=bb122ad25f2ce56d in_seq=0 out_seq=0 2024-09-19T08:06:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.425+0000 7f61ddffb640 10 client.16470 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.425+0000 7f61ddffb640 1 -- 172.21.15.98:0/2146014779 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555632a20860 con 0x555632ac7e30 2024-09-19T08:06:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.425+0000 7f61ddffb640 10 client.16470 handle_command_reply: tid=0 2024-09-19T08:06:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.425+0000 7f61fe660740 1 client.16470 shutdown 2024-09-19T08:06:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.425+0000 7f61fe660740 1 -- 172.21.15.98:0/2146014779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aaeb40 msgr2=0x555632ab4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.425+0000 7f61fe660740 1 --2- 172.21.15.98:0/2146014779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aaeb40 0x555632ab4ff0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f61e40b8750 tx=0x7f61e402a420 comp rx=0 tx=0).stop 2024-09-19T08:06:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.425+0000 7f61fe660740 1 -- 172.21.15.98:0/2146014779 shutdown_connections 2024-09-19T08:06:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.425+0000 7f61fe660740 1 --2- 172.21.15.98:0/2146014779 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555632ac7e30 0x555632aca240 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.425+0000 7f61fe660740 1 --2- 172.21.15.98:0/2146014779 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632ab8370 0x555632ab55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.425+0000 7f61fe660740 1 --2- 172.21.15.98:0/2146014779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aaeb40 0x555632ab4ff0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.425+0000 7f61fe660740 1 --2- 172.21.15.98:0/2146014779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632aad5d0 0x555632ab7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.425+0000 7f61fe660740 1 -- 172.21.15.98:0/2146014779 >> 172.21.15.98:0/2146014779 conn(0x555632a76310 msgr2=0x555632aaf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.425+0000 7f61ddffb640 0 client.16470 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.426+0000 7f61fe660740 1 -- 172.21.15.98:0/2146014779 shutdown_connections 2024-09-19T08:06:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.426+0000 7f61fe660740 1 -- 172.21.15.98:0/2146014779 wait complete. 2024-09-19T08:06:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.426+0000 7f61fe660740 20 client.16470 trim_cache size 0 max 16384 2024-09-19T08:06:57.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.438+0000 7f61fe660740 1 Processor -- start 2024-09-19T08:06:57.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.438+0000 7f61fe660740 1 -- start start 2024-09-19T08:06:57.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.438+0000 7f61fe660740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aad5d0 0x555632ac4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.438+0000 7f61fe660740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632aaeb40 0x555632aacc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.438+0000 7f61fe660740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632ab8370 0x555632ab3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.438+0000 7f61fe660740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61f41afb10 con 0x555632aad5d0 2024-09-19T08:06:57.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.438+0000 7f61fe660740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f61f41b16b0 con 0x555632aaeb40 2024-09-19T08:06:57.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.438+0000 7f61fe660740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f61f41b01e0 con 0x555632ab8370 2024-09-19T08:06:57.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.439+0000 7f61fa023640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aad5d0 0x555632ac4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.439+0000 7f61fa023640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aad5d0 0x555632ac4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40322/0 (socket says 172.21.15.98:40322) 2024-09-19T08:06:57.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.439+0000 7f61fa023640 1 -- 172.21.15.98:0/402692263 learned_addr learned my addr 172.21.15.98:0/402692263 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:57.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.439+0000 7f61f9822640 1 --2- 172.21.15.98:0/402692263 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632aaeb40 0x555632aacc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.439+0000 7f61fa023640 1 -- 172.21.15.98:0/402692263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632ab8370 msgr2=0x555632ab3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:57.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.439+0000 7f61fa023640 1 --2- 172.21.15.98:0/402692263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632ab8370 0x555632ab3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.439+0000 7f61fa023640 1 -- 172.21.15.98:0/402692263 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632aaeb40 msgr2=0x555632aacc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.439+0000 7f61fa023640 1 --2- 172.21.15.98:0/402692263 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632aaeb40 0x555632aacc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.439+0000 7f61fa023640 1 -- 172.21.15.98:0/402692263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555632ab0120 con 0x555632aad5d0 2024-09-19T08:06:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.439+0000 7f61fa023640 1 --2- 172.21.15.98:0/402692263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aad5d0 0x555632ac4310 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f61f000ce90 tx=0x7f61f0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141d8b3c6eca8b46 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.439+0000 7f61dffff640 1 -- 172.21.15.98:0/402692263 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f0015880 con 0x555632aad5d0 2024-09-19T08:06:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.439+0000 7f61dffff640 1 -- 172.21.15.98:0/402692263 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61f0024430 con 0x555632aad5d0 2024-09-19T08:06:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.439+0000 7f61dffff640 1 -- 172.21.15.98:0/402692263 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f001b250 con 0x555632aad5d0 2024-09-19T08:06:57.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.440+0000 7f61fe660740 1 -- 172.21.15.98:0/402692263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aad5d0 msgr2=0x555632ac4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:57.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.440+0000 7f61fe660740 1 --2- 172.21.15.98:0/402692263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aad5d0 0x555632ac4310 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f61f000ce90 tx=0x7f61f0008040 comp rx=0 tx=0).stop 2024-09-19T08:06:57.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.440+0000 7f61fe660740 1 -- 172.21.15.98:0/402692263 shutdown_connections 2024-09-19T08:06:57.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.440+0000 7f61fe660740 1 --2- 172.21.15.98:0/402692263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632ab8370 0x555632ab3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.440+0000 7f61fe660740 1 --2- 172.21.15.98:0/402692263 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632aaeb40 0x555632aacc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.440+0000 7f61fe660740 1 --2- 172.21.15.98:0/402692263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aad5d0 0x555632ac4310 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.440+0000 7f61fe660740 1 -- 172.21.15.98:0/402692263 >> 172.21.15.98:0/402692263 conn(0x555632a76310 msgr2=0x555632ab38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.440+0000 7f61fe660740 1 -- 172.21.15.98:0/402692263 shutdown_connections 2024-09-19T08:06:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.440+0000 7f61fe660740 1 -- 172.21.15.98:0/402692263 wait complete. 2024-09-19T08:06:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.440+0000 7f61fe660740 1 -- start start 2024-09-19T08:06:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.440+0000 7f61fe660740 1 Processor -- start 2024-09-19T08:06:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fe660740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aad5d0 0x555632ac0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fe660740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632aaeb40 0x555632abc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fe660740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632ab8370 0x555632abc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fe660740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61f41b0fb0 con 0x555632aad5d0 2024-09-19T08:06:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fe660740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f61f41b08e0 con 0x555632ab8370 2024-09-19T08:06:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fe660740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f61f41adaf0 con 0x555632aaeb40 2024-09-19T08:06:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fe660740 10 client.0 _ll_register_callbacks cb 0x7f61e4004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fa023640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aad5d0 0x555632ac0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fa023640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aad5d0 0x555632ac0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40326/0 (socket says 172.21.15.98:40326) 2024-09-19T08:06:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fa023640 1 -- 172.21.15.98:0/3379084496 learned_addr learned my addr 172.21.15.98:0/3379084496 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fa824640 1 --2- 172.21.15.98:0/3379084496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632ab8370 0x555632abc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61f9822640 1 --2- 172.21.15.98:0/3379084496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632aaeb40 0x555632abc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fa023640 1 -- 172.21.15.98:0/3379084496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632aaeb40 msgr2=0x555632abc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fa023640 1 --2- 172.21.15.98:0/3379084496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632aaeb40 0x555632abc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fa023640 1 -- 172.21.15.98:0/3379084496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632ab8370 msgr2=0x555632abc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fa023640 1 --2- 172.21.15.98:0/3379084496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632ab8370 0x555632abc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fa023640 1 -- 172.21.15.98:0/3379084496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555632b31ae0 con 0x555632aad5d0 2024-09-19T08:06:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61f9822640 1 --2- 172.21.15.98:0/3379084496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632aaeb40 0x555632abc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:06:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fa023640 1 --2- 172.21.15.98:0/3379084496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aad5d0 0x555632ac0d70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f61f0008010 tx=0x7f61f00245e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=186e237f6c00dc11 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61ddffb640 1 -- 172.21.15.98:0/3379084496 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f002a070 con 0x555632aad5d0 2024-09-19T08:06:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61ddffb640 1 -- 172.21.15.98:0/3379084496 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f61f0015a80 con 0x555632aad5d0 2024-09-19T08:06:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.441+0000 7f61fe660740 1 -- 172.21.15.98:0/3379084496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61f0024430 con 0x555632aad5d0 2024-09-19T08:06:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.442+0000 7f61ddffb640 1 -- 172.21.15.98:0/3379084496 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f61f0015c80 con 0x555632aad5d0 2024-09-19T08:06:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.442+0000 7f61fe660740 10 client.16476 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.442+0000 7f61fe660740 1 -- 172.21.15.98:0/3379084496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555632b32c80 con 0x555632aad5d0 2024-09-19T08:06:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.442+0000 7f61ddffb640 1 -- 172.21.15.98:0/3379084496 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61f0024430 con 0x555632aad5d0 2024-09-19T08:06:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.442+0000 7f61ddffb640 1 -- 172.21.15.98:0/3379084496 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f61f002d360 con 0x555632aad5d0 2024-09-19T08:06:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.442+0000 7f61fe660740 10 client.16476 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.442+0000 7f61fe660740 10 client.16476 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:06:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.442+0000 7f61fe660740 20 client.16476 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.442+0000 7f61fe660740 1 --2- 172.21.15.98:0/3379084496 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555632b332f0 0x555632b35710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.442+0000 7f61fe660740 4 client.16476 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:06:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.442+0000 7f61fe660740 1 -- 172.21.15.98:0/3379084496 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555632a1ad70 con 0x555632b332f0 2024-09-19T08:06:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.442+0000 7f61f9822640 1 --2- 172.21.15.98:0/3379084496 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555632b332f0 0x555632b35710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.442+0000 7f61ddffb640 10 client.16476 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.442+0000 7f61f9822640 1 --2- 172.21.15.98:0/3379084496 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555632b332f0 0x555632b35710 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1af4f9bb269b8001 server_cookie=23ddfe89f76cd9fb in_seq=0 out_seq=0 2024-09-19T08:06:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61ddffb640 1 -- 172.21.15.98:0/3379084496 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555632a1ad70 con 0x555632b332f0 2024-09-19T08:06:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61ddffb640 10 client.16476 handle_command_reply: tid=0 2024-09-19T08:06:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61fe660740 1 client.16476 shutdown 2024-09-19T08:06:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61fe660740 1 -- 172.21.15.98:0/3379084496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aad5d0 msgr2=0x555632ac0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61fe660740 1 --2- 172.21.15.98:0/3379084496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aad5d0 0x555632ac0d70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f61f0008010 tx=0x7f61f00245e0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61fe660740 1 -- 172.21.15.98:0/3379084496 shutdown_connections 2024-09-19T08:06:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61fe660740 1 --2- 172.21.15.98:0/3379084496 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x555632b332f0 0x555632b35710 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61fe660740 1 --2- 172.21.15.98:0/3379084496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555632ab8370 0x555632abc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61fe660740 1 --2- 172.21.15.98:0/3379084496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555632aaeb40 0x555632abc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61ddffb640 0 client.16476 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61fe660740 1 --2- 172.21.15.98:0/3379084496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555632aad5d0 0x555632ac0d70 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61fe660740 1 -- 172.21.15.98:0/3379084496 >> 172.21.15.98:0/3379084496 conn(0x555632a76310 msgr2=0x555632ab40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61fe660740 1 -- 172.21.15.98:0/3379084496 shutdown_connections 2024-09-19T08:06:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61fe660740 1 -- 172.21.15.98:0/3379084496 wait complete. 2024-09-19T08:06:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.443+0000 7f61fe660740 20 client.16476 trim_cache size 0 max 16384 2024-09-19T08:06:57.451 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:06:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.444+0000 7f61fc287640 1 -- 172.21.15.98:0/1645740268 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f61e00a33b0 msgr2=0x7f61e00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.444+0000 7f61fc287640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f61e00a33b0 0x7f61e00a5870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f61ec01d000 tx=0x7f61ec01e000 comp rx=0 tx=0).stop 2024-09-19T08:06:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.444+0000 7f61fc287640 1 -- 172.21.15.98:0/1645740268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f4101420 msgr2=0x7f61f41a09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.444+0000 7f61fc287640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f4101420 0x7f61f41a09a0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f61e4009a50 tx=0x7f61e4004270 comp rx=0 tx=0).stop 2024-09-19T08:06:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.444+0000 7f61fc287640 1 -- 172.21.15.98:0/1645740268 shutdown_connections 2024-09-19T08:06:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.444+0000 7f61fc287640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f61e00a33b0 0x7f61e00a5870 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.444+0000 7f61fc287640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61f4106a80 0x7f61f419a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.444+0000 7f61fc287640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f4101420 0x7f61f41a09a0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.444+0000 7f61fc287640 1 --2- 172.21.15.98:0/1645740268 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61f4100a80 0x7f61f41a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.444+0000 7f61fc287640 1 -- 172.21.15.98:0/1645740268 >> 172.21.15.98:0/1645740268 conn(0x7f61f40fc7a0 msgr2=0x7f61f406a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.444+0000 7f61fc287640 1 -- 172.21.15.98:0/1645740268 shutdown_connections 2024-09-19T08:06:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:57.444+0000 7f61fc287640 1 -- 172.21.15.98:0/1645740268 wait complete. 2024-09-19T08:06:58.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:57 smithi139 ceph-mon[34874]: pgmap v121: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-19T08:06:58.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:57 smithi098 ceph-mon[31333]: pgmap v121: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-19T08:06:58.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:57 smithi162 ceph-mon[35260]: pgmap v121: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-19T08:06:59.251 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:06:59.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:06:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:06:59.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.969+0000 7f78eb015640 1 -- 172.21.15.98:0/3143023330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4102df0 msgr2=0x7f78e41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:59.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.969+0000 7f78eb015640 1 --2- 172.21.15.98:0/3143023330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4102df0 0x7f78e41031d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f78dc009900 tx=0x7f78dc02a410 comp rx=0 tx=0).stop 2024-09-19T08:06:59.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.969+0000 7f78eb015640 1 -- 172.21.15.98:0/3143023330 shutdown_connections 2024-09-19T08:06:59.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.969+0000 7f78eb015640 1 --2- 172.21.15.98:0/3143023330 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78e410c2f0 0x7f78e410e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.969+0000 7f78eb015640 1 --2- 172.21.15.98:0/3143023330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78e41037a0 0x7f78e410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.969+0000 7f78eb015640 1 --2- 172.21.15.98:0/3143023330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4102df0 0x7f78e41031d0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.969+0000 7f78eb015640 1 -- 172.21.15.98:0/3143023330 >> 172.21.15.98:0/3143023330 conn(0x7f78e40fc7d0 msgr2=0x7f78e40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.969+0000 7f78eb015640 1 -- 172.21.15.98:0/3143023330 shutdown_connections 2024-09-19T08:06:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.969+0000 7f78eb015640 1 -- 172.21.15.98:0/3143023330 wait complete. 2024-09-19T08:06:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.969+0000 7f78eb015640 1 Processor -- start 2024-09-19T08:06:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.969+0000 7f78eb015640 1 -- start start 2024-09-19T08:06:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.969+0000 7f78eb015640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4102df0 0x7f78e4100a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.969+0000 7f78eb015640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78e41037a0 0x7f78e4100f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.969+0000 7f78eb015640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78e410c2f0 0x7f78e41014d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78eb015640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78e4112150 con 0x7f78e4102df0 2024-09-19T08:06:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78eb015640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78e4111fd0 con 0x7f78e410c2f0 2024-09-19T08:06:59.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78eb015640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78e41122d0 con 0x7f78e41037a0 2024-09-19T08:06:59.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78e8db1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4102df0 0x7f78e4100a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:59.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78e8db1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4102df0 0x7f78e4100a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40332/0 (socket says 172.21.15.98:40332) 2024-09-19T08:06:59.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78e8db1640 1 -- 172.21.15.98:0/1687185722 learned_addr learned my addr 172.21.15.98:0/1687185722 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:59.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78e3fff640 1 --2- 172.21.15.98:0/1687185722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78e41037a0 0x7f78e4100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:59.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78e95b2640 1 --2- 172.21.15.98:0/1687185722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78e410c2f0 0x7f78e41014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:59.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78e8db1640 1 -- 172.21.15.98:0/1687185722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78e41037a0 msgr2=0x7f78e4100f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:59.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78e8db1640 1 --2- 172.21.15.98:0/1687185722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78e41037a0 0x7f78e4100f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78e8db1640 1 -- 172.21.15.98:0/1687185722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78e410c2f0 msgr2=0x7f78e41014d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:59.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78e8db1640 1 --2- 172.21.15.98:0/1687185722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78e410c2f0 0x7f78e41014d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78e8db1640 1 -- 172.21.15.98:0/1687185722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e41a6c00 con 0x7f78e4102df0 2024-09-19T08:06:59.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78e8db1640 1 --2- 172.21.15.98:0/1687185722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4102df0 0x7f78e4100a50 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f78dc005ab0 tx=0x7f78dc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:59.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78e1ffb640 1 -- 172.21.15.98:0/1687185722 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc03f030 con 0x7f78e4102df0 2024-09-19T08:06:59.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78e1ffb640 1 -- 172.21.15.98:0/1687185722 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78dc003ce0 con 0x7f78e4102df0 2024-09-19T08:06:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78eb015640 1 -- 172.21.15.98:0/1687185722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78e41a6e90 con 0x7f78e4102df0 2024-09-19T08:06:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.970+0000 7f78e1ffb640 1 -- 172.21.15.98:0/1687185722 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc035070 con 0x7f78e4102df0 2024-09-19T08:06:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.971+0000 7f78eb015640 1 -- 172.21.15.98:0/1687185722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78e41a72e0 con 0x7f78e4102df0 2024-09-19T08:06:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.973+0000 7f78e1ffb640 1 -- 172.21.15.98:0/1687185722 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f78dc042030 con 0x7f78e4102df0 2024-09-19T08:06:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.973+0000 7f78ed3ee740 1 Processor -- start 2024-09-19T08:06:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78ed3ee740 1 -- start start 2024-09-19T08:06:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78ed3ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 0x555608485f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78ed3ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 0x555608490800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78ed3ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608490d40 0x555608493130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78ed3ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5556083eb280 con 0x555608490d40 2024-09-19T08:06:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78ed3ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555608392b10 con 0x555608485b60 2024-09-19T08:06:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78ed3ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5556083e7240 con 0x555608486480 2024-09-19T08:06:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78e3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608490d40 0x555608493130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78e3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608490d40 0x555608493130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40334/0 (socket says 172.21.15.98:40334) 2024-09-19T08:06:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78e3fff640 1 -- 172.21.15.98:0/1855011683 learned_addr learned my addr 172.21.15.98:0/1855011683 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78e95b2640 1 --2- 172.21.15.98:0/1855011683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 0x555608485f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78e8db1640 1 --2- 172.21.15.98:0/1855011683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 0x555608490800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78e1ffb640 1 --2- 172.21.15.98:0/1687185722 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f78d00a33b0 0x7f78d00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78e95b2640 1 --2- 172.21.15.98:0/1687185722 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f78d00a33b0 0x7f78d00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78e3fff640 1 -- 172.21.15.98:0/1855011683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 msgr2=0x555608490800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78e3fff640 1 --2- 172.21.15.98:0/1855011683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 0x555608490800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78e3fff640 1 -- 172.21.15.98:0/1855011683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 msgr2=0x555608485f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78e3fff640 1 --2- 172.21.15.98:0/1855011683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 0x555608485f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78e3fff640 1 -- 172.21.15.98:0/1855011683 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55560839f1f0 con 0x555608490d40 2024-09-19T08:06:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.974+0000 7f78e1ffb640 1 -- 172.21.15.98:0/1687185722 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f78dc0f06b0 con 0x7f78e4102df0 2024-09-19T08:06:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.975+0000 7f78e3fff640 1 --2- 172.21.15.98:0/1855011683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608490d40 0x555608493130 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f78d4007d80 tx=0x7f78d400c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40552fd23042991 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.975+0000 7f78e95b2640 1 --2- 172.21.15.98:0/1687185722 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f78d00a33b0 0x7f78d00a5870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f78cc00bfc0 tx=0x7f78cc004730 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.975+0000 7f78daffd640 1 -- 172.21.15.98:0/1855011683 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d4015020 con 0x555608490d40 2024-09-19T08:06:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.975+0000 7f78daffd640 1 -- 172.21.15.98:0/1855011683 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78d4011b10 con 0x555608490d40 2024-09-19T08:06:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.975+0000 7f78daffd640 1 -- 172.21.15.98:0/1855011683 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d40109d0 con 0x555608490d40 2024-09-19T08:06:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.975+0000 7f78ed3ee740 1 -- 172.21.15.98:0/1855011683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608490d40 msgr2=0x555608493130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.975+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/1855011683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608490d40 0x555608493130 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f78d4007d80 tx=0x7f78d400c560 comp rx=0 tx=0).stop 2024-09-19T08:06:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.975+0000 7f78ed3ee740 1 -- 172.21.15.98:0/1855011683 shutdown_connections 2024-09-19T08:06:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.975+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/1855011683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608490d40 0x555608493130 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.976+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/1855011683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 0x555608490800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.976+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/1855011683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 0x555608485f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.976+0000 7f78ed3ee740 1 -- 172.21.15.98:0/1855011683 >> 172.21.15.98:0/1855011683 conn(0x5556084820f0 msgr2=0x5556084824d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.976+0000 7f78ed3ee740 1 -- 172.21.15.98:0/1855011683 shutdown_connections 2024-09-19T08:06:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.976+0000 7f78ed3ee740 1 -- 172.21.15.98:0/1855011683 wait complete. 2024-09-19T08:06:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.976+0000 7f78ed3ee740 1 -- start start 2024-09-19T08:06:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.976+0000 7f78ed3ee740 1 Processor -- start 2024-09-19T08:06:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.976+0000 7f78ed3ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 0x5556084907e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.976+0000 7f78ed3ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608486480 0x55560848db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.976+0000 7f78ed3ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608490d40 0x55560848e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.976+0000 7f78ed3ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555608493910 con 0x555608486480 2024-09-19T08:06:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.976+0000 7f78ed3ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5556083a5f10 con 0x555608485b60 2024-09-19T08:06:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.976+0000 7f78ed3ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555608493a90 con 0x555608490d40 2024-09-19T08:06:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78ed3ee740 10 client.0 _ll_register_callbacks cb 0x55560836ea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:06:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78e3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608486480 0x55560848db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78e3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608486480 0x55560848db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40344/0 (socket says 172.21.15.98:40344) 2024-09-19T08:06:59.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78e3fff640 1 -- 172.21.15.98:0/4074968761 learned_addr learned my addr 172.21.15.98:0/4074968761 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:06:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78e95b2640 1 --2- 172.21.15.98:0/4074968761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608490d40 0x55560848e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78e3fff640 1 -- 172.21.15.98:0/4074968761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608490d40 msgr2=0x55560848e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78e8db1640 1 --2- 172.21.15.98:0/4074968761 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 0x5556084907e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78e3fff640 1 --2- 172.21.15.98:0/4074968761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608490d40 0x55560848e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78e3fff640 1 -- 172.21.15.98:0/4074968761 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 msgr2=0x5556084907e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:06:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78e3fff640 1 --2- 172.21.15.98:0/4074968761 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 0x5556084907e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78e3fff640 1 -- 172.21.15.98:0/4074968761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55560815c6a0 con 0x555608486480 2024-09-19T08:06:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78e8db1640 1 --2- 172.21.15.98:0/4074968761 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 0x5556084907e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:06:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78e3fff640 1 --2- 172.21.15.98:0/4074968761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608486480 0x55560848db30 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f78d4004a80 tx=0x7f78d4011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec7a5868f6b55d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:06:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:06:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78c3fff640 1 -- 172.21.15.98:0/4074968761 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d4015040 con 0x555608486480 2024-09-19T08:06:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78c3fff640 1 -- 172.21.15.98:0/4074968761 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f78d4010770 con 0x555608486480 2024-09-19T08:06:59.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78ed3ee740 1 -- 172.21.15.98:0/4074968761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78d4011b10 con 0x555608486480 2024-09-19T08:06:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78c3fff640 1 -- 172.21.15.98:0/4074968761 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f78d40109b0 con 0x555608486480 2024-09-19T08:06:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78ed3ee740 10 client.16488 fetch_fsmap learned FSMap version 30 2024-09-19T08:06:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.977+0000 7f78ed3ee740 1 -- 172.21.15.98:0/4074968761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55560849f420 con 0x555608486480 2024-09-19T08:06:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.978+0000 7f78c3fff640 1 -- 172.21.15.98:0/4074968761 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78d4011b10 con 0x555608486480 2024-09-19T08:06:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.978+0000 7f78c3fff640 1 -- 172.21.15.98:0/4074968761 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f78d401aa20 con 0x555608486480 2024-09-19T08:06:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.978+0000 7f78ed3ee740 10 client.16488 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:06:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.978+0000 7f78ed3ee740 10 client.16488 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:06:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.978+0000 7f78ed3ee740 20 client.16488 populate_metadata read hostname 'smithi098' 2024-09-19T08:06:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.978+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/4074968761 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5556084a0770 0x5556084a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.978+0000 7f78ed3ee740 4 client.16488 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:06:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.978+0000 7f78ed3ee740 1 -- 172.21.15.98:0/4074968761 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556083f87f0 con 0x5556084a0770 2024-09-19T08:06:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.978+0000 7f78e8db1640 1 --2- 172.21.15.98:0/4074968761 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5556084a0770 0x5556084a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:06:59.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.978+0000 7f78e8db1640 1 --2- 172.21.15.98:0/4074968761 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5556084a0770 0x5556084a2ba0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f439025044ada96b server_cookie=3ea91e8aa784b8c1 in_seq=0 out_seq=0 2024-09-19T08:06:59.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.978+0000 7f78c3fff640 10 client.16488 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:59.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.979+0000 7f78c3fff640 1 -- 172.21.15.98:0/4074968761 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5556083f87f0 con 0x5556084a0770 2024-09-19T08:06:59.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.979+0000 7f78c3fff640 10 client.16488 handle_command_reply: tid=0 2024-09-19T08:06:59.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.979+0000 7f78ed3ee740 1 client.16488 shutdown 2024-09-19T08:06:59.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.979+0000 7f78ed3ee740 1 -- 172.21.15.98:0/4074968761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608486480 msgr2=0x55560848db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:06:59.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.979+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/4074968761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608486480 0x55560848db30 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f78d4004a80 tx=0x7f78d4011a90 comp rx=0 tx=0).stop 2024-09-19T08:06:59.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.979+0000 7f78ed3ee740 1 -- 172.21.15.98:0/4074968761 shutdown_connections 2024-09-19T08:06:59.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.979+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/4074968761 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5556084a0770 0x5556084a2ba0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.979+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/4074968761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608490d40 0x55560848e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.979+0000 7f78c3fff640 0 client.16488 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:06:59.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.979+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/4074968761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608486480 0x55560848db30 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.979+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/4074968761 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 0x5556084907e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:06:59.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.979+0000 7f78ed3ee740 1 -- 172.21.15.98:0/4074968761 >> 172.21.15.98:0/4074968761 conn(0x5556084820f0 msgr2=0x555608487dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:06:59.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.980+0000 7f78ed3ee740 1 -- 172.21.15.98:0/4074968761 shutdown_connections 2024-09-19T08:06:59.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.980+0000 7f78ed3ee740 1 -- 172.21.15.98:0/4074968761 wait complete. 2024-09-19T08:06:59.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.980+0000 7f78ed3ee740 20 client.16488 trim_cache size 0 max 16384 2024-09-19T08:06:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.997+0000 7f78ed3ee740 1 Processor -- start 2024-09-19T08:06:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.997+0000 7f78ed3ee740 1 -- start start 2024-09-19T08:06:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.997+0000 7f78ed3ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 0x55560848cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:06:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.997+0000 7f78ed3ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 0x5556084a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.997+0000 7f78ed3ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608490d40 0x5556084a89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.997+0000 7f78ed3ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78e41afa30 con 0x555608490d40 2024-09-19T08:07:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.997+0000 7f78ed3ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78e41b15d0 con 0x555608485b60 2024-09-19T08:07:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.997+0000 7f78ed3ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78e41b0100 con 0x555608486480 2024-09-19T08:07:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78e95b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608490d40 0x5556084a89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78e95b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608490d40 0x5556084a89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40350/0 (socket says 172.21.15.98:40350) 2024-09-19T08:07:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78e95b2640 1 -- 172.21.15.98:0/646098578 learned_addr learned my addr 172.21.15.98:0/646098578 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78e3fff640 1 --2- 172.21.15.98:0/646098578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 0x5556084a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78e8db1640 1 --2- 172.21.15.98:0/646098578 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 0x55560848cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78e95b2640 1 -- 172.21.15.98:0/646098578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 msgr2=0x5556084a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78e95b2640 1 --2- 172.21.15.98:0/646098578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 0x5556084a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78e95b2640 1 -- 172.21.15.98:0/646098578 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 msgr2=0x55560848cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78e95b2640 1 --2- 172.21.15.98:0/646098578 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 0x55560848cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78e95b2640 1 -- 172.21.15.98:0/646098578 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556084862b0 con 0x555608490d40 2024-09-19T08:07:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78e8db1640 1 --2- 172.21.15.98:0/646098578 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 0x55560848cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78e95b2640 1 --2- 172.21.15.98:0/646098578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608490d40 0x5556084a89f0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f78cc011730 tx=0x7f78cc00b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6a97fd6e9a7ac7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78da7fc640 1 -- 172.21.15.98:0/646098578 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc023030 con 0x555608490d40 2024-09-19T08:07:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78da7fc640 1 -- 172.21.15.98:0/646098578 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78cc018040 con 0x555608490d40 2024-09-19T08:07:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.998+0000 7f78da7fc640 1 -- 172.21.15.98:0/646098578 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc014e60 con 0x555608490d40 2024-09-19T08:07:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.999+0000 7f78ed3ee740 1 -- 172.21.15.98:0/646098578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608490d40 msgr2=0x5556084a89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.999+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/646098578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608490d40 0x5556084a89f0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f78cc011730 tx=0x7f78cc00b340 comp rx=0 tx=0).stop 2024-09-19T08:07:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.999+0000 7f78ed3ee740 1 -- 172.21.15.98:0/646098578 shutdown_connections 2024-09-19T08:07:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.999+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/646098578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608490d40 0x5556084a89f0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.999+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/646098578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 0x5556084a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.999+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/646098578 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608485b60 0x55560848cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.999+0000 7f78ed3ee740 1 -- 172.21.15.98:0/646098578 >> 172.21.15.98:0/646098578 conn(0x5556084820f0 msgr2=0x55560848c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.999+0000 7f78ed3ee740 1 -- 172.21.15.98:0/646098578 shutdown_connections 2024-09-19T08:07:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.999+0000 7f78ed3ee740 1 -- 172.21.15.98:0/646098578 wait complete. 2024-09-19T08:07:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:06:59.999+0000 7f78ed3ee740 1 -- start start 2024-09-19T08:07:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78ed3ee740 1 Processor -- start 2024-09-19T08:07:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78ed3ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608485b60 0x555608494b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78ed3ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 0x555608495040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78ed3ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608490d40 0x555608495580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78ed3ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78e41b0ed0 con 0x555608485b60 2024-09-19T08:07:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78ed3ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78e41b0800 con 0x555608490d40 2024-09-19T08:07:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78ed3ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78e41ada10 con 0x555608486480 2024-09-19T08:07:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78ed3ee740 10 client.0 _ll_register_callbacks cb 0x7f78cc005f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78e8db1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608485b60 0x555608494b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78e8db1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608485b60 0x555608494b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40360/0 (socket says 172.21.15.98:40360) 2024-09-19T08:07:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78e8db1640 1 -- 172.21.15.98:0/1477967324 learned_addr learned my addr 172.21.15.98:0/1477967324 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78e3fff640 1 --2- 172.21.15.98:0/1477967324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 0x555608495040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78e95b2640 1 --2- 172.21.15.98:0/1477967324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608490d40 0x555608495580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78e8db1640 1 -- 172.21.15.98:0/1477967324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 msgr2=0x555608495040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78e8db1640 1 --2- 172.21.15.98:0/1477967324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 0x555608495040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78e8db1640 1 -- 172.21.15.98:0/1477967324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608490d40 msgr2=0x555608495580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78e8db1640 1 --2- 172.21.15.98:0/1477967324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608490d40 0x555608495580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78e8db1640 1 -- 172.21.15.98:0/1477967324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55560850a500 con 0x555608485b60 2024-09-19T08:07:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.000+0000 7f78e95b2640 1 --2- 172.21.15.98:0/1477967324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608490d40 0x555608495580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78e8db1640 1 --2- 172.21.15.98:0/1477967324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608485b60 0x555608494b00 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0bc950 tx=0x7f78dc03b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44049bcb850f77b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78c3fff640 1 -- 172.21.15.98:0/1477967324 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc049040 con 0x555608485b60 2024-09-19T08:07:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78c3fff640 1 -- 172.21.15.98:0/1477967324 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f78dc03bd70 con 0x555608485b60 2024-09-19T08:07:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78ed3ee740 1 -- 172.21.15.98:0/1477967324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78cc018040 con 0x555608485b60 2024-09-19T08:07:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78c3fff640 1 -- 172.21.15.98:0/1477967324 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f78dc0044e0 con 0x555608485b60 2024-09-19T08:07:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78ed3ee740 10 client.16494 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78ed3ee740 1 -- 172.21.15.98:0/1477967324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55560850b790 con 0x555608485b60 2024-09-19T08:07:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78c3fff640 1 -- 172.21.15.98:0/1477967324 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78cc018040 con 0x555608485b60 2024-09-19T08:07:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78c3fff640 1 -- 172.21.15.98:0/1477967324 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f78dc0ed2f0 con 0x555608485b60 2024-09-19T08:07:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78ed3ee740 10 client.16494 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78ed3ee740 10 client.16494 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78ed3ee740 20 client.16494 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/1477967324 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55560850beb0 0x55560850e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78ed3ee740 4 client.16494 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:07:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.001+0000 7f78ed3ee740 1 -- 172.21.15.98:0/1477967324 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5556083f2d00 con 0x55560850beb0 2024-09-19T08:07:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.002+0000 7f78e3fff640 1 --2- 172.21.15.98:0/1477967324 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55560850beb0 0x55560850e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.002+0000 7f78e3fff640 1 --2- 172.21.15.98:0/1477967324 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55560850beb0 0x55560850e2d0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7001aac73b1cadc server_cookie=834db5f83896d3ea in_seq=0 out_seq=0 2024-09-19T08:07:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.002+0000 7f78c3fff640 10 client.16494 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.003+0000 7f78c3fff640 1 -- 172.21.15.98:0/1477967324 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5556083f2d00 con 0x55560850beb0 2024-09-19T08:07:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.003+0000 7f78c3fff640 10 client.16494 handle_command_reply: tid=0 2024-09-19T08:07:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.003+0000 7f78ed3ee740 1 client.16494 shutdown 2024-09-19T08:07:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.003+0000 7f78ed3ee740 1 -- 172.21.15.98:0/1477967324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608485b60 msgr2=0x555608494b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.003+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/1477967324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608485b60 0x555608494b00 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0bc950 tx=0x7f78dc03b800 comp rx=0 tx=0).stop 2024-09-19T08:07:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.003+0000 7f78ed3ee740 1 -- 172.21.15.98:0/1477967324 shutdown_connections 2024-09-19T08:07:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.003+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/1477967324 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55560850beb0 0x55560850e2d0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.003+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/1477967324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555608490d40 0x555608495580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.003+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/1477967324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555608486480 0x555608495040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.003+0000 7f78ed3ee740 1 --2- 172.21.15.98:0/1477967324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555608485b60 0x555608494b00 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.003+0000 7f78ed3ee740 1 -- 172.21.15.98:0/1477967324 >> 172.21.15.98:0/1477967324 conn(0x5556084820f0 msgr2=0x55560848cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.003+0000 7f78c3fff640 0 client.16494 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.003+0000 7f78ed3ee740 1 -- 172.21.15.98:0/1477967324 shutdown_connections 2024-09-19T08:07:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.003+0000 7f78ed3ee740 1 -- 172.21.15.98:0/1477967324 wait complete. 2024-09-19T08:07:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.004+0000 7f78ed3ee740 20 client.16494 trim_cache size 0 max 16384 2024-09-19T08:07:00.009 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:07:00.010 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:07:00.010 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "d909e8c6-e120-45b5-b5dc-dc14050ec12e", 2024-09-19T08:07:00.010 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:07:00.010 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:07:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.004+0000 7f78eb015640 1 -- 172.21.15.98:0/1687185722 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f78d00a33b0 msgr2=0x7f78d00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.004+0000 7f78eb015640 1 --2- 172.21.15.98:0/1687185722 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f78d00a33b0 0x7f78d00a5870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f78cc00bfc0 tx=0x7f78cc004730 comp rx=0 tx=0).stop 2024-09-19T08:07:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.004+0000 7f78eb015640 1 -- 172.21.15.98:0/1687185722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4102df0 msgr2=0x7f78e4100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.004+0000 7f78eb015640 1 --2- 172.21.15.98:0/1687185722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4102df0 0x7f78e4100a50 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f78dc005ab0 tx=0x7f78dc0042f0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.004+0000 7f78eb015640 1 -- 172.21.15.98:0/1687185722 shutdown_connections 2024-09-19T08:07:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.004+0000 7f78eb015640 1 --2- 172.21.15.98:0/1687185722 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f78d00a33b0 0x7f78d00a5870 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.004+0000 7f78eb015640 1 --2- 172.21.15.98:0/1687185722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78e410c2f0 0x7f78e41014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.004+0000 7f78eb015640 1 --2- 172.21.15.98:0/1687185722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78e41037a0 0x7f78e4100f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.004+0000 7f78eb015640 1 --2- 172.21.15.98:0/1687185722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4102df0 0x7f78e4100a50 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.004+0000 7f78eb015640 1 -- 172.21.15.98:0/1687185722 >> 172.21.15.98:0/1687185722 conn(0x7f78e40fc7d0 msgr2=0x7f78e40fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.004+0000 7f78eb015640 1 -- 172.21.15.98:0/1687185722 shutdown_connections 2024-09-19T08:07:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:00.005+0000 7f78eb015640 1 -- 172.21.15.98:0/1687185722 wait complete. 2024-09-19T08:07:00.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:06:59 smithi139 ceph-mon[34874]: pgmap v122: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:07:00.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:06:59 smithi098 ceph-mon[31333]: pgmap v122: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:07:00.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:06:59 smithi162 ceph-mon[35260]: pgmap v122: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:07:00.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:07:01.013 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:01.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:00 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:07:01.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:00 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:01.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:00 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:07:01.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:00 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:01.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:00 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:07:01.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:00 smithi139 ceph-mon[34874]: pgmap v123: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:01.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:00 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:07:01.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:00 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:01.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:00 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:07:01.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:00 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:01.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:00 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:07:01.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:00 smithi098 ceph-mon[31333]: pgmap v123: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:01.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:00 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:07:01.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:00 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:01.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:00 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:07:01.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:00 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:01.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:00 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:07:01.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:00 smithi162 ceph-mon[35260]: pgmap v123: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:01.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.457+0000 7f7e7c722640 1 -- 172.21.15.98:0/1638450399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e74100a80 msgr2=0x7f7e74100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:01.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.457+0000 7f7e7c722640 1 --2- 172.21.15.98:0/1638450399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e74100a80 0x7f7e74100ee0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009930 tx=0x7f7e6402a420 comp rx=0 tx=0).stop 2024-09-19T08:07:01.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.457+0000 7f7e7c722640 1 -- 172.21.15.98:0/1638450399 shutdown_connections 2024-09-19T08:07:01.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.457+0000 7f7e7c722640 1 --2- 172.21.15.98:0/1638450399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e74101420 0x7f7e7410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.457+0000 7f7e7c722640 1 --2- 172.21.15.98:0/1638450399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e74100a80 0x7f7e74100ee0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.457+0000 7f7e7c722640 1 --2- 172.21.15.98:0/1638450399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e74106a80 0x7f7e74106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.457+0000 7f7e7c722640 1 -- 172.21.15.98:0/1638450399 >> 172.21.15.98:0/1638450399 conn(0x7f7e740fc7a0 msgr2=0x7f7e740febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:01.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.457+0000 7f7e7c722640 1 -- 172.21.15.98:0/1638450399 shutdown_connections 2024-09-19T08:07:01.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.457+0000 7f7e7c722640 1 -- 172.21.15.98:0/1638450399 wait complete. 2024-09-19T08:07:01.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e7c722640 1 Processor -- start 2024-09-19T08:07:01.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e7c722640 1 -- start start 2024-09-19T08:07:01.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e7c722640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e74100a80 0x7f7e74199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e7c722640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e74101420 0x7f7e74199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e7c722640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e74106a80 0x7f7e741a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e7c722640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e74110e80 con 0x7f7e74101420 2024-09-19T08:07:01.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e7c722640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7e74110d00 con 0x7f7e74100a80 2024-09-19T08:07:01.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e7c722640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7e74111000 con 0x7f7e74106a80 2024-09-19T08:07:01.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e79cbd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e74101420 0x7f7e74199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e79cbd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e74101420 0x7f7e74199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40374/0 (socket says 172.21.15.98:40374) 2024-09-19T08:07:01.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e79cbd640 1 -- 172.21.15.98:0/2894246386 learned_addr learned my addr 172.21.15.98:0/2894246386 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:01.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e7acbf640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e74106a80 0x7f7e741a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e7a4be640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e74100a80 0x7f7e74199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e79cbd640 1 -- 172.21.15.98:0/2894246386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e74106a80 msgr2=0x7f7e741a4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:01.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e79cbd640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e74106a80 0x7f7e741a4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e79cbd640 1 -- 172.21.15.98:0/2894246386 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e74100a80 msgr2=0x7f7e74199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:01.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e79cbd640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e74100a80 0x7f7e74199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e79cbd640 1 -- 172.21.15.98:0/2894246386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e741a48d0 con 0x7f7e74101420 2024-09-19T08:07:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e7acbf640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e74106a80 0x7f7e741a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.458+0000 7f7e7a4be640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e74100a80 0x7f7e74199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.459+0000 7f7e79cbd640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e74101420 0x7f7e74199850 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009230 tx=0x7f7e64004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.459+0000 7f7e6b7fe640 1 -- 172.21.15.98:0/2894246386 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6403f030 con 0x7f7e74101420 2024-09-19T08:07:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.459+0000 7f7e7c722640 1 -- 172.21.15.98:0/2894246386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e741a4aa0 con 0x7f7e74101420 2024-09-19T08:07:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.459+0000 7f7e6b7fe640 1 -- 172.21.15.98:0/2894246386 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e64003ad0 con 0x7f7e74101420 2024-09-19T08:07:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.459+0000 7f7e6b7fe640 1 -- 172.21.15.98:0/2894246386 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e64035070 con 0x7f7e74101420 2024-09-19T08:07:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.459+0000 7f7e7c722640 1 -- 172.21.15.98:0/2894246386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e741a4f00 con 0x7f7e74101420 2024-09-19T08:07:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.461+0000 7f7e6b7fe640 1 -- 172.21.15.98:0/2894246386 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f7e64042030 con 0x7f7e74101420 2024-09-19T08:07:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.461+0000 7f7e7eafb740 1 Processor -- start 2024-09-19T08:07:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7eafb740 1 -- start start 2024-09-19T08:07:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7eafb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 0x560547c85760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7eafb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c85ca0 0x560547c94550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7eafb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c94a90 0x560547c96e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7eafb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560547beb2e0 con 0x560547c94a90 2024-09-19T08:07:01.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7eafb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560547b92ce0 con 0x560547c85380 2024-09-19T08:07:01.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7eafb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560547be72a0 con 0x560547c85ca0 2024-09-19T08:07:01.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e6b7fe640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7e600a33b0 0x7f7e600a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7acbf640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 0x560547c85760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e79cbd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c94a90 0x560547c96e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7acbf640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 0x560547c85760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:49826/0 (socket says 172.21.15.98:49826) 2024-09-19T08:07:01.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7acbf640 1 -- 172.21.15.98:0/1173833939 learned_addr learned my addr 172.21.15.98:0/1173833939 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:01.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7acbf640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7e600a33b0 0x7f7e600a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7a4be640 1 --2- 172.21.15.98:0/1173833939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c85ca0 0x560547c94550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e6b7fe640 1 -- 172.21.15.98:0/2894246386 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f7e640f0290 con 0x7f7e74101420 2024-09-19T08:07:01.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7acbf640 1 -- 172.21.15.98:0/1173833939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c85ca0 msgr2=0x560547c94550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:01.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7acbf640 1 --2- 172.21.15.98:0/1173833939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c85ca0 0x560547c94550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7acbf640 1 -- 172.21.15.98:0/1173833939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c94a90 msgr2=0x560547c96e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:01.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7acbf640 1 --2- 172.21.15.98:0/1173833939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c94a90 0x560547c96e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e7acbf640 1 -- 172.21.15.98:0/1173833939 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560547b77440 con 0x560547c85380 2024-09-19T08:07:01.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.462+0000 7f7e79cbd640 1 --2- 172.21.15.98:0/1173833939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c94a90 0x560547c96e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:07:01.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.463+0000 7f7e7acbf640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7e600a33b0 0x7f7e600a5870 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c012ca0 tx=0x7f7e6c009ad0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:01.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.463+0000 7f7e7acbf640 1 --2- 172.21.15.98:0/1173833939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 0x560547c85760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c012850 tx=0x7f7e6c01c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d8e112bbddce7e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:01.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.463+0000 7f7e68ff9640 1 -- 172.21.15.98:0/1173833939 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c024070 con 0x560547c85380 2024-09-19T08:07:01.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.463+0000 7f7e68ff9640 1 -- 172.21.15.98:0/1173833939 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e6c012cd0 con 0x560547c85380 2024-09-19T08:07:01.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.463+0000 7f7e68ff9640 1 -- 172.21.15.98:0/1173833939 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c005280 con 0x560547c85380 2024-09-19T08:07:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.463+0000 7f7e7eafb740 1 -- 172.21.15.98:0/1173833939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 msgr2=0x560547c85760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/1173833939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 0x560547c85760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c012850 tx=0x7f7e6c01c000 comp rx=0 tx=0).stop 2024-09-19T08:07:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 -- 172.21.15.98:0/1173833939 shutdown_connections 2024-09-19T08:07:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/1173833939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c94a90 0x560547c96e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/1173833939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c85ca0 0x560547c94550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/1173833939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 0x560547c85760 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 -- 172.21.15.98:0/1173833939 >> 172.21.15.98:0/1173833939 conn(0x560547c81a30 msgr2=0x560547c81e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 -- 172.21.15.98:0/1173833939 shutdown_connections 2024-09-19T08:07:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 -- 172.21.15.98:0/1173833939 wait complete. 2024-09-19T08:07:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 -- start start 2024-09-19T08:07:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 Processor -- start 2024-09-19T08:07:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 0x560547c94470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c85ca0 0x560547c91770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c94a90 0x560547c91dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560547c97660 con 0x560547c85ca0 2024-09-19T08:07:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560547ba6050 con 0x560547c85380 2024-09-19T08:07:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560547c977e0 con 0x560547c94a90 2024-09-19T08:07:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e7eafb740 10 client.0 _ll_register_callbacks cb 0x560547be6160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e79cbd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c85ca0 0x560547c91770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e79cbd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c85ca0 0x560547c91770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40392/0 (socket says 172.21.15.98:40392) 2024-09-19T08:07:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.464+0000 7f7e79cbd640 1 -- 172.21.15.98:0/3991508086 learned_addr learned my addr 172.21.15.98:0/3991508086 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e7a4be640 1 --2- 172.21.15.98:0/3991508086 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 0x560547c94470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e79cbd640 1 -- 172.21.15.98:0/3991508086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c94a90 msgr2=0x560547c91dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e79cbd640 1 --2- 172.21.15.98:0/3991508086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c94a90 0x560547c91dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e79cbd640 1 -- 172.21.15.98:0/3991508086 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 msgr2=0x560547c94470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:01.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e79cbd640 1 --2- 172.21.15.98:0/3991508086 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 0x560547c94470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e79cbd640 1 -- 172.21.15.98:0/3991508086 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56054795c3c0 con 0x560547c85ca0 2024-09-19T08:07:01.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e79cbd640 1 --2- 172.21.15.98:0/3991508086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c85ca0 0x560547c91770 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7e640eda70 tx=0x7f7e6402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=513ea19488cb0c5c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:01.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:01.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e5dffb640 1 -- 172.21.15.98:0/3991508086 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e64049040 con 0x560547c85ca0 2024-09-19T08:07:01.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e5dffb640 1 -- 172.21.15.98:0/3991508086 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f7e640edc90 con 0x560547c85ca0 2024-09-19T08:07:01.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e7eafb740 1 -- 172.21.15.98:0/3991508086 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e6c012cd0 con 0x560547c85ca0 2024-09-19T08:07:01.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e5dffb640 1 -- 172.21.15.98:0/3991508086 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7e640bcca0 con 0x560547c85ca0 2024-09-19T08:07:01.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e7eafb740 10 client.16506 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:01.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e7eafb740 1 -- 172.21.15.98:0/3991508086 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560547ca3180 con 0x560547c85ca0 2024-09-19T08:07:01.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e5dffb640 1 -- 172.21.15.98:0/3991508086 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e6c012cd0 con 0x560547c85ca0 2024-09-19T08:07:01.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.465+0000 7f7e5dffb640 1 -- 172.21.15.98:0/3991508086 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f7e64003c80 con 0x560547c85ca0 2024-09-19T08:07:01.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.466+0000 7f7e7eafb740 10 client.16506 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:01.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.466+0000 7f7e7eafb740 10 client.16506 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:01.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.466+0000 7f7e7eafb740 20 client.16506 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:01.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.466+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/3991508086 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560547ca45c0 0x560547ca6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.466+0000 7f7e7eafb740 4 client.16506 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:01.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.466+0000 7f7e7eafb740 1 -- 172.21.15.98:0/3991508086 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560547bf8850 con 0x560547ca45c0 2024-09-19T08:07:01.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.466+0000 7f7e7a4be640 1 --2- 172.21.15.98:0/3991508086 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560547ca45c0 0x560547ca6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.466+0000 7f7e5dffb640 10 client.16506 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:01.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.466+0000 7f7e7a4be640 1 --2- 172.21.15.98:0/3991508086 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560547ca45c0 0x560547ca6a30 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76293b71c1584667 server_cookie=8d236f116e166be0 in_seq=0 out_seq=0 2024-09-19T08:07:01.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e5dffb640 1 -- 172.21.15.98:0/3991508086 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560547bf8850 con 0x560547ca45c0 2024-09-19T08:07:01.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e5dffb640 10 client.16506 handle_command_reply: tid=0 2024-09-19T08:07:01.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e7eafb740 1 client.16506 shutdown 2024-09-19T08:07:01.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e7eafb740 1 -- 172.21.15.98:0/3991508086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c85ca0 msgr2=0x560547c91770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:01.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/3991508086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c85ca0 0x560547c91770 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7e640eda70 tx=0x7f7e6402a420 comp rx=0 tx=0).stop 2024-09-19T08:07:01.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e7eafb740 1 -- 172.21.15.98:0/3991508086 shutdown_connections 2024-09-19T08:07:01.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/3991508086 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560547ca45c0 0x560547ca6a30 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/3991508086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c94a90 0x560547c91dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/3991508086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c85ca0 0x560547c91770 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/3991508086 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 0x560547c94470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e5dffb640 0 client.16506 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:01.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e7eafb740 1 -- 172.21.15.98:0/3991508086 >> 172.21.15.98:0/3991508086 conn(0x560547c81a30 msgr2=0x560547c87790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:01.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e7eafb740 1 -- 172.21.15.98:0/3991508086 shutdown_connections 2024-09-19T08:07:01.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e7eafb740 1 -- 172.21.15.98:0/3991508086 wait complete. 2024-09-19T08:07:01.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.467+0000 7f7e7eafb740 20 client.16506 trim_cache size 0 max 16384 2024-09-19T08:07:01.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.484+0000 7f7e7eafb740 1 Processor -- start 2024-09-19T08:07:01.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.484+0000 7f7e7eafb740 1 -- start start 2024-09-19T08:07:01.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.484+0000 7f7e7eafb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c85380 0x560547ca0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.484+0000 7f7e7eafb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85ca0 0x560547cac340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.484+0000 7f7e7eafb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c94a90 0x560547cac880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.484+0000 7f7e7eafb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741ab5f0 con 0x560547c85380 2024-09-19T08:07:01.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.484+0000 7f7e7eafb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741ad190 con 0x560547c85ca0 2024-09-19T08:07:01.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.484+0000 7f7e7eafb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741abcc0 con 0x560547c94a90 2024-09-19T08:07:01.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.485+0000 7f7e7a4be640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c85380 0x560547ca0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.485+0000 7f7e7a4be640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c85380 0x560547ca0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40402/0 (socket says 172.21.15.98:40402) 2024-09-19T08:07:01.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.485+0000 7f7e7a4be640 1 -- 172.21.15.98:0/689917398 learned_addr learned my addr 172.21.15.98:0/689917398 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:01.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.485+0000 7f7e79cbd640 1 --2- 172.21.15.98:0/689917398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85ca0 0x560547cac340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.485+0000 7f7e7acbf640 1 --2- 172.21.15.98:0/689917398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c94a90 0x560547cac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.485+0000 7f7e7a4be640 1 -- 172.21.15.98:0/689917398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c94a90 msgr2=0x560547cac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.485+0000 7f7e7a4be640 1 --2- 172.21.15.98:0/689917398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c94a90 0x560547cac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.485+0000 7f7e7a4be640 1 -- 172.21.15.98:0/689917398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85ca0 msgr2=0x560547cac340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.485+0000 7f7e7a4be640 1 --2- 172.21.15.98:0/689917398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85ca0 0x560547cac340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.485+0000 7f7e7a4be640 1 -- 172.21.15.98:0/689917398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560547c92140 con 0x560547c85380 2024-09-19T08:07:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.485+0000 7f7e7a4be640 1 --2- 172.21.15.98:0/689917398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c85380 0x560547ca0950 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7e70005d80 tx=0x7f7e700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297200ea516caf73 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.485+0000 7f7e5ffff640 1 -- 172.21.15.98:0/689917398 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e700042f0 con 0x560547c85380 2024-09-19T08:07:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.485+0000 7f7e5ffff640 1 -- 172.21.15.98:0/689917398 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e70021070 con 0x560547c85380 2024-09-19T08:07:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.485+0000 7f7e5ffff640 1 -- 172.21.15.98:0/689917398 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7001c840 con 0x560547c85380 2024-09-19T08:07:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.486+0000 7f7e7eafb740 1 -- 172.21.15.98:0/689917398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c85380 msgr2=0x560547ca0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.486+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/689917398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c85380 0x560547ca0950 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7e70005d80 tx=0x7f7e700046c0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.486+0000 7f7e7eafb740 1 -- 172.21.15.98:0/689917398 shutdown_connections 2024-09-19T08:07:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.486+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/689917398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c94a90 0x560547cac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.486+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/689917398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85ca0 0x560547cac340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.486+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/689917398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c85380 0x560547ca0950 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.486+0000 7f7e7eafb740 1 -- 172.21.15.98:0/689917398 >> 172.21.15.98:0/689917398 conn(0x560547c81a30 msgr2=0x560547c8cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:01.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.486+0000 7f7e7eafb740 1 -- 172.21.15.98:0/689917398 shutdown_connections 2024-09-19T08:07:01.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.486+0000 7f7e7eafb740 1 -- 172.21.15.98:0/689917398 wait complete. 2024-09-19T08:07:01.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.486+0000 7f7e7eafb740 1 -- start start 2024-09-19T08:07:01.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7eafb740 1 Processor -- start 2024-09-19T08:07:01.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7eafb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 0x560547c8cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7eafb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c85ca0 0x560547c9cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7eafb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c94a90 0x560547c9d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7eafb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741aca90 con 0x560547c94a90 2024-09-19T08:07:01.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7eafb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741ac3c0 con 0x560547c85380 2024-09-19T08:07:01.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7eafb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741a95d0 con 0x560547c85ca0 2024-09-19T08:07:01.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7eafb740 10 client.0 _ll_register_callbacks cb 0x7f7e640b88a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:01.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7acbf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c94a90 0x560547c9d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7acbf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c94a90 0x560547c9d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40416/0 (socket says 172.21.15.98:40416) 2024-09-19T08:07:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7acbf640 1 -- 172.21.15.98:0/4253906958 learned_addr learned my addr 172.21.15.98:0/4253906958 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7a4be640 1 --2- 172.21.15.98:0/4253906958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 0x560547c8cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e79cbd640 1 --2- 172.21.15.98:0/4253906958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c85ca0 0x560547c9cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7acbf640 1 -- 172.21.15.98:0/4253906958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c85ca0 msgr2=0x560547c9cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7acbf640 1 --2- 172.21.15.98:0/4253906958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c85ca0 0x560547c9cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7acbf640 1 -- 172.21.15.98:0/4253906958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 msgr2=0x560547c8cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7acbf640 1 --2- 172.21.15.98:0/4253906958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 0x560547c8cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.487+0000 7f7e7acbf640 1 -- 172.21.15.98:0/4253906958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560547c8a100 con 0x560547c94a90 2024-09-19T08:07:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e7acbf640 1 --2- 172.21.15.98:0/4253906958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c94a90 0x560547c9d000 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c025000 tx=0x7f7e6c012f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c63d33a8a2338 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e5dffb640 1 -- 172.21.15.98:0/4253906958 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c024040 con 0x560547c94a90 2024-09-19T08:07:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e5dffb640 1 -- 172.21.15.98:0/4253906958 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f7e6c00dd50 con 0x560547c94a90 2024-09-19T08:07:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e7eafb740 1 -- 172.21.15.98:0/4253906958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e70021070 con 0x560547c94a90 2024-09-19T08:07:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e5dffb640 1 -- 172.21.15.98:0/4253906958 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7e6c0093c0 con 0x560547c94a90 2024-09-19T08:07:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e7eafb740 10 client.16512 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e7eafb740 1 -- 172.21.15.98:0/4253906958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560547d13cb0 con 0x560547c94a90 2024-09-19T08:07:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e5dffb640 1 -- 172.21.15.98:0/4253906958 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e70021070 con 0x560547c94a90 2024-09-19T08:07:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e5dffb640 1 -- 172.21.15.98:0/4253906958 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f7e6c00c030 con 0x560547c94a90 2024-09-19T08:07:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e7eafb740 10 client.16512 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e7eafb740 10 client.16512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e7eafb740 20 client.16512 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/4253906958 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560547d14040 0x560547d16460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e7eafb740 4 client.16512 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:07:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.488+0000 7f7e7eafb740 1 -- 172.21.15.98:0/4253906958 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560547bf2d60 con 0x560547d14040 2024-09-19T08:07:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.489+0000 7f7e7a4be640 1 --2- 172.21.15.98:0/4253906958 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560547d14040 0x560547d16460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.489+0000 7f7e5dffb640 10 client.16512 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.489+0000 7f7e7a4be640 1 --2- 172.21.15.98:0/4253906958 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560547d14040 0x560547d16460 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d86c647386bf925f server_cookie=a4535f5829e64d2f in_seq=0 out_seq=0 2024-09-19T08:07:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.489+0000 7f7e5dffb640 1 -- 172.21.15.98:0/4253906958 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560547bf2d60 con 0x560547d14040 2024-09-19T08:07:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.489+0000 7f7e5dffb640 10 client.16512 handle_command_reply: tid=0 2024-09-19T08:07:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.489+0000 7f7e7eafb740 1 client.16512 shutdown 2024-09-19T08:07:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.489+0000 7f7e7eafb740 1 -- 172.21.15.98:0/4253906958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c94a90 msgr2=0x560547c9d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.489+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/4253906958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c94a90 0x560547c9d000 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c025000 tx=0x7f7e6c012f00 comp rx=0 tx=0).stop 2024-09-19T08:07:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.489+0000 7f7e7eafb740 1 -- 172.21.15.98:0/4253906958 shutdown_connections 2024-09-19T08:07:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.489+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/4253906958 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560547d14040 0x560547d16460 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.489+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/4253906958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560547c94a90 0x560547c9d000 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.489+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/4253906958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560547c85ca0 0x560547c9cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7eafb740 1 --2- 172.21.15.98:0/4253906958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560547c85380 0x560547c8cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7eafb740 1 -- 172.21.15.98:0/4253906958 >> 172.21.15.98:0/4253906958 conn(0x560547c81a30 msgr2=0x560547c86ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:01.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e5dffb640 0 client.16512 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:01.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7eafb740 1 -- 172.21.15.98:0/4253906958 shutdown_connections 2024-09-19T08:07:01.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7eafb740 1 -- 172.21.15.98:0/4253906958 wait complete. 2024-09-19T08:07:01.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7eafb740 20 client.16512 trim_cache size 0 max 16384 2024-09-19T08:07:01.499 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:07:01.499 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:07:01.499 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:07:01.499 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:07:01.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7c722640 1 -- 172.21.15.98:0/2894246386 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7e600a33b0 msgr2=0x7f7e600a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:01.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7c722640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7e600a33b0 0x7f7e600a5870 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c012ca0 tx=0x7f7e6c009ad0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7c722640 1 -- 172.21.15.98:0/2894246386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e74101420 msgr2=0x7f7e74199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:01.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7c722640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e74101420 0x7f7e74199850 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009230 tx=0x7f7e64004270 comp rx=0 tx=0).stop 2024-09-19T08:07:01.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7c722640 1 -- 172.21.15.98:0/2894246386 shutdown_connections 2024-09-19T08:07:01.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7c722640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7e600a33b0 0x7f7e600a5870 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7c722640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e74106a80 0x7f7e741a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7c722640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e74101420 0x7f7e74199850 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7c722640 1 --2- 172.21.15.98:0/2894246386 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e74100a80 0x7f7e74199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7c722640 1 -- 172.21.15.98:0/2894246386 >> 172.21.15.98:0/2894246386 conn(0x7f7e740fc7a0 msgr2=0x7f7e740fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7c722640 1 -- 172.21.15.98:0/2894246386 shutdown_connections 2024-09-19T08:07:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:01.490+0000 7f7e7c722640 1 -- 172.21.15.98:0/2894246386 wait complete. 2024-09-19T08:07:02.251 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:07:02.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:07:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.006+0000 7fb06377d640 1 -- 172.21.15.98:0/3992283972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb05c1082a0 msgr2=0x7fb05c10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.006+0000 7fb06377d640 1 --2- 172.21.15.98:0/3992283972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb05c1082a0 0x7fb05c10ec40 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb04c009cd0 tx=0x7fb04c02a8e0 comp rx=0 tx=0).stop 2024-09-19T08:07:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.007+0000 7fb06377d640 1 -- 172.21.15.98:0/3992283972 shutdown_connections 2024-09-19T08:07:03.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.007+0000 7fb06377d640 1 --2- 172.21.15.98:0/3992283972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb05c1082a0 0x7fb05c10ec40 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:03.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.007+0000 7fb06377d640 1 --2- 172.21.15.98:0/3992283972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb05c1036e0 0x7fb05c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:03.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.007+0000 7fb06377d640 1 --2- 172.21.15.98:0/3992283972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb05c102d30 0x7fb05c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:03.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.007+0000 7fb06377d640 1 -- 172.21.15.98:0/3992283972 >> 172.21.15.98:0/3992283972 conn(0x7fb05c0fc7d0 msgr2=0x7fb05c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:03.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.007+0000 7fb06377d640 1 -- 172.21.15.98:0/3992283972 shutdown_connections 2024-09-19T08:07:03.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.007+0000 7fb06377d640 1 -- 172.21.15.98:0/3992283972 wait complete. 2024-09-19T08:07:03.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.007+0000 7fb06377d640 1 Processor -- start 2024-09-19T08:07:03.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.007+0000 7fb06377d640 1 -- start start 2024-09-19T08:07:03.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb06377d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb05c102d30 0x7fb05c1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:03.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb06377d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb05c1036e0 0x7fb05c198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:03.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb06377d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb05c1082a0 0x7fb05c19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb06377d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c1120a0 con 0x7fb05c102d30 2024-09-19T08:07:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb06377d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c111f20 con 0x7fb05c1036e0 2024-09-19T08:07:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb06377d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c112220 con 0x7fb05c1082a0 2024-09-19T08:07:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb061519640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb05c102d30 0x7fb05c1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb061519640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb05c102d30 0x7fb05c1984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40422/0 (socket says 172.21.15.98:40422) 2024-09-19T08:07:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb061519640 1 -- 172.21.15.98:0/1959115904 learned_addr learned my addr 172.21.15.98:0/1959115904 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb060d18640 1 --2- 172.21.15.98:0/1959115904 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb05c1036e0 0x7fb05c198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb061d1a640 1 --2- 172.21.15.98:0/1959115904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb05c1082a0 0x7fb05c19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb061519640 1 -- 172.21.15.98:0/1959115904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb05c1082a0 msgr2=0x7fb05c19cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb061519640 1 --2- 172.21.15.98:0/1959115904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb05c1082a0 0x7fb05c19cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb061519640 1 -- 172.21.15.98:0/1959115904 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb05c1036e0 msgr2=0x7fb05c198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb061519640 1 --2- 172.21.15.98:0/1959115904 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb05c1036e0 0x7fb05c198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:03.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb061519640 1 -- 172.21.15.98:0/1959115904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05c19d470 con 0x7fb05c102d30 2024-09-19T08:07:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.008+0000 7fb061519640 1 --2- 172.21.15.98:0/1959115904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb05c102d30 0x7fb05c1984c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb05400c920 tx=0x7fb05400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.009+0000 7fb05a7fc640 1 -- 172.21.15.98:0/1959115904 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb054007c70 con 0x7fb05c102d30 2024-09-19T08:07:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.009+0000 7fb05a7fc640 1 -- 172.21.15.98:0/1959115904 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb054007e10 con 0x7fb05c102d30 2024-09-19T08:07:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.009+0000 7fb05a7fc640 1 -- 172.21.15.98:0/1959115904 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb054018890 con 0x7fb05c102d30 2024-09-19T08:07:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.009+0000 7fb06377d640 1 -- 172.21.15.98:0/1959115904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb05c19d760 con 0x7fb05c102d30 2024-09-19T08:07:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.009+0000 7fb06377d640 1 -- 172.21.15.98:0/1959115904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05c1a5040 con 0x7fb05c102d30 2024-09-19T08:07:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.009+0000 7fb06377d640 1 -- 172.21.15.98:0/1959115904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb05c103110 con 0x7fb05c102d30 2024-09-19T08:07:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.011+0000 7fb05a7fc640 1 -- 172.21.15.98:0/1959115904 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fb054018a50 con 0x7fb05c102d30 2024-09-19T08:07:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.011+0000 7fb05a7fc640 1 --2- 172.21.15.98:0/1959115904 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb0440a33b0 0x7fb0440a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.011+0000 7fb060d18640 1 --2- 172.21.15.98:0/1959115904 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb0440a33b0 0x7fb0440a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.011+0000 7fb05a7fc640 1 -- 172.21.15.98:0/1959115904 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fb0540d0190 con 0x7fb05c102d30 2024-09-19T08:07:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.012+0000 7fb060d18640 1 --2- 172.21.15.98:0/1959115904 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb0440a33b0 0x7fb0440a5870 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb050005e00 tx=0x7fb050005c50 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.015+0000 7fb05a7fc640 1 -- 172.21.15.98:0/1959115904 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb054014030 con 0x7fb05c102d30 2024-09-19T08:07:03.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:02 smithi139 ceph-mon[34874]: pgmap v124: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:07:03.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:02 smithi098 ceph-mon[31333]: pgmap v124: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:07:03.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.215+0000 7fb06377d640 1 -- 172.21.15.98:0/1959115904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb05c199820 con 0x7fb05c102d30 2024-09-19T08:07:03.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.215+0000 7fb05a7fc640 1 -- 172.21.15.98:0/1959115904 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11985 (secure 0 0 0) 0x7fb05409cc30 con 0x7fb05c102d30 2024-09-19T08:07:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:07:03.217 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":30,"btime":"2024-09-19T08:05:21:864704+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:05:21.551452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:07:03.217 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 30 2024-09-19T08:07:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.217+0000 7fb06377d640 1 -- 172.21.15.98:0/1959115904 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb0440a33b0 msgr2=0x7fb0440a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.218+0000 7fb06377d640 1 --2- 172.21.15.98:0/1959115904 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb0440a33b0 0x7fb0440a5870 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb050005e00 tx=0x7fb050005c50 comp rx=0 tx=0).stop 2024-09-19T08:07:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.218+0000 7fb06377d640 1 -- 172.21.15.98:0/1959115904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb05c102d30 msgr2=0x7fb05c1984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.218+0000 7fb06377d640 1 --2- 172.21.15.98:0/1959115904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb05c102d30 0x7fb05c1984c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb05400c920 tx=0x7fb05400cde0 comp rx=0 tx=0).stop 2024-09-19T08:07:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.218+0000 7fb06377d640 1 -- 172.21.15.98:0/1959115904 shutdown_connections 2024-09-19T08:07:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.218+0000 7fb06377d640 1 --2- 172.21.15.98:0/1959115904 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb0440a33b0 0x7fb0440a5870 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.218+0000 7fb06377d640 1 --2- 172.21.15.98:0/1959115904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb05c1082a0 0x7fb05c19cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.218+0000 7fb06377d640 1 --2- 172.21.15.98:0/1959115904 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb05c1036e0 0x7fb05c198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.218+0000 7fb06377d640 1 --2- 172.21.15.98:0/1959115904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb05c102d30 0x7fb05c1984c0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.218+0000 7fb06377d640 1 -- 172.21.15.98:0/1959115904 >> 172.21.15.98:0/1959115904 conn(0x7fb05c0fc7d0 msgr2=0x7fb05c106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:03.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.218+0000 7fb06377d640 1 -- 172.21.15.98:0/1959115904 shutdown_connections 2024-09-19T08:07:03.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:03.218+0000 7fb06377d640 1 -- 172.21.15.98:0/1959115904 wait complete. 2024-09-19T08:07:03.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:02 smithi162 ceph-mon[35260]: pgmap v124: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:07:04.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:07:04.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:03 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1959115904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:07:04.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:03 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1959115904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:07:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:04.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:03 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1959115904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:07:04.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.752+0000 7f89e6c8c640 1 -- 172.21.15.98:0/345732811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e010c180 msgr2=0x7f89e010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:04.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.752+0000 7f89e6c8c640 1 --2- 172.21.15.98:0/345732811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e010c180 0x7f89e010e610 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f89d0009900 tx=0x7f89d002a480 comp rx=0 tx=0).stop 2024-09-19T08:07:04.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.752+0000 7f89e6c8c640 1 -- 172.21.15.98:0/345732811 shutdown_connections 2024-09-19T08:07:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.752+0000 7f89e6c8c640 1 --2- 172.21.15.98:0/345732811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e010c180 0x7f89e010e610 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.752+0000 7f89e6c8c640 1 --2- 172.21.15.98:0/345732811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89e0103700 0x7f89e010bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.752+0000 7f89e6c8c640 1 --2- 172.21.15.98:0/345732811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89e0102de0 0x7f89e01031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.752+0000 7f89e6c8c640 1 -- 172.21.15.98:0/345732811 >> 172.21.15.98:0/345732811 conn(0x7f89e00fc7a0 msgr2=0x7f89e00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.752+0000 7f89e6c8c640 1 -- 172.21.15.98:0/345732811 shutdown_connections 2024-09-19T08:07:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.752+0000 7f89e6c8c640 1 -- 172.21.15.98:0/345732811 wait complete. 2024-09-19T08:07:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.753+0000 7f89e6c8c640 1 Processor -- start 2024-09-19T08:07:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.753+0000 7f89e6c8c640 1 -- start start 2024-09-19T08:07:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.753+0000 7f89e6c8c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89e0102de0 0x7f89e0100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.753+0000 7f89e6c8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e0103700 0x7f89e0100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.753+0000 7f89e6c8c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89e010c180 0x7f89e01014c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.753+0000 7f89e6c8c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89e0111f30 con 0x7f89e0103700 2024-09-19T08:07:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.753+0000 7f89e6c8c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f89e0111db0 con 0x7f89e0102de0 2024-09-19T08:07:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.753+0000 7f89e6c8c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f89e01120b0 con 0x7f89e010c180 2024-09-19T08:07:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.753+0000 7f89dffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e0103700 0x7f89e0100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.753+0000 7f89dffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e0103700 0x7f89e0100f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40434/0 (socket says 172.21.15.98:40434) 2024-09-19T08:07:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.753+0000 7f89dffff640 1 -- 172.21.15.98:0/2681351593 learned_addr learned my addr 172.21.15.98:0/2681351593 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.754+0000 7f89e4a28640 1 --2- 172.21.15.98:0/2681351593 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89e0102de0 0x7f89e0100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.754+0000 7f89dffff640 1 -- 172.21.15.98:0/2681351593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89e010c180 msgr2=0x7f89e01014c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.754+0000 7f89dffff640 1 --2- 172.21.15.98:0/2681351593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89e010c180 0x7f89e01014c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.754+0000 7f89dffff640 1 -- 172.21.15.98:0/2681351593 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89e0102de0 msgr2=0x7f89e0100a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.754+0000 7f89dffff640 1 --2- 172.21.15.98:0/2681351593 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89e0102de0 0x7f89e0100a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.754+0000 7f89dffff640 1 -- 172.21.15.98:0/2681351593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e01a68b0 con 0x7f89e0103700 2024-09-19T08:07:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.754+0000 7f89e4a28640 1 --2- 172.21.15.98:0/2681351593 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89e0102de0 0x7f89e0100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.754+0000 7f89dffff640 1 --2- 172.21.15.98:0/2681351593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e0103700 0x7f89e0100f80 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f89d400d950 tx=0x7f89d400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.754+0000 7f89ddffb640 1 -- 172.21.15.98:0/2681351593 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4017070 con 0x7f89e0103700 2024-09-19T08:07:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.754+0000 7f89ddffb640 1 -- 172.21.15.98:0/2681351593 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d4009c30 con 0x7f89e0103700 2024-09-19T08:07:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.754+0000 7f89ddffb640 1 -- 172.21.15.98:0/2681351593 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d400f6b0 con 0x7f89e0103700 2024-09-19T08:07:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.754+0000 7f89e6c8c640 1 -- 172.21.15.98:0/2681351593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e01a6ba0 con 0x7f89e0103700 2024-09-19T08:07:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.754+0000 7f89e6c8c640 1 -- 172.21.15.98:0/2681351593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e01a7090 con 0x7f89e0103700 2024-09-19T08:07:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.756+0000 7f89ddffb640 1 -- 172.21.15.98:0/2681351593 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f89d40109d0 con 0x7f89e0103700 2024-09-19T08:07:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89e9065740 1 Processor -- start 2024-09-19T08:07:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89e9065740 1 -- start start 2024-09-19T08:07:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89e9065740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb3d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89e9065740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb3da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89e9065740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb48370 0x55bdfbb4a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89e9065740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bdfbaa32f0 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89e9065740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bdfba4acf0 con 0x55bdfbb3eb40 2024-09-19T08:07:04.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89e9065740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bdfba9f2b0 con 0x55bdfbb48370 2024-09-19T08:07:04.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89ddffb640 1 --2- 172.21.15.98:0/2681351593 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f89c80a3430 0x7f89c80a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89e4a28640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb3da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:04.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89e4a28640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb3da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40436/0 (socket says 172.21.15.98:40436) 2024-09-19T08:07:04.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89e4a28640 1 -- 172.21.15.98:0/1605372890 learned_addr learned my addr 172.21.15.98:0/1605372890 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:04.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89e5229640 1 --2- 172.21.15.98:0/2681351593 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f89c80a3430 0x7f89c80a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:04.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89e5229640 1 --2- 172.21.15.98:0/1605372890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb3d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:04.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.758+0000 7f89dffff640 1 --2- 172.21.15.98:0/1605372890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb48370 0x55bdfbb4a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:04.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.757+0000 7f89ddffb640 1 -- 172.21.15.98:0/2681351593 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f89d40cf920 con 0x7f89e0103700 2024-09-19T08:07:04.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.758+0000 7f89e4a28640 1 -- 172.21.15.98:0/1605372890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb48370 msgr2=0x55bdfbb4a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:04.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.758+0000 7f89e4a28640 1 --2- 172.21.15.98:0/1605372890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb48370 0x55bdfbb4a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.758+0000 7f89e4a28640 1 -- 172.21.15.98:0/1605372890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb3eb40 msgr2=0x55bdfbb3d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:04.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.758+0000 7f89e4a28640 1 --2- 172.21.15.98:0/1605372890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb3d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.758+0000 7f89e4a28640 1 -- 172.21.15.98:0/1605372890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdfba2f3a0 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.758+0000 7f89e5229640 1 --2- 172.21.15.98:0/1605372890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb3d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:04.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.758+0000 7f89dffff640 1 --2- 172.21.15.98:0/1605372890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb48370 0x55bdfbb4a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:04.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.758+0000 7f89e4a28640 1 --2- 172.21.15.98:0/1605372890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb3da10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f89d8008e70 tx=0x7f89d800d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439505dc4b149183 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:04.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.758+0000 7f89e5229640 1 --2- 172.21.15.98:0/2681351593 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f89c80a3430 0x7f89c80a58f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f89d00096c0 tx=0x7f89d0031490 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:04.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.758+0000 7f89c6ffd640 1 -- 172.21.15.98:0/1605372890 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d800f830 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.758+0000 7f89c6ffd640 1 -- 172.21.15.98:0/1605372890 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d8018480 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.758+0000 7f89c6ffd640 1 -- 172.21.15.98:0/1605372890 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d800e610 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.759+0000 7f89e9065740 1 -- 172.21.15.98:0/1605372890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3d5d0 msgr2=0x55bdfbb3da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:04.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.759+0000 7f89e9065740 1 --2- 172.21.15.98:0/1605372890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb3da10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f89d8008e70 tx=0x7f89d800d710 comp rx=0 tx=0).stop 2024-09-19T08:07:04.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.759+0000 7f89e9065740 1 -- 172.21.15.98:0/1605372890 shutdown_connections 2024-09-19T08:07:04.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.759+0000 7f89e9065740 1 --2- 172.21.15.98:0/1605372890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb48370 0x55bdfbb4a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.759+0000 7f89e9065740 1 --2- 172.21.15.98:0/1605372890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb3da10 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.759+0000 7f89e9065740 1 --2- 172.21.15.98:0/1605372890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb3d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.759+0000 7f89e9065740 1 -- 172.21.15.98:0/1605372890 >> 172.21.15.98:0/1605372890 conn(0x55bdfbb06310 msgr2=0x55bdfbb39c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:04.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.759+0000 7f89e9065740 1 -- 172.21.15.98:0/1605372890 shutdown_connections 2024-09-19T08:07:04.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.759+0000 7f89e9065740 1 -- 172.21.15.98:0/1605372890 wait complete. 2024-09-19T08:07:04.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.759+0000 7f89e9065740 1 -- start start 2024-09-19T08:07:04.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.759+0000 7f89e9065740 1 Processor -- start 2024-09-19T08:07:04.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.759+0000 7f89e9065740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb47ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89e9065740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb44f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89e9065740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb48370 0x55bdfbb454f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89e9065740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bdfbb4b070 con 0x55bdfbb3eb40 2024-09-19T08:07:04.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89e9065740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bdfba5e060 con 0x55bdfbb48370 2024-09-19T08:07:04.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89e9065740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bdfbb4b1f0 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89e9065740 10 client.0 _ll_register_callbacks cb 0x55bdfb7b18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:04.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89dffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb44f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:04.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89dffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb44f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40438/0 (socket says 172.21.15.98:40438) 2024-09-19T08:07:04.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89dffff640 1 -- 172.21.15.98:0/2653362965 learned_addr learned my addr 172.21.15.98:0/2653362965 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:04.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89e5229640 1 --2- 172.21.15.98:0/2653362965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb48370 0x55bdfbb454f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:04.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89e4a28640 1 --2- 172.21.15.98:0/2653362965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb47ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:04.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89dffff640 1 -- 172.21.15.98:0/2653362965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb3d5d0 msgr2=0x55bdfbb47ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:04.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89dffff640 1 --2- 172.21.15.98:0/2653362965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb47ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89dffff640 1 -- 172.21.15.98:0/2653362965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb48370 msgr2=0x55bdfbb454f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:04.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89dffff640 1 --2- 172.21.15.98:0/2653362965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb48370 0x55bdfbb454f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89dffff640 1 -- 172.21.15.98:0/2653362965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdfb8143c0 con 0x55bdfbb3eb40 2024-09-19T08:07:04.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89dffff640 1 --2- 172.21.15.98:0/2653362965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb44f90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f89d409bb60 tx=0x7f89d409c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd1d4a28a774eb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:04.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89bffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:04.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89bffff640 1 -- 172.21.15.98:0/2653362965 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4020040 con 0x55bdfbb3eb40 2024-09-19T08:07:04.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89bffff640 1 -- 172.21.15.98:0/2653362965 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f89d4086070 con 0x55bdfbb3eb40 2024-09-19T08:07:04.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89e9065740 1 -- 172.21.15.98:0/2653362965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89d8018480 con 0x55bdfbb3eb40 2024-09-19T08:07:04.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89bffff640 1 -- 172.21.15.98:0/2653362965 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f89d40cd4c0 con 0x55bdfbb3eb40 2024-09-19T08:07:04.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89e9065740 10 client.16530 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:04.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.760+0000 7f89e9065740 1 -- 172.21.15.98:0/2653362965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55bdfbb56a50 con 0x55bdfbb3eb40 2024-09-19T08:07:04.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.761+0000 7f89bffff640 1 -- 172.21.15.98:0/2653362965 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d40b2020 con 0x55bdfbb3eb40 2024-09-19T08:07:04.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.761+0000 7f89bffff640 1 -- 172.21.15.98:0/2653362965 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f89d4010460 con 0x55bdfbb3eb40 2024-09-19T08:07:04.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.761+0000 7f89e9065740 10 client.16530 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:04.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.761+0000 7f89e9065740 10 client.16530 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:07:04.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.761+0000 7f89e9065740 20 client.16530 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:04.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.761+0000 7f89e9065740 1 --2- 172.21.15.98:0/2653362965 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55bdfbb57f60 0x55bdfbb5a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.761+0000 7f89e9065740 4 client.16530 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:04.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.761+0000 7f89e9065740 1 -- 172.21.15.98:0/2653362965 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdfbab0860 con 0x55bdfbb57f60 2024-09-19T08:07:04.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.761+0000 7f89e4a28640 1 --2- 172.21.15.98:0/2653362965 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55bdfbb57f60 0x55bdfbb5a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:04.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.761+0000 7f89bffff640 10 client.16530 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:07:04.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.761+0000 7f89e4a28640 1 --2- 172.21.15.98:0/2653362965 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55bdfbb57f60 0x55bdfbb5a370 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ce05dfdf8f4578ec server_cookie=f18cfff879c1f13d in_seq=0 out_seq=0 2024-09-19T08:07:04.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.762+0000 7f89bffff640 1 -- 172.21.15.98:0/2653362965 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bdfbab0860 con 0x55bdfbb57f60 2024-09-19T08:07:04.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.762+0000 7f89bffff640 10 client.16530 handle_command_reply: tid=0 2024-09-19T08:07:04.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.762+0000 7f89e9065740 1 client.16530 shutdown 2024-09-19T08:07:04.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.762+0000 7f89e9065740 1 -- 172.21.15.98:0/2653362965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3eb40 msgr2=0x55bdfbb44f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:04.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.762+0000 7f89e9065740 1 --2- 172.21.15.98:0/2653362965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb44f90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f89d409bb60 tx=0x7f89d409c420 comp rx=0 tx=0).stop 2024-09-19T08:07:04.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.762+0000 7f89e9065740 1 -- 172.21.15.98:0/2653362965 shutdown_connections 2024-09-19T08:07:04.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.762+0000 7f89e9065740 1 --2- 172.21.15.98:0/2653362965 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55bdfbb57f60 0x55bdfbb5a370 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.762+0000 7f89e9065740 1 --2- 172.21.15.98:0/2653362965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb48370 0x55bdfbb454f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.762+0000 7f89e9065740 1 --2- 172.21.15.98:0/2653362965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb44f90 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.762+0000 7f89e9065740 1 --2- 172.21.15.98:0/2653362965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb47ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.762+0000 7f89e9065740 1 -- 172.21.15.98:0/2653362965 >> 172.21.15.98:0/2653362965 conn(0x55bdfbb06310 msgr2=0x55bdfbb3f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:04.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.762+0000 7f89bffff640 0 client.16530 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:07:04.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.762+0000 7f89e9065740 1 -- 172.21.15.98:0/2653362965 shutdown_connections 2024-09-19T08:07:04.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.763+0000 7f89e9065740 1 -- 172.21.15.98:0/2653362965 wait complete. 2024-09-19T08:07:04.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.763+0000 7f89e9065740 20 client.16530 trim_cache size 0 max 16384 2024-09-19T08:07:04.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.775+0000 7f89e9065740 1 Processor -- start 2024-09-19T08:07:04.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.775+0000 7f89e9065740 1 -- start start 2024-09-19T08:07:04.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89e9065740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb39e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89e9065740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb54370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89e9065740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb48370 0x55bdfbb44850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89e9065740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89e01af9d0 con 0x55bdfbb3eb40 2024-09-19T08:07:04.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89e9065740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f89e01b1570 con 0x55bdfbb48370 2024-09-19T08:07:04.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89e9065740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f89e01b00a0 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89dffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb54370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:04.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89dffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb54370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40448/0 (socket says 172.21.15.98:40448) 2024-09-19T08:07:04.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89dffff640 1 -- 172.21.15.98:0/1198259442 learned_addr learned my addr 172.21.15.98:0/1198259442 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:04.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89dffff640 1 -- 172.21.15.98:0/1198259442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb3d5d0 msgr2=0x55bdfbb39e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:04.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89dffff640 1 --2- 172.21.15.98:0/1198259442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb39e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89dffff640 1 -- 172.21.15.98:0/1198259442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb48370 msgr2=0x55bdfbb44850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:04.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89dffff640 1 --2- 172.21.15.98:0/1198259442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb48370 0x55bdfbb44850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89dffff640 1 -- 172.21.15.98:0/1198259442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdfbb434e0 con 0x55bdfbb3eb40 2024-09-19T08:07:04.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89dffff640 1 --2- 172.21.15.98:0/1198259442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb54370 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f89d409bb10 tx=0x7f89d400bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bab9d64512857c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:04.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89c67fc640 1 -- 172.21.15.98:0/1198259442 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d40a4840 con 0x55bdfbb3eb40 2024-09-19T08:07:04.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89c67fc640 1 -- 172.21.15.98:0/1198259442 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d40b2070 con 0x55bdfbb3eb40 2024-09-19T08:07:04.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.776+0000 7f89c67fc640 1 -- 172.21.15.98:0/1198259442 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d40cd2c0 con 0x55bdfbb3eb40 2024-09-19T08:07:04.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.777+0000 7f89e9065740 1 -- 172.21.15.98:0/1198259442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3eb40 msgr2=0x55bdfbb54370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:04.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.777+0000 7f89e9065740 1 --2- 172.21.15.98:0/1198259442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb54370 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f89d409bb10 tx=0x7f89d400bf60 comp rx=0 tx=0).stop 2024-09-19T08:07:04.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.777+0000 7f89e9065740 1 -- 172.21.15.98:0/1198259442 shutdown_connections 2024-09-19T08:07:04.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.777+0000 7f89e9065740 1 --2- 172.21.15.98:0/1198259442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb48370 0x55bdfbb44850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.777+0000 7f89e9065740 1 --2- 172.21.15.98:0/1198259442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb54370 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.777+0000 7f89e9065740 1 --2- 172.21.15.98:0/1198259442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb39e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.777+0000 7f89e9065740 1 -- 172.21.15.98:0/1198259442 >> 172.21.15.98:0/1198259442 conn(0x55bdfbb06310 msgr2=0x55bdfbb39700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:04.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.777+0000 7f89e9065740 1 -- 172.21.15.98:0/1198259442 shutdown_connections 2024-09-19T08:07:04.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.777+0000 7f89e9065740 1 -- 172.21.15.98:0/1198259442 wait complete. 2024-09-19T08:07:04.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.777+0000 7f89e9065740 1 -- start start 2024-09-19T08:07:04.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.777+0000 7f89e9065740 1 Processor -- start 2024-09-19T08:07:04.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e9065740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb506c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e9065740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb50c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e9065740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb48370 0x55bdfbb41200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e9065740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89e01b0e70 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e9065740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f89e01b07a0 con 0x55bdfbb3eb40 2024-09-19T08:07:04.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e9065740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f89e01ad9b0 con 0x55bdfbb48370 2024-09-19T08:07:04.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e9065740 10 client.0 _ll_register_callbacks cb 0x7f89d409b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:04.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e4a28640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:04.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e4a28640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb506c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40450/0 (socket says 172.21.15.98:40450) 2024-09-19T08:07:04.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e4a28640 1 -- 172.21.15.98:0/3031325424 learned_addr learned my addr 172.21.15.98:0/3031325424 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:04.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89dffff640 1 --2- 172.21.15.98:0/3031325424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb50c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:04.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e4a28640 1 -- 172.21.15.98:0/3031325424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb48370 msgr2=0x55bdfbb41200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:04.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e4a28640 1 --2- 172.21.15.98:0/3031325424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb48370 0x55bdfbb41200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e4a28640 1 -- 172.21.15.98:0/3031325424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb3eb40 msgr2=0x55bdfbb50c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:04.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e4a28640 1 --2- 172.21.15.98:0/3031325424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb50c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e4a28640 1 -- 172.21.15.98:0/3031325424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdfbb41870 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89dffff640 1 --2- 172.21.15.98:0/3031325424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb50c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:04.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e4a28640 1 --2- 172.21.15.98:0/3031325424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb506c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f89d8008d90 tx=0x7f89d800dee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb6d0e7b7bf7d3f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:04.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89bffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:04.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89bffff640 1 -- 172.21.15.98:0/3031325424 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8018830 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89bffff640 1 -- 172.21.15.98:0/3031325424 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f89d800f630 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e9065740 1 -- 172.21.15.98:0/3031325424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89d40b2070 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89bffff640 1 -- 172.21.15.98:0/3031325424 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f89d80189d0 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e9065740 10 client.16536 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:04.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.778+0000 7f89e9065740 1 -- 172.21.15.98:0/3031325424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55bdfbb42b70 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.779+0000 7f89bffff640 1 -- 172.21.15.98:0/3031325424 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d40b2070 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.779+0000 7f89bffff640 1 -- 172.21.15.98:0/3031325424 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f89d802e910 con 0x55bdfbb3d5d0 2024-09-19T08:07:04.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.779+0000 7f89e9065740 10 client.16536 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:04.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.779+0000 7f89e9065740 10 client.16536 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:07:04.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.779+0000 7f89e9065740 20 client.16536 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:04.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.779+0000 7f89e9065740 1 --2- 172.21.15.98:0/3031325424 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55bdfbbcc430 0x55bdfbbce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:04.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.779+0000 7f89e9065740 4 client.16536 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:07:04.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.779+0000 7f89e9065740 1 -- 172.21.15.98:0/3031325424 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bdfbaaad70 con 0x55bdfbbcc430 2024-09-19T08:07:04.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.780+0000 7f89dffff640 1 --2- 172.21.15.98:0/3031325424 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55bdfbbcc430 0x55bdfbbce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:04.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.780+0000 7f89dffff640 1 --2- 172.21.15.98:0/3031325424 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55bdfbbcc430 0x55bdfbbce820 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7d3264c4fa23b89c server_cookie=719a040fa69ad7c0 in_seq=0 out_seq=0 2024-09-19T08:07:04.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.780+0000 7f89bffff640 10 client.16536 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:07:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.780+0000 7f89bffff640 1 -- 172.21.15.98:0/3031325424 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bdfbaaad70 con 0x55bdfbbcc430 2024-09-19T08:07:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.780+0000 7f89bffff640 10 client.16536 handle_command_reply: tid=0 2024-09-19T08:07:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.780+0000 7f89e9065740 1 client.16536 shutdown 2024-09-19T08:07:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.780+0000 7f89e9065740 1 -- 172.21.15.98:0/3031325424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3d5d0 msgr2=0x55bdfbb506c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.780+0000 7f89e9065740 1 --2- 172.21.15.98:0/3031325424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb506c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f89d8008d90 tx=0x7f89d800dee0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89e9065740 1 -- 172.21.15.98:0/3031325424 shutdown_connections 2024-09-19T08:07:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89e9065740 1 --2- 172.21.15.98:0/3031325424 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55bdfbbcc430 0x55bdfbbce820 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89e9065740 1 --2- 172.21.15.98:0/3031325424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdfbb48370 0x55bdfbb41200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89bffff640 0 client.16536 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:07:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89e9065740 1 --2- 172.21.15.98:0/3031325424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bdfbb3eb40 0x55bdfbb50c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89e9065740 1 --2- 172.21.15.98:0/3031325424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bdfbb3d5d0 0x55bdfbb506c0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89e9065740 1 -- 172.21.15.98:0/3031325424 >> 172.21.15.98:0/3031325424 conn(0x55bdfbb06310 msgr2=0x55bdfbb467f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89e9065740 1 -- 172.21.15.98:0/3031325424 shutdown_connections 2024-09-19T08:07:04.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89e9065740 1 -- 172.21.15.98:0/3031325424 wait complete. 2024-09-19T08:07:04.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89e9065740 20 client.16536 trim_cache size 0 max 16384 2024-09-19T08:07:04.799 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:07:04.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89e6c8c640 1 -- 172.21.15.98:0/2681351593 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f89c80a3430 msgr2=0x7f89c80a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:04.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89e6c8c640 1 --2- 172.21.15.98:0/2681351593 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f89c80a3430 0x7f89c80a58f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f89d00096c0 tx=0x7f89d0031490 comp rx=0 tx=0).stop 2024-09-19T08:07:04.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89e6c8c640 1 -- 172.21.15.98:0/2681351593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e0103700 msgr2=0x7f89e0100f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:04.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89e6c8c640 1 --2- 172.21.15.98:0/2681351593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e0103700 0x7f89e0100f80 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f89d400d950 tx=0x7f89d400de10 comp rx=0 tx=0).stop 2024-09-19T08:07:04.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.781+0000 7f89e6c8c640 1 -- 172.21.15.98:0/2681351593 shutdown_connections 2024-09-19T08:07:04.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.782+0000 7f89e6c8c640 1 --2- 172.21.15.98:0/2681351593 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f89c80a3430 0x7f89c80a58f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.782+0000 7f89e6c8c640 1 --2- 172.21.15.98:0/2681351593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89e010c180 0x7f89e01014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.782+0000 7f89e6c8c640 1 --2- 172.21.15.98:0/2681351593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e0103700 0x7f89e0100f80 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.782+0000 7f89e6c8c640 1 --2- 172.21.15.98:0/2681351593 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89e0102de0 0x7f89e0100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:04.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.782+0000 7f89e6c8c640 1 -- 172.21.15.98:0/2681351593 >> 172.21.15.98:0/2681351593 conn(0x7f89e00fc7a0 msgr2=0x7f89e00fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.782+0000 7f89e6c8c640 1 -- 172.21.15.98:0/2681351593 shutdown_connections 2024-09-19T08:07:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:04.782+0000 7f89e6c8c640 1 -- 172.21.15.98:0/2681351593 wait complete. 2024-09-19T08:07:04.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:04 smithi098 ceph-mon[31333]: pgmap v125: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:07:05.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:04 smithi139 ceph-mon[34874]: pgmap v125: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:07:05.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:04 smithi162 ceph-mon[35260]: pgmap v125: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:07:05.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:07:05.715 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:06.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.219+0000 7ff5bd30e640 1 -- 172.21.15.98:0/1031291607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b8106f10 msgr2=0x7ff5b810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:06.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.219+0000 7ff5bd30e640 1 --2- 172.21.15.98:0/1031291607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b8106f10 0x7ff5b810ea90 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009ca0 tx=0x7ff5b002a8c0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.219+0000 7ff5bd30e640 1 -- 172.21.15.98:0/1031291607 shutdown_connections 2024-09-19T08:07:06.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.219+0000 7ff5bd30e640 1 --2- 172.21.15.98:0/1031291607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b8106f10 0x7ff5b810ea90 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.219+0000 7ff5bd30e640 1 --2- 172.21.15.98:0/1031291607 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5b8102790 0x7ff5b81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.219+0000 7ff5bd30e640 1 --2- 172.21.15.98:0/1031291607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b8101de0 0x7ff5b81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.219+0000 7ff5bd30e640 1 -- 172.21.15.98:0/1031291607 >> 172.21.15.98:0/1031291607 conn(0x7ff5b80fd8c0 msgr2=0x7ff5b80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:06.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.219+0000 7ff5bd30e640 1 -- 172.21.15.98:0/1031291607 shutdown_connections 2024-09-19T08:07:06.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.219+0000 7ff5bd30e640 1 -- 172.21.15.98:0/1031291607 wait complete. 2024-09-19T08:07:06.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.219+0000 7ff5bd30e640 1 Processor -- start 2024-09-19T08:07:06.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5bd30e640 1 -- start start 2024-09-19T08:07:06.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5bd30e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b8101de0 0x7ff5b81983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5bd30e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5b8102790 0x7ff5b8198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5bd30e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b8106f10 0x7ff5b819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5bd30e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b8111f70 con 0x7ff5b8106f10 2024-09-19T08:07:06.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5bd30e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b8111df0 con 0x7ff5b8102790 2024-09-19T08:07:06.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5bd30e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b81120f0 con 0x7ff5b8101de0 2024-09-19T08:07:06.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5b77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b8106f10 0x7ff5b819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5b77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b8106f10 0x7ff5b819ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40466/0 (socket says 172.21.15.98:40466) 2024-09-19T08:07:06.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5b77fe640 1 -- 172.21.15.98:0/3640052075 learned_addr learned my addr 172.21.15.98:0/3640052075 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:06.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/3640052075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5b8102790 0x7ff5b8198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5b6ffd640 1 --2- 172.21.15.98:0/3640052075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b8101de0 0x7ff5b81983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5b77fe640 1 -- 172.21.15.98:0/3640052075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b8101de0 msgr2=0x7ff5b81983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:06.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5b77fe640 1 --2- 172.21.15.98:0/3640052075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b8101de0 0x7ff5b81983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5b77fe640 1 -- 172.21.15.98:0/3640052075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5b8102790 msgr2=0x7ff5b8198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:06.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5b77fe640 1 --2- 172.21.15.98:0/3640052075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5b8102790 0x7ff5b8198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.220+0000 7ff5b77fe640 1 -- 172.21.15.98:0/3640052075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b819d3a0 con 0x7ff5b8106f10 2024-09-19T08:07:06.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.221+0000 7ff5b77fe640 1 --2- 172.21.15.98:0/3640052075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b8106f10 0x7ff5b819ccc0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0031a20 tx=0x7ff5b0031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:06.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.221+0000 7ff5affff640 1 -- 172.21.15.98:0/3640052075 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0003c30 con 0x7ff5b8106f10 2024-09-19T08:07:06.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.221+0000 7ff5bd30e640 1 -- 172.21.15.98:0/3640052075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5b819d630 con 0x7ff5b8106f10 2024-09-19T08:07:06.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.221+0000 7ff5affff640 1 -- 172.21.15.98:0/3640052075 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5b0003dd0 con 0x7ff5b8106f10 2024-09-19T08:07:06.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.221+0000 7ff5affff640 1 -- 172.21.15.98:0/3640052075 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b002fcf0 con 0x7ff5b8106f10 2024-09-19T08:07:06.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.221+0000 7ff5bd30e640 1 -- 172.21.15.98:0/3640052075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5b81a4f70 con 0x7ff5b8106f10 2024-09-19T08:07:06.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.223+0000 7ff5affff640 1 -- 172.21.15.98:0/3640052075 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7ff5b0004050 con 0x7ff5b8106f10 2024-09-19T08:07:06.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.223+0000 7ff5bf6e7740 1 Processor -- start 2024-09-19T08:07:06.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.223+0000 7ff5bf6e7740 1 -- start start 2024-09-19T08:07:06.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5bf6e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d295bb40 0x55f2d295a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5bf6e7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295a5d0 0x55f2d295aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5bf6e7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d2969780 0x55f2d296bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5bf6e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f2d28c02f0 con 0x55f2d295bb40 2024-09-19T08:07:06.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5bf6e7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f2d2867cf0 con 0x55f2d2969780 2024-09-19T08:07:06.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5bf6e7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f2d28bc2b0 con 0x55f2d295a5d0 2024-09-19T08:07:06.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d295bb40 0x55f2d295a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5affff640 1 --2- 172.21.15.98:0/3640052075 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff5a40a33b0 0x7ff5a40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d295bb40 0x55f2d295a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40468/0 (socket says 172.21.15.98:40468) 2024-09-19T08:07:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5b67fc640 1 -- 172.21.15.98:0/2830153680 learned_addr learned my addr 172.21.15.98:0/2830153680 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5b6ffd640 1 --2- 172.21.15.98:0/2830153680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295a5d0 0x55f2d295aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/2830153680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d2969780 0x55f2d296bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5b77fe640 1 --2- 172.21.15.98:0/3640052075 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff5a40a33b0 0x7ff5a40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5b67fc640 1 -- 172.21.15.98:0/2830153680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295a5d0 msgr2=0x55f2d295aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/2830153680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295a5d0 0x55f2d295aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5b67fc640 1 -- 172.21.15.98:0/2830153680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d2969780 msgr2=0x55f2d296bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/2830153680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d2969780 0x55f2d296bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5b67fc640 1 -- 172.21.15.98:0/2830153680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2d284c3a0 con 0x55f2d295bb40 2024-09-19T08:07:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5affff640 1 -- 172.21.15.98:0/3640052075 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff5b00f0e80 con 0x7ff5b8106f10 2024-09-19T08:07:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/2830153680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d295bb40 0x55f2d295a090 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00058a0 tx=0x7ff5a000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5913a9c4d06102 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5ad7fa640 1 -- 172.21.15.98:0/2830153680 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0004030 con 0x55f2d295bb40 2024-09-19T08:07:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5ad7fa640 1 -- 172.21.15.98:0/2830153680 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5a00041d0 con 0x55f2d295bb40 2024-09-19T08:07:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.224+0000 7ff5ad7fa640 1 -- 172.21.15.98:0/2830153680 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a000f6c0 con 0x55f2d295bb40 2024-09-19T08:07:06.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.225+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/2830153680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d295bb40 msgr2=0x55f2d295a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:06.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.225+0000 7ff5b77fe640 1 --2- 172.21.15.98:0/3640052075 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff5a40a33b0 0x7ff5a40a5870 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00eec00 tx=0x7ff5b00b96c0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:06.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.225+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/2830153680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d295bb40 0x55f2d295a090 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00058a0 tx=0x7ff5a000b7c0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.225+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/2830153680 shutdown_connections 2024-09-19T08:07:06.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.225+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/2830153680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d2969780 0x55f2d296bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.225+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/2830153680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295a5d0 0x55f2d295aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.225+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/2830153680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d295bb40 0x55f2d295a090 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.225+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/2830153680 >> 172.21.15.98:0/2830153680 conn(0x55f2d2923310 msgr2=0x55f2d2956c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:06.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.225+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/2830153680 shutdown_connections 2024-09-19T08:07:06.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.225+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/2830153680 wait complete. 2024-09-19T08:07:06.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.226+0000 7ff5bf6e7740 1 -- start start 2024-09-19T08:07:06.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.226+0000 7ff5bf6e7740 1 Processor -- start 2024-09-19T08:07:06.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.226+0000 7ff5bf6e7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d295a5d0 0x55f2d29691e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.226+0000 7ff5bf6e7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 0x55f2d2966490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.226+0000 7ff5bf6e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d2969780 0x55f2d2966c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.226+0000 7ff5bf6e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f2d296c480 con 0x55f2d2969780 2024-09-19T08:07:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.226+0000 7ff5bf6e7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f2d287b060 con 0x55f2d295a5d0 2024-09-19T08:07:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.226+0000 7ff5bf6e7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f2d296c600 con 0x55f2d295bb40 2024-09-19T08:07:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.226+0000 7ff5bf6e7740 10 client.0 _ll_register_callbacks cb 0x55f2d25ce8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.226+0000 7ff5b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d2969780 0x55f2d2966c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.226+0000 7ff5b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d2969780 0x55f2d2966c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40472/0 (socket says 172.21.15.98:40472) 2024-09-19T08:07:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.226+0000 7ff5b67fc640 1 -- 172.21.15.98:0/1512316041 learned_addr learned my addr 172.21.15.98:0/1512316041 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.226+0000 7ff5b6ffd640 1 --2- 172.21.15.98:0/1512316041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 0x55f2d2966490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.226+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/1512316041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d295a5d0 0x55f2d29691e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff5b67fc640 1 -- 172.21.15.98:0/1512316041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 msgr2=0x55f2d2966490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/1512316041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 0x55f2d2966490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff5b67fc640 1 -- 172.21.15.98:0/1512316041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d295a5d0 msgr2=0x55f2d29691e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:06.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/1512316041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d295a5d0 0x55f2d29691e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff5b67fc640 1 -- 172.21.15.98:0/1512316041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2d26313c0 con 0x55f2d2969780 2024-09-19T08:07:06.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/1512316041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d2969780 0x55f2d2966c20 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000be60 tx=0x7ff5a0003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c67588d5830c2a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:06.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff59e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:06.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff59e7fc640 1 -- 172.21.15.98:0/1512316041 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a00053f0 con 0x55f2d2969780 2024-09-19T08:07:06.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff59e7fc640 1 -- 172.21.15.98:0/1512316041 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff5a0005590 con 0x55f2d2969780 2024-09-19T08:07:06.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/1512316041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5a00041d0 con 0x55f2d2969780 2024-09-19T08:07:06.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff59e7fc640 1 -- 172.21.15.98:0/1512316041 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff5a00035f0 con 0x55f2d2969780 2024-09-19T08:07:06.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff5bf6e7740 10 client.16548 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:06.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/1512316041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f2d2977fd0 con 0x55f2d2969780 2024-09-19T08:07:06.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff59e7fc640 1 -- 172.21.15.98:0/1512316041 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5a00041d0 con 0x55f2d2969780 2024-09-19T08:07:06.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff59e7fc640 1 -- 172.21.15.98:0/1512316041 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7ff5a001d070 con 0x55f2d2969780 2024-09-19T08:07:06.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff5bf6e7740 10 client.16548 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:06.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff5bf6e7740 10 client.16548 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:07:06.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.227+0000 7ff5bf6e7740 20 client.16548 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:06.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.228+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/1512316041 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f2d2979320 0x55f2d297b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.228+0000 7ff5bf6e7740 4 client.16548 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:06.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.228+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/1512316041 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2d28cd860 con 0x55f2d2979320 2024-09-19T08:07:06.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.228+0000 7ff5b6ffd640 1 --2- 172.21.15.98:0/1512316041 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f2d2979320 0x55f2d297b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.228+0000 7ff5b6ffd640 1 --2- 172.21.15.98:0/1512316041 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f2d2979320 0x55f2d297b710 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bd3e50bf975866cf server_cookie=6a9c41cfc6d9ea14 in_seq=0 out_seq=0 2024-09-19T08:07:06.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.228+0000 7ff59e7fc640 10 client.16548 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:07:06.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.229+0000 7ff59e7fc640 1 -- 172.21.15.98:0/1512316041 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f2d28cd860 con 0x55f2d2979320 2024-09-19T08:07:06.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.229+0000 7ff59e7fc640 10 client.16548 handle_command_reply: tid=0 2024-09-19T08:07:06.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.229+0000 7ff5bf6e7740 1 client.16548 shutdown 2024-09-19T08:07:06.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.229+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/1512316041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d2969780 msgr2=0x55f2d2966c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:06.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.229+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/1512316041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d2969780 0x55f2d2966c20 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000be60 tx=0x7ff5a0003d80 comp rx=0 tx=0).stop 2024-09-19T08:07:06.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.229+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/1512316041 shutdown_connections 2024-09-19T08:07:06.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.229+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/1512316041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d2969780 0x55f2d2966c20 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.229+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/1512316041 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f2d2979320 0x55f2d297b710 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.230+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/1512316041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 0x55f2d2966490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.230+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/1512316041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d295a5d0 0x55f2d29691e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.230+0000 7ff59e7fc640 0 client.16548 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:07:06.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.230+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/1512316041 >> 172.21.15.98:0/1512316041 conn(0x55f2d2923310 msgr2=0x55f2d295c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:06.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.230+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/1512316041 shutdown_connections 2024-09-19T08:07:06.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.230+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/1512316041 wait complete. 2024-09-19T08:07:06.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.230+0000 7ff5bf6e7740 20 client.16548 trim_cache size 0 max 16384 2024-09-19T08:07:06.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.243+0000 7ff5bf6e7740 1 Processor -- start 2024-09-19T08:07:06.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.243+0000 7ff5bf6e7740 1 -- start start 2024-09-19T08:07:06.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.243+0000 7ff5bf6e7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d295a5d0 0x55f2d2980e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.243+0000 7ff5bf6e7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 0x55f2d29813c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.243+0000 7ff5bf6e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d2969780 0x55f2d2963710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.243+0000 7ff5bf6e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b81ab640 con 0x55f2d2969780 2024-09-19T08:07:06.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.243+0000 7ff5bf6e7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b81ad1e0 con 0x55f2d295a5d0 2024-09-19T08:07:06.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.243+0000 7ff5bf6e7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b81abd10 con 0x55f2d295bb40 2024-09-19T08:07:06.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.243+0000 7ff5b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d2969780 0x55f2d2963710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.243+0000 7ff5b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d2969780 0x55f2d2963710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40484/0 (socket says 172.21.15.98:40484) 2024-09-19T08:07:06.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.243+0000 7ff5b67fc640 1 -- 172.21.15.98:0/3527232171 learned_addr learned my addr 172.21.15.98:0/3527232171 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:06.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.243+0000 7ff5b6ffd640 1 --2- 172.21.15.98:0/3527232171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 0x55f2d29813c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.244+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/3527232171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d295a5d0 0x55f2d2980e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.244+0000 7ff5b67fc640 1 -- 172.21.15.98:0/3527232171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 msgr2=0x55f2d29813c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:06.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.244+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/3527232171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 0x55f2d29813c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.244+0000 7ff5b67fc640 1 -- 172.21.15.98:0/3527232171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d295a5d0 msgr2=0x55f2d2980e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:06.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.244+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/3527232171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d295a5d0 0x55f2d2980e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.244+0000 7ff5b67fc640 1 -- 172.21.15.98:0/3527232171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2d2956ed0 con 0x55f2d2969780 2024-09-19T08:07:06.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.244+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/3527232171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d2969780 0x55f2d2963710 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0012e60 tx=0x7ff5a000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9020e4f9feb07da6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:06.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.244+0000 7ff5acff9640 1 -- 172.21.15.98:0/3527232171 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a002f480 con 0x55f2d2969780 2024-09-19T08:07:06.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.244+0000 7ff5acff9640 1 -- 172.21.15.98:0/3527232171 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5a0038070 con 0x55f2d2969780 2024-09-19T08:07:06.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.244+0000 7ff5acff9640 1 -- 172.21.15.98:0/3527232171 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a00106c0 con 0x55f2d2969780 2024-09-19T08:07:06.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.245+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/3527232171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d2969780 msgr2=0x55f2d2963710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:06.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.245+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/3527232171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d2969780 0x55f2d2963710 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0012e60 tx=0x7ff5a000b390 comp rx=0 tx=0).stop 2024-09-19T08:07:06.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.245+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/3527232171 shutdown_connections 2024-09-19T08:07:06.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.245+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/3527232171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d2969780 0x55f2d2963710 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.245+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/3527232171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 0x55f2d29813c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.245+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/3527232171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d295a5d0 0x55f2d2980e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.245+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/3527232171 >> 172.21.15.98:0/3527232171 conn(0x55f2d2923310 msgr2=0x55f2d2964d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:06.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.245+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/3527232171 shutdown_connections 2024-09-19T08:07:06.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.245+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/3527232171 wait complete. 2024-09-19T08:07:06.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.245+0000 7ff5bf6e7740 1 -- start start 2024-09-19T08:07:06.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.245+0000 7ff5bf6e7740 1 Processor -- start 2024-09-19T08:07:06.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.245+0000 7ff5bf6e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d295a5d0 0x55f2d2971e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5bf6e7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 0x55f2d2963010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5bf6e7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d2969780 0x55f2d2963550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5bf6e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b81acae0 con 0x55f2d295a5d0 2024-09-19T08:07:06.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5bf6e7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b81ac410 con 0x55f2d2969780 2024-09-19T08:07:06.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5bf6e7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b81a9620 con 0x55f2d295bb40 2024-09-19T08:07:06.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5bf6e7740 10 client.0 _ll_register_callbacks cb 0x7ff5a00123f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:06.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d295a5d0 0x55f2d2971e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d295a5d0 0x55f2d2971e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40500/0 (socket says 172.21.15.98:40500) 2024-09-19T08:07:06.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5b67fc640 1 -- 172.21.15.98:0/3679676714 learned_addr learned my addr 172.21.15.98:0/3679676714 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:06.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5b6ffd640 1 --2- 172.21.15.98:0/3679676714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 0x55f2d2963010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5b67fc640 1 -- 172.21.15.98:0/3679676714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 msgr2=0x55f2d2963010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:06.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/3679676714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 0x55f2d2963010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5b67fc640 1 -- 172.21.15.98:0/3679676714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d2969780 msgr2=0x55f2d2963550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:06.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/3679676714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d2969780 0x55f2d2963550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5b67fc640 1 -- 172.21.15.98:0/3679676714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2d2963b40 con 0x55f2d295a5d0 2024-09-19T08:07:06.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5b67fc640 1 --2- 172.21.15.98:0/3679676714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d295a5d0 0x55f2d2971e00 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0005bf0 tx=0x7ff5a0005c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b9cf9ab2f58d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:06.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff59e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:06.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff59e7fc640 1 -- 172.21.15.98:0/3679676714 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0043060 con 0x55f2d295a5d0 2024-09-19T08:07:06.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff59e7fc640 1 -- 172.21.15.98:0/3679676714 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff5a0010870 con 0x55f2d295a5d0 2024-09-19T08:07:06.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/3679676714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5a0038070 con 0x55f2d295a5d0 2024-09-19T08:07:06.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff59e7fc640 1 -- 172.21.15.98:0/3679676714 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff5a002f480 con 0x55f2d295a5d0 2024-09-19T08:07:06.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5bf6e7740 10 client.16554 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:06.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.246+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/3679676714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f2d29e3c50 con 0x55f2d295a5d0 2024-09-19T08:07:06.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.247+0000 7ff59e7fc640 1 -- 172.21.15.98:0/3679676714 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5a0038070 con 0x55f2d295a5d0 2024-09-19T08:07:06.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.247+0000 7ff59e7fc640 1 -- 172.21.15.98:0/3679676714 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7ff5a001d070 con 0x55f2d295a5d0 2024-09-19T08:07:06.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.247+0000 7ff5bf6e7740 10 client.16554 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:06.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.247+0000 7ff5bf6e7740 10 client.16554 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:07:06.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.247+0000 7ff5bf6e7740 20 client.16554 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.247+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/3679676714 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f2d29e40c0 0x55f2d29e64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.247+0000 7ff5bf6e7740 4 client.16554 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:07:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.247+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/3679676714 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f2d28c7d70 con 0x55f2d29e40c0 2024-09-19T08:07:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.247+0000 7ff5b6ffd640 1 --2- 172.21.15.98:0/3679676714 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f2d29e40c0 0x55f2d29e64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff59e7fc640 10 client.16554 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:07:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff5b6ffd640 1 --2- 172.21.15.98:0/3679676714 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f2d29e40c0 0x55f2d29e64b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3abc87bc73f57189 server_cookie=b064f63fa7346efb in_seq=0 out_seq=0 2024-09-19T08:07:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff59e7fc640 1 -- 172.21.15.98:0/3679676714 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f2d28c7d70 con 0x55f2d29e40c0 2024-09-19T08:07:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff59e7fc640 10 client.16554 handle_command_reply: tid=0 2024-09-19T08:07:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff5bf6e7740 1 client.16554 shutdown 2024-09-19T08:07:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/3679676714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d295a5d0 msgr2=0x55f2d2971e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/3679676714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d295a5d0 0x55f2d2971e00 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0005bf0 tx=0x7ff5a0005c20 comp rx=0 tx=0).stop 2024-09-19T08:07:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/3679676714 shutdown_connections 2024-09-19T08:07:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/3679676714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d2969780 0x55f2d2963550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/3679676714 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55f2d29e40c0 0x55f2d29e64b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/3679676714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d295bb40 0x55f2d2963010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff5bf6e7740 1 --2- 172.21.15.98:0/3679676714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d295a5d0 0x55f2d2971e00 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/3679676714 >> 172.21.15.98:0/3679676714 conn(0x55f2d2923310 msgr2=0x55f2d2968510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:06.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff59e7fc640 0 client.16554 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:07:06.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/3679676714 shutdown_connections 2024-09-19T08:07:06.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff5bf6e7740 1 -- 172.21.15.98:0/3679676714 wait complete. 2024-09-19T08:07:06.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.248+0000 7ff5bf6e7740 20 client.16554 trim_cache size 0 max 16384 2024-09-19T08:07:06.259 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:07:06.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.249+0000 7ff5bd30e640 1 -- 172.21.15.98:0/3640052075 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff5a40a33b0 msgr2=0x7ff5a40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:06.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.249+0000 7ff5bd30e640 1 --2- 172.21.15.98:0/3640052075 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff5a40a33b0 0x7ff5a40a5870 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00eec00 tx=0x7ff5b00b96c0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.249+0000 7ff5bd30e640 1 -- 172.21.15.98:0/3640052075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b8106f10 msgr2=0x7ff5b819ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:06.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.249+0000 7ff5bd30e640 1 --2- 172.21.15.98:0/3640052075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b8106f10 0x7ff5b819ccc0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0031a20 tx=0x7ff5b0031a50 comp rx=0 tx=0).stop 2024-09-19T08:07:06.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.249+0000 7ff5bd30e640 1 -- 172.21.15.98:0/3640052075 shutdown_connections 2024-09-19T08:07:06.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.249+0000 7ff5bd30e640 1 --2- 172.21.15.98:0/3640052075 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff5a40a33b0 0x7ff5a40a5870 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.249+0000 7ff5bd30e640 1 --2- 172.21.15.98:0/3640052075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b8106f10 0x7ff5b819ccc0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.249+0000 7ff5bd30e640 1 --2- 172.21.15.98:0/3640052075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5b8102790 0x7ff5b8198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.249+0000 7ff5bd30e640 1 --2- 172.21.15.98:0/3640052075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b8101de0 0x7ff5b81983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:06.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.249+0000 7ff5bd30e640 1 -- 172.21.15.98:0/3640052075 >> 172.21.15.98:0/3640052075 conn(0x7ff5b80fd8c0 msgr2=0x7ff5b80feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:06.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.249+0000 7ff5bd30e640 1 -- 172.21.15.98:0/3640052075 shutdown_connections 2024-09-19T08:07:06.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:06.249+0000 7ff5bd30e640 1 -- 172.21.15.98:0/3640052075 wait complete. 2024-09-19T08:07:06.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:07:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:07.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:06 smithi139 ceph-mon[34874]: pgmap v126: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-09-19T08:07:07.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:06 smithi098 ceph-mon[31333]: pgmap v126: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-09-19T08:07:07.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:06 smithi162 ceph-mon[35260]: pgmap v126: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-09-19T08:07:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.560+0000 7fd82ba40640 1 -- 172.21.15.98:0/3817741973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8241082a0 msgr2=0x7fd82410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.560+0000 7fd82ba40640 1 --2- 172.21.15.98:0/3817741973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8241082a0 0x7fd82410ec40 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009930 tx=0x7fd81c02a420 comp rx=0 tx=0).stop 2024-09-19T08:07:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.560+0000 7fd82ba40640 1 -- 172.21.15.98:0/3817741973 shutdown_connections 2024-09-19T08:07:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.560+0000 7fd82ba40640 1 --2- 172.21.15.98:0/3817741973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8241082a0 0x7fd82410ec40 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.560+0000 7fd82ba40640 1 --2- 172.21.15.98:0/3817741973 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8241036e0 0x7fd824107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.560+0000 7fd82ba40640 1 --2- 172.21.15.98:0/3817741973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd824102d30 0x7fd824103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.560+0000 7fd82ba40640 1 -- 172.21.15.98:0/3817741973 >> 172.21.15.98:0/3817741973 conn(0x7fd8240fc7d0 msgr2=0x7fd8240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.560+0000 7fd82ba40640 1 -- 172.21.15.98:0/3817741973 shutdown_connections 2024-09-19T08:07:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.560+0000 7fd82ba40640 1 -- 172.21.15.98:0/3817741973 wait complete. 2024-09-19T08:07:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.560+0000 7fd82ba40640 1 Processor -- start 2024-09-19T08:07:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.560+0000 7fd82ba40640 1 -- start start 2024-09-19T08:07:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd82ba40640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd824102d30 0x7fd8241984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd82ba40640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8241036e0 0x7fd824198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd82ba40640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8241082a0 0x7fd82419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd82ba40640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8241120a0 con 0x7fd8241082a0 2024-09-19T08:07:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd82ba40640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd824111f20 con 0x7fd8241036e0 2024-09-19T08:07:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd82ba40640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd824112220 con 0x7fd824102d30 2024-09-19T08:07:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd829fdd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8241082a0 0x7fd82419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd829fdd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8241082a0 0x7fd82419cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34490/0 (socket says 172.21.15.98:34490) 2024-09-19T08:07:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd829fdd640 1 -- 172.21.15.98:0/2645886714 learned_addr learned my addr 172.21.15.98:0/2645886714 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd8297dc640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd824102d30 0x7fd8241984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd828fdb640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8241036e0 0x7fd824198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd829fdd640 1 -- 172.21.15.98:0/2645886714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd824102d30 msgr2=0x7fd8241984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd829fdd640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd824102d30 0x7fd8241984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd829fdd640 1 -- 172.21.15.98:0/2645886714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8241036e0 msgr2=0x7fd824198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd829fdd640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8241036e0 0x7fd824198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd829fdd640 1 -- 172.21.15.98:0/2645886714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd82419d480 con 0x7fd8241082a0 2024-09-19T08:07:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd828fdb640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8241036e0 0x7fd824198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd8297dc640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd824102d30 0x7fd8241984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd829fdd640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8241082a0 0x7fd82419cda0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd81c005ab0 tx=0x7fd81c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd8227fc640 1 -- 172.21.15.98:0/2645886714 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c03f030 con 0x7fd8241082a0 2024-09-19T08:07:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd8227fc640 1 -- 172.21.15.98:0/2645886714 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd81c003ce0 con 0x7fd8241082a0 2024-09-19T08:07:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd82ba40640 1 -- 172.21.15.98:0/2645886714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd82419d710 con 0x7fd8241082a0 2024-09-19T08:07:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd8227fc640 1 -- 172.21.15.98:0/2645886714 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c035070 con 0x7fd8241082a0 2024-09-19T08:07:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.561+0000 7fd82ba40640 1 -- 172.21.15.98:0/2645886714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8241a5050 con 0x7fd8241082a0 2024-09-19T08:07:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.564+0000 7fd8227fc640 1 -- 172.21.15.98:0/2645886714 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fd81c042030 con 0x7fd8241082a0 2024-09-19T08:07:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.564+0000 7fd82de19740 1 Processor -- start 2024-09-19T08:07:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.564+0000 7fd82de19740 1 -- start start 2024-09-19T08:07:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.564+0000 7fd82de19740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff7b40 0x560caeff6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.564+0000 7fd82de19740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff65d0 0x560caeff6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.564+0000 7fd82de19740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caf001370 0x560caf003760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.564+0000 7fd82de19740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560caef5c2f0 con 0x560caeff65d0 2024-09-19T08:07:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.564+0000 7fd82de19740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560caef03cf0 con 0x560caf001370 2024-09-19T08:07:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.564+0000 7fd82de19740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560caef582b0 con 0x560caeff7b40 2024-09-19T08:07:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.564+0000 7fd8297dc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff65d0 0x560caeff6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.564+0000 7fd8297dc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff65d0 0x560caeff6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34494/0 (socket says 172.21.15.98:34494) 2024-09-19T08:07:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.564+0000 7fd8297dc640 1 -- 172.21.15.98:0/3671346296 learned_addr learned my addr 172.21.15.98:0/3671346296 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd828fdb640 1 --2- 172.21.15.98:0/3671346296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff7b40 0x560caeff6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd8227fc640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd80c0a33b0 0x7fd80c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd828fdb640 1 --2- 172.21.15.98:0/3671346296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caf001370 0x560caf003760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd829fdd640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd80c0a33b0 0x7fd80c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd8297dc640 1 -- 172.21.15.98:0/3671346296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff7b40 msgr2=0x560caeff6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd8297dc640 1 --2- 172.21.15.98:0/3671346296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff7b40 0x560caeff6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd8297dc640 1 -- 172.21.15.98:0/3671346296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caf001370 msgr2=0x560caf003760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd8297dc640 1 --2- 172.21.15.98:0/3671346296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caf001370 0x560caf003760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd8297dc640 1 -- 172.21.15.98:0/3671346296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560caeee83a0 con 0x560caeff65d0 2024-09-19T08:07:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd8227fc640 1 -- 172.21.15.98:0/2645886714 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fd81c0f0340 con 0x7fd8241082a0 2024-09-19T08:07:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd828fdb640 1 --2- 172.21.15.98:0/3671346296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caf001370 0x560caf003760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd8297dc640 1 --2- 172.21.15.98:0/3671346296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff65d0 0x560caeff6a10 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd814007f70 tx=0x7fd81400e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb01c1a734d8584 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd829fdd640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd80c0a33b0 0x7fd80c0a5870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd81c0c4d90 tx=0x7fd81c08e3d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd80b7fe640 1 -- 172.21.15.98:0/3671346296 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814014070 con 0x560caeff65d0 2024-09-19T08:07:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd80b7fe640 1 -- 172.21.15.98:0/3671346296 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd814004d20 con 0x560caeff65d0 2024-09-19T08:07:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.565+0000 7fd80b7fe640 1 -- 172.21.15.98:0/3671346296 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814005520 con 0x560caeff65d0 2024-09-19T08:07:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.566+0000 7fd82de19740 1 -- 172.21.15.98:0/3671346296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff65d0 msgr2=0x560caeff6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.566+0000 7fd82de19740 1 --2- 172.21.15.98:0/3671346296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff65d0 0x560caeff6a10 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd814007f70 tx=0x7fd81400e7e0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.566+0000 7fd82de19740 1 -- 172.21.15.98:0/3671346296 shutdown_connections 2024-09-19T08:07:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.566+0000 7fd82de19740 1 --2- 172.21.15.98:0/3671346296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caf001370 0x560caf003760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.566+0000 7fd82de19740 1 --2- 172.21.15.98:0/3671346296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff65d0 0x560caeff6a10 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.566+0000 7fd82de19740 1 --2- 172.21.15.98:0/3671346296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff7b40 0x560caeff6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.566+0000 7fd82de19740 1 -- 172.21.15.98:0/3671346296 >> 172.21.15.98:0/3671346296 conn(0x560caefbf310 msgr2=0x560caeff2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.566+0000 7fd82de19740 1 -- 172.21.15.98:0/3671346296 shutdown_connections 2024-09-19T08:07:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.566+0000 7fd82de19740 1 -- 172.21.15.98:0/3671346296 wait complete. 2024-09-19T08:07:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.566+0000 7fd82de19740 1 -- start start 2024-09-19T08:07:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd82de19740 1 Processor -- start 2024-09-19T08:07:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd82de19740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff65d0 0x560caf000d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd82de19740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff7b40 0x560caeffe030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd82de19740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caf001370 0x560caeffe640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd82de19740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560caf004070 con 0x560caeff65d0 2024-09-19T08:07:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd82de19740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560caef17060 con 0x560caf001370 2024-09-19T08:07:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd82de19740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560caf0041f0 con 0x560caeff7b40 2024-09-19T08:07:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd82de19740 10 client.0 _ll_register_callbacks cb 0x560caec6a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd828fdb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff65d0 0x560caf000d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd828fdb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff65d0 0x560caf000d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34504/0 (socket says 172.21.15.98:34504) 2024-09-19T08:07:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd828fdb640 1 -- 172.21.15.98:0/336044403 learned_addr learned my addr 172.21.15.98:0/336044403 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd8297dc640 1 --2- 172.21.15.98:0/336044403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff7b40 0x560caeffe030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd828fdb640 1 -- 172.21.15.98:0/336044403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff7b40 msgr2=0x560caeffe030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd828fdb640 1 --2- 172.21.15.98:0/336044403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff7b40 0x560caeffe030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd828fdb640 1 -- 172.21.15.98:0/336044403 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caf001370 msgr2=0x560caeffe640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd828fdb640 1 --2- 172.21.15.98:0/336044403 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caf001370 0x560caeffe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd828fdb640 1 -- 172.21.15.98:0/336044403 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560caeccd3c0 con 0x560caeff65d0 2024-09-19T08:07:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.567+0000 7fd8297dc640 1 --2- 172.21.15.98:0/336044403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff7b40 0x560caeffe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd828fdb640 1 --2- 172.21.15.98:0/336044403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff65d0 0x560caf000d80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fd8240697d0 tx=0x7fd818010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df698c6a9760cd96 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd808ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd808ff9640 1 -- 172.21.15.98:0/336044403 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81801e070 con 0x560caeff65d0 2024-09-19T08:07:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd808ff9640 1 -- 172.21.15.98:0/336044403 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fd818004020 con 0x560caeff65d0 2024-09-19T08:07:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd82de19740 1 -- 172.21.15.98:0/336044403 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd814004d20 con 0x560caeff65d0 2024-09-19T08:07:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd808ff9640 1 -- 172.21.15.98:0/336044403 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd818029050 con 0x560caeff65d0 2024-09-19T08:07:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd82de19740 10 client.16566 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd82de19740 1 -- 172.21.15.98:0/336044403 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560caf00fba0 con 0x560caeff65d0 2024-09-19T08:07:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd808ff9640 1 -- 172.21.15.98:0/336044403 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd814004d20 con 0x560caeff65d0 2024-09-19T08:07:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd808ff9640 1 -- 172.21.15.98:0/336044403 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fd818003bd0 con 0x560caeff65d0 2024-09-19T08:07:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd82de19740 10 client.16566 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd82de19740 10 client.16566 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:07:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd82de19740 20 client.16566 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd82de19740 1 --2- 172.21.15.98:0/336044403 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560caf010fb0 0x560caf0133e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd82de19740 4 client.16566 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.568+0000 7fd82de19740 1 -- 172.21.15.98:0/336044403 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560caef69860 con 0x560caf010fb0 2024-09-19T08:07:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.569+0000 7fd8297dc640 1 --2- 172.21.15.98:0/336044403 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560caf010fb0 0x560caf0133e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.569+0000 7fd8297dc640 1 --2- 172.21.15.98:0/336044403 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560caf010fb0 0x560caf0133e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b7bd748f0f6ba7ff server_cookie=afc581c4e082136f in_seq=0 out_seq=0 2024-09-19T08:07:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.569+0000 7fd808ff9640 10 client.16566 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:07:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd808ff9640 1 -- 172.21.15.98:0/336044403 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560caef69860 con 0x560caf010fb0 2024-09-19T08:07:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd808ff9640 10 client.16566 handle_command_reply: tid=0 2024-09-19T08:07:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd82de19740 1 client.16566 shutdown 2024-09-19T08:07:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd82de19740 1 -- 172.21.15.98:0/336044403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff65d0 msgr2=0x560caf000d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd82de19740 1 --2- 172.21.15.98:0/336044403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff65d0 0x560caf000d80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fd8240697d0 tx=0x7fd818010800 comp rx=0 tx=0).stop 2024-09-19T08:07:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd82de19740 1 -- 172.21.15.98:0/336044403 shutdown_connections 2024-09-19T08:07:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd82de19740 1 --2- 172.21.15.98:0/336044403 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560caf010fb0 0x560caf0133e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd82de19740 1 --2- 172.21.15.98:0/336044403 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caf001370 0x560caeffe640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd82de19740 1 --2- 172.21.15.98:0/336044403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff7b40 0x560caeffe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd82de19740 1 --2- 172.21.15.98:0/336044403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff65d0 0x560caf000d80 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd808ff9640 0 client.16566 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:07:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd82de19740 1 -- 172.21.15.98:0/336044403 >> 172.21.15.98:0/336044403 conn(0x560caefbf310 msgr2=0x560caeff8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd82de19740 1 -- 172.21.15.98:0/336044403 shutdown_connections 2024-09-19T08:07:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd82de19740 1 -- 172.21.15.98:0/336044403 wait complete. 2024-09-19T08:07:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.570+0000 7fd82de19740 20 client.16566 trim_cache size 0 max 16384 2024-09-19T08:07:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.583+0000 7fd82de19740 1 Processor -- start 2024-09-19T08:07:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.583+0000 7fd82de19740 1 -- start start 2024-09-19T08:07:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.583+0000 7fd82de19740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff65d0 0x560caeff1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.583+0000 7fd82de19740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caeff7b40 0x560caeff2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.583+0000 7fd82de19740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caf001370 0x560caeff9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.583+0000 7fd82de19740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8241ab640 con 0x560caf001370 2024-09-19T08:07:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.583+0000 7fd82de19740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd8241ad1e0 con 0x560caeff7b40 2024-09-19T08:07:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.583+0000 7fd82de19740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd8241abd10 con 0x560caeff65d0 2024-09-19T08:07:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.583+0000 7fd828fdb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caf001370 0x560caeff9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.583+0000 7fd828fdb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caf001370 0x560caeff9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34518/0 (socket says 172.21.15.98:34518) 2024-09-19T08:07:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.583+0000 7fd828fdb640 1 -- 172.21.15.98:0/1861912685 learned_addr learned my addr 172.21.15.98:0/1861912685 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.584+0000 7fd828fdb640 1 --2- 172.21.15.98:0/1861912685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff65d0 0x560caeff1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.584+0000 7fd828fdb640 1 -- 172.21.15.98:0/1861912685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff65d0 msgr2=0x560caeff1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.584+0000 7fd828fdb640 1 --2- 172.21.15.98:0/1861912685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff65d0 0x560caeff1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.584+0000 7fd828fdb640 1 -- 172.21.15.98:0/1861912685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caeff7b40 msgr2=0x560caeff2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.584+0000 7fd828fdb640 1 --2- 172.21.15.98:0/1861912685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caeff7b40 0x560caeff2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.584+0000 7fd828fdb640 1 -- 172.21.15.98:0/1861912685 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560caf013a60 con 0x560caf001370 2024-09-19T08:07:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.584+0000 7fd828fdb640 1 --2- 172.21.15.98:0/1861912685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caf001370 0x560caeff9120 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd818010fd0 tx=0x7fd818005c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18364fa0ca9e18a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.584+0000 7fd80affd640 1 -- 172.21.15.98:0/1861912685 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818031020 con 0x560caf001370 2024-09-19T08:07:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.584+0000 7fd80affd640 1 -- 172.21.15.98:0/1861912685 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd81801a640 con 0x560caf001370 2024-09-19T08:07:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.584+0000 7fd80affd640 1 -- 172.21.15.98:0/1861912685 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81801e070 con 0x560caf001370 2024-09-19T08:07:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.585+0000 7fd82de19740 1 -- 172.21.15.98:0/1861912685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caf001370 msgr2=0x560caeff9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.585+0000 7fd82de19740 1 --2- 172.21.15.98:0/1861912685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caf001370 0x560caeff9120 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd818010fd0 tx=0x7fd818005c60 comp rx=0 tx=0).stop 2024-09-19T08:07:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.585+0000 7fd82de19740 1 -- 172.21.15.98:0/1861912685 shutdown_connections 2024-09-19T08:07:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.585+0000 7fd82de19740 1 --2- 172.21.15.98:0/1861912685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caf001370 0x560caeff9120 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.585+0000 7fd82de19740 1 --2- 172.21.15.98:0/1861912685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caeff7b40 0x560caeff2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.585+0000 7fd82de19740 1 --2- 172.21.15.98:0/1861912685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caeff65d0 0x560caeff1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.585+0000 7fd82de19740 1 -- 172.21.15.98:0/1861912685 >> 172.21.15.98:0/1861912685 conn(0x560caefbf310 msgr2=0x560caeff14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.585+0000 7fd82de19740 1 -- 172.21.15.98:0/1861912685 shutdown_connections 2024-09-19T08:07:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.585+0000 7fd82de19740 1 -- 172.21.15.98:0/1861912685 wait complete. 2024-09-19T08:07:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.585+0000 7fd82de19740 1 -- start start 2024-09-19T08:07:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.585+0000 7fd82de19740 1 Processor -- start 2024-09-19T08:07:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd82de19740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caeff65d0 0x560caf009ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd82de19740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff7b40 0x560caeffacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd82de19740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caf001370 0x560caeffb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd82de19740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8241acae0 con 0x560caeff7b40 2024-09-19T08:07:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd82de19740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd8241ac410 con 0x560caeff65d0 2024-09-19T08:07:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd82de19740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd8241a9620 con 0x560caf001370 2024-09-19T08:07:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd82de19740 10 client.0 _ll_register_callbacks cb 0x7fd81800e350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd8297dc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff7b40 0x560caeffacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd8297dc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff7b40 0x560caeffacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34528/0 (socket says 172.21.15.98:34528) 2024-09-19T08:07:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd8297dc640 1 -- 172.21.15.98:0/3730681105 learned_addr learned my addr 172.21.15.98:0/3730681105 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd828fdb640 1 --2- 172.21.15.98:0/3730681105 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caeff65d0 0x560caf009ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd8297dc640 1 -- 172.21.15.98:0/3730681105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caf001370 msgr2=0x560caeffb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd828fdb640 1 --2- 172.21.15.98:0/3730681105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caf001370 0x560caeffb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd8297dc640 1 --2- 172.21.15.98:0/3730681105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caf001370 0x560caeffb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd8297dc640 1 -- 172.21.15.98:0/3730681105 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caeff65d0 msgr2=0x560caf009ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd8297dc640 1 --2- 172.21.15.98:0/3730681105 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caeff65d0 0x560caf009ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd8297dc640 1 -- 172.21.15.98:0/3730681105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560caeffb940 con 0x560caeff7b40 2024-09-19T08:07:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd828fdb640 1 --2- 172.21.15.98:0/3730681105 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caeff65d0 0x560caf009ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd8297dc640 1 --2- 172.21.15.98:0/3730681105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff7b40 0x560caeffacc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd814007d00 tx=0x7fd814007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d4de673557424c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd808ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd808ff9640 1 -- 172.21.15.98:0/3730681105 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814014050 con 0x560caeff7b40 2024-09-19T08:07:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd808ff9640 1 -- 172.21.15.98:0/3730681105 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fd814010640 con 0x560caeff7b40 2024-09-19T08:07:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd82de19740 1 -- 172.21.15.98:0/3730681105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd81801a640 con 0x560caeff7b40 2024-09-19T08:07:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd808ff9640 1 -- 172.21.15.98:0/3730681105 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd814010840 con 0x560caeff7b40 2024-09-19T08:07:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd82de19740 10 client.16572 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.586+0000 7fd82de19740 1 -- 172.21.15.98:0/3730681105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560caf07bb20 con 0x560caeff7b40 2024-09-19T08:07:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.587+0000 7fd808ff9640 1 -- 172.21.15.98:0/3730681105 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd81801a640 con 0x560caeff7b40 2024-09-19T08:07:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.587+0000 7fd808ff9640 1 -- 172.21.15.98:0/3730681105 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fd814018410 con 0x560caeff7b40 2024-09-19T08:07:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.587+0000 7fd82de19740 10 client.16572 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.587+0000 7fd82de19740 10 client.16572 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:07:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.587+0000 7fd82de19740 20 client.16572 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.587+0000 7fd82de19740 1 --2- 172.21.15.98:0/3730681105 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560caf07c060 0x560caf07e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.587+0000 7fd82de19740 4 client.16572 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:07:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.587+0000 7fd82de19740 1 -- 172.21.15.98:0/3730681105 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560caef63d70 con 0x560caf07c060 2024-09-19T08:07:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.587+0000 7fd828fdb640 1 --2- 172.21.15.98:0/3730681105 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560caf07c060 0x560caf07e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd828fdb640 1 --2- 172.21.15.98:0/3730681105 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560caf07c060 0x560caf07e480 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=37ee889bbadc0dc4 server_cookie=9d1d6a6ef7f9fdca in_seq=0 out_seq=0 2024-09-19T08:07:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd808ff9640 10 client.16572 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:07:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd808ff9640 1 -- 172.21.15.98:0/3730681105 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560caef63d70 con 0x560caf07c060 2024-09-19T08:07:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd808ff9640 10 client.16572 handle_command_reply: tid=0 2024-09-19T08:07:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd82de19740 1 client.16572 shutdown 2024-09-19T08:07:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd82de19740 1 -- 172.21.15.98:0/3730681105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff7b40 msgr2=0x560caeffacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd82de19740 1 --2- 172.21.15.98:0/3730681105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff7b40 0x560caeffacc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd814007d00 tx=0x7fd814007680 comp rx=0 tx=0).stop 2024-09-19T08:07:07.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd82de19740 1 -- 172.21.15.98:0/3730681105 shutdown_connections 2024-09-19T08:07:07.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd82de19740 1 --2- 172.21.15.98:0/3730681105 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x560caf07c060 0x560caf07e480 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd82de19740 1 --2- 172.21.15.98:0/3730681105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560caf001370 0x560caeffb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd82de19740 1 --2- 172.21.15.98:0/3730681105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560caeff7b40 0x560caeffacc0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd808ff9640 0 client.16572 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:07:07.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd82de19740 1 --2- 172.21.15.98:0/3730681105 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560caeff65d0 0x560caf009ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd82de19740 1 -- 172.21.15.98:0/3730681105 >> 172.21.15.98:0/3730681105 conn(0x560caefbf310 msgr2=0x560caeff1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:07.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd82de19740 1 -- 172.21.15.98:0/3730681105 shutdown_connections 2024-09-19T08:07:07.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd82de19740 1 -- 172.21.15.98:0/3730681105 wait complete. 2024-09-19T08:07:07.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.588+0000 7fd82de19740 20 client.16572 trim_cache size 0 max 16384 2024-09-19T08:07:07.618 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:07:07.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.589+0000 7fd82ba40640 1 -- 172.21.15.98:0/2645886714 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd80c0a33b0 msgr2=0x7fd80c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:07.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.589+0000 7fd82ba40640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd80c0a33b0 0x7fd80c0a5870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd81c0c4d90 tx=0x7fd81c08e3d0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.589+0000 7fd82ba40640 1 -- 172.21.15.98:0/2645886714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8241082a0 msgr2=0x7fd82419cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:07.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.589+0000 7fd82ba40640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8241082a0 0x7fd82419cda0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd81c005ab0 tx=0x7fd81c0042f0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.589+0000 7fd82ba40640 1 -- 172.21.15.98:0/2645886714 shutdown_connections 2024-09-19T08:07:07.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.589+0000 7fd82ba40640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd80c0a33b0 0x7fd80c0a5870 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.589+0000 7fd82ba40640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8241082a0 0x7fd82419cda0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.589+0000 7fd82ba40640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8241036e0 0x7fd824198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.589+0000 7fd82ba40640 1 --2- 172.21.15.98:0/2645886714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd824102d30 0x7fd8241984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:07.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.589+0000 7fd82ba40640 1 -- 172.21.15.98:0/2645886714 >> 172.21.15.98:0/2645886714 conn(0x7fd8240fc7d0 msgr2=0x7fd824106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:07.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.589+0000 7fd82ba40640 1 -- 172.21.15.98:0/2645886714 shutdown_connections 2024-09-19T08:07:07.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:07.589+0000 7fd82ba40640 1 -- 172.21.15.98:0/2645886714 wait complete. 2024-09-19T08:07:08.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:07:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:09.106 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:08 smithi098 ceph-mon[31333]: pgmap v127: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-09-19T08:07:09.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.111+0000 7f9f49468640 1 -- 172.21.15.98:0/472593719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f441036e0 msgr2=0x7f9f44107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:09.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.111+0000 7f9f49468640 1 --2- 172.21.15.98:0/472593719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f441036e0 0x7f9f44107b90 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009930 tx=0x7f9f3802a420 comp rx=0 tx=0).stop 2024-09-19T08:07:09.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.111+0000 7f9f49468640 1 -- 172.21.15.98:0/472593719 shutdown_connections 2024-09-19T08:07:09.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.111+0000 7f9f49468640 1 --2- 172.21.15.98:0/472593719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f441082a0 0x7f9f4410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.111+0000 7f9f49468640 1 --2- 172.21.15.98:0/472593719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f441036e0 0x7f9f44107b90 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.111+0000 7f9f49468640 1 --2- 172.21.15.98:0/472593719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f44102d30 0x7f9f44103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.111+0000 7f9f49468640 1 -- 172.21.15.98:0/472593719 >> 172.21.15.98:0/472593719 conn(0x7f9f440fc7d0 msgr2=0x7f9f440febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:09.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.111+0000 7f9f49468640 1 -- 172.21.15.98:0/472593719 shutdown_connections 2024-09-19T08:07:09.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.111+0000 7f9f49468640 1 -- 172.21.15.98:0/472593719 wait complete. 2024-09-19T08:07:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.112+0000 7f9f49468640 1 Processor -- start 2024-09-19T08:07:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.112+0000 7f9f49468640 1 -- start start 2024-09-19T08:07:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.112+0000 7f9f49468640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f44102d30 0x7f9f44100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.112+0000 7f9f49468640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f441036e0 0x7f9f44100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.112+0000 7f9f49468640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f441082a0 0x7f9f44101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.112+0000 7f9f49468640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9f441120c0 con 0x7f9f441082a0 2024-09-19T08:07:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.112+0000 7f9f49468640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9f44111f40 con 0x7f9f44102d30 2024-09-19T08:07:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.112+0000 7f9f49468640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9f44112240 con 0x7f9f441036e0 2024-09-19T08:07:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f441082a0 0x7f9f44101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f427fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f441036e0 0x7f9f44100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f441082a0 0x7f9f44101410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34552/0 (socket says 172.21.15.98:34552) 2024-09-19T08:07:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f427fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f441036e0 0x7f9f44100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:47184/0 (socket says 172.21.15.98:47184) 2024-09-19T08:07:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f437fe640 1 -- 172.21.15.98:0/3198622806 learned_addr learned my addr 172.21.15.98:0/3198622806 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f42ffd640 1 --2- 172.21.15.98:0/3198622806 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f44102d30 0x7f9f44100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f437fe640 1 -- 172.21.15.98:0/3198622806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f441036e0 msgr2=0x7f9f44100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f437fe640 1 --2- 172.21.15.98:0/3198622806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f441036e0 0x7f9f44100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f437fe640 1 -- 172.21.15.98:0/3198622806 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f44102d30 msgr2=0x7f9f44100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f437fe640 1 --2- 172.21.15.98:0/3198622806 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f44102d30 0x7f9f44100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f437fe640 1 -- 172.21.15.98:0/3198622806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f441a6ae0 con 0x7f9f441082a0 2024-09-19T08:07:09.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f437fe640 1 --2- 172.21.15.98:0/3198622806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f441082a0 0x7f9f44101410 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c009fd0 tx=0x7f9f3c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:09.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f2bfff640 1 -- 172.21.15.98:0/3198622806 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c011830 con 0x7f9f441082a0 2024-09-19T08:07:09.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f49468640 1 -- 172.21.15.98:0/3198622806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f441a6dd0 con 0x7f9f441082a0 2024-09-19T08:07:09.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f2bfff640 1 -- 172.21.15.98:0/3198622806 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f3c016070 con 0x7f9f441082a0 2024-09-19T08:07:09.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f2bfff640 1 -- 172.21.15.98:0/3198622806 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c00f2e0 con 0x7f9f441082a0 2024-09-19T08:07:09.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.113+0000 7f9f49468640 1 -- 172.21.15.98:0/3198622806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f441a7290 con 0x7f9f441082a0 2024-09-19T08:07:09.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.116+0000 7f9f2bfff640 1 -- 172.21.15.98:0/3198622806 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f9f3c0119d0 con 0x7f9f441082a0 2024-09-19T08:07:09.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.116+0000 7f9f4b841740 1 Processor -- start 2024-09-19T08:07:09.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.116+0000 7f9f4b841740 1 -- start start 2024-09-19T08:07:09.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f4b841740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538abab40 0x565538ab9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f4b841740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538ab95d0 0x565538ab9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f4b841740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ac4370 0x565538ac6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f4b841740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x565538a1f2f0 con 0x565538abab40 2024-09-19T08:07:09.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f4b841740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5655389c6cf0 con 0x565538ac4370 2024-09-19T08:07:09.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f4b841740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x565538a1b2b0 con 0x565538ab95d0 2024-09-19T08:07:09.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f427fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538abab40 0x565538ab9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f427fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538abab40 0x565538ab9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34564/0 (socket says 172.21.15.98:34564) 2024-09-19T08:07:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f427fc640 1 -- 172.21.15.98:0/1553343142 learned_addr learned my addr 172.21.15.98:0/1553343142 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f2bfff640 1 --2- 172.21.15.98:0/3198622806 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9f2c0a3430 0x7f9f2c0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f42ffd640 1 --2- 172.21.15.98:0/1553343142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538ab95d0 0x565538ab9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f427fc640 1 --2- 172.21.15.98:0/1553343142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ac4370 0x565538ac6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f437fe640 1 --2- 172.21.15.98:0/3198622806 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9f2c0a3430 0x7f9f2c0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f42ffd640 1 -- 172.21.15.98:0/1553343142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ac4370 msgr2=0x565538ac6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f42ffd640 1 --2- 172.21.15.98:0/1553343142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ac4370 0x565538ac6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f42ffd640 1 -- 172.21.15.98:0/1553343142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538abab40 msgr2=0x565538ab9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f42ffd640 1 --2- 172.21.15.98:0/1553343142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538abab40 0x565538ab9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f2bfff640 1 -- 172.21.15.98:0/3198622806 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f9f3c0cf0c0 con 0x7f9f441082a0 2024-09-19T08:07:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f42ffd640 1 -- 172.21.15.98:0/1553343142 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5655389ab3a0 con 0x565538ab95d0 2024-09-19T08:07:09.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.117+0000 7f9f427fc640 1 --2- 172.21.15.98:0/1553343142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538abab40 0x565538ab9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:07:09.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.118+0000 7f9f42ffd640 1 --2- 172.21.15.98:0/1553343142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538ab95d0 0x565538ab9a10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009fb0 tx=0x7f9f34003d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4a437273fb4f4ff server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:09.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.118+0000 7f9f437fe640 1 --2- 172.21.15.98:0/3198622806 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9f2c0a3430 0x7f9f2c0a58f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c0ccaa0 tx=0x7f9f3c0047d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:09.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.118+0000 7f9f297fa640 1 -- 172.21.15.98:0/1553343142 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f34004820 con 0x565538ab95d0 2024-09-19T08:07:09.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.118+0000 7f9f297fa640 1 -- 172.21.15.98:0/1553343142 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f34005020 con 0x565538ab95d0 2024-09-19T08:07:09.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.118+0000 7f9f297fa640 1 -- 172.21.15.98:0/1553343142 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f340126b0 con 0x565538ab95d0 2024-09-19T08:07:09.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.118+0000 7f9f4b841740 1 -- 172.21.15.98:0/1553343142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538ab95d0 msgr2=0x565538ab9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:09.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.118+0000 7f9f4b841740 1 --2- 172.21.15.98:0/1553343142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538ab95d0 0x565538ab9a10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009fb0 tx=0x7f9f34003d40 comp rx=0 tx=0).stop 2024-09-19T08:07:09.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 -- 172.21.15.98:0/1553343142 shutdown_connections 2024-09-19T08:07:09.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 --2- 172.21.15.98:0/1553343142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ac4370 0x565538ac6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 --2- 172.21.15.98:0/1553343142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538ab95d0 0x565538ab9a10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 --2- 172.21.15.98:0/1553343142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538abab40 0x565538ab9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 -- 172.21.15.98:0/1553343142 >> 172.21.15.98:0/1553343142 conn(0x565538a82310 msgr2=0x565538ab5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:09.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 -- 172.21.15.98:0/1553343142 shutdown_connections 2024-09-19T08:07:09.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 -- 172.21.15.98:0/1553343142 wait complete. 2024-09-19T08:07:09.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 -- start start 2024-09-19T08:07:09.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 Processor -- start 2024-09-19T08:07:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538ab95d0 0x565538ac3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538abab40 0x565538ac0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ac4370 0x565538ac14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x565538ac7070 con 0x565538ab95d0 2024-09-19T08:07:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5655389da060 con 0x565538ac4370 2024-09-19T08:07:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x565538ac71f0 con 0x565538abab40 2024-09-19T08:07:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f4b841740 10 client.0 _ll_register_callbacks cb 0x56553872d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.119+0000 7f9f427fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538ab95d0 0x565538ac3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f42ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538abab40 0x565538ac0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f427fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538ab95d0 0x565538ac3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34576/0 (socket says 172.21.15.98:34576) 2024-09-19T08:07:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f427fc640 1 -- 172.21.15.98:0/1892793417 learned_addr learned my addr 172.21.15.98:0/1892793417 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:09.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f427fc640 1 -- 172.21.15.98:0/1892793417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538abab40 msgr2=0x565538ac0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f427fc640 1 --2- 172.21.15.98:0/1892793417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538abab40 0x565538ac0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f427fc640 1 -- 172.21.15.98:0/1892793417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ac4370 msgr2=0x565538ac14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f427fc640 1 --2- 172.21.15.98:0/1892793417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ac4370 0x565538ac14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f427fc640 1 -- 172.21.15.98:0/1892793417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5655387903c0 con 0x565538ab95d0 2024-09-19T08:07:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f427fc640 1 --2- 172.21.15.98:0/1892793417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538ab95d0 0x565538ac3cb0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009230 tx=0x7f9f380092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9af288d8e8a3cf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f26ff5640 1 -- 172.21.15.98:0/1892793417 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f38044030 con 0x565538ab95d0 2024-09-19T08:07:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f26ff5640 1 -- 172.21.15.98:0/1892793417 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f9f38005020 con 0x565538ab95d0 2024-09-19T08:07:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f4b841740 1 -- 172.21.15.98:0/1892793417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f34005020 con 0x565538ab95d0 2024-09-19T08:07:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f26ff5640 1 -- 172.21.15.98:0/1892793417 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9f38035040 con 0x565538ab95d0 2024-09-19T08:07:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f4b841740 10 client.16584 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.120+0000 7f9f4b841740 1 -- 172.21.15.98:0/1892793417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x565538ad2a90 con 0x565538ab95d0 2024-09-19T08:07:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.121+0000 7f9f26ff5640 1 -- 172.21.15.98:0/1892793417 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f34005020 con 0x565538ab95d0 2024-09-19T08:07:09.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.121+0000 7f9f26ff5640 1 -- 172.21.15.98:0/1892793417 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f9f3803b460 con 0x565538ab95d0 2024-09-19T08:07:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.121+0000 7f9f4b841740 10 client.16584 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.121+0000 7f9f4b841740 10 client.16584 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:07:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.121+0000 7f9f4b841740 20 client.16584 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.121+0000 7f9f4b841740 1 --2- 172.21.15.98:0/1892793417 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x565538ad3ea0 0x565538ad6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.121+0000 7f9f4b841740 4 client.16584 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.121+0000 7f9f4b841740 1 -- 172.21.15.98:0/1892793417 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565538a2c860 con 0x565538ad3ea0 2024-09-19T08:07:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.121+0000 7f9f42ffd640 1 --2- 172.21.15.98:0/1892793417 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x565538ad3ea0 0x565538ad6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.121+0000 7f9f42ffd640 1 --2- 172.21.15.98:0/1892793417 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x565538ad3ea0 0x565538ad6290 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a3e6730dcd7fdfb6 server_cookie=483262d827266b17 in_seq=0 out_seq=0 2024-09-19T08:07:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.121+0000 7f9f26ff5640 10 client.16584 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:07:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.122+0000 7f9f26ff5640 1 -- 172.21.15.98:0/1892793417 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x565538a2c860 con 0x565538ad3ea0 2024-09-19T08:07:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.122+0000 7f9f26ff5640 10 client.16584 handle_command_reply: tid=0 2024-09-19T08:07:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.122+0000 7f9f4b841740 1 client.16584 shutdown 2024-09-19T08:07:09.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.122+0000 7f9f4b841740 1 -- 172.21.15.98:0/1892793417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538ab95d0 msgr2=0x565538ac3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:09.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.122+0000 7f9f4b841740 1 --2- 172.21.15.98:0/1892793417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538ab95d0 0x565538ac3cb0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009230 tx=0x7f9f380092d0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.122+0000 7f9f4b841740 1 -- 172.21.15.98:0/1892793417 shutdown_connections 2024-09-19T08:07:09.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.122+0000 7f9f4b841740 1 --2- 172.21.15.98:0/1892793417 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x565538ad3ea0 0x565538ad6290 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.122+0000 7f9f4b841740 1 --2- 172.21.15.98:0/1892793417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ac4370 0x565538ac14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.122+0000 7f9f4b841740 1 --2- 172.21.15.98:0/1892793417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538abab40 0x565538ac0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.122+0000 7f9f4b841740 1 --2- 172.21.15.98:0/1892793417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538ab95d0 0x565538ac3cb0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.122+0000 7f9f4b841740 1 -- 172.21.15.98:0/1892793417 >> 172.21.15.98:0/1892793417 conn(0x565538a82310 msgr2=0x565538abb930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:09.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.122+0000 7f9f26ff5640 0 client.16584 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:07:09.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.123+0000 7f9f4b841740 1 -- 172.21.15.98:0/1892793417 shutdown_connections 2024-09-19T08:07:09.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.123+0000 7f9f4b841740 1 -- 172.21.15.98:0/1892793417 wait complete. 2024-09-19T08:07:09.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.123+0000 7f9f4b841740 20 client.16584 trim_cache size 0 max 16384 2024-09-19T08:07:09.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f4b841740 1 Processor -- start 2024-09-19T08:07:09.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f4b841740 1 -- start start 2024-09-19T08:07:09.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f4b841740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ab95d0 0x565538ab4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f4b841740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538abab40 0x565538ac5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f4b841740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538ac4370 0x565538ac5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f4b841740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9f441afaa0 con 0x565538abab40 2024-09-19T08:07:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f4b841740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9f441b1640 con 0x565538ab95d0 2024-09-19T08:07:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f4b841740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9f441b0170 con 0x565538ac4370 2024-09-19T08:07:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f42ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538abab40 0x565538ac5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f42ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538abab40 0x565538ac5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34578/0 (socket says 172.21.15.98:34578) 2024-09-19T08:07:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f42ffd640 1 -- 172.21.15.98:0/2410439534 learned_addr learned my addr 172.21.15.98:0/2410439534 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f427fc640 1 --2- 172.21.15.98:0/2410439534 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ab95d0 0x565538ab4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f427fc640 1 --2- 172.21.15.98:0/2410439534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538ac4370 0x565538ac5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f42ffd640 1 -- 172.21.15.98:0/2410439534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538ac4370 msgr2=0x565538ac5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f42ffd640 1 --2- 172.21.15.98:0/2410439534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538ac4370 0x565538ac5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f42ffd640 1 -- 172.21.15.98:0/2410439534 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ab95d0 msgr2=0x565538ab4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f42ffd640 1 --2- 172.21.15.98:0/2410439534 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ab95d0 0x565538ab4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f42ffd640 1 -- 172.21.15.98:0/2410439534 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565538ac37d0 con 0x565538abab40 2024-09-19T08:07:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f427fc640 1 --2- 172.21.15.98:0/2410439534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538ac4370 0x565538ac5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f42ffd640 1 --2- 172.21.15.98:0/2410439534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538abab40 0x565538ac5760 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009fb0 tx=0x7f9f34005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779d653f9e5085cb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f28ff9640 1 -- 172.21.15.98:0/2410439534 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f34012450 con 0x565538abab40 2024-09-19T08:07:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f28ff9640 1 -- 172.21.15.98:0/2410439534 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f34004440 con 0x565538abab40 2024-09-19T08:07:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.136+0000 7f9f28ff9640 1 -- 172.21.15.98:0/2410439534 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f34028890 con 0x565538abab40 2024-09-19T08:07:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 -- 172.21.15.98:0/2410439534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538abab40 msgr2=0x565538ac5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 --2- 172.21.15.98:0/2410439534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538abab40 0x565538ac5760 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009fb0 tx=0x7f9f34005970 comp rx=0 tx=0).stop 2024-09-19T08:07:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 -- 172.21.15.98:0/2410439534 shutdown_connections 2024-09-19T08:07:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 --2- 172.21.15.98:0/2410439534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538ac4370 0x565538ac5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 --2- 172.21.15.98:0/2410439534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538abab40 0x565538ac5760 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 --2- 172.21.15.98:0/2410439534 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ab95d0 0x565538ab4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 -- 172.21.15.98:0/2410439534 >> 172.21.15.98:0/2410439534 conn(0x565538a82310 msgr2=0x565538ab4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 -- 172.21.15.98:0/2410439534 shutdown_connections 2024-09-19T08:07:09.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 -- 172.21.15.98:0/2410439534 wait complete. 2024-09-19T08:07:09.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 -- start start 2024-09-19T08:07:09.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 Processor -- start 2024-09-19T08:07:09.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ab95d0 0x565538accd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538abab40 0x565538adb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538ac4370 0x565538adbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9f441b0f40 con 0x565538ac4370 2024-09-19T08:07:09.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9f441b0870 con 0x565538ab95d0 2024-09-19T08:07:09.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9f441adaa0 con 0x565538abab40 2024-09-19T08:07:09.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.137+0000 7f9f4b841740 10 client.0 _ll_register_callbacks cb 0x7f9f38032d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:09.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f427fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538ac4370 0x565538adbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f427fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538ac4370 0x565538adbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34588/0 (socket says 172.21.15.98:34588) 2024-09-19T08:07:09.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f427fc640 1 -- 172.21.15.98:0/876296177 learned_addr learned my addr 172.21.15.98:0/876296177 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:09.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f427fc640 1 --2- 172.21.15.98:0/876296177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ab95d0 0x565538accd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f42ffd640 1 --2- 172.21.15.98:0/876296177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538abab40 0x565538adb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f427fc640 1 -- 172.21.15.98:0/876296177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538abab40 msgr2=0x565538adb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:09.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f427fc640 1 --2- 172.21.15.98:0/876296177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538abab40 0x565538adb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f427fc640 1 -- 172.21.15.98:0/876296177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ab95d0 msgr2=0x565538accd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:09.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f427fc640 1 --2- 172.21.15.98:0/876296177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ab95d0 0x565538accd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f427fc640 1 -- 172.21.15.98:0/876296177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565538adc340 con 0x565538ac4370 2024-09-19T08:07:09.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f427fc640 1 --2- 172.21.15.98:0/876296177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538ac4370 0x565538adbb10 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9f3803b9c0 tx=0x7f9f380058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb824f90026510dd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:09.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:09.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f26ff5640 1 -- 172.21.15.98:0/876296177 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f38044040 con 0x565538ac4370 2024-09-19T08:07:09.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f26ff5640 1 -- 172.21.15.98:0/876296177 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f9f3803bbf0 con 0x565538ac4370 2024-09-19T08:07:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f4b841740 1 -- 172.21.15.98:0/876296177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f34004440 con 0x565538ac4370 2024-09-19T08:07:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f26ff5640 1 -- 172.21.15.98:0/876296177 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9f3803bdf0 con 0x565538ac4370 2024-09-19T08:07:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f4b841740 10 client.16590 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.138+0000 7f9f4b841740 1 -- 172.21.15.98:0/876296177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x565538b3eb90 con 0x565538ac4370 2024-09-19T08:07:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.139+0000 7f9f26ff5640 1 -- 172.21.15.98:0/876296177 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f34004440 con 0x565538ac4370 2024-09-19T08:07:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.139+0000 7f9f26ff5640 1 -- 172.21.15.98:0/876296177 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f9f38003bf0 con 0x565538ac4370 2024-09-19T08:07:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.139+0000 7f9f4b841740 10 client.16590 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.139+0000 7f9f4b841740 10 client.16590 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:07:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.139+0000 7f9f4b841740 20 client.16590 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.139+0000 7f9f4b841740 1 --2- 172.21.15.98:0/876296177 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x565538b3f1b0 0x565538b415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.139+0000 7f9f4b841740 4 client.16590 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:07:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.139+0000 7f9f4b841740 1 -- 172.21.15.98:0/876296177 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x565538a26d70 con 0x565538b3f1b0 2024-09-19T08:07:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.139+0000 7f9f42ffd640 1 --2- 172.21.15.98:0/876296177 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x565538b3f1b0 0x565538b415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.139+0000 7f9f42ffd640 1 --2- 172.21.15.98:0/876296177 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x565538b3f1b0 0x565538b415c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6102dd6e0224f78d server_cookie=ed83a66e063c2b90 in_seq=0 out_seq=0 2024-09-19T08:07:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.139+0000 7f9f26ff5640 10 client.16590 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:07:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f26ff5640 1 -- 172.21.15.98:0/876296177 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x565538a26d70 con 0x565538b3f1b0 2024-09-19T08:07:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f26ff5640 10 client.16590 handle_command_reply: tid=0 2024-09-19T08:07:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f4b841740 1 client.16590 shutdown 2024-09-19T08:07:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f4b841740 1 -- 172.21.15.98:0/876296177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538ac4370 msgr2=0x565538adbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f4b841740 1 --2- 172.21.15.98:0/876296177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538ac4370 0x565538adbb10 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9f3803b9c0 tx=0x7f9f380058d0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f4b841740 1 -- 172.21.15.98:0/876296177 shutdown_connections 2024-09-19T08:07:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f4b841740 1 --2- 172.21.15.98:0/876296177 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x565538b3f1b0 0x565538b415c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f4b841740 1 --2- 172.21.15.98:0/876296177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565538ac4370 0x565538adbb10 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f4b841740 1 --2- 172.21.15.98:0/876296177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565538abab40 0x565538adb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f4b841740 1 --2- 172.21.15.98:0/876296177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565538ab95d0 0x565538accd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f4b841740 1 -- 172.21.15.98:0/876296177 >> 172.21.15.98:0/876296177 conn(0x565538a82310 msgr2=0x565538ac2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:09.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f26ff5640 0 client.16590 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:07:09.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f4b841740 1 -- 172.21.15.98:0/876296177 shutdown_connections 2024-09-19T08:07:09.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f4b841740 1 -- 172.21.15.98:0/876296177 wait complete. 2024-09-19T08:07:09.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f4b841740 20 client.16590 trim_cache size 0 max 16384 2024-09-19T08:07:09.148 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:07:09.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f49468640 1 -- 172.21.15.98:0/3198622806 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9f2c0a3430 msgr2=0x7f9f2c0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:09.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f49468640 1 --2- 172.21.15.98:0/3198622806 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9f2c0a3430 0x7f9f2c0a58f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c0ccaa0 tx=0x7f9f3c0047d0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f49468640 1 -- 172.21.15.98:0/3198622806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f441082a0 msgr2=0x7f9f44101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:09.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.140+0000 7f9f49468640 1 --2- 172.21.15.98:0/3198622806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f441082a0 0x7f9f44101410 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c009fd0 tx=0x7f9f3c00b8e0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.141+0000 7f9f49468640 1 -- 172.21.15.98:0/3198622806 shutdown_connections 2024-09-19T08:07:09.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.141+0000 7f9f49468640 1 --2- 172.21.15.98:0/3198622806 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9f2c0a3430 0x7f9f2c0a58f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.141+0000 7f9f49468640 1 --2- 172.21.15.98:0/3198622806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f441082a0 0x7f9f44101410 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.141+0000 7f9f49468640 1 --2- 172.21.15.98:0/3198622806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f441036e0 0x7f9f44100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.141+0000 7f9f49468640 1 --2- 172.21.15.98:0/3198622806 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f44102d30 0x7f9f44100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:09.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.141+0000 7f9f49468640 1 -- 172.21.15.98:0/3198622806 >> 172.21.15.98:0/3198622806 conn(0x7f9f440fc7d0 msgr2=0x7f9f44105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:09.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.141+0000 7f9f49468640 1 -- 172.21.15.98:0/3198622806 shutdown_connections 2024-09-19T08:07:09.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:09.141+0000 7f9f49468640 1 -- 172.21.15.98:0/3198622806 wait complete. 2024-09-19T08:07:09.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:08 smithi139 ceph-mon[34874]: pgmap v127: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-09-19T08:07:09.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:08 smithi162 ceph-mon[35260]: pgmap v127: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-09-19T08:07:09.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:07:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.621+0000 7fac42b88640 1 -- 172.21.15.98:0/2900430376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c106a80 msgr2=0x7fac3c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.621+0000 7fac42b88640 1 --2- 172.21.15.98:0/2900430376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c106a80 0x7fac3c106e60 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fac30009930 tx=0x7fac3002a420 comp rx=0 tx=0).stop 2024-09-19T08:07:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.621+0000 7fac42b88640 1 -- 172.21.15.98:0/2900430376 shutdown_connections 2024-09-19T08:07:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.621+0000 7fac42b88640 1 --2- 172.21.15.98:0/2900430376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac3c101420 0x7fac3c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.621+0000 7fac42b88640 1 --2- 172.21.15.98:0/2900430376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac3c100a80 0x7fac3c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.621+0000 7fac42b88640 1 --2- 172.21.15.98:0/2900430376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c106a80 0x7fac3c106e60 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.621+0000 7fac42b88640 1 -- 172.21.15.98:0/2900430376 >> 172.21.15.98:0/2900430376 conn(0x7fac3c0fc7a0 msgr2=0x7fac3c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.621+0000 7fac42b88640 1 -- 172.21.15.98:0/2900430376 shutdown_connections 2024-09-19T08:07:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.621+0000 7fac42b88640 1 -- 172.21.15.98:0/2900430376 wait complete. 2024-09-19T08:07:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac42b88640 1 Processor -- start 2024-09-19T08:07:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac42b88640 1 -- start start 2024-09-19T08:07:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac42b88640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac3c100a80 0x7fac3c198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac42b88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c101420 0x7fac3c198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac42b88640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac3c106a80 0x7fac3c19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac42b88640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c110e80 con 0x7fac3c101420 2024-09-19T08:07:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac42b88640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c110d00 con 0x7fac3c100a80 2024-09-19T08:07:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac42b88640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c111000 con 0x7fac3c106a80 2024-09-19T08:07:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac3bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c101420 0x7fac3c198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac3bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c101420 0x7fac3c198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34598/0 (socket says 172.21.15.98:34598) 2024-09-19T08:07:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac3bfff640 1 -- 172.21.15.98:0/393562582 learned_addr learned my addr 172.21.15.98:0/393562582 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac40924640 1 --2- 172.21.15.98:0/393562582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac3c100a80 0x7fac3c198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac41125640 1 --2- 172.21.15.98:0/393562582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac3c106a80 0x7fac3c19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac3bfff640 1 -- 172.21.15.98:0/393562582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac3c106a80 msgr2=0x7fac3c19cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac3bfff640 1 --2- 172.21.15.98:0/393562582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac3c106a80 0x7fac3c19cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac3bfff640 1 -- 172.21.15.98:0/393562582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac3c100a80 msgr2=0x7fac3c198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac3bfff640 1 --2- 172.21.15.98:0/393562582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac3c100a80 0x7fac3c198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.622+0000 7fac3bfff640 1 -- 172.21.15.98:0/393562582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3c19d3f0 con 0x7fac3c101420 2024-09-19T08:07:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.623+0000 7fac3bfff640 1 --2- 172.21.15.98:0/393562582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c101420 0x7fac3c198980 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fac2800d950 tx=0x7fac2800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.623+0000 7fac39ffb640 1 -- 172.21.15.98:0/393562582 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28017030 con 0x7fac3c101420 2024-09-19T08:07:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.623+0000 7fac39ffb640 1 -- 172.21.15.98:0/393562582 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac28004020 con 0x7fac3c101420 2024-09-19T08:07:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.623+0000 7fac42b88640 1 -- 172.21.15.98:0/393562582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac3c19d6e0 con 0x7fac3c101420 2024-09-19T08:07:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.623+0000 7fac39ffb640 1 -- 172.21.15.98:0/393562582 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac280047a0 con 0x7fac3c101420 2024-09-19T08:07:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.623+0000 7fac42b88640 1 -- 172.21.15.98:0/393562582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac3c1a4fc0 con 0x7fac3c101420 2024-09-19T08:07:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.625+0000 7fac39ffb640 1 -- 172.21.15.98:0/393562582 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fac280041d0 con 0x7fac3c101420 2024-09-19T08:07:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.625+0000 7fac44f61740 1 Processor -- start 2024-09-19T08:07:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac44f61740 1 -- start start 2024-09-19T08:07:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac39ffb640 1 --2- 172.21.15.98:0/393562582 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fac2c0a33b0 0x7fac2c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac44f61740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 0x5611badaf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac44f61740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badaf5d0 0x5611badafa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac44f61740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badc0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac44f61740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5611bad152f0 con 0x5611badbe780 2024-09-19T08:07:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac44f61740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5611bacbccf0 con 0x5611badb0b40 2024-09-19T08:07:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac44f61740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5611bad112b0 con 0x5611badaf5d0 2024-09-19T08:07:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac41125640 1 --2- 172.21.15.98:0/393562582 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fac2c0a33b0 0x7fac2c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac39ffb640 1 -- 172.21.15.98:0/393562582 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fac280cf6d0 con 0x7fac3c101420 2024-09-19T08:07:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac40924640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 0x5611badaf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac3bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badaf5d0 0x5611badafa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac40924640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 0x5611badaf090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:40808/0 (socket says 172.21.15.98:40808) 2024-09-19T08:07:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac3bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badaf5d0 0x5611badafa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:47232/0 (socket says 172.21.15.98:47232) 2024-09-19T08:07:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac3bfff640 1 -- 172.21.15.98:0/3033589837 learned_addr learned my addr 172.21.15.98:0/3033589837 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac41125640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badc0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac3bfff640 1 -- 172.21.15.98:0/3033589837 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 msgr2=0x5611badaf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac3bfff640 1 --2- 172.21.15.98:0/3033589837 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 0x5611badaf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac3bfff640 1 -- 172.21.15.98:0/3033589837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 msgr2=0x5611badc0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac3bfff640 1 --2- 172.21.15.98:0/3033589837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badc0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac3bfff640 1 -- 172.21.15.98:0/3033589837 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611baca13a0 con 0x5611badaf5d0 2024-09-19T08:07:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.626+0000 7fac41125640 1 --2- 172.21.15.98:0/3033589837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badc0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:07:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.627+0000 7fac41125640 1 --2- 172.21.15.98:0/393562582 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fac2c0a33b0 0x7fac2c0a5870 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac34005c40 tx=0x7fac34009d30 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.627+0000 7fac3bfff640 1 --2- 172.21.15.98:0/3033589837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badaf5d0 0x5611badafa10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fac28000c00 tx=0x7fac2809b5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8357ea4ff47c46d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.627+0000 7fac26ffd640 1 -- 172.21.15.98:0/3033589837 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2806da10 con 0x5611badaf5d0 2024-09-19T08:07:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.627+0000 7fac26ffd640 1 -- 172.21.15.98:0/3033589837 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac2809c0e0 con 0x5611badaf5d0 2024-09-19T08:07:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.627+0000 7fac26ffd640 1 -- 172.21.15.98:0/3033589837 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2809fd90 con 0x5611badaf5d0 2024-09-19T08:07:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.628+0000 7fac44f61740 1 -- 172.21.15.98:0/3033589837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badaf5d0 msgr2=0x5611badafa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.628+0000 7fac44f61740 1 --2- 172.21.15.98:0/3033589837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badaf5d0 0x5611badafa10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fac28000c00 tx=0x7fac2809b5a0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.628+0000 7fac44f61740 1 -- 172.21.15.98:0/3033589837 shutdown_connections 2024-09-19T08:07:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.628+0000 7fac44f61740 1 --2- 172.21.15.98:0/3033589837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badc0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.628+0000 7fac44f61740 1 --2- 172.21.15.98:0/3033589837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badaf5d0 0x5611badafa10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.628+0000 7fac44f61740 1 --2- 172.21.15.98:0/3033589837 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 0x5611badaf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.628+0000 7fac44f61740 1 -- 172.21.15.98:0/3033589837 >> 172.21.15.98:0/3033589837 conn(0x5611bad78310 msgr2=0x5611badabc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.628+0000 7fac44f61740 1 -- 172.21.15.98:0/3033589837 shutdown_connections 2024-09-19T08:07:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.628+0000 7fac44f61740 1 -- 172.21.15.98:0/3033589837 wait complete. 2024-09-19T08:07:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.628+0000 7fac44f61740 1 -- start start 2024-09-19T08:07:10.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.628+0000 7fac44f61740 1 Processor -- start 2024-09-19T08:07:10.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.628+0000 7fac44f61740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badaf5d0 0x5611badbe210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac44f61740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badb0b40 0x5611badbb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac44f61740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badbbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac44f61740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5611badc1480 con 0x5611badbe780 2024-09-19T08:07:10.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac44f61740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5611bacd0060 con 0x5611badaf5d0 2024-09-19T08:07:10.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac44f61740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5611badc1600 con 0x5611badb0b40 2024-09-19T08:07:10.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac44f61740 10 client.0 _ll_register_callbacks cb 0x5611baa238c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:10.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac41125640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badbbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac3bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badb0b40 0x5611badbb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac3bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badb0b40 0x5611badbb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:47240/0 (socket says 172.21.15.98:47240) 2024-09-19T08:07:10.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac41125640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badbbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34618/0 (socket says 172.21.15.98:34618) 2024-09-19T08:07:10.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac3bfff640 1 -- 172.21.15.98:0/3994069398 learned_addr learned my addr 172.21.15.98:0/3994069398 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:10.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac40924640 1 --2- 172.21.15.98:0/3994069398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badaf5d0 0x5611badbe210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac41125640 1 -- 172.21.15.98:0/3994069398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badb0b40 msgr2=0x5611badbb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:10.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac41125640 1 --2- 172.21.15.98:0/3994069398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badb0b40 0x5611badbb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac41125640 1 -- 172.21.15.98:0/3994069398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badaf5d0 msgr2=0x5611badbe210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:10.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac41125640 1 --2- 172.21.15.98:0/3994069398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badaf5d0 0x5611badbe210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac41125640 1 -- 172.21.15.98:0/3994069398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611baa863c0 con 0x5611badbe780 2024-09-19T08:07:10.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac41125640 1 --2- 172.21.15.98:0/3994069398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badbbc50 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fac34005fd0 tx=0x7fac34004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ed3e40eb70abcf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:10.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac1bfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:10.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac1bfff640 1 -- 172.21.15.98:0/3994069398 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac34027070 con 0x5611badbe780 2024-09-19T08:07:10.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac1bfff640 1 -- 172.21.15.98:0/3994069398 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fac34004b20 con 0x5611badbe780 2024-09-19T08:07:10.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac44f61740 1 -- 172.21.15.98:0/3994069398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac2809c0e0 con 0x5611badbe780 2024-09-19T08:07:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac1bfff640 1 -- 172.21.15.98:0/3994069398 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fac34032050 con 0x5611badbe780 2024-09-19T08:07:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac44f61740 10 client.16602 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.629+0000 7fac44f61740 1 -- 172.21.15.98:0/3994069398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5611badccfd0 con 0x5611badbe780 2024-09-19T08:07:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.630+0000 7fac1bfff640 1 -- 172.21.15.98:0/3994069398 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac2809c0e0 con 0x5611badbe780 2024-09-19T08:07:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.630+0000 7fac1bfff640 1 -- 172.21.15.98:0/3994069398 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fac34023430 con 0x5611badbe780 2024-09-19T08:07:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.630+0000 7fac44f61740 10 client.16602 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.630+0000 7fac44f61740 10 client.16602 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.630+0000 7fac44f61740 20 client.16602 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.630+0000 7fac44f61740 1 --2- 172.21.15.98:0/3994069398 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5611badce430 0x5611badd0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.630+0000 7fac44f61740 4 client.16602 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.630+0000 7fac44f61740 1 -- 172.21.15.98:0/3994069398 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611bad22860 con 0x5611badce430 2024-09-19T08:07:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.630+0000 7fac40924640 1 --2- 172.21.15.98:0/3994069398 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5611badce430 0x5611badd0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.631+0000 7fac40924640 1 --2- 172.21.15.98:0/3994069398 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5611badce430 0x5611badd0860 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51f046c2a3ab565e server_cookie=b5abf99584706d41 in_seq=0 out_seq=0 2024-09-19T08:07:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.631+0000 7fac1bfff640 10 client.16602 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.631+0000 7fac1bfff640 1 -- 172.21.15.98:0/3994069398 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5611bad22860 con 0x5611badce430 2024-09-19T08:07:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.631+0000 7fac1bfff640 10 client.16602 handle_command_reply: tid=0 2024-09-19T08:07:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.631+0000 7fac44f61740 1 client.16602 shutdown 2024-09-19T08:07:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.631+0000 7fac44f61740 1 -- 172.21.15.98:0/3994069398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 msgr2=0x5611badbbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.631+0000 7fac44f61740 1 --2- 172.21.15.98:0/3994069398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badbbc50 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fac34005fd0 tx=0x7fac34004ab0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.631+0000 7fac44f61740 1 -- 172.21.15.98:0/3994069398 shutdown_connections 2024-09-19T08:07:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.631+0000 7fac44f61740 1 --2- 172.21.15.98:0/3994069398 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5611badce430 0x5611badd0860 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.631+0000 7fac44f61740 1 --2- 172.21.15.98:0/3994069398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badbbc50 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.631+0000 7fac44f61740 1 --2- 172.21.15.98:0/3994069398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badb0b40 0x5611badbb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.631+0000 7fac44f61740 1 --2- 172.21.15.98:0/3994069398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badaf5d0 0x5611badbe210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.631+0000 7fac1bfff640 0 client.16602 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.631+0000 7fac44f61740 1 -- 172.21.15.98:0/3994069398 >> 172.21.15.98:0/3994069398 conn(0x5611bad78310 msgr2=0x5611badb1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:10.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.632+0000 7fac44f61740 1 -- 172.21.15.98:0/3994069398 shutdown_connections 2024-09-19T08:07:10.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.632+0000 7fac44f61740 1 -- 172.21.15.98:0/3994069398 wait complete. 2024-09-19T08:07:10.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.632+0000 7fac44f61740 20 client.16602 trim_cache size 0 max 16384 2024-09-19T08:07:10.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac44f61740 1 Processor -- start 2024-09-19T08:07:10.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac44f61740 1 -- start start 2024-09-19T08:07:10.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac44f61740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badaf5d0 0x5611badba4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac44f61740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 0x5611badd6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac44f61740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badd6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac44f61740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c1ab640 con 0x5611badbe780 2024-09-19T08:07:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac44f61740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c1ad1e0 con 0x5611badb0b40 2024-09-19T08:07:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac44f61740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c1abd10 con 0x5611badaf5d0 2024-09-19T08:07:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac41125640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badd6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac41125640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badd6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34626/0 (socket says 172.21.15.98:34626) 2024-09-19T08:07:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac41125640 1 -- 172.21.15.98:0/168140312 learned_addr learned my addr 172.21.15.98:0/168140312 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac40924640 1 --2- 172.21.15.98:0/168140312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badaf5d0 0x5611badba4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac3bfff640 1 --2- 172.21.15.98:0/168140312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 0x5611badd6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac41125640 1 -- 172.21.15.98:0/168140312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badaf5d0 msgr2=0x5611badba4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac41125640 1 --2- 172.21.15.98:0/168140312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badaf5d0 0x5611badba4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac41125640 1 -- 172.21.15.98:0/168140312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 msgr2=0x5611badd6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac41125640 1 --2- 172.21.15.98:0/168140312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 0x5611badd6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.645+0000 7fac41125640 1 -- 172.21.15.98:0/168140312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611badac960 con 0x5611badbe780 2024-09-19T08:07:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.646+0000 7fac41125640 1 --2- 172.21.15.98:0/168140312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badd6670 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fac34023850 tx=0x7fac34023640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84ff74a32ac6be1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.646+0000 7fac267fc640 1 -- 172.21.15.98:0/168140312 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3403a030 con 0x5611badbe780 2024-09-19T08:07:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.646+0000 7fac267fc640 1 -- 172.21.15.98:0/168140312 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac34013c90 con 0x5611badbe780 2024-09-19T08:07:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.646+0000 7fac267fc640 1 -- 172.21.15.98:0/168140312 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac34027040 con 0x5611badbe780 2024-09-19T08:07:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.646+0000 7fac44f61740 1 -- 172.21.15.98:0/168140312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 msgr2=0x5611badd6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.646+0000 7fac44f61740 1 --2- 172.21.15.98:0/168140312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badd6670 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fac34023850 tx=0x7fac34023640 comp rx=0 tx=0).stop 2024-09-19T08:07:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.646+0000 7fac44f61740 1 -- 172.21.15.98:0/168140312 shutdown_connections 2024-09-19T08:07:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.646+0000 7fac44f61740 1 --2- 172.21.15.98:0/168140312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badbe780 0x5611badd6670 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.646+0000 7fac44f61740 1 --2- 172.21.15.98:0/168140312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 0x5611badd6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.646+0000 7fac44f61740 1 --2- 172.21.15.98:0/168140312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badaf5d0 0x5611badba4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.646+0000 7fac44f61740 1 -- 172.21.15.98:0/168140312 >> 172.21.15.98:0/168140312 conn(0x5611bad78310 msgr2=0x5611badb9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.647+0000 7fac44f61740 1 -- 172.21.15.98:0/168140312 shutdown_connections 2024-09-19T08:07:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.647+0000 7fac44f61740 1 -- 172.21.15.98:0/168140312 wait complete. 2024-09-19T08:07:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.647+0000 7fac44f61740 1 -- start start 2024-09-19T08:07:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.647+0000 7fac44f61740 1 Processor -- start 2024-09-19T08:07:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.647+0000 7fac44f61740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badaf5d0 0x5611badc7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.647+0000 7fac44f61740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 0x5611badb8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.647+0000 7fac44f61740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badbe780 0x5611badb8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.647+0000 7fac44f61740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c1acae0 con 0x5611badaf5d0 2024-09-19T08:07:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.647+0000 7fac44f61740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c1ac410 con 0x5611badb0b40 2024-09-19T08:07:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.647+0000 7fac44f61740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c1a9640 con 0x5611badbe780 2024-09-19T08:07:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.647+0000 7fac44f61740 10 client.0 _ll_register_callbacks cb 0x7fac34009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.647+0000 7fac40924640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badaf5d0 0x5611badc7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.647+0000 7fac40924640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badaf5d0 0x5611badc7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34634/0 (socket says 172.21.15.98:34634) 2024-09-19T08:07:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.647+0000 7fac40924640 1 -- 172.21.15.98:0/861526078 learned_addr learned my addr 172.21.15.98:0/861526078 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac41125640 1 --2- 172.21.15.98:0/861526078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badbe780 0x5611badb8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac3bfff640 1 --2- 172.21.15.98:0/861526078 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 0x5611badb8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac40924640 1 -- 172.21.15.98:0/861526078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badbe780 msgr2=0x5611badb8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac40924640 1 --2- 172.21.15.98:0/861526078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badbe780 0x5611badb8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac40924640 1 -- 172.21.15.98:0/861526078 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 msgr2=0x5611badb8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac40924640 1 --2- 172.21.15.98:0/861526078 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 0x5611badb8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac40924640 1 -- 172.21.15.98:0/861526078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611bae38100 con 0x5611badaf5d0 2024-09-19T08:07:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac41125640 1 --2- 172.21.15.98:0/861526078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badbe780 0x5611badb8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac40924640 1 --2- 172.21.15.98:0/861526078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badaf5d0 0x5611badc7170 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fac30009900 tx=0x7fac30031b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=360f34dfe874a380 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac1bfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac1bfff640 1 -- 172.21.15.98:0/861526078 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac30035040 con 0x5611badaf5d0 2024-09-19T08:07:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac1bfff640 1 -- 172.21.15.98:0/861526078 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fac30005020 con 0x5611badaf5d0 2024-09-19T08:07:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac44f61740 1 -- 172.21.15.98:0/861526078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac34013c90 con 0x5611badaf5d0 2024-09-19T08:07:10.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac1bfff640 1 -- 172.21.15.98:0/861526078 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fac30005220 con 0x5611badaf5d0 2024-09-19T08:07:10.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac44f61740 10 client.16608 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:10.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac44f61740 1 -- 172.21.15.98:0/861526078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5611bae39300 con 0x5611badaf5d0 2024-09-19T08:07:10.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.648+0000 7fac1bfff640 1 -- 172.21.15.98:0/861526078 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac34013c90 con 0x5611badaf5d0 2024-09-19T08:07:10.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.649+0000 7fac1bfff640 1 -- 172.21.15.98:0/861526078 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fac300035f0 con 0x5611badaf5d0 2024-09-19T08:07:10.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.649+0000 7fac44f61740 10 client.16608 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:10.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.649+0000 7fac44f61740 10 client.16608 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:10.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.649+0000 7fac44f61740 20 client.16608 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:10.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.649+0000 7fac44f61740 1 --2- 172.21.15.98:0/861526078 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5611bae39a50 0x5611bae3be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:10.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.649+0000 7fac44f61740 4 client.16608 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:07:10.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.649+0000 7fac44f61740 1 -- 172.21.15.98:0/861526078 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5611bad1cd70 con 0x5611bae39a50 2024-09-19T08:07:10.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.649+0000 7fac3bfff640 1 --2- 172.21.15.98:0/861526078 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5611bae39a50 0x5611bae3be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:10.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.649+0000 7fac1bfff640 10 client.16608 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:10.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.649+0000 7fac3bfff640 1 --2- 172.21.15.98:0/861526078 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5611bae39a50 0x5611bae3be70 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=907193bc5f7089b2 server_cookie=5079e18121a18b6f in_seq=0 out_seq=0 2024-09-19T08:07:10.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac1bfff640 1 -- 172.21.15.98:0/861526078 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5611bad1cd70 con 0x5611bae39a50 2024-09-19T08:07:10.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac1bfff640 10 client.16608 handle_command_reply: tid=0 2024-09-19T08:07:10.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac44f61740 1 client.16608 shutdown 2024-09-19T08:07:10.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac44f61740 1 -- 172.21.15.98:0/861526078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badaf5d0 msgr2=0x5611badc7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:10.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac44f61740 1 --2- 172.21.15.98:0/861526078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badaf5d0 0x5611badc7170 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fac30009900 tx=0x7fac30031b40 comp rx=0 tx=0).stop 2024-09-19T08:07:10.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac44f61740 1 -- 172.21.15.98:0/861526078 shutdown_connections 2024-09-19T08:07:10.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac44f61740 1 --2- 172.21.15.98:0/861526078 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5611bae39a50 0x5611bae3be70 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac44f61740 1 --2- 172.21.15.98:0/861526078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611badbe780 0x5611badb8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac44f61740 1 --2- 172.21.15.98:0/861526078 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5611badb0b40 0x5611badb8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac1bfff640 0 client.16608 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:10.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac44f61740 1 --2- 172.21.15.98:0/861526078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611badaf5d0 0x5611badc7170 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac44f61740 1 -- 172.21.15.98:0/861526078 >> 172.21.15.98:0/861526078 conn(0x5611bad78310 msgr2=0x5611badba5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:10.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac44f61740 1 -- 172.21.15.98:0/861526078 shutdown_connections 2024-09-19T08:07:10.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac44f61740 1 -- 172.21.15.98:0/861526078 wait complete. 2024-09-19T08:07:10.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac44f61740 20 client.16608 trim_cache size 0 max 16384 2024-09-19T08:07:10.661 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:07:10.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac42b88640 1 -- 172.21.15.98:0/393562582 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fac2c0a33b0 msgr2=0x7fac2c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:10.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac42b88640 1 --2- 172.21.15.98:0/393562582 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fac2c0a33b0 0x7fac2c0a5870 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac34005c40 tx=0x7fac34009d30 comp rx=0 tx=0).stop 2024-09-19T08:07:10.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac42b88640 1 -- 172.21.15.98:0/393562582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c101420 msgr2=0x7fac3c198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:10.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.650+0000 7fac42b88640 1 --2- 172.21.15.98:0/393562582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c101420 0x7fac3c198980 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fac2800d950 tx=0x7fac2800de10 comp rx=0 tx=0).stop 2024-09-19T08:07:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.651+0000 7fac42b88640 1 -- 172.21.15.98:0/393562582 shutdown_connections 2024-09-19T08:07:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.651+0000 7fac42b88640 1 --2- 172.21.15.98:0/393562582 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fac2c0a33b0 0x7fac2c0a5870 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.651+0000 7fac42b88640 1 --2- 172.21.15.98:0/393562582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac3c106a80 0x7fac3c19cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.651+0000 7fac42b88640 1 --2- 172.21.15.98:0/393562582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c101420 0x7fac3c198980 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.651+0000 7fac42b88640 1 --2- 172.21.15.98:0/393562582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac3c100a80 0x7fac3c198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.651+0000 7fac42b88640 1 -- 172.21.15.98:0/393562582 >> 172.21.15.98:0/393562582 conn(0x7fac3c0fc7a0 msgr2=0x7fac3c104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.651+0000 7fac42b88640 1 -- 172.21.15.98:0/393562582 shutdown_connections 2024-09-19T08:07:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:10.651+0000 7fac42b88640 1 -- 172.21.15.98:0/393562582 wait complete. 2024-09-19T08:07:11.172 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:10 smithi162 ceph-mon[35260]: pgmap v128: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:07:11.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:10 smithi139 ceph-mon[34874]: pgmap v128: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:07:11.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:10 smithi098 ceph-mon[31333]: pgmap v128: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:07:12.407 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:07:12.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:07:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:12.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:12 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:07:12.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:12 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:07:12.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:12 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:07:13.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.161+0000 7f3de725c640 1 -- 172.21.15.98:0/458170665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de01027a0 msgr2=0x7f3de01067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:13.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.161+0000 7f3de725c640 1 --2- 172.21.15.98:0/458170665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de01027a0 0x7f3de01067f0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc009900 tx=0x7f3dcc02a410 comp rx=0 tx=0).stop 2024-09-19T08:07:13.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.161+0000 7f3de725c640 1 -- 172.21.15.98:0/458170665 shutdown_connections 2024-09-19T08:07:13.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.161+0000 7f3de725c640 1 --2- 172.21.15.98:0/458170665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3de0106f20 0x7f3de010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.161+0000 7f3de725c640 1 --2- 172.21.15.98:0/458170665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de01027a0 0x7f3de01067f0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.161+0000 7f3de725c640 1 --2- 172.21.15.98:0/458170665 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de0101df0 0x7f3de01021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.161+0000 7f3de725c640 1 -- 172.21.15.98:0/458170665 >> 172.21.15.98:0/458170665 conn(0x7f3de00fd8b0 msgr2=0x7f3de00ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:13.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.161+0000 7f3de725c640 1 -- 172.21.15.98:0/458170665 shutdown_connections 2024-09-19T08:07:13.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.161+0000 7f3de725c640 1 -- 172.21.15.98:0/458170665 wait complete. 2024-09-19T08:07:13.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.161+0000 7f3de725c640 1 Processor -- start 2024-09-19T08:07:13.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.161+0000 7f3de725c640 1 -- start start 2024-09-19T08:07:13.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3de725c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3de0101df0 0x7f3de0198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3de725c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de01027a0 0x7f3de0198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3de725c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de0106f20 0x7f3de019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3de725c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3de0111f80 con 0x7f3de01027a0 2024-09-19T08:07:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3de725c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3de0111e00 con 0x7f3de0106f20 2024-09-19T08:07:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3de725c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3de0112100 con 0x7f3de0101df0 2024-09-19T08:07:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3ddffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de01027a0 0x7f3de0198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3de4ff8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3de0101df0 0x7f3de0198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3ddffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de01027a0 0x7f3de0198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34652/0 (socket says 172.21.15.98:34652) 2024-09-19T08:07:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3ddffff640 1 -- 172.21.15.98:0/2940255423 learned_addr learned my addr 172.21.15.98:0/2940255423 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3de57f9640 1 --2- 172.21.15.98:0/2940255423 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de0106f20 0x7f3de019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3ddffff640 1 -- 172.21.15.98:0/2940255423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3de0101df0 msgr2=0x7f3de0198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3ddffff640 1 --2- 172.21.15.98:0/2940255423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3de0101df0 0x7f3de0198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3ddffff640 1 -- 172.21.15.98:0/2940255423 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de0106f20 msgr2=0x7f3de019ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3ddffff640 1 --2- 172.21.15.98:0/2940255423 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de0106f20 0x7f3de019ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3ddffff640 1 -- 172.21.15.98:0/2940255423 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de019d3b0 con 0x7f3de01027a0 2024-09-19T08:07:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3ddffff640 1 --2- 172.21.15.98:0/2940255423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de01027a0 0x7f3de0198940 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc006010 tx=0x7f3dcc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3dddffb640 1 -- 172.21.15.98:0/2940255423 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc03f030 con 0x7f3de01027a0 2024-09-19T08:07:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.162+0000 7f3dddffb640 1 -- 172.21.15.98:0/2940255423 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dcc003ce0 con 0x7f3de01027a0 2024-09-19T08:07:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.163+0000 7f3de725c640 1 -- 172.21.15.98:0/2940255423 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de019d640 con 0x7f3de01027a0 2024-09-19T08:07:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.163+0000 7f3dddffb640 1 -- 172.21.15.98:0/2940255423 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc035070 con 0x7f3de01027a0 2024-09-19T08:07:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.163+0000 7f3de725c640 1 -- 172.21.15.98:0/2940255423 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de01a4f80 con 0x7f3de01027a0 2024-09-19T08:07:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.165+0000 7f3dddffb640 1 -- 172.21.15.98:0/2940255423 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f3dcc042030 con 0x7f3de01027a0 2024-09-19T08:07:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.165+0000 7f3de9635740 1 Processor -- start 2024-09-19T08:07:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.165+0000 7f3de9635740 1 -- start start 2024-09-19T08:07:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.165+0000 7f3de9635740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa6ca20 0x55885aa6ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.165+0000 7f3de9635740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6d340 0x55885aa7bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de9635740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa7c150 0x55885aa7e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de9635740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55885a9d2280 con 0x55885aa6ca20 2024-09-19T08:07:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de9635740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55885a979b10 con 0x55885aa7c150 2024-09-19T08:07:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de9635740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55885a9ce240 con 0x55885aa6d340 2024-09-19T08:07:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3dddffb640 1 --2- 172.21.15.98:0/2940255423 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3dd00a3430 0x7f3dd00a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de57f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa6ca20 0x55885aa6ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de57f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa6ca20 0x55885aa6ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34658/0 (socket says 172.21.15.98:34658) 2024-09-19T08:07:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de57f9640 1 -- 172.21.15.98:0/3379613329 learned_addr learned my addr 172.21.15.98:0/3379613329 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3dddffb640 1 -- 172.21.15.98:0/2940255423 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f3dcc0f01b0 con 0x7f3de01027a0 2024-09-19T08:07:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3ddffff640 1 --2- 172.21.15.98:0/3379613329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa7c150 0x55885aa7e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de4ff8640 1 --2- 172.21.15.98:0/3379613329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6d340 0x55885aa7bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de57f9640 1 --2- 172.21.15.98:0/2940255423 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3dd00a3430 0x7f3dd00a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de57f9640 1 -- 172.21.15.98:0/3379613329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6d340 msgr2=0x55885aa7bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de57f9640 1 --2- 172.21.15.98:0/3379613329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6d340 0x55885aa7bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de57f9640 1 -- 172.21.15.98:0/3379613329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa7c150 msgr2=0x55885aa7e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de57f9640 1 --2- 172.21.15.98:0/3379613329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa7c150 0x55885aa7e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de57f9640 1 -- 172.21.15.98:0/3379613329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55885a9861f0 con 0x55885aa6ca20 2024-09-19T08:07:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de57f9640 1 --2- 172.21.15.98:0/3379613329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa6ca20 0x55885aa6ce00 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8014940 tx=0x7f3dd8014e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2c4fce7da67c29 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3dcaffd640 1 -- 172.21.15.98:0/3379613329 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8014050 con 0x55885aa6ca20 2024-09-19T08:07:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3dcaffd640 1 -- 172.21.15.98:0/3379613329 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dd80141f0 con 0x55885aa6ca20 2024-09-19T08:07:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3dcaffd640 1 -- 172.21.15.98:0/3379613329 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd801e650 con 0x55885aa6ca20 2024-09-19T08:07:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.166+0000 7f3de57f9640 1 --2- 172.21.15.98:0/2940255423 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3dd00a3430 0x7f3dd00a58f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8021000 tx=0x7f3dd8020000 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.167+0000 7f3de9635740 1 -- 172.21.15.98:0/3379613329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa6ca20 msgr2=0x55885aa6ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.167+0000 7f3de9635740 1 --2- 172.21.15.98:0/3379613329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa6ca20 0x55885aa6ce00 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8014940 tx=0x7f3dd8014e00 comp rx=0 tx=0).stop 2024-09-19T08:07:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.167+0000 7f3de9635740 1 -- 172.21.15.98:0/3379613329 shutdown_connections 2024-09-19T08:07:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.167+0000 7f3de9635740 1 --2- 172.21.15.98:0/3379613329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa7c150 0x55885aa7e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.167+0000 7f3de9635740 1 --2- 172.21.15.98:0/3379613329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6d340 0x55885aa7bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.167+0000 7f3de9635740 1 --2- 172.21.15.98:0/3379613329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa6ca20 0x55885aa6ce00 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.167+0000 7f3de9635740 1 -- 172.21.15.98:0/3379613329 >> 172.21.15.98:0/3379613329 conn(0x55885aa690f0 msgr2=0x55885aa694d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.167+0000 7f3de9635740 1 -- 172.21.15.98:0/3379613329 shutdown_connections 2024-09-19T08:07:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.167+0000 7f3de9635740 1 -- 172.21.15.98:0/3379613329 wait complete. 2024-09-19T08:07:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.167+0000 7f3de9635740 1 -- start start 2024-09-19T08:07:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de9635740 1 Processor -- start 2024-09-19T08:07:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de9635740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6ca20 0x55885aa790b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de9635740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6d340 0x55885aa795f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de9635740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa7c150 0x55885aa79b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de9635740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55885aa7ed20 con 0x55885aa7c150 2024-09-19T08:07:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de9635740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55885a98cf10 con 0x55885aa6ca20 2024-09-19T08:07:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de9635740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55885aa7eea0 con 0x55885aa6d340 2024-09-19T08:07:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de9635740 10 client.0 _ll_register_callbacks cb 0x55885a955a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de57f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa7c150 0x55885aa79b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de57f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa7c150 0x55885aa79b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34668/0 (socket says 172.21.15.98:34668) 2024-09-19T08:07:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de57f9640 1 -- 172.21.15.98:0/2237710802 learned_addr learned my addr 172.21.15.98:0/2237710802 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3ddffff640 1 --2- 172.21.15.98:0/2237710802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6d340 0x55885aa795f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de4ff8640 1 --2- 172.21.15.98:0/2237710802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6ca20 0x55885aa790b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de57f9640 1 -- 172.21.15.98:0/2237710802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6d340 msgr2=0x55885aa795f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de57f9640 1 --2- 172.21.15.98:0/2237710802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6d340 0x55885aa795f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de57f9640 1 -- 172.21.15.98:0/2237710802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6ca20 msgr2=0x55885aa790b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de57f9640 1 --2- 172.21.15.98:0/2237710802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6ca20 0x55885aa790b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3de57f9640 1 -- 172.21.15.98:0/2237710802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55885a7436a0 con 0x55885aa7c150 2024-09-19T08:07:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.168+0000 7f3ddffff640 1 --2- 172.21.15.98:0/2237710802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6d340 0x55885aa795f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3de57f9640 1 --2- 172.21.15.98:0/2237710802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa7c150 0x55885aa79b30 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80096b0 tx=0x7f3dd8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e0d8cfd5f22ace server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3dbffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3dbffff640 1 -- 172.21.15.98:0/2237710802 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80143a0 con 0x55885aa7c150 2024-09-19T08:07:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3dbffff640 1 -- 172.21.15.98:0/2237710802 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f3dd801e3f0 con 0x55885aa7c150 2024-09-19T08:07:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3de9635740 1 -- 172.21.15.98:0/2237710802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dd80141f0 con 0x55885aa7c150 2024-09-19T08:07:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3dbffff640 1 -- 172.21.15.98:0/2237710802 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3dd801e5f0 con 0x55885aa7c150 2024-09-19T08:07:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3de9635740 10 client.16620 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3de9635740 1 -- 172.21.15.98:0/2237710802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55885aa8aa90 con 0x55885aa7c150 2024-09-19T08:07:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3dbffff640 1 -- 172.21.15.98:0/2237710802 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dd80141f0 con 0x55885aa7c150 2024-09-19T08:07:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3dbffff640 1 -- 172.21.15.98:0/2237710802 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f3dd800f040 con 0x55885aa7c150 2024-09-19T08:07:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3de9635740 10 client.16620 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3de9635740 10 client.16620 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3de9635740 20 client.16620 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3de9635740 1 --2- 172.21.15.98:0/2237710802 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55885aa8bea0 0x55885aa8e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3de9635740 4 client.16620 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.169+0000 7f3de9635740 1 -- 172.21.15.98:0/2237710802 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55885a9df7f0 con 0x55885aa8bea0 2024-09-19T08:07:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.170+0000 7f3de4ff8640 1 --2- 172.21.15.98:0/2237710802 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55885aa8bea0 0x55885aa8e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.170+0000 7f3dbffff640 10 client.16620 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.170+0000 7f3de4ff8640 1 --2- 172.21.15.98:0/2237710802 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55885aa8bea0 0x55885aa8e290 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2554e88d3dbe4c8 server_cookie=ac4810152d1222fa in_seq=0 out_seq=0 2024-09-19T08:07:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3dbffff640 1 -- 172.21.15.98:0/2237710802 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55885a9df7f0 con 0x55885aa8bea0 2024-09-19T08:07:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3dbffff640 10 client.16620 handle_command_reply: tid=0 2024-09-19T08:07:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3de9635740 1 client.16620 shutdown 2024-09-19T08:07:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3de9635740 1 -- 172.21.15.98:0/2237710802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa7c150 msgr2=0x55885aa79b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3de9635740 1 --2- 172.21.15.98:0/2237710802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa7c150 0x55885aa79b30 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80096b0 tx=0x7f3dd8005f90 comp rx=0 tx=0).stop 2024-09-19T08:07:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3de9635740 1 -- 172.21.15.98:0/2237710802 shutdown_connections 2024-09-19T08:07:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3de9635740 1 --2- 172.21.15.98:0/2237710802 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55885aa8bea0 0x55885aa8e290 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3dbffff640 0 client.16620 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:13.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3de9635740 1 --2- 172.21.15.98:0/2237710802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa7c150 0x55885aa79b30 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3de9635740 1 --2- 172.21.15.98:0/2237710802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6d340 0x55885aa795f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3de9635740 1 --2- 172.21.15.98:0/2237710802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6ca20 0x55885aa790b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3de9635740 1 -- 172.21.15.98:0/2237710802 >> 172.21.15.98:0/2237710802 conn(0x55885aa690f0 msgr2=0x55885aa6ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:13.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3de9635740 1 -- 172.21.15.98:0/2237710802 shutdown_connections 2024-09-19T08:07:13.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3de9635740 1 -- 172.21.15.98:0/2237710802 wait complete. 2024-09-19T08:07:13.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.171+0000 7f3de9635740 20 client.16620 trim_cache size 0 max 16384 2024-09-19T08:07:13.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.188+0000 7f3de9635740 1 Processor -- start 2024-09-19T08:07:13.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.188+0000 7f3de9635740 1 -- start start 2024-09-19T08:07:13.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de9635740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6ca20 0x55885aa68160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de9635740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6d340 0x55885aa93a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de9635740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa7c150 0x55885aa93fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de9635740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3de01ab5b0 con 0x55885aa7c150 2024-09-19T08:07:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de9635740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3de01ad150 con 0x55885aa6d340 2024-09-19T08:07:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de9635740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3de01abc80 con 0x55885aa6ca20 2024-09-19T08:07:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de57f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa7c150 0x55885aa93fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de57f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa7c150 0x55885aa93fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34676/0 (socket says 172.21.15.98:34676) 2024-09-19T08:07:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de57f9640 1 -- 172.21.15.98:0/2604591996 learned_addr learned my addr 172.21.15.98:0/2604591996 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de4ff8640 1 --2- 172.21.15.98:0/2604591996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6ca20 0x55885aa68160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3ddffff640 1 --2- 172.21.15.98:0/2604591996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6d340 0x55885aa93a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de57f9640 1 -- 172.21.15.98:0/2604591996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6ca20 msgr2=0x55885aa68160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de57f9640 1 --2- 172.21.15.98:0/2604591996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6ca20 0x55885aa68160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de57f9640 1 -- 172.21.15.98:0/2604591996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6d340 msgr2=0x55885aa93a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de57f9640 1 --2- 172.21.15.98:0/2604591996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6d340 0x55885aa93a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de57f9640 1 -- 172.21.15.98:0/2604591996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55885aa77600 con 0x55885aa7c150 2024-09-19T08:07:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3de57f9640 1 --2- 172.21.15.98:0/2604591996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa7c150 0x55885aa93fc0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8015010 tx=0x7f3dd8012320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e48cb0d40f35d97 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3dca7fc640 1 -- 172.21.15.98:0/2604591996 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd801e3f0 con 0x55885aa7c150 2024-09-19T08:07:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3dca7fc640 1 -- 172.21.15.98:0/2604591996 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dd8009800 con 0x55885aa7c150 2024-09-19T08:07:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.189+0000 7f3dca7fc640 1 -- 172.21.15.98:0/2604591996 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80340f0 con 0x55885aa7c150 2024-09-19T08:07:13.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.190+0000 7f3de9635740 1 -- 172.21.15.98:0/2604591996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa7c150 msgr2=0x55885aa93fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.190+0000 7f3de9635740 1 --2- 172.21.15.98:0/2604591996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa7c150 0x55885aa93fc0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8015010 tx=0x7f3dd8012320 comp rx=0 tx=0).stop 2024-09-19T08:07:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.190+0000 7f3de9635740 1 -- 172.21.15.98:0/2604591996 shutdown_connections 2024-09-19T08:07:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.190+0000 7f3de9635740 1 --2- 172.21.15.98:0/2604591996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa7c150 0x55885aa93fc0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.190+0000 7f3de9635740 1 --2- 172.21.15.98:0/2604591996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6d340 0x55885aa93a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.190+0000 7f3de9635740 1 --2- 172.21.15.98:0/2604591996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa6ca20 0x55885aa68160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.190+0000 7f3de9635740 1 -- 172.21.15.98:0/2604591996 >> 172.21.15.98:0/2604591996 conn(0x55885aa690f0 msgr2=0x55885aa67a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.190+0000 7f3de9635740 1 -- 172.21.15.98:0/2604591996 shutdown_connections 2024-09-19T08:07:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.190+0000 7f3de9635740 1 -- 172.21.15.98:0/2604591996 wait complete. 2024-09-19T08:07:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.191+0000 7f3de9635740 1 -- start start 2024-09-19T08:07:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.191+0000 7f3de9635740 1 Processor -- start 2024-09-19T08:07:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.191+0000 7f3de9635740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa6ca20 0x55885aa84620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.191+0000 7f3de9635740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6d340 0x55885aa81100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.191+0000 7f3de9635740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa7c150 0x55885aa81640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.191+0000 7f3de9635740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3de01aca50 con 0x55885aa6ca20 2024-09-19T08:07:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.191+0000 7f3de9635740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3de01ac380 con 0x55885aa6d340 2024-09-19T08:07:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.191+0000 7f3de9635740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3de01a95b0 con 0x55885aa7c150 2024-09-19T08:07:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.191+0000 7f3de9635740 10 client.0 _ll_register_callbacks cb 0x7f3dcc005750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.191+0000 7f3de4ff8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa6ca20 0x55885aa84620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.191+0000 7f3de4ff8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa6ca20 0x55885aa84620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34690/0 (socket says 172.21.15.98:34690) 2024-09-19T08:07:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.191+0000 7f3de4ff8640 1 -- 172.21.15.98:0/75049063 learned_addr learned my addr 172.21.15.98:0/75049063 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.191+0000 7f3de57f9640 1 --2- 172.21.15.98:0/75049063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa7c150 0x55885aa81640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.191+0000 7f3ddffff640 1 --2- 172.21.15.98:0/75049063 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6d340 0x55885aa81100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3de4ff8640 1 -- 172.21.15.98:0/75049063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa7c150 msgr2=0x55885aa81640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3de4ff8640 1 --2- 172.21.15.98:0/75049063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa7c150 0x55885aa81640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3de4ff8640 1 -- 172.21.15.98:0/75049063 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6d340 msgr2=0x55885aa81100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3de4ff8640 1 --2- 172.21.15.98:0/75049063 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6d340 0x55885aa81100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3de4ff8640 1 -- 172.21.15.98:0/75049063 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55885aa81c00 con 0x55885aa6ca20 2024-09-19T08:07:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3de57f9640 1 --2- 172.21.15.98:0/75049063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa7c150 0x55885aa81640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:07:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3de4ff8640 1 --2- 172.21.15.98:0/75049063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa6ca20 0x55885aa84620 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4005fd0 tx=0x7f3dd401fa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3128fffa31012d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3dbffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3dbffff640 1 -- 172.21.15.98:0/75049063 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd4011d50 con 0x55885aa6ca20 2024-09-19T08:07:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3dbffff640 1 -- 172.21.15.98:0/75049063 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f3dd4004020 con 0x55885aa6ca20 2024-09-19T08:07:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3de9635740 1 -- 172.21.15.98:0/75049063 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dd8009800 con 0x55885aa6ca20 2024-09-19T08:07:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3dbffff640 1 -- 172.21.15.98:0/75049063 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3dd4004220 con 0x55885aa6ca20 2024-09-19T08:07:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3de9635740 10 client.16626 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3de9635740 1 -- 172.21.15.98:0/75049063 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55885aaf78d0 con 0x55885aa6ca20 2024-09-19T08:07:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3dbffff640 1 -- 172.21.15.98:0/75049063 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dd8009800 con 0x55885aa6ca20 2024-09-19T08:07:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3dbffff640 1 -- 172.21.15.98:0/75049063 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f3dd4026450 con 0x55885aa6ca20 2024-09-19T08:07:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.192+0000 7f3de9635740 10 client.16626 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.193+0000 7f3de9635740 10 client.16626 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.193+0000 7f3de9635740 20 client.16626 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.193+0000 7f3de9635740 1 --2- 172.21.15.98:0/75049063 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55885aa8bea0 0x55885aafa120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.193+0000 7f3de9635740 4 client.16626 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:07:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.193+0000 7f3de9635740 1 -- 172.21.15.98:0/75049063 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55885a9d9d00 con 0x55885aa8bea0 2024-09-19T08:07:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.193+0000 7f3ddffff640 1 --2- 172.21.15.98:0/75049063 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55885aa8bea0 0x55885aafa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.193+0000 7f3ddffff640 1 --2- 172.21.15.98:0/75049063 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55885aa8bea0 0x55885aafa120 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52c50b86c1dcc586 server_cookie=97f2b4ec1ebf779f in_seq=0 out_seq=0 2024-09-19T08:07:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.193+0000 7f3dbffff640 10 client.16626 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.194+0000 7f3dbffff640 1 -- 172.21.15.98:0/75049063 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55885a9d9d00 con 0x55885aa8bea0 2024-09-19T08:07:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.194+0000 7f3dbffff640 10 client.16626 handle_command_reply: tid=0 2024-09-19T08:07:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.194+0000 7f3de9635740 1 client.16626 shutdown 2024-09-19T08:07:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.194+0000 7f3de9635740 1 -- 172.21.15.98:0/75049063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa6ca20 msgr2=0x55885aa84620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:13.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.194+0000 7f3de9635740 1 --2- 172.21.15.98:0/75049063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa6ca20 0x55885aa84620 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4005fd0 tx=0x7f3dd401fa50 comp rx=0 tx=0).stop 2024-09-19T08:07:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.194+0000 7f3de9635740 1 -- 172.21.15.98:0/75049063 shutdown_connections 2024-09-19T08:07:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.194+0000 7f3de9635740 1 --2- 172.21.15.98:0/75049063 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55885aa8bea0 0x55885aafa120 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.195+0000 7f3de9635740 1 --2- 172.21.15.98:0/75049063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55885aa7c150 0x55885aa81640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.195+0000 7f3dbffff640 0 client.16626 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.195+0000 7f3de9635740 1 --2- 172.21.15.98:0/75049063 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55885aa6d340 0x55885aa81100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.195+0000 7f3de9635740 1 --2- 172.21.15.98:0/75049063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55885aa6ca20 0x55885aa84620 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.195+0000 7f3de9635740 1 -- 172.21.15.98:0/75049063 >> 172.21.15.98:0/75049063 conn(0x55885aa690f0 msgr2=0x55885aa68270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.195+0000 7f3de9635740 1 -- 172.21.15.98:0/75049063 shutdown_connections 2024-09-19T08:07:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.195+0000 7f3de9635740 1 -- 172.21.15.98:0/75049063 wait complete. 2024-09-19T08:07:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.195+0000 7f3de9635740 20 client.16626 trim_cache size 0 max 16384 2024-09-19T08:07:13.199 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:07:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:07:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "4e6388ce-428e-46b7-a8c7-482948457f5a", 2024-09-19T08:07:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:07:13.200 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:07:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.195+0000 7f3de725c640 1 -- 172.21.15.98:0/2940255423 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3dd00a3430 msgr2=0x7f3dd00a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.195+0000 7f3de725c640 1 --2- 172.21.15.98:0/2940255423 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3dd00a3430 0x7f3dd00a58f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8021000 tx=0x7f3dd8020000 comp rx=0 tx=0).stop 2024-09-19T08:07:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.195+0000 7f3de725c640 1 -- 172.21.15.98:0/2940255423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de01027a0 msgr2=0x7f3de0198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.195+0000 7f3de725c640 1 --2- 172.21.15.98:0/2940255423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de01027a0 0x7f3de0198940 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc006010 tx=0x7f3dcc0042f0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.195+0000 7f3de725c640 1 -- 172.21.15.98:0/2940255423 shutdown_connections 2024-09-19T08:07:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.195+0000 7f3de725c640 1 --2- 172.21.15.98:0/2940255423 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3dd00a3430 0x7f3dd00a58f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.196+0000 7f3de725c640 1 --2- 172.21.15.98:0/2940255423 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de0106f20 0x7f3de019ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.196+0000 7f3de725c640 1 --2- 172.21.15.98:0/2940255423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de01027a0 0x7f3de0198940 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.196+0000 7f3de725c640 1 --2- 172.21.15.98:0/2940255423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3de0101df0 0x7f3de0198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.196+0000 7f3de725c640 1 -- 172.21.15.98:0/2940255423 >> 172.21.15.98:0/2940255423 conn(0x7f3de00fd8b0 msgr2=0x7f3de00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.196+0000 7f3de725c640 1 -- 172.21.15.98:0/2940255423 shutdown_connections 2024-09-19T08:07:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:13.196+0000 7f3de725c640 1 -- 172.21.15.98:0/2940255423 wait complete. 2024-09-19T08:07:13.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:13 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:07:13.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:13 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:07:13.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:13 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:07:13.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:13 smithi139 ceph-mon[34874]: pgmap v129: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:07:13.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:13 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:07:13.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:13 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:07:13.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:13 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:07:13.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:13 smithi098 ceph-mon[31333]: pgmap v129: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:07:13.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:13 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:07:13.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:13 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:07:13.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:13 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:07:13.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:13 smithi162 ceph-mon[35260]: pgmap v129: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:07:13.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:07:14.162 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:14.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:14 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:07:14.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:14 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:14.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:14 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:07:14.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:14 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.590+0000 7fc59b7ec640 1 -- 172.21.15.98:0/3582955490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc594106fc0 msgr2=0x7fc59410eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.590+0000 7fc59b7ec640 1 --2- 172.21.15.98:0/3582955490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc594106fc0 0x7fc59410eaf0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009930 tx=0x7fc58c02a420 comp rx=0 tx=0).stop 2024-09-19T08:07:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.590+0000 7fc59b7ec640 1 -- 172.21.15.98:0/3582955490 shutdown_connections 2024-09-19T08:07:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.590+0000 7fc59b7ec640 1 --2- 172.21.15.98:0/3582955490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc594106fc0 0x7fc59410eaf0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.590+0000 7fc59b7ec640 1 --2- 172.21.15.98:0/3582955490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc594100d20 0x7fc594106860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.590+0000 7fc59b7ec640 1 --2- 172.21.15.98:0/3582955490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc594100370 0x7fc594100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.590+0000 7fc59b7ec640 1 -- 172.21.15.98:0/3582955490 >> 172.21.15.98:0/3582955490 conn(0x7fc594077210 msgr2=0x7fc5940fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.590+0000 7fc59b7ec640 1 -- 172.21.15.98:0/3582955490 shutdown_connections 2024-09-19T08:07:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.591+0000 7fc59b7ec640 1 -- 172.21.15.98:0/3582955490 wait complete. 2024-09-19T08:07:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc59b7ec640 1 Processor -- start 2024-09-19T08:07:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc59b7ec640 1 -- start start 2024-09-19T08:07:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc59b7ec640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc594100370 0x7fc5941012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc59b7ec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc594100d20 0x7fc594101830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc59b7ec640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc594106fc0 0x7fc594101d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc59b7ec640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc594111fa0 con 0x7fc594100d20 2024-09-19T08:07:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc59b7ec640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc594111e20 con 0x7fc594100370 2024-09-19T08:07:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc59b7ec640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc594112120 con 0x7fc594106fc0 2024-09-19T08:07:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc598d87640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc594100d20 0x7fc594101830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc598d87640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc594100d20 0x7fc594101830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34712/0 (socket says 172.21.15.98:34712) 2024-09-19T08:07:14.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc598d87640 1 -- 172.21.15.98:0/3763374088 learned_addr learned my addr 172.21.15.98:0/3763374088 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:14.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc599d89640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc594106fc0 0x7fc594101d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc599588640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc594100370 0x7fc5941012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc598d87640 1 -- 172.21.15.98:0/3763374088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc594106fc0 msgr2=0x7fc594101d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:14.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc598d87640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc594106fc0 0x7fc594101d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc598d87640 1 -- 172.21.15.98:0/3763374088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc594100370 msgr2=0x7fc5941012f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:14.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc598d87640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc594100370 0x7fc5941012f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc598d87640 1 -- 172.21.15.98:0/3763374088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5941a6880 con 0x7fc594100d20 2024-09-19T08:07:14.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc599588640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc594100370 0x7fc5941012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:14.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.592+0000 7fc598d87640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc594100d20 0x7fc594101830 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc58800c970 tx=0x7fc58800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:14.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.593+0000 7fc5927fc640 1 -- 172.21.15.98:0/3763374088 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc588007bb0 con 0x7fc594100d20 2024-09-19T08:07:14.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.593+0000 7fc59b7ec640 1 -- 172.21.15.98:0/3763374088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5941a6b70 con 0x7fc594100d20 2024-09-19T08:07:14.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.593+0000 7fc5927fc640 1 -- 172.21.15.98:0/3763374088 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc588007d50 con 0x7fc594100d20 2024-09-19T08:07:14.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.593+0000 7fc5927fc640 1 -- 172.21.15.98:0/3763374088 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58800f450 con 0x7fc594100d20 2024-09-19T08:07:14.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.593+0000 7fc59b7ec640 1 -- 172.21.15.98:0/3763374088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5941a7100 con 0x7fc594100d20 2024-09-19T08:07:14.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.595+0000 7fc5927fc640 1 -- 172.21.15.98:0/3763374088 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fc588004f90 con 0x7fc594100d20 2024-09-19T08:07:14.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.595+0000 7fc59dbc5740 1 Processor -- start 2024-09-19T08:07:14.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.595+0000 7fc59dbc5740 1 -- start start 2024-09-19T08:07:14.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc59dbc5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a3fb4a0 0x56226a3fb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc59dbc5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fbdc0 0x56226a406140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc59dbc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a406680 0x56226a408a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc59dbc5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56226a3612e0 con 0x56226a3fbdc0 2024-09-19T08:07:14.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc59dbc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56226a308ce0 con 0x56226a406680 2024-09-19T08:07:14.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc59dbc5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56226a35d2a0 con 0x56226a3fb4a0 2024-09-19T08:07:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc599588640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fbdc0 0x56226a406140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc599588640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fbdc0 0x56226a406140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34722/0 (socket says 172.21.15.98:34722) 2024-09-19T08:07:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc599588640 1 -- 172.21.15.98:0/2796853674 learned_addr learned my addr 172.21.15.98:0/2796853674 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc599d89640 1 --2- 172.21.15.98:0/2796853674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a3fb4a0 0x56226a3fb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc598d87640 1 --2- 172.21.15.98:0/2796853674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a406680 0x56226a408a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc599588640 1 -- 172.21.15.98:0/2796853674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a3fb4a0 msgr2=0x56226a3fb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc599588640 1 --2- 172.21.15.98:0/2796853674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a3fb4a0 0x56226a3fb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc5927fc640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc57c0a33b0 0x7fc57c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc599588640 1 -- 172.21.15.98:0/2796853674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a406680 msgr2=0x56226a408a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc599588640 1 --2- 172.21.15.98:0/2796853674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a406680 0x56226a408a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc599588640 1 -- 172.21.15.98:0/2796853674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56226a2ed440 con 0x56226a3fbdc0 2024-09-19T08:07:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc598d87640 1 --2- 172.21.15.98:0/2796853674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a406680 0x56226a408a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc599d89640 1 --2- 172.21.15.98:0/2796853674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a3fb4a0 0x56226a3fb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc5927fc640 1 -- 172.21.15.98:0/3763374088 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fc5880cfa40 con 0x7fc594100d20 2024-09-19T08:07:14.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc599d89640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc57c0a33b0 0x7fc57c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.596+0000 7fc599588640 1 --2- 172.21.15.98:0/2796853674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fbdc0 0x56226a406140 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fc58400f3e0 tx=0x7fc58400f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209c26b7d4f3bc4a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:14.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.597+0000 7fc57b7fe640 1 -- 172.21.15.98:0/2796853674 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc584004d20 con 0x56226a3fbdc0 2024-09-19T08:07:14.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.597+0000 7fc57b7fe640 1 -- 172.21.15.98:0/2796853674 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5840119e0 con 0x56226a3fbdc0 2024-09-19T08:07:14.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.597+0000 7fc57b7fe640 1 -- 172.21.15.98:0/2796853674 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5840055a0 con 0x56226a3fbdc0 2024-09-19T08:07:14.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.597+0000 7fc59dbc5740 1 -- 172.21.15.98:0/2796853674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fbdc0 msgr2=0x56226a406140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:14.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.597+0000 7fc599d89640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc57c0a33b0 0x7fc57c0a5870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc58c006010 tx=0x7fc58c0095c0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:14.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.597+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/2796853674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fbdc0 0x56226a406140 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fc58400f3e0 tx=0x7fc58400f8a0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.597+0000 7fc59dbc5740 1 -- 172.21.15.98:0/2796853674 shutdown_connections 2024-09-19T08:07:14.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.597+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/2796853674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a406680 0x56226a408a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.597+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/2796853674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fbdc0 0x56226a406140 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.597+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/2796853674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a3fb4a0 0x56226a3fb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.597+0000 7fc59dbc5740 1 -- 172.21.15.98:0/2796853674 >> 172.21.15.98:0/2796853674 conn(0x56226a3f7a30 msgr2=0x56226a3f7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:14.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.597+0000 7fc59dbc5740 1 -- 172.21.15.98:0/2796853674 shutdown_connections 2024-09-19T08:07:14.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.597+0000 7fc59dbc5740 1 -- 172.21.15.98:0/2796853674 wait complete. 2024-09-19T08:07:14.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.598+0000 7fc59dbc5740 1 -- start start 2024-09-19T08:07:14.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.598+0000 7fc59dbc5740 1 Processor -- start 2024-09-19T08:07:14.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.598+0000 7fc59dbc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fb4a0 0x56226a405f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.598+0000 7fc59dbc5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fbdc0 0x56226a4031b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.598+0000 7fc59dbc5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a406680 0x56226a4036f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.598+0000 7fc59dbc5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56226a409250 con 0x56226a3fbdc0 2024-09-19T08:07:14.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.598+0000 7fc59dbc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56226a31c050 con 0x56226a3fb4a0 2024-09-19T08:07:14.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.598+0000 7fc59dbc5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56226a4093d0 con 0x56226a406680 2024-09-19T08:07:14.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.598+0000 7fc59dbc5740 10 client.0 _ll_register_callbacks cb 0x56226a35c160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.598+0000 7fc598d87640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fbdc0 0x56226a4031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.598+0000 7fc598d87640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fbdc0 0x56226a4031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34730/0 (socket says 172.21.15.98:34730) 2024-09-19T08:07:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.598+0000 7fc598d87640 1 -- 172.21.15.98:0/447356195 learned_addr learned my addr 172.21.15.98:0/447356195 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc599d89640 1 --2- 172.21.15.98:0/447356195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a406680 0x56226a4036f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc598d87640 1 -- 172.21.15.98:0/447356195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a406680 msgr2=0x56226a4036f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc598d87640 1 --2- 172.21.15.98:0/447356195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a406680 0x56226a4036f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc599588640 1 --2- 172.21.15.98:0/447356195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fb4a0 0x56226a405f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc598d87640 1 -- 172.21.15.98:0/447356195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fb4a0 msgr2=0x56226a405f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc598d87640 1 --2- 172.21.15.98:0/447356195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fb4a0 0x56226a405f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc598d87640 1 -- 172.21.15.98:0/447356195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56226a0d23c0 con 0x56226a3fbdc0 2024-09-19T08:07:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc599d89640 1 --2- 172.21.15.98:0/447356195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a406680 0x56226a4036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc598d87640 1 --2- 172.21.15.98:0/447356195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fbdc0 0x56226a4031b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc588007590 tx=0x7fc588098050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a26286cedb7956f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc578ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:14.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc578ff9640 1 -- 172.21.15.98:0/447356195 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58801d040 con 0x56226a3fbdc0 2024-09-19T08:07:14.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc578ff9640 1 -- 172.21.15.98:0/447356195 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fc58809b140 con 0x56226a3fbdc0 2024-09-19T08:07:14.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc59dbc5740 1 -- 172.21.15.98:0/447356195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5840119e0 con 0x56226a3fbdc0 2024-09-19T08:07:14.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc578ff9640 1 -- 172.21.15.98:0/447356195 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc5880cd5e0 con 0x56226a3fbdc0 2024-09-19T08:07:14.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc59dbc5740 10 client.16638 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:14.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc59dbc5740 1 -- 172.21.15.98:0/447356195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56226a414b90 con 0x56226a3fbdc0 2024-09-19T08:07:14.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc578ff9640 1 -- 172.21.15.98:0/447356195 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5840119e0 con 0x56226a3fbdc0 2024-09-19T08:07:14.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc578ff9640 1 -- 172.21.15.98:0/447356195 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fc5880b3080 con 0x56226a3fbdc0 2024-09-19T08:07:14.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc59dbc5740 10 client.16638 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:14.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.599+0000 7fc59dbc5740 10 client.16638 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:14.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.600+0000 7fc59dbc5740 20 client.16638 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:14.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.600+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/447356195 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56226a416050 0x56226a418460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.600+0000 7fc59dbc5740 4 client.16638 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.600+0000 7fc59dbc5740 1 -- 172.21.15.98:0/447356195 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56226a36e850 con 0x56226a416050 2024-09-19T08:07:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.600+0000 7fc599588640 1 --2- 172.21.15.98:0/447356195 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56226a416050 0x56226a418460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.600+0000 7fc578ff9640 10 client.16638 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.600+0000 7fc599588640 1 --2- 172.21.15.98:0/447356195 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56226a416050 0x56226a418460 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eba9ac42f235d83c server_cookie=7850dae2a7633247 in_seq=0 out_seq=0 2024-09-19T08:07:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc578ff9640 1 -- 172.21.15.98:0/447356195 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56226a36e850 con 0x56226a416050 2024-09-19T08:07:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc578ff9640 10 client.16638 handle_command_reply: tid=0 2024-09-19T08:07:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc59dbc5740 1 client.16638 shutdown 2024-09-19T08:07:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc59dbc5740 1 -- 172.21.15.98:0/447356195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fbdc0 msgr2=0x56226a4031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/447356195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fbdc0 0x56226a4031b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc588007590 tx=0x7fc588098050 comp rx=0 tx=0).stop 2024-09-19T08:07:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc59dbc5740 1 -- 172.21.15.98:0/447356195 shutdown_connections 2024-09-19T08:07:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/447356195 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56226a416050 0x56226a418460 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/447356195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a406680 0x56226a4036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc578ff9640 0 client.16638 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/447356195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fbdc0 0x56226a4031b0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/447356195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fb4a0 0x56226a405f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc59dbc5740 1 -- 172.21.15.98:0/447356195 >> 172.21.15.98:0/447356195 conn(0x56226a3f7a30 msgr2=0x56226a3fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc59dbc5740 1 -- 172.21.15.98:0/447356195 shutdown_connections 2024-09-19T08:07:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc59dbc5740 1 -- 172.21.15.98:0/447356195 wait complete. 2024-09-19T08:07:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.601+0000 7fc59dbc5740 20 client.16638 trim_cache size 0 max 16384 2024-09-19T08:07:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc59dbc5740 1 Processor -- start 2024-09-19T08:07:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc59dbc5740 1 -- start start 2024-09-19T08:07:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc59dbc5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a3fb4a0 0x56226a41dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc59dbc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fbdc0 0x56226a41e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc59dbc5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a406680 0x56226a400430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc59dbc5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc5941af9b0 con 0x56226a406680 2024-09-19T08:07:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc59dbc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc5941b1550 con 0x56226a3fbdc0 2024-09-19T08:07:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc59dbc5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc5941b0080 con 0x56226a3fb4a0 2024-09-19T08:07:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc599d89640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a406680 0x56226a400430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc599d89640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a406680 0x56226a400430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34734/0 (socket says 172.21.15.98:34734) 2024-09-19T08:07:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc599d89640 1 -- 172.21.15.98:0/173715334 learned_addr learned my addr 172.21.15.98:0/173715334 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc599588640 1 --2- 172.21.15.98:0/173715334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a3fb4a0 0x56226a41dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc599d89640 1 -- 172.21.15.98:0/173715334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a3fb4a0 msgr2=0x56226a41dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc599d89640 1 --2- 172.21.15.98:0/173715334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a3fb4a0 0x56226a41dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc599d89640 1 -- 172.21.15.98:0/173715334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fbdc0 msgr2=0x56226a41e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc599d89640 1 --2- 172.21.15.98:0/173715334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fbdc0 0x56226a41e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc599d89640 1 -- 172.21.15.98:0/173715334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56226a403f70 con 0x56226a406680 2024-09-19T08:07:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.619+0000 7fc599d89640 1 --2- 172.21.15.98:0/173715334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a406680 0x56226a400430 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009930 tx=0x7fc58c03e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39cac7cff8ddf0b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.620+0000 7fc57affd640 1 -- 172.21.15.98:0/173715334 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c035040 con 0x56226a406680 2024-09-19T08:07:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.620+0000 7fc57affd640 1 -- 172.21.15.98:0/173715334 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc58c03c440 con 0x56226a406680 2024-09-19T08:07:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.620+0000 7fc57affd640 1 -- 172.21.15.98:0/173715334 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c049680 con 0x56226a406680 2024-09-19T08:07:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.620+0000 7fc59dbc5740 1 -- 172.21.15.98:0/173715334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a406680 msgr2=0x56226a400430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.620+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/173715334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a406680 0x56226a400430 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009930 tx=0x7fc58c03e000 comp rx=0 tx=0).stop 2024-09-19T08:07:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.620+0000 7fc59dbc5740 1 -- 172.21.15.98:0/173715334 shutdown_connections 2024-09-19T08:07:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.620+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/173715334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a406680 0x56226a400430 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.620+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/173715334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fbdc0 0x56226a41e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.620+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/173715334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a3fb4a0 0x56226a41dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.620+0000 7fc59dbc5740 1 -- 172.21.15.98:0/173715334 >> 172.21.15.98:0/173715334 conn(0x56226a3f7a30 msgr2=0x56226a405230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.620+0000 7fc59dbc5740 1 -- 172.21.15.98:0/173715334 shutdown_connections 2024-09-19T08:07:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.620+0000 7fc59dbc5740 1 -- 172.21.15.98:0/173715334 wait complete. 2024-09-19T08:07:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.621+0000 7fc59dbc5740 1 -- start start 2024-09-19T08:07:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.621+0000 7fc59dbc5740 1 Processor -- start 2024-09-19T08:07:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.621+0000 7fc59dbc5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fb4a0 0x56226a40cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.621+0000 7fc59dbc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fbdc0 0x56226a40d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.621+0000 7fc59dbc5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a406680 0x56226a47fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.621+0000 7fc59dbc5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc5941b0e50 con 0x56226a3fb4a0 2024-09-19T08:07:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.621+0000 7fc59dbc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc5941b0780 con 0x56226a3fbdc0 2024-09-19T08:07:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.621+0000 7fc59dbc5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc5941ad990 con 0x56226a406680 2024-09-19T08:07:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.621+0000 7fc59dbc5740 10 client.0 _ll_register_callbacks cb 0x7fc58800c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.621+0000 7fc599588640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fb4a0 0x56226a40cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.621+0000 7fc599588640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fb4a0 0x56226a40cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34740/0 (socket says 172.21.15.98:34740) 2024-09-19T08:07:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.621+0000 7fc599588640 1 -- 172.21.15.98:0/566162371 learned_addr learned my addr 172.21.15.98:0/566162371 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.621+0000 7fc599d89640 1 --2- 172.21.15.98:0/566162371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a406680 0x56226a47fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc599588640 1 -- 172.21.15.98:0/566162371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a406680 msgr2=0x56226a47fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc599588640 1 --2- 172.21.15.98:0/566162371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a406680 0x56226a47fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc598d87640 1 --2- 172.21.15.98:0/566162371 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fbdc0 0x56226a40d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc599588640 1 -- 172.21.15.98:0/566162371 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fbdc0 msgr2=0x56226a40d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc599588640 1 --2- 172.21.15.98:0/566162371 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fbdc0 0x56226a40d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc599588640 1 -- 172.21.15.98:0/566162371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56226a480200 con 0x56226a3fb4a0 2024-09-19T08:07:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc598d87640 1 --2- 172.21.15.98:0/566162371 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fbdc0 0x56226a40d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc599588640 1 --2- 172.21.15.98:0/566162371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fb4a0 0x56226a40cae0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc584009e40 tx=0x7fc584005160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613dac21aaea922c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc578ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc578ff9640 1 -- 172.21.15.98:0/566162371 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58402d060 con 0x56226a3fb4a0 2024-09-19T08:07:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc578ff9640 1 -- 172.21.15.98:0/566162371 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fc584003d80 con 0x56226a3fb4a0 2024-09-19T08:07:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc59dbc5740 1 -- 172.21.15.98:0/566162371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc58c03c440 con 0x56226a3fb4a0 2024-09-19T08:07:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc578ff9640 1 -- 172.21.15.98:0/566162371 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc584004a00 con 0x56226a3fb4a0 2024-09-19T08:07:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc59dbc5740 10 client.16644 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc59dbc5740 1 -- 172.21.15.98:0/566162371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56226a4814e0 con 0x56226a3fb4a0 2024-09-19T08:07:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc578ff9640 1 -- 172.21.15.98:0/566162371 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc58c03c440 con 0x56226a3fb4a0 2024-09-19T08:07:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc578ff9640 1 -- 172.21.15.98:0/566162371 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fc584005300 con 0x56226a3fb4a0 2024-09-19T08:07:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc59dbc5740 10 client.16644 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc59dbc5740 10 client.16644 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.622+0000 7fc59dbc5740 20 client.16644 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/566162371 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56226a481930 0x56226a483d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc59dbc5740 4 client.16644 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:07:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc59dbc5740 1 -- 172.21.15.98:0/566162371 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56226a368d60 con 0x56226a481930 2024-09-19T08:07:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc598d87640 1 --2- 172.21.15.98:0/566162371 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56226a481930 0x56226a483d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc578ff9640 10 client.16644 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc598d87640 1 --2- 172.21.15.98:0/566162371 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56226a481930 0x56226a483d50 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3461fac947d2208 server_cookie=19066c6d8558c22 in_seq=0 out_seq=0 2024-09-19T08:07:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc578ff9640 1 -- 172.21.15.98:0/566162371 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56226a368d60 con 0x56226a481930 2024-09-19T08:07:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc578ff9640 10 client.16644 handle_command_reply: tid=0 2024-09-19T08:07:14.643 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:07:14.643 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-19T08:07:14.643 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-09-19T08:07:14.643 INFO:teuthology.orchestra.run.smithi098.stdout: "4e6388ce-428e-46b7-a8c7-482948457f5a": { 2024-09-19T08:07:14.643 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-09-19T08:07:14.643 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "4e6388ce-428e-46b7-a8c7-482948457f5a", 2024-09-19T08:07:14.643 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-09-19T08:07:14.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:07:14.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:07:14.643 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:07:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc59dbc5740 1 client.16644 shutdown 2024-09-19T08:07:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc59dbc5740 1 -- 172.21.15.98:0/566162371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fb4a0 msgr2=0x56226a40cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/566162371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fb4a0 0x56226a40cae0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc584009e40 tx=0x7fc584005160 comp rx=0 tx=0).stop 2024-09-19T08:07:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc59dbc5740 1 -- 172.21.15.98:0/566162371 shutdown_connections 2024-09-19T08:07:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/566162371 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x56226a481930 0x56226a483d50 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/566162371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56226a406680 0x56226a47fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/566162371 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56226a3fbdc0 0x56226a40d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc578ff9640 0 client.16644 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc59dbc5740 1 --2- 172.21.15.98:0/566162371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56226a3fb4a0 0x56226a40cae0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.623+0000 7fc59dbc5740 1 -- 172.21.15.98:0/566162371 >> 172.21.15.98:0/566162371 conn(0x56226a3f7a30 msgr2=0x56226a3f6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59dbc5740 1 -- 172.21.15.98:0/566162371 shutdown_connections 2024-09-19T08:07:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59dbc5740 1 -- 172.21.15.98:0/566162371 wait complete. 2024-09-19T08:07:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59dbc5740 20 client.16644 trim_cache size 0 max 16384 2024-09-19T08:07:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59b7ec640 1 -- 172.21.15.98:0/3763374088 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc57c0a33b0 msgr2=0x7fc57c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59b7ec640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc57c0a33b0 0x7fc57c0a5870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc58c006010 tx=0x7fc58c0095c0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59b7ec640 1 -- 172.21.15.98:0/3763374088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc594100d20 msgr2=0x7fc594101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59b7ec640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc594100d20 0x7fc594101830 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc58800c970 tx=0x7fc58800ce30 comp rx=0 tx=0).stop 2024-09-19T08:07:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59b7ec640 1 -- 172.21.15.98:0/3763374088 shutdown_connections 2024-09-19T08:07:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59b7ec640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc57c0a33b0 0x7fc57c0a5870 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59b7ec640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc594106fc0 0x7fc594101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59b7ec640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc594100d20 0x7fc594101830 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59b7ec640 1 --2- 172.21.15.98:0/3763374088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc594100370 0x7fc5941012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59b7ec640 1 -- 172.21.15.98:0/3763374088 >> 172.21.15.98:0/3763374088 conn(0x7fc594077210 msgr2=0x7fc5940fc760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59b7ec640 1 -- 172.21.15.98:0/3763374088 shutdown_connections 2024-09-19T08:07:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:14.624+0000 7fc59b7ec640 1 -- 172.21.15.98:0/3763374088 wait complete. 2024-09-19T08:07:14.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:14 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:07:14.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:14 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:14.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:14 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:07:14.678 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:14 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:14.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:14 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:07:14.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:14 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:14.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:14 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:07:14.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:14 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:15.377 INFO:tasks.cephfs.filesystem:scrub status for tag:4e6388ce-428e-46b7-a8c7-482948457f5a - {'path': '/', 'tag': '4e6388ce-428e-46b7-a8c7-482948457f5a', 'options': 'recursive,force'} 2024-09-19T08:07:15.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:15 smithi139 ceph-mon[34874]: pgmap v130: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-19T08:07:15.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:15 smithi098 ceph-mon[31333]: pgmap v130: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-19T08:07:15.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:15 smithi162 ceph-mon[35260]: pgmap v130: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-19T08:07:16.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:16 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:07:16.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:16 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:07:16.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:16 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:07:17.677 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:17 smithi139 ceph-mon[34874]: pgmap v131: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-19T08:07:17.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:17 smithi098 ceph-mon[31333]: pgmap v131: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-19T08:07:17.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:17 smithi162 ceph-mon[35260]: pgmap v131: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-19T08:07:19.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:18 smithi139 ceph-mon[34874]: pgmap v132: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-19T08:07:19.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:18 smithi098 ceph-mon[31333]: pgmap v132: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-19T08:07:19.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:18 smithi162 ceph-mon[35260]: pgmap v132: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-19T08:07:21.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:20 smithi139 ceph-mon[34874]: pgmap v133: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:07:21.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:20 smithi098 ceph-mon[31333]: pgmap v133: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:07:21.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:20 smithi162 ceph-mon[35260]: pgmap v133: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:07:23.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:22 smithi139 ceph-mon[34874]: pgmap v134: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:07:23.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:22 smithi098 ceph-mon[31333]: pgmap v134: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:07:23.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:22 smithi162 ceph-mon[35260]: pgmap v134: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:07:25.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:24 smithi139 ceph-mon[34874]: pgmap v135: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:07:25.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:24 smithi098 ceph-mon[31333]: pgmap v135: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:07:25.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:24 smithi162 ceph-mon[35260]: pgmap v135: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:07:27.184 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:26 smithi139 ceph-mon[34874]: pgmap v136: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-09-19T08:07:27.188 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:26 smithi098 ceph-mon[31333]: pgmap v136: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-09-19T08:07:27.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:26 smithi162 ceph-mon[35260]: pgmap v136: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-09-19T08:07:29.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:28 smithi139 ceph-mon[34874]: pgmap v137: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-19T08:07:29.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:28 smithi098 ceph-mon[31333]: pgmap v137: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-19T08:07:29.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:28 smithi162 ceph-mon[35260]: pgmap v137: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-19T08:07:31.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:30 smithi139 ceph-mon[34874]: pgmap v138: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-19T08:07:31.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:30 smithi098 ceph-mon[31333]: pgmap v138: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-19T08:07:31.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:30 smithi162 ceph-mon[35260]: pgmap v138: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-19T08:07:33.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:32 smithi139 ceph-mon[34874]: pgmap v139: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-19T08:07:33.184 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:32 smithi098 ceph-mon[31333]: pgmap v139: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-19T08:07:33.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:32 smithi162 ceph-mon[35260]: pgmap v139: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-19T08:07:35.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:34 smithi139 ceph-mon[34874]: pgmap v140: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-19T08:07:35.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:34 smithi098 ceph-mon[31333]: pgmap v140: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-19T08:07:35.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:34 smithi162 ceph-mon[35260]: pgmap v140: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-19T08:07:37.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:36 smithi139 ceph-mon[34874]: pgmap v141: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-19T08:07:37.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:36 smithi098 ceph-mon[31333]: pgmap v141: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-19T08:07:37.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:36 smithi162 ceph-mon[35260]: pgmap v141: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-19T08:07:39.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:38 smithi139 ceph-mon[34874]: pgmap v142: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-19T08:07:39.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:38 smithi098 ceph-mon[31333]: pgmap v142: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-19T08:07:39.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:38 smithi162 ceph-mon[35260]: pgmap v142: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-19T08:07:42.182 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:41 smithi139 ceph-mon[34874]: pgmap v143: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:42.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:41 smithi098 ceph-mon[31333]: pgmap v143: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:42.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:41 smithi162 ceph-mon[35260]: pgmap v143: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:44.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:43 smithi139 ceph-mon[34874]: pgmap v144: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:07:44.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:43 smithi098 ceph-mon[31333]: pgmap v144: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:07:44.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:43 smithi162 ceph-mon[35260]: pgmap v144: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:07:45.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:07:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:46.084 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:45 smithi098 ceph-mon[31333]: pgmap v145: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.169+0000 7feb46199640 1 -- 172.21.15.98:0/1037719603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb40100a90 msgr2=0x7feb40100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.169+0000 7feb46199640 1 --2- 172.21.15.98:0/1037719603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb40100a90 0x7feb40100ef0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7feb34009930 tx=0x7feb3402a420 comp rx=0 tx=0).stop 2024-09-19T08:07:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.169+0000 7feb46199640 1 -- 172.21.15.98:0/1037719603 shutdown_connections 2024-09-19T08:07:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.169+0000 7feb46199640 1 --2- 172.21.15.98:0/1037719603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb40101430 0x7feb4010d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.169+0000 7feb46199640 1 --2- 172.21.15.98:0/1037719603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb40100a90 0x7feb40100ef0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.169+0000 7feb46199640 1 --2- 172.21.15.98:0/1037719603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb40106a90 0x7feb40106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.169+0000 7feb46199640 1 -- 172.21.15.98:0/1037719603 >> 172.21.15.98:0/1037719603 conn(0x7feb400fc7d0 msgr2=0x7feb400febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.169+0000 7feb46199640 1 -- 172.21.15.98:0/1037719603 shutdown_connections 2024-09-19T08:07:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.169+0000 7feb46199640 1 -- 172.21.15.98:0/1037719603 wait complete. 2024-09-19T08:07:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.170+0000 7feb46199640 1 Processor -- start 2024-09-19T08:07:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.170+0000 7feb46199640 1 -- start start 2024-09-19T08:07:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.170+0000 7feb46199640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb40100a90 0x7feb40197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.170+0000 7feb46199640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb40101430 0x7feb401978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.170+0000 7feb46199640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb40106a90 0x7feb4019bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.170+0000 7feb46199640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb40110e90 con 0x7feb40106a90 2024-09-19T08:07:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.170+0000 7feb46199640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feb40110d10 con 0x7feb40101430 2024-09-19T08:07:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.170+0000 7feb46199640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb40111010 con 0x7feb40100a90 2024-09-19T08:07:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.170+0000 7feb3ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb40106a90 0x7feb4019bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb3ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb40106a90 0x7feb4019bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58472/0 (socket says 172.21.15.98:58472) 2024-09-19T08:07:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb3ffff640 1 -- 172.21.15.98:0/652529775 learned_addr learned my addr 172.21.15.98:0/652529775 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb3f7fe640 1 --2- 172.21.15.98:0/652529775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb40100a90 0x7feb40197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb3effd640 1 --2- 172.21.15.98:0/652529775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb40101430 0x7feb401978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb3ffff640 1 -- 172.21.15.98:0/652529775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb40100a90 msgr2=0x7feb40197370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb3ffff640 1 --2- 172.21.15.98:0/652529775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb40100a90 0x7feb40197370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb3ffff640 1 -- 172.21.15.98:0/652529775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb40101430 msgr2=0x7feb401978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb3ffff640 1 --2- 172.21.15.98:0/652529775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb40101430 0x7feb401978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb3ffff640 1 -- 172.21.15.98:0/652529775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4019c320 con 0x7feb40106a90 2024-09-19T08:07:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb3ffff640 1 --2- 172.21.15.98:0/652529775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb40106a90 0x7feb4019bc40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7feb3800ebb0 tx=0x7feb38007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb3cff9640 1 -- 172.21.15.98:0/652529775 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb38007ca0 con 0x7feb40106a90 2024-09-19T08:07:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb3cff9640 1 -- 172.21.15.98:0/652529775 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb38007e40 con 0x7feb40106a90 2024-09-19T08:07:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb46199640 1 -- 172.21.15.98:0/652529775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb4019c5b0 con 0x7feb40106a90 2024-09-19T08:07:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb3cff9640 1 -- 172.21.15.98:0/652529775 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb380058a0 con 0x7feb40106a90 2024-09-19T08:07:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.171+0000 7feb46199640 1 -- 172.21.15.98:0/652529775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb4019ca70 con 0x7feb40106a90 2024-09-19T08:07:46.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:45 smithi139 ceph-mon[34874]: pgmap v145: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.174+0000 7feb3cff9640 1 -- 172.21.15.98:0/652529775 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7feb3800c460 con 0x7feb40106a90 2024-09-19T08:07:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.174+0000 7feb48572740 1 Processor -- start 2024-09-19T08:07:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.174+0000 7feb48572740 1 -- start start 2024-09-19T08:07:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.174+0000 7feb48572740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b314a0 0x563547b31880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.174+0000 7feb48572740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b31dc0 0x563547b3c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.174+0000 7feb48572740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b3ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.174+0000 7feb48572740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563547a972e0 con 0x563547b314a0 2024-09-19T08:07:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.174+0000 7feb48572740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563547a3ece0 con 0x563547b31dc0 2024-09-19T08:07:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.174+0000 7feb48572740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563547a932a0 con 0x563547b3c680 2024-09-19T08:07:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b314a0 0x563547b31880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b314a0 0x563547b31880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58482/0 (socket says 172.21.15.98:58482) 2024-09-19T08:07:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3effd640 1 -- 172.21.15.98:0/3806853142 learned_addr learned my addr 172.21.15.98:0/3806853142 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3f7fe640 1 --2- 172.21.15.98:0/3806853142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b31dc0 0x563547b3c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3cff9640 1 --2- 172.21.15.98:0/652529775 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feb280a33b0 0x7feb280a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3effd640 1 --2- 172.21.15.98:0/3806853142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b3ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3effd640 1 -- 172.21.15.98:0/3806853142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 msgr2=0x563547b3ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3effd640 1 --2- 172.21.15.98:0/3806853142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b3ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3effd640 1 -- 172.21.15.98:0/3806853142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b31dc0 msgr2=0x563547b3c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3effd640 1 --2- 172.21.15.98:0/3806853142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b31dc0 0x563547b3c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3effd640 1 -- 172.21.15.98:0/3806853142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563547a23440 con 0x563547b314a0 2024-09-19T08:07:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3ffff640 1 --2- 172.21.15.98:0/652529775 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feb280a33b0 0x7feb280a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3f7fe640 1 --2- 172.21.15.98:0/3806853142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b31dc0 0x563547b3c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3effd640 1 --2- 172.21.15.98:0/3806853142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b314a0 0x563547b31880 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7feb34005c40 tx=0x7feb34005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452478f826732d35 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3cff9640 1 -- 172.21.15.98:0/652529775 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7feb380d11f0 con 0x7feb40106a90 2024-09-19T08:07:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb25ffb640 1 -- 172.21.15.98:0/3806853142 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34035040 con 0x563547b314a0 2024-09-19T08:07:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb3ffff640 1 --2- 172.21.15.98:0/652529775 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feb280a33b0 0x7feb280a5870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feb3809d2b0 tx=0x7feb380998d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb25ffb640 1 -- 172.21.15.98:0/3806853142 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb34005020 con 0x563547b314a0 2024-09-19T08:07:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.175+0000 7feb25ffb640 1 -- 172.21.15.98:0/3806853142 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34004c60 con 0x563547b314a0 2024-09-19T08:07:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.176+0000 7feb48572740 1 -- 172.21.15.98:0/3806853142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b314a0 msgr2=0x563547b31880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.176+0000 7feb48572740 1 --2- 172.21.15.98:0/3806853142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b314a0 0x563547b31880 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7feb34005c40 tx=0x7feb34005fd0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.176+0000 7feb48572740 1 -- 172.21.15.98:0/3806853142 shutdown_connections 2024-09-19T08:07:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.176+0000 7feb48572740 1 --2- 172.21.15.98:0/3806853142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b3ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.176+0000 7feb48572740 1 --2- 172.21.15.98:0/3806853142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b31dc0 0x563547b3c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.176+0000 7feb48572740 1 --2- 172.21.15.98:0/3806853142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b314a0 0x563547b31880 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.176+0000 7feb48572740 1 -- 172.21.15.98:0/3806853142 >> 172.21.15.98:0/3806853142 conn(0x563547b2da30 msgr2=0x563547b2de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.176+0000 7feb48572740 1 -- 172.21.15.98:0/3806853142 shutdown_connections 2024-09-19T08:07:46.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.176+0000 7feb48572740 1 -- 172.21.15.98:0/3806853142 wait complete. 2024-09-19T08:07:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.177+0000 7feb48572740 1 -- start start 2024-09-19T08:07:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.177+0000 7feb48572740 1 Processor -- start 2024-09-19T08:07:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.177+0000 7feb48572740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b314a0 0x563547b3c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.177+0000 7feb48572740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b31dc0 0x563547b392b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.177+0000 7feb48572740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b39830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.177+0000 7feb48572740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563547b3f1f0 con 0x563547b31dc0 2024-09-19T08:07:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.177+0000 7feb48572740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563547a52050 con 0x563547b314a0 2024-09-19T08:07:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.177+0000 7feb48572740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563547b3f370 con 0x563547b3c680 2024-09-19T08:07:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.177+0000 7feb48572740 10 client.0 _ll_register_callbacks cb 0x563547a92160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.177+0000 7feb3f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b31dc0 0x563547b392b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.177+0000 7feb3f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b31dc0 0x563547b392b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58484/0 (socket says 172.21.15.98:58484) 2024-09-19T08:07:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.177+0000 7feb3f7fe640 1 -- 172.21.15.98:0/3480743127 learned_addr learned my addr 172.21.15.98:0/3480743127 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.177+0000 7feb3effd640 1 --2- 172.21.15.98:0/3480743127 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b314a0 0x563547b3c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.178+0000 7feb3effd640 1 --2- 172.21.15.98:0/3480743127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b39830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.178+0000 7feb3f7fe640 1 -- 172.21.15.98:0/3480743127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 msgr2=0x563547b39830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.178+0000 7feb3f7fe640 1 --2- 172.21.15.98:0/3480743127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b39830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.178+0000 7feb3f7fe640 1 -- 172.21.15.98:0/3480743127 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b314a0 msgr2=0x563547b3c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.178+0000 7feb3f7fe640 1 --2- 172.21.15.98:0/3480743127 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b314a0 0x563547b3c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.178+0000 7feb3f7fe640 1 -- 172.21.15.98:0/3480743127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635478083c0 con 0x563547b31dc0 2024-09-19T08:07:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.178+0000 7feb3f7fe640 1 --2- 172.21.15.98:0/3480743127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b31dc0 0x563547b392b0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7feb30012700 tx=0x7feb30012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7573f1d34844e25c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.178+0000 7feb1effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.178+0000 7feb1effd640 1 -- 172.21.15.98:0/3480743127 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3000f920 con 0x563547b31dc0 2024-09-19T08:07:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.178+0000 7feb1effd640 1 -- 172.21.15.98:0/3480743127 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7feb3000fac0 con 0x563547b31dc0 2024-09-19T08:07:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.178+0000 7feb48572740 1 -- 172.21.15.98:0/3480743127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb34005020 con 0x563547b31dc0 2024-09-19T08:07:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.178+0000 7feb1effd640 1 -- 172.21.15.98:0/3480743127 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7feb3000fcc0 con 0x563547b31dc0 2024-09-19T08:07:46.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.178+0000 7feb48572740 10 client.16656 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:46.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.178+0000 7feb48572740 1 -- 172.21.15.98:0/3480743127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563547b4ac40 con 0x563547b31dc0 2024-09-19T08:07:46.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.179+0000 7feb1effd640 1 -- 172.21.15.98:0/3480743127 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb34005020 con 0x563547b31dc0 2024-09-19T08:07:46.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.179+0000 7feb1effd640 1 -- 172.21.15.98:0/3480743127 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7feb30010430 con 0x563547b31dc0 2024-09-19T08:07:46.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.179+0000 7feb48572740 10 client.16656 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:46.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.179+0000 7feb48572740 10 client.16656 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:46.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.179+0000 7feb48572740 20 client.16656 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:46.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.179+0000 7feb48572740 1 --2- 172.21.15.98:0/3480743127 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563547b4c1a0 0x563547b4e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.179+0000 7feb48572740 4 client.16656 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:46.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.179+0000 7feb48572740 1 -- 172.21.15.98:0/3480743127 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563547aa4850 con 0x563547b4c1a0 2024-09-19T08:07:46.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.179+0000 7feb3effd640 1 --2- 172.21.15.98:0/3480743127 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563547b4c1a0 0x563547b4e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.179+0000 7feb3effd640 1 --2- 172.21.15.98:0/3480743127 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563547b4c1a0 0x563547b4e590 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b076d7afd2be385 server_cookie=31e45e52484831d3 in_seq=0 out_seq=0 2024-09-19T08:07:46.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.180+0000 7feb1effd640 10 client.16656 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:46.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.180+0000 7feb1effd640 1 -- 172.21.15.98:0/3480743127 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563547aa4850 con 0x563547b4c1a0 2024-09-19T08:07:46.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.180+0000 7feb1effd640 10 client.16656 handle_command_reply: tid=0 2024-09-19T08:07:46.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.180+0000 7feb48572740 1 client.16656 shutdown 2024-09-19T08:07:46.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.180+0000 7feb48572740 1 -- 172.21.15.98:0/3480743127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b31dc0 msgr2=0x563547b392b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:46.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.180+0000 7feb48572740 1 --2- 172.21.15.98:0/3480743127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b31dc0 0x563547b392b0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7feb30012700 tx=0x7feb30012bc0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.180+0000 7feb48572740 1 -- 172.21.15.98:0/3480743127 shutdown_connections 2024-09-19T08:07:46.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.180+0000 7feb48572740 1 --2- 172.21.15.98:0/3480743127 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563547b4c1a0 0x563547b4e590 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.180+0000 7feb48572740 1 --2- 172.21.15.98:0/3480743127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b39830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.180+0000 7feb48572740 1 --2- 172.21.15.98:0/3480743127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b31dc0 0x563547b392b0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.180+0000 7feb48572740 1 --2- 172.21.15.98:0/3480743127 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b314a0 0x563547b3c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.180+0000 7feb48572740 1 -- 172.21.15.98:0/3480743127 >> 172.21.15.98:0/3480743127 conn(0x563547b2da30 msgr2=0x563547b2cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:46.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.180+0000 7feb1effd640 0 client.16656 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:46.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.181+0000 7feb48572740 1 -- 172.21.15.98:0/3480743127 shutdown_connections 2024-09-19T08:07:46.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.181+0000 7feb48572740 1 -- 172.21.15.98:0/3480743127 wait complete. 2024-09-19T08:07:46.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.181+0000 7feb48572740 20 client.16656 trim_cache size 0 max 16384 2024-09-19T08:07:46.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.197+0000 7feb48572740 1 Processor -- start 2024-09-19T08:07:46.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb48572740 1 -- start start 2024-09-19T08:07:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb48572740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b314a0 0x563547b48450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb48572740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b31dc0 0x563547b3b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb48572740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b53e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb48572740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb401ab600 con 0x563547b314a0 2024-09-19T08:07:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb48572740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feb401ad1a0 con 0x563547b31dc0 2024-09-19T08:07:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb48572740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb401abcd0 con 0x563547b3c680 2024-09-19T08:07:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb3effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b314a0 0x563547b48450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb3effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b314a0 0x563547b48450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58490/0 (socket says 172.21.15.98:58490) 2024-09-19T08:07:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb3effd640 1 -- 172.21.15.98:0/1954510518 learned_addr learned my addr 172.21.15.98:0/1954510518 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb3f7fe640 1 --2- 172.21.15.98:0/1954510518 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b31dc0 0x563547b3b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb3effd640 1 --2- 172.21.15.98:0/1954510518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b53e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb3effd640 1 -- 172.21.15.98:0/1954510518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 msgr2=0x563547b53e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb3effd640 1 --2- 172.21.15.98:0/1954510518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b53e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb3effd640 1 -- 172.21.15.98:0/1954510518 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b31dc0 msgr2=0x563547b3b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb3effd640 1 --2- 172.21.15.98:0/1954510518 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b31dc0 0x563547b3b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb3effd640 1 -- 172.21.15.98:0/1954510518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563547b3bc20 con 0x563547b314a0 2024-09-19T08:07:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.198+0000 7feb3effd640 1 --2- 172.21.15.98:0/1954510518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b314a0 0x563547b48450 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7feb34004810 tx=0x7feb34031740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c657ea9bd27abf63 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.199+0000 7feb257fa640 1 -- 172.21.15.98:0/1954510518 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34035070 con 0x563547b314a0 2024-09-19T08:07:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.199+0000 7feb257fa640 1 -- 172.21.15.98:0/1954510518 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb34004370 con 0x563547b314a0 2024-09-19T08:07:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.199+0000 7feb257fa640 1 -- 172.21.15.98:0/1954510518 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34046720 con 0x563547b314a0 2024-09-19T08:07:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.199+0000 7feb48572740 1 -- 172.21.15.98:0/1954510518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b314a0 msgr2=0x563547b48450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.199+0000 7feb48572740 1 --2- 172.21.15.98:0/1954510518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b314a0 0x563547b48450 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7feb34004810 tx=0x7feb34031740 comp rx=0 tx=0).stop 2024-09-19T08:07:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.199+0000 7feb48572740 1 -- 172.21.15.98:0/1954510518 shutdown_connections 2024-09-19T08:07:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.199+0000 7feb48572740 1 --2- 172.21.15.98:0/1954510518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b53e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.199+0000 7feb48572740 1 --2- 172.21.15.98:0/1954510518 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b31dc0 0x563547b3b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.199+0000 7feb48572740 1 --2- 172.21.15.98:0/1954510518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b314a0 0x563547b48450 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.199+0000 7feb48572740 1 -- 172.21.15.98:0/1954510518 >> 172.21.15.98:0/1954510518 conn(0x563547b2da30 msgr2=0x563547b348b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.200+0000 7feb48572740 1 -- 172.21.15.98:0/1954510518 shutdown_connections 2024-09-19T08:07:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.200+0000 7feb48572740 1 -- 172.21.15.98:0/1954510518 wait complete. 2024-09-19T08:07:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.200+0000 7feb48572740 1 -- start start 2024-09-19T08:07:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.200+0000 7feb48572740 1 Processor -- start 2024-09-19T08:07:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.200+0000 7feb48572740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b314a0 0x563547b44cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.200+0000 7feb48572740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b31dc0 0x563547b35fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.200+0000 7feb48572740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b364e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.200+0000 7feb48572740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb401acaa0 con 0x563547b31dc0 2024-09-19T08:07:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.200+0000 7feb48572740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feb401ac3d0 con 0x563547b314a0 2024-09-19T08:07:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.200+0000 7feb48572740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb401a95e0 con 0x563547b3c680 2024-09-19T08:07:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.200+0000 7feb48572740 10 client.0 _ll_register_callbacks cb 0x7feb300097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb3f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b31dc0 0x563547b35fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb3effd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b314a0 0x563547b44cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb3f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b31dc0 0x563547b35fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58500/0 (socket says 172.21.15.98:58500) 2024-09-19T08:07:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb3effd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b314a0 0x563547b44cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:53008/0 (socket says 172.21.15.98:53008) 2024-09-19T08:07:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb3f7fe640 1 -- 172.21.15.98:0/2476777424 learned_addr learned my addr 172.21.15.98:0/2476777424 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb3effd640 1 --2- 172.21.15.98:0/2476777424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b364e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb3effd640 1 -- 172.21.15.98:0/2476777424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 msgr2=0x563547b364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb3effd640 1 --2- 172.21.15.98:0/2476777424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b364e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb3effd640 1 -- 172.21.15.98:0/2476777424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b31dc0 msgr2=0x563547b35fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb3effd640 1 --2- 172.21.15.98:0/2476777424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b31dc0 0x563547b35fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb3effd640 1 -- 172.21.15.98:0/2476777424 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563547b36ce0 con 0x563547b314a0 2024-09-19T08:07:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb3f7fe640 1 --2- 172.21.15.98:0/2476777424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b31dc0 0x563547b35fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:07:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb3effd640 1 --2- 172.21.15.98:0/2476777424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b314a0 0x563547b44cd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feb34031710 tx=0x7feb34003dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bacd4d9d0e120ff server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb1effd640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-09-19T08:07:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb1effd640 1 -- 172.21.15.98:0/2476777424 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34062020 con 0x563547b314a0 2024-09-19T08:07:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb48572740 1 -- 172.21.15.98:0/2476777424 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb34004370 con 0x563547b314a0 2024-09-19T08:07:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.201+0000 7feb1effd640 1 -- 172.21.15.98:0/2476777424 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7feb34003a00 con 0x563547b314a0 2024-09-19T08:07:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.202+0000 7feb1effd640 1 -- 172.21.15.98:0/2476777424 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7feb34035070 con 0x563547b314a0 2024-09-19T08:07:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.202+0000 7feb48572740 10 client.25685 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.202+0000 7feb48572740 1 -- 172.21.15.98:0/2476777424 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563547bb6dd0 con 0x563547b314a0 2024-09-19T08:07:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.202+0000 7feb1effd640 1 -- 172.21.15.98:0/2476777424 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb34004370 con 0x563547b314a0 2024-09-19T08:07:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.202+0000 7feb1effd640 1 -- 172.21.15.98:0/2476777424 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7feb340544d0 con 0x563547b314a0 2024-09-19T08:07:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.202+0000 7feb48572740 10 client.25685 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.202+0000 7feb48572740 10 client.25685 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.202+0000 7feb48572740 20 client.25685 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.202+0000 7feb48572740 1 --2- 172.21.15.98:0/2476777424 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563547bb70a0 0x563547bb9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.202+0000 7feb48572740 4 client.25685 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:07:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.202+0000 7feb48572740 1 -- 172.21.15.98:0/2476777424 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563547a9ed60 con 0x563547bb70a0 2024-09-19T08:07:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.202+0000 7feb3f7fe640 1 --2- 172.21.15.98:0/2476777424 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563547bb70a0 0x563547bb9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb1effd640 10 client.25685 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb3f7fe640 1 --2- 172.21.15.98:0/2476777424 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563547bb70a0 0x563547bb9490 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88a9a0f54e683d8b server_cookie=5d576d61b8402aac in_seq=0 out_seq=0 2024-09-19T08:07:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb1effd640 1 -- 172.21.15.98:0/2476777424 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563547a9ed60 con 0x563547bb70a0 2024-09-19T08:07:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb1effd640 10 client.25685 handle_command_reply: tid=0 2024-09-19T08:07:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb48572740 1 client.25685 shutdown 2024-09-19T08:07:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb48572740 1 -- 172.21.15.98:0/2476777424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b314a0 msgr2=0x563547b44cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb48572740 1 --2- 172.21.15.98:0/2476777424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b314a0 0x563547b44cd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feb34031710 tx=0x7feb34003dd0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb48572740 1 -- 172.21.15.98:0/2476777424 shutdown_connections 2024-09-19T08:07:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb48572740 1 --2- 172.21.15.98:0/2476777424 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x563547bb70a0 0x563547bb9490 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb48572740 1 --2- 172.21.15.98:0/2476777424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563547b3c680 0x563547b364e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb48572740 1 --2- 172.21.15.98:0/2476777424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563547b31dc0 0x563547b35fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb48572740 1 --2- 172.21.15.98:0/2476777424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563547b314a0 0x563547b44cd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb1effd640 0 client.25685 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb48572740 1 -- 172.21.15.98:0/2476777424 >> 172.21.15.98:0/2476777424 conn(0x563547b2da30 msgr2=0x563547b39390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb48572740 1 -- 172.21.15.98:0/2476777424 shutdown_connections 2024-09-19T08:07:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb48572740 1 -- 172.21.15.98:0/2476777424 wait complete. 2024-09-19T08:07:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.203+0000 7feb48572740 20 client.25685 trim_cache size 0 max 16384 2024-09-19T08:07:46.210 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:07:46.210 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:07:46.210 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:07:46.210 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:07:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.204+0000 7feb46199640 1 -- 172.21.15.98:0/652529775 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feb280a33b0 msgr2=0x7feb280a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.204+0000 7feb46199640 1 --2- 172.21.15.98:0/652529775 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feb280a33b0 0x7feb280a5870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feb3809d2b0 tx=0x7feb380998d0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.204+0000 7feb46199640 1 -- 172.21.15.98:0/652529775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb40106a90 msgr2=0x7feb4019bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.204+0000 7feb46199640 1 --2- 172.21.15.98:0/652529775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb40106a90 0x7feb4019bc40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7feb3800ebb0 tx=0x7feb38007590 comp rx=0 tx=0).stop 2024-09-19T08:07:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.204+0000 7feb46199640 1 -- 172.21.15.98:0/652529775 shutdown_connections 2024-09-19T08:07:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.204+0000 7feb46199640 1 --2- 172.21.15.98:0/652529775 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feb280a33b0 0x7feb280a5870 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.204+0000 7feb46199640 1 --2- 172.21.15.98:0/652529775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb40106a90 0x7feb4019bc40 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.204+0000 7feb46199640 1 --2- 172.21.15.98:0/652529775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb40101430 0x7feb401978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.204+0000 7feb46199640 1 --2- 172.21.15.98:0/652529775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb40100a90 0x7feb40197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.204+0000 7feb46199640 1 -- 172.21.15.98:0/652529775 >> 172.21.15.98:0/652529775 conn(0x7feb400fc7d0 msgr2=0x7feb400fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.204+0000 7feb46199640 1 -- 172.21.15.98:0/652529775 shutdown_connections 2024-09-19T08:07:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:46.204+0000 7feb46199640 1 -- 172.21.15.98:0/652529775 wait complete. 2024-09-19T08:07:46.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:45 smithi162 ceph-mon[35260]: pgmap v145: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:46.985 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:07:46.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:07:47.237 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:47.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.724+0000 7fccdf79a640 1 -- 172.21.15.98:0/607359098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd8102dd0 msgr2=0x7fccd81031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:47.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.724+0000 7fccdf79a640 1 --2- 172.21.15.98:0/607359098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd8102dd0 0x7fccd81031b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fccd0009930 tx=0x7fccd002a420 comp rx=0 tx=0).stop 2024-09-19T08:07:47.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.724+0000 7fccdf79a640 1 -- 172.21.15.98:0/607359098 shutdown_connections 2024-09-19T08:07:47.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.724+0000 7fccdf79a640 1 --2- 172.21.15.98:0/607359098 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccd8108300 0x7fccd810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:47.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.724+0000 7fccdf79a640 1 --2- 172.21.15.98:0/607359098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccd8103780 0x7fccd8107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:47.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.724+0000 7fccdf79a640 1 --2- 172.21.15.98:0/607359098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd8102dd0 0x7fccd81031b0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:47.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.724+0000 7fccdf79a640 1 -- 172.21.15.98:0/607359098 >> 172.21.15.98:0/607359098 conn(0x7fccd80fc7d0 msgr2=0x7fccd80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:47.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.724+0000 7fccdf79a640 1 -- 172.21.15.98:0/607359098 shutdown_connections 2024-09-19T08:07:47.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.724+0000 7fccdf79a640 1 -- 172.21.15.98:0/607359098 wait complete. 2024-09-19T08:07:47.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccdf79a640 1 Processor -- start 2024-09-19T08:07:47.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccdf79a640 1 -- start start 2024-09-19T08:07:47.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccdf79a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccd8102dd0 0x7fccd8198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:47.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccdf79a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccd8103780 0x7fccd8198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:47.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccdf79a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd8108300 0x7fccd819ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:47.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccdf79a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fccd81120b0 con 0x7fccd8108300 2024-09-19T08:07:47.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccdf79a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fccd8111f30 con 0x7fccd8103780 2024-09-19T08:07:47.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccdf79a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fccd8112230 con 0x7fccd8102dd0 2024-09-19T08:07:47.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccddd37640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd8108300 0x7fccd819ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:47.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccddd37640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd8108300 0x7fccd819ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53426/0 (socket says 172.21.15.98:53426) 2024-09-19T08:07:47.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccddd37640 1 -- 172.21.15.98:0/2430794426 learned_addr learned my addr 172.21.15.98:0/2430794426 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:47.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccdd536640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccd8102dd0 0x7fccd8198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:47.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccdcd35640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccd8103780 0x7fccd8198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:47.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccddd37640 1 -- 172.21.15.98:0/2430794426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccd8102dd0 msgr2=0x7fccd8198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:47.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccddd37640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccd8102dd0 0x7fccd8198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:47.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccddd37640 1 -- 172.21.15.98:0/2430794426 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccd8103780 msgr2=0x7fccd8198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:47.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccddd37640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccd8103780 0x7fccd8198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:47.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccddd37640 1 -- 172.21.15.98:0/2430794426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd819d520 con 0x7fccd8108300 2024-09-19T08:07:47.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccdcd35640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccd8103780 0x7fccd8198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:47.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.725+0000 7fccddd37640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd8108300 0x7fccd819ce40 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fccc800cce0 tx=0x7fccc800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:47.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.726+0000 7fccd67fc640 1 -- 172.21.15.98:0/2430794426 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc800f450 con 0x7fccd8108300 2024-09-19T08:07:47.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.726+0000 7fccdf79a640 1 -- 172.21.15.98:0/2430794426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccd819d810 con 0x7fccd8108300 2024-09-19T08:07:47.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.726+0000 7fccd67fc640 1 -- 172.21.15.98:0/2430794426 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccc800fad0 con 0x7fccd8108300 2024-09-19T08:07:47.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.726+0000 7fccd67fc640 1 -- 172.21.15.98:0/2430794426 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc8018c10 con 0x7fccd8108300 2024-09-19T08:07:47.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.726+0000 7fccdf79a640 1 -- 172.21.15.98:0/2430794426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccd81a50f0 con 0x7fccd8108300 2024-09-19T08:07:47.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.727+0000 7fccdf79a640 1 -- 172.21.15.98:0/2430794426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccd806a7b0 con 0x7fccd8108300 2024-09-19T08:07:47.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.728+0000 7fccd67fc640 1 -- 172.21.15.98:0/2430794426 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fccc800f5f0 con 0x7fccd8108300 2024-09-19T08:07:47.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.729+0000 7fccd67fc640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fccc00a33b0 0x7fccc00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:47.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.729+0000 7fccdd536640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fccc00a33b0 0x7fccc00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:47.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.729+0000 7fccd67fc640 1 -- 172.21.15.98:0/2430794426 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fccc80d0400 con 0x7fccd8108300 2024-09-19T08:07:47.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.729+0000 7fccdd536640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fccc00a33b0 0x7fccc00a5870 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fccd0009900 tx=0x7fccd00058d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:47.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.732+0000 7fccd67fc640 1 -- 172.21.15.98:0/2430794426 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fccc8014030 con 0x7fccd8108300 2024-09-19T08:07:47.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.941+0000 7fccdf79a640 1 -- 172.21.15.98:0/2430794426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fccd81031b0 con 0x7fccd8108300 2024-09-19T08:07:47.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.941+0000 7fccd67fc640 1 -- 172.21.15.98:0/2430794426 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11985 (secure 0 0 0) 0x7fccd81031b0 con 0x7fccd8108300 2024-09-19T08:07:47.942 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:07:47.942 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":30,"btime":"2024-09-19T08:05:21:864704+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:05:21.551452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:07:47.942 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 30 2024-09-19T08:07:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.944+0000 7fccdf79a640 1 -- 172.21.15.98:0/2430794426 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fccc00a33b0 msgr2=0x7fccc00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.944+0000 7fccdf79a640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fccc00a33b0 0x7fccc00a5870 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fccd0009900 tx=0x7fccd00058d0 comp rx=0 tx=0).stop 2024-09-19T08:07:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.944+0000 7fccdf79a640 1 -- 172.21.15.98:0/2430794426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd8108300 msgr2=0x7fccd819ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.944+0000 7fccdf79a640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd8108300 0x7fccd819ce40 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fccc800cce0 tx=0x7fccc800a620 comp rx=0 tx=0).stop 2024-09-19T08:07:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.944+0000 7fccdf79a640 1 -- 172.21.15.98:0/2430794426 shutdown_connections 2024-09-19T08:07:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.944+0000 7fccdf79a640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fccc00a33b0 0x7fccc00a5870 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.944+0000 7fccdf79a640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd8108300 0x7fccd819ce40 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.944+0000 7fccdf79a640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccd8103780 0x7fccd8198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.944+0000 7fccdf79a640 1 --2- 172.21.15.98:0/2430794426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccd8102dd0 0x7fccd8198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.944+0000 7fccdf79a640 1 -- 172.21.15.98:0/2430794426 >> 172.21.15.98:0/2430794426 conn(0x7fccd80fc7d0 msgr2=0x7fccd8105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.944+0000 7fccdf79a640 1 -- 172.21.15.98:0/2430794426 shutdown_connections 2024-09-19T08:07:47.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:47.944+0000 7fccdf79a640 1 -- 172.21.15.98:0/2430794426 wait complete. 2024-09-19T08:07:48.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:47 smithi139 ceph-mon[34874]: pgmap v146: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:07:48.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:47 smithi098 ceph-mon[31333]: pgmap v146: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:07:48.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:47 smithi162 ceph-mon[35260]: pgmap v146: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:07:48.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:07:48.967 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:49.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:48 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2430794426' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:07:49.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:48 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2430794426' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:07:49.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:48 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2430794426' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:07:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.449+0000 7f92552ff640 1 -- 172.21.15.98:0/3399690949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92501002a0 msgr2=0x7f92501006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.449+0000 7f92552ff640 1 --2- 172.21.15.98:0/3399690949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92501002a0 0x7f92501006c0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9238009930 tx=0x7f923802a420 comp rx=0 tx=0).stop 2024-09-19T08:07:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.450+0000 7f92552ff640 1 -- 172.21.15.98:0/3399690949 shutdown_connections 2024-09-19T08:07:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.450+0000 7f92552ff640 1 --2- 172.21.15.98:0/3399690949 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9250100c00 0x7f925010d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.450+0000 7f92552ff640 1 --2- 172.21.15.98:0/3399690949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92501002a0 0x7f92501006c0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.450+0000 7f92552ff640 1 --2- 172.21.15.98:0/3399690949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92501062b0 0x7f9250106690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.450+0000 7f92552ff640 1 -- 172.21.15.98:0/3399690949 >> 172.21.15.98:0/3399690949 conn(0x7f92500fc5c0 msgr2=0x7f92500fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.450+0000 7f92552ff640 1 -- 172.21.15.98:0/3399690949 shutdown_connections 2024-09-19T08:07:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.450+0000 7f92552ff640 1 -- 172.21.15.98:0/3399690949 wait complete. 2024-09-19T08:07:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.450+0000 7f92552ff640 1 Processor -- start 2024-09-19T08:07:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.450+0000 7f92552ff640 1 -- start start 2024-09-19T08:07:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.450+0000 7f92552ff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92501002a0 0x7f9250199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f92552ff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9250100c00 0x7f92501998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f92552ff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92501062b0 0x7f92501a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f92552ff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9250110eb0 con 0x7f92501002a0 2024-09-19T08:07:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f92552ff640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9250110d30 con 0x7f9250100c00 2024-09-19T08:07:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f92552ff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9250111030 con 0x7f92501062b0 2024-09-19T08:07:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f924effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92501002a0 0x7f9250199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f924effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92501002a0 0x7f9250199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53434/0 (socket says 172.21.15.98:53434) 2024-09-19T08:07:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f924effd640 1 -- 172.21.15.98:0/3773735356 learned_addr learned my addr 172.21.15.98:0/3773735356 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f92467fc640 1 --2- 172.21.15.98:0/3773735356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9250100c00 0x7f92501998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f924f7fe640 1 --2- 172.21.15.98:0/3773735356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92501062b0 0x7f92501a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f924effd640 1 -- 172.21.15.98:0/3773735356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92501062b0 msgr2=0x7f92501a43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f924effd640 1 --2- 172.21.15.98:0/3773735356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92501062b0 0x7f92501a43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f924effd640 1 -- 172.21.15.98:0/3773735356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9250100c00 msgr2=0x7f92501998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f924effd640 1 --2- 172.21.15.98:0/3773735356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9250100c00 0x7f92501998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f924effd640 1 -- 172.21.15.98:0/3773735356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92501a4920 con 0x7f92501002a0 2024-09-19T08:07:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f924effd640 1 --2- 172.21.15.98:0/3773735356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92501002a0 0x7f9250199360 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f924000c970 tx=0x7f924000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f924cff9640 1 -- 172.21.15.98:0/3773735356 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9240007bb0 con 0x7f92501002a0 2024-09-19T08:07:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f924cff9640 1 -- 172.21.15.98:0/3773735356 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9240007d50 con 0x7f92501002a0 2024-09-19T08:07:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f92552ff640 1 -- 172.21.15.98:0/3773735356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92501a4af0 con 0x7f92501002a0 2024-09-19T08:07:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f924cff9640 1 -- 172.21.15.98:0/3773735356 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924000f450 con 0x7f92501002a0 2024-09-19T08:07:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.451+0000 7f92552ff640 1 -- 172.21.15.98:0/3773735356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92501a4fb0 con 0x7f92501002a0 2024-09-19T08:07:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.453+0000 7f924cff9640 1 -- 172.21.15.98:0/3773735356 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f9240005020 con 0x7f92501002a0 2024-09-19T08:07:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.454+0000 7f92576d8740 1 Processor -- start 2024-09-19T08:07:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.454+0000 7f92576d8740 1 -- start start 2024-09-19T08:07:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.454+0000 7f92576d8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2deb40 0x55f31c2dd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.454+0000 7f92576d8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2dd5d0 0x55f31c2dda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.454+0000 7f92576d8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2ec780 0x55f31c2eeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.454+0000 7f92576d8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f31c2432f0 con 0x55f31c2ec780 2024-09-19T08:07:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.454+0000 7f92576d8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f31c1eacf0 con 0x55f31c2deb40 2024-09-19T08:07:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.454+0000 7f92576d8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f31c23f2b0 con 0x55f31c2dd5d0 2024-09-19T08:07:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f92467fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2ec780 0x55f31c2eeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f92467fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2ec780 0x55f31c2eeb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53446/0 (socket says 172.21.15.98:53446) 2024-09-19T08:07:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f92467fc640 1 -- 172.21.15.98:0/3544885273 learned_addr learned my addr 172.21.15.98:0/3544885273 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f924effd640 1 --2- 172.21.15.98:0/3544885273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2dd5d0 0x55f31c2dda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f924cff9640 1 --2- 172.21.15.98:0/3773735356 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f923c0a33b0 0x7f923c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f924f7fe640 1 --2- 172.21.15.98:0/3544885273 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2deb40 0x55f31c2dd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f92467fc640 1 -- 172.21.15.98:0/3544885273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2dd5d0 msgr2=0x55f31c2dda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f924f7fe640 1 --2- 172.21.15.98:0/3773735356 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f923c0a33b0 0x7f923c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f92467fc640 1 --2- 172.21.15.98:0/3544885273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2dd5d0 0x55f31c2dda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f92467fc640 1 -- 172.21.15.98:0/3544885273 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2deb40 msgr2=0x55f31c2dd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f92467fc640 1 --2- 172.21.15.98:0/3544885273 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2deb40 0x55f31c2dd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f92467fc640 1 -- 172.21.15.98:0/3544885273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f31c1cf3a0 con 0x55f31c2ec780 2024-09-19T08:07:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f924cff9640 1 -- 172.21.15.98:0/3773735356 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f92400cfac0 con 0x7f92501002a0 2024-09-19T08:07:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f92467fc640 1 --2- 172.21.15.98:0/3544885273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2ec780 0x55f31c2eeb70 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f923802a940 tx=0x7f92380316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92690d79144531b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f92457fa640 1 -- 172.21.15.98:0/3544885273 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238040070 con 0x55f31c2ec780 2024-09-19T08:07:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f924f7fe640 1 --2- 172.21.15.98:0/3773735356 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f923c0a33b0 0x7f923c0a5870 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9248008f90 tx=0x7f9248004ab0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f92457fa640 1 -- 172.21.15.98:0/3544885273 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9238005020 con 0x55f31c2ec780 2024-09-19T08:07:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.455+0000 7f92457fa640 1 -- 172.21.15.98:0/3544885273 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238035040 con 0x55f31c2ec780 2024-09-19T08:07:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.456+0000 7f92576d8740 1 -- 172.21.15.98:0/3544885273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2ec780 msgr2=0x55f31c2eeb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.456+0000 7f92576d8740 1 --2- 172.21.15.98:0/3544885273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2ec780 0x55f31c2eeb70 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f923802a940 tx=0x7f92380316e0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.456+0000 7f92576d8740 1 -- 172.21.15.98:0/3544885273 shutdown_connections 2024-09-19T08:07:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.456+0000 7f92576d8740 1 --2- 172.21.15.98:0/3544885273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2ec780 0x55f31c2eeb70 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.456+0000 7f92576d8740 1 --2- 172.21.15.98:0/3544885273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2dd5d0 0x55f31c2dda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.456+0000 7f92576d8740 1 --2- 172.21.15.98:0/3544885273 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2deb40 0x55f31c2dd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.456+0000 7f92576d8740 1 -- 172.21.15.98:0/3544885273 >> 172.21.15.98:0/3544885273 conn(0x55f31c2a6310 msgr2=0x55f31c2d9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.456+0000 7f92576d8740 1 -- 172.21.15.98:0/3544885273 shutdown_connections 2024-09-19T08:07:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.456+0000 7f92576d8740 1 -- 172.21.15.98:0/3544885273 wait complete. 2024-09-19T08:07:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.456+0000 7f92576d8740 1 -- start start 2024-09-19T08:07:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92576d8740 1 Processor -- start 2024-09-19T08:07:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92576d8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 0x55f31c2ec060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92576d8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c2e9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92576d8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 0x55f31c2e9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92576d8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f31c2ef420 con 0x55f31c2deb40 2024-09-19T08:07:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92576d8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f31c1fe060 con 0x55f31c2dd5d0 2024-09-19T08:07:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92576d8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f31c2ef5a0 con 0x55f31c2ec780 2024-09-19T08:07:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92576d8740 10 client.0 _ll_register_callbacks cb 0x55f31bf518c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92467fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c2e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92467fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c2e9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53452/0 (socket says 172.21.15.98:53452) 2024-09-19T08:07:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92467fc640 1 -- 172.21.15.98:0/1107056794 learned_addr learned my addr 172.21.15.98:0/1107056794 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f924f7fe640 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 0x55f31c2e9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f924effd640 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 0x55f31c2ec060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92467fc640 1 -- 172.21.15.98:0/1107056794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 msgr2=0x55f31c2e9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92467fc640 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 0x55f31c2e9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92467fc640 1 -- 172.21.15.98:0/1107056794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 msgr2=0x55f31c2ec060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92467fc640 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 0x55f31c2ec060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92467fc640 1 -- 172.21.15.98:0/1107056794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f31bfb43c0 con 0x55f31c2deb40 2024-09-19T08:07:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f924f7fe640 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 0x55f31c2e9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f924effd640 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 0x55f31c2ec060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92467fc640 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c2e9310 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f9238006010 tx=0x7f92380314b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e688605c6cbeeed8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92367fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.457+0000 7f92367fc640 1 -- 172.21.15.98:0/1107056794 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238040070 con 0x55f31c2deb40 2024-09-19T08:07:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.458+0000 7f92367fc640 1 -- 172.21.15.98:0/1107056794 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f9238004030 con 0x55f31c2deb40 2024-09-19T08:07:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.458+0000 7f92576d8740 1 -- 172.21.15.98:0/1107056794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9238005020 con 0x55f31c2deb40 2024-09-19T08:07:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.458+0000 7f92367fc640 1 -- 172.21.15.98:0/1107056794 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9238056050 con 0x55f31c2deb40 2024-09-19T08:07:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.458+0000 7f92576d8740 10 client.16680 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.458+0000 7f92576d8740 1 -- 172.21.15.98:0/1107056794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f31c2fae00 con 0x55f31c2deb40 2024-09-19T08:07:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.458+0000 7f92367fc640 1 -- 172.21.15.98:0/1107056794 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9238005020 con 0x55f31c2deb40 2024-09-19T08:07:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.458+0000 7f92367fc640 1 -- 172.21.15.98:0/1107056794 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f92380037b0 con 0x55f31c2deb40 2024-09-19T08:07:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.458+0000 7f92576d8740 10 client.16680 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.458+0000 7f92576d8740 10 client.16680 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:07:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.458+0000 7f92576d8740 20 client.16680 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.458+0000 7f92576d8740 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55f31c2fc150 0x55f31c2fe540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.458+0000 7f92576d8740 4 client.16680 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.458+0000 7f92576d8740 1 -- 172.21.15.98:0/1107056794 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f31c250860 con 0x55f31c2fc150 2024-09-19T08:07:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.458+0000 7f924effd640 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55f31c2fc150 0x55f31c2fe540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.459+0000 7f92367fc640 10 client.16680 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:07:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.459+0000 7f924effd640 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55f31c2fc150 0x55f31c2fe540 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8d42428b3d36b99e server_cookie=c4fe11e2f49baf2c in_seq=0 out_seq=0 2024-09-19T08:07:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.459+0000 7f92367fc640 1 -- 172.21.15.98:0/1107056794 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f31c250860 con 0x55f31c2fc150 2024-09-19T08:07:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.459+0000 7f92367fc640 10 client.16680 handle_command_reply: tid=0 2024-09-19T08:07:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.460+0000 7f92576d8740 1 client.16680 shutdown 2024-09-19T08:07:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.460+0000 7f92576d8740 1 -- 172.21.15.98:0/1107056794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 msgr2=0x55f31c2e9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.460+0000 7f92576d8740 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c2e9310 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f9238006010 tx=0x7f92380314b0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.460+0000 7f92576d8740 1 -- 172.21.15.98:0/1107056794 shutdown_connections 2024-09-19T08:07:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.460+0000 7f92576d8740 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55f31c2fc150 0x55f31c2fe540 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.460+0000 7f92576d8740 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 0x55f31c2e9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.460+0000 7f92576d8740 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c2e9310 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.460+0000 7f92367fc640 0 client.16680 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:07:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.460+0000 7f92576d8740 1 --2- 172.21.15.98:0/1107056794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 0x55f31c2ec060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.460+0000 7f92576d8740 1 -- 172.21.15.98:0/1107056794 >> 172.21.15.98:0/1107056794 conn(0x55f31c2a6310 msgr2=0x55f31c2df960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.460+0000 7f92576d8740 1 -- 172.21.15.98:0/1107056794 shutdown_connections 2024-09-19T08:07:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.460+0000 7f92576d8740 1 -- 172.21.15.98:0/1107056794 wait complete. 2024-09-19T08:07:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.460+0000 7f92576d8740 20 client.16680 trim_cache size 0 max 16384 2024-09-19T08:07:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.472+0000 7f92576d8740 1 Processor -- start 2024-09-19T08:07:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.472+0000 7f92576d8740 1 -- start start 2024-09-19T08:07:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92576d8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 0x55f31c2f86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92576d8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c303b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92576d8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 0x55f31c3040b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92576d8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f92501ab5d0 con 0x55f31c2deb40 2024-09-19T08:07:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92576d8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f92501ad170 con 0x55f31c2dd5d0 2024-09-19T08:07:49.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92576d8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f92501abca0 con 0x55f31c2ec780 2024-09-19T08:07:49.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92467fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c303b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92467fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c303b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53462/0 (socket says 172.21.15.98:53462) 2024-09-19T08:07:49.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92467fc640 1 -- 172.21.15.98:0/1542156605 learned_addr learned my addr 172.21.15.98:0/1542156605 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:49.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f924effd640 1 --2- 172.21.15.98:0/1542156605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 0x55f31c2f86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f924f7fe640 1 --2- 172.21.15.98:0/1542156605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 0x55f31c3040b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92467fc640 1 -- 172.21.15.98:0/1542156605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 msgr2=0x55f31c3040b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:49.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92467fc640 1 --2- 172.21.15.98:0/1542156605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 0x55f31c3040b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92467fc640 1 -- 172.21.15.98:0/1542156605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 msgr2=0x55f31c2f86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:49.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92467fc640 1 --2- 172.21.15.98:0/1542156605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 0x55f31c2f86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92467fc640 1 -- 172.21.15.98:0/1542156605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f31c2da1a0 con 0x55f31c2deb40 2024-09-19T08:07:49.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f924effd640 1 --2- 172.21.15.98:0/1542156605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 0x55f31c2f86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:49.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f92467fc640 1 --2- 172.21.15.98:0/1542156605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c303b70 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f923802a940 tx=0x7f923802a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc8b58c86edcc7a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:49.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.473+0000 7f9244ff9640 1 -- 172.21.15.98:0/1542156605 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238040040 con 0x55f31c2deb40 2024-09-19T08:07:49.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.474+0000 7f9244ff9640 1 -- 172.21.15.98:0/1542156605 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f923804d070 con 0x55f31c2deb40 2024-09-19T08:07:49.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.474+0000 7f9244ff9640 1 -- 172.21.15.98:0/1542156605 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238035040 con 0x55f31c2deb40 2024-09-19T08:07:49.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.474+0000 7f92576d8740 1 -- 172.21.15.98:0/1542156605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 msgr2=0x55f31c303b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:49.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.474+0000 7f92576d8740 1 --2- 172.21.15.98:0/1542156605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c303b70 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f923802a940 tx=0x7f923802a420 comp rx=0 tx=0).stop 2024-09-19T08:07:49.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.474+0000 7f92576d8740 1 -- 172.21.15.98:0/1542156605 shutdown_connections 2024-09-19T08:07:49.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.474+0000 7f92576d8740 1 --2- 172.21.15.98:0/1542156605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 0x55f31c3040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.474+0000 7f92576d8740 1 --2- 172.21.15.98:0/1542156605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c303b70 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.474+0000 7f92576d8740 1 --2- 172.21.15.98:0/1542156605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 0x55f31c2f86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.474+0000 7f92576d8740 1 -- 172.21.15.98:0/1542156605 >> 172.21.15.98:0/1542156605 conn(0x55f31c2a6310 msgr2=0x55f31c2e8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:49.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.474+0000 7f92576d8740 1 -- 172.21.15.98:0/1542156605 shutdown_connections 2024-09-19T08:07:49.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.474+0000 7f92576d8740 1 -- 172.21.15.98:0/1542156605 wait complete. 2024-09-19T08:07:49.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92576d8740 1 -- start start 2024-09-19T08:07:49.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92576d8740 1 Processor -- start 2024-09-19T08:07:49.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92576d8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 0x55f31c2f0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92576d8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c2f0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92576d8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 0x55f31c2f1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92576d8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f92501aca70 con 0x55f31c2deb40 2024-09-19T08:07:49.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92576d8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f92501ac3a0 con 0x55f31c2dd5d0 2024-09-19T08:07:49.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92576d8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f92501a95b0 con 0x55f31c2ec780 2024-09-19T08:07:49.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92576d8740 10 client.0 _ll_register_callbacks cb 0x7f9248008550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:49.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92467fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c2f0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92467fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c2f0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53476/0 (socket says 172.21.15.98:53476) 2024-09-19T08:07:49.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92467fc640 1 -- 172.21.15.98:0/3999885126 learned_addr learned my addr 172.21.15.98:0/3999885126 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:49.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f924effd640 1 --2- 172.21.15.98:0/3999885126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 0x55f31c2f0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92467fc640 1 -- 172.21.15.98:0/3999885126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 msgr2=0x55f31c2f1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:49.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92467fc640 1 --2- 172.21.15.98:0/3999885126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 0x55f31c2f1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92467fc640 1 -- 172.21.15.98:0/3999885126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 msgr2=0x55f31c2f0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:49.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92467fc640 1 --2- 172.21.15.98:0/3999885126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 0x55f31c2f0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.475+0000 7f92467fc640 1 -- 172.21.15.98:0/3999885126 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f31c365e10 con 0x55f31c2deb40 2024-09-19T08:07:49.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92467fc640 1 --2- 172.21.15.98:0/3999885126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c2f0dd0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9238003ca0 tx=0x7f9238000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1ec59476037b44 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:49.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92367fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:49.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92367fc640 1 -- 172.21.15.98:0/3999885126 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238040030 con 0x55f31c2deb40 2024-09-19T08:07:49.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92367fc640 1 -- 172.21.15.98:0/3999885126 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f92380035f0 con 0x55f31c2deb40 2024-09-19T08:07:49.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92576d8740 1 -- 172.21.15.98:0/3999885126 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f923804d070 con 0x55f31c2deb40 2024-09-19T08:07:49.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92367fc640 1 -- 172.21.15.98:0/3999885126 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9238035040 con 0x55f31c2deb40 2024-09-19T08:07:49.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92576d8740 10 client.16686 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:49.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92576d8740 1 -- 172.21.15.98:0/3999885126 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f31c367080 con 0x55f31c2deb40 2024-09-19T08:07:49.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92367fc640 1 -- 172.21.15.98:0/3999885126 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f923804d070 con 0x55f31c2deb40 2024-09-19T08:07:49.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92367fc640 1 -- 172.21.15.98:0/3999885126 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f9238061030 con 0x55f31c2deb40 2024-09-19T08:07:49.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92576d8740 10 client.16686 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:49.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92576d8740 10 client.16686 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:07:49.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92576d8740 20 client.16686 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:49.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92576d8740 1 --2- 172.21.15.98:0/3999885126 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55f31c3677b0 0x55f31c369bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:49.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92576d8740 4 client.16686 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:07:49.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.476+0000 7f92576d8740 1 -- 172.21.15.98:0/3999885126 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f31c24ad70 con 0x55f31c3677b0 2024-09-19T08:07:49.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.477+0000 7f924effd640 1 --2- 172.21.15.98:0/3999885126 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55f31c3677b0 0x55f31c369bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:49.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.477+0000 7f924effd640 1 --2- 172.21.15.98:0/3999885126 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55f31c3677b0 0x55f31c369bf0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b07c0e2c7f1fa095 server_cookie=51164855577602fd in_seq=0 out_seq=0 2024-09-19T08:07:49.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.477+0000 7f92367fc640 10 client.16686 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:07:49.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.477+0000 7f92367fc640 1 -- 172.21.15.98:0/3999885126 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f31c24ad70 con 0x55f31c3677b0 2024-09-19T08:07:49.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.477+0000 7f92367fc640 10 client.16686 handle_command_reply: tid=0 2024-09-19T08:07:49.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.477+0000 7f92576d8740 1 client.16686 shutdown 2024-09-19T08:07:49.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92576d8740 1 -- 172.21.15.98:0/3999885126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 msgr2=0x55f31c2f0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:49.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92576d8740 1 --2- 172.21.15.98:0/3999885126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c2f0dd0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9238003ca0 tx=0x7f9238000bc0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92576d8740 1 -- 172.21.15.98:0/3999885126 shutdown_connections 2024-09-19T08:07:49.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92576d8740 1 --2- 172.21.15.98:0/3999885126 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55f31c3677b0 0x55f31c369bf0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92576d8740 1 --2- 172.21.15.98:0/3999885126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f31c2ec780 0x55f31c2f1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92576d8740 1 --2- 172.21.15.98:0/3999885126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f31c2deb40 0x55f31c2f0dd0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92367fc640 0 client.16686 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:07:49.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92576d8740 1 --2- 172.21.15.98:0/3999885126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f31c2dd5d0 0x55f31c2f0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92576d8740 1 -- 172.21.15.98:0/3999885126 >> 172.21.15.98:0/3999885126 conn(0x55f31c2a6310 msgr2=0x55f31c2d84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:49.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92576d8740 1 -- 172.21.15.98:0/3999885126 shutdown_connections 2024-09-19T08:07:49.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92576d8740 1 -- 172.21.15.98:0/3999885126 wait complete. 2024-09-19T08:07:49.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92576d8740 20 client.16686 trim_cache size 0 max 16384 2024-09-19T08:07:49.485 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:07:49.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92552ff640 1 -- 172.21.15.98:0/3773735356 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f923c0a33b0 msgr2=0x7f923c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:49.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92552ff640 1 --2- 172.21.15.98:0/3773735356 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f923c0a33b0 0x7f923c0a5870 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9248008f90 tx=0x7f9248004ab0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92552ff640 1 -- 172.21.15.98:0/3773735356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92501002a0 msgr2=0x7f9250199360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:49.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92552ff640 1 --2- 172.21.15.98:0/3773735356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92501002a0 0x7f9250199360 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f924000c970 tx=0x7f924000ce30 comp rx=0 tx=0).stop 2024-09-19T08:07:49.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92552ff640 1 -- 172.21.15.98:0/3773735356 shutdown_connections 2024-09-19T08:07:49.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92552ff640 1 --2- 172.21.15.98:0/3773735356 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f923c0a33b0 0x7f923c0a5870 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92552ff640 1 --2- 172.21.15.98:0/3773735356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92501062b0 0x7f92501a43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92552ff640 1 --2- 172.21.15.98:0/3773735356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9250100c00 0x7f92501998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92552ff640 1 --2- 172.21.15.98:0/3773735356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92501002a0 0x7f9250199360 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:49.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.478+0000 7f92552ff640 1 -- 172.21.15.98:0/3773735356 >> 172.21.15.98:0/3773735356 conn(0x7f92500fc5c0 msgr2=0x7f92500feee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:49.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.479+0000 7f92552ff640 1 -- 172.21.15.98:0/3773735356 shutdown_connections 2024-09-19T08:07:49.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:49.479+0000 7f92552ff640 1 -- 172.21.15.98:0/3773735356 wait complete. 2024-09-19T08:07:50.149 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:49 smithi098 ceph-mon[31333]: pgmap v147: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:07:50.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:49 smithi139 ceph-mon[34874]: pgmap v147: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:07:50.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:07:50.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:49 smithi162 ceph-mon[35260]: pgmap v147: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:07:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:50.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.902+0000 7f8e819bf640 1 -- 172.21.15.98:0/774544713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e7c106a80 msgr2=0x7f8e7c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.902+0000 7f8e819bf640 1 --2- 172.21.15.98:0/774544713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e7c106a80 0x7f8e7c106e60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009930 tx=0x7f8e6c02a420 comp rx=0 tx=0).stop 2024-09-19T08:07:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.902+0000 7f8e819bf640 1 -- 172.21.15.98:0/774544713 shutdown_connections 2024-09-19T08:07:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.902+0000 7f8e819bf640 1 --2- 172.21.15.98:0/774544713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e7c101420 0x7f8e7c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.902+0000 7f8e819bf640 1 --2- 172.21.15.98:0/774544713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e7c100a80 0x7f8e7c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.902+0000 7f8e819bf640 1 --2- 172.21.15.98:0/774544713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e7c106a80 0x7f8e7c106e60 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.902+0000 7f8e819bf640 1 -- 172.21.15.98:0/774544713 >> 172.21.15.98:0/774544713 conn(0x7f8e7c0fc7a0 msgr2=0x7f8e7c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.902+0000 7f8e819bf640 1 -- 172.21.15.98:0/774544713 shutdown_connections 2024-09-19T08:07:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.902+0000 7f8e819bf640 1 -- 172.21.15.98:0/774544713 wait complete. 2024-09-19T08:07:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.903+0000 7f8e819bf640 1 Processor -- start 2024-09-19T08:07:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.903+0000 7f8e819bf640 1 -- start start 2024-09-19T08:07:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.903+0000 7f8e819bf640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e7c100a80 0x7f8e7c199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.903+0000 7f8e819bf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e7c101420 0x7f8e7c199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.903+0000 7f8e819bf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e7c106a80 0x7f8e7c1a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.903+0000 7f8e819bf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c110e80 con 0x7f8e7c106a80 2024-09-19T08:07:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.903+0000 7f8e819bf640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c110d00 con 0x7f8e7c100a80 2024-09-19T08:07:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.903+0000 7f8e819bf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c111000 con 0x7f8e7c101420 2024-09-19T08:07:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.903+0000 7f8e7b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e7c106a80 0x7f8e7c1a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.903+0000 7f8e7b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e7c106a80 0x7f8e7c1a42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53490/0 (socket says 172.21.15.98:53490) 2024-09-19T08:07:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.903+0000 7f8e7b7fe640 1 -- 172.21.15.98:0/1438717564 learned_addr learned my addr 172.21.15.98:0/1438717564 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.903+0000 7f8e73fff640 1 --2- 172.21.15.98:0/1438717564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e7c101420 0x7f8e7c199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.904+0000 7f8e7affd640 1 --2- 172.21.15.98:0/1438717564 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e7c100a80 0x7f8e7c199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.904+0000 7f8e7b7fe640 1 -- 172.21.15.98:0/1438717564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e7c101420 msgr2=0x7f8e7c199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.904+0000 7f8e7b7fe640 1 --2- 172.21.15.98:0/1438717564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e7c101420 0x7f8e7c199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.904+0000 7f8e7b7fe640 1 -- 172.21.15.98:0/1438717564 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e7c100a80 msgr2=0x7f8e7c199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.904+0000 7f8e7b7fe640 1 --2- 172.21.15.98:0/1438717564 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e7c100a80 0x7f8e7c199250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.904+0000 7f8e7b7fe640 1 -- 172.21.15.98:0/1438717564 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7c1a4810 con 0x7f8e7c106a80 2024-09-19T08:07:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.904+0000 7f8e7b7fe640 1 --2- 172.21.15.98:0/1438717564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e7c106a80 0x7f8e7c1a42d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400eaa0 tx=0x7f8e7400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.904+0000 7f8e809bd640 1 -- 172.21.15.98:0/1438717564 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74004d20 con 0x7f8e7c106a80 2024-09-19T08:07:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.904+0000 7f8e819bf640 1 -- 172.21.15.98:0/1438717564 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e7c1a49e0 con 0x7f8e7c106a80 2024-09-19T08:07:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.904+0000 7f8e809bd640 1 -- 172.21.15.98:0/1438717564 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e74005020 con 0x7f8e7c106a80 2024-09-19T08:07:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.904+0000 7f8e809bd640 1 -- 172.21.15.98:0/1438717564 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e740116d0 con 0x7f8e7c106a80 2024-09-19T08:07:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.904+0000 7f8e819bf640 1 -- 172.21.15.98:0/1438717564 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e7c1a4f20 con 0x7f8e7c106a80 2024-09-19T08:07:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.907+0000 7f8e809bd640 1 -- 172.21.15.98:0/1438717564 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f8e740051d0 con 0x7f8e7c106a80 2024-09-19T08:07:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.907+0000 7f8e83d98740 1 Processor -- start 2024-09-19T08:07:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.907+0000 7f8e83d98740 1 -- start start 2024-09-19T08:07:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.907+0000 7f8e83d98740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887112090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.907+0000 7f8e83d98740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558871125d0 0x555887112a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.907+0000 7f8e83d98740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55588711d370 0x55588711f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.907+0000 7f8e83d98740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5558870782f0 con 0x555887113b40 2024-09-19T08:07:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.907+0000 7f8e83d98740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55588701fcf0 con 0x5558871125d0 2024-09-19T08:07:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.907+0000 7f8e83d98740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5558870742b0 con 0x55588711d370 2024-09-19T08:07:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e73fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887112090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e73fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887112090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53500/0 (socket says 172.21.15.98:53500) 2024-09-19T08:07:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e73fff640 1 -- 172.21.15.98:0/1976563438 learned_addr learned my addr 172.21.15.98:0/1976563438 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e7affd640 1 --2- 172.21.15.98:0/1976563438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558871125d0 0x555887112a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e73fff640 1 --2- 172.21.15.98:0/1976563438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55588711d370 0x55588711f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e73fff640 1 -- 172.21.15.98:0/1976563438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55588711d370 msgr2=0x55588711f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e73fff640 1 --2- 172.21.15.98:0/1976563438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55588711d370 0x55588711f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e73fff640 1 -- 172.21.15.98:0/1976563438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558871125d0 msgr2=0x555887112a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e73fff640 1 --2- 172.21.15.98:0/1976563438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558871125d0 0x555887112a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e73fff640 1 -- 172.21.15.98:0/1976563438 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558870043a0 con 0x555887113b40 2024-09-19T08:07:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e809bd640 1 --2- 172.21.15.98:0/1438717564 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8e680a33b0 0x7f8e680a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e73fff640 1 --2- 172.21.15.98:0/1976563438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887112090 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8e64013560 tx=0x7f8e6400a800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=975894e613d278c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e7affd640 1 --2- 172.21.15.98:0/1438717564 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8e680a33b0 0x7f8e680a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e809bd640 1 -- 172.21.15.98:0/1438717564 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f8e740cfe60 con 0x7f8e7c106a80 2024-09-19T08:07:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e71ffb640 1 -- 172.21.15.98:0/1976563438 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e64019030 con 0x555887113b40 2024-09-19T08:07:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e71ffb640 1 -- 172.21.15.98:0/1976563438 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e640110d0 con 0x555887113b40 2024-09-19T08:07:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.908+0000 7f8e71ffb640 1 -- 172.21.15.98:0/1976563438 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6400f8e0 con 0x555887113b40 2024-09-19T08:07:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.909+0000 7f8e83d98740 1 -- 172.21.15.98:0/1976563438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 msgr2=0x555887112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.909+0000 7f8e7affd640 1 --2- 172.21.15.98:0/1438717564 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8e680a33b0 0x7f8e680a5870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c000c00 tx=0x7f8e6c005d90 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.909+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1976563438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887112090 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8e64013560 tx=0x7f8e6400a800 comp rx=0 tx=0).stop 2024-09-19T08:07:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.909+0000 7f8e83d98740 1 -- 172.21.15.98:0/1976563438 shutdown_connections 2024-09-19T08:07:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.909+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1976563438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55588711d370 0x55588711f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.909+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1976563438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558871125d0 0x555887112a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.909+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1976563438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887112090 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.909+0000 7f8e83d98740 1 -- 172.21.15.98:0/1976563438 >> 172.21.15.98:0/1976563438 conn(0x5558870db310 msgr2=0x55588710ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.909+0000 7f8e83d98740 1 -- 172.21.15.98:0/1976563438 shutdown_connections 2024-09-19T08:07:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.909+0000 7f8e83d98740 1 -- 172.21.15.98:0/1976563438 wait complete. 2024-09-19T08:07:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.910+0000 7f8e83d98740 1 -- start start 2024-09-19T08:07:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.910+0000 7f8e83d98740 1 Processor -- start 2024-09-19T08:07:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.910+0000 7f8e83d98740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558871125d0 0x55588711cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.910+0000 7f8e83d98740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555887113b40 0x55588711a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.910+0000 7f8e83d98740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588711d370 0x55588711a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.910+0000 7f8e83d98740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555887120070 con 0x55588711d370 2024-09-19T08:07:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.910+0000 7f8e83d98740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555887033060 con 0x5558871125d0 2024-09-19T08:07:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.910+0000 7f8e83d98740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5558871201f0 con 0x555887113b40 2024-09-19T08:07:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.910+0000 7f8e83d98740 10 client.0 _ll_register_callbacks cb 0x555886d868c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.910+0000 7f8e7b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588711d370 0x55588711a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.910+0000 7f8e7b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588711d370 0x55588711a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53510/0 (socket says 172.21.15.98:53510) 2024-09-19T08:07:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.910+0000 7f8e7affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558871125d0 0x55588711cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.910+0000 7f8e7b7fe640 1 -- 172.21.15.98:0/1414388577 learned_addr learned my addr 172.21.15.98:0/1414388577 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.910+0000 7f8e73fff640 1 --2- 172.21.15.98:0/1414388577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555887113b40 0x55588711a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.911+0000 7f8e7b7fe640 1 -- 172.21.15.98:0/1414388577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555887113b40 msgr2=0x55588711a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.911+0000 7f8e7b7fe640 1 --2- 172.21.15.98:0/1414388577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555887113b40 0x55588711a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.911+0000 7f8e7b7fe640 1 -- 172.21.15.98:0/1414388577 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558871125d0 msgr2=0x55588711cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.911+0000 7f8e7b7fe640 1 --2- 172.21.15.98:0/1414388577 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558871125d0 0x55588711cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.911+0000 7f8e7b7fe640 1 -- 172.21.15.98:0/1414388577 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555886de93c0 con 0x55588711d370 2024-09-19T08:07:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.911+0000 7f8e7b7fe640 1 --2- 172.21.15.98:0/1414388577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588711d370 0x55588711a6d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f8e740a48b0 tx=0x7f8e7409c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f969e3094b2b76 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.911+0000 7f8e5affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e5affd640 1 -- 172.21.15.98:0/1414388577 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7401f030 con 0x55588711d370 2024-09-19T08:07:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e5affd640 1 -- 172.21.15.98:0/1414388577 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f8e740bc070 con 0x55588711d370 2024-09-19T08:07:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e83d98740 1 -- 172.21.15.98:0/1414388577 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e640110d0 con 0x55588711d370 2024-09-19T08:07:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e5affd640 1 -- 172.21.15.98:0/1414388577 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8e740bc270 con 0x55588711d370 2024-09-19T08:07:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e83d98740 10 client.16698 fetch_fsmap learned FSMap version 30 2024-09-19T08:07:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e83d98740 1 -- 172.21.15.98:0/1414388577 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55588712bbb0 con 0x55588711d370 2024-09-19T08:07:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e5affd640 1 -- 172.21.15.98:0/1414388577 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e640110d0 con 0x55588711d370 2024-09-19T08:07:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e5affd640 1 -- 172.21.15.98:0/1414388577 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f8e74096ce0 con 0x55588711d370 2024-09-19T08:07:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e83d98740 10 client.16698 fetch_fsmap finished waiting for FSMap version 30 2024-09-19T08:07:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e83d98740 10 client.16698 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:07:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e83d98740 20 client.16698 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1414388577 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55588712d070 0x55588712f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e83d98740 4 client.16698 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e83d98740 1 -- 172.21.15.98:0/1414388577 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555887085860 con 0x55588712d070 2024-09-19T08:07:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.912+0000 7f8e7affd640 1 --2- 172.21.15.98:0/1414388577 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55588712d070 0x55588712f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.913+0000 7f8e7affd640 1 --2- 172.21.15.98:0/1414388577 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55588712d070 0x55588712f4f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ea3cbe7f4b0dd01d server_cookie=ba937070612ade97 in_seq=0 out_seq=0 2024-09-19T08:07:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.913+0000 7f8e5affd640 10 client.16698 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:07:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.914+0000 7f8e5affd640 1 -- 172.21.15.98:0/1414388577 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555887085860 con 0x55588712d070 2024-09-19T08:07:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.914+0000 7f8e5affd640 10 client.16698 handle_command_reply: tid=0 2024-09-19T08:07:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.914+0000 7f8e83d98740 1 client.16698 shutdown 2024-09-19T08:07:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.914+0000 7f8e83d98740 1 -- 172.21.15.98:0/1414388577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588711d370 msgr2=0x55588711a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.914+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1414388577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588711d370 0x55588711a6d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f8e740a48b0 tx=0x7f8e7409c870 comp rx=0 tx=0).stop 2024-09-19T08:07:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.914+0000 7f8e83d98740 1 -- 172.21.15.98:0/1414388577 shutdown_connections 2024-09-19T08:07:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.914+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1414388577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588711d370 0x55588711a6d0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.914+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1414388577 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55588712d070 0x55588712f4f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.914+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1414388577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555887113b40 0x55588711a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.914+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1414388577 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558871125d0 0x55588711cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.914+0000 7f8e5affd640 0 client.16698 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:07:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.914+0000 7f8e83d98740 1 -- 172.21.15.98:0/1414388577 >> 172.21.15.98:0/1414388577 conn(0x5558870db310 msgr2=0x555887114930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.914+0000 7f8e83d98740 1 -- 172.21.15.98:0/1414388577 shutdown_connections 2024-09-19T08:07:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.914+0000 7f8e83d98740 1 -- 172.21.15.98:0/1414388577 wait complete. 2024-09-19T08:07:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.915+0000 7f8e83d98740 20 client.16698 trim_cache size 0 max 16384 2024-09-19T08:07:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.927+0000 7f8e83d98740 1 Processor -- start 2024-09-19T08:07:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.927+0000 7f8e83d98740 1 -- start start 2024-09-19T08:07:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e83d98740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558871125d0 0x555887129310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e83d98740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887119050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e83d98740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55588711d370 0x555887134e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e83d98740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c1ad880 con 0x555887113b40 2024-09-19T08:07:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e83d98740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c1af420 con 0x55588711d370 2024-09-19T08:07:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e83d98740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c1adf50 con 0x5558871125d0 2024-09-19T08:07:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e73fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887119050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e73fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887119050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53518/0 (socket says 172.21.15.98:53518) 2024-09-19T08:07:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e73fff640 1 -- 172.21.15.98:0/3982691082 learned_addr learned my addr 172.21.15.98:0/3982691082 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e7b7fe640 1 --2- 172.21.15.98:0/3982691082 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55588711d370 0x555887134e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e7affd640 1 --2- 172.21.15.98:0/3982691082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558871125d0 0x555887129310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e73fff640 1 -- 172.21.15.98:0/3982691082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558871125d0 msgr2=0x555887129310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e73fff640 1 --2- 172.21.15.98:0/3982691082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558871125d0 0x555887129310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e73fff640 1 -- 172.21.15.98:0/3982691082 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55588711d370 msgr2=0x555887134e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e73fff640 1 --2- 172.21.15.98:0/3982691082 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55588711d370 0x555887134e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e73fff640 1 -- 172.21.15.98:0/3982691082 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55588711a710 con 0x555887113b40 2024-09-19T08:07:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.928+0000 7f8e73fff640 1 --2- 172.21.15.98:0/3982691082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887119050 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f8e6400f460 tx=0x7f8e6400a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46261877c5c547b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.929+0000 7f8e717fa640 1 -- 172.21.15.98:0/3982691082 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e64019040 con 0x555887113b40 2024-09-19T08:07:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.929+0000 7f8e717fa640 1 -- 172.21.15.98:0/3982691082 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e640110c0 con 0x555887113b40 2024-09-19T08:07:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.929+0000 7f8e717fa640 1 -- 172.21.15.98:0/3982691082 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e64011e10 con 0x555887113b40 2024-09-19T08:07:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.930+0000 7f8e83d98740 1 -- 172.21.15.98:0/3982691082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 msgr2=0x555887119050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.930+0000 7f8e83d98740 1 --2- 172.21.15.98:0/3982691082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887119050 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f8e6400f460 tx=0x7f8e6400a2d0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.930+0000 7f8e83d98740 1 -- 172.21.15.98:0/3982691082 shutdown_connections 2024-09-19T08:07:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.930+0000 7f8e83d98740 1 --2- 172.21.15.98:0/3982691082 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55588711d370 0x555887134e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.930+0000 7f8e83d98740 1 --2- 172.21.15.98:0/3982691082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887119050 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.930+0000 7f8e83d98740 1 --2- 172.21.15.98:0/3982691082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558871125d0 0x555887129310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.930+0000 7f8e83d98740 1 -- 172.21.15.98:0/3982691082 >> 172.21.15.98:0/3982691082 conn(0x5558870db310 msgr2=0x555887118940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.930+0000 7f8e83d98740 1 -- 172.21.15.98:0/3982691082 shutdown_connections 2024-09-19T08:07:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.930+0000 7f8e83d98740 1 -- 172.21.15.98:0/3982691082 wait complete. 2024-09-19T08:07:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.930+0000 7f8e83d98740 1 -- start start 2024-09-19T08:07:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e83d98740 1 Processor -- start 2024-09-19T08:07:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e83d98740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558871125d0 0x5558871258b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e83d98740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887120c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e83d98740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55588711d370 0x555887121170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e83d98740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c1aed20 con 0x555887113b40 2024-09-19T08:07:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e83d98740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c1ae650 con 0x5558871125d0 2024-09-19T08:07:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e83d98740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c1ab860 con 0x55588711d370 2024-09-19T08:07:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e83d98740 10 client.0 _ll_register_callbacks cb 0x7f8e74004ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e73fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887120c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e73fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887120c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53522/0 (socket says 172.21.15.98:53522) 2024-09-19T08:07:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e73fff640 1 -- 172.21.15.98:0/1946129050 learned_addr learned my addr 172.21.15.98:0/1946129050 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e73fff640 1 -- 172.21.15.98:0/1946129050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55588711d370 msgr2=0x555887121170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e73fff640 1 --2- 172.21.15.98:0/1946129050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55588711d370 0x555887121170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e73fff640 1 -- 172.21.15.98:0/1946129050 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558871125d0 msgr2=0x5558871258b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e73fff640 1 --2- 172.21.15.98:0/1946129050 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558871125d0 0x5558871258b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.931+0000 7f8e73fff640 1 -- 172.21.15.98:0/1946129050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558871216b0 con 0x555887113b40 2024-09-19T08:07:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e73fff640 1 --2- 172.21.15.98:0/1946129050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887120c30 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8e64013eb0 tx=0x7f8e6400aae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ebee3ce11c1ecfa server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e5affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e5affd640 1 -- 172.21.15.98:0/1946129050 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e64019070 con 0x555887113b40 2024-09-19T08:07:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e5affd640 1 -- 172.21.15.98:0/1946129050 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f8e64011d90 con 0x555887113b40 2024-09-19T08:07:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e83d98740 1 -- 172.21.15.98:0/1946129050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e640110c0 con 0x555887113b40 2024-09-19T08:07:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e5affd640 1 -- 172.21.15.98:0/1946129050 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8e64005040 con 0x555887113b40 2024-09-19T08:07:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e83d98740 10 client.16704 fetch_fsmap learned FSMap version 31 2024-09-19T08:07:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e83d98740 1 -- 172.21.15.98:0/1946129050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555887197ad0 con 0x555887113b40 2024-09-19T08:07:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e5affd640 1 -- 172.21.15.98:0/1946129050 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e640110c0 con 0x555887113b40 2024-09-19T08:07:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e5affd640 1 -- 172.21.15.98:0/1946129050 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f8e64004c60 con 0x555887113b40 2024-09-19T08:07:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e83d98740 10 client.16704 fetch_fsmap finished waiting for FSMap version 31 2024-09-19T08:07:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e83d98740 10 client.16704 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:07:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e83d98740 20 client.16704 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1946129050 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x555887197f00 0x55588719a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e83d98740 4 client.16704 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:07:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.932+0000 7f8e83d98740 1 -- 172.21.15.98:0/1946129050 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55588707fd70 con 0x555887197f00 2024-09-19T08:07:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.933+0000 7f8e7affd640 1 --2- 172.21.15.98:0/1946129050 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x555887197f00 0x55588719a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.933+0000 7f8e5affd640 10 client.16704 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:07:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.933+0000 7f8e7affd640 1 --2- 172.21.15.98:0/1946129050 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x555887197f00 0x55588719a340 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=22168166deb98164 server_cookie=d6ca67cf29a49d4b in_seq=0 out_seq=0 2024-09-19T08:07:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.933+0000 7f8e5affd640 1 -- 172.21.15.98:0/1946129050 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55588707fd70 con 0x555887197f00 2024-09-19T08:07:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.933+0000 7f8e5affd640 10 client.16704 handle_command_reply: tid=0 2024-09-19T08:07:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.933+0000 7f8e83d98740 1 client.16704 shutdown 2024-09-19T08:07:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.933+0000 7f8e83d98740 1 -- 172.21.15.98:0/1946129050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 msgr2=0x555887120c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.933+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1946129050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887120c30 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8e64013eb0 tx=0x7f8e6400aae0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.933+0000 7f8e83d98740 1 -- 172.21.15.98:0/1946129050 shutdown_connections 2024-09-19T08:07:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.933+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1946129050 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x555887197f00 0x55588719a340 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.933+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1946129050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55588711d370 0x555887121170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.933+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1946129050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555887113b40 0x555887120c30 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.933+0000 7f8e83d98740 1 --2- 172.21.15.98:0/1946129050 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558871125d0 0x5558871258b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.933+0000 7f8e83d98740 1 -- 172.21.15.98:0/1946129050 >> 172.21.15.98:0/1946129050 conn(0x5558870db310 msgr2=0x555887119160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e5affd640 0 client.16704 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:07:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e83d98740 1 -- 172.21.15.98:0/1946129050 shutdown_connections 2024-09-19T08:07:50.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e83d98740 1 -- 172.21.15.98:0/1946129050 wait complete. 2024-09-19T08:07:50.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e83d98740 20 client.16704 trim_cache size 0 max 16384 2024-09-19T08:07:50.951 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:07:50.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e819bf640 1 -- 172.21.15.98:0/1438717564 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8e680a33b0 msgr2=0x7f8e680a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:50.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e819bf640 1 --2- 172.21.15.98:0/1438717564 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8e680a33b0 0x7f8e680a5870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c000c00 tx=0x7f8e6c005d90 comp rx=0 tx=0).stop 2024-09-19T08:07:50.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e819bf640 1 -- 172.21.15.98:0/1438717564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e7c106a80 msgr2=0x7f8e7c1a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:50.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e819bf640 1 --2- 172.21.15.98:0/1438717564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e7c106a80 0x7f8e7c1a42d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400eaa0 tx=0x7f8e7400eea0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e819bf640 1 -- 172.21.15.98:0/1438717564 shutdown_connections 2024-09-19T08:07:50.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e819bf640 1 --2- 172.21.15.98:0/1438717564 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8e680a33b0 0x7f8e680a5870 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e819bf640 1 --2- 172.21.15.98:0/1438717564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e7c106a80 0x7f8e7c1a42d0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e819bf640 1 --2- 172.21.15.98:0/1438717564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e7c101420 0x7f8e7c199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e819bf640 1 --2- 172.21.15.98:0/1438717564 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e7c100a80 0x7f8e7c199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:50.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e819bf640 1 -- 172.21.15.98:0/1438717564 >> 172.21.15.98:0/1438717564 conn(0x7f8e7c0fc7a0 msgr2=0x7f8e7c0fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:50.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e819bf640 1 -- 172.21.15.98:0/1438717564 shutdown_connections 2024-09-19T08:07:50.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:50.934+0000 7f8e819bf640 1 -- 172.21.15.98:0/1438717564 wait complete. 2024-09-19T08:07:51.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:07:51.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:51 smithi098 ceph-mon[31333]: pgmap v148: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:51.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:51 smithi098 ceph-mon[31333]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:07:51.952 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:52.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:51 smithi139 ceph-mon[34874]: pgmap v148: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:52.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:51 smithi139 ceph-mon[34874]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:07:52.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:51 smithi162 ceph-mon[35260]: pgmap v148: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:52.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:51 smithi162 ceph-mon[35260]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:07:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.428+0000 7fe1308ea640 1 -- 172.21.15.98:0/1212405739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe12c106a80 msgr2=0x7fe12c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.428+0000 7fe1308ea640 1 --2- 172.21.15.98:0/1212405739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe12c106a80 0x7fe12c106e60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fe120009930 tx=0x7fe12002a420 comp rx=0 tx=0).stop 2024-09-19T08:07:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.428+0000 7fe1308ea640 1 -- 172.21.15.98:0/1212405739 shutdown_connections 2024-09-19T08:07:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.428+0000 7fe1308ea640 1 --2- 172.21.15.98:0/1212405739 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe12c101420 0x7fe12c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.428+0000 7fe1308ea640 1 --2- 172.21.15.98:0/1212405739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe12c100a80 0x7fe12c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.428+0000 7fe1308ea640 1 --2- 172.21.15.98:0/1212405739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe12c106a80 0x7fe12c106e60 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.428+0000 7fe1308ea640 1 -- 172.21.15.98:0/1212405739 >> 172.21.15.98:0/1212405739 conn(0x7fe12c0fc7a0 msgr2=0x7fe12c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.428+0000 7fe1308ea640 1 -- 172.21.15.98:0/1212405739 shutdown_connections 2024-09-19T08:07:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.428+0000 7fe1308ea640 1 -- 172.21.15.98:0/1212405739 wait complete. 2024-09-19T08:07:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.429+0000 7fe1308ea640 1 Processor -- start 2024-09-19T08:07:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.429+0000 7fe1308ea640 1 -- start start 2024-09-19T08:07:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.429+0000 7fe1308ea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe12c100a80 0x7fe12c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.429+0000 7fe1308ea640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe12c101420 0x7fe12c198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.429+0000 7fe1308ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe12c106a80 0x7fe12c19cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.429+0000 7fe1308ea640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c110e40 con 0x7fe12c106a80 2024-09-19T08:07:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.429+0000 7fe1308ea640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c110cc0 con 0x7fe12c101420 2024-09-19T08:07:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.429+0000 7fe1308ea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c110fc0 con 0x7fe12c100a80 2024-09-19T08:07:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.429+0000 7fe12ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe12c106a80 0x7fe12c19cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe12ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe12c106a80 0x7fe12c19cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53536/0 (socket says 172.21.15.98:53536) 2024-09-19T08:07:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe12ad9d640 1 -- 172.21.15.98:0/3300664591 learned_addr learned my addr 172.21.15.98:0/3300664591 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe129d9b640 1 --2- 172.21.15.98:0/3300664591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe12c101420 0x7fe12c198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe12a59c640 1 --2- 172.21.15.98:0/3300664591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe12c100a80 0x7fe12c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe12ad9d640 1 -- 172.21.15.98:0/3300664591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe12c100a80 msgr2=0x7fe12c198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe12ad9d640 1 --2- 172.21.15.98:0/3300664591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe12c100a80 0x7fe12c198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe12ad9d640 1 -- 172.21.15.98:0/3300664591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe12c101420 msgr2=0x7fe12c198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe12ad9d640 1 --2- 172.21.15.98:0/3300664591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe12c101420 0x7fe12c198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe12ad9d640 1 -- 172.21.15.98:0/3300664591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe12c19d3e0 con 0x7fe12c106a80 2024-09-19T08:07:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe12ad9d640 1 --2- 172.21.15.98:0/3300664591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe12c106a80 0x7fe12c19cd00 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe12400d750 tx=0x7fe12400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe11b7fe640 1 -- 172.21.15.98:0/3300664591 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe124017070 con 0x7fe12c106a80 2024-09-19T08:07:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe1308ea640 1 -- 172.21.15.98:0/3300664591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe12c19d6d0 con 0x7fe12c106a80 2024-09-19T08:07:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe11b7fe640 1 -- 172.21.15.98:0/3300664591 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe124005500 con 0x7fe12c106a80 2024-09-19T08:07:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe11b7fe640 1 -- 172.21.15.98:0/3300664591 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe124004d90 con 0x7fe12c106a80 2024-09-19T08:07:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.430+0000 7fe1308ea640 1 -- 172.21.15.98:0/3300664591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe12c1a4fb0 con 0x7fe12c106a80 2024-09-19T08:07:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.432+0000 7fe11b7fe640 1 -- 172.21.15.98:0/3300664591 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fe124004f30 con 0x7fe12c106a80 2024-09-19T08:07:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.433+0000 7fe132cc3740 1 Processor -- start 2024-09-19T08:07:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.433+0000 7fe132cc3740 1 -- start start 2024-09-19T08:07:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.433+0000 7fe132cc3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30cbb40 0x55b5b30ca090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.433+0000 7fe132cc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30ca5d0 0x55b5b30caa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.433+0000 7fe132cc3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30d9780 0x55b5b30dbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.433+0000 7fe132cc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b5b30302f0 con 0x55b5b30ca5d0 2024-09-19T08:07:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.433+0000 7fe132cc3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b5b2fd7cf0 con 0x55b5b30d9780 2024-09-19T08:07:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.433+0000 7fe132cc3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b5b302c2b0 con 0x55b5b30cbb40 2024-09-19T08:07:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.433+0000 7fe12a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30ca5d0 0x55b5b30caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.433+0000 7fe12a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30ca5d0 0x55b5b30caa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53548/0 (socket says 172.21.15.98:53548) 2024-09-19T08:07:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.433+0000 7fe12a59c640 1 -- 172.21.15.98:0/1639467296 learned_addr learned my addr 172.21.15.98:0/1639467296 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.433+0000 7fe129d9b640 1 --2- 172.21.15.98:0/1639467296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30cbb40 0x55b5b30ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.433+0000 7fe11b7fe640 1 --2- 172.21.15.98:0/3300664591 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1140a33b0 0x7fe1140a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.434+0000 7fe129d9b640 1 --2- 172.21.15.98:0/1639467296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30d9780 0x55b5b30dbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.434+0000 7fe12ad9d640 1 --2- 172.21.15.98:0/3300664591 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1140a33b0 0x7fe1140a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.434+0000 7fe12a59c640 1 -- 172.21.15.98:0/1639467296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30cbb40 msgr2=0x55b5b30ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.434+0000 7fe12a59c640 1 --2- 172.21.15.98:0/1639467296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30cbb40 0x55b5b30ca090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.434+0000 7fe12a59c640 1 -- 172.21.15.98:0/1639467296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30d9780 msgr2=0x55b5b30dbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.434+0000 7fe12a59c640 1 --2- 172.21.15.98:0/1639467296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30d9780 0x55b5b30dbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.434+0000 7fe12a59c640 1 -- 172.21.15.98:0/1639467296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5b2fbc3a0 con 0x55b5b30ca5d0 2024-09-19T08:07:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.434+0000 7fe11b7fe640 1 -- 172.21.15.98:0/3300664591 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fe1240cf660 con 0x7fe12c106a80 2024-09-19T08:07:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.434+0000 7fe12a59c640 1 --2- 172.21.15.98:0/1639467296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30ca5d0 0x55b5b30caa10 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe12002a940 tx=0x7fe120005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968d2ce9c2cd1f9d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.434+0000 7fe118ff9640 1 -- 172.21.15.98:0/1639467296 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12003f070 con 0x55b5b30ca5d0 2024-09-19T08:07:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.434+0000 7fe12ad9d640 1 --2- 172.21.15.98:0/3300664591 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1140a33b0 0x7fe1140a5870 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe12400d720 tx=0x7fe1240c6740 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.434+0000 7fe118ff9640 1 -- 172.21.15.98:0/1639467296 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe120004030 con 0x55b5b30ca5d0 2024-09-19T08:07:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.434+0000 7fe118ff9640 1 -- 172.21.15.98:0/1639467296 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe120035070 con 0x55b5b30ca5d0 2024-09-19T08:07:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.435+0000 7fe132cc3740 1 -- 172.21.15.98:0/1639467296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30ca5d0 msgr2=0x55b5b30caa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.435+0000 7fe132cc3740 1 --2- 172.21.15.98:0/1639467296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30ca5d0 0x55b5b30caa10 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe12002a940 tx=0x7fe120005e80 comp rx=0 tx=0).stop 2024-09-19T08:07:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.435+0000 7fe132cc3740 1 -- 172.21.15.98:0/1639467296 shutdown_connections 2024-09-19T08:07:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.435+0000 7fe132cc3740 1 --2- 172.21.15.98:0/1639467296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30d9780 0x55b5b30dbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.435+0000 7fe132cc3740 1 --2- 172.21.15.98:0/1639467296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30ca5d0 0x55b5b30caa10 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.435+0000 7fe132cc3740 1 --2- 172.21.15.98:0/1639467296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30cbb40 0x55b5b30ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.435+0000 7fe132cc3740 1 -- 172.21.15.98:0/1639467296 >> 172.21.15.98:0/1639467296 conn(0x55b5b3093310 msgr2=0x55b5b30c6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.435+0000 7fe132cc3740 1 -- 172.21.15.98:0/1639467296 shutdown_connections 2024-09-19T08:07:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.435+0000 7fe132cc3740 1 -- 172.21.15.98:0/1639467296 wait complete. 2024-09-19T08:07:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.435+0000 7fe132cc3740 1 -- start start 2024-09-19T08:07:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe132cc3740 1 Processor -- start 2024-09-19T08:07:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe132cc3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30ca5d0 0x55b5b30d91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe132cc3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30cbb40 0x55b5b30d6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe132cc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30d9780 0x55b5b30d6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe132cc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b5b30dc420 con 0x55b5b30d9780 2024-09-19T08:07:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe132cc3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b5b2feb060 con 0x55b5b30cbb40 2024-09-19T08:07:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe132cc3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b5b30dc5a0 con 0x55b5b30ca5d0 2024-09-19T08:07:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe132cc3740 10 client.0 _ll_register_callbacks cb 0x55b5b2d3e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe129d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30d9780 0x55b5b30d6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe129d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30d9780 0x55b5b30d6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53564/0 (socket says 172.21.15.98:53564) 2024-09-19T08:07:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe129d9b640 1 -- 172.21.15.98:0/189451173 learned_addr learned my addr 172.21.15.98:0/189451173 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe129d9b640 1 --2- 172.21.15.98:0/189451173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30ca5d0 0x55b5b30d91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe12a59c640 1 --2- 172.21.15.98:0/189451173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30cbb40 0x55b5b30d6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe129d9b640 1 -- 172.21.15.98:0/189451173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30ca5d0 msgr2=0x55b5b30d91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe129d9b640 1 --2- 172.21.15.98:0/189451173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30ca5d0 0x55b5b30d91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe129d9b640 1 -- 172.21.15.98:0/189451173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30cbb40 msgr2=0x55b5b30d6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe129d9b640 1 --2- 172.21.15.98:0/189451173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30cbb40 0x55b5b30d6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe129d9b640 1 -- 172.21.15.98:0/189451173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5b2da13c0 con 0x55b5b30d9780 2024-09-19T08:07:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe129d9b640 1 --2- 172.21.15.98:0/189451173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30d9780 0x55b5b30d6a60 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe11c012e90 tx=0x7fe11c009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77bd2cd86aa613b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe111ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe111ffb640 1 -- 172.21.15.98:0/189451173 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11c004b20 con 0x55b5b30d9780 2024-09-19T08:07:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe111ffb640 1 -- 172.21.15.98:0/189451173 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fe11c004cc0 con 0x55b5b30d9780 2024-09-19T08:07:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.436+0000 7fe132cc3740 1 -- 172.21.15.98:0/189451173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe120004030 con 0x55b5b30d9780 2024-09-19T08:07:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.437+0000 7fe111ffb640 1 -- 172.21.15.98:0/189451173 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe11c010440 con 0x55b5b30d9780 2024-09-19T08:07:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.437+0000 7fe132cc3740 10 client.16716 fetch_fsmap learned FSMap version 31 2024-09-19T08:07:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.437+0000 7fe132cc3740 1 -- 172.21.15.98:0/189451173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b5b30e7f40 con 0x55b5b30d9780 2024-09-19T08:07:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.437+0000 7fe111ffb640 1 -- 172.21.15.98:0/189451173 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe120004030 con 0x55b5b30d9780 2024-09-19T08:07:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.437+0000 7fe111ffb640 1 -- 172.21.15.98:0/189451173 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fe11c029650 con 0x55b5b30d9780 2024-09-19T08:07:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.437+0000 7fe132cc3740 10 client.16716 fetch_fsmap finished waiting for FSMap version 31 2024-09-19T08:07:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.437+0000 7fe132cc3740 10 client.16716 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:07:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.437+0000 7fe132cc3740 20 client.16716 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.437+0000 7fe132cc3740 1 --2- 172.21.15.98:0/189451173 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55b5b30e9450 0x55b5b30eb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.437+0000 7fe132cc3740 4 client.16716 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.437+0000 7fe132cc3740 1 -- 172.21.15.98:0/189451173 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5b303d860 con 0x55b5b30e9450 2024-09-19T08:07:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.437+0000 7fe12a59c640 1 --2- 172.21.15.98:0/189451173 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55b5b30e9450 0x55b5b30eb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.438+0000 7fe12a59c640 1 --2- 172.21.15.98:0/189451173 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55b5b30e9450 0x55b5b30eb8d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d7631220f4ed8df6 server_cookie=30052d1c079512a6 in_seq=0 out_seq=0 2024-09-19T08:07:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.438+0000 7fe111ffb640 10 client.16716 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:07:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.438+0000 7fe111ffb640 1 -- 172.21.15.98:0/189451173 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b5b303d860 con 0x55b5b30e9450 2024-09-19T08:07:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.438+0000 7fe111ffb640 10 client.16716 handle_command_reply: tid=0 2024-09-19T08:07:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.439+0000 7fe132cc3740 1 client.16716 shutdown 2024-09-19T08:07:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.439+0000 7fe132cc3740 1 -- 172.21.15.98:0/189451173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30d9780 msgr2=0x55b5b30d6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.439+0000 7fe132cc3740 1 --2- 172.21.15.98:0/189451173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30d9780 0x55b5b30d6a60 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe11c012e90 tx=0x7fe11c009ee0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.439+0000 7fe132cc3740 1 -- 172.21.15.98:0/189451173 shutdown_connections 2024-09-19T08:07:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.439+0000 7fe132cc3740 1 --2- 172.21.15.98:0/189451173 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55b5b30e9450 0x55b5b30eb8d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.439+0000 7fe132cc3740 1 --2- 172.21.15.98:0/189451173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30d9780 0x55b5b30d6a60 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.439+0000 7fe132cc3740 1 --2- 172.21.15.98:0/189451173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30cbb40 0x55b5b30d6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.439+0000 7fe132cc3740 1 --2- 172.21.15.98:0/189451173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30ca5d0 0x55b5b30d91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.439+0000 7fe132cc3740 1 -- 172.21.15.98:0/189451173 >> 172.21.15.98:0/189451173 conn(0x55b5b3093310 msgr2=0x55b5b30cc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.439+0000 7fe111ffb640 0 client.16716 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:07:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.439+0000 7fe132cc3740 1 -- 172.21.15.98:0/189451173 shutdown_connections 2024-09-19T08:07:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.439+0000 7fe132cc3740 1 -- 172.21.15.98:0/189451173 wait complete. 2024-09-19T08:07:52.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.439+0000 7fe132cc3740 20 client.16716 trim_cache size 0 max 16384 2024-09-19T08:07:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe132cc3740 1 Processor -- start 2024-09-19T08:07:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe132cc3740 1 -- start start 2024-09-19T08:07:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe132cc3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30ca5d0 0x55b5b30e56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe132cc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30cbb40 0x55b5b30c6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe132cc3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30d9780 0x55b5b30c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe132cc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c1ab570 con 0x55b5b30cbb40 2024-09-19T08:07:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe132cc3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c1ad0e0 con 0x55b5b30d9780 2024-09-19T08:07:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe132cc3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c1abc40 con 0x55b5b30ca5d0 2024-09-19T08:07:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe12a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30cbb40 0x55b5b30c6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe12a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30cbb40 0x55b5b30c6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53580/0 (socket says 172.21.15.98:53580) 2024-09-19T08:07:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe12a59c640 1 -- 172.21.15.98:0/1140180042 learned_addr learned my addr 172.21.15.98:0/1140180042 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe129d9b640 1 --2- 172.21.15.98:0/1140180042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30ca5d0 0x55b5b30e56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe12a59c640 1 -- 172.21.15.98:0/1140180042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30ca5d0 msgr2=0x55b5b30e56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe129d9b640 1 --2- 172.21.15.98:0/1140180042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30d9780 0x55b5b30c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe12a59c640 1 --2- 172.21.15.98:0/1140180042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30ca5d0 0x55b5b30e56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe12a59c640 1 -- 172.21.15.98:0/1140180042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30d9780 msgr2=0x55b5b30c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe12a59c640 1 --2- 172.21.15.98:0/1140180042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30d9780 0x55b5b30c6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe12a59c640 1 -- 172.21.15.98:0/1140180042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5b30cd120 con 0x55b5b30cbb40 2024-09-19T08:07:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe129d9b640 1 --2- 172.21.15.98:0/1140180042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30d9780 0x55b5b30c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.452+0000 7fe12a59c640 1 --2- 172.21.15.98:0/1140180042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30cbb40 0x55b5b30c6700 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x55b5b30c7ac0 tx=0x7fe120000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8bf267597f1f8bf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.453+0000 7fe113fff640 1 -- 172.21.15.98:0/1140180042 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe120035070 con 0x55b5b30cbb40 2024-09-19T08:07:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.453+0000 7fe113fff640 1 -- 172.21.15.98:0/1140180042 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe120004460 con 0x55b5b30cbb40 2024-09-19T08:07:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.453+0000 7fe113fff640 1 -- 172.21.15.98:0/1140180042 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12003f040 con 0x55b5b30cbb40 2024-09-19T08:07:52.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.453+0000 7fe132cc3740 1 -- 172.21.15.98:0/1140180042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30cbb40 msgr2=0x55b5b30c6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:52.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.453+0000 7fe132cc3740 1 --2- 172.21.15.98:0/1140180042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30cbb40 0x55b5b30c6700 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x55b5b30c7ac0 tx=0x7fe120000bc0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.453+0000 7fe132cc3740 1 -- 172.21.15.98:0/1140180042 shutdown_connections 2024-09-19T08:07:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.453+0000 7fe132cc3740 1 --2- 172.21.15.98:0/1140180042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30d9780 0x55b5b30c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.453+0000 7fe132cc3740 1 --2- 172.21.15.98:0/1140180042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30cbb40 0x55b5b30c6700 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.453+0000 7fe132cc3740 1 --2- 172.21.15.98:0/1140180042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30ca5d0 0x55b5b30e56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.453+0000 7fe132cc3740 1 -- 172.21.15.98:0/1140180042 >> 172.21.15.98:0/1140180042 conn(0x55b5b3093310 msgr2=0x55b5b30d6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.453+0000 7fe132cc3740 1 -- 172.21.15.98:0/1140180042 shutdown_connections 2024-09-19T08:07:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.453+0000 7fe132cc3740 1 -- 172.21.15.98:0/1140180042 wait complete. 2024-09-19T08:07:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe132cc3740 1 -- start start 2024-09-19T08:07:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe132cc3740 1 Processor -- start 2024-09-19T08:07:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe132cc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30ca5d0 0x55b5b30e1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe132cc3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30cbb40 0x55b5b30d2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe132cc3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30d9780 0x55b5b30d3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe132cc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c1aca10 con 0x55b5b30ca5d0 2024-09-19T08:07:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe132cc3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c1ac340 con 0x55b5b30cbb40 2024-09-19T08:07:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe132cc3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c1a9550 con 0x55b5b30d9780 2024-09-19T08:07:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe132cc3740 10 client.0 _ll_register_callbacks cb 0x7fe11c00e350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe129d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30ca5d0 0x55b5b30e1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe129d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30ca5d0 0x55b5b30e1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53596/0 (socket says 172.21.15.98:53596) 2024-09-19T08:07:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe129d9b640 1 -- 172.21.15.98:0/2663204413 learned_addr learned my addr 172.21.15.98:0/2663204413 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe12a59c640 1 --2- 172.21.15.98:0/2663204413 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30cbb40 0x55b5b30d2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe129d9b640 1 -- 172.21.15.98:0/2663204413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30d9780 msgr2=0x55b5b30d3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe129d9b640 1 --2- 172.21.15.98:0/2663204413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30d9780 0x55b5b30d3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe129d9b640 1 -- 172.21.15.98:0/2663204413 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30cbb40 msgr2=0x55b5b30d2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe129d9b640 1 --2- 172.21.15.98:0/2663204413 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30cbb40 0x55b5b30d2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe129d9b640 1 -- 172.21.15.98:0/2663204413 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5b30d3a30 con 0x55b5b30ca5d0 2024-09-19T08:07:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.454+0000 7fe12a59c640 1 --2- 172.21.15.98:0/2663204413 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30cbb40 0x55b5b30d2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe129d9b640 1 --2- 172.21.15.98:0/2663204413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30ca5d0 0x55b5b30e1d00 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fe11c00ec60 tx=0x7fe11c012640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b799a150b21f37 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe111ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe111ffb640 1 -- 172.21.15.98:0/2663204413 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11c0122c0 con 0x55b5b30ca5d0 2024-09-19T08:07:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe111ffb640 1 -- 172.21.15.98:0/2663204413 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fe11c029430 con 0x55b5b30ca5d0 2024-09-19T08:07:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe132cc3740 1 -- 172.21.15.98:0/2663204413 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe120004460 con 0x55b5b30ca5d0 2024-09-19T08:07:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe111ffb640 1 -- 172.21.15.98:0/2663204413 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe11c029630 con 0x55b5b30ca5d0 2024-09-19T08:07:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe132cc3740 10 client.16722 fetch_fsmap learned FSMap version 31 2024-09-19T08:07:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe132cc3740 1 -- 172.21.15.98:0/2663204413 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b5b3153ec0 con 0x55b5b30ca5d0 2024-09-19T08:07:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe111ffb640 1 -- 172.21.15.98:0/2663204413 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe120004460 con 0x55b5b30ca5d0 2024-09-19T08:07:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe111ffb640 1 -- 172.21.15.98:0/2663204413 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fe11c005040 con 0x55b5b30ca5d0 2024-09-19T08:07:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe132cc3740 10 client.16722 fetch_fsmap finished waiting for FSMap version 31 2024-09-19T08:07:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe132cc3740 10 client.16722 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:07:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe132cc3740 20 client.16722 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe132cc3740 1 --2- 172.21.15.98:0/2663204413 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55b5b3154640 0x55b5b3156a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe132cc3740 4 client.16722 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:07:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.455+0000 7fe132cc3740 1 -- 172.21.15.98:0/2663204413 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b5b3037d70 con 0x55b5b3154640 2024-09-19T08:07:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe12a59c640 1 --2- 172.21.15.98:0/2663204413 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55b5b3154640 0x55b5b3156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe12a59c640 1 --2- 172.21.15.98:0/2663204413 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55b5b3154640 0x55b5b3156a60 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=83d410139f89764 server_cookie=f2b683a28e21ff92 in_seq=0 out_seq=0 2024-09-19T08:07:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe111ffb640 10 client.16722 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:07:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe111ffb640 1 -- 172.21.15.98:0/2663204413 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b5b3037d70 con 0x55b5b3154640 2024-09-19T08:07:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe111ffb640 10 client.16722 handle_command_reply: tid=0 2024-09-19T08:07:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe132cc3740 1 client.16722 shutdown 2024-09-19T08:07:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe132cc3740 1 -- 172.21.15.98:0/2663204413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30ca5d0 msgr2=0x55b5b30e1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe132cc3740 1 --2- 172.21.15.98:0/2663204413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30ca5d0 0x55b5b30e1d00 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fe11c00ec60 tx=0x7fe11c012640 comp rx=0 tx=0).stop 2024-09-19T08:07:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe132cc3740 1 -- 172.21.15.98:0/2663204413 shutdown_connections 2024-09-19T08:07:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe132cc3740 1 --2- 172.21.15.98:0/2663204413 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55b5b3154640 0x55b5b3156a60 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe111ffb640 0 client.16722 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:07:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe132cc3740 1 --2- 172.21.15.98:0/2663204413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5b30d9780 0x55b5b30d3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe132cc3740 1 --2- 172.21.15.98:0/2663204413 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b5b30cbb40 0x55b5b30d2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe132cc3740 1 --2- 172.21.15.98:0/2663204413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b5b30ca5d0 0x55b5b30e1d00 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.456+0000 7fe132cc3740 1 -- 172.21.15.98:0/2663204413 >> 172.21.15.98:0/2663204413 conn(0x55b5b3093310 msgr2=0x55b5b30c6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe132cc3740 1 -- 172.21.15.98:0/2663204413 shutdown_connections 2024-09-19T08:07:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe132cc3740 1 -- 172.21.15.98:0/2663204413 wait complete. 2024-09-19T08:07:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe132cc3740 20 client.16722 trim_cache size 0 max 16384 2024-09-19T08:07:52.464 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:07:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe1308ea640 1 -- 172.21.15.98:0/3300664591 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1140a33b0 msgr2=0x7fe1140a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe1308ea640 1 --2- 172.21.15.98:0/3300664591 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1140a33b0 0x7fe1140a5870 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe12400d720 tx=0x7fe1240c6740 comp rx=0 tx=0).stop 2024-09-19T08:07:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe1308ea640 1 -- 172.21.15.98:0/3300664591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe12c106a80 msgr2=0x7fe12c19cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe1308ea640 1 --2- 172.21.15.98:0/3300664591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe12c106a80 0x7fe12c19cd00 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe12400d750 tx=0x7fe12400dc10 comp rx=0 tx=0).stop 2024-09-19T08:07:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe1308ea640 1 -- 172.21.15.98:0/3300664591 shutdown_connections 2024-09-19T08:07:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe1308ea640 1 --2- 172.21.15.98:0/3300664591 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1140a33b0 0x7fe1140a5870 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe1308ea640 1 --2- 172.21.15.98:0/3300664591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe12c106a80 0x7fe12c19cd00 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe1308ea640 1 --2- 172.21.15.98:0/3300664591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe12c101420 0x7fe12c198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe1308ea640 1 --2- 172.21.15.98:0/3300664591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe12c100a80 0x7fe12c198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe1308ea640 1 -- 172.21.15.98:0/3300664591 >> 172.21.15.98:0/3300664591 conn(0x7fe12c0fc7a0 msgr2=0x7fe12c104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe1308ea640 1 -- 172.21.15.98:0/3300664591 shutdown_connections 2024-09-19T08:07:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:52.457+0000 7fe1308ea640 1 -- 172.21.15.98:0/3300664591 wait complete. 2024-09-19T08:07:53.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:07:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.969+0000 7f78289aa640 1 -- 172.21.15.98:0/237191728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7824102790 msgr2=0x7f78241067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.969+0000 7f78289aa640 1 --2- 172.21.15.98:0/237191728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7824102790 0x7f78241067e0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f7818009900 tx=0x7f781802a410 comp rx=0 tx=0).stop 2024-09-19T08:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.969+0000 7f78289aa640 1 -- 172.21.15.98:0/237191728 shutdown_connections 2024-09-19T08:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.969+0000 7f78289aa640 1 --2- 172.21.15.98:0/237191728 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7824106f10 0x7f782410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.969+0000 7f78289aa640 1 --2- 172.21.15.98:0/237191728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7824102790 0x7f78241067e0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.969+0000 7f78289aa640 1 --2- 172.21.15.98:0/237191728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7824101de0 0x7f78241021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.969+0000 7f78289aa640 1 -- 172.21.15.98:0/237191728 >> 172.21.15.98:0/237191728 conn(0x7f78240fd8c0 msgr2=0x7f78240ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.969+0000 7f78289aa640 1 -- 172.21.15.98:0/237191728 shutdown_connections 2024-09-19T08:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.969+0000 7f78289aa640 1 -- 172.21.15.98:0/237191728 wait complete. 2024-09-19T08:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.970+0000 7f78289aa640 1 Processor -- start 2024-09-19T08:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.970+0000 7f78289aa640 1 -- start start 2024-09-19T08:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f78289aa640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7824101de0 0x7f78241985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f78289aa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7824102790 0x7f7824198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f78289aa640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7824106f10 0x7f782419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f78289aa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7824112010 con 0x7f7824102790 2024-09-19T08:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f78289aa640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7824111e90 con 0x7f7824106f10 2024-09-19T08:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f78289aa640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7824112190 con 0x7f7824101de0 2024-09-19T08:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f7821d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7824102790 0x7f7824198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f7821d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7824102790 0x7f7824198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53626/0 (socket says 172.21.15.98:53626) 2024-09-19T08:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f7821d9b640 1 -- 172.21.15.98:0/1124862139 learned_addr learned my addr 172.21.15.98:0/1124862139 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f7822d9d640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7824106f10 0x7f782419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f782259c640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7824101de0 0x7f78241985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f7821d9b640 1 -- 172.21.15.98:0/1124862139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7824101de0 msgr2=0x7f78241985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f7821d9b640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7824101de0 0x7f78241985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f7821d9b640 1 -- 172.21.15.98:0/1124862139 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7824106f10 msgr2=0x7f782419ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f7821d9b640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7824106f10 0x7f782419ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f7821d9b640 1 -- 172.21.15.98:0/1124862139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f782419d560 con 0x7f7824102790 2024-09-19T08:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f782259c640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7824101de0 0x7f78241985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f7821d9b640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7824102790 0x7f7824198af0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7818005ab0 tx=0x7f78180042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f78137fe640 1 -- 172.21.15.98:0/1124862139 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781803f030 con 0x7f7824102790 2024-09-19T08:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f78137fe640 1 -- 172.21.15.98:0/1124862139 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7818003ce0 con 0x7f7824102790 2024-09-19T08:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f78289aa640 1 -- 172.21.15.98:0/1124862139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f782419d7f0 con 0x7f7824102790 2024-09-19T08:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.971+0000 7f78137fe640 1 -- 172.21.15.98:0/1124862139 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818035070 con 0x7f7824102790 2024-09-19T08:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.972+0000 7f78289aa640 1 -- 172.21.15.98:0/1124862139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78241a5130 con 0x7f7824102790 2024-09-19T08:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.974+0000 7f78137fe640 1 -- 172.21.15.98:0/1124862139 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f7818042030 con 0x7f7824102790 2024-09-19T08:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.974+0000 7f782ad83740 1 Processor -- start 2024-09-19T08:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.974+0000 7f782ad83740 1 -- start start 2024-09-19T08:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.974+0000 7f782ad83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfa1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.974+0000 7f782ad83740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfa1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.974+0000 7f782ad83740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 0x55cbcbfb2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.974+0000 7f782ad83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cbcbf072f0 con 0x55cbcbfa2b40 2024-09-19T08:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.974+0000 7f782ad83740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cbcbeaecf0 con 0x55cbcbfa15d0 2024-09-19T08:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.974+0000 7f782ad83740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cbcbf032b0 con 0x55cbcbfb0780 2024-09-19T08:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.974+0000 7f7822d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfa1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.974+0000 7f7822d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfa1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53638/0 (socket says 172.21.15.98:53638) 2024-09-19T08:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.974+0000 7f7822d9d640 1 -- 172.21.15.98:0/3537035368 learned_addr learned my addr 172.21.15.98:0/3537035368 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f782259c640 1 --2- 172.21.15.98:0/3537035368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfa1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f7821d9b640 1 --2- 172.21.15.98:0/3537035368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 0x55cbcbfb2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f7822d9d640 1 -- 172.21.15.98:0/3537035368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 msgr2=0x55cbcbfb2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f7822d9d640 1 --2- 172.21.15.98:0/3537035368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 0x55cbcbfb2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f7822d9d640 1 -- 172.21.15.98:0/3537035368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 msgr2=0x55cbcbfa1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f7822d9d640 1 --2- 172.21.15.98:0/3537035368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfa1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f7822d9d640 1 -- 172.21.15.98:0/3537035368 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbcbe933a0 con 0x55cbcbfa2b40 2024-09-19T08:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f7821d9b640 1 --2- 172.21.15.98:0/3537035368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 0x55cbcbfb2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f78137fe640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f780c0a33b0 0x7f780c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f7822d9d640 1 --2- 172.21.15.98:0/3537035368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfa1090 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f781c00a430 tx=0x7f781c009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2841576d12d2fc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f782259c640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f780c0a33b0 0x7f780c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f78137fe640 1 -- 172.21.15.98:0/1124862139 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f78180f05e0 con 0x7f7824102790 2024-09-19T08:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f7810ff9640 1 -- 172.21.15.98:0/3537035368 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c013790 con 0x55cbcbfa2b40 2024-09-19T08:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f7810ff9640 1 -- 172.21.15.98:0/3537035368 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f781c019070 con 0x55cbcbfa2b40 2024-09-19T08:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.975+0000 7f7810ff9640 1 -- 172.21.15.98:0/3537035368 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c014de0 con 0x55cbcbfa2b40 2024-09-19T08:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.976+0000 7f782ad83740 1 -- 172.21.15.98:0/3537035368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa2b40 msgr2=0x55cbcbfa1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.976+0000 7f782259c640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f780c0a33b0 0x7f780c0a5870 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55cbcbf9eac0 tx=0x7f7814008040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.976+0000 7f782ad83740 1 --2- 172.21.15.98:0/3537035368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfa1090 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f781c00a430 tx=0x7f781c009650 comp rx=0 tx=0).stop 2024-09-19T08:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.976+0000 7f782ad83740 1 -- 172.21.15.98:0/3537035368 shutdown_connections 2024-09-19T08:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.976+0000 7f782ad83740 1 --2- 172.21.15.98:0/3537035368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 0x55cbcbfb2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.976+0000 7f782ad83740 1 --2- 172.21.15.98:0/3537035368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfa1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.976+0000 7f782ad83740 1 --2- 172.21.15.98:0/3537035368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfa1090 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.976+0000 7f782ad83740 1 -- 172.21.15.98:0/3537035368 >> 172.21.15.98:0/3537035368 conn(0x55cbcbf6a310 msgr2=0x55cbcbf9dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.976+0000 7f782ad83740 1 -- 172.21.15.98:0/3537035368 shutdown_connections 2024-09-19T08:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.976+0000 7f782ad83740 1 -- 172.21.15.98:0/3537035368 wait complete. 2024-09-19T08:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.976+0000 7f782ad83740 1 -- start start 2024-09-19T08:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.976+0000 7f782ad83740 1 Processor -- start 2024-09-19T08:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782ad83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfb01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782ad83740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfad480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782ad83740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 0x55cbcbfadc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782ad83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cbcbfb3480 con 0x55cbcbfa15d0 2024-09-19T08:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782ad83740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cbcbec2060 con 0x55cbcbfa2b40 2024-09-19T08:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782ad83740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cbcbfb3600 con 0x55cbcbfb0780 2024-09-19T08:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782ad83740 10 client.0 _ll_register_callbacks cb 0x55cbcbc158c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfb01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfb01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53644/0 (socket says 172.21.15.98:53644) 2024-09-19T08:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782259c640 1 -- 172.21.15.98:0/1114088943 learned_addr learned my addr 172.21.15.98:0/1114088943 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f7822d9d640 1 --2- 172.21.15.98:0/1114088943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 0x55cbcbfadc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f7821d9b640 1 --2- 172.21.15.98:0/1114088943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782259c640 1 -- 172.21.15.98:0/1114088943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 msgr2=0x55cbcbfadc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782259c640 1 --2- 172.21.15.98:0/1114088943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 0x55cbcbfadc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782259c640 1 -- 172.21.15.98:0/1114088943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa2b40 msgr2=0x55cbcbfad480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782259c640 1 --2- 172.21.15.98:0/1114088943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfad480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782259c640 1 -- 172.21.15.98:0/1114088943 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbcbc783c0 con 0x55cbcbfa15d0 2024-09-19T08:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f7821d9b640 1 --2- 172.21.15.98:0/1114088943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f782259c640 1 --2- 172.21.15.98:0/1114088943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfb01d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7814014710 tx=0x7f78140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3c7657660e275 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.977+0000 7f7809ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.978+0000 7f7809ffb640 1 -- 172.21.15.98:0/1114088943 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781401b050 con 0x55cbcbfa15d0 2024-09-19T08:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.978+0000 7f7809ffb640 1 -- 172.21.15.98:0/1114088943 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f7814015820 con 0x55cbcbfa15d0 2024-09-19T08:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.978+0000 7f782ad83740 1 -- 172.21.15.98:0/1114088943 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f781c019070 con 0x55cbcbfa15d0 2024-09-19T08:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.978+0000 7f7809ffb640 1 -- 172.21.15.98:0/1114088943 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f781402e050 con 0x55cbcbfa15d0 2024-09-19T08:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.978+0000 7f782ad83740 10 client.16734 fetch_fsmap learned FSMap version 31 2024-09-19T08:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.978+0000 7f782ad83740 1 -- 172.21.15.98:0/1114088943 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cbcbfbef60 con 0x55cbcbfa15d0 2024-09-19T08:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.978+0000 7f7809ffb640 1 -- 172.21.15.98:0/1114088943 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f781c019070 con 0x55cbcbfa15d0 2024-09-19T08:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.978+0000 7f7809ffb640 1 -- 172.21.15.98:0/1114088943 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f78140133c0 con 0x55cbcbfa15d0 2024-09-19T08:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.978+0000 7f782ad83740 10 client.16734 fetch_fsmap finished waiting for FSMap version 31 2024-09-19T08:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.978+0000 7f782ad83740 10 client.16734 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.978+0000 7f782ad83740 20 client.16734 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.978+0000 7f782ad83740 1 --2- 172.21.15.98:0/1114088943 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55cbcbfc0470 0x55cbcbfc28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.978+0000 7f782ad83740 4 client.16734 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.978+0000 7f782ad83740 1 -- 172.21.15.98:0/1114088943 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbcbf14860 con 0x55cbcbfc0470 2024-09-19T08:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.979+0000 7f7821d9b640 1 --2- 172.21.15.98:0/1114088943 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55cbcbfc0470 0x55cbcbfc28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.979+0000 7f7809ffb640 10 client.16734 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.979+0000 7f7821d9b640 1 --2- 172.21.15.98:0/1114088943 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55cbcbfc0470 0x55cbcbfc28f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=96310e5f048b2a10 server_cookie=c875c1acb4289f79 in_seq=0 out_seq=0 2024-09-19T08:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f7809ffb640 1 -- 172.21.15.98:0/1114088943 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cbcbf14860 con 0x55cbcbfc0470 2024-09-19T08:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f7809ffb640 10 client.16734 handle_command_reply: tid=0 2024-09-19T08:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f782ad83740 1 client.16734 shutdown 2024-09-19T08:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f782ad83740 1 -- 172.21.15.98:0/1114088943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa15d0 msgr2=0x55cbcbfb01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f782ad83740 1 --2- 172.21.15.98:0/1114088943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfb01d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7814014710 tx=0x7f78140046c0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f782ad83740 1 -- 172.21.15.98:0/1114088943 shutdown_connections 2024-09-19T08:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f782ad83740 1 --2- 172.21.15.98:0/1114088943 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55cbcbfc0470 0x55cbcbfc28f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f782ad83740 1 --2- 172.21.15.98:0/1114088943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 0x55cbcbfadc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f782ad83740 1 --2- 172.21.15.98:0/1114088943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f7809ffb640 0 client.16734 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f782ad83740 1 --2- 172.21.15.98:0/1114088943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfb01d0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f782ad83740 1 -- 172.21.15.98:0/1114088943 >> 172.21.15.98:0/1114088943 conn(0x55cbcbf6a310 msgr2=0x55cbcbfa3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f782ad83740 1 -- 172.21.15.98:0/1114088943 shutdown_connections 2024-09-19T08:07:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f782ad83740 1 -- 172.21.15.98:0/1114088943 wait complete. 2024-09-19T08:07:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.980+0000 7f782ad83740 20 client.16734 trim_cache size 0 max 16384 2024-09-19T08:07:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f782ad83740 1 Processor -- start 2024-09-19T08:07:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f782ad83740 1 -- start start 2024-09-19T08:07:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f782ad83740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f782ad83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfc81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f782ad83740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 0x55cbcbfc8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f782ad83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78241ab6c0 con 0x55cbcbfa2b40 2024-09-19T08:07:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f782ad83740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78241ad230 con 0x55cbcbfa15d0 2024-09-19T08:07:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f782ad83740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78241abd90 con 0x55cbcbfb0780 2024-09-19T08:07:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f7821d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfc81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f7821d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfc81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53660/0 (socket says 172.21.15.98:53660) 2024-09-19T08:07:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f7821d9b640 1 -- 172.21.15.98:0/1946843865 learned_addr learned my addr 172.21.15.98:0/1946843865 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f782259c640 1 --2- 172.21.15.98:0/1946843865 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f7822d9d640 1 --2- 172.21.15.98:0/1946843865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 0x55cbcbfc8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f7821d9b640 1 -- 172.21.15.98:0/1946843865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 msgr2=0x55cbcbfc8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f7821d9b640 1 --2- 172.21.15.98:0/1946843865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 0x55cbcbfc8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f7821d9b640 1 -- 172.21.15.98:0/1946843865 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 msgr2=0x55cbcbfac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f7821d9b640 1 --2- 172.21.15.98:0/1946843865 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfac330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f7821d9b640 1 -- 172.21.15.98:0/1946843865 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbcbf9d4f0 con 0x55cbcbfa2b40 2024-09-19T08:07:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f7822d9d640 1 --2- 172.21.15.98:0/1946843865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 0x55cbcbfc8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.993+0000 7f7821d9b640 1 --2- 172.21.15.98:0/1946843865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfc81c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f7818006010 tx=0x7f78180045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c4e419b3a4a032 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.994+0000 7f780bfff640 1 -- 172.21.15.98:0/1946843865 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818049040 con 0x55cbcbfa2b40 2024-09-19T08:07:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.994+0000 7f780bfff640 1 -- 172.21.15.98:0/1946843865 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78180c0ca0 con 0x55cbcbfa2b40 2024-09-19T08:07:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.994+0000 7f780bfff640 1 -- 172.21.15.98:0/1946843865 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78180b8f60 con 0x55cbcbfa2b40 2024-09-19T08:07:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.994+0000 7f782ad83740 1 -- 172.21.15.98:0/1946843865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa2b40 msgr2=0x55cbcbfc81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.994+0000 7f782ad83740 1 --2- 172.21.15.98:0/1946843865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfc81c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f7818006010 tx=0x7f78180045e0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.994+0000 7f782ad83740 1 -- 172.21.15.98:0/1946843865 shutdown_connections 2024-09-19T08:07:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.994+0000 7f782ad83740 1 --2- 172.21.15.98:0/1946843865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfb0780 0x55cbcbfc8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.994+0000 7f782ad83740 1 --2- 172.21.15.98:0/1946843865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfc81c0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.994+0000 7f782ad83740 1 --2- 172.21.15.98:0/1946843865 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.994+0000 7f782ad83740 1 -- 172.21.15.98:0/1946843865 >> 172.21.15.98:0/1946843865 conn(0x55cbcbf6a310 msgr2=0x55cbcbfabc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.994+0000 7f782ad83740 1 -- 172.21.15.98:0/1946843865 shutdown_connections 2024-09-19T08:07:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.994+0000 7f782ad83740 1 -- 172.21.15.98:0/1946843865 wait complete. 2024-09-19T08:07:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.994+0000 7f782ad83740 1 -- start start 2024-09-19T08:07:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f782ad83740 1 Processor -- start 2024-09-19T08:07:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f782ad83740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfb8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f782ad83740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfb4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f782ad83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfb0780 0x55cbcbfb46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f782ad83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78241acb60 con 0x55cbcbfb0780 2024-09-19T08:07:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f782ad83740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78241ac490 con 0x55cbcbfa15d0 2024-09-19T08:07:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f782ad83740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78241a96a0 con 0x55cbcbfa2b40 2024-09-19T08:07:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f782ad83740 10 client.0 _ll_register_callbacks cb 0x7f781400b6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f7822d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfb0780 0x55cbcbfb46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f7822d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfb0780 0x55cbcbfb46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53668/0 (socket says 172.21.15.98:53668) 2024-09-19T08:07:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f7822d9d640 1 -- 172.21.15.98:0/3457304375 learned_addr learned my addr 172.21.15.98:0/3457304375 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:54.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f7821d9b640 1 --2- 172.21.15.98:0/3457304375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfb4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f782259c640 1 --2- 172.21.15.98:0/3457304375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfb8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f7822d9d640 1 -- 172.21.15.98:0/3457304375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfa2b40 msgr2=0x55cbcbfb4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:54.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f7822d9d640 1 --2- 172.21.15.98:0/3457304375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfb4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f7822d9d640 1 -- 172.21.15.98:0/3457304375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 msgr2=0x55cbcbfb8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:54.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f7822d9d640 1 --2- 172.21.15.98:0/3457304375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfb8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f7822d9d640 1 -- 172.21.15.98:0/3457304375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbcbfb4d70 con 0x55cbcbfb0780 2024-09-19T08:07:54.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f7822d9d640 1 --2- 172.21.15.98:0/3457304375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfb0780 0x55cbcbfb46d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f781c00a430 tx=0x7f781c0137c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de018d12f7c7c7c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f7809ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f7809ffb640 1 -- 172.21.15.98:0/3457304375 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c02c060 con 0x55cbcbfb0780 2024-09-19T08:07:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f7809ffb640 1 -- 172.21.15.98:0/3457304375 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f781c013bc0 con 0x55cbcbfb0780 2024-09-19T08:07:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.995+0000 7f782ad83740 1 -- 172.21.15.98:0/3457304375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78180c0ca0 con 0x55cbcbfb0780 2024-09-19T08:07:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.996+0000 7f7809ffb640 1 -- 172.21.15.98:0/3457304375 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f781c013dc0 con 0x55cbcbfb0780 2024-09-19T08:07:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.996+0000 7f782ad83740 10 client.16740 fetch_fsmap learned FSMap version 31 2024-09-19T08:07:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.996+0000 7f782ad83740 1 -- 172.21.15.98:0/3457304375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cbcc02b000 con 0x55cbcbfb0780 2024-09-19T08:07:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.996+0000 7f7809ffb640 1 -- 172.21.15.98:0/3457304375 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78180c0ca0 con 0x55cbcbfb0780 2024-09-19T08:07:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.996+0000 7f7809ffb640 1 -- 172.21.15.98:0/3457304375 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f781c01e490 con 0x55cbcbfb0780 2024-09-19T08:07:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.996+0000 7f782ad83740 10 client.16740 fetch_fsmap finished waiting for FSMap version 31 2024-09-19T08:07:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.996+0000 7f782ad83740 10 client.16740 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:07:54.033 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:07:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.996+0000 7f782ad83740 20 client.16740 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.996+0000 7f782ad83740 1 --2- 172.21.15.98:0/3457304375 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55cbcc02b590 0x55cbcc02d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.996+0000 7f782ad83740 4 client.16740 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:07:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.996+0000 7f782ad83740 1 -- 172.21.15.98:0/3457304375 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cbcbf0ed70 con 0x55cbcc02b590 2024-09-19T08:07:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.996+0000 7f782259c640 1 --2- 172.21.15.98:0/3457304375 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55cbcc02b590 0x55cbcc02d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f7809ffb640 10 client.16740 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:07:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f782259c640 1 --2- 172.21.15.98:0/3457304375 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55cbcc02b590 0x55cbcc02d9b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6d3fe219c52c202d server_cookie=274ee9c7be7155c9 in_seq=0 out_seq=0 2024-09-19T08:07:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f7809ffb640 1 -- 172.21.15.98:0/3457304375 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cbcbf0ed70 con 0x55cbcc02b590 2024-09-19T08:07:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f7809ffb640 10 client.16740 handle_command_reply: tid=0 2024-09-19T08:07:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f782ad83740 1 client.16740 shutdown 2024-09-19T08:07:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f782ad83740 1 -- 172.21.15.98:0/3457304375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfb0780 msgr2=0x55cbcbfb46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f782ad83740 1 --2- 172.21.15.98:0/3457304375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfb0780 0x55cbcbfb46d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f781c00a430 tx=0x7f781c0137c0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f782ad83740 1 -- 172.21.15.98:0/3457304375 shutdown_connections 2024-09-19T08:07:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f782ad83740 1 --2- 172.21.15.98:0/3457304375 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55cbcc02b590 0x55cbcc02d9b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f782ad83740 1 --2- 172.21.15.98:0/3457304375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbcbfb0780 0x55cbcbfb46d0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f7809ffb640 0 client.16740 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:07:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f782ad83740 1 --2- 172.21.15.98:0/3457304375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbcbfa2b40 0x55cbcbfb4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f782ad83740 1 --2- 172.21.15.98:0/3457304375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cbcbfa15d0 0x55cbcbfb8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f782ad83740 1 -- 172.21.15.98:0/3457304375 >> 172.21.15.98:0/3457304375 conn(0x55cbcbf6a310 msgr2=0x55cbcbf9cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f782ad83740 1 -- 172.21.15.98:0/3457304375 shutdown_connections 2024-09-19T08:07:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f782ad83740 1 -- 172.21.15.98:0/3457304375 wait complete. 2024-09-19T08:07:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.997+0000 7f782ad83740 20 client.16740 trim_cache size 0 max 16384 2024-09-19T08:07:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.998+0000 7f78289aa640 1 -- 172.21.15.98:0/1124862139 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f780c0a33b0 msgr2=0x7f780c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.998+0000 7f78289aa640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f780c0a33b0 0x7f780c0a5870 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55cbcbf9eac0 tx=0x7f7814008040 comp rx=0 tx=0).stop 2024-09-19T08:07:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.998+0000 7f78289aa640 1 -- 172.21.15.98:0/1124862139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7824102790 msgr2=0x7f7824198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.998+0000 7f78289aa640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7824102790 0x7f7824198af0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7818005ab0 tx=0x7f78180042f0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.998+0000 7f78289aa640 1 -- 172.21.15.98:0/1124862139 shutdown_connections 2024-09-19T08:07:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.998+0000 7f78289aa640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f780c0a33b0 0x7f780c0a5870 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.998+0000 7f78289aa640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7824106f10 0x7f782419ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.998+0000 7f78289aa640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7824102790 0x7f7824198af0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.998+0000 7f78289aa640 1 --2- 172.21.15.98:0/1124862139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7824101de0 0x7f78241985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.998+0000 7f78289aa640 1 -- 172.21.15.98:0/1124862139 >> 172.21.15.98:0/1124862139 conn(0x7f78240fd8c0 msgr2=0x7f78240fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.998+0000 7f78289aa640 1 -- 172.21.15.98:0/1124862139 shutdown_connections 2024-09-19T08:07:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:53.998+0000 7f78289aa640 1 -- 172.21.15.98:0/1124862139 wait complete. 2024-09-19T08:07:54.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:53 smithi139 ceph-mon[34874]: pgmap v149: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:07:54.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:53 smithi098 ceph-mon[31333]: pgmap v149: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:07:54.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:53 smithi162 ceph-mon[35260]: pgmap v149: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:07:54.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:07:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:55.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.394+0000 7fd3cc235640 1 -- 172.21.15.98:0/1869629670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c41082a0 msgr2=0x7fd3c410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:55.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.394+0000 7fd3cc235640 1 --2- 172.21.15.98:0/1869629670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c41082a0 0x7fd3c410ec40 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc009930 tx=0x7fd3bc02a420 comp rx=0 tx=0).stop 2024-09-19T08:07:55.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.394+0000 7fd3cc235640 1 -- 172.21.15.98:0/1869629670 shutdown_connections 2024-09-19T08:07:55.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.394+0000 7fd3cc235640 1 --2- 172.21.15.98:0/1869629670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c41082a0 0x7fd3c410ec40 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.394+0000 7fd3cc235640 1 --2- 172.21.15.98:0/1869629670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3c41036e0 0x7fd3c4107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.394+0000 7fd3cc235640 1 --2- 172.21.15.98:0/1869629670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3c4102d30 0x7fd3c4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.394+0000 7fd3cc235640 1 -- 172.21.15.98:0/1869629670 >> 172.21.15.98:0/1869629670 conn(0x7fd3c40fc7d0 msgr2=0x7fd3c40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:55.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.395+0000 7fd3cc235640 1 -- 172.21.15.98:0/1869629670 shutdown_connections 2024-09-19T08:07:55.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.395+0000 7fd3cc235640 1 -- 172.21.15.98:0/1869629670 wait complete. 2024-09-19T08:07:55.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.395+0000 7fd3cc235640 1 Processor -- start 2024-09-19T08:07:55.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.395+0000 7fd3cc235640 1 -- start start 2024-09-19T08:07:55.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.395+0000 7fd3cc235640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3c4102d30 0x7fd3c4100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.395+0000 7fd3cc235640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3c41036e0 0x7fd3c4100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.395+0000 7fd3cc235640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c41082a0 0x7fd3c4101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.395+0000 7fd3cc235640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c4112100 con 0x7fd3c41082a0 2024-09-19T08:07:55.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.395+0000 7fd3cc235640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c4111f80 con 0x7fd3c41036e0 2024-09-19T08:07:55.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3cc235640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c4112280 con 0x7fd3c4102d30 2024-09-19T08:07:55.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3ca7d2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c41082a0 0x7fd3c4101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3ca7d2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c41082a0 0x7fd3c4101410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53688/0 (socket says 172.21.15.98:53688) 2024-09-19T08:07:55.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3ca7d2640 1 -- 172.21.15.98:0/3451584205 learned_addr learned my addr 172.21.15.98:0/3451584205 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:55.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3c41036e0 0x7fd3c4100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3c9fd1640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3c4102d30 0x7fd3c4100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3ca7d2640 1 -- 172.21.15.98:0/3451584205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3c4102d30 msgr2=0x7fd3c4100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:55.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3ca7d2640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3c4102d30 0x7fd3c4100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3ca7d2640 1 -- 172.21.15.98:0/3451584205 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3c41036e0 msgr2=0x7fd3c4100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:55.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3ca7d2640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3c41036e0 0x7fd3c4100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3ca7d2640 1 -- 172.21.15.98:0/3451584205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c41a6a10 con 0x7fd3c41082a0 2024-09-19T08:07:55.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3c41036e0 0x7fd3c4100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:55.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3ca7d2640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c41082a0 0x7fd3c4101410 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc005ab0 tx=0x7fd3bc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:55.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3c2ffd640 1 -- 172.21.15.98:0/3451584205 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc03f030 con 0x7fd3c41082a0 2024-09-19T08:07:55.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3c2ffd640 1 -- 172.21.15.98:0/3451584205 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3bc003ce0 con 0x7fd3c41082a0 2024-09-19T08:07:55.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3c2ffd640 1 -- 172.21.15.98:0/3451584205 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc035070 con 0x7fd3c41082a0 2024-09-19T08:07:55.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3cc235640 1 -- 172.21.15.98:0/3451584205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3c41a6ca0 con 0x7fd3c41082a0 2024-09-19T08:07:55.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.396+0000 7fd3cc235640 1 -- 172.21.15.98:0/3451584205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c41a7180 con 0x7fd3c41082a0 2024-09-19T08:07:55.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.398+0000 7fd3c2ffd640 1 -- 172.21.15.98:0/3451584205 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fd3bc042030 con 0x7fd3c41082a0 2024-09-19T08:07:55.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.399+0000 7fd3ce60e740 1 Processor -- start 2024-09-19T08:07:55.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.399+0000 7fd3ce60e740 1 -- start start 2024-09-19T08:07:55.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.399+0000 7fd3ce60e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c5b40 0x5581e61c4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3ce60e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c45d0 0x5581e61c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3ce60e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61cf370 0x5581e61d1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3ce60e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5581e612a2f0 con 0x5581e61cf370 2024-09-19T08:07:55.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3ce60e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5581e60d1cf0 con 0x5581e61c45d0 2024-09-19T08:07:55.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3ce60e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5581e61262b0 con 0x5581e61c5b40 2024-09-19T08:07:55.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3c2ffd640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd3ac0a33b0 0x7fd3ac0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3c97d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61cf370 0x5581e61d1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3c97d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61cf370 0x5581e61d1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53704/0 (socket says 172.21.15.98:53704) 2024-09-19T08:07:55.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3c97d0640 1 -- 172.21.15.98:0/3087528960 learned_addr learned my addr 172.21.15.98:0/3087528960 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:55.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3c9fd1640 1 --2- 172.21.15.98:0/3087528960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c45d0 0x5581e61c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3ca7d2640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd3ac0a33b0 0x7fd3ac0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/3087528960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c5b40 0x5581e61c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3c2ffd640 1 -- 172.21.15.98:0/3451584205 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fd3bc0f03a0 con 0x7fd3c41082a0 2024-09-19T08:07:55.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3c9fd1640 1 -- 172.21.15.98:0/3087528960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c5b40 msgr2=0x5581e61c4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:55.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3c9fd1640 1 --2- 172.21.15.98:0/3087528960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c5b40 0x5581e61c4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3c9fd1640 1 -- 172.21.15.98:0/3087528960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61cf370 msgr2=0x5581e61d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:55.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3c9fd1640 1 --2- 172.21.15.98:0/3087528960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61cf370 0x5581e61d1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3c9fd1640 1 -- 172.21.15.98:0/3087528960 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581e60b63a0 con 0x5581e61c45d0 2024-09-19T08:07:55.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/3087528960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c5b40 0x5581e61c4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:55.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3c9fd1640 1 --2- 172.21.15.98:0/3087528960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c45d0 0x5581e61c4a10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4009ee0 tx=0x7fd3b400c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=235717824834f920 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:55.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.400+0000 7fd3ca7d2640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd3ac0a33b0 0x7fd3ac0a5870 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc006010 tx=0x7fd3bc08e3d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:55.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.401+0000 7fd3abfff640 1 -- 172.21.15.98:0/3087528960 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b401a070 con 0x5581e61c45d0 2024-09-19T08:07:55.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.401+0000 7fd3abfff640 1 -- 172.21.15.98:0/3087528960 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3b40158f0 con 0x5581e61c45d0 2024-09-19T08:07:55.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.401+0000 7fd3abfff640 1 -- 172.21.15.98:0/3087528960 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b400a9d0 con 0x5581e61c45d0 2024-09-19T08:07:55.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.401+0000 7fd3ce60e740 1 -- 172.21.15.98:0/3087528960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c45d0 msgr2=0x5581e61c4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:55.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.401+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/3087528960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c45d0 0x5581e61c4a10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4009ee0 tx=0x7fd3b400c7f0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.401+0000 7fd3ce60e740 1 -- 172.21.15.98:0/3087528960 shutdown_connections 2024-09-19T08:07:55.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.401+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/3087528960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61cf370 0x5581e61d1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.401+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/3087528960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c45d0 0x5581e61c4a10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.401+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/3087528960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c5b40 0x5581e61c4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.401+0000 7fd3ce60e740 1 -- 172.21.15.98:0/3087528960 >> 172.21.15.98:0/3087528960 conn(0x5581e618d310 msgr2=0x5581e61c0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:55.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.401+0000 7fd3ce60e740 1 -- 172.21.15.98:0/3087528960 shutdown_connections 2024-09-19T08:07:55.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.401+0000 7fd3ce60e740 1 -- 172.21.15.98:0/3087528960 wait complete. 2024-09-19T08:07:55.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.402+0000 7fd3ce60e740 1 -- start start 2024-09-19T08:07:55.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.402+0000 7fd3ce60e740 1 Processor -- start 2024-09-19T08:07:55.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.402+0000 7fd3ce60e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c45d0 0x5581e61ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.402+0000 7fd3ce60e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c5b40 0x5581e61cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.402+0000 7fd3ce60e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61cf370 0x5581e61cc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.402+0000 7fd3ce60e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5581e61d2070 con 0x5581e61cf370 2024-09-19T08:07:55.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.402+0000 7fd3ce60e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5581e60e5060 con 0x5581e61c45d0 2024-09-19T08:07:55.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.402+0000 7fd3ce60e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5581e61d21f0 con 0x5581e61c5b40 2024-09-19T08:07:55.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.402+0000 7fd3ce60e740 10 client.0 _ll_register_callbacks cb 0x5581e5e388c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:55.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.402+0000 7fd3c97d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61cf370 0x5581e61cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.402+0000 7fd3c97d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61cf370 0x5581e61cc570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53718/0 (socket says 172.21.15.98:53718) 2024-09-19T08:07:55.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.402+0000 7fd3c97d0640 1 -- 172.21.15.98:0/1111560527 learned_addr learned my addr 172.21.15.98:0/1111560527 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:55.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.402+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/1111560527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c45d0 0x5581e61ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3c9fd1640 1 --2- 172.21.15.98:0/1111560527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c5b40 0x5581e61cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3c97d0640 1 -- 172.21.15.98:0/1111560527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c5b40 msgr2=0x5581e61cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:55.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/1111560527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c5b40 0x5581e61cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3c97d0640 1 -- 172.21.15.98:0/1111560527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c45d0 msgr2=0x5581e61ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:55.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/1111560527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c45d0 0x5581e61ced20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3c97d0640 1 -- 172.21.15.98:0/1111560527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581e5e9b3c0 con 0x5581e61cf370 2024-09-19T08:07:55.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/1111560527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61cf370 0x5581e61cc570 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8011460 tx=0x7fd3b8008350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4fa78e5fc476b50 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:55.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:55.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3a97fa640 1 -- 172.21.15.98:0/1111560527 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8013030 con 0x5581e61cf370 2024-09-19T08:07:55.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3a97fa640 1 -- 172.21.15.98:0/1111560527 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fd3b80131d0 con 0x5581e61cf370 2024-09-19T08:07:55.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1111560527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3b40158f0 con 0x5581e61cf370 2024-09-19T08:07:55.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3a97fa640 1 -- 172.21.15.98:0/1111560527 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd3b8005020 con 0x5581e61cf370 2024-09-19T08:07:55.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3ce60e740 10 client.16752 fetch_fsmap learned FSMap version 31 2024-09-19T08:07:55.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1111560527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5581e61ddb50 con 0x5581e61cf370 2024-09-19T08:07:55.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3a97fa640 1 -- 172.21.15.98:0/1111560527 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3b40158f0 con 0x5581e61cf370 2024-09-19T08:07:55.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.403+0000 7fd3a97fa640 1 -- 172.21.15.98:0/1111560527 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fd3b801e020 con 0x5581e61cf370 2024-09-19T08:07:55.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.404+0000 7fd3ce60e740 10 client.16752 fetch_fsmap finished waiting for FSMap version 31 2024-09-19T08:07:55.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.404+0000 7fd3ce60e740 10 client.16752 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:55.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.404+0000 7fd3ce60e740 20 client.16752 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:55.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.404+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/1111560527 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5581e61df010 0x5581e61e1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.404+0000 7fd3ce60e740 4 client.16752 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:55.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.404+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1111560527 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581e6137860 con 0x5581e61df010 2024-09-19T08:07:55.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.404+0000 7fd3c9fd1640 1 --2- 172.21.15.98:0/1111560527 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5581e61df010 0x5581e61e1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.404+0000 7fd3c9fd1640 1 --2- 172.21.15.98:0/1111560527 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5581e61df010 0x5581e61e1400 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81f7f9e5f690fdb2 server_cookie=d59e6cd2d17ed99a in_seq=0 out_seq=0 2024-09-19T08:07:55.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.404+0000 7fd3a97fa640 10 client.16752 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:55.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3a97fa640 1 -- 172.21.15.98:0/1111560527 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5581e6137860 con 0x5581e61df010 2024-09-19T08:07:55.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3a97fa640 10 client.16752 handle_command_reply: tid=0 2024-09-19T08:07:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3ce60e740 1 client.16752 shutdown 2024-09-19T08:07:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1111560527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61cf370 msgr2=0x5581e61cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/1111560527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61cf370 0x5581e61cc570 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8011460 tx=0x7fd3b8008350 comp rx=0 tx=0).stop 2024-09-19T08:07:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1111560527 shutdown_connections 2024-09-19T08:07:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/1111560527 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5581e61df010 0x5581e61e1400 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/1111560527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61cf370 0x5581e61cc570 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/1111560527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c5b40 0x5581e61cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/1111560527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c45d0 0x5581e61ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3a97fa640 0 client.16752 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1111560527 >> 172.21.15.98:0/1111560527 conn(0x5581e618d310 msgr2=0x5581e61c6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1111560527 shutdown_connections 2024-09-19T08:07:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1111560527 wait complete. 2024-09-19T08:07:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.405+0000 7fd3ce60e740 20 client.16752 trim_cache size 0 max 16384 2024-09-19T08:07:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.418+0000 7fd3ce60e740 1 Processor -- start 2024-09-19T08:07:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.418+0000 7fd3ce60e740 1 -- start start 2024-09-19T08:07:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.418+0000 7fd3ce60e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61c45d0 0x5581e61cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.418+0000 7fd3ce60e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c5b40 0x5581e61c7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.418+0000 7fd3ce60e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61cf370 0x5581e61c7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.418+0000 7fd3ce60e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c41af970 con 0x5581e61c45d0 2024-09-19T08:07:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.418+0000 7fd3ce60e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c41b14e0 con 0x5581e61c5b40 2024-09-19T08:07:55.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.418+0000 7fd3ce60e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c41b0040 con 0x5581e61cf370 2024-09-19T08:07:55.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.418+0000 7fd3c97d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61c45d0 0x5581e61cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.418+0000 7fd3c97d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61c45d0 0x5581e61cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53728/0 (socket says 172.21.15.98:53728) 2024-09-19T08:07:55.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.418+0000 7fd3c97d0640 1 -- 172.21.15.98:0/4227290204 learned_addr learned my addr 172.21.15.98:0/4227290204 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:55.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.419+0000 7fd3c9fd1640 1 --2- 172.21.15.98:0/4227290204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c5b40 0x5581e61c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.419+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/4227290204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61cf370 0x5581e61c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.419+0000 7fd3c97d0640 1 -- 172.21.15.98:0/4227290204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61cf370 msgr2=0x5581e61c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:55.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.419+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/4227290204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61cf370 0x5581e61c7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.419+0000 7fd3c97d0640 1 -- 172.21.15.98:0/4227290204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c5b40 msgr2=0x5581e61c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:55.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.419+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/4227290204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c5b40 0x5581e61c7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.419+0000 7fd3c97d0640 1 -- 172.21.15.98:0/4227290204 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581e61cf090 con 0x5581e61c45d0 2024-09-19T08:07:55.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.419+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/4227290204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61c45d0 0x5581e61cdf40 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8008980 tx=0x7fd3b8013b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c23651f7f1fc66 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:55.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.419+0000 7fd3ab7fe640 1 -- 172.21.15.98:0/4227290204 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b80110b0 con 0x5581e61c45d0 2024-09-19T08:07:55.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.419+0000 7fd3ab7fe640 1 -- 172.21.15.98:0/4227290204 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3b8011250 con 0x5581e61c45d0 2024-09-19T08:07:55.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.419+0000 7fd3ab7fe640 1 -- 172.21.15.98:0/4227290204 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b80038e0 con 0x5581e61c45d0 2024-09-19T08:07:55.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.420+0000 7fd3ce60e740 1 -- 172.21.15.98:0/4227290204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61c45d0 msgr2=0x5581e61cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:55.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.420+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/4227290204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61c45d0 0x5581e61cdf40 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8008980 tx=0x7fd3b8013b90 comp rx=0 tx=0).stop 2024-09-19T08:07:55.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.420+0000 7fd3ce60e740 1 -- 172.21.15.98:0/4227290204 shutdown_connections 2024-09-19T08:07:55.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.420+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/4227290204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61cf370 0x5581e61c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.420+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/4227290204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61c5b40 0x5581e61c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.420+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/4227290204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61c45d0 0x5581e61cdf40 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.420+0000 7fd3ce60e740 1 -- 172.21.15.98:0/4227290204 >> 172.21.15.98:0/4227290204 conn(0x5581e618d310 msgr2=0x5581e61cd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:55.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.420+0000 7fd3ce60e740 1 -- 172.21.15.98:0/4227290204 shutdown_connections 2024-09-19T08:07:55.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.420+0000 7fd3ce60e740 1 -- 172.21.15.98:0/4227290204 wait complete. 2024-09-19T08:07:55.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.420+0000 7fd3ce60e740 1 -- start start 2024-09-19T08:07:55.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.420+0000 7fd3ce60e740 1 Processor -- start 2024-09-19T08:07:55.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3ce60e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c45d0 0x5581e61d7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3ce60e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61c5b40 0x5581e61d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3ce60e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61cf370 0x5581e61d3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3ce60e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c41b0e10 con 0x5581e61c5b40 2024-09-19T08:07:55.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3ce60e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c41b0740 con 0x5581e61cf370 2024-09-19T08:07:55.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3ce60e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c41ad950 con 0x5581e61c45d0 2024-09-19T08:07:55.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3ce60e740 10 client.0 _ll_register_callbacks cb 0x7fd3b8008de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:55.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3c9fd1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61c5b40 0x5581e61d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3c9fd1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61c5b40 0x5581e61d2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53738/0 (socket says 172.21.15.98:53738) 2024-09-19T08:07:55.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3c9fd1640 1 -- 172.21.15.98:0/1251782070 learned_addr learned my addr 172.21.15.98:0/1251782070 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:55.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/1251782070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61cf370 0x5581e61d3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3c9fd1640 1 -- 172.21.15.98:0/1251782070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c45d0 msgr2=0x5581e61d7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:55.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/1251782070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c45d0 0x5581e61d7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3c9fd1640 1 --2- 172.21.15.98:0/1251782070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c45d0 0x5581e61d7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3c9fd1640 1 -- 172.21.15.98:0/1251782070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61cf370 msgr2=0x5581e61d3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:55.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3c9fd1640 1 --2- 172.21.15.98:0/1251782070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61cf370 0x5581e61d3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3c9fd1640 1 -- 172.21.15.98:0/1251782070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581e61d3a40 con 0x5581e61c5b40 2024-09-19T08:07:55.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/1251782070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c45d0 0x5581e61d7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:55.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3c9fd1640 1 --2- 172.21.15.98:0/1251782070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61c5b40 0x5581e61d2e10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4009e80 tx=0x7fd3b400a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613cb7a2ad86e875 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:55.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:55.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3a97fa640 1 -- 172.21.15.98:0/1251782070 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b401a040 con 0x5581e61c5b40 2024-09-19T08:07:55.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3a97fa640 1 -- 172.21.15.98:0/1251782070 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fd3b400a6c0 con 0x5581e61c5b40 2024-09-19T08:07:55.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1251782070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3b8011250 con 0x5581e61c5b40 2024-09-19T08:07:55.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3a97fa640 1 -- 172.21.15.98:0/1251782070 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd3b40098e0 con 0x5581e61c5b40 2024-09-19T08:07:55.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3ce60e740 10 client.16758 fetch_fsmap learned FSMap version 31 2024-09-19T08:07:55.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.421+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1251782070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5581e6249a70 con 0x5581e61c5b40 2024-09-19T08:07:55.457 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:07:55.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.422+0000 7fd3a97fa640 1 -- 172.21.15.98:0/1251782070 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3b8011250 con 0x5581e61c5b40 2024-09-19T08:07:55.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.422+0000 7fd3a97fa640 1 -- 172.21.15.98:0/1251782070 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fd3b4033af0 con 0x5581e61c5b40 2024-09-19T08:07:55.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.422+0000 7fd3ce60e740 10 client.16758 fetch_fsmap finished waiting for FSMap version 31 2024-09-19T08:07:55.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.422+0000 7fd3ce60e740 10 client.16758 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:55.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.422+0000 7fd3ce60e740 20 client.16758 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:55.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.422+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/1251782070 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5581e6249e60 0x5581e624c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:55.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.422+0000 7fd3ce60e740 4 client.16758 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:07:55.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.422+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1251782070 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5581e6131d70 con 0x5581e6249e60 2024-09-19T08:07:55.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.422+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/1251782070 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5581e6249e60 0x5581e624c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:55.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.422+0000 7fd3a97fa640 10 client.16758 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:55.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.422+0000 7fd3c97d0640 1 --2- 172.21.15.98:0/1251782070 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5581e6249e60 0x5581e624c280 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6cebacb215aa862 server_cookie=743cbdaeaea0df6a in_seq=0 out_seq=0 2024-09-19T08:07:55.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3a97fa640 1 -- 172.21.15.98:0/1251782070 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5581e6131d70 con 0x5581e6249e60 2024-09-19T08:07:55.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3a97fa640 10 client.16758 handle_command_reply: tid=0 2024-09-19T08:07:55.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3ce60e740 1 client.16758 shutdown 2024-09-19T08:07:55.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1251782070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61c5b40 msgr2=0x5581e61d2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:55.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/1251782070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61c5b40 0x5581e61d2e10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4009e80 tx=0x7fd3b400a400 comp rx=0 tx=0).stop 2024-09-19T08:07:55.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1251782070 shutdown_connections 2024-09-19T08:07:55.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/1251782070 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5581e6249e60 0x5581e624c280 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/1251782070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581e61cf370 0x5581e61d3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/1251782070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581e61c5b40 0x5581e61d2e10 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3ce60e740 1 --2- 172.21.15.98:0/1251782070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5581e61c45d0 0x5581e61d7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3a97fa640 0 client.16758 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:55.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1251782070 >> 172.21.15.98:0/1251782070 conn(0x5581e618d310 msgr2=0x5581e61ce050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:55.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1251782070 shutdown_connections 2024-09-19T08:07:55.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3ce60e740 1 -- 172.21.15.98:0/1251782070 wait complete. 2024-09-19T08:07:55.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3ce60e740 20 client.16758 trim_cache size 0 max 16384 2024-09-19T08:07:55.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.423+0000 7fd3cc235640 1 -- 172.21.15.98:0/3451584205 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd3ac0a33b0 msgr2=0x7fd3ac0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:55.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.424+0000 7fd3cc235640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd3ac0a33b0 0x7fd3ac0a5870 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc006010 tx=0x7fd3bc08e3d0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.424+0000 7fd3cc235640 1 -- 172.21.15.98:0/3451584205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c41082a0 msgr2=0x7fd3c4101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:55.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.424+0000 7fd3cc235640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c41082a0 0x7fd3c4101410 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc005ab0 tx=0x7fd3bc0042f0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.424+0000 7fd3cc235640 1 -- 172.21.15.98:0/3451584205 shutdown_connections 2024-09-19T08:07:55.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.424+0000 7fd3cc235640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd3ac0a33b0 0x7fd3ac0a5870 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.424+0000 7fd3cc235640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c41082a0 0x7fd3c4101410 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.424+0000 7fd3cc235640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3c41036e0 0x7fd3c4100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.424+0000 7fd3cc235640 1 --2- 172.21.15.98:0/3451584205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3c4102d30 0x7fd3c4100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:55.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.424+0000 7fd3cc235640 1 -- 172.21.15.98:0/3451584205 >> 172.21.15.98:0/3451584205 conn(0x7fd3c40fc7d0 msgr2=0x7fd3c4106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:55.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.424+0000 7fd3cc235640 1 -- 172.21.15.98:0/3451584205 shutdown_connections 2024-09-19T08:07:55.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:55.424+0000 7fd3cc235640 1 -- 172.21.15.98:0/3451584205 wait complete. 2024-09-19T08:07:56.128 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:55 smithi098 ceph-mon[31333]: pgmap v150: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:56.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:55 smithi139 ceph-mon[34874]: pgmap v150: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:56.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:55 smithi162 ceph-mon[35260]: pgmap v150: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:07:57.155 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:07:57.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:07:57.404 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:57.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.916+0000 7f4efc227640 1 -- 172.21.15.98:0/4182702624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4101de0 msgr2=0x7f4ef41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:57.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.916+0000 7f4efc227640 1 --2- 172.21.15.98:0/4182702624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4101de0 0x7f4ef41021c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009900 tx=0x7f4ef002a410 comp rx=0 tx=0).stop 2024-09-19T08:07:57.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.916+0000 7f4efc227640 1 -- 172.21.15.98:0/4182702624 shutdown_connections 2024-09-19T08:07:57.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.916+0000 7f4efc227640 1 --2- 172.21.15.98:0/4182702624 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ef4106f10 0x7f4ef410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.916+0000 7f4efc227640 1 --2- 172.21.15.98:0/4182702624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ef4102790 0x7f4ef41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.916+0000 7f4efc227640 1 --2- 172.21.15.98:0/4182702624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4101de0 0x7f4ef41021c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.916+0000 7f4efc227640 1 -- 172.21.15.98:0/4182702624 >> 172.21.15.98:0/4182702624 conn(0x7f4ef40fd8c0 msgr2=0x7f4ef40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:57.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.916+0000 7f4efc227640 1 -- 172.21.15.98:0/4182702624 shutdown_connections 2024-09-19T08:07:57.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.916+0000 7f4efc227640 1 -- 172.21.15.98:0/4182702624 wait complete. 2024-09-19T08:07:57.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4efc227640 1 Processor -- start 2024-09-19T08:07:57.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4efc227640 1 -- start start 2024-09-19T08:07:57.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4efc227640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ef4101de0 0x7f4ef4198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4efc227640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4102790 0x7f4ef41988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4efc227640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ef4106f10 0x7f4ef419cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4efc227640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef4111f30 con 0x7f4ef4102790 2024-09-19T08:07:57.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4efc227640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef4111db0 con 0x7f4ef4101de0 2024-09-19T08:07:57.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4efc227640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41120b0 con 0x7f4ef4106f10 2024-09-19T08:07:57.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4ef97c2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4102790 0x7f4ef41988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4ef97c2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4102790 0x7f4ef41988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43078/0 (socket says 172.21.15.98:43078) 2024-09-19T08:07:57.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4ef97c2640 1 -- 172.21.15.98:0/3262996374 learned_addr learned my addr 172.21.15.98:0/3262996374 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:57.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4efa7c4640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ef4106f10 0x7f4ef419cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4ef9fc3640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ef4101de0 0x7f4ef4198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4ef97c2640 1 -- 172.21.15.98:0/3262996374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ef4106f10 msgr2=0x7f4ef419cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:57.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ef4106f10 0x7f4ef419cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4ef97c2640 1 -- 172.21.15.98:0/3262996374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ef4101de0 msgr2=0x7f4ef4198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:57.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ef4101de0 0x7f4ef4198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4ef97c2640 1 -- 172.21.15.98:0/3262996374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef419d340 con 0x7f4ef4102790 2024-09-19T08:07:57.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4ef9fc3640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ef4101de0 0x7f4ef4198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:57.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.917+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4102790 0x7f4ef41988d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800c920 tx=0x7f4ee800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:57.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.918+0000 7f4ee6ffd640 1 -- 172.21.15.98:0/3262996374 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8007c70 con 0x7f4ef4102790 2024-09-19T08:07:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.918+0000 7f4efc227640 1 -- 172.21.15.98:0/3262996374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef419d630 con 0x7f4ef4102790 2024-09-19T08:07:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.918+0000 7f4ee6ffd640 1 -- 172.21.15.98:0/3262996374 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee8007e10 con 0x7f4ef4102790 2024-09-19T08:07:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.918+0000 7f4ee6ffd640 1 -- 172.21.15.98:0/3262996374 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8018890 con 0x7f4ef4102790 2024-09-19T08:07:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.918+0000 7f4efc227640 1 -- 172.21.15.98:0/3262996374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef41a4f10 con 0x7f4ef4102790 2024-09-19T08:07:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.920+0000 7f4ee6ffd640 1 -- 172.21.15.98:0/3262996374 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f4ee8018a50 con 0x7f4ef4102790 2024-09-19T08:07:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.920+0000 7f4efe600740 1 Processor -- start 2024-09-19T08:07:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.920+0000 7f4efe600740 1 -- start start 2024-09-19T08:07:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4ee6ffd640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4ee00a33b0 0x7f4ee00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4efe600740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7ba40 0x556773b7be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4efe600740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7c360 0x556773b8ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4efe600740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 0x556773b8d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4efe600740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556773ae1280 con 0x556773b7c360 2024-09-19T08:07:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4efe600740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556773a88b10 con 0x556773b7ba40 2024-09-19T08:07:57.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4efe600740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556773add240 con 0x556773b8b150 2024-09-19T08:07:57.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4efa7c4640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4ee00a33b0 0x7f4ee00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4ef97c2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7c360 0x556773b8ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4ef97c2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7c360 0x556773b8ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43092/0 (socket says 172.21.15.98:43092) 2024-09-19T08:07:57.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4ef97c2640 1 -- 172.21.15.98:0/2319011260 learned_addr learned my addr 172.21.15.98:0/2319011260 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:57.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4ee6ffd640 1 -- 172.21.15.98:0/3262996374 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f4ee80cfe60 con 0x7f4ef4102790 2024-09-19T08:07:57.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4efa7c4640 1 --2- 172.21.15.98:0/2319011260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 0x556773b8d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4ef9fc3640 1 --2- 172.21.15.98:0/2319011260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7ba40 0x556773b7be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4ef97c2640 1 -- 172.21.15.98:0/2319011260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 msgr2=0x556773b8d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:57.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/2319011260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 0x556773b8d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4ef97c2640 1 -- 172.21.15.98:0/2319011260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7ba40 msgr2=0x556773b7be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:57.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/2319011260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7ba40 0x556773b7be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4ef97c2640 1 -- 172.21.15.98:0/2319011260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556773a951f0 con 0x556773b7c360 2024-09-19T08:07:57.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/2319011260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7c360 0x556773b8ac10 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8000c00 tx=0x7f4ee80148a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc85c2bcf153d033 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:57.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.921+0000 7f4efa7c4640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4ee00a33b0 0x7f4ee00a5870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x556773b79470 tx=0x7f4eec009a60 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:57.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.922+0000 7f4edffff640 1 -- 172.21.15.98:0/2319011260 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee80c6bb0 con 0x556773b7c360 2024-09-19T08:07:57.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.922+0000 7f4edffff640 1 -- 172.21.15.98:0/2319011260 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee8012110 con 0x556773b7c360 2024-09-19T08:07:57.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.922+0000 7f4edffff640 1 -- 172.21.15.98:0/2319011260 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee809a500 con 0x556773b7c360 2024-09-19T08:07:57.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.922+0000 7f4efe600740 1 -- 172.21.15.98:0/2319011260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7c360 msgr2=0x556773b8ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:57.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.922+0000 7f4efe600740 1 --2- 172.21.15.98:0/2319011260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7c360 0x556773b8ac10 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8000c00 tx=0x7f4ee80148a0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.923+0000 7f4efe600740 1 -- 172.21.15.98:0/2319011260 shutdown_connections 2024-09-19T08:07:57.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.923+0000 7f4efe600740 1 --2- 172.21.15.98:0/2319011260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 0x556773b8d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.923+0000 7f4efe600740 1 --2- 172.21.15.98:0/2319011260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7c360 0x556773b8ac10 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.923+0000 7f4efe600740 1 --2- 172.21.15.98:0/2319011260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7ba40 0x556773b7be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.923+0000 7f4efe600740 1 -- 172.21.15.98:0/2319011260 >> 172.21.15.98:0/2319011260 conn(0x556773b780f0 msgr2=0x556773b784d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:57.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.923+0000 7f4efe600740 1 -- 172.21.15.98:0/2319011260 shutdown_connections 2024-09-19T08:07:57.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.923+0000 7f4efe600740 1 -- 172.21.15.98:0/2319011260 wait complete. 2024-09-19T08:07:57.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.923+0000 7f4efe600740 1 -- start start 2024-09-19T08:07:57.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.923+0000 7f4efe600740 1 Processor -- start 2024-09-19T08:07:57.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4efe600740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7ba40 0x556773b880d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4efe600740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b7c360 0x556773b88610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4efe600740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b8b150 0x556773b88b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4efe600740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556773b8dd20 con 0x556773b7ba40 2024-09-19T08:07:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4efe600740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556773a9bf10 con 0x556773b8b150 2024-09-19T08:07:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4efe600740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556773b8dea0 con 0x556773b7c360 2024-09-19T08:07:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4efe600740 10 client.0 _ll_register_callbacks cb 0x556773a64a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4ef9fc3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7ba40 0x556773b880d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4ef9fc3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7ba40 0x556773b880d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43104/0 (socket says 172.21.15.98:43104) 2024-09-19T08:07:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4ef9fc3640 1 -- 172.21.15.98:0/470183186 learned_addr learned my addr 172.21.15.98:0/470183186 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/470183186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b7c360 0x556773b88610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4efa7c4640 1 --2- 172.21.15.98:0/470183186 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b8b150 0x556773b88b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4ef9fc3640 1 -- 172.21.15.98:0/470183186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b7c360 msgr2=0x556773b88610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4ef9fc3640 1 --2- 172.21.15.98:0/470183186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b7c360 0x556773b88610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4ef9fc3640 1 -- 172.21.15.98:0/470183186 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b8b150 msgr2=0x556773b88b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:57.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4ef9fc3640 1 --2- 172.21.15.98:0/470183186 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b8b150 0x556773b88b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4ef9fc3640 1 -- 172.21.15.98:0/470183186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567738526a0 con 0x556773b7ba40 2024-09-19T08:07:57.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4ef9fc3640 1 --2- 172.21.15.98:0/470183186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7ba40 0x556773b880d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0033000 tx=0x7f4ef0031a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac11816e324d717c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:57.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.924+0000 7f4edd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:57.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.927+0000 7f4edd7fa640 1 -- 172.21.15.98:0/470183186 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0035040 con 0x556773b7ba40 2024-09-19T08:07:57.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.927+0000 7f4edd7fa640 1 -- 172.21.15.98:0/470183186 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f4ef0004340 con 0x556773b7ba40 2024-09-19T08:07:57.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.927+0000 7f4efe600740 1 -- 172.21.15.98:0/470183186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556773aee7f0 con 0x556773b7ba40 2024-09-19T08:07:57.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.927+0000 7f4edd7fa640 1 -- 172.21.15.98:0/470183186 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4ef0005030 con 0x556773b7ba40 2024-09-19T08:07:57.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.927+0000 7f4efe600740 10 client.16770 fetch_fsmap learned FSMap version 31 2024-09-19T08:07:57.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.927+0000 7f4efe600740 1 -- 172.21.15.98:0/470183186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556773b99a00 con 0x556773b7ba40 2024-09-19T08:07:57.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.927+0000 7f4edd7fa640 1 -- 172.21.15.98:0/470183186 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556773aee7f0 con 0x556773b7ba40 2024-09-19T08:07:57.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.928+0000 7f4edd7fa640 1 -- 172.21.15.98:0/470183186 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f4ef00035f0 con 0x556773b7ba40 2024-09-19T08:07:57.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.928+0000 7f4efe600740 10 client.16770 fetch_fsmap finished waiting for FSMap version 31 2024-09-19T08:07:57.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.928+0000 7f4efe600740 10 client.16770 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:57.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.928+0000 7f4efe600740 20 client.16770 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:57.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.928+0000 7f4efe600740 1 --2- 172.21.15.98:0/470183186 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x556773b9ae10 0x556773b9d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.928+0000 7f4efe600740 4 client.16770 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:57.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.928+0000 7f4efe600740 1 -- 172.21.15.98:0/470183186 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556773ae8d00 con 0x556773b9ae10 2024-09-19T08:07:57.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.928+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/470183186 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x556773b9ae10 0x556773b9d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.928+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/470183186 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x556773b9ae10 0x556773b9d220 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=762084d1495f5a79 server_cookie=a84b11170f746f77 in_seq=0 out_seq=0 2024-09-19T08:07:57.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.928+0000 7f4edd7fa640 10 client.16770 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:57.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.929+0000 7f4edd7fa640 1 -- 172.21.15.98:0/470183186 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556773ae8d00 con 0x556773b9ae10 2024-09-19T08:07:57.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.929+0000 7f4edd7fa640 10 client.16770 handle_command_reply: tid=0 2024-09-19T08:07:57.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.929+0000 7f4efe600740 1 client.16770 shutdown 2024-09-19T08:07:57.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.929+0000 7f4efe600740 1 -- 172.21.15.98:0/470183186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7ba40 msgr2=0x556773b880d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:57.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.929+0000 7f4efe600740 1 --2- 172.21.15.98:0/470183186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7ba40 0x556773b880d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0033000 tx=0x7f4ef0031a60 comp rx=0 tx=0).stop 2024-09-19T08:07:57.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.929+0000 7f4efe600740 1 -- 172.21.15.98:0/470183186 shutdown_connections 2024-09-19T08:07:57.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.929+0000 7f4efe600740 1 --2- 172.21.15.98:0/470183186 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x556773b9ae10 0x556773b9d220 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.929+0000 7f4efe600740 1 --2- 172.21.15.98:0/470183186 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b8b150 0x556773b88b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.929+0000 7f4efe600740 1 --2- 172.21.15.98:0/470183186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b7c360 0x556773b88610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.929+0000 7f4edd7fa640 0 client.16770 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:57.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.929+0000 7f4efe600740 1 --2- 172.21.15.98:0/470183186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7ba40 0x556773b880d0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.929+0000 7f4efe600740 1 -- 172.21.15.98:0/470183186 >> 172.21.15.98:0/470183186 conn(0x556773b780f0 msgr2=0x556773b7de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:57.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.929+0000 7f4efe600740 1 -- 172.21.15.98:0/470183186 shutdown_connections 2024-09-19T08:07:57.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.929+0000 7f4efe600740 1 -- 172.21.15.98:0/470183186 wait complete. 2024-09-19T08:07:57.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.930+0000 7f4efe600740 20 client.16770 trim_cache size 0 max 16384 2024-09-19T08:07:57.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.946+0000 7f4efe600740 1 Processor -- start 2024-09-19T08:07:57.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.946+0000 7f4efe600740 1 -- start start 2024-09-19T08:07:57.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4efe600740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7ba40 0x556773b887a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4efe600740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7c360 0x556773b88ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4efe600740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 0x556773ba2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4efe600740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41ab620 con 0x556773b7c360 2024-09-19T08:07:57.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4efe600740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41ad190 con 0x556773b7ba40 2024-09-19T08:07:57.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4efe600740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41abcf0 con 0x556773b8b150 2024-09-19T08:07:57.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4ef97c2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7c360 0x556773b88ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4ef97c2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7c360 0x556773b88ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43116/0 (socket says 172.21.15.98:43116) 2024-09-19T08:07:57.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4ef97c2640 1 -- 172.21.15.98:0/618404395 learned_addr learned my addr 172.21.15.98:0/618404395 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:57.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4ef9fc3640 1 --2- 172.21.15.98:0/618404395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7ba40 0x556773b887a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4ef97c2640 1 -- 172.21.15.98:0/618404395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 msgr2=0x556773ba2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:57.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4efa7c4640 1 --2- 172.21.15.98:0/618404395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 0x556773ba2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/618404395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 0x556773ba2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4ef97c2640 1 -- 172.21.15.98:0/618404395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7ba40 msgr2=0x556773b887a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/618404395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7ba40 0x556773b887a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4ef97c2640 1 -- 172.21.15.98:0/618404395 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556773b77b90 con 0x556773b7c360 2024-09-19T08:07:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4efa7c4640 1 --2- 172.21.15.98:0/618404395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 0x556773ba2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/618404395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7c360 0x556773b88ce0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4ee809be70 tx=0x7f4ee80cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c476fbb5ae1038 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4edf7fe640 1 -- 172.21.15.98:0/618404395 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee80a0520 con 0x556773b7c360 2024-09-19T08:07:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4edf7fe640 1 -- 172.21.15.98:0/618404395 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee809a470 con 0x556773b7c360 2024-09-19T08:07:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.947+0000 7f4edf7fe640 1 -- 172.21.15.98:0/618404395 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee80a0160 con 0x556773b7c360 2024-09-19T08:07:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.948+0000 7f4efe600740 1 -- 172.21.15.98:0/618404395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7c360 msgr2=0x556773b88ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.948+0000 7f4efe600740 1 --2- 172.21.15.98:0/618404395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7c360 0x556773b88ce0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4ee809be70 tx=0x7f4ee80cf040 comp rx=0 tx=0).stop 2024-09-19T08:07:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.948+0000 7f4efe600740 1 -- 172.21.15.98:0/618404395 shutdown_connections 2024-09-19T08:07:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.948+0000 7f4efe600740 1 --2- 172.21.15.98:0/618404395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 0x556773ba2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.948+0000 7f4efe600740 1 --2- 172.21.15.98:0/618404395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7c360 0x556773b88ce0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.948+0000 7f4efe600740 1 --2- 172.21.15.98:0/618404395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7ba40 0x556773b887a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.948+0000 7f4efe600740 1 -- 172.21.15.98:0/618404395 >> 172.21.15.98:0/618404395 conn(0x556773b780f0 msgr2=0x556773b88090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.948+0000 7f4efe600740 1 -- 172.21.15.98:0/618404395 shutdown_connections 2024-09-19T08:07:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.948+0000 7f4efe600740 1 -- 172.21.15.98:0/618404395 wait complete. 2024-09-19T08:07:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.948+0000 7f4efe600740 1 -- start start 2024-09-19T08:07:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.948+0000 7f4efe600740 1 Processor -- start 2024-09-19T08:07:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4efe600740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7ba40 0x556773b8f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4efe600740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7c360 0x556773b8fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4efe600740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 0x556773bfea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4efe600740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41acac0 con 0x556773b7ba40 2024-09-19T08:07:57.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4efe600740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41ac3f0 con 0x556773b7c360 2024-09-19T08:07:57.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4efe600740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41a9600 con 0x556773b8b150 2024-09-19T08:07:57.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4efe600740 10 client.0 _ll_register_callbacks cb 0x7f4ef0005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:57.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4ef9fc3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7ba40 0x556773b8f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4ef9fc3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7ba40 0x556773b8f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43118/0 (socket says 172.21.15.98:43118) 2024-09-19T08:07:57.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4ef9fc3640 1 -- 172.21.15.98:0/2856908520 learned_addr learned my addr 172.21.15.98:0/2856908520 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:57.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4efa7c4640 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 0x556773bfea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7c360 0x556773b8fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4ef9fc3640 1 -- 172.21.15.98:0/2856908520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 msgr2=0x556773bfea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:57.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4ef9fc3640 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 0x556773bfea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4ef9fc3640 1 -- 172.21.15.98:0/2856908520 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7c360 msgr2=0x556773b8fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:57.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4ef9fc3640 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7c360 0x556773b8fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4ef9fc3640 1 -- 172.21.15.98:0/2856908520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556773bff0d0 con 0x556773b7ba40 2024-09-19T08:07:57.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4efa7c4640 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 0x556773bfea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:57.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.949+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7c360 0x556773b8fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:07:57.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4ef9fc3640 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7ba40 0x556773b8f710 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4ef002a930 tx=0x7f4ef00315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c41404a4ec47cbc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:57.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4edd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:57.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4edd7fa640 1 -- 172.21.15.98:0/2856908520 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0053030 con 0x556773b7ba40 2024-09-19T08:07:57.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4efe600740 1 -- 172.21.15.98:0/2856908520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ee809a470 con 0x556773b7ba40 2024-09-19T08:07:57.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4edd7fa640 1 -- 172.21.15.98:0/2856908520 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f4ef0003b90 con 0x556773b7ba40 2024-09-19T08:07:57.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4edd7fa640 1 -- 172.21.15.98:0/2856908520 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4ef005d050 con 0x556773b7ba40 2024-09-19T08:07:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4efe600740 10 client.16776 fetch_fsmap learned FSMap version 31 2024-09-19T08:07:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4efe600740 1 -- 172.21.15.98:0/2856908520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556773c063f0 con 0x556773b7ba40 2024-09-19T08:07:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4edd7fa640 1 -- 172.21.15.98:0/2856908520 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee809a470 con 0x556773b7ba40 2024-09-19T08:07:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4edd7fa640 1 -- 172.21.15.98:0/2856908520 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f4ef0004360 con 0x556773b7ba40 2024-09-19T08:07:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4efe600740 10 client.16776 fetch_fsmap finished waiting for FSMap version 31 2024-09-19T08:07:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4efe600740 10 client.16776 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4efe600740 20 client.16776 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4efe600740 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x556773c06b60 0x556773c08f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4efe600740 4 client.16776 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:07:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.950+0000 7f4efe600740 1 -- 172.21.15.98:0/2856908520 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556773a65110 con 0x556773c06b60 2024-09-19T08:07:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.951+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x556773c06b60 0x556773c08f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.951+0000 7f4edd7fa640 10 client.16776 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.951+0000 7f4ef97c2640 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x556773c06b60 0x556773c08f50 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9db20b1e312a3f9e server_cookie=30966f4b86708376 in_seq=0 out_seq=0 2024-09-19T08:07:57.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4edd7fa640 1 -- 172.21.15.98:0/2856908520 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556773a65110 con 0x556773c06b60 2024-09-19T08:07:57.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4edd7fa640 10 client.16776 handle_command_reply: tid=0 2024-09-19T08:07:57.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4efe600740 1 client.16776 shutdown 2024-09-19T08:07:57.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4efe600740 1 -- 172.21.15.98:0/2856908520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7ba40 msgr2=0x556773b8f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:57.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4efe600740 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7ba40 0x556773b8f710 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4ef002a930 tx=0x7f4ef00315b0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4efe600740 1 -- 172.21.15.98:0/2856908520 shutdown_connections 2024-09-19T08:07:57.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4efe600740 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x556773c06b60 0x556773c08f50 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4efe600740 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556773b8b150 0x556773bfea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4efe600740 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556773b7c360 0x556773b8fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4efe600740 1 --2- 172.21.15.98:0/2856908520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556773b7ba40 0x556773b8f710 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4efe600740 1 -- 172.21.15.98:0/2856908520 >> 172.21.15.98:0/2856908520 conn(0x556773b780f0 msgr2=0x556773b967d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:57.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4edd7fa640 0 client.16776 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:57.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4efe600740 1 -- 172.21.15.98:0/2856908520 shutdown_connections 2024-09-19T08:07:57.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4efe600740 1 -- 172.21.15.98:0/2856908520 wait complete. 2024-09-19T08:07:57.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.952+0000 7f4efe600740 20 client.16776 trim_cache size 0 max 16384 2024-09-19T08:07:57.970 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:07:57.971 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:07:57.971 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "f503195e-7f23-4a19-8b3a-9a33e73db8fe", 2024-09-19T08:07:57.971 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:07:57.971 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:07:57.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.953+0000 7f4efc227640 1 -- 172.21.15.98:0/3262996374 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4ee00a33b0 msgr2=0x7f4ee00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:57.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.953+0000 7f4efc227640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4ee00a33b0 0x7f4ee00a5870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x556773b79470 tx=0x7f4eec009a60 comp rx=0 tx=0).stop 2024-09-19T08:07:57.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.953+0000 7f4efc227640 1 -- 172.21.15.98:0/3262996374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4102790 msgr2=0x7f4ef41988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:57.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.953+0000 7f4efc227640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4102790 0x7f4ef41988d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800c920 tx=0x7f4ee800cde0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.953+0000 7f4efc227640 1 -- 172.21.15.98:0/3262996374 shutdown_connections 2024-09-19T08:07:57.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.953+0000 7f4efc227640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4ee00a33b0 0x7f4ee00a5870 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.953+0000 7f4efc227640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ef4106f10 0x7f4ef419cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.953+0000 7f4efc227640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4102790 0x7f4ef41988d0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.953+0000 7f4efc227640 1 --2- 172.21.15.98:0/3262996374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ef4101de0 0x7f4ef4198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:57.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.953+0000 7f4efc227640 1 -- 172.21.15.98:0/3262996374 >> 172.21.15.98:0/3262996374 conn(0x7f4ef40fd8c0 msgr2=0x7f4ef40fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:57.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.953+0000 7f4efc227640 1 -- 172.21.15.98:0/3262996374 shutdown_connections 2024-09-19T08:07:57.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:57.953+0000 7f4efc227640 1 -- 172.21.15.98:0/3262996374 wait complete. 2024-09-19T08:07:58.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:57 smithi139 ceph-mon[34874]: pgmap v151: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:07:58.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:57 smithi098 ceph-mon[31333]: pgmap v151: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:07:58.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:57 smithi162 ceph-mon[35260]: pgmap v151: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:07:58.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:07:59.017 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:07:59.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:58 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:07:59.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:58 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:59.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:58 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:07:59.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:58 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:59.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:58 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:07:59.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:58 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:59.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:58 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:07:59.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:58 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:59.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:58 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:07:59.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:58 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:59.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:58 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:07:59.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:58 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:07:59.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.501+0000 7efe6cf74640 1 -- 172.21.15.98:0/1958024689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe68102570 msgr2=0x7efe681029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:59.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.501+0000 7efe6cf74640 1 --2- 172.21.15.98:0/1958024689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe68102570 0x7efe681029b0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7efe50009900 tx=0x7efe5002a480 comp rx=0 tx=0).stop 2024-09-19T08:07:59.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.501+0000 7efe6cf74640 1 -- 172.21.15.98:0/1958024689 shutdown_connections 2024-09-19T08:07:59.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.501+0000 7efe6cf74640 1 --2- 172.21.15.98:0/1958024689 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe68102ef0 0x7efe6810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.501+0000 7efe6cf74640 1 --2- 172.21.15.98:0/1958024689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe68102570 0x7efe681029b0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.501+0000 7efe6cf74640 1 --2- 172.21.15.98:0/1958024689 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe68100e20 0x7efe68101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.501+0000 7efe6cf74640 1 -- 172.21.15.98:0/1958024689 >> 172.21.15.98:0/1958024689 conn(0x7efe680fc7d0 msgr2=0x7efe680febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:59.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.501+0000 7efe6cf74640 1 -- 172.21.15.98:0/1958024689 shutdown_connections 2024-09-19T08:07:59.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.501+0000 7efe6cf74640 1 -- 172.21.15.98:0/1958024689 wait complete. 2024-09-19T08:07:59.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.502+0000 7efe6cf74640 1 Processor -- start 2024-09-19T08:07:59.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.502+0000 7efe6cf74640 1 -- start start 2024-09-19T08:07:59.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.502+0000 7efe6cf74640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe68100e20 0x7efe680fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.502+0000 7efe6cf74640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe68102570 0x7efe680fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.502+0000 7efe6cf74640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe68102ef0 0x7efe680fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.502+0000 7efe6cf74640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe68110f80 con 0x7efe68102570 2024-09-19T08:07:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.502+0000 7efe6cf74640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efe68110e00 con 0x7efe68100e20 2024-09-19T08:07:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.502+0000 7efe6cf74640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efe68111100 con 0x7efe68102ef0 2024-09-19T08:07:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.502+0000 7efe5ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe68102570 0x7efe680fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.502+0000 7efe5ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe68102570 0x7efe680fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43130/0 (socket says 172.21.15.98:43130) 2024-09-19T08:07:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.502+0000 7efe5ffff640 1 -- 172.21.15.98:0/1044673686 learned_addr learned my addr 172.21.15.98:0/1044673686 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.502+0000 7efe6659c640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe68100e20 0x7efe680fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.503+0000 7efe66d9d640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe68102ef0 0x7efe680fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.503+0000 7efe5ffff640 1 -- 172.21.15.98:0/1044673686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe68102ef0 msgr2=0x7efe680fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.503+0000 7efe5ffff640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe68102ef0 0x7efe680fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.503+0000 7efe5ffff640 1 -- 172.21.15.98:0/1044673686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe68100e20 msgr2=0x7efe680fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:59.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.503+0000 7efe5ffff640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe68100e20 0x7efe680fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.503+0000 7efe5ffff640 1 -- 172.21.15.98:0/1044673686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe681a69c0 con 0x7efe68102570 2024-09-19T08:07:59.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.503+0000 7efe66d9d640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe68102ef0 0x7efe680fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:59.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.503+0000 7efe5ffff640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe68102570 0x7efe680fd1c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7efe50006010 tx=0x7efe50003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:59.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.503+0000 7efe5f7fe640 1 -- 172.21.15.98:0/1044673686 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5003f070 con 0x7efe68102570 2024-09-19T08:07:59.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.503+0000 7efe6cf74640 1 -- 172.21.15.98:0/1044673686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe681a6c50 con 0x7efe68102570 2024-09-19T08:07:59.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.503+0000 7efe5f7fe640 1 -- 172.21.15.98:0/1044673686 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe50031c30 con 0x7efe68102570 2024-09-19T08:07:59.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.503+0000 7efe5f7fe640 1 -- 172.21.15.98:0/1044673686 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe50035030 con 0x7efe68102570 2024-09-19T08:07:59.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.503+0000 7efe6cf74640 1 -- 172.21.15.98:0/1044673686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe681a7180 con 0x7efe68102570 2024-09-19T08:07:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.505+0000 7efe5f7fe640 1 -- 172.21.15.98:0/1044673686 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7efe50042070 con 0x7efe68102570 2024-09-19T08:07:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.506+0000 7efe5f7fe640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7efe540a33b0 0x7efe540a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.506+0000 7efe6659c640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7efe540a33b0 0x7efe540a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.506+0000 7efe5f7fe640 1 -- 172.21.15.98:0/1044673686 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7efe500f05b0 con 0x7efe68102570 2024-09-19T08:07:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.507+0000 7efe6f34d740 1 Processor -- start 2024-09-19T08:07:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.507+0000 7efe6659c640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7efe540a33b0 0x7efe540a5870 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7efe5800bbe0 tx=0x7efe5800b390 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.507+0000 7efe6f34d740 1 -- start start 2024-09-19T08:07:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.507+0000 7efe6f34d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1b4a0 0x561e7cf1b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.507+0000 7efe6f34d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1bdc0 0x561e7cf26140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.507+0000 7efe6f34d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf28a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.507+0000 7efe6f34d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561e7ce812e0 con 0x561e7cf26680 2024-09-19T08:07:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.507+0000 7efe6f34d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561e7ce28ce0 con 0x561e7cf1b4a0 2024-09-19T08:07:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.507+0000 7efe6f34d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561e7ce7d2a0 con 0x561e7cf1bdc0 2024-09-19T08:07:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.507+0000 7efe66d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf28a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.507+0000 7efe66d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf28a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43140/0 (socket says 172.21.15.98:43140) 2024-09-19T08:07:59.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.507+0000 7efe66d9d640 1 -- 172.21.15.98:0/1805650993 learned_addr learned my addr 172.21.15.98:0/1805650993 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.507+0000 7efe6659c640 1 --2- 172.21.15.98:0/1805650993 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1b4a0 0x561e7cf1b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.507+0000 7efe5ffff640 1 --2- 172.21.15.98:0/1805650993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1bdc0 0x561e7cf26140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.508+0000 7efe66d9d640 1 -- 172.21.15.98:0/1805650993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1bdc0 msgr2=0x561e7cf26140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.508+0000 7efe66d9d640 1 --2- 172.21.15.98:0/1805650993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1bdc0 0x561e7cf26140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.508+0000 7efe66d9d640 1 -- 172.21.15.98:0/1805650993 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1b4a0 msgr2=0x561e7cf1b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.508+0000 7efe66d9d640 1 --2- 172.21.15.98:0/1805650993 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1b4a0 0x561e7cf1b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.508+0000 7efe66d9d640 1 -- 172.21.15.98:0/1805650993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e7ce0d440 con 0x561e7cf26680 2024-09-19T08:07:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.508+0000 7efe6659c640 1 --2- 172.21.15.98:0/1805650993 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1b4a0 0x561e7cf1b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:07:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.508+0000 7efe66d9d640 1 --2- 172.21.15.98:0/1805650993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf28a70 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7efe60007e60 tx=0x7efe6000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b2a3b127678a07 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.508+0000 7efe5cff9640 1 -- 172.21.15.98:0/1805650993 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe60014070 con 0x561e7cf26680 2024-09-19T08:07:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.508+0000 7efe5cff9640 1 -- 172.21.15.98:0/1805650993 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe60003e50 con 0x561e7cf26680 2024-09-19T08:07:59.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.508+0000 7efe5cff9640 1 -- 172.21.15.98:0/1805650993 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe600047b0 con 0x561e7cf26680 2024-09-19T08:07:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.509+0000 7efe6f34d740 1 -- 172.21.15.98:0/1805650993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 msgr2=0x561e7cf28a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.509+0000 7efe6f34d740 1 --2- 172.21.15.98:0/1805650993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf28a70 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7efe60007e60 tx=0x7efe6000e720 comp rx=0 tx=0).stop 2024-09-19T08:07:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.509+0000 7efe6f34d740 1 -- 172.21.15.98:0/1805650993 shutdown_connections 2024-09-19T08:07:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.509+0000 7efe6f34d740 1 --2- 172.21.15.98:0/1805650993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf28a70 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.509+0000 7efe6f34d740 1 --2- 172.21.15.98:0/1805650993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1bdc0 0x561e7cf26140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.509+0000 7efe6f34d740 1 --2- 172.21.15.98:0/1805650993 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1b4a0 0x561e7cf1b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.509+0000 7efe6f34d740 1 -- 172.21.15.98:0/1805650993 >> 172.21.15.98:0/1805650993 conn(0x561e7cf17a30 msgr2=0x561e7cf17e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.509+0000 7efe6f34d740 1 -- 172.21.15.98:0/1805650993 shutdown_connections 2024-09-19T08:07:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.509+0000 7efe6f34d740 1 -- 172.21.15.98:0/1805650993 wait complete. 2024-09-19T08:07:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.509+0000 7efe6f34d740 1 -- start start 2024-09-19T08:07:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe6f34d740 1 Processor -- start 2024-09-19T08:07:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe6f34d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1b4a0 0x561e7cf25f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe6f34d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1bdc0 0x561e7cf231f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe6f34d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf23730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe6f34d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561e7cf29250 con 0x561e7cf26680 2024-09-19T08:07:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe6f34d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561e7ce3c050 con 0x561e7cf1b4a0 2024-09-19T08:07:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe6f34d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561e7cf293d0 con 0x561e7cf1bdc0 2024-09-19T08:07:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe6f34d740 10 client.0 _ll_register_callbacks cb 0x561e7ce7c160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe66d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf23730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe66d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf23730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43152/0 (socket says 172.21.15.98:43152) 2024-09-19T08:07:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe66d9d640 1 -- 172.21.15.98:0/2589039854 learned_addr learned my addr 172.21.15.98:0/2589039854 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe6659c640 1 --2- 172.21.15.98:0/2589039854 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1b4a0 0x561e7cf25f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe5ffff640 1 --2- 172.21.15.98:0/2589039854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1bdc0 0x561e7cf231f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe66d9d640 1 -- 172.21.15.98:0/2589039854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1bdc0 msgr2=0x561e7cf231f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe66d9d640 1 --2- 172.21.15.98:0/2589039854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1bdc0 0x561e7cf231f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe66d9d640 1 -- 172.21.15.98:0/2589039854 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1b4a0 msgr2=0x561e7cf25f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe66d9d640 1 --2- 172.21.15.98:0/2589039854 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1b4a0 0x561e7cf25f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe66d9d640 1 -- 172.21.15.98:0/2589039854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e7cbf23c0 con 0x561e7cf26680 2024-09-19T08:07:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe66d9d640 1 --2- 172.21.15.98:0/2589039854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf23730 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7efe60000f80 tx=0x7efe60007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8819fecda4e9270d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.510+0000 7efe4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.511+0000 7efe4dffb640 1 -- 172.21.15.98:0/2589039854 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe60014030 con 0x561e7cf26680 2024-09-19T08:07:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.511+0000 7efe4dffb640 1 -- 172.21.15.98:0/2589039854 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7efe60010850 con 0x561e7cf26680 2024-09-19T08:07:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.511+0000 7efe6f34d740 1 -- 172.21.15.98:0/2589039854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe60003e50 con 0x561e7cf26680 2024-09-19T08:07:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.511+0000 7efe4dffb640 1 -- 172.21.15.98:0/2589039854 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efe60010a90 con 0x561e7cf26680 2024-09-19T08:07:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.511+0000 7efe6f34d740 10 client.16788 fetch_fsmap learned FSMap version 31 2024-09-19T08:07:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.511+0000 7efe6f34d740 1 -- 172.21.15.98:0/2589039854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561e7cf34bd0 con 0x561e7cf26680 2024-09-19T08:07:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.511+0000 7efe4dffb640 1 -- 172.21.15.98:0/2589039854 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe60003e50 con 0x561e7cf26680 2024-09-19T08:07:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.511+0000 7efe4dffb640 1 -- 172.21.15.98:0/2589039854 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7efe60005050 con 0x561e7cf26680 2024-09-19T08:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.512+0000 7efe6f34d740 10 client.16788 fetch_fsmap finished waiting for FSMap version 31 2024-09-19T08:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.512+0000 7efe6f34d740 10 client.16788 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.512+0000 7efe6f34d740 20 client.16788 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.512+0000 7efe6f34d740 1 --2- 172.21.15.98:0/2589039854 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e7cf36090 0x561e7cf38480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.512+0000 7efe6f34d740 4 client.16788 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.512+0000 7efe6f34d740 1 -- 172.21.15.98:0/2589039854 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e7ce8e850 con 0x561e7cf36090 2024-09-19T08:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.512+0000 7efe6659c640 1 --2- 172.21.15.98:0/2589039854 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e7cf36090 0x561e7cf38480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.512+0000 7efe6659c640 1 --2- 172.21.15.98:0/2589039854 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e7cf36090 0x561e7cf38480 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2c7de7e54a37e95 server_cookie=e6b1c9faa9c350ae in_seq=0 out_seq=0 2024-09-19T08:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.512+0000 7efe4dffb640 10 client.16788 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.513+0000 7efe4dffb640 1 -- 172.21.15.98:0/2589039854 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561e7ce8e850 con 0x561e7cf36090 2024-09-19T08:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.513+0000 7efe4dffb640 10 client.16788 handle_command_reply: tid=0 2024-09-19T08:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.513+0000 7efe6f34d740 1 client.16788 shutdown 2024-09-19T08:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.513+0000 7efe6f34d740 1 -- 172.21.15.98:0/2589039854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 msgr2=0x561e7cf23730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.513+0000 7efe6f34d740 1 --2- 172.21.15.98:0/2589039854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf23730 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7efe60000f80 tx=0x7efe60007bd0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.513+0000 7efe6f34d740 1 -- 172.21.15.98:0/2589039854 shutdown_connections 2024-09-19T08:07:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.513+0000 7efe6f34d740 1 --2- 172.21.15.98:0/2589039854 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e7cf36090 0x561e7cf38480 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.513+0000 7efe6f34d740 1 --2- 172.21.15.98:0/2589039854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf23730 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.513+0000 7efe6f34d740 1 --2- 172.21.15.98:0/2589039854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1bdc0 0x561e7cf231f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.513+0000 7efe6f34d740 1 --2- 172.21.15.98:0/2589039854 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1b4a0 0x561e7cf25f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.513+0000 7efe4dffb640 0 client.16788 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.513+0000 7efe6f34d740 1 -- 172.21.15.98:0/2589039854 >> 172.21.15.98:0/2589039854 conn(0x561e7cf17a30 msgr2=0x561e7cf1d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.513+0000 7efe6f34d740 1 -- 172.21.15.98:0/2589039854 shutdown_connections 2024-09-19T08:07:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.514+0000 7efe6f34d740 1 -- 172.21.15.98:0/2589039854 wait complete. 2024-09-19T08:07:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.514+0000 7efe6f34d740 20 client.16788 trim_cache size 0 max 16384 2024-09-19T08:07:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.545+0000 7efe6f34d740 1 Processor -- start 2024-09-19T08:07:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe6f34d740 1 -- start start 2024-09-19T08:07:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe6f34d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1b4a0 0x561e7cf324e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe6f34d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf1bdc0 0x561e7cf3dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe6f34d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf26680 0x561e7cf3e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe6f34d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe681afa20 con 0x561e7cf1bdc0 2024-09-19T08:07:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe6f34d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efe681b1590 con 0x561e7cf26680 2024-09-19T08:07:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe6f34d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efe681b00f0 con 0x561e7cf1b4a0 2024-09-19T08:07:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe5ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf1bdc0 0x561e7cf3dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe5ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf1bdc0 0x561e7cf3dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43164/0 (socket says 172.21.15.98:43164) 2024-09-19T08:07:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe5ffff640 1 -- 172.21.15.98:0/3797211804 learned_addr learned my addr 172.21.15.98:0/3797211804 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe6659c640 1 --2- 172.21.15.98:0/3797211804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1b4a0 0x561e7cf324e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe66d9d640 1 --2- 172.21.15.98:0/3797211804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf26680 0x561e7cf3e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe5ffff640 1 -- 172.21.15.98:0/3797211804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1b4a0 msgr2=0x561e7cf324e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe5ffff640 1 --2- 172.21.15.98:0/3797211804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1b4a0 0x561e7cf324e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe5ffff640 1 -- 172.21.15.98:0/3797211804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf26680 msgr2=0x561e7cf3e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe5ffff640 1 --2- 172.21.15.98:0/3797211804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf26680 0x561e7cf3e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.546+0000 7efe5ffff640 1 -- 172.21.15.98:0/3797211804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e7cf262b0 con 0x561e7cf1bdc0 2024-09-19T08:07:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.547+0000 7efe6659c640 1 --2- 172.21.15.98:0/3797211804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1b4a0 0x561e7cf324e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:07:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.547+0000 7efe5ffff640 1 --2- 172.21.15.98:0/3797211804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf1bdc0 0x561e7cf3dc10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7efe500b9230 tx=0x7efe50009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da59f8a7285b640 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.547+0000 7efe4ffff640 1 -- 172.21.15.98:0/3797211804 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe50049040 con 0x561e7cf1bdc0 2024-09-19T08:07:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.547+0000 7efe4ffff640 1 -- 172.21.15.98:0/3797211804 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe500bc860 con 0x561e7cf1bdc0 2024-09-19T08:07:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.547+0000 7efe4ffff640 1 -- 172.21.15.98:0/3797211804 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe500bcac0 con 0x561e7cf1bdc0 2024-09-19T08:07:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 -- 172.21.15.98:0/3797211804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf1bdc0 msgr2=0x561e7cf3dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:59.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 --2- 172.21.15.98:0/3797211804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf1bdc0 0x561e7cf3dc10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7efe500b9230 tx=0x7efe50009560 comp rx=0 tx=0).stop 2024-09-19T08:07:59.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 -- 172.21.15.98:0/3797211804 shutdown_connections 2024-09-19T08:07:59.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 --2- 172.21.15.98:0/3797211804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf26680 0x561e7cf3e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 --2- 172.21.15.98:0/3797211804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf1bdc0 0x561e7cf3dc10 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 --2- 172.21.15.98:0/3797211804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1b4a0 0x561e7cf324e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 -- 172.21.15.98:0/3797211804 >> 172.21.15.98:0/3797211804 conn(0x561e7cf17a30 msgr2=0x561e7cf222e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:59.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 -- 172.21.15.98:0/3797211804 shutdown_connections 2024-09-19T08:07:59.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 -- 172.21.15.98:0/3797211804 wait complete. 2024-09-19T08:07:59.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 -- start start 2024-09-19T08:07:59.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 Processor -- start 2024-09-19T08:07:59.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1b4a0 0x561e7cf2ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1bdc0 0x561e7cf2a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf2a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe681b0ec0 con 0x561e7cf26680 2024-09-19T08:07:59.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efe681b07f0 con 0x561e7cf1bdc0 2024-09-19T08:07:59.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.548+0000 7efe6f34d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efe681ada00 con 0x561e7cf1b4a0 2024-09-19T08:07:59.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe6f34d740 10 client.0 _ll_register_callbacks cb 0x7efe50005750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:07:59.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe66d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf2a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe66d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf2a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43170/0 (socket says 172.21.15.98:43170) 2024-09-19T08:07:59.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe66d9d640 1 -- 172.21.15.98:0/3638805427 learned_addr learned my addr 172.21.15.98:0/3638805427 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:07:59.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe6659c640 1 --2- 172.21.15.98:0/3638805427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1b4a0 0x561e7cf2ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe5ffff640 1 --2- 172.21.15.98:0/3638805427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1bdc0 0x561e7cf2a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe66d9d640 1 -- 172.21.15.98:0/3638805427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1b4a0 msgr2=0x561e7cf2ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:59.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe66d9d640 1 --2- 172.21.15.98:0/3638805427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1b4a0 0x561e7cf2ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe66d9d640 1 -- 172.21.15.98:0/3638805427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1bdc0 msgr2=0x561e7cf2a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:07:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe66d9d640 1 --2- 172.21.15.98:0/3638805427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1bdc0 0x561e7cf2a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe66d9d640 1 -- 172.21.15.98:0/3638805427 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e7cf2adf0 con 0x561e7cf26680 2024-09-19T08:07:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe66d9d640 1 --2- 172.21.15.98:0/3638805427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf2a610 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x561e7cf1b340 tx=0x7efe60007810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0006dda78602e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:07:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:07:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe4dffb640 1 -- 172.21.15.98:0/3638805427 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe60030030 con 0x561e7cf26680 2024-09-19T08:07:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe4dffb640 1 -- 172.21.15.98:0/3638805427 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7efe60002030 con 0x561e7cf26680 2024-09-19T08:07:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.549+0000 7efe6f34d740 1 -- 172.21.15.98:0/3638805427 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe500bc860 con 0x561e7cf26680 2024-09-19T08:07:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.550+0000 7efe4dffb640 1 -- 172.21.15.98:0/3638805427 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efe6003a050 con 0x561e7cf26680 2024-09-19T08:07:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.550+0000 7efe6f34d740 10 client.16794 fetch_fsmap learned FSMap version 31 2024-09-19T08:07:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.550+0000 7efe6f34d740 1 -- 172.21.15.98:0/3638805427 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561e7cfa0ba0 con 0x561e7cf26680 2024-09-19T08:07:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.550+0000 7efe4dffb640 1 -- 172.21.15.98:0/3638805427 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe500bc860 con 0x561e7cf26680 2024-09-19T08:07:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.550+0000 7efe4dffb640 1 -- 172.21.15.98:0/3638805427 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7efe6002cca0 con 0x561e7cf26680 2024-09-19T08:07:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.550+0000 7efe6f34d740 10 client.16794 fetch_fsmap finished waiting for FSMap version 31 2024-09-19T08:07:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.550+0000 7efe6f34d740 10 client.16794 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:07:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.550+0000 7efe6f34d740 20 client.16794 populate_metadata read hostname 'smithi098' 2024-09-19T08:07:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.550+0000 7efe6f34d740 1 --2- 172.21.15.98:0/3638805427 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e7cfa12b0 0x561e7cfa36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:07:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.550+0000 7efe6f34d740 4 client.16794 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:07:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.550+0000 7efe6f34d740 1 -- 172.21.15.98:0/3638805427 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561e7ce88d60 con 0x561e7cfa12b0 2024-09-19T08:07:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.551+0000 7efe6659c640 1 --2- 172.21.15.98:0/3638805427 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e7cfa12b0 0x561e7cfa36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:07:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.551+0000 7efe6659c640 1 --2- 172.21.15.98:0/3638805427 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e7cfa12b0 0x561e7cfa36d0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2628cc03e869970 server_cookie=d4bc9b2f81f194ef in_seq=0 out_seq=0 2024-09-19T08:07:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.551+0000 7efe4dffb640 10 client.16794 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.551+0000 7efe4dffb640 1 -- 172.21.15.98:0/3638805427 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561e7ce88d60 con 0x561e7cfa12b0 2024-09-19T08:07:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.551+0000 7efe4dffb640 10 client.16794 handle_command_reply: tid=0 2024-09-19T08:07:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.551+0000 7efe6f34d740 1 client.16794 shutdown 2024-09-19T08:07:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6f34d740 1 -- 172.21.15.98:0/3638805427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 msgr2=0x561e7cf2a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6f34d740 1 --2- 172.21.15.98:0/3638805427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf2a610 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x561e7cf1b340 tx=0x7efe60007810 comp rx=0 tx=0).stop 2024-09-19T08:07:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6f34d740 1 -- 172.21.15.98:0/3638805427 shutdown_connections 2024-09-19T08:07:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6f34d740 1 --2- 172.21.15.98:0/3638805427 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561e7cfa12b0 0x561e7cfa36d0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe4dffb640 0 client.16794 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:07:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6f34d740 1 --2- 172.21.15.98:0/3638805427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561e7cf26680 0x561e7cf2a610 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6f34d740 1 --2- 172.21.15.98:0/3638805427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561e7cf1bdc0 0x561e7cf2a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6f34d740 1 --2- 172.21.15.98:0/3638805427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561e7cf1b4a0 0x561e7cf2ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6f34d740 1 -- 172.21.15.98:0/3638805427 >> 172.21.15.98:0/3638805427 conn(0x561e7cf17a30 msgr2=0x561e7cf24a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6f34d740 1 -- 172.21.15.98:0/3638805427 shutdown_connections 2024-09-19T08:07:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6f34d740 1 -- 172.21.15.98:0/3638805427 wait complete. 2024-09-19T08:07:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6f34d740 20 client.16794 trim_cache size 0 max 16384 2024-09-19T08:07:59.558 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:07:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-19T08:07:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-09-19T08:07:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: "f503195e-7f23-4a19-8b3a-9a33e73db8fe": { 2024-09-19T08:07:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-09-19T08:07:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "f503195e-7f23-4a19-8b3a-9a33e73db8fe", 2024-09-19T08:07:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-09-19T08:07:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:07:59.558 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:07:59.558 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:07:59.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6cf74640 1 -- 172.21.15.98:0/1044673686 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7efe540a33b0 msgr2=0x7efe540a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:59.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6cf74640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7efe540a33b0 0x7efe540a5870 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7efe5800bbe0 tx=0x7efe5800b390 comp rx=0 tx=0).stop 2024-09-19T08:07:59.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6cf74640 1 -- 172.21.15.98:0/1044673686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe68102570 msgr2=0x7efe680fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:07:59.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.552+0000 7efe6cf74640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe68102570 0x7efe680fd1c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7efe50006010 tx=0x7efe50003c10 comp rx=0 tx=0).stop 2024-09-19T08:07:59.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.553+0000 7efe6cf74640 1 -- 172.21.15.98:0/1044673686 shutdown_connections 2024-09-19T08:07:59.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.553+0000 7efe6cf74640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7efe540a33b0 0x7efe540a5870 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.553+0000 7efe6cf74640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe68102ef0 0x7efe680fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.553+0000 7efe6cf74640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe68102570 0x7efe680fd1c0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.553+0000 7efe6cf74640 1 --2- 172.21.15.98:0/1044673686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe68100e20 0x7efe680fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:07:59.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.553+0000 7efe6cf74640 1 -- 172.21.15.98:0/1044673686 >> 172.21.15.98:0/1044673686 conn(0x7efe680fc7d0 msgr2=0x7efe6806a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:07:59.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.553+0000 7efe6cf74640 1 -- 172.21.15.98:0/1044673686 shutdown_connections 2024-09-19T08:07:59.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:07:59.553+0000 7efe6cf74640 1 -- 172.21.15.98:0/1044673686 wait complete. 2024-09-19T08:08:00.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:07:59 smithi098 ceph-mon[31333]: pgmap v152: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:08:00.258 INFO:tasks.cephfs.filesystem:scrub status for tag:f503195e-7f23-4a19-8b3a-9a33e73db8fe - {'path': '/', 'tag': 'f503195e-7f23-4a19-8b3a-9a33e73db8fe', 'options': 'recursive,force'} 2024-09-19T08:08:00.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:07:59 smithi162 ceph-mon[35260]: pgmap v152: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:08:00.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:07:59 smithi139 ceph-mon[34874]: pgmap v152: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:08:01.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:00 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:08:01.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:00 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:08:01.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:00 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:08:02.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:01 smithi162 ceph-mon[35260]: pgmap v153: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:02.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:01 smithi139 ceph-mon[34874]: pgmap v153: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:02.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:01 smithi098 ceph-mon[31333]: pgmap v153: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:04.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:03 smithi162 ceph-mon[35260]: pgmap v154: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:08:04.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:03 smithi139 ceph-mon[34874]: pgmap v154: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:08:04.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:03 smithi098 ceph-mon[31333]: pgmap v154: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:08:06.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:05 smithi162 ceph-mon[35260]: pgmap v155: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:08:06.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:05 smithi139 ceph-mon[34874]: pgmap v155: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:08:06.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:05 smithi098 ceph-mon[31333]: pgmap v155: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:08:08.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:07 smithi162 ceph-mon[35260]: pgmap v156: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-09-19T08:08:08.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:07 smithi139 ceph-mon[34874]: pgmap v156: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-09-19T08:08:08.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:07 smithi098 ceph-mon[31333]: pgmap v156: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-09-19T08:08:10.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:09 smithi162 ceph-mon[35260]: pgmap v157: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-19T08:08:10.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:09 smithi139 ceph-mon[34874]: pgmap v157: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-19T08:08:10.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:09 smithi098 ceph-mon[31333]: pgmap v157: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-19T08:08:12.230 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:11 smithi139 ceph-mon[34874]: pgmap v158: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:08:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:11 smithi162 ceph-mon[35260]: pgmap v158: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:08:12.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:11 smithi098 ceph-mon[31333]: pgmap v158: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:08:13.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:12 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:08:13.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:12 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:08:13.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:12 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:08:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:13 smithi162 ceph-mon[35260]: pgmap v159: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:08:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:13 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:08:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:13 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:08:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:13 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:08:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:13 smithi139 ceph-mon[34874]: pgmap v159: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:08:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:13 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:08:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:13 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:08:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:13 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:08:14.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:13 smithi098 ceph-mon[31333]: pgmap v159: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:08:14.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:13 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:08:14.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:13 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:08:14.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:13 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:08:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:15 smithi162 ceph-mon[35260]: pgmap v160: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-19T08:08:16.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:15 smithi139 ceph-mon[34874]: pgmap v160: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-19T08:08:16.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:15 smithi098 ceph-mon[31333]: pgmap v160: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-19T08:08:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:17 smithi162 ceph-mon[35260]: pgmap v161: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-19T08:08:18.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:17 smithi139 ceph-mon[34874]: pgmap v161: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-19T08:08:18.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:17 smithi098 ceph-mon[31333]: pgmap v161: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-19T08:08:20.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:19 smithi162 ceph-mon[35260]: pgmap v162: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:08:20.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:19 smithi139 ceph-mon[34874]: pgmap v162: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:08:20.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:19 smithi098 ceph-mon[31333]: pgmap v162: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:08:22.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:21 smithi162 ceph-mon[35260]: pgmap v163: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:22.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:21 smithi139 ceph-mon[34874]: pgmap v163: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:22.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:21 smithi098 ceph-mon[31333]: pgmap v163: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:24.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:23 smithi162 ceph-mon[35260]: pgmap v164: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:08:24.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:23 smithi139 ceph-mon[34874]: pgmap v164: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:08:24.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:23 smithi098 ceph-mon[31333]: pgmap v164: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:08:26.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:25 smithi162 ceph-mon[35260]: pgmap v165: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:26.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:25 smithi139 ceph-mon[34874]: pgmap v165: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:26.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:25 smithi098 ceph-mon[31333]: pgmap v165: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:27 smithi162 ceph-mon[35260]: pgmap v166: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:08:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:27 smithi162 ceph-mon[35260]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:08:28.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:27 smithi139 ceph-mon[34874]: pgmap v166: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:08:28.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:27 smithi139 ceph-mon[34874]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:08:28.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:27 smithi098 ceph-mon[31333]: pgmap v166: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:08:28.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:27 smithi098 ceph-mon[31333]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T08:08:30.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:08:30.288 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:29 smithi098 ceph-mon[31333]: pgmap v167: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:08:30.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:29 smithi162 ceph-mon[35260]: pgmap v167: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:08:30.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:29 smithi139 ceph-mon[34874]: pgmap v167: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:08:30.522 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:08:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.993+0000 7fb2c87b1640 1 -- 172.21.15.98:0/2333174058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2c0102570 msgr2=0x7fb2c01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.993+0000 7fb2c87b1640 1 --2- 172.21.15.98:0/2333174058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2c0102570 0x7fb2c01029b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009900 tx=0x7fb2b002a480 comp rx=0 tx=0).stop 2024-09-19T08:08:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 -- 172.21.15.98:0/2333174058 shutdown_connections 2024-09-19T08:08:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 --2- 172.21.15.98:0/2333174058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c0102ef0 0x7fb2c010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 --2- 172.21.15.98:0/2333174058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2c0102570 0x7fb2c01029b0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 --2- 172.21.15.98:0/2333174058 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2c0100e20 0x7fb2c0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 -- 172.21.15.98:0/2333174058 >> 172.21.15.98:0/2333174058 conn(0x7fb2c00fc7d0 msgr2=0x7fb2c00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 -- 172.21.15.98:0/2333174058 shutdown_connections 2024-09-19T08:08:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 -- 172.21.15.98:0/2333174058 wait complete. 2024-09-19T08:08:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 Processor -- start 2024-09-19T08:08:31.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 -- start start 2024-09-19T08:08:31.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2c0100e20 0x7fb2c00fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c0102570 0x7fb2c00fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2c0102ef0 0x7fb2c00fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c0110fa0 con 0x7fb2c0102ef0 2024-09-19T08:08:31.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c0110e20 con 0x7fb2c0100e20 2024-09-19T08:08:31.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.994+0000 7fb2c87b1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c0111120 con 0x7fb2c0102570 2024-09-19T08:08:31.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2c6d4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2c0102ef0 0x7fb2c00fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2c5d4c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c0102570 0x7fb2c00fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2c6d4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2c0102ef0 0x7fb2c00fd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55558/0 (socket says 172.21.15.98:55558) 2024-09-19T08:08:31.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2c5d4c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c0102570 0x7fb2c00fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:58114/0 (socket says 172.21.15.98:58114) 2024-09-19T08:08:31.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2c6d4e640 1 -- 172.21.15.98:0/1165967427 learned_addr learned my addr 172.21.15.98:0/1165967427 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:31.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2c654d640 1 --2- 172.21.15.98:0/1165967427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2c0100e20 0x7fb2c00fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2c6d4e640 1 -- 172.21.15.98:0/1165967427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c0102570 msgr2=0x7fb2c00fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2c6d4e640 1 --2- 172.21.15.98:0/1165967427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c0102570 0x7fb2c00fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2c6d4e640 1 -- 172.21.15.98:0/1165967427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2c0100e20 msgr2=0x7fb2c00fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2c6d4e640 1 --2- 172.21.15.98:0/1165967427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2c0100e20 0x7fb2c00fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2c6d4e640 1 -- 172.21.15.98:0/1165967427 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c01a6a20 con 0x7fb2c0102ef0 2024-09-19T08:08:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2c6d4e640 1 --2- 172.21.15.98:0/1165967427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2c0102ef0 0x7fb2c00fd700 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb2b800b970 tx=0x7fb2b800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2b77fe640 1 -- 172.21.15.98:0/1165967427 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8019070 con 0x7fb2c0102ef0 2024-09-19T08:08:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2c87b1640 1 -- 172.21.15.98:0/1165967427 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c01a6d10 con 0x7fb2c0102ef0 2024-09-19T08:08:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2b77fe640 1 -- 172.21.15.98:0/1165967427 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2b800cb40 con 0x7fb2c0102ef0 2024-09-19T08:08:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.995+0000 7fb2b77fe640 1 -- 172.21.15.98:0/1165967427 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8007650 con 0x7fb2c0102ef0 2024-09-19T08:08:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.996+0000 7fb2c87b1640 1 -- 172.21.15.98:0/1165967427 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c01a7220 con 0x7fb2c0102ef0 2024-09-19T08:08:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.998+0000 7fb2b77fe640 1 -- 172.21.15.98:0/1165967427 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fb2b801c070 con 0x7fb2c0102ef0 2024-09-19T08:08:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.998+0000 7fb2cab8a740 1 Processor -- start 2024-09-19T08:08:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.998+0000 7fb2cab8a740 1 -- start start 2024-09-19T08:08:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2cab8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e6c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2cab8a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e6cdc0 0x560411e77140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2cab8a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e77680 0x560411e79a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2cab8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560411dd22e0 con 0x560411e6c4a0 2024-09-19T08:08:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2cab8a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560411d79ce0 con 0x560411e6cdc0 2024-09-19T08:08:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2cab8a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560411dce2a0 con 0x560411e77680 2024-09-19T08:08:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2c5d4c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e6c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2c5d4c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e6c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55562/0 (socket says 172.21.15.98:55562) 2024-09-19T08:08:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2c5d4c640 1 -- 172.21.15.98:0/1843905412 learned_addr learned my addr 172.21.15.98:0/1843905412 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2c5d4c640 1 --2- 172.21.15.98:0/1843905412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e77680 0x560411e79a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2c654d640 1 --2- 172.21.15.98:0/1843905412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e6cdc0 0x560411e77140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2c5d4c640 1 -- 172.21.15.98:0/1843905412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e77680 msgr2=0x560411e79a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2c5d4c640 1 --2- 172.21.15.98:0/1843905412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e77680 0x560411e79a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2c5d4c640 1 -- 172.21.15.98:0/1843905412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e6cdc0 msgr2=0x560411e77140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2c5d4c640 1 --2- 172.21.15.98:0/1843905412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e6cdc0 0x560411e77140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2b77fe640 1 --2- 172.21.15.98:0/1165967427 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb2ac0a3430 0x7fb2ac0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:30.999+0000 7fb2c5d4c640 1 -- 172.21.15.98:0/1843905412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560411d5e440 con 0x560411e6c4a0 2024-09-19T08:08:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.000+0000 7fb2c5d4c640 1 --2- 172.21.15.98:0/1843905412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e6c880 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fb2b002a9a0 tx=0x7fb2b0005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333f6d0545200bf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.000+0000 7fb2b77fe640 1 -- 172.21.15.98:0/1165967427 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fb2b80cf140 con 0x7fb2c0102ef0 2024-09-19T08:08:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.000+0000 7fb2c5d4c640 1 --2- 172.21.15.98:0/1165967427 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb2ac0a3430 0x7fb2ac0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.000+0000 7fb2b4ff9640 1 -- 172.21.15.98:0/1843905412 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b0035070 con 0x560411e6c4a0 2024-09-19T08:08:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.000+0000 7fb2b4ff9640 1 -- 172.21.15.98:0/1843905412 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2b00315c0 con 0x560411e6c4a0 2024-09-19T08:08:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.000+0000 7fb2b4ff9640 1 -- 172.21.15.98:0/1843905412 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b003b940 con 0x560411e6c4a0 2024-09-19T08:08:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.001+0000 7fb2c5d4c640 1 --2- 172.21.15.98:0/1165967427 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb2ac0a3430 0x7fb2ac0a58f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0046000 tx=0x7fb2b003bc90 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.001+0000 7fb2cab8a740 1 -- 172.21.15.98:0/1843905412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 msgr2=0x560411e6c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.001+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/1843905412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e6c880 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fb2b002a9a0 tx=0x7fb2b0005c00 comp rx=0 tx=0).stop 2024-09-19T08:08:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.001+0000 7fb2cab8a740 1 -- 172.21.15.98:0/1843905412 shutdown_connections 2024-09-19T08:08:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.001+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/1843905412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e77680 0x560411e79a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.001+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/1843905412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e6cdc0 0x560411e77140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.001+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/1843905412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e6c880 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.001+0000 7fb2cab8a740 1 -- 172.21.15.98:0/1843905412 >> 172.21.15.98:0/1843905412 conn(0x560411e68a30 msgr2=0x560411e68e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.001+0000 7fb2cab8a740 1 -- 172.21.15.98:0/1843905412 shutdown_connections 2024-09-19T08:08:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.001+0000 7fb2cab8a740 1 -- 172.21.15.98:0/1843905412 wait complete. 2024-09-19T08:08:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.001+0000 7fb2cab8a740 1 -- start start 2024-09-19T08:08:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.001+0000 7fb2cab8a740 1 Processor -- start 2024-09-19T08:08:31.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2cab8a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6c4a0 0x560411e76f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2cab8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6cdc0 0x560411e741b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2cab8a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 0x560411e746f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2cab8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560411e7a250 con 0x560411e6cdc0 2024-09-19T08:08:31.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2cab8a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560411d8d050 con 0x560411e77680 2024-09-19T08:08:31.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2cab8a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560411e7a3d0 con 0x560411e6c4a0 2024-09-19T08:08:31.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2cab8a740 10 client.0 _ll_register_callbacks cb 0x560411dcd160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:31.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2c5d4c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6cdc0 0x560411e741b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2c654d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6c4a0 0x560411e76f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2c654d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6c4a0 0x560411e76f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:58134/0 (socket says 172.21.15.98:58134) 2024-09-19T08:08:31.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2c5d4c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6cdc0 0x560411e741b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55568/0 (socket says 172.21.15.98:55568) 2024-09-19T08:08:31.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2c654d640 1 -- 172.21.15.98:0/902404517 learned_addr learned my addr 172.21.15.98:0/902404517 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:31.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2c6d4e640 1 --2- 172.21.15.98:0/902404517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 0x560411e746f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2c654d640 1 -- 172.21.15.98:0/902404517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 msgr2=0x560411e746f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:31.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2c654d640 1 --2- 172.21.15.98:0/902404517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 0x560411e746f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2c654d640 1 -- 172.21.15.98:0/902404517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6cdc0 msgr2=0x560411e741b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:31.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2c654d640 1 --2- 172.21.15.98:0/902404517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6cdc0 0x560411e741b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.002+0000 7fb2c654d640 1 -- 172.21.15.98:0/902404517 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560411b433c0 con 0x560411e6c4a0 2024-09-19T08:08:31.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.003+0000 7fb2c654d640 1 --2- 172.21.15.98:0/902404517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6c4a0 0x560411e76f00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc012790 tx=0x7fb2bc012c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b236c2e58995e05 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:31.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.003+0000 7fb2a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-09-19T08:08:31.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.003+0000 7fb2a9ffb640 1 -- 172.21.15.98:0/902404517 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc010dc0 con 0x560411e6c4a0 2024-09-19T08:08:31.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.003+0000 7fb2cab8a740 1 -- 172.21.15.98:0/902404517 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2b00315c0 con 0x560411e6c4a0 2024-09-19T08:08:31.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.003+0000 7fb2a9ffb640 1 -- 172.21.15.98:0/902404517 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fb2bc005020 con 0x560411e6c4a0 2024-09-19T08:08:31.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.003+0000 7fb2a9ffb640 1 -- 172.21.15.98:0/902404517 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb2bc005260 con 0x560411e6c4a0 2024-09-19T08:08:31.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.003+0000 7fb2cab8a740 10 client.25894 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:31.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.003+0000 7fb2cab8a740 1 -- 172.21.15.98:0/902404517 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560411e85c20 con 0x560411e6c4a0 2024-09-19T08:08:31.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.003+0000 7fb2a9ffb640 1 -- 172.21.15.98:0/902404517 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2b00315c0 con 0x560411e6c4a0 2024-09-19T08:08:31.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.003+0000 7fb2a9ffb640 1 -- 172.21.15.98:0/902404517 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb2bc01a4c0 con 0x560411e6c4a0 2024-09-19T08:08:31.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.003+0000 7fb2cab8a740 10 client.25894 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:31.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.003+0000 7fb2cab8a740 10 client.25894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:08:31.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.004+0000 7fb2cab8a740 20 client.25894 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:31.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.004+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/902404517 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560411e87180 0x560411e89600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.004+0000 7fb2cab8a740 4 client.25894 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:08:31.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.004+0000 7fb2cab8a740 1 -- 172.21.15.98:0/902404517 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560411ddf850 con 0x560411e87180 2024-09-19T08:08:31.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.004+0000 7fb2c5d4c640 1 --2- 172.21.15.98:0/902404517 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560411e87180 0x560411e89600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.004+0000 7fb2c5d4c640 1 --2- 172.21.15.98:0/902404517 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560411e87180 0x560411e89600 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bc4547bfdfd4492 server_cookie=8f6d0ed0441d586f in_seq=0 out_seq=0 2024-09-19T08:08:31.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.004+0000 7fb2a9ffb640 10 client.25894 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:31.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.005+0000 7fb2a9ffb640 1 -- 172.21.15.98:0/902404517 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560411ddf850 con 0x560411e87180 2024-09-19T08:08:31.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.005+0000 7fb2a9ffb640 10 client.25894 handle_command_reply: tid=0 2024-09-19T08:08:31.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.005+0000 7fb2cab8a740 1 client.25894 shutdown 2024-09-19T08:08:31.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.005+0000 7fb2cab8a740 1 -- 172.21.15.98:0/902404517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6c4a0 msgr2=0x560411e76f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:31.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.005+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/902404517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6c4a0 0x560411e76f00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc012790 tx=0x7fb2bc012c50 comp rx=0 tx=0).stop 2024-09-19T08:08:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.005+0000 7fb2cab8a740 1 -- 172.21.15.98:0/902404517 shutdown_connections 2024-09-19T08:08:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.005+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/902404517 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560411e87180 0x560411e89600 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.005+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/902404517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 0x560411e746f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.005+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/902404517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6cdc0 0x560411e741b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.005+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/902404517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6c4a0 0x560411e76f00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.005+0000 7fb2cab8a740 1 -- 172.21.15.98:0/902404517 >> 172.21.15.98:0/902404517 conn(0x560411e68a30 msgr2=0x560411e6e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.005+0000 7fb2a9ffb640 0 client.25894 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.005+0000 7fb2cab8a740 1 -- 172.21.15.98:0/902404517 shutdown_connections 2024-09-19T08:08:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.006+0000 7fb2cab8a740 1 -- 172.21.15.98:0/902404517 wait complete. 2024-09-19T08:08:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.006+0000 7fb2cab8a740 20 client.25894 trim_cache size 0 max 16384 2024-09-19T08:08:31.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2cab8a740 1 Processor -- start 2024-09-19T08:08:31.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2cab8a740 1 -- start start 2024-09-19T08:08:31.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2cab8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e83510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2cab8a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6cdc0 0x560411e6f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2cab8a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 0x560411e67a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2cab8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c01afab0 con 0x560411e6c4a0 2024-09-19T08:08:31.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2cab8a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c01b15f0 con 0x560411e77680 2024-09-19T08:08:31.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2cab8a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c01b0180 con 0x560411e6cdc0 2024-09-19T08:08:31.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2c654d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e83510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2c654d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e83510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55584/0 (socket says 172.21.15.98:55584) 2024-09-19T08:08:31.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2c654d640 1 -- 172.21.15.98:0/514459414 learned_addr learned my addr 172.21.15.98:0/514459414 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:31.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2c6d4e640 1 --2- 172.21.15.98:0/514459414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 0x560411e67a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2c5d4c640 1 --2- 172.21.15.98:0/514459414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6cdc0 0x560411e6f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2c654d640 1 -- 172.21.15.98:0/514459414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6cdc0 msgr2=0x560411e6f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:31.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2c654d640 1 --2- 172.21.15.98:0/514459414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6cdc0 0x560411e6f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2c654d640 1 -- 172.21.15.98:0/514459414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 msgr2=0x560411e67a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:31.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2c654d640 1 --2- 172.21.15.98:0/514459414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 0x560411e67a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2c654d640 1 -- 172.21.15.98:0/514459414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560411e73160 con 0x560411e6c4a0 2024-09-19T08:08:31.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.023+0000 7fb2c6d4e640 1 --2- 172.21.15.98:0/514459414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 0x560411e67a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:08:31.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.024+0000 7fb2c654d640 1 --2- 172.21.15.98:0/514459414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e83510 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0058c0 tx=0x7fb2bc00ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd56e30e515170cf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:31.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.024+0000 7fb2abfff640 1 -- 172.21.15.98:0/514459414 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc01a6f0 con 0x560411e6c4a0 2024-09-19T08:08:31.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.024+0000 7fb2abfff640 1 -- 172.21.15.98:0/514459414 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2bc005400 con 0x560411e6c4a0 2024-09-19T08:08:31.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.024+0000 7fb2abfff640 1 -- 172.21.15.98:0/514459414 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc0035f0 con 0x560411e6c4a0 2024-09-19T08:08:31.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.024+0000 7fb2cab8a740 1 -- 172.21.15.98:0/514459414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 msgr2=0x560411e83510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:31.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.024+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/514459414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e83510 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0058c0 tx=0x7fb2bc00ac00 comp rx=0 tx=0).stop 2024-09-19T08:08:31.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 -- 172.21.15.98:0/514459414 shutdown_connections 2024-09-19T08:08:31.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/514459414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 0x560411e67a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/514459414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6cdc0 0x560411e6f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/514459414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e83510 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 -- 172.21.15.98:0/514459414 >> 172.21.15.98:0/514459414 conn(0x560411e68a30 msgr2=0x560411e67370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:31.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 -- 172.21.15.98:0/514459414 shutdown_connections 2024-09-19T08:08:31.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 -- 172.21.15.98:0/514459414 wait complete. 2024-09-19T08:08:31.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 -- start start 2024-09-19T08:08:31.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 Processor -- start 2024-09-19T08:08:31.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e7b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6cdc0 0x560411e7be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 0x560411ef0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c01b0f20 con 0x560411e6c4a0 2024-09-19T08:08:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c01b0850 con 0x560411e77680 2024-09-19T08:08:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c01ada90 con 0x560411e6cdc0 2024-09-19T08:08:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2cab8a740 10 client.0 _ll_register_callbacks cb 0x7fb2bc00a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2c654d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e7b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2c654d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e7b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55592/0 (socket says 172.21.15.98:55592) 2024-09-19T08:08:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.025+0000 7fb2c654d640 1 -- 172.21.15.98:0/3070452965 learned_addr learned my addr 172.21.15.98:0/3070452965 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2c6d4e640 1 --2- 172.21.15.98:0/3070452965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 0x560411ef0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2c5d4c640 1 --2- 172.21.15.98:0/3070452965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6cdc0 0x560411e7be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2c654d640 1 -- 172.21.15.98:0/3070452965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6cdc0 msgr2=0x560411e7be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2c654d640 1 --2- 172.21.15.98:0/3070452965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6cdc0 0x560411e7be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2c654d640 1 -- 172.21.15.98:0/3070452965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 msgr2=0x560411ef0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2c654d640 1 --2- 172.21.15.98:0/3070452965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 0x560411ef0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2c654d640 1 -- 172.21.15.98:0/3070452965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560411ef13e0 con 0x560411e6c4a0 2024-09-19T08:08:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2c654d640 1 --2- 172.21.15.98:0/3070452965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e7b8c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0124b0 tx=0x7fb2bc01a960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83084ae09eb8d858 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2a9ffb640 1 -- 172.21.15.98:0/3070452965 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc010dc0 con 0x560411e6c4a0 2024-09-19T08:08:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2a9ffb640 1 -- 172.21.15.98:0/3070452965 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fb2bc012030 con 0x560411e6c4a0 2024-09-19T08:08:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2cab8a740 1 -- 172.21.15.98:0/3070452965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2bc005400 con 0x560411e6c4a0 2024-09-19T08:08:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2a9ffb640 1 -- 172.21.15.98:0/3070452965 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb2bc044050 con 0x560411e6c4a0 2024-09-19T08:08:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2cab8a740 10 client.16812 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2cab8a740 1 -- 172.21.15.98:0/3070452965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560411ef25d0 con 0x560411e6c4a0 2024-09-19T08:08:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.026+0000 7fb2a9ffb640 1 -- 172.21.15.98:0/3070452965 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2bc005400 con 0x560411e6c4a0 2024-09-19T08:08:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.027+0000 7fb2a9ffb640 1 -- 172.21.15.98:0/3070452965 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb2bc003900 con 0x560411e6c4a0 2024-09-19T08:08:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.027+0000 7fb2cab8a740 10 client.16812 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.027+0000 7fb2cab8a740 10 client.16812 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:08:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.027+0000 7fb2cab8a740 20 client.16812 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.027+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/3070452965 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560411ef2c80 0x560411ef5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.027+0000 7fb2cab8a740 4 client.16812 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:08:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.027+0000 7fb2cab8a740 1 -- 172.21.15.98:0/3070452965 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560411dd9d60 con 0x560411ef2c80 2024-09-19T08:08:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.027+0000 7fb2c5d4c640 1 --2- 172.21.15.98:0/3070452965 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560411ef2c80 0x560411ef5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.027+0000 7fb2a9ffb640 10 client.16812 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.027+0000 7fb2c5d4c640 1 --2- 172.21.15.98:0/3070452965 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560411ef2c80 0x560411ef5070 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38612af8051410d1 server_cookie=df539c65d736dd0d in_seq=0 out_seq=0 2024-09-19T08:08:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2a9ffb640 1 -- 172.21.15.98:0/3070452965 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560411dd9d60 con 0x560411ef2c80 2024-09-19T08:08:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2a9ffb640 10 client.16812 handle_command_reply: tid=0 2024-09-19T08:08:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2cab8a740 1 client.16812 shutdown 2024-09-19T08:08:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2cab8a740 1 -- 172.21.15.98:0/3070452965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 msgr2=0x560411e7b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/3070452965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e7b8c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0124b0 tx=0x7fb2bc01a960 comp rx=0 tx=0).stop 2024-09-19T08:08:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2cab8a740 1 -- 172.21.15.98:0/3070452965 shutdown_connections 2024-09-19T08:08:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/3070452965 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560411ef2c80 0x560411ef5070 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/3070452965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560411e77680 0x560411ef0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/3070452965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560411e6cdc0 0x560411e7be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2cab8a740 1 --2- 172.21.15.98:0/3070452965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560411e6c4a0 0x560411e7b8c0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2cab8a740 1 -- 172.21.15.98:0/3070452965 >> 172.21.15.98:0/3070452965 conn(0x560411e68a30 msgr2=0x560411e75a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2a9ffb640 0 client.16812 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2cab8a740 1 -- 172.21.15.98:0/3070452965 shutdown_connections 2024-09-19T08:08:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2cab8a740 1 -- 172.21.15.98:0/3070452965 wait complete. 2024-09-19T08:08:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2cab8a740 20 client.16812 trim_cache size 0 max 16384 2024-09-19T08:08:31.036 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:08:31.037 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:08:31.037 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:08:31.037 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:08:31.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2c87b1640 1 -- 172.21.15.98:0/1165967427 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb2ac0a3430 msgr2=0x7fb2ac0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2c87b1640 1 --2- 172.21.15.98:0/1165967427 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb2ac0a3430 0x7fb2ac0a58f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0046000 tx=0x7fb2b003bc90 comp rx=0 tx=0).stop 2024-09-19T08:08:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2c87b1640 1 -- 172.21.15.98:0/1165967427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2c0102ef0 msgr2=0x7fb2c00fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2c87b1640 1 --2- 172.21.15.98:0/1165967427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2c0102ef0 0x7fb2c00fd700 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb2b800b970 tx=0x7fb2b800be30 comp rx=0 tx=0).stop 2024-09-19T08:08:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2c87b1640 1 -- 172.21.15.98:0/1165967427 shutdown_connections 2024-09-19T08:08:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2c87b1640 1 --2- 172.21.15.98:0/1165967427 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb2ac0a3430 0x7fb2ac0a58f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2c87b1640 1 --2- 172.21.15.98:0/1165967427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2c0102ef0 0x7fb2c00fd700 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2c87b1640 1 --2- 172.21.15.98:0/1165967427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c0102570 0x7fb2c00fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.028+0000 7fb2c87b1640 1 --2- 172.21.15.98:0/1165967427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2c0100e20 0x7fb2c00fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.029+0000 7fb2c87b1640 1 -- 172.21.15.98:0/1165967427 >> 172.21.15.98:0/1165967427 conn(0x7fb2c00fc7d0 msgr2=0x7fb2c006a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.029+0000 7fb2c87b1640 1 -- 172.21.15.98:0/1165967427 shutdown_connections 2024-09-19T08:08:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:31.029+0000 7fb2c87b1640 1 -- 172.21.15.98:0/1165967427 wait complete. 2024-09-19T08:08:31.828 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:08:31.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:08:32.129 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:08:32.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:32 smithi098 ceph-mon[31333]: pgmap v168: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:32.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:32 smithi162 ceph-mon[35260]: pgmap v168: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:32.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:32 smithi139 ceph-mon[34874]: pgmap v168: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:32.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.623+0000 7fda1cc28640 1 -- 172.21.15.98:0/3494722102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda18101df0 msgr2=0x7fda181021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:32.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.623+0000 7fda1cc28640 1 --2- 172.21.15.98:0/3494722102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda18101df0 0x7fda181021d0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fda10009ca0 tx=0x7fda1002a8c0 comp rx=0 tx=0).stop 2024-09-19T08:08:32.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.623+0000 7fda1cc28640 1 -- 172.21.15.98:0/3494722102 shutdown_connections 2024-09-19T08:08:32.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.623+0000 7fda1cc28640 1 --2- 172.21.15.98:0/3494722102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda18106f20 0x7fda1810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:32.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.623+0000 7fda1cc28640 1 --2- 172.21.15.98:0/3494722102 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda181027a0 0x7fda181067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:32.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.623+0000 7fda1cc28640 1 --2- 172.21.15.98:0/3494722102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda18101df0 0x7fda181021d0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:32.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.623+0000 7fda1cc28640 1 -- 172.21.15.98:0/3494722102 >> 172.21.15.98:0/3494722102 conn(0x7fda180fd8f0 msgr2=0x7fda180ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:32.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.623+0000 7fda1cc28640 1 -- 172.21.15.98:0/3494722102 shutdown_connections 2024-09-19T08:08:32.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.623+0000 7fda1cc28640 1 -- 172.21.15.98:0/3494722102 wait complete. 2024-09-19T08:08:32.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda1cc28640 1 Processor -- start 2024-09-19T08:08:32.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda1cc28640 1 -- start start 2024-09-19T08:08:32.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda1cc28640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda18101df0 0x7fda18198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:32.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda1cc28640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda181027a0 0x7fda181989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:32.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda1cc28640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda18106f20 0x7fda1819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:32.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda1cc28640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fda18111f80 con 0x7fda18106f20 2024-09-19T08:08:32.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda1cc28640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fda18111e00 con 0x7fda181027a0 2024-09-19T08:08:32.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda1cc28640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fda18112100 con 0x7fda18101df0 2024-09-19T08:08:32.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda16d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda18106f20 0x7fda1819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:32.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda16d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda18106f20 0x7fda1819cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55608/0 (socket says 172.21.15.98:55608) 2024-09-19T08:08:32.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda16d9d640 1 -- 172.21.15.98:0/3954360259 learned_addr learned my addr 172.21.15.98:0/3954360259 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:32.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda1659c640 1 --2- 172.21.15.98:0/3954360259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda18101df0 0x7fda18198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:32.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda15d9b640 1 --2- 172.21.15.98:0/3954360259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda181027a0 0x7fda181989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:32.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda16d9d640 1 -- 172.21.15.98:0/3954360259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda18101df0 msgr2=0x7fda18198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:32.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda16d9d640 1 --2- 172.21.15.98:0/3954360259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda18101df0 0x7fda18198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:32.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda16d9d640 1 -- 172.21.15.98:0/3954360259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda181027a0 msgr2=0x7fda181989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:32.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda16d9d640 1 --2- 172.21.15.98:0/3954360259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda181027a0 0x7fda181989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:32.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.624+0000 7fda16d9d640 1 -- 172.21.15.98:0/3954360259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1819d410 con 0x7fda18106f20 2024-09-19T08:08:32.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.625+0000 7fda16d9d640 1 --2- 172.21.15.98:0/3954360259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda18106f20 0x7fda1819cd30 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fda0800d750 tx=0x7fda0800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:32.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.625+0000 7fda0f7fe640 1 -- 172.21.15.98:0/3954360259 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda08017070 con 0x7fda18106f20 2024-09-19T08:08:32.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.625+0000 7fda0f7fe640 1 -- 172.21.15.98:0/3954360259 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda08005500 con 0x7fda18106f20 2024-09-19T08:08:32.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.625+0000 7fda1cc28640 1 -- 172.21.15.98:0/3954360259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda1819d700 con 0x7fda18106f20 2024-09-19T08:08:32.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.625+0000 7fda0f7fe640 1 -- 172.21.15.98:0/3954360259 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda08004d90 con 0x7fda18106f20 2024-09-19T08:08:32.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.625+0000 7fda1cc28640 1 -- 172.21.15.98:0/3954360259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda181a4fe0 con 0x7fda18106f20 2024-09-19T08:08:32.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.626+0000 7fda1cc28640 1 -- 172.21.15.98:0/3954360259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda181021d0 con 0x7fda18106f20 2024-09-19T08:08:32.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.627+0000 7fda0f7fe640 1 -- 172.21.15.98:0/3954360259 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fda08005020 con 0x7fda18106f20 2024-09-19T08:08:32.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.628+0000 7fda0f7fe640 1 --2- 172.21.15.98:0/3954360259 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fda040a33b0 0x7fda040a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:32.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.628+0000 7fda1659c640 1 --2- 172.21.15.98:0/3954360259 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fda040a33b0 0x7fda040a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:32.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.628+0000 7fda0f7fe640 1 -- 172.21.15.98:0/3954360259 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fda080cf6f0 con 0x7fda18106f20 2024-09-19T08:08:32.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.628+0000 7fda1659c640 1 --2- 172.21.15.98:0/3954360259 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fda040a33b0 0x7fda040a5870 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fda10000f80 tx=0x7fda100093f0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:32.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.632+0000 7fda0f7fe640 1 -- 172.21.15.98:0/3954360259 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fda0809c100 con 0x7fda18106f20 2024-09-19T08:08:32.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.831+0000 7fda1cc28640 1 -- 172.21.15.98:0/3954360259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda1810a160 con 0x7fda18106f20 2024-09-19T08:08:32.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.831+0000 7fda0f7fe640 1 -- 172.21.15.98:0/3954360259 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11983 (secure 0 0 0) 0x7fda0809ba00 con 0x7fda18106f20 2024-09-19T08:08:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:08:32.832 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"btime":"2024-09-19T08:08:26:985720+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:08:26.555658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:08:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2024-09-19T08:08:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.834+0000 7fda1cc28640 1 -- 172.21.15.98:0/3954360259 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fda040a33b0 msgr2=0x7fda040a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.834+0000 7fda1cc28640 1 --2- 172.21.15.98:0/3954360259 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fda040a33b0 0x7fda040a5870 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fda10000f80 tx=0x7fda100093f0 comp rx=0 tx=0).stop 2024-09-19T08:08:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.834+0000 7fda1cc28640 1 -- 172.21.15.98:0/3954360259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda18106f20 msgr2=0x7fda1819cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.834+0000 7fda1cc28640 1 --2- 172.21.15.98:0/3954360259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda18106f20 0x7fda1819cd30 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fda0800d750 tx=0x7fda0800dc10 comp rx=0 tx=0).stop 2024-09-19T08:08:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.834+0000 7fda1cc28640 1 -- 172.21.15.98:0/3954360259 shutdown_connections 2024-09-19T08:08:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.834+0000 7fda1cc28640 1 --2- 172.21.15.98:0/3954360259 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fda040a33b0 0x7fda040a5870 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.834+0000 7fda1cc28640 1 --2- 172.21.15.98:0/3954360259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda18106f20 0x7fda1819cd30 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.834+0000 7fda1cc28640 1 --2- 172.21.15.98:0/3954360259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda181027a0 0x7fda181989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.834+0000 7fda1cc28640 1 --2- 172.21.15.98:0/3954360259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda18101df0 0x7fda18198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.834+0000 7fda1cc28640 1 -- 172.21.15.98:0/3954360259 >> 172.21.15.98:0/3954360259 conn(0x7fda180fd8f0 msgr2=0x7fda180fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.834+0000 7fda1cc28640 1 -- 172.21.15.98:0/3954360259 shutdown_connections 2024-09-19T08:08:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:32.834+0000 7fda1cc28640 1 -- 172.21.15.98:0/3954360259 wait complete. 2024-09-19T08:08:33.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:33 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3954360259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:08:33.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:33 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3954360259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:08:33.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:33 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3954360259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:08:33.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:08:33.729 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:08:34.134 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:34 smithi098 ceph-mon[31333]: pgmap v169: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:08:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.214+0000 7f5263564640 1 -- 172.21.15.98:0/121031892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f525c101420 msgr2=0x7f525c10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.214+0000 7f5263564640 1 --2- 172.21.15.98:0/121031892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f525c101420 0x7f525c10d950 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f5254009cd0 tx=0x7f525402a8e0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.214+0000 7f5263564640 1 -- 172.21.15.98:0/121031892 shutdown_connections 2024-09-19T08:08:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.214+0000 7f5263564640 1 --2- 172.21.15.98:0/121031892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f525c101420 0x7f525c10d950 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.214+0000 7f5263564640 1 --2- 172.21.15.98:0/121031892 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f525c100a80 0x7f525c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.214+0000 7f5263564640 1 --2- 172.21.15.98:0/121031892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f525c106a80 0x7f525c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.215+0000 7f5263564640 1 -- 172.21.15.98:0/121031892 >> 172.21.15.98:0/121031892 conn(0x7f525c0fc7a0 msgr2=0x7f525c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.215+0000 7f5263564640 1 -- 172.21.15.98:0/121031892 shutdown_connections 2024-09-19T08:08:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.215+0000 7f5263564640 1 -- 172.21.15.98:0/121031892 wait complete. 2024-09-19T08:08:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.215+0000 7f5263564640 1 Processor -- start 2024-09-19T08:08:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.215+0000 7f5263564640 1 -- start start 2024-09-19T08:08:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.215+0000 7f5263564640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f525c100a80 0x7f525c197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.215+0000 7f5263564640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f525c101420 0x7f525c1978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.215+0000 7f5263564640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f525c106a80 0x7f525c19bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.215+0000 7f5263564640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f525c110ec0 con 0x7f525c100a80 2024-09-19T08:08:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.215+0000 7f5263564640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f525c110d40 con 0x7f525c106a80 2024-09-19T08:08:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.215+0000 7f5263564640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f525c111040 con 0x7f525c101420 2024-09-19T08:08:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f5261300640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f525c100a80 0x7f525c197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f5261300640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f525c100a80 0x7f525c197380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55638/0 (socket says 172.21.15.98:55638) 2024-09-19T08:08:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f5261300640 1 -- 172.21.15.98:0/4235928375 learned_addr learned my addr 172.21.15.98:0/4235928375 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f5260aff640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f525c101420 0x7f525c1978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f5261b01640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f525c106a80 0x7f525c19bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f5261300640 1 -- 172.21.15.98:0/4235928375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f525c101420 msgr2=0x7f525c1978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f5261300640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f525c101420 0x7f525c1978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f5261300640 1 -- 172.21.15.98:0/4235928375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f525c106a80 msgr2=0x7f525c19bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f5261300640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f525c106a80 0x7f525c19bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f5261300640 1 -- 172.21.15.98:0/4235928375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525c19c330 con 0x7f525c100a80 2024-09-19T08:08:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f5261b01640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f525c106a80 0x7f525c19bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:08:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f5261300640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f525c100a80 0x7f525c197380 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f525800d770 tx=0x7f525800db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f52527fc640 1 -- 172.21.15.98:0/4235928375 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5258017070 con 0x7f525c100a80 2024-09-19T08:08:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f5263564640 1 -- 172.21.15.98:0/4235928375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f525c19c620 con 0x7f525c100a80 2024-09-19T08:08:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f52527fc640 1 -- 172.21.15.98:0/4235928375 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5258009820 con 0x7f525c100a80 2024-09-19T08:08:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f52527fc640 1 -- 172.21.15.98:0/4235928375 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5258004cd0 con 0x7f525c100a80 2024-09-19T08:08:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.216+0000 7f5263564640 1 -- 172.21.15.98:0/4235928375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f525c1a3f00 con 0x7f525c100a80 2024-09-19T08:08:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.218+0000 7f52527fc640 1 -- 172.21.15.98:0/4235928375 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f52580099d0 con 0x7f525c100a80 2024-09-19T08:08:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.219+0000 7f526593d740 1 Processor -- start 2024-09-19T08:08:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.219+0000 7f526593d740 1 -- start start 2024-09-19T08:08:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.219+0000 7f526593d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df9b40 0x557022df8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.219+0000 7f526593d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df85d0 0x557022df8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.219+0000 7f526593d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022e05580 0x557022e07970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.219+0000 7f526593d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557022d5e2f0 con 0x557022df9b40 2024-09-19T08:08:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.219+0000 7f526593d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557022d05cf0 con 0x557022e05580 2024-09-19T08:08:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.219+0000 7f526593d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557022d5a2b0 con 0x557022df85d0 2024-09-19T08:08:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.219+0000 7f52527fc640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f52480a33b0 0x7f52480a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.219+0000 7f5261b01640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df9b40 0x557022df8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.219+0000 7f5261b01640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df9b40 0x557022df8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55642/0 (socket says 172.21.15.98:55642) 2024-09-19T08:08:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.219+0000 7f5261b01640 1 -- 172.21.15.98:0/2713716761 learned_addr learned my addr 172.21.15.98:0/2713716761 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.219+0000 7f5261b01640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f52480a33b0 0x7f52480a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.220+0000 7f52527fc640 1 -- 172.21.15.98:0/4235928375 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f52580cf730 con 0x7f525c100a80 2024-09-19T08:08:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.220+0000 7f5261300640 1 --2- 172.21.15.98:0/2713716761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df85d0 0x557022df8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.220+0000 7f5260aff640 1 --2- 172.21.15.98:0/2713716761 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022e05580 0x557022e07970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.220+0000 7f5261b01640 1 -- 172.21.15.98:0/2713716761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df85d0 msgr2=0x557022df8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.220+0000 7f5261b01640 1 --2- 172.21.15.98:0/2713716761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df85d0 0x557022df8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.220+0000 7f5261b01640 1 -- 172.21.15.98:0/2713716761 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022e05580 msgr2=0x557022e07970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.220+0000 7f5261b01640 1 --2- 172.21.15.98:0/2713716761 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022e05580 0x557022e07970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.220+0000 7f5261b01640 1 -- 172.21.15.98:0/2713716761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557022cea3a0 con 0x557022df9b40 2024-09-19T08:08:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.220+0000 7f5260aff640 1 --2- 172.21.15.98:0/2713716761 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022e05580 0x557022e07970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:08:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.220+0000 7f5261b01640 1 --2- 172.21.15.98:0/2713716761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df9b40 0x557022df8090 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f5254009a60 tx=0x7f5254031d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ea5b614283b401 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.220+0000 7f5261b01640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f52480a33b0 0x7f52480a5870 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5254006010 tx=0x7f5254047000 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.220+0000 7f52477fe640 1 -- 172.21.15.98:0/2713716761 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525404d070 con 0x557022df9b40 2024-09-19T08:08:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.220+0000 7f52477fe640 1 -- 172.21.15.98:0/2713716761 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5254036040 con 0x557022df9b40 2024-09-19T08:08:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.220+0000 7f52477fe640 1 -- 172.21.15.98:0/2713716761 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254030df0 con 0x557022df9b40 2024-09-19T08:08:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.221+0000 7f526593d740 1 -- 172.21.15.98:0/2713716761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df9b40 msgr2=0x557022df8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.221+0000 7f526593d740 1 --2- 172.21.15.98:0/2713716761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df9b40 0x557022df8090 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f5254009a60 tx=0x7f5254031d70 comp rx=0 tx=0).stop 2024-09-19T08:08:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.221+0000 7f526593d740 1 -- 172.21.15.98:0/2713716761 shutdown_connections 2024-09-19T08:08:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.221+0000 7f526593d740 1 --2- 172.21.15.98:0/2713716761 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022e05580 0x557022e07970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.221+0000 7f526593d740 1 --2- 172.21.15.98:0/2713716761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df85d0 0x557022df8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.221+0000 7f526593d740 1 --2- 172.21.15.98:0/2713716761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df9b40 0x557022df8090 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.221+0000 7f526593d740 1 -- 172.21.15.98:0/2713716761 >> 172.21.15.98:0/2713716761 conn(0x557022dc1310 msgr2=0x557022df4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.221+0000 7f526593d740 1 -- 172.21.15.98:0/2713716761 shutdown_connections 2024-09-19T08:08:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.221+0000 7f526593d740 1 -- 172.21.15.98:0/2713716761 wait complete. 2024-09-19T08:08:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.221+0000 7f526593d740 1 -- start start 2024-09-19T08:08:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.221+0000 7f526593d740 1 Processor -- start 2024-09-19T08:08:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f526593d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022df85d0 0x557022e04fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f526593d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df9b40 0x557022e02260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f526593d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022e05580 0x557022e029f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f526593d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557022e08220 con 0x557022e05580 2024-09-19T08:08:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f526593d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557022d19060 con 0x557022df85d0 2024-09-19T08:08:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f526593d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557022e083a0 con 0x557022df9b40 2024-09-19T08:08:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f526593d740 10 client.0 _ll_register_callbacks cb 0x557022a6c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f5261b01640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022e05580 0x557022e029f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f5261b01640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022e05580 0x557022e029f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55648/0 (socket says 172.21.15.98:55648) 2024-09-19T08:08:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f5261b01640 1 -- 172.21.15.98:0/2116159932 learned_addr learned my addr 172.21.15.98:0/2116159932 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f5260aff640 1 --2- 172.21.15.98:0/2116159932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df9b40 0x557022e02260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f5261300640 1 --2- 172.21.15.98:0/2116159932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022df85d0 0x557022e04fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f5261b01640 1 -- 172.21.15.98:0/2116159932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df9b40 msgr2=0x557022e02260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f5261b01640 1 --2- 172.21.15.98:0/2116159932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df9b40 0x557022e02260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f5261b01640 1 -- 172.21.15.98:0/2116159932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022df85d0 msgr2=0x557022e04fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f5261b01640 1 --2- 172.21.15.98:0/2116159932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022df85d0 0x557022e04fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f5261b01640 1 -- 172.21.15.98:0/2116159932 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557022acf3c0 con 0x557022e05580 2024-09-19T08:08:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.222+0000 7f5261300640 1 --2- 172.21.15.98:0/2116159932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022df85d0 0x557022e04fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:08:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f5261b01640 1 --2- 172.21.15.98:0/2116159932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022e05580 0x557022e029f0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f5254031920 tx=0x7f5254004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21cb41fcc78736c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f5244ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f5244ff9640 1 -- 172.21.15.98:0/2116159932 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525404d050 con 0x557022e05580 2024-09-19T08:08:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f5244ff9640 1 -- 172.21.15.98:0/2116159932 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f5254030af0 con 0x557022e05580 2024-09-19T08:08:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f526593d740 1 -- 172.21.15.98:0/2116159932 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5254036040 con 0x557022e05580 2024-09-19T08:08:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f5244ff9640 1 -- 172.21.15.98:0/2116159932 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f525402af20 con 0x557022e05580 2024-09-19T08:08:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f526593d740 10 client.16830 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f526593d740 1 -- 172.21.15.98:0/2116159932 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557022e13c70 con 0x557022e05580 2024-09-19T08:08:34.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f5244ff9640 1 -- 172.21.15.98:0/2116159932 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5254036040 con 0x557022e05580 2024-09-19T08:08:34.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f5244ff9640 1 -- 172.21.15.98:0/2116159932 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5254048070 con 0x557022e05580 2024-09-19T08:08:34.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f526593d740 10 client.16830 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:34.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f526593d740 10 client.16830 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:08:34.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f526593d740 20 client.16830 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:34.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f526593d740 1 --2- 172.21.15.98:0/2116159932 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557022e14fc0 0x557022e173b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f526593d740 4 client.16830 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:08:34.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.223+0000 7f526593d740 1 -- 172.21.15.98:0/2116159932 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557022d6b860 con 0x557022e14fc0 2024-09-19T08:08:34.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.224+0000 7f5261300640 1 --2- 172.21.15.98:0/2116159932 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557022e14fc0 0x557022e173b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.224+0000 7f5244ff9640 10 client.16830 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:08:34.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.224+0000 7f5261300640 1 --2- 172.21.15.98:0/2116159932 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557022e14fc0 0x557022e173b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d9d87f8fc1deff92 server_cookie=b84fadc30cf5cd9d in_seq=0 out_seq=0 2024-09-19T08:08:34.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.225+0000 7f5244ff9640 1 -- 172.21.15.98:0/2116159932 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557022d6b860 con 0x557022e14fc0 2024-09-19T08:08:34.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.225+0000 7f5244ff9640 10 client.16830 handle_command_reply: tid=0 2024-09-19T08:08:34.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.225+0000 7f526593d740 1 client.16830 shutdown 2024-09-19T08:08:34.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.225+0000 7f526593d740 1 -- 172.21.15.98:0/2116159932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022e05580 msgr2=0x557022e029f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:34.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.225+0000 7f526593d740 1 --2- 172.21.15.98:0/2116159932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022e05580 0x557022e029f0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f5254031920 tx=0x7f5254004d70 comp rx=0 tx=0).stop 2024-09-19T08:08:34.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.225+0000 7f526593d740 1 -- 172.21.15.98:0/2116159932 shutdown_connections 2024-09-19T08:08:34.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.225+0000 7f526593d740 1 --2- 172.21.15.98:0/2116159932 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557022e14fc0 0x557022e173b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.225+0000 7f526593d740 1 --2- 172.21.15.98:0/2116159932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022e05580 0x557022e029f0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.225+0000 7f5244ff9640 0 client.16830 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:08:34.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.225+0000 7f526593d740 1 --2- 172.21.15.98:0/2116159932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df9b40 0x557022e02260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.225+0000 7f526593d740 1 --2- 172.21.15.98:0/2116159932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022df85d0 0x557022e04fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.225+0000 7f526593d740 1 -- 172.21.15.98:0/2116159932 >> 172.21.15.98:0/2116159932 conn(0x557022dc1310 msgr2=0x557022dfa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:34.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.226+0000 7f526593d740 1 -- 172.21.15.98:0/2116159932 shutdown_connections 2024-09-19T08:08:34.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.226+0000 7f526593d740 1 -- 172.21.15.98:0/2116159932 wait complete. 2024-09-19T08:08:34.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.226+0000 7f526593d740 20 client.16830 trim_cache size 0 max 16384 2024-09-19T08:08:34.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.238+0000 7f526593d740 1 Processor -- start 2024-09-19T08:08:34.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.238+0000 7f526593d740 1 -- start start 2024-09-19T08:08:34.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.238+0000 7f526593d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df85d0 0x557022df4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f526593d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df9b40 0x557022e1c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f526593d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022e05580 0x557022e1cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f526593d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f525c1ab5c0 con 0x557022df85d0 2024-09-19T08:08:34.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f526593d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f525c1ad100 con 0x557022e05580 2024-09-19T08:08:34.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f526593d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f525c1abc90 con 0x557022df9b40 2024-09-19T08:08:34.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f5261300640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df85d0 0x557022df4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f5261300640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df85d0 0x557022df4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55660/0 (socket says 172.21.15.98:55660) 2024-09-19T08:08:34.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f5261300640 1 -- 172.21.15.98:0/1554496770 learned_addr learned my addr 172.21.15.98:0/1554496770 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:34.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f5260aff640 1 --2- 172.21.15.98:0/1554496770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df9b40 0x557022e1c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f5261300640 1 -- 172.21.15.98:0/1554496770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df9b40 msgr2=0x557022e1c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:34.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f5261300640 1 --2- 172.21.15.98:0/1554496770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df9b40 0x557022e1c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f5261300640 1 -- 172.21.15.98:0/1554496770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022e05580 msgr2=0x557022e1cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:34.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f5261300640 1 --2- 172.21.15.98:0/1554496770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022e05580 0x557022e1cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f5261300640 1 -- 172.21.15.98:0/1554496770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557022e01270 con 0x557022df85d0 2024-09-19T08:08:34.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f5261300640 1 --2- 172.21.15.98:0/1554496770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df85d0 0x557022df4e10 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f52580ccf10 tx=0x7f5258086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a92165c4fd9ffd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:34.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f5246ffd640 1 -- 172.21.15.98:0/1554496770 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52580b9070 con 0x557022df85d0 2024-09-19T08:08:34.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f5246ffd640 1 -- 172.21.15.98:0/1554496770 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5258099d40 con 0x557022df85d0 2024-09-19T08:08:34.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.239+0000 7f5246ffd640 1 -- 172.21.15.98:0/1554496770 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5258020040 con 0x557022df85d0 2024-09-19T08:08:34.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.240+0000 7f526593d740 1 -- 172.21.15.98:0/1554496770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df85d0 msgr2=0x557022df4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:34.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.240+0000 7f526593d740 1 --2- 172.21.15.98:0/1554496770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df85d0 0x557022df4e10 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f52580ccf10 tx=0x7f5258086040 comp rx=0 tx=0).stop 2024-09-19T08:08:34.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.240+0000 7f526593d740 1 -- 172.21.15.98:0/1554496770 shutdown_connections 2024-09-19T08:08:34.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.240+0000 7f526593d740 1 --2- 172.21.15.98:0/1554496770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022e05580 0x557022e1cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.240+0000 7f526593d740 1 --2- 172.21.15.98:0/1554496770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022df9b40 0x557022e1c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.240+0000 7f526593d740 1 --2- 172.21.15.98:0/1554496770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df85d0 0x557022df4e10 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.240+0000 7f526593d740 1 -- 172.21.15.98:0/1554496770 >> 172.21.15.98:0/1554496770 conn(0x557022dc1310 msgr2=0x557022df4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:34.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.240+0000 7f526593d740 1 -- 172.21.15.98:0/1554496770 shutdown_connections 2024-09-19T08:08:34.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.240+0000 7f526593d740 1 -- 172.21.15.98:0/1554496770 wait complete. 2024-09-19T08:08:34.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.240+0000 7f526593d740 1 -- start start 2024-09-19T08:08:34.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.240+0000 7f526593d740 1 Processor -- start 2024-09-19T08:08:34.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f526593d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022df85d0 0x557022e0d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f526593d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df9b40 0x557022e0dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f526593d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022e05580 0x557022e0a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f526593d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f525c1aca30 con 0x557022df9b40 2024-09-19T08:08:34.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f526593d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f525c1ac360 con 0x557022df85d0 2024-09-19T08:08:34.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f526593d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f525c1a95a0 con 0x557022e05580 2024-09-19T08:08:34.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f526593d740 10 client.0 _ll_register_callbacks cb 0x7f5254031bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:34.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f5260aff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df9b40 0x557022e0dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f5260aff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df9b40 0x557022e0dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55670/0 (socket says 172.21.15.98:55670) 2024-09-19T08:08:34.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f5260aff640 1 -- 172.21.15.98:0/1562362329 learned_addr learned my addr 172.21.15.98:0/1562362329 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:34.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f5261300640 1 --2- 172.21.15.98:0/1562362329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022df85d0 0x557022e0d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f5261b01640 1 --2- 172.21.15.98:0/1562362329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022e05580 0x557022e0a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f5260aff640 1 -- 172.21.15.98:0/1562362329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022e05580 msgr2=0x557022e0a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:34.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f5260aff640 1 --2- 172.21.15.98:0/1562362329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022e05580 0x557022e0a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f5260aff640 1 -- 172.21.15.98:0/1562362329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022df85d0 msgr2=0x557022e0d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:34.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f5260aff640 1 --2- 172.21.15.98:0/1562362329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022df85d0 0x557022e0d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f5260aff640 1 -- 172.21.15.98:0/1562362329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557022e0ac90 con 0x557022df9b40 2024-09-19T08:08:34.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f5261300640 1 --2- 172.21.15.98:0/1562362329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022df85d0 0x557022e0d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:08:34.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f5260aff640 1 --2- 172.21.15.98:0/1562362329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df9b40 0x557022e0dd50 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f525c0fed90 tx=0x7f524c0167a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba2a08cb52aedc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:34.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.241+0000 7f5244ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:34.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.242+0000 7f5244ff9640 1 -- 172.21.15.98:0/1562362329 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c017640 con 0x557022df9b40 2024-09-19T08:08:34.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.242+0000 7f5244ff9640 1 -- 172.21.15.98:0/1562362329 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f524c0177e0 con 0x557022df9b40 2024-09-19T08:08:34.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.242+0000 7f526593d740 1 -- 172.21.15.98:0/1562362329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5258099d40 con 0x557022df9b40 2024-09-19T08:08:34.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.242+0000 7f5244ff9640 1 -- 172.21.15.98:0/1562362329 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f524c017a00 con 0x557022df9b40 2024-09-19T08:08:34.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.242+0000 7f526593d740 10 client.16836 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:34.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.242+0000 7f526593d740 1 -- 172.21.15.98:0/1562362329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557022e80340 con 0x557022df9b40 2024-09-19T08:08:34.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.242+0000 7f5244ff9640 1 -- 172.21.15.98:0/1562362329 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5258099d40 con 0x557022df9b40 2024-09-19T08:08:34.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.242+0000 7f5244ff9640 1 -- 172.21.15.98:0/1562362329 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f524c0283d0 con 0x557022df9b40 2024-09-19T08:08:34.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.242+0000 7f526593d740 10 client.16836 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:34.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.242+0000 7f526593d740 10 client.16836 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:08:34.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.242+0000 7f526593d740 20 client.16836 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:34.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.242+0000 7f526593d740 1 --2- 172.21.15.98:0/1562362329 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557022e14fc0 0x557022e82b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:34.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.242+0000 7f526593d740 4 client.16836 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:08:34.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.242+0000 7f526593d740 1 -- 172.21.15.98:0/1562362329 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557022d65d70 con 0x557022e14fc0 2024-09-19T08:08:34.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f5261300640 1 --2- 172.21.15.98:0/1562362329 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557022e14fc0 0x557022e82b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:34.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f5244ff9640 10 client.16836 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:08:34.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f5261300640 1 --2- 172.21.15.98:0/1562362329 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557022e14fc0 0x557022e82b20 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3118a1ac4bdab706 server_cookie=a8d13ecddfa24be7 in_seq=0 out_seq=0 2024-09-19T08:08:34.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f5244ff9640 1 -- 172.21.15.98:0/1562362329 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557022d65d70 con 0x557022e14fc0 2024-09-19T08:08:34.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f5244ff9640 10 client.16836 handle_command_reply: tid=0 2024-09-19T08:08:34.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f526593d740 1 client.16836 shutdown 2024-09-19T08:08:34.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f526593d740 1 -- 172.21.15.98:0/1562362329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df9b40 msgr2=0x557022e0dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:34.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f526593d740 1 --2- 172.21.15.98:0/1562362329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df9b40 0x557022e0dd50 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f525c0fed90 tx=0x7f524c0167a0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f526593d740 1 -- 172.21.15.98:0/1562362329 shutdown_connections 2024-09-19T08:08:34.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f526593d740 1 --2- 172.21.15.98:0/1562362329 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x557022e14fc0 0x557022e82b20 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f5244ff9640 0 client.16836 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:08:34.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f526593d740 1 --2- 172.21.15.98:0/1562362329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557022e05580 0x557022e0a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f526593d740 1 --2- 172.21.15.98:0/1562362329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557022df9b40 0x557022e0dd50 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f526593d740 1 --2- 172.21.15.98:0/1562362329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557022df85d0 0x557022e0d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.243+0000 7f526593d740 1 -- 172.21.15.98:0/1562362329 >> 172.21.15.98:0/1562362329 conn(0x557022dc1310 msgr2=0x557022e03ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:34.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.244+0000 7f526593d740 1 -- 172.21.15.98:0/1562362329 shutdown_connections 2024-09-19T08:08:34.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.244+0000 7f526593d740 1 -- 172.21.15.98:0/1562362329 wait complete. 2024-09-19T08:08:34.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.244+0000 7f526593d740 20 client.16836 trim_cache size 0 max 16384 2024-09-19T08:08:34.250 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:08:34.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.244+0000 7f5263564640 1 -- 172.21.15.98:0/4235928375 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f52480a33b0 msgr2=0x7f52480a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:34.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.244+0000 7f5263564640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f52480a33b0 0x7f52480a5870 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5254006010 tx=0x7f5254047000 comp rx=0 tx=0).stop 2024-09-19T08:08:34.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.244+0000 7f5263564640 1 -- 172.21.15.98:0/4235928375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f525c100a80 msgr2=0x7f525c197380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:34.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.244+0000 7f5263564640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f525c100a80 0x7f525c197380 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f525800d770 tx=0x7f525800db70 comp rx=0 tx=0).stop 2024-09-19T08:08:34.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.244+0000 7f5263564640 1 -- 172.21.15.98:0/4235928375 shutdown_connections 2024-09-19T08:08:34.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.244+0000 7f5263564640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f52480a33b0 0x7f52480a5870 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.244+0000 7f5263564640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f525c106a80 0x7f525c19bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.244+0000 7f5263564640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f525c101420 0x7f525c1978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.244+0000 7f5263564640 1 --2- 172.21.15.98:0/4235928375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f525c100a80 0x7f525c197380 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:34.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.244+0000 7f5263564640 1 -- 172.21.15.98:0/4235928375 >> 172.21.15.98:0/4235928375 conn(0x7f525c0fc7a0 msgr2=0x7f525c0fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:34.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.245+0000 7f5263564640 1 -- 172.21.15.98:0/4235928375 shutdown_connections 2024-09-19T08:08:34.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:34.245+0000 7f5263564640 1 -- 172.21.15.98:0/4235928375 wait complete. 2024-09-19T08:08:34.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:34 smithi162 ceph-mon[35260]: pgmap v169: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:08:34.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:34 smithi139 ceph-mon[34874]: pgmap v169: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:08:34.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:08:35.149 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:08:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.641+0000 7efe51087640 1 -- 172.21.15.98:0/3218883599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe4c102790 msgr2=0x7efe4c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.641+0000 7efe51087640 1 --2- 172.21.15.98:0/3218883599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe4c102790 0x7efe4c1067e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7efe34009900 tx=0x7efe3402a410 comp rx=0 tx=0).stop 2024-09-19T08:08:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.641+0000 7efe51087640 1 -- 172.21.15.98:0/3218883599 shutdown_connections 2024-09-19T08:08:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.641+0000 7efe51087640 1 --2- 172.21.15.98:0/3218883599 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe4c106f10 0x7efe4c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.641+0000 7efe51087640 1 --2- 172.21.15.98:0/3218883599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe4c102790 0x7efe4c1067e0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.641+0000 7efe51087640 1 --2- 172.21.15.98:0/3218883599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe4c101de0 0x7efe4c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.641+0000 7efe51087640 1 -- 172.21.15.98:0/3218883599 >> 172.21.15.98:0/3218883599 conn(0x7efe4c0fd8c0 msgr2=0x7efe4c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.641+0000 7efe51087640 1 -- 172.21.15.98:0/3218883599 shutdown_connections 2024-09-19T08:08:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.641+0000 7efe51087640 1 -- 172.21.15.98:0/3218883599 wait complete. 2024-09-19T08:08:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.642+0000 7efe51087640 1 Processor -- start 2024-09-19T08:08:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.642+0000 7efe51087640 1 -- start start 2024-09-19T08:08:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.642+0000 7efe51087640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe4c101de0 0x7efe4c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.642+0000 7efe51087640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe4c102790 0x7efe4c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.642+0000 7efe51087640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe4c106f10 0x7efe4c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.642+0000 7efe51087640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe4c111f30 con 0x7efe4c102790 2024-09-19T08:08:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.642+0000 7efe51087640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efe4c111db0 con 0x7efe4c106f10 2024-09-19T08:08:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.642+0000 7efe51087640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efe4c1120b0 con 0x7efe4c101de0 2024-09-19T08:08:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe4259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe4c102790 0x7efe4c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe4259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe4c102790 0x7efe4c198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55700/0 (socket says 172.21.15.98:55700) 2024-09-19T08:08:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe4259c640 1 -- 172.21.15.98:0/381611986 learned_addr learned my addr 172.21.15.98:0/381611986 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe4b59e640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe4c106f10 0x7efe4c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe4ad9d640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe4c101de0 0x7efe4c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe4259c640 1 -- 172.21.15.98:0/381611986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe4c101de0 msgr2=0x7efe4c198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe4259c640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe4c101de0 0x7efe4c198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe4259c640 1 -- 172.21.15.98:0/381611986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe4c106f10 msgr2=0x7efe4c19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe4259c640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe4c106f10 0x7efe4c19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe4259c640 1 -- 172.21.15.98:0/381611986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe4c19d400 con 0x7efe4c102790 2024-09-19T08:08:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe4b59e640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe4c106f10 0x7efe4c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:08:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe4259c640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe4c102790 0x7efe4c198990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7efe34006010 tx=0x7efe340042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe48d99640 1 -- 172.21.15.98:0/381611986 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3403f030 con 0x7efe4c102790 2024-09-19T08:08:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe48d99640 1 -- 172.21.15.98:0/381611986 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe34003ce0 con 0x7efe4c102790 2024-09-19T08:08:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe51087640 1 -- 172.21.15.98:0/381611986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe4c19d690 con 0x7efe4c102790 2024-09-19T08:08:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe48d99640 1 -- 172.21.15.98:0/381611986 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe34035070 con 0x7efe4c102790 2024-09-19T08:08:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.643+0000 7efe51087640 1 -- 172.21.15.98:0/381611986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe4c1a4fd0 con 0x7efe4c102790 2024-09-19T08:08:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.646+0000 7efe48d99640 1 -- 172.21.15.98:0/381611986 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7efe34042030 con 0x7efe4c102790 2024-09-19T08:08:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.646+0000 7efe53460740 1 Processor -- start 2024-09-19T08:08:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.646+0000 7efe53460740 1 -- start start 2024-09-19T08:08:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.646+0000 7efe48d99640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7efe380a33b0 0x7efe380a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.646+0000 7efe4b59e640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7efe380a33b0 0x7efe380a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.646+0000 7efe48d99640 1 -- 172.21.15.98:0/381611986 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7efe340f0380 con 0x7efe4c102790 2024-09-19T08:08:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe53460740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 0x558c0c485090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe53460740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c4855d0 0x558c0c485a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe53460740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c494780 0x558c0c496b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe53460740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558c0c3eb2f0 con 0x558c0c494780 2024-09-19T08:08:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe53460740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558c0c392cf0 con 0x558c0c486b40 2024-09-19T08:08:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe53460740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558c0c3e72b0 con 0x558c0c4855d0 2024-09-19T08:08:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe4b59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c494780 0x558c0c496b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe4b59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c494780 0x558c0c496b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55704/0 (socket says 172.21.15.98:55704) 2024-09-19T08:08:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe4b59e640 1 -- 172.21.15.98:0/1618499393 learned_addr learned my addr 172.21.15.98:0/1618499393 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe4259c640 1 --2- 172.21.15.98:0/1618499393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c4855d0 0x558c0c485a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe4b59e640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7efe380a33b0 0x7efe380a5870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efe440092f0 tx=0x7efe44009280 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe4ad9d640 1 --2- 172.21.15.98:0/1618499393 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 0x558c0c485090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe4b59e640 1 -- 172.21.15.98:0/1618499393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c4855d0 msgr2=0x558c0c485a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe4b59e640 1 --2- 172.21.15.98:0/1618499393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c4855d0 0x558c0c485a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe4b59e640 1 -- 172.21.15.98:0/1618499393 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 msgr2=0x558c0c485090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe4b59e640 1 --2- 172.21.15.98:0/1618499393 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 0x558c0c485090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe4b59e640 1 -- 172.21.15.98:0/1618499393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c0c3773a0 con 0x558c0c494780 2024-09-19T08:08:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe4b59e640 1 --2- 172.21.15.98:0/1618499393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c494780 0x558c0c496b70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7efe440125e0 tx=0x7efe44013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28859ab294c8bbf6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe4159a640 1 -- 172.21.15.98:0/1618499393 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe44023070 con 0x558c0c494780 2024-09-19T08:08:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.647+0000 7efe4159a640 1 -- 172.21.15.98:0/1618499393 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe44011c00 con 0x558c0c494780 2024-09-19T08:08:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.648+0000 7efe4159a640 1 -- 172.21.15.98:0/1618499393 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4401fe20 con 0x558c0c494780 2024-09-19T08:08:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.648+0000 7efe53460740 1 -- 172.21.15.98:0/1618499393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c494780 msgr2=0x558c0c496b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.648+0000 7efe53460740 1 --2- 172.21.15.98:0/1618499393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c494780 0x558c0c496b70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7efe440125e0 tx=0x7efe44013000 comp rx=0 tx=0).stop 2024-09-19T08:08:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.648+0000 7efe53460740 1 -- 172.21.15.98:0/1618499393 shutdown_connections 2024-09-19T08:08:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.648+0000 7efe53460740 1 --2- 172.21.15.98:0/1618499393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c494780 0x558c0c496b70 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.648+0000 7efe53460740 1 --2- 172.21.15.98:0/1618499393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c4855d0 0x558c0c485a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.648+0000 7efe53460740 1 --2- 172.21.15.98:0/1618499393 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 0x558c0c485090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.648+0000 7efe53460740 1 -- 172.21.15.98:0/1618499393 >> 172.21.15.98:0/1618499393 conn(0x558c0c44e310 msgr2=0x558c0c481c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.648+0000 7efe53460740 1 -- 172.21.15.98:0/1618499393 shutdown_connections 2024-09-19T08:08:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.648+0000 7efe53460740 1 -- 172.21.15.98:0/1618499393 wait complete. 2024-09-19T08:08:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe53460740 1 -- start start 2024-09-19T08:08:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe53460740 1 Processor -- start 2024-09-19T08:08:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe53460740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c4855d0 0x558c0c494110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe53460740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c486b40 0x558c0c4913c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe53460740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c494780 0x558c0c491940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe53460740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558c0c497480 con 0x558c0c486b40 2024-09-19T08:08:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe53460740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558c0c3a6060 con 0x558c0c494780 2024-09-19T08:08:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe53460740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558c0c497600 con 0x558c0c4855d0 2024-09-19T08:08:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe53460740 10 client.0 _ll_register_callbacks cb 0x558c0c0f98c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe4259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c486b40 0x558c0c4913c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe4259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c486b40 0x558c0c4913c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55716/0 (socket says 172.21.15.98:55716) 2024-09-19T08:08:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe4259c640 1 -- 172.21.15.98:0/66557929 learned_addr learned my addr 172.21.15.98:0/66557929 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe4b59e640 1 --2- 172.21.15.98:0/66557929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c494780 0x558c0c491940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe4ad9d640 1 --2- 172.21.15.98:0/66557929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c4855d0 0x558c0c494110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe4259c640 1 -- 172.21.15.98:0/66557929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c4855d0 msgr2=0x558c0c494110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe4259c640 1 --2- 172.21.15.98:0/66557929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c4855d0 0x558c0c494110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe4259c640 1 -- 172.21.15.98:0/66557929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c494780 msgr2=0x558c0c491940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe4259c640 1 --2- 172.21.15.98:0/66557929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c494780 0x558c0c491940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.649+0000 7efe4259c640 1 -- 172.21.15.98:0/66557929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c0c15c3c0 con 0x558c0c486b40 2024-09-19T08:08:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe4259c640 1 --2- 172.21.15.98:0/66557929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c486b40 0x558c0c4913c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7efe340b9120 tx=0x7efe340b8a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9475070ed081f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe327fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe327fc640 1 -- 172.21.15.98:0/66557929 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe34049050 con 0x558c0c486b40 2024-09-19T08:08:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe327fc640 1 -- 172.21.15.98:0/66557929 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7efe340bc9b0 con 0x558c0c486b40 2024-09-19T08:08:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe53460740 1 -- 172.21.15.98:0/66557929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe44011c00 con 0x558c0c486b40 2024-09-19T08:08:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe327fc640 1 -- 172.21.15.98:0/66557929 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efe340bc0b0 con 0x558c0c486b40 2024-09-19T08:08:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe53460740 10 client.16848 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe53460740 1 -- 172.21.15.98:0/66557929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558c0c4a2eb0 con 0x558c0c486b40 2024-09-19T08:08:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe327fc640 1 -- 172.21.15.98:0/66557929 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe44011c00 con 0x558c0c486b40 2024-09-19T08:08:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe327fc640 1 -- 172.21.15.98:0/66557929 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7efe340c4f30 con 0x558c0c486b40 2024-09-19T08:08:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe53460740 10 client.16848 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe53460740 10 client.16848 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:08:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe53460740 20 client.16848 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe53460740 1 --2- 172.21.15.98:0/66557929 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x558c0c4a43b0 0x558c0c4a67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe53460740 4 client.16848 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:08:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.650+0000 7efe53460740 1 -- 172.21.15.98:0/66557929 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c0c3f8860 con 0x558c0c4a43b0 2024-09-19T08:08:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.651+0000 7efe4ad9d640 1 --2- 172.21.15.98:0/66557929 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x558c0c4a43b0 0x558c0c4a67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.651+0000 7efe327fc640 10 client.16848 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:08:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.651+0000 7efe4ad9d640 1 --2- 172.21.15.98:0/66557929 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x558c0c4a43b0 0x558c0c4a67e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9351de167a5218a9 server_cookie=c71d15f770a95cc6 in_seq=0 out_seq=0 2024-09-19T08:08:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe327fc640 1 -- 172.21.15.98:0/66557929 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558c0c3f8860 con 0x558c0c4a43b0 2024-09-19T08:08:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe327fc640 10 client.16848 handle_command_reply: tid=0 2024-09-19T08:08:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe53460740 1 client.16848 shutdown 2024-09-19T08:08:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe53460740 1 -- 172.21.15.98:0/66557929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c486b40 msgr2=0x558c0c4913c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe53460740 1 --2- 172.21.15.98:0/66557929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c486b40 0x558c0c4913c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7efe340b9120 tx=0x7efe340b8a70 comp rx=0 tx=0).stop 2024-09-19T08:08:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe53460740 1 -- 172.21.15.98:0/66557929 shutdown_connections 2024-09-19T08:08:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe53460740 1 --2- 172.21.15.98:0/66557929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c494780 0x558c0c491940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe53460740 1 --2- 172.21.15.98:0/66557929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c486b40 0x558c0c4913c0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe53460740 1 --2- 172.21.15.98:0/66557929 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x558c0c4a43b0 0x558c0c4a67e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe53460740 1 --2- 172.21.15.98:0/66557929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c4855d0 0x558c0c494110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe327fc640 0 client.16848 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:08:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe53460740 1 -- 172.21.15.98:0/66557929 >> 172.21.15.98:0/66557929 conn(0x558c0c44e310 msgr2=0x558c0c487960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:35.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe53460740 1 -- 172.21.15.98:0/66557929 shutdown_connections 2024-09-19T08:08:35.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe53460740 1 -- 172.21.15.98:0/66557929 wait complete. 2024-09-19T08:08:35.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.652+0000 7efe53460740 20 client.16848 trim_cache size 0 max 16384 2024-09-19T08:08:35.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe53460740 1 Processor -- start 2024-09-19T08:08:35.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe53460740 1 -- start start 2024-09-19T08:08:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe53460740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c4855d0 0x558c0c4a0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe53460740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 0x558c0c490c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe53460740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c494780 0x558c0c4ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe53460740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe4c1ab660 con 0x558c0c4855d0 2024-09-19T08:08:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe53460740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efe4c1ad1a0 con 0x558c0c486b40 2024-09-19T08:08:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe53460740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efe4c1abd30 con 0x558c0c494780 2024-09-19T08:08:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe4ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c4855d0 0x558c0c4a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe4ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c4855d0 0x558c0c4a0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55720/0 (socket says 172.21.15.98:55720) 2024-09-19T08:08:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe4ad9d640 1 -- 172.21.15.98:0/3502704377 learned_addr learned my addr 172.21.15.98:0/3502704377 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe4259c640 1 --2- 172.21.15.98:0/3502704377 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 0x558c0c490c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe4b59e640 1 --2- 172.21.15.98:0/3502704377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c494780 0x558c0c4ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe4ad9d640 1 -- 172.21.15.98:0/3502704377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c494780 msgr2=0x558c0c4ac0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:35.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe4ad9d640 1 --2- 172.21.15.98:0/3502704377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c494780 0x558c0c4ac0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe4ad9d640 1 -- 172.21.15.98:0/3502704377 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 msgr2=0x558c0c490c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:35.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe4ad9d640 1 --2- 172.21.15.98:0/3502704377 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 0x558c0c490c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.665+0000 7efe4ad9d640 1 -- 172.21.15.98:0/3502704377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c0c482960 con 0x558c0c4855d0 2024-09-19T08:08:35.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.666+0000 7efe4ad9d640 1 --2- 172.21.15.98:0/3502704377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c4855d0 0x558c0c4a0720 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7efe3c009a30 tx=0x7efe3c004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf574866f9649c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:35.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.666+0000 7efe40d99640 1 -- 172.21.15.98:0/3502704377 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3c003c40 con 0x558c0c4855d0 2024-09-19T08:08:35.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.666+0000 7efe40d99640 1 -- 172.21.15.98:0/3502704377 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe3c022070 con 0x558c0c4855d0 2024-09-19T08:08:35.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.666+0000 7efe40d99640 1 -- 172.21.15.98:0/3502704377 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3c012740 con 0x558c0c4855d0 2024-09-19T08:08:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.666+0000 7efe53460740 1 -- 172.21.15.98:0/3502704377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c4855d0 msgr2=0x558c0c4a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.666+0000 7efe53460740 1 --2- 172.21.15.98:0/3502704377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c4855d0 0x558c0c4a0720 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7efe3c009a30 tx=0x7efe3c004410 comp rx=0 tx=0).stop 2024-09-19T08:08:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.666+0000 7efe53460740 1 -- 172.21.15.98:0/3502704377 shutdown_connections 2024-09-19T08:08:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 1 --2- 172.21.15.98:0/3502704377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c494780 0x558c0c4ac0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 1 --2- 172.21.15.98:0/3502704377 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 0x558c0c490c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 1 --2- 172.21.15.98:0/3502704377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c4855d0 0x558c0c4a0720 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 1 -- 172.21.15.98:0/3502704377 >> 172.21.15.98:0/3502704377 conn(0x558c0c44e310 msgr2=0x558c0c48fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 1 -- 172.21.15.98:0/3502704377 shutdown_connections 2024-09-19T08:08:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 1 -- 172.21.15.98:0/3502704377 wait complete. 2024-09-19T08:08:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 1 -- start start 2024-09-19T08:08:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 1 Processor -- start 2024-09-19T08:08:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c4855d0 0x558c0c49cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 0x558c0c498210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c494780 0x558c0c498750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe4c1acad0 con 0x558c0c4855d0 2024-09-19T08:08:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efe4c1ac400 con 0x558c0c486b40 2024-09-19T08:08:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efe4c1a9660 con 0x558c0c494780 2024-09-19T08:08:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe53460740 10 client.0 _ll_register_callbacks cb 0x7efe3403b460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe4ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c4855d0 0x558c0c49cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe4ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c4855d0 0x558c0c49cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55732/0 (socket says 172.21.15.98:55732) 2024-09-19T08:08:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe4ad9d640 1 -- 172.21.15.98:0/1629002976 learned_addr learned my addr 172.21.15.98:0/1629002976 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.667+0000 7efe4b59e640 1 --2- 172.21.15.98:0/1629002976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c494780 0x558c0c498750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe4259c640 1 --2- 172.21.15.98:0/1629002976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 0x558c0c498210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe4ad9d640 1 -- 172.21.15.98:0/1629002976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c494780 msgr2=0x558c0c498750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe4ad9d640 1 --2- 172.21.15.98:0/1629002976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c494780 0x558c0c498750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe4ad9d640 1 -- 172.21.15.98:0/1629002976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 msgr2=0x558c0c498210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe4ad9d640 1 --2- 172.21.15.98:0/1629002976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 0x558c0c498210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe4ad9d640 1 -- 172.21.15.98:0/1629002976 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c0c498f80 con 0x558c0c4855d0 2024-09-19T08:08:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe4ad9d640 1 --2- 172.21.15.98:0/1629002976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c4855d0 0x558c0c49cee0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7efe3c0057a0 tx=0x7efe3c009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5a09e1c3184263 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe327fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe327fc640 1 -- 172.21.15.98:0/1629002976 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3c01d3b0 con 0x558c0c4855d0 2024-09-19T08:08:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe327fc640 1 -- 172.21.15.98:0/1629002976 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7efe3c01d550 con 0x558c0c4855d0 2024-09-19T08:08:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe53460740 1 -- 172.21.15.98:0/1629002976 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe3c022070 con 0x558c0c4855d0 2024-09-19T08:08:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe327fc640 1 -- 172.21.15.98:0/1629002976 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efe3c01d750 con 0x558c0c4855d0 2024-09-19T08:08:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe53460740 10 client.16854 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe53460740 1 -- 172.21.15.98:0/1629002976 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558c0c50ef80 con 0x558c0c4855d0 2024-09-19T08:08:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe327fc640 1 -- 172.21.15.98:0/1629002976 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe3c022070 con 0x558c0c4855d0 2024-09-19T08:08:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.668+0000 7efe327fc640 1 -- 172.21.15.98:0/1629002976 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7efe3c038430 con 0x558c0c4855d0 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.669+0000 7efe53460740 10 client.16854 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.669+0000 7efe53460740 10 client.16854 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.669+0000 7efe53460740 20 client.16854 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.669+0000 7efe53460740 1 --2- 172.21.15.98:0/1629002976 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x558c0c50f690 0x558c0c511ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.669+0000 7efe53460740 4 client.16854 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.669+0000 7efe53460740 1 -- 172.21.15.98:0/1629002976 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558c0c3f2d70 con 0x558c0c50f690 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.669+0000 7efe4259c640 1 --2- 172.21.15.98:0/1629002976 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x558c0c50f690 0x558c0c511ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.669+0000 7efe327fc640 10 client.16854 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.669+0000 7efe4259c640 1 --2- 172.21.15.98:0/1629002976 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x558c0c50f690 0x558c0c511ab0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=57239bd3e909e743 server_cookie=106051fc85cc3af0 in_seq=0 out_seq=0 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.669+0000 7efe327fc640 1 -- 172.21.15.98:0/1629002976 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558c0c3f2d70 con 0x558c0c50f690 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.669+0000 7efe327fc640 10 client.16854 handle_command_reply: tid=0 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.669+0000 7efe53460740 1 client.16854 shutdown 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe53460740 1 -- 172.21.15.98:0/1629002976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c4855d0 msgr2=0x558c0c49cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe53460740 1 --2- 172.21.15.98:0/1629002976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c4855d0 0x558c0c49cee0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7efe3c0057a0 tx=0x7efe3c009cd0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe53460740 1 -- 172.21.15.98:0/1629002976 shutdown_connections 2024-09-19T08:08:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe53460740 1 --2- 172.21.15.98:0/1629002976 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x558c0c50f690 0x558c0c511ab0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe53460740 1 --2- 172.21.15.98:0/1629002976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0c494780 0x558c0c498750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe53460740 1 --2- 172.21.15.98:0/1629002976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c0c486b40 0x558c0c498210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe327fc640 0 client.16854 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:08:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe53460740 1 --2- 172.21.15.98:0/1629002976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c0c4855d0 0x558c0c49cee0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe53460740 1 -- 172.21.15.98:0/1629002976 >> 172.21.15.98:0/1629002976 conn(0x558c0c44e310 msgr2=0x558c0c487930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe53460740 1 -- 172.21.15.98:0/1629002976 shutdown_connections 2024-09-19T08:08:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe53460740 1 -- 172.21.15.98:0/1629002976 wait complete. 2024-09-19T08:08:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe53460740 20 client.16854 trim_cache size 0 max 16384 2024-09-19T08:08:35.675 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:08:35.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe51087640 1 -- 172.21.15.98:0/381611986 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7efe380a33b0 msgr2=0x7efe380a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:35.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe51087640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7efe380a33b0 0x7efe380a5870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efe440092f0 tx=0x7efe44009280 comp rx=0 tx=0).stop 2024-09-19T08:08:35.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe51087640 1 -- 172.21.15.98:0/381611986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe4c102790 msgr2=0x7efe4c198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:35.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.670+0000 7efe51087640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe4c102790 0x7efe4c198990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7efe34006010 tx=0x7efe340042f0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.671+0000 7efe51087640 1 -- 172.21.15.98:0/381611986 shutdown_connections 2024-09-19T08:08:35.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.671+0000 7efe51087640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7efe380a33b0 0x7efe380a5870 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.671+0000 7efe51087640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe4c106f10 0x7efe4c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.671+0000 7efe51087640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe4c102790 0x7efe4c198990 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.671+0000 7efe51087640 1 --2- 172.21.15.98:0/381611986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe4c101de0 0x7efe4c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:35.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.671+0000 7efe51087640 1 -- 172.21.15.98:0/381611986 >> 172.21.15.98:0/381611986 conn(0x7efe4c0fd8c0 msgr2=0x7efe4c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:35.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.671+0000 7efe51087640 1 -- 172.21.15.98:0/381611986 shutdown_connections 2024-09-19T08:08:35.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:35.671+0000 7efe51087640 1 -- 172.21.15.98:0/381611986 wait complete. 2024-09-19T08:08:36.300 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:36 smithi098 ceph-mon[31333]: pgmap v170: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:36.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:08:36.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:36 smithi162 ceph-mon[35260]: pgmap v170: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:36.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:36 smithi139 ceph-mon[34874]: pgmap v170: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:08:37.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.037+0000 7fcc27986640 1 -- 172.21.15.98:0/107644816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc20102ef0 msgr2=0x7fcc2010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:37.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.037+0000 7fcc27986640 1 --2- 172.21.15.98:0/107644816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc20102ef0 0x7fcc2010da40 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009ca0 tx=0x7fcc1802a950 comp rx=0 tx=0).stop 2024-09-19T08:08:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.037+0000 7fcc27986640 1 -- 172.21.15.98:0/107644816 shutdown_connections 2024-09-19T08:08:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.037+0000 7fcc27986640 1 --2- 172.21.15.98:0/107644816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc20102ef0 0x7fcc2010da40 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.037+0000 7fcc27986640 1 --2- 172.21.15.98:0/107644816 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc20102570 0x7fcc201029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.037+0000 7fcc27986640 1 --2- 172.21.15.98:0/107644816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc20100e20 0x7fcc20101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.037+0000 7fcc27986640 1 -- 172.21.15.98:0/107644816 >> 172.21.15.98:0/107644816 conn(0x7fcc200fc7d0 msgr2=0x7fcc200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.037+0000 7fcc27986640 1 -- 172.21.15.98:0/107644816 shutdown_connections 2024-09-19T08:08:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.037+0000 7fcc27986640 1 -- 172.21.15.98:0/107644816 wait complete. 2024-09-19T08:08:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.038+0000 7fcc27986640 1 Processor -- start 2024-09-19T08:08:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.038+0000 7fcc27986640 1 -- start start 2024-09-19T08:08:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.038+0000 7fcc27986640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc20100e20 0x7fcc201972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.038+0000 7fcc27986640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc20102570 0x7fcc20197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.038+0000 7fcc27986640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc20102ef0 0x7fcc2019bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.038+0000 7fcc27986640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc20110f80 con 0x7fcc20102ef0 2024-09-19T08:08:37.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.038+0000 7fcc27986640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcc20110e00 con 0x7fcc20100e20 2024-09-19T08:08:37.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.038+0000 7fcc27986640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcc20111100 con 0x7fcc20102570 2024-09-19T08:08:37.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc25f23640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc20102ef0 0x7fcc2019bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc25f23640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc20102ef0 0x7fcc2019bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53196/0 (socket says 172.21.15.98:53196) 2024-09-19T08:08:37.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc25f23640 1 -- 172.21.15.98:0/4111118856 learned_addr learned my addr 172.21.15.98:0/4111118856 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:37.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc25722640 1 --2- 172.21.15.98:0/4111118856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc20100e20 0x7fcc201972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc24f21640 1 --2- 172.21.15.98:0/4111118856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc20102570 0x7fcc20197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc25f23640 1 -- 172.21.15.98:0/4111118856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc20102570 msgr2=0x7fcc20197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:37.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc25f23640 1 --2- 172.21.15.98:0/4111118856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc20102570 0x7fcc20197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc25f23640 1 -- 172.21.15.98:0/4111118856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc20100e20 msgr2=0x7fcc201972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:37.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc25f23640 1 --2- 172.21.15.98:0/4111118856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc20100e20 0x7fcc201972e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc25f23640 1 -- 172.21.15.98:0/4111118856 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2019c290 con 0x7fcc20102ef0 2024-09-19T08:08:37.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc25f23640 1 --2- 172.21.15.98:0/4111118856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc20102ef0 0x7fcc2019bbb0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fcc18004af0 tx=0x7fcc1802f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:37.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc167fc640 1 -- 172.21.15.98:0/4111118856 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18034040 con 0x7fcc20102ef0 2024-09-19T08:08:37.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc27986640 1 -- 172.21.15.98:0/4111118856 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2019c520 con 0x7fcc20102ef0 2024-09-19T08:08:37.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc167fc640 1 -- 172.21.15.98:0/4111118856 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc18030920 con 0x7fcc20102ef0 2024-09-19T08:08:37.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc167fc640 1 -- 172.21.15.98:0/4111118856 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18004820 con 0x7fcc20102ef0 2024-09-19T08:08:37.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.039+0000 7fcc27986640 1 -- 172.21.15.98:0/4111118856 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc201a3e60 con 0x7fcc20102ef0 2024-09-19T08:08:37.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc167fc640 1 -- 172.21.15.98:0/4111118856 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fcc18037020 con 0x7fcc20102ef0 2024-09-19T08:08:37.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc29d5f740 1 Processor -- start 2024-09-19T08:08:37.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc29d5f740 1 -- start start 2024-09-19T08:08:37.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc29d5f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd102d50 0x55cafd1012a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc29d5f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd1017e0 0x55cafd101c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc29d5f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd10c580 0x55cafd10e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc29d5f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cafd0652f0 con 0x55cafd10c580 2024-09-19T08:08:37.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc29d5f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cafd00ccf0 con 0x55cafd1017e0 2024-09-19T08:08:37.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc29d5f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cafd0612b0 con 0x55cafd102d50 2024-09-19T08:08:37.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc24f21640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd10c580 0x55cafd10e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc24f21640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd10c580 0x55cafd10e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53206/0 (socket says 172.21.15.98:53206) 2024-09-19T08:08:37.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc24f21640 1 -- 172.21.15.98:0/1346792235 learned_addr learned my addr 172.21.15.98:0/1346792235 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:37.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc24f21640 1 --2- 172.21.15.98:0/1346792235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd102d50 0x55cafd1012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc25722640 1 --2- 172.21.15.98:0/1346792235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd1017e0 0x55cafd101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.042+0000 7fcc167fc640 1 --2- 172.21.15.98:0/4111118856 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcc0c0a33b0 0x7fcc0c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.043+0000 7fcc25f23640 1 --2- 172.21.15.98:0/4111118856 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcc0c0a33b0 0x7fcc0c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.043+0000 7fcc24f21640 1 -- 172.21.15.98:0/1346792235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd102d50 msgr2=0x55cafd1012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:37.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.043+0000 7fcc24f21640 1 --2- 172.21.15.98:0/1346792235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd102d50 0x55cafd1012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.043+0000 7fcc24f21640 1 -- 172.21.15.98:0/1346792235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd1017e0 msgr2=0x55cafd101c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:37.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.043+0000 7fcc24f21640 1 --2- 172.21.15.98:0/1346792235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd1017e0 0x55cafd101c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.043+0000 7fcc24f21640 1 -- 172.21.15.98:0/1346792235 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cafcff13a0 con 0x55cafd10c580 2024-09-19T08:08:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.043+0000 7fcc167fc640 1 -- 172.21.15.98:0/4111118856 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fcc180f0a80 con 0x7fcc20102ef0 2024-09-19T08:08:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.043+0000 7fcc24f21640 1 --2- 172.21.15.98:0/1346792235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd10c580 0x55cafd10e970 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fcc10013930 tx=0x7fcc1000b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70101bcd22e3ab73 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.043+0000 7fcc0b7fe640 1 -- 172.21.15.98:0/1346792235 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1000e950 con 0x55cafd10c580 2024-09-19T08:08:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.043+0000 7fcc0b7fe640 1 -- 172.21.15.98:0/1346792235 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc10010430 con 0x55cafd10c580 2024-09-19T08:08:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.043+0000 7fcc0b7fe640 1 -- 172.21.15.98:0/1346792235 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1000f620 con 0x55cafd10c580 2024-09-19T08:08:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.043+0000 7fcc25f23640 1 --2- 172.21.15.98:0/4111118856 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcc0c0a33b0 0x7fcc0c0a5870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcc180bcba0 tx=0x7fcc1808ec20 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.044+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1346792235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd10c580 msgr2=0x55cafd10e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.044+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/1346792235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd10c580 0x55cafd10e970 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fcc10013930 tx=0x7fcc1000b460 comp rx=0 tx=0).stop 2024-09-19T08:08:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.044+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1346792235 shutdown_connections 2024-09-19T08:08:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.044+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/1346792235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd10c580 0x55cafd10e970 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.044+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/1346792235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd1017e0 0x55cafd101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.044+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/1346792235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd102d50 0x55cafd1012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.044+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1346792235 >> 172.21.15.98:0/1346792235 conn(0x55cafd0c8310 msgr2=0x55cafd0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.044+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1346792235 shutdown_connections 2024-09-19T08:08:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.044+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1346792235 wait complete. 2024-09-19T08:08:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.044+0000 7fcc29d5f740 1 -- start start 2024-09-19T08:08:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.044+0000 7fcc29d5f740 1 Processor -- start 2024-09-19T08:08:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc29d5f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd1017e0 0x55cafd107710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc29d5f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd102d50 0x55cafd107c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc29d5f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd10c580 0x55cafd1081b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc29d5f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cafd0fb870 con 0x55cafd102d50 2024-09-19T08:08:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc29d5f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cafd020060 con 0x55cafd1017e0 2024-09-19T08:08:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc29d5f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cafd0fb9f0 con 0x55cafd10c580 2024-09-19T08:08:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc29d5f740 10 client.0 _ll_register_callbacks cb 0x55cafcd738c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc25722640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd102d50 0x55cafd107c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc25722640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd102d50 0x55cafd107c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53210/0 (socket says 172.21.15.98:53210) 2024-09-19T08:08:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc25722640 1 -- 172.21.15.98:0/2412294094 learned_addr learned my addr 172.21.15.98:0/2412294094 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc24f21640 1 --2- 172.21.15.98:0/2412294094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd1017e0 0x55cafd107710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc24f21640 1 --2- 172.21.15.98:0/2412294094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd10c580 0x55cafd1081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc25722640 1 -- 172.21.15.98:0/2412294094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd10c580 msgr2=0x55cafd1081b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc25722640 1 --2- 172.21.15.98:0/2412294094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd10c580 0x55cafd1081b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc25722640 1 -- 172.21.15.98:0/2412294094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd1017e0 msgr2=0x55cafd107710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc25722640 1 --2- 172.21.15.98:0/2412294094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd1017e0 0x55cafd107710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc25722640 1 -- 172.21.15.98:0/2412294094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cafcdd63c0 con 0x55cafd102d50 2024-09-19T08:08:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc24f21640 1 --2- 172.21.15.98:0/2412294094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd10c580 0x55cafd1081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:08:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc25722640 1 --2- 172.21.15.98:0/2412294094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd102d50 0x55cafd107c50 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c012740 tx=0x7fcc1c012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e7d513a10cbe258 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc08ff9640 1 -- 172.21.15.98:0/2412294094 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c00f920 con 0x55cafd102d50 2024-09-19T08:08:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc29d5f740 1 -- 172.21.15.98:0/2412294094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc10010430 con 0x55cafd102d50 2024-09-19T08:08:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.045+0000 7fcc08ff9640 1 -- 172.21.15.98:0/2412294094 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fcc1c00fac0 con 0x55cafd102d50 2024-09-19T08:08:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.046+0000 7fcc08ff9640 1 -- 172.21.15.98:0/2412294094 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcc1c00fd00 con 0x55cafd102d50 2024-09-19T08:08:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.046+0000 7fcc29d5f740 10 client.16866 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.046+0000 7fcc29d5f740 1 -- 172.21.15.98:0/2412294094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cafd118b90 con 0x55cafd102d50 2024-09-19T08:08:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.046+0000 7fcc08ff9640 1 -- 172.21.15.98:0/2412294094 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc10010430 con 0x55cafd102d50 2024-09-19T08:08:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.046+0000 7fcc08ff9640 1 -- 172.21.15.98:0/2412294094 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcc1c01c420 con 0x55cafd102d50 2024-09-19T08:08:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.046+0000 7fcc29d5f740 10 client.16866 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.046+0000 7fcc29d5f740 10 client.16866 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:08:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.046+0000 7fcc29d5f740 20 client.16866 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.046+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/2412294094 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55cafd11a0f0 0x55cafd11c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.046+0000 7fcc29d5f740 4 client.16866 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.046+0000 7fcc29d5f740 1 -- 172.21.15.98:0/2412294094 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cafd072860 con 0x55cafd11a0f0 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.046+0000 7fcc24f21640 1 --2- 172.21.15.98:0/2412294094 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55cafd11a0f0 0x55cafd11c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.047+0000 7fcc24f21640 1 --2- 172.21.15.98:0/2412294094 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55cafd11a0f0 0x55cafd11c510 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=75aaf379fe039035 server_cookie=31a7a4d723d06161 in_seq=0 out_seq=0 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.047+0000 7fcc08ff9640 10 client.16866 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.047+0000 7fcc08ff9640 1 -- 172.21.15.98:0/2412294094 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cafd072860 con 0x55cafd11a0f0 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.047+0000 7fcc08ff9640 10 client.16866 handle_command_reply: tid=0 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.047+0000 7fcc29d5f740 1 client.16866 shutdown 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.047+0000 7fcc29d5f740 1 -- 172.21.15.98:0/2412294094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd102d50 msgr2=0x55cafd107c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.047+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/2412294094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd102d50 0x55cafd107c50 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c012740 tx=0x7fcc1c012c00 comp rx=0 tx=0).stop 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.047+0000 7fcc29d5f740 1 -- 172.21.15.98:0/2412294094 shutdown_connections 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.047+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/2412294094 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55cafd11a0f0 0x55cafd11c510 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.047+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/2412294094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd10c580 0x55cafd1081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.047+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/2412294094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd102d50 0x55cafd107c50 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.047+0000 7fcc08ff9640 0 client.16866 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:08:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.047+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/2412294094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd1017e0 0x55cafd107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.047+0000 7fcc29d5f740 1 -- 172.21.15.98:0/2412294094 >> 172.21.15.98:0/2412294094 conn(0x55cafd0c8310 msgr2=0x55cafd103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.048+0000 7fcc29d5f740 1 -- 172.21.15.98:0/2412294094 shutdown_connections 2024-09-19T08:08:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.048+0000 7fcc29d5f740 1 -- 172.21.15.98:0/2412294094 wait complete. 2024-09-19T08:08:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.048+0000 7fcc29d5f740 20 client.16866 trim_cache size 0 max 16384 2024-09-19T08:08:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc29d5f740 1 Processor -- start 2024-09-19T08:08:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc29d5f740 1 -- start start 2024-09-19T08:08:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc29d5f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd1017e0 0x55cafd0fcc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc29d5f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd102d50 0x55cafd115070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc29d5f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd10c580 0x55cafd1155b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc29d5f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc201ab430 con 0x55cafd10c580 2024-09-19T08:08:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc29d5f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcc201acf70 con 0x55cafd102d50 2024-09-19T08:08:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc29d5f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcc201abb00 con 0x55cafd1017e0 2024-09-19T08:08:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc24f21640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd10c580 0x55cafd1155b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc24f21640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd10c580 0x55cafd1155b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53222/0 (socket says 172.21.15.98:53222) 2024-09-19T08:08:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc24f21640 1 -- 172.21.15.98:0/1320348385 learned_addr learned my addr 172.21.15.98:0/1320348385 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc25722640 1 --2- 172.21.15.98:0/1320348385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd102d50 0x55cafd115070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc24f21640 1 --2- 172.21.15.98:0/1320348385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd1017e0 0x55cafd0fcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc24f21640 1 -- 172.21.15.98:0/1320348385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd1017e0 msgr2=0x55cafd0fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc24f21640 1 --2- 172.21.15.98:0/1320348385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd1017e0 0x55cafd0fcc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc24f21640 1 -- 172.21.15.98:0/1320348385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd102d50 msgr2=0x55cafd115070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc24f21640 1 --2- 172.21.15.98:0/1320348385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd102d50 0x55cafd115070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc24f21640 1 -- 172.21.15.98:0/1320348385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cafd101610 con 0x55cafd10c580 2024-09-19T08:08:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.061+0000 7fcc24f21640 1 --2- 172.21.15.98:0/1320348385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd10c580 0x55cafd1155b0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fcc10005f10 tx=0x7fcc10010de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad94f0a464347c4b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.062+0000 7fcc0affd640 1 -- 172.21.15.98:0/1320348385 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1000bc70 con 0x55cafd10c580 2024-09-19T08:08:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.062+0000 7fcc0affd640 1 -- 172.21.15.98:0/1320348385 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc10005040 con 0x55cafd10c580 2024-09-19T08:08:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.062+0000 7fcc0affd640 1 -- 172.21.15.98:0/1320348385 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1000e720 con 0x55cafd10c580 2024-09-19T08:08:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.062+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1320348385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd10c580 msgr2=0x55cafd1155b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.062+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/1320348385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd10c580 0x55cafd1155b0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fcc10005f10 tx=0x7fcc10010de0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.062+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1320348385 shutdown_connections 2024-09-19T08:08:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.062+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/1320348385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd10c580 0x55cafd1155b0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.062+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/1320348385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd102d50 0x55cafd115070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.062+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/1320348385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd1017e0 0x55cafd0fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.062+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1320348385 >> 172.21.15.98:0/1320348385 conn(0x55cafd0c8310 msgr2=0x55cafd10b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.062+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1320348385 shutdown_connections 2024-09-19T08:08:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.062+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1320348385 wait complete. 2024-09-19T08:08:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.063+0000 7fcc29d5f740 1 -- start start 2024-09-19T08:08:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.063+0000 7fcc29d5f740 1 Processor -- start 2024-09-19T08:08:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.063+0000 7fcc29d5f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd1017e0 0x55cafd122450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.063+0000 7fcc29d5f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd102d50 0x55cafd120db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.063+0000 7fcc29d5f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd10c580 0x55cafd1212f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.063+0000 7fcc29d5f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc201ac8a0 con 0x55cafd1017e0 2024-09-19T08:08:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.063+0000 7fcc29d5f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcc201ac1d0 con 0x55cafd102d50 2024-09-19T08:08:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.063+0000 7fcc29d5f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcc201a9430 con 0x55cafd10c580 2024-09-19T08:08:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.063+0000 7fcc29d5f740 10 client.0 _ll_register_callbacks cb 0x7fcc1c009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.063+0000 7fcc24f21640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd1017e0 0x55cafd122450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.063+0000 7fcc24f21640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd1017e0 0x55cafd122450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53226/0 (socket says 172.21.15.98:53226) 2024-09-19T08:08:37.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.063+0000 7fcc24f21640 1 -- 172.21.15.98:0/1220585737 learned_addr learned my addr 172.21.15.98:0/1220585737 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:37.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc25722640 1 --2- 172.21.15.98:0/1220585737 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd102d50 0x55cafd120db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc24f21640 1 -- 172.21.15.98:0/1220585737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd10c580 msgr2=0x55cafd1212f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:37.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc24f21640 1 --2- 172.21.15.98:0/1220585737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd10c580 0x55cafd1212f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc24f21640 1 -- 172.21.15.98:0/1220585737 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd102d50 msgr2=0x55cafd120db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:37.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc24f21640 1 --2- 172.21.15.98:0/1220585737 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd102d50 0x55cafd120db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc24f21640 1 -- 172.21.15.98:0/1220585737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cafd121af0 con 0x55cafd1017e0 2024-09-19T08:08:37.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc24f21640 1 --2- 172.21.15.98:0/1220585737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd1017e0 0x55cafd122450 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fcc10000c00 tx=0x7fcc1000ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcddf63d6d30eb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:37.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:37.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc08ff9640 1 -- 172.21.15.98:0/1220585737 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc100052b0 con 0x55cafd1017e0 2024-09-19T08:08:37.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc08ff9640 1 -- 172.21.15.98:0/1220585737 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fcc1002de80 con 0x55cafd1017e0 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1220585737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc10005040 con 0x55cafd1017e0 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc08ff9640 1 -- 172.21.15.98:0/1220585737 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcc10005450 con 0x55cafd1017e0 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc29d5f740 10 client.16872 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1220585737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cafd184d90 con 0x55cafd1017e0 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc08ff9640 1 -- 172.21.15.98:0/1220585737 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc10005040 con 0x55cafd1017e0 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.064+0000 7fcc08ff9640 1 -- 172.21.15.98:0/1220585737 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcc1000eb70 con 0x55cafd1017e0 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc29d5f740 10 client.16872 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc29d5f740 10 client.16872 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc29d5f740 20 client.16872 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/1220585737 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55cafd1851b0 0x55cafd1875d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc29d5f740 4 client.16872 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1220585737 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cafd06cd70 con 0x55cafd1851b0 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc25722640 1 --2- 172.21.15.98:0/1220585737 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55cafd1851b0 0x55cafd1875d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc08ff9640 10 client.16872 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:08:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc25722640 1 --2- 172.21.15.98:0/1220585737 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55cafd1851b0 0x55cafd1875d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=26b150c1eb954911 server_cookie=75c8d771d24f5e04 in_seq=0 out_seq=0 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc08ff9640 1 -- 172.21.15.98:0/1220585737 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cafd06cd70 con 0x55cafd1851b0 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc08ff9640 10 client.16872 handle_command_reply: tid=0 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc29d5f740 1 client.16872 shutdown 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1220585737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd1017e0 msgr2=0x55cafd122450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/1220585737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd1017e0 0x55cafd122450 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fcc10000c00 tx=0x7fcc1000ba30 comp rx=0 tx=0).stop 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1220585737 shutdown_connections 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/1220585737 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55cafd1851b0 0x55cafd1875d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/1220585737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cafd10c580 0x55cafd1212f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/1220585737 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cafd102d50 0x55cafd120db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc08ff9640 0 client.16872 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.065+0000 7fcc29d5f740 1 --2- 172.21.15.98:0/1220585737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cafd1017e0 0x55cafd122450 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1220585737 >> 172.21.15.98:0/1220585737 conn(0x55cafd0c8310 msgr2=0x55cafd1032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1220585737 shutdown_connections 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc29d5f740 1 -- 172.21.15.98:0/1220585737 wait complete. 2024-09-19T08:08:37.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc29d5f740 20 client.16872 trim_cache size 0 max 16384 2024-09-19T08:08:37.075 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:08:37.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc27986640 1 -- 172.21.15.98:0/4111118856 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcc0c0a33b0 msgr2=0x7fcc0c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:37.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc27986640 1 --2- 172.21.15.98:0/4111118856 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcc0c0a33b0 0x7fcc0c0a5870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcc180bcba0 tx=0x7fcc1808ec20 comp rx=0 tx=0).stop 2024-09-19T08:08:37.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc27986640 1 -- 172.21.15.98:0/4111118856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc20102ef0 msgr2=0x7fcc2019bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:37.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc27986640 1 --2- 172.21.15.98:0/4111118856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc20102ef0 0x7fcc2019bbb0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fcc18004af0 tx=0x7fcc1802f5b0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc27986640 1 -- 172.21.15.98:0/4111118856 shutdown_connections 2024-09-19T08:08:37.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc27986640 1 --2- 172.21.15.98:0/4111118856 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcc0c0a33b0 0x7fcc0c0a5870 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc27986640 1 --2- 172.21.15.98:0/4111118856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc20102ef0 0x7fcc2019bbb0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc27986640 1 --2- 172.21.15.98:0/4111118856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc20102570 0x7fcc20197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc27986640 1 --2- 172.21.15.98:0/4111118856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc20100e20 0x7fcc201972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:37.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc27986640 1 -- 172.21.15.98:0/4111118856 >> 172.21.15.98:0/4111118856 conn(0x7fcc200fc7d0 msgr2=0x7fcc20106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:37.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc27986640 1 -- 172.21.15.98:0/4111118856 shutdown_connections 2024-09-19T08:08:37.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:37.066+0000 7fcc27986640 1 -- 172.21.15.98:0/4111118856 wait complete. 2024-09-19T08:08:37.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:08:38.067 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:08:38.287 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:38 smithi098 ceph-mon[31333]: pgmap v171: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:08:38.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:38 smithi162 ceph-mon[35260]: pgmap v171: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:08:38.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:38 smithi139 ceph-mon[34874]: pgmap v171: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:08:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.510+0000 7f4d58ae2640 1 -- 172.21.15.98:0/1177090393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54101de0 msgr2=0x7f4d541021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.510+0000 7f4d58ae2640 1 --2- 172.21.15.98:0/1177090393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54101de0 0x7f4d541021c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009900 tx=0x7f4d4402a410 comp rx=0 tx=0).stop 2024-09-19T08:08:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.510+0000 7f4d58ae2640 1 -- 172.21.15.98:0/1177090393 shutdown_connections 2024-09-19T08:08:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.510+0000 7f4d58ae2640 1 --2- 172.21.15.98:0/1177090393 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d54106f10 0x7f4d5410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.510+0000 7f4d58ae2640 1 --2- 172.21.15.98:0/1177090393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d54102790 0x7f4d541067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.510+0000 7f4d58ae2640 1 --2- 172.21.15.98:0/1177090393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54101de0 0x7f4d541021c0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.510+0000 7f4d58ae2640 1 -- 172.21.15.98:0/1177090393 >> 172.21.15.98:0/1177090393 conn(0x7f4d540fd8c0 msgr2=0x7f4d540ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.510+0000 7f4d58ae2640 1 -- 172.21.15.98:0/1177090393 shutdown_connections 2024-09-19T08:08:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.510+0000 7f4d58ae2640 1 -- 172.21.15.98:0/1177090393 wait complete. 2024-09-19T08:08:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.511+0000 7f4d58ae2640 1 Processor -- start 2024-09-19T08:08:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.511+0000 7f4d58ae2640 1 -- start start 2024-09-19T08:08:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.511+0000 7f4d58ae2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d54101de0 0x7f4d54198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.511+0000 7f4d58ae2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54102790 0x7f4d541988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.511+0000 7f4d58ae2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d54106f10 0x7f4d5419cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.511+0000 7f4d58ae2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d54111f30 con 0x7f4d54102790 2024-09-19T08:08:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.511+0000 7f4d58ae2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4d54111db0 con 0x7f4d54106f10 2024-09-19T08:08:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.511+0000 7f4d58ae2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541120b0 con 0x7f4d54101de0 2024-09-19T08:08:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d51d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54102790 0x7f4d541988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d51d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54102790 0x7f4d541988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53238/0 (socket says 172.21.15.98:53238) 2024-09-19T08:08:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d51d9b640 1 -- 172.21.15.98:0/786421461 learned_addr learned my addr 172.21.15.98:0/786421461 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d52d9d640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d54106f10 0x7f4d5419cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d5259c640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d54101de0 0x7f4d54198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d51d9b640 1 -- 172.21.15.98:0/786421461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d54101de0 msgr2=0x7f4d54198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d51d9b640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d54101de0 0x7f4d54198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d51d9b640 1 -- 172.21.15.98:0/786421461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d54106f10 msgr2=0x7f4d5419cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d51d9b640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d54106f10 0x7f4d5419cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d51d9b640 1 -- 172.21.15.98:0/786421461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5419d340 con 0x7f4d54102790 2024-09-19T08:08:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d5259c640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d54101de0 0x7f4d54198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:08:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d51d9b640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54102790 0x7f4d541988d0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f4d4800c920 tx=0x7f4d4800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d437fe640 1 -- 172.21.15.98:0/786421461 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d48007c70 con 0x7f4d54102790 2024-09-19T08:08:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d437fe640 1 -- 172.21.15.98:0/786421461 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d48007e10 con 0x7f4d54102790 2024-09-19T08:08:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d58ae2640 1 -- 172.21.15.98:0/786421461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5419d630 con 0x7f4d54102790 2024-09-19T08:08:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d437fe640 1 -- 172.21.15.98:0/786421461 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d48018890 con 0x7f4d54102790 2024-09-19T08:08:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.512+0000 7f4d58ae2640 1 -- 172.21.15.98:0/786421461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d541a4f10 con 0x7f4d54102790 2024-09-19T08:08:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.514+0000 7f4d437fe640 1 -- 172.21.15.98:0/786421461 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f4d48018a70 con 0x7f4d54102790 2024-09-19T08:08:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.515+0000 7f4d5aebb740 1 Processor -- start 2024-09-19T08:08:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.515+0000 7f4d5aebb740 1 -- start start 2024-09-19T08:08:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.515+0000 7f4d5aebb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 0x560a558a0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.515+0000 7f4d5aebb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558a05d0 0x560a558a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.515+0000 7f4d5aebb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558ab370 0x560a558ad760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.515+0000 7f4d5aebb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560a558062f0 con 0x560a558a05d0 2024-09-19T08:08:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.515+0000 7f4d5aebb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560a557adcf0 con 0x560a558a1b40 2024-09-19T08:08:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.515+0000 7f4d5aebb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560a558022b0 con 0x560a558ab370 2024-09-19T08:08:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.515+0000 7f4d437fe640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4d3c0a33b0 0x7f4d3c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.515+0000 7f4d5259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558a05d0 0x560a558a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.515+0000 7f4d5259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558a05d0 0x560a558a0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53246/0 (socket says 172.21.15.98:53246) 2024-09-19T08:08:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.515+0000 7f4d5259c640 1 -- 172.21.15.98:0/3354973431 learned_addr learned my addr 172.21.15.98:0/3354973431 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.515+0000 7f4d52d9d640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4d3c0a33b0 0x7f4d3c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.516+0000 7f4d51d9b640 1 --2- 172.21.15.98:0/3354973431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558ab370 0x560a558ad760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.516+0000 7f4d52d9d640 1 --2- 172.21.15.98:0/3354973431 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 0x560a558a0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.516+0000 7f4d437fe640 1 -- 172.21.15.98:0/786421461 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f4d480cfe40 con 0x7f4d54102790 2024-09-19T08:08:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.516+0000 7f4d5259c640 1 -- 172.21.15.98:0/3354973431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558ab370 msgr2=0x560a558ad760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.516+0000 7f4d5259c640 1 --2- 172.21.15.98:0/3354973431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558ab370 0x560a558ad760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.516+0000 7f4d5259c640 1 -- 172.21.15.98:0/3354973431 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 msgr2=0x560a558a0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.516+0000 7f4d5259c640 1 --2- 172.21.15.98:0/3354973431 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 0x560a558a0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.516+0000 7f4d5259c640 1 -- 172.21.15.98:0/3354973431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a557923a0 con 0x560a558a05d0 2024-09-19T08:08:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.516+0000 7f4d51d9b640 1 --2- 172.21.15.98:0/3354973431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558ab370 0x560a558ad760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:08:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.516+0000 7f4d52d9d640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4d3c0a33b0 0x7f4d3c0a5870 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c0098b0 tx=0x7f4d4c009d90 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.516+0000 7f4d5259c640 1 --2- 172.21.15.98:0/3354973431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558a05d0 0x560a558a0a10 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f4d44033000 tx=0x7f4d44032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306bfeb5c28f7794 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.516+0000 7f4d40ff9640 1 -- 172.21.15.98:0/3354973431 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4403f070 con 0x560a558a05d0 2024-09-19T08:08:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.516+0000 7f4d40ff9640 1 -- 172.21.15.98:0/3354973431 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d440317b0 con 0x560a558a05d0 2024-09-19T08:08:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.516+0000 7f4d40ff9640 1 -- 172.21.15.98:0/3354973431 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d44035070 con 0x560a558a05d0 2024-09-19T08:08:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.517+0000 7f4d5aebb740 1 -- 172.21.15.98:0/3354973431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558a05d0 msgr2=0x560a558a0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.517+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/3354973431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558a05d0 0x560a558a0a10 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f4d44033000 tx=0x7f4d44032d60 comp rx=0 tx=0).stop 2024-09-19T08:08:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.517+0000 7f4d5aebb740 1 -- 172.21.15.98:0/3354973431 shutdown_connections 2024-09-19T08:08:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.517+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/3354973431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558ab370 0x560a558ad760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.517+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/3354973431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558a05d0 0x560a558a0a10 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.517+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/3354973431 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 0x560a558a0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.517+0000 7f4d5aebb740 1 -- 172.21.15.98:0/3354973431 >> 172.21.15.98:0/3354973431 conn(0x560a55869310 msgr2=0x560a5589cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.517+0000 7f4d5aebb740 1 -- 172.21.15.98:0/3354973431 shutdown_connections 2024-09-19T08:08:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.517+0000 7f4d5aebb740 1 -- 172.21.15.98:0/3354973431 wait complete. 2024-09-19T08:08:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.517+0000 7f4d5aebb740 1 -- start start 2024-09-19T08:08:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.517+0000 7f4d5aebb740 1 Processor -- start 2024-09-19T08:08:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d5aebb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 0x560a558aad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d5aebb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 0x560a558a7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d5aebb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558ab370 0x560a558a85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d5aebb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560a558ae010 con 0x560a558ab370 2024-09-19T08:08:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d5aebb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560a557c1060 con 0x560a558a1b40 2024-09-19T08:08:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d5aebb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560a558ae190 con 0x560a558a05d0 2024-09-19T08:08:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d5aebb740 10 client.0 _ll_register_callbacks cb 0x560a555148c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d52d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558ab370 0x560a558a85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d52d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558ab370 0x560a558a85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53254/0 (socket says 172.21.15.98:53254) 2024-09-19T08:08:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d52d9d640 1 -- 172.21.15.98:0/2045373487 learned_addr learned my addr 172.21.15.98:0/2045373487 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d5259c640 1 --2- 172.21.15.98:0/2045373487 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 0x560a558aad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d51d9b640 1 --2- 172.21.15.98:0/2045373487 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 0x560a558a7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d52d9d640 1 -- 172.21.15.98:0/2045373487 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 msgr2=0x560a558aad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d52d9d640 1 --2- 172.21.15.98:0/2045373487 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 0x560a558aad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d52d9d640 1 -- 172.21.15.98:0/2045373487 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 msgr2=0x560a558a7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d52d9d640 1 --2- 172.21.15.98:0/2045373487 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 0x560a558a7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d52d9d640 1 -- 172.21.15.98:0/2045373487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a555773c0 con 0x560a558ab370 2024-09-19T08:08:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d52d9d640 1 --2- 172.21.15.98:0/2045373487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558ab370 0x560a558a85e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c0092f0 tx=0x7f4d4c009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69dde21ea4daad27 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.518+0000 7f4d39ffb640 1 -- 172.21.15.98:0/2045373487 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c004b20 con 0x560a558ab370 2024-09-19T08:08:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.519+0000 7f4d39ffb640 1 -- 172.21.15.98:0/2045373487 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f4d4c004cc0 con 0x560a558ab370 2024-09-19T08:08:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.519+0000 7f4d5aebb740 1 -- 172.21.15.98:0/2045373487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d440317b0 con 0x560a558ab370 2024-09-19T08:08:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.519+0000 7f4d39ffb640 1 -- 172.21.15.98:0/2045373487 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4d4c004500 con 0x560a558ab370 2024-09-19T08:08:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.519+0000 7f4d5aebb740 10 client.16884 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.519+0000 7f4d5aebb740 1 -- 172.21.15.98:0/2045373487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560a558b99f0 con 0x560a558ab370 2024-09-19T08:08:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.519+0000 7f4d39ffb640 1 -- 172.21.15.98:0/2045373487 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d440317b0 con 0x560a558ab370 2024-09-19T08:08:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.519+0000 7f4d39ffb640 1 -- 172.21.15.98:0/2045373487 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4d4c01f070 con 0x560a558ab370 2024-09-19T08:08:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.519+0000 7f4d5aebb740 10 client.16884 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.519+0000 7f4d5aebb740 10 client.16884 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:08:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.519+0000 7f4d5aebb740 20 client.16884 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.519+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/2045373487 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x560a558bae00 0x560a558bd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.519+0000 7f4d5aebb740 4 client.16884 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:08:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.519+0000 7f4d5aebb740 1 -- 172.21.15.98:0/2045373487 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a55813860 con 0x560a558bae00 2024-09-19T08:08:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.519+0000 7f4d5259c640 1 --2- 172.21.15.98:0/2045373487 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x560a558bae00 0x560a558bd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.520+0000 7f4d39ffb640 10 client.16884 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:08:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.520+0000 7f4d5259c640 1 --2- 172.21.15.98:0/2045373487 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x560a558bae00 0x560a558bd2a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=122b0c82c7d8fb28 server_cookie=82fdf82dd36b578f in_seq=0 out_seq=0 2024-09-19T08:08:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.520+0000 7f4d39ffb640 1 -- 172.21.15.98:0/2045373487 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560a55813860 con 0x560a558bae00 2024-09-19T08:08:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.520+0000 7f4d39ffb640 10 client.16884 handle_command_reply: tid=0 2024-09-19T08:08:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.521+0000 7f4d5aebb740 1 client.16884 shutdown 2024-09-19T08:08:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.521+0000 7f4d5aebb740 1 -- 172.21.15.98:0/2045373487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558ab370 msgr2=0x560a558a85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.521+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/2045373487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558ab370 0x560a558a85e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c0092f0 tx=0x7f4d4c009600 comp rx=0 tx=0).stop 2024-09-19T08:08:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.521+0000 7f4d5aebb740 1 -- 172.21.15.98:0/2045373487 shutdown_connections 2024-09-19T08:08:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.521+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/2045373487 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x560a558bae00 0x560a558bd2a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.521+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/2045373487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558ab370 0x560a558a85e0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.521+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/2045373487 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 0x560a558a7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.521+0000 7f4d39ffb640 0 client.16884 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:08:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.521+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/2045373487 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 0x560a558aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.521+0000 7f4d5aebb740 1 -- 172.21.15.98:0/2045373487 >> 172.21.15.98:0/2045373487 conn(0x560a55869310 msgr2=0x560a558a2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.521+0000 7f4d5aebb740 1 -- 172.21.15.98:0/2045373487 shutdown_connections 2024-09-19T08:08:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.521+0000 7f4d5aebb740 1 -- 172.21.15.98:0/2045373487 wait complete. 2024-09-19T08:08:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.521+0000 7f4d5aebb740 20 client.16884 trim_cache size 0 max 16384 2024-09-19T08:08:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.534+0000 7f4d5aebb740 1 Processor -- start 2024-09-19T08:08:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.534+0000 7f4d5aebb740 1 -- start start 2024-09-19T08:08:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.534+0000 7f4d5aebb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 0x560a558a9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.534+0000 7f4d5aebb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558a1b40 0x560a558a3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.534+0000 7f4d5aebb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558ab370 0x560a558a3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.534+0000 7f4d5aebb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541ad6e0 con 0x560a558a1b40 2024-09-19T08:08:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.534+0000 7f4d5aebb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541af220 con 0x560a558ab370 2024-09-19T08:08:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.534+0000 7f4d5aebb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541addb0 con 0x560a558a05d0 2024-09-19T08:08:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d51d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558a1b40 0x560a558a3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d51d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558a1b40 0x560a558a3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53266/0 (socket says 172.21.15.98:53266) 2024-09-19T08:08:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d51d9b640 1 -- 172.21.15.98:0/686623058 learned_addr learned my addr 172.21.15.98:0/686623058 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d52d9d640 1 --2- 172.21.15.98:0/686623058 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558ab370 0x560a558a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d5259c640 1 --2- 172.21.15.98:0/686623058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 0x560a558a9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d51d9b640 1 -- 172.21.15.98:0/686623058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 msgr2=0x560a558a9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d51d9b640 1 --2- 172.21.15.98:0/686623058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 0x560a558a9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d51d9b640 1 -- 172.21.15.98:0/686623058 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558ab370 msgr2=0x560a558a3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d51d9b640 1 --2- 172.21.15.98:0/686623058 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558ab370 0x560a558a3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d51d9b640 1 -- 172.21.15.98:0/686623058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a558a8be0 con 0x560a558a1b40 2024-09-19T08:08:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d52d9d640 1 --2- 172.21.15.98:0/686623058 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558ab370 0x560a558a3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:08:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d5259c640 1 --2- 172.21.15.98:0/686623058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 0x560a558a9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:08:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d51d9b640 1 --2- 172.21.15.98:0/686623058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558a1b40 0x560a558a3120 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f4d4809c0e0 tx=0x7f4d48013b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db281fcae1fd350 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d3bfff640 1 -- 172.21.15.98:0/686623058 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d480b4070 con 0x560a558a1b40 2024-09-19T08:08:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d3bfff640 1 -- 172.21.15.98:0/686623058 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d480a0500 con 0x560a558a1b40 2024-09-19T08:08:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.535+0000 7f4d3bfff640 1 -- 172.21.15.98:0/686623058 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d480cd840 con 0x560a558a1b40 2024-09-19T08:08:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.536+0000 7f4d5aebb740 1 -- 172.21.15.98:0/686623058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558a1b40 msgr2=0x560a558a3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.536+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/686623058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558a1b40 0x560a558a3120 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f4d4809c0e0 tx=0x7f4d48013b80 comp rx=0 tx=0).stop 2024-09-19T08:08:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.536+0000 7f4d5aebb740 1 -- 172.21.15.98:0/686623058 shutdown_connections 2024-09-19T08:08:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.536+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/686623058 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558ab370 0x560a558a3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.536+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/686623058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558a1b40 0x560a558a3120 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.536+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/686623058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 0x560a558a9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.536+0000 7f4d5aebb740 1 -- 172.21.15.98:0/686623058 >> 172.21.15.98:0/686623058 conn(0x560a55869310 msgr2=0x560a558a9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:38.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.536+0000 7f4d5aebb740 1 -- 172.21.15.98:0/686623058 shutdown_connections 2024-09-19T08:08:38.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.536+0000 7f4d5aebb740 1 -- 172.21.15.98:0/686623058 wait complete. 2024-09-19T08:08:38.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.536+0000 7f4d5aebb740 1 -- start start 2024-09-19T08:08:38.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.536+0000 7f4d5aebb740 1 Processor -- start 2024-09-19T08:08:38.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d5aebb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 0x560a558b3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d5aebb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 0x560a558b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d5aebb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558ab370 0x560a558b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d5aebb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541aeb50 con 0x560a558ab370 2024-09-19T08:08:38.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d5aebb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541ae480 con 0x560a558a1b40 2024-09-19T08:08:38.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d5aebb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541ab6c0 con 0x560a558a05d0 2024-09-19T08:08:38.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d5aebb740 10 client.0 _ll_register_callbacks cb 0x7f4d4c0125b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d52d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558ab370 0x560a558b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d51d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 0x560a558b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d52d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558ab370 0x560a558b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53282/0 (socket says 172.21.15.98:53282) 2024-09-19T08:08:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d52d9d640 1 -- 172.21.15.98:0/356871173 learned_addr learned my addr 172.21.15.98:0/356871173 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d5259c640 1 --2- 172.21.15.98:0/356871173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 0x560a558b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d52d9d640 1 -- 172.21.15.98:0/356871173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 msgr2=0x560a558b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d52d9d640 1 --2- 172.21.15.98:0/356871173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 0x560a558b3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d52d9d640 1 -- 172.21.15.98:0/356871173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 msgr2=0x560a558b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d52d9d640 1 --2- 172.21.15.98:0/356871173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 0x560a558b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d52d9d640 1 -- 172.21.15.98:0/356871173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a558b0bd0 con 0x560a558ab370 2024-09-19T08:08:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d52d9d640 1 --2- 172.21.15.98:0/356871173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558ab370 0x560a558b03f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c0092f0 tx=0x7f4d4c004400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7dcf89419165028 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.537+0000 7f4d39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.538+0000 7f4d39ffb640 1 -- 172.21.15.98:0/356871173 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c012880 con 0x560a558ab370 2024-09-19T08:08:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.538+0000 7f4d39ffb640 1 -- 172.21.15.98:0/356871173 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f4d4c02a8b0 con 0x560a558ab370 2024-09-19T08:08:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.538+0000 7f4d5aebb740 1 -- 172.21.15.98:0/356871173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d480a0500 con 0x560a558ab370 2024-09-19T08:08:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.538+0000 7f4d39ffb640 1 -- 172.21.15.98:0/356871173 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4d4c02aab0 con 0x560a558ab370 2024-09-19T08:08:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.538+0000 7f4d5aebb740 10 client.16890 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.538+0000 7f4d5aebb740 1 -- 172.21.15.98:0/356871173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560a55926600 con 0x560a558ab370 2024-09-19T08:08:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.538+0000 7f4d39ffb640 1 -- 172.21.15.98:0/356871173 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d480a0500 con 0x560a558ab370 2024-09-19T08:08:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.538+0000 7f4d39ffb640 1 -- 172.21.15.98:0/356871173 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4d4c00f040 con 0x560a558ab370 2024-09-19T08:08:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.538+0000 7f4d5aebb740 10 client.16890 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.538+0000 7f4d5aebb740 10 client.16890 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:08:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.538+0000 7f4d5aebb740 20 client.16890 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.538+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/356871173 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x560a55926cf0 0x560a55929110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.538+0000 7f4d5aebb740 4 client.16890 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:08:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.538+0000 7f4d5aebb740 1 -- 172.21.15.98:0/356871173 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560a5580dd70 con 0x560a55926cf0 2024-09-19T08:08:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.539+0000 7f4d5259c640 1 --2- 172.21.15.98:0/356871173 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x560a55926cf0 0x560a55929110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.539+0000 7f4d39ffb640 10 client.16890 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:08:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.539+0000 7f4d5259c640 1 --2- 172.21.15.98:0/356871173 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x560a55926cf0 0x560a55929110 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8b59258665fb06dc server_cookie=ab132f0fc4a345a6 in_seq=0 out_seq=0 2024-09-19T08:08:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.539+0000 7f4d39ffb640 1 -- 172.21.15.98:0/356871173 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560a5580dd70 con 0x560a55926cf0 2024-09-19T08:08:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.539+0000 7f4d39ffb640 10 client.16890 handle_command_reply: tid=0 2024-09-19T08:08:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.539+0000 7f4d5aebb740 1 client.16890 shutdown 2024-09-19T08:08:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.539+0000 7f4d5aebb740 1 -- 172.21.15.98:0/356871173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558ab370 msgr2=0x560a558b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.539+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/356871173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558ab370 0x560a558b03f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c0092f0 tx=0x7f4d4c004400 comp rx=0 tx=0).stop 2024-09-19T08:08:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.539+0000 7f4d5aebb740 1 -- 172.21.15.98:0/356871173 shutdown_connections 2024-09-19T08:08:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.539+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/356871173 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x560a55926cf0 0x560a55929110 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/356871173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a558ab370 0x560a558b03f0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/356871173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a558a1b40 0x560a558b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d5aebb740 1 --2- 172.21.15.98:0/356871173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a558a05d0 0x560a558b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d5aebb740 1 -- 172.21.15.98:0/356871173 >> 172.21.15.98:0/356871173 conn(0x560a55869310 msgr2=0x560a558aa070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:38.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d39ffb640 0 client.16890 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:08:38.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d5aebb740 1 -- 172.21.15.98:0/356871173 shutdown_connections 2024-09-19T08:08:38.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d5aebb740 1 -- 172.21.15.98:0/356871173 wait complete. 2024-09-19T08:08:38.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d5aebb740 20 client.16890 trim_cache size 0 max 16384 2024-09-19T08:08:38.546 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:08:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d58ae2640 1 -- 172.21.15.98:0/786421461 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4d3c0a33b0 msgr2=0x7f4d3c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d58ae2640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4d3c0a33b0 0x7f4d3c0a5870 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c0098b0 tx=0x7f4d4c009d90 comp rx=0 tx=0).stop 2024-09-19T08:08:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d58ae2640 1 -- 172.21.15.98:0/786421461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54102790 msgr2=0x7f4d541988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d58ae2640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54102790 0x7f4d541988d0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f4d4800c920 tx=0x7f4d4800cde0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d58ae2640 1 -- 172.21.15.98:0/786421461 shutdown_connections 2024-09-19T08:08:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d58ae2640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4d3c0a33b0 0x7f4d3c0a5870 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d58ae2640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d54106f10 0x7f4d5419cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d58ae2640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54102790 0x7f4d541988d0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d58ae2640 1 --2- 172.21.15.98:0/786421461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d54101de0 0x7f4d54198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.540+0000 7f4d58ae2640 1 -- 172.21.15.98:0/786421461 >> 172.21.15.98:0/786421461 conn(0x7f4d540fd8c0 msgr2=0x7f4d540fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.541+0000 7f4d58ae2640 1 -- 172.21.15.98:0/786421461 shutdown_connections 2024-09-19T08:08:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:38.541+0000 7f4d58ae2640 1 -- 172.21.15.98:0/786421461 wait complete. 2024-09-19T08:08:39.171 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:08:39.493 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:08:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.987+0000 7f2985a08640 1 -- 172.21.15.98:0/3250828505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2980100e20 msgr2=0x7f2980101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.987+0000 7f2985a08640 1 --2- 172.21.15.98:0/3250828505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2980100e20 0x7f2980101200 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f2970009900 tx=0x7f297002a480 comp rx=0 tx=0).stop 2024-09-19T08:08:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.987+0000 7f2985a08640 1 -- 172.21.15.98:0/3250828505 shutdown_connections 2024-09-19T08:08:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.987+0000 7f2985a08640 1 --2- 172.21.15.98:0/3250828505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2980102ef0 0x7f298010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.987+0000 7f2985a08640 1 --2- 172.21.15.98:0/3250828505 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2980102570 0x7f29801029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.987+0000 7f2985a08640 1 --2- 172.21.15.98:0/3250828505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2980100e20 0x7f2980101200 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.987+0000 7f2985a08640 1 -- 172.21.15.98:0/3250828505 >> 172.21.15.98:0/3250828505 conn(0x7f29800fc7d0 msgr2=0x7f29800febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.987+0000 7f2985a08640 1 -- 172.21.15.98:0/3250828505 shutdown_connections 2024-09-19T08:08:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.987+0000 7f2985a08640 1 -- 172.21.15.98:0/3250828505 wait complete. 2024-09-19T08:08:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.988+0000 7f2985a08640 1 Processor -- start 2024-09-19T08:08:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.988+0000 7f2985a08640 1 -- start start 2024-09-19T08:08:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.988+0000 7f2985a08640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2980100e20 0x7f2980198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.988+0000 7f2985a08640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2980102570 0x7f2980198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.988+0000 7f2985a08640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2980102ef0 0x7f298019cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.988+0000 7f2985a08640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2980111020 con 0x7f2980102ef0 2024-09-19T08:08:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.988+0000 7f2985a08640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2980110ea0 con 0x7f2980102570 2024-09-19T08:08:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.988+0000 7f2985a08640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f29801111a0 con 0x7f2980100e20 2024-09-19T08:08:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.988+0000 7f297f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2980102ef0 0x7f298019cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f297f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2980102ef0 0x7f298019cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53300/0 (socket says 172.21.15.98:53300) 2024-09-19T08:08:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f297f7fe640 1 -- 172.21.15.98:0/784411276 learned_addr learned my addr 172.21.15.98:0/784411276 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f297effd640 1 --2- 172.21.15.98:0/784411276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2980100e20 0x7f2980198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f29767fc640 1 --2- 172.21.15.98:0/784411276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2980102570 0x7f2980198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f297f7fe640 1 -- 172.21.15.98:0/784411276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2980100e20 msgr2=0x7f2980198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f297f7fe640 1 --2- 172.21.15.98:0/784411276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2980100e20 0x7f2980198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f297f7fe640 1 -- 172.21.15.98:0/784411276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2980102570 msgr2=0x7f2980198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f297f7fe640 1 --2- 172.21.15.98:0/784411276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2980102570 0x7f2980198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f297f7fe640 1 -- 172.21.15.98:0/784411276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f298019d6c0 con 0x7f2980102ef0 2024-09-19T08:08:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f297f7fe640 1 --2- 172.21.15.98:0/784411276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2980102ef0 0x7f298019cfe0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f297800b970 tx=0x7f297800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f2984a06640 1 -- 172.21.15.98:0/784411276 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2978019070 con 0x7f2980102ef0 2024-09-19T08:08:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f2984a06640 1 -- 172.21.15.98:0/784411276 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f297800cb40 con 0x7f2980102ef0 2024-09-19T08:08:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f2984a06640 1 -- 172.21.15.98:0/784411276 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2978007650 con 0x7f2980102ef0 2024-09-19T08:08:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f2985a08640 1 -- 172.21.15.98:0/784411276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f298019d9b0 con 0x7f2980102ef0 2024-09-19T08:08:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.989+0000 7f2985a08640 1 -- 172.21.15.98:0/784411276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29801a5290 con 0x7f2980102ef0 2024-09-19T08:08:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.992+0000 7f2984a06640 1 -- 172.21.15.98:0/784411276 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f297801c070 con 0x7f2980102ef0 2024-09-19T08:08:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.992+0000 7f2987de1740 1 Processor -- start 2024-09-19T08:08:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.992+0000 7f2987de1740 1 -- start start 2024-09-19T08:08:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.992+0000 7f2987de1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a33674090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.992+0000 7f2987de1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 0x561a33674a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.992+0000 7f2987de1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a3367f370 0x561a33681760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.992+0000 7f2987de1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561a335da2f0 con 0x561a33675b40 2024-09-19T08:08:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.992+0000 7f2987de1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561a33581cf0 con 0x561a3367f370 2024-09-19T08:08:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.992+0000 7f2987de1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561a335d62b0 con 0x561a336745d0 2024-09-19T08:08:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f29767fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a33674090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f29767fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a33674090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53312/0 (socket says 172.21.15.98:53312) 2024-09-19T08:08:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f29767fc640 1 -- 172.21.15.98:0/4125678348 learned_addr learned my addr 172.21.15.98:0/4125678348 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f297effd640 1 --2- 172.21.15.98:0/4125678348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 0x561a33674a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f29767fc640 1 --2- 172.21.15.98:0/4125678348 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a3367f370 0x561a33681760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f2984a06640 1 --2- 172.21.15.98:0/784411276 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f296c0a33b0 0x7f296c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f29767fc640 1 -- 172.21.15.98:0/4125678348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 msgr2=0x561a33674a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f29767fc640 1 --2- 172.21.15.98:0/4125678348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 0x561a33674a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f29767fc640 1 -- 172.21.15.98:0/4125678348 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a3367f370 msgr2=0x561a33681760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f29767fc640 1 --2- 172.21.15.98:0/4125678348 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a3367f370 0x561a33681760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f29767fc640 1 -- 172.21.15.98:0/4125678348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a335663a0 con 0x561a33675b40 2024-09-19T08:08:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f297f7fe640 1 --2- 172.21.15.98:0/784411276 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f296c0a33b0 0x7f296c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f29767fc640 1 --2- 172.21.15.98:0/4125678348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a33674090 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2968006e10 tx=0x7f2968009420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2e7b011a6211ea server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f2984a06640 1 -- 172.21.15.98:0/784411276 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f29780cf1f0 con 0x7f2980102ef0 2024-09-19T08:08:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f2975ffb640 1 -- 172.21.15.98:0/4125678348 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296801c070 con 0x561a33675b40 2024-09-19T08:08:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f2975ffb640 1 -- 172.21.15.98:0/4125678348 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2968010040 con 0x561a33675b40 2024-09-19T08:08:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.993+0000 7f2975ffb640 1 -- 172.21.15.98:0/4125678348 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296801c070 con 0x561a33675b40 2024-09-19T08:08:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.994+0000 7f2987de1740 1 -- 172.21.15.98:0/4125678348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 msgr2=0x561a33674090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.994+0000 7f297f7fe640 1 --2- 172.21.15.98:0/784411276 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f296c0a33b0 0x7f296c0a5870 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f29780ccf70 tx=0x7f297806d390 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.994+0000 7f2987de1740 1 --2- 172.21.15.98:0/4125678348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a33674090 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2968006e10 tx=0x7f2968009420 comp rx=0 tx=0).stop 2024-09-19T08:08:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.994+0000 7f2987de1740 1 -- 172.21.15.98:0/4125678348 shutdown_connections 2024-09-19T08:08:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.994+0000 7f2987de1740 1 --2- 172.21.15.98:0/4125678348 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a3367f370 0x561a33681760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.994+0000 7f2987de1740 1 --2- 172.21.15.98:0/4125678348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 0x561a33674a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.994+0000 7f2987de1740 1 --2- 172.21.15.98:0/4125678348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a33674090 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.994+0000 7f2987de1740 1 -- 172.21.15.98:0/4125678348 >> 172.21.15.98:0/4125678348 conn(0x561a3363d310 msgr2=0x561a33670c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.994+0000 7f2987de1740 1 -- 172.21.15.98:0/4125678348 shutdown_connections 2024-09-19T08:08:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.994+0000 7f2987de1740 1 -- 172.21.15.98:0/4125678348 wait complete. 2024-09-19T08:08:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.994+0000 7f2987de1740 1 -- start start 2024-09-19T08:08:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f2987de1740 1 Processor -- start 2024-09-19T08:08:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f2987de1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 0x561a3367ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f2987de1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a33675b40 0x561a3367c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f2987de1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a3367f370 0x561a3367c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f2987de1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561a33682070 con 0x561a3367f370 2024-09-19T08:08:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f2987de1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561a33595060 con 0x561a33675b40 2024-09-19T08:08:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f2987de1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561a336821f0 con 0x561a336745d0 2024-09-19T08:08:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f2987de1740 10 client.0 _ll_register_callbacks cb 0x561a332e88c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f29767fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a3367f370 0x561a3367c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f29767fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a3367f370 0x561a3367c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53324/0 (socket says 172.21.15.98:53324) 2024-09-19T08:08:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f29767fc640 1 -- 172.21.15.98:0/4131014540 learned_addr learned my addr 172.21.15.98:0/4131014540 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f29767fc640 1 --2- 172.21.15.98:0/4131014540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 0x561a3367ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f297effd640 1 --2- 172.21.15.98:0/4131014540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a33675b40 0x561a3367c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f29767fc640 1 -- 172.21.15.98:0/4131014540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 msgr2=0x561a3367ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f29767fc640 1 --2- 172.21.15.98:0/4131014540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 0x561a3367ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f29767fc640 1 -- 172.21.15.98:0/4131014540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a33675b40 msgr2=0x561a3367c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f29767fc640 1 --2- 172.21.15.98:0/4131014540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a33675b40 0x561a3367c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.995+0000 7f29767fc640 1 -- 172.21.15.98:0/4131014540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a3334b3c0 con 0x561a3367f370 2024-09-19T08:08:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f29767fc640 1 --2- 172.21.15.98:0/4131014540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a3367f370 0x561a3367c640 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2968017e70 tx=0x7f29680143b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bddc8376b906b557 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f295effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f295effd640 1 -- 172.21.15.98:0/4131014540 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296801c040 con 0x561a3367f370 2024-09-19T08:08:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f295effd640 1 -- 172.21.15.98:0/4131014540 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f2968014690 con 0x561a3367f370 2024-09-19T08:08:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f2987de1740 1 -- 172.21.15.98:0/4131014540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2968010040 con 0x561a3367f370 2024-09-19T08:08:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f295effd640 1 -- 172.21.15.98:0/4131014540 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2968014890 con 0x561a3367f370 2024-09-19T08:08:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f2987de1740 10 client.16902 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f2987de1740 1 -- 172.21.15.98:0/4131014540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561a3368db40 con 0x561a3367f370 2024-09-19T08:08:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f295effd640 1 -- 172.21.15.98:0/4131014540 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2968010040 con 0x561a3367f370 2024-09-19T08:08:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f295effd640 1 -- 172.21.15.98:0/4131014540 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2968022040 con 0x561a3367f370 2024-09-19T08:08:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f2987de1740 10 client.16902 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f2987de1740 10 client.16902 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:08:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f2987de1740 20 client.16902 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f2987de1740 1 --2- 172.21.15.98:0/4131014540 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561a3368ee90 0x561a33691280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f2987de1740 4 client.16902 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:08:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f2987de1740 1 -- 172.21.15.98:0/4131014540 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a335e7860 con 0x561a3368ee90 2024-09-19T08:08:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.996+0000 7f297effd640 1 --2- 172.21.15.98:0/4131014540 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561a3368ee90 0x561a33691280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.997+0000 7f295effd640 10 client.16902 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.997+0000 7f297effd640 1 --2- 172.21.15.98:0/4131014540 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561a3368ee90 0x561a33691280 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d728603bdaf396d server_cookie=bc149df4ead06e in_seq=0 out_seq=0 2024-09-19T08:08:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f295effd640 1 -- 172.21.15.98:0/4131014540 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561a335e7860 con 0x561a3368ee90 2024-09-19T08:08:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f295effd640 10 client.16902 handle_command_reply: tid=0 2024-09-19T08:08:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f2987de1740 1 client.16902 shutdown 2024-09-19T08:08:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f2987de1740 1 -- 172.21.15.98:0/4131014540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a3367f370 msgr2=0x561a3367c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f2987de1740 1 --2- 172.21.15.98:0/4131014540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a3367f370 0x561a3367c640 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2968017e70 tx=0x7f29680143b0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f2987de1740 1 -- 172.21.15.98:0/4131014540 shutdown_connections 2024-09-19T08:08:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f2987de1740 1 --2- 172.21.15.98:0/4131014540 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561a3368ee90 0x561a33691280 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f2987de1740 1 --2- 172.21.15.98:0/4131014540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a3367f370 0x561a3367c640 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f2987de1740 1 --2- 172.21.15.98:0/4131014540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a33675b40 0x561a3367c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f2987de1740 1 --2- 172.21.15.98:0/4131014540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 0x561a3367ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f295effd640 0 client.16902 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f2987de1740 1 -- 172.21.15.98:0/4131014540 >> 172.21.15.98:0/4131014540 conn(0x561a3363d310 msgr2=0x561a33676960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f2987de1740 1 -- 172.21.15.98:0/4131014540 shutdown_connections 2024-09-19T08:08:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f2987de1740 1 -- 172.21.15.98:0/4131014540 wait complete. 2024-09-19T08:08:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:39.998+0000 7f2987de1740 20 client.16902 trim_cache size 0 max 16384 2024-09-19T08:08:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.010+0000 7f2987de1740 1 Processor -- start 2024-09-19T08:08:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.010+0000 7f2987de1740 1 -- start start 2024-09-19T08:08:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f2987de1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a336745d0 0x561a3368b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f2987de1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a3367dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f2987de1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a3367f370 0x561a33677120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f2987de1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f29801ada10 con 0x561a33675b40 2024-09-19T08:08:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f2987de1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f29801af550 con 0x561a336745d0 2024-09-19T08:08:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f2987de1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f29801ae0e0 con 0x561a3367f370 2024-09-19T08:08:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f297effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a3367dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f297effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a3367dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53340/0 (socket says 172.21.15.98:53340) 2024-09-19T08:08:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f297effd640 1 -- 172.21.15.98:0/3053298945 learned_addr learned my addr 172.21.15.98:0/3053298945 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f29767fc640 1 --2- 172.21.15.98:0/3053298945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a336745d0 0x561a3368b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f29767fc640 1 --2- 172.21.15.98:0/3053298945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a3367f370 0x561a33677120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f297effd640 1 -- 172.21.15.98:0/3053298945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a3367f370 msgr2=0x561a33677120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f297effd640 1 --2- 172.21.15.98:0/3053298945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a3367f370 0x561a33677120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f297effd640 1 -- 172.21.15.98:0/3053298945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a336745d0 msgr2=0x561a3368b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f297effd640 1 --2- 172.21.15.98:0/3053298945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a336745d0 0x561a3368b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f297effd640 1 -- 172.21.15.98:0/3053298945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a33670510 con 0x561a33675b40 2024-09-19T08:08:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f29767fc640 1 --2- 172.21.15.98:0/3053298945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a336745d0 0x561a3368b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:08:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f297effd640 1 --2- 172.21.15.98:0/3053298945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a3367dfa0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f2970005940 tx=0x7f29700058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a704dbe5c6c110 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f29757fa640 1 -- 172.21.15.98:0/3053298945 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2970035040 con 0x561a33675b40 2024-09-19T08:08:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f29757fa640 1 -- 172.21.15.98:0/3053298945 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2970005500 con 0x561a33675b40 2024-09-19T08:08:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.011+0000 7f29757fa640 1 -- 172.21.15.98:0/3053298945 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29700035f0 con 0x561a33675b40 2024-09-19T08:08:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.012+0000 7f2987de1740 1 -- 172.21.15.98:0/3053298945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 msgr2=0x561a3367dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.012+0000 7f2987de1740 1 --2- 172.21.15.98:0/3053298945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a3367dfa0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f2970005940 tx=0x7f29700058b0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.012+0000 7f2987de1740 1 -- 172.21.15.98:0/3053298945 shutdown_connections 2024-09-19T08:08:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.012+0000 7f2987de1740 1 --2- 172.21.15.98:0/3053298945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a3367f370 0x561a33677120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.012+0000 7f2987de1740 1 --2- 172.21.15.98:0/3053298945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a3367dfa0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.012+0000 7f2987de1740 1 --2- 172.21.15.98:0/3053298945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a336745d0 0x561a3368b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.012+0000 7f2987de1740 1 -- 172.21.15.98:0/3053298945 >> 172.21.15.98:0/3053298945 conn(0x561a3363d310 msgr2=0x561a3367d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.012+0000 7f2987de1740 1 -- 172.21.15.98:0/3053298945 shutdown_connections 2024-09-19T08:08:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.012+0000 7f2987de1740 1 -- 172.21.15.98:0/3053298945 wait complete. 2024-09-19T08:08:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.012+0000 7f2987de1740 1 -- start start 2024-09-19T08:08:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.012+0000 7f2987de1740 1 Processor -- start 2024-09-19T08:08:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f2987de1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 0x561a33682480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f2987de1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a336829c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f2987de1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a3367f370 0x561a336f7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f2987de1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f29801aee80 con 0x561a33675b40 2024-09-19T08:08:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f2987de1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f29801ae7b0 con 0x561a3367f370 2024-09-19T08:08:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f2987de1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f29801ab9f0 con 0x561a336745d0 2024-09-19T08:08:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f2987de1740 10 client.0 _ll_register_callbacks cb 0x7f296801a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f297effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a336829c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f297effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a336829c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53344/0 (socket says 172.21.15.98:53344) 2024-09-19T08:08:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f297effd640 1 -- 172.21.15.98:0/3780430662 learned_addr learned my addr 172.21.15.98:0/3780430662 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f29767fc640 1 --2- 172.21.15.98:0/3780430662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 0x561a33682480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f297effd640 1 -- 172.21.15.98:0/3780430662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 msgr2=0x561a33682480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f297effd640 1 --2- 172.21.15.98:0/3780430662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 0x561a33682480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f297effd640 1 -- 172.21.15.98:0/3780430662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a3367f370 msgr2=0x561a336f7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f297effd640 1 --2- 172.21.15.98:0/3780430662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a3367f370 0x561a336f7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f297effd640 1 -- 172.21.15.98:0/3780430662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a33683250 con 0x561a33675b40 2024-09-19T08:08:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f29767fc640 1 --2- 172.21.15.98:0/3780430662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 0x561a33682480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:08:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f297effd640 1 --2- 172.21.15.98:0/3780430662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a336829c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f2970009230 tx=0x7f2970004460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d938b6efd00520ae server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f295effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f295effd640 1 -- 172.21.15.98:0/3780430662 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2970053060 con 0x561a33675b40 2024-09-19T08:08:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f2987de1740 1 -- 172.21.15.98:0/3780430662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2970005500 con 0x561a33675b40 2024-09-19T08:08:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.013+0000 7f295effd640 1 -- 172.21.15.98:0/3780430662 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f2970005020 con 0x561a33675b40 2024-09-19T08:08:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.014+0000 7f295effd640 1 -- 172.21.15.98:0/3780430662 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2970035070 con 0x561a33675b40 2024-09-19T08:08:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.014+0000 7f2987de1740 10 client.16908 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.014+0000 7f2987de1740 1 -- 172.21.15.98:0/3780430662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561a336f8f70 con 0x561a33675b40 2024-09-19T08:08:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.014+0000 7f295effd640 1 -- 172.21.15.98:0/3780430662 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2970005500 con 0x561a33675b40 2024-09-19T08:08:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.014+0000 7f295effd640 1 -- 172.21.15.98:0/3780430662 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2970031540 con 0x561a33675b40 2024-09-19T08:08:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.014+0000 7f2987de1740 10 client.16908 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.014+0000 7f2987de1740 10 client.16908 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:08:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.014+0000 7f2987de1740 20 client.16908 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.014+0000 7f2987de1740 1 --2- 172.21.15.98:0/3780430662 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561a336f96d0 0x561a336fbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.014+0000 7f2987de1740 4 client.16908 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:08:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.014+0000 7f2987de1740 1 -- 172.21.15.98:0/3780430662 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561a335e1d70 con 0x561a336f96d0 2024-09-19T08:08:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.014+0000 7f29767fc640 1 --2- 172.21.15.98:0/3780430662 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561a336f96d0 0x561a336fbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f295effd640 10 client.16908 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f29767fc640 1 --2- 172.21.15.98:0/3780430662 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561a336f96d0 0x561a336fbaf0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84a4eb823c31ed2e server_cookie=fac7bf40349f1d85 in_seq=0 out_seq=0 2024-09-19T08:08:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f295effd640 1 -- 172.21.15.98:0/3780430662 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561a335e1d70 con 0x561a336f96d0 2024-09-19T08:08:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f295effd640 10 client.16908 handle_command_reply: tid=0 2024-09-19T08:08:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f2987de1740 1 client.16908 shutdown 2024-09-19T08:08:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f2987de1740 1 -- 172.21.15.98:0/3780430662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 msgr2=0x561a336829c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f2987de1740 1 --2- 172.21.15.98:0/3780430662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a336829c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f2970009230 tx=0x7f2970004460 comp rx=0 tx=0).stop 2024-09-19T08:08:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f2987de1740 1 -- 172.21.15.98:0/3780430662 shutdown_connections 2024-09-19T08:08:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f2987de1740 1 --2- 172.21.15.98:0/3780430662 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561a336f96d0 0x561a336fbaf0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f2987de1740 1 --2- 172.21.15.98:0/3780430662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a3367f370 0x561a336f7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f2987de1740 1 --2- 172.21.15.98:0/3780430662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a33675b40 0x561a336829c0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f295effd640 0 client.16908 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f2987de1740 1 --2- 172.21.15.98:0/3780430662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561a336745d0 0x561a33682480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f2987de1740 1 -- 172.21.15.98:0/3780430662 >> 172.21.15.98:0/3780430662 conn(0x561a3363d310 msgr2=0x561a3367e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f2987de1740 1 -- 172.21.15.98:0/3780430662 shutdown_connections 2024-09-19T08:08:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f2987de1740 1 -- 172.21.15.98:0/3780430662 wait complete. 2024-09-19T08:08:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.015+0000 7f2987de1740 20 client.16908 trim_cache size 0 max 16384 2024-09-19T08:08:40.033 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:08:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.016+0000 7f2985a08640 1 -- 172.21.15.98:0/784411276 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f296c0a33b0 msgr2=0x7f296c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.016+0000 7f2985a08640 1 --2- 172.21.15.98:0/784411276 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f296c0a33b0 0x7f296c0a5870 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f29780ccf70 tx=0x7f297806d390 comp rx=0 tx=0).stop 2024-09-19T08:08:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.016+0000 7f2985a08640 1 -- 172.21.15.98:0/784411276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2980102ef0 msgr2=0x7f298019cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.016+0000 7f2985a08640 1 --2- 172.21.15.98:0/784411276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2980102ef0 0x7f298019cfe0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f297800b970 tx=0x7f297800be30 comp rx=0 tx=0).stop 2024-09-19T08:08:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.016+0000 7f2985a08640 1 -- 172.21.15.98:0/784411276 shutdown_connections 2024-09-19T08:08:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.016+0000 7f2985a08640 1 --2- 172.21.15.98:0/784411276 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f296c0a33b0 0x7f296c0a5870 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.016+0000 7f2985a08640 1 --2- 172.21.15.98:0/784411276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2980102ef0 0x7f298019cfe0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.016+0000 7f2985a08640 1 --2- 172.21.15.98:0/784411276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2980102570 0x7f2980198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.016+0000 7f2985a08640 1 --2- 172.21.15.98:0/784411276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2980100e20 0x7f2980198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.016+0000 7f2985a08640 1 -- 172.21.15.98:0/784411276 >> 172.21.15.98:0/784411276 conn(0x7f29800fc7d0 msgr2=0x7f298006a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.016+0000 7f2985a08640 1 -- 172.21.15.98:0/784411276 shutdown_connections 2024-09-19T08:08:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:40.016+0000 7f2985a08640 1 -- 172.21.15.98:0/784411276 wait complete. 2024-09-19T08:08:40.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:40 smithi098 ceph-mon[31333]: pgmap v172: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:08:40.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:40 smithi162 ceph-mon[35260]: pgmap v172: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:08:40.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:40 smithi139 ceph-mon[34874]: pgmap v172: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:08:41.647 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:08:41.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:08:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:08:42.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:42 smithi098 ceph-mon[31333]: pgmap v173: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:42.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:42 smithi162 ceph-mon[35260]: pgmap v173: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:42.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:42 smithi139 ceph-mon[34874]: pgmap v173: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:42.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.405+0000 7f07d0862640 1 -- 172.21.15.98:0/2499769259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07cc101430 msgr2=0x7f07cc10d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:42.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.405+0000 7f07d0862640 1 --2- 172.21.15.98:0/2499769259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07cc101430 0x7f07cc10d960 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009cd0 tx=0x7f07c402a8e0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.405+0000 7f07d0862640 1 -- 172.21.15.98:0/2499769259 shutdown_connections 2024-09-19T08:08:42.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.405+0000 7f07d0862640 1 --2- 172.21.15.98:0/2499769259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07cc101430 0x7f07cc10d960 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.405+0000 7f07d0862640 1 --2- 172.21.15.98:0/2499769259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07cc100a90 0x7f07cc100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.405+0000 7f07d0862640 1 --2- 172.21.15.98:0/2499769259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07cc106a90 0x7f07cc106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.405+0000 7f07d0862640 1 -- 172.21.15.98:0/2499769259 >> 172.21.15.98:0/2499769259 conn(0x7f07cc0fc7d0 msgr2=0x7f07cc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:42.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.405+0000 7f07d0862640 1 -- 172.21.15.98:0/2499769259 shutdown_connections 2024-09-19T08:08:42.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.405+0000 7f07d0862640 1 -- 172.21.15.98:0/2499769259 wait complete. 2024-09-19T08:08:42.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.406+0000 7f07d0862640 1 Processor -- start 2024-09-19T08:08:42.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.406+0000 7f07d0862640 1 -- start start 2024-09-19T08:08:42.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07d0862640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07cc100a90 0x7f07cc1972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07d0862640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07cc101430 0x7f07cc197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07d0862640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07cc106a90 0x7f07cc19bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07d0862640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f07cc110e90 con 0x7f07cc101430 2024-09-19T08:08:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07d0862640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f07cc110d10 con 0x7f07cc100a90 2024-09-19T08:08:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07d0862640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f07cc111010 con 0x7f07cc106a90 2024-09-19T08:08:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07c3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07cc101430 0x7f07cc197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07c3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07cc101430 0x7f07cc197830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53358/0 (socket says 172.21.15.98:53358) 2024-09-19T08:08:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07c3fff640 1 -- 172.21.15.98:0/1726321832 learned_addr learned my addr 172.21.15.98:0/1726321832 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07cad9d640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07cc106a90 0x7f07cc19bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07ca59c640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07cc100a90 0x7f07cc1972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07c3fff640 1 -- 172.21.15.98:0/1726321832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07cc106a90 msgr2=0x7f07cc19bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07c3fff640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07cc106a90 0x7f07cc19bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07c3fff640 1 -- 172.21.15.98:0/1726321832 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07cc100a90 msgr2=0x7f07cc1972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07c3fff640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07cc100a90 0x7f07cc1972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07c3fff640 1 -- 172.21.15.98:0/1726321832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07cc19c2a0 con 0x7f07cc101430 2024-09-19T08:08:42.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07ca59c640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07cc100a90 0x7f07cc1972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:08:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07c3fff640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07cc101430 0x7f07cc197830 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f07b400d770 tx=0x7f07b400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07c37fe640 1 -- 172.21.15.98:0/1726321832 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b4017070 con 0x7f07cc101430 2024-09-19T08:08:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07c37fe640 1 -- 172.21.15.98:0/1726321832 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07b4005500 con 0x7f07cc101430 2024-09-19T08:08:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07d0862640 1 -- 172.21.15.98:0/1726321832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07cc19c590 con 0x7f07cc101430 2024-09-19T08:08:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07c37fe640 1 -- 172.21.15.98:0/1726321832 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b4004e00 con 0x7f07cc101430 2024-09-19T08:08:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.407+0000 7f07d0862640 1 -- 172.21.15.98:0/1726321832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07cc1a3e70 con 0x7f07cc101430 2024-09-19T08:08:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.409+0000 7f07c37fe640 1 -- 172.21.15.98:0/1726321832 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f07b4004fa0 con 0x7f07cc101430 2024-09-19T08:08:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.410+0000 7f07d2c3b740 1 Processor -- start 2024-09-19T08:08:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.410+0000 7f07d2c3b740 1 -- start start 2024-09-19T08:08:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.410+0000 7f07d2c3b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a49a40 0x5642f9a49e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.410+0000 7f07d2c3b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 0x5642f9a56a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.410+0000 7f07d2c3b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a56f50 0x5642f9a59340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.410+0000 7f07d2c3b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5642f99af280 con 0x5642f9a49a40 2024-09-19T08:08:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.410+0000 7f07d2c3b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5642f9956b10 con 0x5642f9a4a360 2024-09-19T08:08:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.410+0000 7f07d2c3b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5642f99ab240 con 0x5642f9a56f50 2024-09-19T08:08:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.410+0000 7f07cad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a49a40 0x5642f9a49e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.410+0000 7f07cad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a49a40 0x5642f9a49e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53370/0 (socket says 172.21.15.98:53370) 2024-09-19T08:08:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.410+0000 7f07cad9d640 1 -- 172.21.15.98:0/1502819707 learned_addr learned my addr 172.21.15.98:0/1502819707 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.410+0000 7f07c37fe640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f07b80a3430 0x7f07b80a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.410+0000 7f07c3fff640 1 --2- 172.21.15.98:0/1502819707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a56f50 0x5642f9a59340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.411+0000 7f07cad9d640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f07b80a3430 0x7f07b80a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.411+0000 7f07cad9d640 1 -- 172.21.15.98:0/1502819707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a56f50 msgr2=0x5642f9a59340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.411+0000 7f07cad9d640 1 --2- 172.21.15.98:0/1502819707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a56f50 0x5642f9a59340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.411+0000 7f07cad9d640 1 -- 172.21.15.98:0/1502819707 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 msgr2=0x5642f9a56a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.411+0000 7f07c37fe640 1 -- 172.21.15.98:0/1726321832 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f07b40cf770 con 0x7f07cc101430 2024-09-19T08:08:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.411+0000 7f07cad9d640 1 --2- 172.21.15.98:0/1502819707 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 0x5642f9a56a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.411+0000 7f07cad9d640 1 -- 172.21.15.98:0/1502819707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642f99631f0 con 0x5642f9a49a40 2024-09-19T08:08:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.411+0000 7f07cad9d640 1 --2- 172.21.15.98:0/1502819707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a49a40 0x5642f9a49e20 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f07c4004af0 tx=0x7f07c40319c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224cd50f3b3ce9e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.411+0000 7f07c0ff9640 1 -- 172.21.15.98:0/1502819707 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c4031430 con 0x5642f9a49a40 2024-09-19T08:08:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.411+0000 7f07c0ff9640 1 -- 172.21.15.98:0/1502819707 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07c4036040 con 0x5642f9a49a40 2024-09-19T08:08:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.411+0000 7f07c0ff9640 1 -- 172.21.15.98:0/1502819707 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c4030a80 con 0x5642f9a49a40 2024-09-19T08:08:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.411+0000 7f07cad9d640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f07b80a3430 0x7f07b80a58f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f07c404d000 tx=0x7f07c404b000 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.412+0000 7f07d2c3b740 1 -- 172.21.15.98:0/1502819707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a49a40 msgr2=0x5642f9a49e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.412+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/1502819707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a49a40 0x5642f9a49e20 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f07c4004af0 tx=0x7f07c40319c0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.412+0000 7f07d2c3b740 1 -- 172.21.15.98:0/1502819707 shutdown_connections 2024-09-19T08:08:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.412+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/1502819707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a56f50 0x5642f9a59340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.412+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/1502819707 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 0x5642f9a56a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.412+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/1502819707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a49a40 0x5642f9a49e20 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.412+0000 7f07d2c3b740 1 -- 172.21.15.98:0/1502819707 >> 172.21.15.98:0/1502819707 conn(0x5642f9a460f0 msgr2=0x5642f9a464d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.412+0000 7f07d2c3b740 1 -- 172.21.15.98:0/1502819707 shutdown_connections 2024-09-19T08:08:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.412+0000 7f07d2c3b740 1 -- 172.21.15.98:0/1502819707 wait complete. 2024-09-19T08:08:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.412+0000 7f07d2c3b740 1 -- start start 2024-09-19T08:08:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07d2c3b740 1 Processor -- start 2024-09-19T08:08:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07d2c3b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a49a40 0x5642f9a53ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07d2c3b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 0x5642f9a54400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07d2c3b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a56f50 0x5642f9a54940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07d2c3b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5642f9a59b20 con 0x5642f9a56f50 2024-09-19T08:08:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07d2c3b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5642f9969f10 con 0x5642f9a4a360 2024-09-19T08:08:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07d2c3b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5642f9a59ca0 con 0x5642f9a49a40 2024-09-19T08:08:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07d2c3b740 10 client.0 _ll_register_callbacks cb 0x5642f9932a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07cad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a56f50 0x5642f9a54940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07cad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a56f50 0x5642f9a54940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53378/0 (socket says 172.21.15.98:53378) 2024-09-19T08:08:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07cad9d640 1 -- 172.21.15.98:0/3381643799 learned_addr learned my addr 172.21.15.98:0/3381643799 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07ca59c640 1 --2- 172.21.15.98:0/3381643799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a49a40 0x5642f9a53ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07c3fff640 1 --2- 172.21.15.98:0/3381643799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 0x5642f9a54400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07cad9d640 1 -- 172.21.15.98:0/3381643799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a49a40 msgr2=0x5642f9a53ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07cad9d640 1 --2- 172.21.15.98:0/3381643799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a49a40 0x5642f9a53ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07cad9d640 1 -- 172.21.15.98:0/3381643799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 msgr2=0x5642f9a54400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07cad9d640 1 --2- 172.21.15.98:0/3381643799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 0x5642f9a54400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07cad9d640 1 -- 172.21.15.98:0/3381643799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642f97206a0 con 0x5642f9a56f50 2024-09-19T08:08:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07c3fff640 1 --2- 172.21.15.98:0/3381643799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 0x5642f9a54400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:08:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07cad9d640 1 --2- 172.21.15.98:0/3381643799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a56f50 0x5642f9a54940 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f07c4031b80 tx=0x7f07c4030640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca51f4e26c2525b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.413+0000 7f07b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.414+0000 7f07b1ffb640 1 -- 172.21.15.98:0/3381643799 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c4030860 con 0x5642f9a56f50 2024-09-19T08:08:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.414+0000 7f07b1ffb640 1 -- 172.21.15.98:0/3381643799 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f07c4030a00 con 0x5642f9a56f50 2024-09-19T08:08:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.414+0000 7f07d2c3b740 1 -- 172.21.15.98:0/3381643799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07c4036040 con 0x5642f9a56f50 2024-09-19T08:08:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.414+0000 7f07b1ffb640 1 -- 172.21.15.98:0/3381643799 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f07c4004b40 con 0x5642f9a56f50 2024-09-19T08:08:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.414+0000 7f07d2c3b740 10 client.16920 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.414+0000 7f07d2c3b740 1 -- 172.21.15.98:0/3381643799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5642f9a657e0 con 0x5642f9a56f50 2024-09-19T08:08:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.414+0000 7f07b1ffb640 1 -- 172.21.15.98:0/3381643799 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07c4036040 con 0x5642f9a56f50 2024-09-19T08:08:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.414+0000 7f07b1ffb640 1 -- 172.21.15.98:0/3381643799 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f07c4030070 con 0x5642f9a56f50 2024-09-19T08:08:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.414+0000 7f07d2c3b740 10 client.16920 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.414+0000 7f07d2c3b740 10 client.16920 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:08:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.414+0000 7f07d2c3b740 20 client.16920 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.414+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/3381643799 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5642f9a66b30 0x5642f9a68f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.414+0000 7f07d2c3b740 4 client.16920 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:08:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.414+0000 7f07d2c3b740 1 -- 172.21.15.98:0/3381643799 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642f99bc7f0 con 0x5642f9a66b30 2024-09-19T08:08:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.415+0000 7f07ca59c640 1 --2- 172.21.15.98:0/3381643799 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5642f9a66b30 0x5642f9a68f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.415+0000 7f07b1ffb640 10 client.16920 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.415+0000 7f07ca59c640 1 --2- 172.21.15.98:0/3381643799 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5642f9a66b30 0x5642f9a68f20 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6aadd6c74b29a031 server_cookie=5d2bb126ca499071 in_seq=0 out_seq=0 2024-09-19T08:08:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07b1ffb640 1 -- 172.21.15.98:0/3381643799 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5642f99bc7f0 con 0x5642f9a66b30 2024-09-19T08:08:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07b1ffb640 10 client.16920 handle_command_reply: tid=0 2024-09-19T08:08:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07d2c3b740 1 client.16920 shutdown 2024-09-19T08:08:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07d2c3b740 1 -- 172.21.15.98:0/3381643799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a56f50 msgr2=0x5642f9a54940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/3381643799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a56f50 0x5642f9a54940 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f07c4031b80 tx=0x7f07c4030640 comp rx=0 tx=0).stop 2024-09-19T08:08:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07d2c3b740 1 -- 172.21.15.98:0/3381643799 shutdown_connections 2024-09-19T08:08:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/3381643799 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5642f9a66b30 0x5642f9a68f20 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/3381643799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a56f50 0x5642f9a54940 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/3381643799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 0x5642f9a54400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/3381643799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a49a40 0x5642f9a53ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07b1ffb640 0 client.16920 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07d2c3b740 1 -- 172.21.15.98:0/3381643799 >> 172.21.15.98:0/3381643799 conn(0x5642f9a460f0 msgr2=0x5642f9a4be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07d2c3b740 1 -- 172.21.15.98:0/3381643799 shutdown_connections 2024-09-19T08:08:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07d2c3b740 1 -- 172.21.15.98:0/3381643799 wait complete. 2024-09-19T08:08:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.416+0000 7f07d2c3b740 20 client.16920 trim_cache size 0 max 16384 2024-09-19T08:08:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.433+0000 7f07d2c3b740 1 Processor -- start 2024-09-19T08:08:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.433+0000 7f07d2c3b740 1 -- start start 2024-09-19T08:08:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.433+0000 7f07d2c3b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a49a40 0x5642f9a54790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.433+0000 7f07d2c3b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 0x5642f9a62640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.433+0000 7f07d2c3b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a56f50 0x5642f9a51120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.433+0000 7f07d2c3b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f07cc1ab5c0 con 0x5642f9a56f50 2024-09-19T08:08:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.433+0000 7f07d2c3b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f07cc1ad100 con 0x5642f9a4a360 2024-09-19T08:08:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.433+0000 7f07d2c3b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f07cc1abc90 con 0x5642f9a49a40 2024-09-19T08:08:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.433+0000 7f07cad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a56f50 0x5642f9a51120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.433+0000 7f07cad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a56f50 0x5642f9a51120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53386/0 (socket says 172.21.15.98:53386) 2024-09-19T08:08:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.433+0000 7f07cad9d640 1 -- 172.21.15.98:0/4205098015 learned_addr learned my addr 172.21.15.98:0/4205098015 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07c3fff640 1 --2- 172.21.15.98:0/4205098015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 0x5642f9a62640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07ca59c640 1 --2- 172.21.15.98:0/4205098015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a49a40 0x5642f9a54790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07cad9d640 1 -- 172.21.15.98:0/4205098015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a49a40 msgr2=0x5642f9a54790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07cad9d640 1 --2- 172.21.15.98:0/4205098015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a49a40 0x5642f9a54790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07cad9d640 1 -- 172.21.15.98:0/4205098015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 msgr2=0x5642f9a62640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07cad9d640 1 --2- 172.21.15.98:0/4205098015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 0x5642f9a62640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07cad9d640 1 -- 172.21.15.98:0/4205098015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642f9a4ed20 con 0x5642f9a56f50 2024-09-19T08:08:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07ca59c640 1 --2- 172.21.15.98:0/4205098015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a49a40 0x5642f9a54790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:08:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07cad9d640 1 --2- 172.21.15.98:0/4205098015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a56f50 0x5642f9a51120 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f07c4002230 tx=0x7f07c4030f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826085a4144bdeaf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07b3fff640 1 -- 172.21.15.98:0/4205098015 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c4003d40 con 0x5642f9a56f50 2024-09-19T08:08:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07b3fff640 1 -- 172.21.15.98:0/4205098015 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07c406b020 con 0x5642f9a56f50 2024-09-19T08:08:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07b3fff640 1 -- 172.21.15.98:0/4205098015 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c405f290 con 0x5642f9a56f50 2024-09-19T08:08:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07d2c3b740 1 -- 172.21.15.98:0/4205098015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a56f50 msgr2=0x5642f9a51120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/4205098015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a56f50 0x5642f9a51120 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f07c4002230 tx=0x7f07c4030f10 comp rx=0 tx=0).stop 2024-09-19T08:08:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07d2c3b740 1 -- 172.21.15.98:0/4205098015 shutdown_connections 2024-09-19T08:08:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.434+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/4205098015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a56f50 0x5642f9a51120 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/4205098015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a4a360 0x5642f9a62640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/4205098015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a49a40 0x5642f9a54790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07d2c3b740 1 -- 172.21.15.98:0/4205098015 >> 172.21.15.98:0/4205098015 conn(0x5642f9a460f0 msgr2=0x5642f9a61f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07d2c3b740 1 -- 172.21.15.98:0/4205098015 shutdown_connections 2024-09-19T08:08:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07d2c3b740 1 -- 172.21.15.98:0/4205098015 wait complete. 2024-09-19T08:08:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07d2c3b740 1 -- start start 2024-09-19T08:08:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07d2c3b740 1 Processor -- start 2024-09-19T08:08:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07d2c3b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a49a40 0x5642f9a6c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07d2c3b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a4a360 0x5642f9a6c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07d2c3b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a56f50 0x5642f9a6a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07d2c3b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f07cc1aca30 con 0x5642f9a49a40 2024-09-19T08:08:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07d2c3b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f07cc1ac360 con 0x5642f9a56f50 2024-09-19T08:08:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07d2c3b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f07cc1a95a0 con 0x5642f9a4a360 2024-09-19T08:08:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07d2c3b740 10 client.0 _ll_register_callbacks cb 0x7f07c4031780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07ca59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a49a40 0x5642f9a6c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07ca59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a49a40 0x5642f9a6c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53392/0 (socket says 172.21.15.98:53392) 2024-09-19T08:08:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07ca59c640 1 -- 172.21.15.98:0/850585012 learned_addr learned my addr 172.21.15.98:0/850585012 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07c3fff640 1 --2- 172.21.15.98:0/850585012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a4a360 0x5642f9a6c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.435+0000 7f07cad9d640 1 --2- 172.21.15.98:0/850585012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a56f50 0x5642f9a6a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07ca59c640 1 -- 172.21.15.98:0/850585012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a4a360 msgr2=0x5642f9a6c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07ca59c640 1 --2- 172.21.15.98:0/850585012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a4a360 0x5642f9a6c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07ca59c640 1 -- 172.21.15.98:0/850585012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a56f50 msgr2=0x5642f9a6a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07ca59c640 1 --2- 172.21.15.98:0/850585012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a56f50 0x5642f9a6a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07ca59c640 1 -- 172.21.15.98:0/850585012 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642f9a6ad50 con 0x5642f9a49a40 2024-09-19T08:08:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07cad9d640 1 --2- 172.21.15.98:0/850585012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a56f50 0x5642f9a6a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:08:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07ca59c640 1 --2- 172.21.15.98:0/850585012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a49a40 0x5642f9a6c130 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f07bc009ad0 tx=0x7f07bc012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acc21e4945a0647 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07b1ffb640 1 -- 172.21.15.98:0/850585012 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07bc01f070 con 0x5642f9a49a40 2024-09-19T08:08:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07b1ffb640 1 -- 172.21.15.98:0/850585012 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f07bc004220 con 0x5642f9a49a40 2024-09-19T08:08:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07d2c3b740 1 -- 172.21.15.98:0/850585012 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07c406b020 con 0x5642f9a49a40 2024-09-19T08:08:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07b1ffb640 1 -- 172.21.15.98:0/850585012 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f07bc02a050 con 0x5642f9a49a40 2024-09-19T08:08:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07d2c3b740 10 client.16926 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07d2c3b740 1 -- 172.21.15.98:0/850585012 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5642f9ad1fc0 con 0x5642f9a49a40 2024-09-19T08:08:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07b1ffb640 1 -- 172.21.15.98:0/850585012 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07c406b020 con 0x5642f9a49a40 2024-09-19T08:08:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07b1ffb640 1 -- 172.21.15.98:0/850585012 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f07bc00f040 con 0x5642f9a49a40 2024-09-19T08:08:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07d2c3b740 10 client.16926 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07d2c3b740 10 client.16926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:08:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07d2c3b740 20 client.16926 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/850585012 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5642f9ad26e0 0x5642f9ad4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07d2c3b740 4 client.16926 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:08:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.436+0000 7f07d2c3b740 1 -- 172.21.15.98:0/850585012 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5642f99b6d00 con 0x5642f9ad26e0 2024-09-19T08:08:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.437+0000 7f07c3fff640 1 --2- 172.21.15.98:0/850585012 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5642f9ad26e0 0x5642f9ad4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.437+0000 7f07b1ffb640 10 client.16926 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.437+0000 7f07c3fff640 1 --2- 172.21.15.98:0/850585012 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5642f9ad26e0 0x5642f9ad4ad0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9d929a7d91bdba server_cookie=5fc62ee5194a67ae in_seq=0 out_seq=0 2024-09-19T08:08:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07b1ffb640 1 -- 172.21.15.98:0/850585012 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5642f99b6d00 con 0x5642f9ad26e0 2024-09-19T08:08:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07b1ffb640 10 client.16926 handle_command_reply: tid=0 2024-09-19T08:08:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07d2c3b740 1 client.16926 shutdown 2024-09-19T08:08:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07d2c3b740 1 -- 172.21.15.98:0/850585012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a49a40 msgr2=0x5642f9a6c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/850585012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a49a40 0x5642f9a6c130 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f07bc009ad0 tx=0x7f07bc012820 comp rx=0 tx=0).stop 2024-09-19T08:08:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07d2c3b740 1 -- 172.21.15.98:0/850585012 shutdown_connections 2024-09-19T08:08:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/850585012 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5642f9ad26e0 0x5642f9ad4ad0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/850585012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5642f9a56f50 0x5642f9a6a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/850585012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642f9a4a360 0x5642f9a6c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07b1ffb640 0 client.16926 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07d2c3b740 1 --2- 172.21.15.98:0/850585012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642f9a49a40 0x5642f9a6c130 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07d2c3b740 1 -- 172.21.15.98:0/850585012 >> 172.21.15.98:0/850585012 conn(0x5642f9a460f0 msgr2=0x5642f9a62750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07d2c3b740 1 -- 172.21.15.98:0/850585012 shutdown_connections 2024-09-19T08:08:42.456 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:08:42.456 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:08:42.456 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "a1ba20d0-ccd9-4e1b-ba27-db4320c183b5", 2024-09-19T08:08:42.456 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:08:42.456 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:08:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07d2c3b740 1 -- 172.21.15.98:0/850585012 wait complete. 2024-09-19T08:08:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.438+0000 7f07d2c3b740 20 client.16926 trim_cache size 0 max 16384 2024-09-19T08:08:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.439+0000 7f07d0862640 1 -- 172.21.15.98:0/1726321832 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f07b80a3430 msgr2=0x7f07b80a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.439+0000 7f07d0862640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f07b80a3430 0x7f07b80a58f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f07c404d000 tx=0x7f07c404b000 comp rx=0 tx=0).stop 2024-09-19T08:08:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.439+0000 7f07d0862640 1 -- 172.21.15.98:0/1726321832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07cc101430 msgr2=0x7f07cc197830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.439+0000 7f07d0862640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07cc101430 0x7f07cc197830 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f07b400d770 tx=0x7f07b400dc30 comp rx=0 tx=0).stop 2024-09-19T08:08:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.439+0000 7f07d0862640 1 -- 172.21.15.98:0/1726321832 shutdown_connections 2024-09-19T08:08:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.439+0000 7f07d0862640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f07b80a3430 0x7f07b80a58f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.439+0000 7f07d0862640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07cc106a90 0x7f07cc19bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.439+0000 7f07d0862640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07cc101430 0x7f07cc197830 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.439+0000 7f07d0862640 1 --2- 172.21.15.98:0/1726321832 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07cc100a90 0x7f07cc1972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:42.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.439+0000 7f07d0862640 1 -- 172.21.15.98:0/1726321832 >> 172.21.15.98:0/1726321832 conn(0x7f07cc0fc7d0 msgr2=0x7f07cc0fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:42.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.439+0000 7f07d0862640 1 -- 172.21.15.98:0/1726321832 shutdown_connections 2024-09-19T08:08:42.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:42.439+0000 7f07d0862640 1 -- 172.21.15.98:0/1726321832 wait complete. 2024-09-19T08:08:43.171 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:08:43.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:43 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:08:43.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:43 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:08:43.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:43 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:08:43.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:43 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:08:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:08:43.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:43 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:08:43.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:43 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:08:43.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:43 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:08:43.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:43 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:08:43.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:43 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:08:43.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:43 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:08:43.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:43 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:08:43.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:43 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:08:43.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.879+0000 7fd76e79a640 1 -- 172.21.15.98:0/1661973014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd768102790 msgr2=0x7fd7681067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:43.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.879+0000 7fd76e79a640 1 --2- 172.21.15.98:0/1661973014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd768102790 0x7fd7681067e0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009900 tx=0x7fd75c02a410 comp rx=0 tx=0).stop 2024-09-19T08:08:43.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.879+0000 7fd76e79a640 1 -- 172.21.15.98:0/1661973014 shutdown_connections 2024-09-19T08:08:43.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.879+0000 7fd76e79a640 1 --2- 172.21.15.98:0/1661973014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd768106f10 0x7fd76810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.879+0000 7fd76e79a640 1 --2- 172.21.15.98:0/1661973014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd768102790 0x7fd7681067e0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.879+0000 7fd76e79a640 1 --2- 172.21.15.98:0/1661973014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd768101de0 0x7fd7681021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.879+0000 7fd76e79a640 1 -- 172.21.15.98:0/1661973014 >> 172.21.15.98:0/1661973014 conn(0x7fd7680fd8c0 msgr2=0x7fd7680ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:43.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.879+0000 7fd76e79a640 1 -- 172.21.15.98:0/1661973014 shutdown_connections 2024-09-19T08:08:43.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.879+0000 7fd76e79a640 1 -- 172.21.15.98:0/1661973014 wait complete. 2024-09-19T08:08:43.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.880+0000 7fd76e79a640 1 Processor -- start 2024-09-19T08:08:43.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.880+0000 7fd76e79a640 1 -- start start 2024-09-19T08:08:43.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.880+0000 7fd76e79a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd768101de0 0x7fd7680744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.880+0000 7fd76e79a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd768102790 0x7fd768074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.880+0000 7fd76e79a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd768106f10 0x7fd768074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.880+0000 7fd76e79a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd768111f90 con 0x7fd768102790 2024-09-19T08:08:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.880+0000 7fd76e79a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd768111e10 con 0x7fd768101de0 2024-09-19T08:08:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.880+0000 7fd76e79a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd768112110 con 0x7fd768106f10 2024-09-19T08:08:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.880+0000 7fd7677fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd768102790 0x7fd768074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.880+0000 7fd7677fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd768102790 0x7fd768074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53412/0 (socket says 172.21.15.98:53412) 2024-09-19T08:08:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.880+0000 7fd7677fe640 1 -- 172.21.15.98:0/931327617 learned_addr learned my addr 172.21.15.98:0/931327617 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.881+0000 7fd767fff640 1 --2- 172.21.15.98:0/931327617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd768101de0 0x7fd7680744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.881+0000 7fd76cd37640 1 --2- 172.21.15.98:0/931327617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd768106f10 0x7fd768074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.881+0000 7fd7677fe640 1 -- 172.21.15.98:0/931327617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd768106f10 msgr2=0x7fd768074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.881+0000 7fd7677fe640 1 --2- 172.21.15.98:0/931327617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd768106f10 0x7fd768074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.881+0000 7fd7677fe640 1 -- 172.21.15.98:0/931327617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd768101de0 msgr2=0x7fd7680744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.881+0000 7fd7677fe640 1 --2- 172.21.15.98:0/931327617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd768101de0 0x7fd7680744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.881+0000 7fd7677fe640 1 -- 172.21.15.98:0/931327617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7681a68e0 con 0x7fd768102790 2024-09-19T08:08:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.881+0000 7fd7677fe640 1 --2- 172.21.15.98:0/931327617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd768102790 0x7fd768074a30 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fd75c005ab0 tx=0x7fd75c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.881+0000 7fd7657fa640 1 -- 172.21.15.98:0/931327617 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c03f030 con 0x7fd768102790 2024-09-19T08:08:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.881+0000 7fd7657fa640 1 -- 172.21.15.98:0/931327617 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd75c003ce0 con 0x7fd768102790 2024-09-19T08:08:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.881+0000 7fd76e79a640 1 -- 172.21.15.98:0/931327617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7681a6b70 con 0x7fd768102790 2024-09-19T08:08:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.881+0000 7fd7657fa640 1 -- 172.21.15.98:0/931327617 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c035070 con 0x7fd768102790 2024-09-19T08:08:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.881+0000 7fd76e79a640 1 -- 172.21.15.98:0/931327617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7681a7050 con 0x7fd768102790 2024-09-19T08:08:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.883+0000 7fd7657fa640 1 -- 172.21.15.98:0/931327617 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fd75c042030 con 0x7fd768102790 2024-09-19T08:08:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd770b73740 1 Processor -- start 2024-09-19T08:08:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd770b73740 1 -- start start 2024-09-19T08:08:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd770b73740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23b74a0 0x55f2d23b7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd770b73740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b7dc0 0x55f2d23c2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd770b73740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23c2680 0x55f2d23c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd770b73740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f2d231d2e0 con 0x55f2d23b7dc0 2024-09-19T08:08:43.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd770b73740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f2d22c4ce0 con 0x55f2d23b74a0 2024-09-19T08:08:43.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd770b73740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f2d23192a0 con 0x55f2d23c2680 2024-09-19T08:08:43.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd7657fa640 1 --2- 172.21.15.98:0/931327617 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd7500a33b0 0x7fd7500a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd767fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b7dc0 0x55f2d23c2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd767fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b7dc0 0x55f2d23c2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53428/0 (socket says 172.21.15.98:53428) 2024-09-19T08:08:43.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd767fff640 1 -- 172.21.15.98:0/1433447251 learned_addr learned my addr 172.21.15.98:0/1433447251 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:43.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd76cd37640 1 --2- 172.21.15.98:0/931327617 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd7500a33b0 0x7fd7500a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd76cd37640 1 --2- 172.21.15.98:0/1433447251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23b74a0 0x55f2d23b7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd7657fa640 1 -- 172.21.15.98:0/931327617 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fd75c0f06b0 con 0x7fd768102790 2024-09-19T08:08:43.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd7677fe640 1 --2- 172.21.15.98:0/1433447251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23c2680 0x55f2d23c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd767fff640 1 -- 172.21.15.98:0/1433447251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23c2680 msgr2=0x55f2d23c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:43.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd767fff640 1 --2- 172.21.15.98:0/1433447251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23c2680 0x55f2d23c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd767fff640 1 -- 172.21.15.98:0/1433447251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23b74a0 msgr2=0x55f2d23b7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:43.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd767fff640 1 --2- 172.21.15.98:0/1433447251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23b74a0 0x55f2d23b7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.884+0000 7fd767fff640 1 -- 172.21.15.98:0/1433447251 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2d22a9440 con 0x55f2d23b7dc0 2024-09-19T08:08:43.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.885+0000 7fd76cd37640 1 --2- 172.21.15.98:0/931327617 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd7500a33b0 0x7fd7500a5870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd760009fd0 tx=0x7fd760009d80 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:43.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.885+0000 7fd767fff640 1 --2- 172.21.15.98:0/1433447251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b7dc0 0x55f2d23c2140 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fd75800cc60 tx=0x7fd75800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66c715bd95df2f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:43.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.885+0000 7fd74e7fc640 1 -- 172.21.15.98:0/1433447251 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd758014070 con 0x55f2d23b7dc0 2024-09-19T08:08:43.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.885+0000 7fd74e7fc640 1 -- 172.21.15.98:0/1433447251 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7580047a0 con 0x55f2d23b7dc0 2024-09-19T08:08:43.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.885+0000 7fd74e7fc640 1 -- 172.21.15.98:0/1433447251 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7580106c0 con 0x55f2d23b7dc0 2024-09-19T08:08:43.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.885+0000 7fd770b73740 1 -- 172.21.15.98:0/1433447251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b7dc0 msgr2=0x55f2d23c2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:43.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.885+0000 7fd770b73740 1 --2- 172.21.15.98:0/1433447251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b7dc0 0x55f2d23c2140 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fd75800cc60 tx=0x7fd75800e720 comp rx=0 tx=0).stop 2024-09-19T08:08:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.886+0000 7fd770b73740 1 -- 172.21.15.98:0/1433447251 shutdown_connections 2024-09-19T08:08:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.886+0000 7fd770b73740 1 --2- 172.21.15.98:0/1433447251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23c2680 0x55f2d23c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.886+0000 7fd770b73740 1 --2- 172.21.15.98:0/1433447251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b7dc0 0x55f2d23c2140 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.886+0000 7fd770b73740 1 --2- 172.21.15.98:0/1433447251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23b74a0 0x55f2d23b7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.886+0000 7fd770b73740 1 -- 172.21.15.98:0/1433447251 >> 172.21.15.98:0/1433447251 conn(0x55f2d23b3a30 msgr2=0x55f2d23b3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.886+0000 7fd770b73740 1 -- 172.21.15.98:0/1433447251 shutdown_connections 2024-09-19T08:08:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.886+0000 7fd770b73740 1 -- 172.21.15.98:0/1433447251 wait complete. 2024-09-19T08:08:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.886+0000 7fd770b73740 1 -- start start 2024-09-19T08:08:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.886+0000 7fd770b73740 1 Processor -- start 2024-09-19T08:08:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.886+0000 7fd770b73740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.886+0000 7fd770b73740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23b7dc0 0x55f2d23bf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.886+0000 7fd770b73740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23c2680 0x55f2d23bf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd770b73740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f2d23c5250 con 0x55f2d23b74a0 2024-09-19T08:08:43.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd770b73740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f2d22d8050 con 0x55f2d23b7dc0 2024-09-19T08:08:43.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd770b73740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f2d23c53d0 con 0x55f2d23c2680 2024-09-19T08:08:43.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd770b73740 10 client.0 _ll_register_callbacks cb 0x55f2d2318160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:43.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd767fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd767fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23c1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53432/0 (socket says 172.21.15.98:53432) 2024-09-19T08:08:43.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd767fff640 1 -- 172.21.15.98:0/3023244093 learned_addr learned my addr 172.21.15.98:0/3023244093 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:43.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd76cd37640 1 --2- 172.21.15.98:0/3023244093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23c2680 0x55f2d23bf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd767fff640 1 -- 172.21.15.98:0/3023244093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23c2680 msgr2=0x55f2d23bf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:43.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd7677fe640 1 --2- 172.21.15.98:0/3023244093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23b7dc0 0x55f2d23bf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd767fff640 1 --2- 172.21.15.98:0/3023244093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23c2680 0x55f2d23bf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd767fff640 1 -- 172.21.15.98:0/3023244093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23b7dc0 msgr2=0x55f2d23bf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:43.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd767fff640 1 --2- 172.21.15.98:0/3023244093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23b7dc0 0x55f2d23bf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd767fff640 1 -- 172.21.15.98:0/3023244093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2d208e3c0 con 0x55f2d23b74a0 2024-09-19T08:08:43.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd7677fe640 1 --2- 172.21.15.98:0/3023244093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23b7dc0 0x55f2d23bf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:08:43.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd767fff640 1 --2- 172.21.15.98:0/3023244093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23c1f50 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fd75800c9d0 tx=0x7fd7580046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3efa196baf4e611e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:43.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd7477fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:43.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd7477fe640 1 -- 172.21.15.98:0/3023244093 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd758027060 con 0x55f2d23b74a0 2024-09-19T08:08:43.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd7477fe640 1 -- 172.21.15.98:0/3023244093 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fd758005210 con 0x55f2d23b74a0 2024-09-19T08:08:43.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd770b73740 1 -- 172.21.15.98:0/3023244093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7580047a0 con 0x55f2d23b74a0 2024-09-19T08:08:43.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd7477fe640 1 -- 172.21.15.98:0/3023244093 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd758014070 con 0x55f2d23b74a0 2024-09-19T08:08:43.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd770b73740 10 client.16938 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:43.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.887+0000 7fd770b73740 1 -- 172.21.15.98:0/3023244093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f2d23d0ba0 con 0x55f2d23b74a0 2024-09-19T08:08:43.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.888+0000 7fd7477fe640 1 -- 172.21.15.98:0/3023244093 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7580047a0 con 0x55f2d23b74a0 2024-09-19T08:08:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.888+0000 7fd7477fe640 1 -- 172.21.15.98:0/3023244093 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd75802b430 con 0x55f2d23b74a0 2024-09-19T08:08:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.888+0000 7fd770b73740 10 client.16938 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.888+0000 7fd770b73740 10 client.16938 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:08:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.888+0000 7fd770b73740 20 client.16938 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.888+0000 7fd770b73740 1 --2- 172.21.15.98:0/3023244093 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55f2d23d2060 0x55f2d23d4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.888+0000 7fd770b73740 4 client.16938 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:08:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.888+0000 7fd770b73740 1 -- 172.21.15.98:0/3023244093 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2d232a850 con 0x55f2d23d2060 2024-09-19T08:08:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.888+0000 7fd7677fe640 1 --2- 172.21.15.98:0/3023244093 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55f2d23d2060 0x55f2d23d4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.888+0000 7fd7477fe640 10 client.16938 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.888+0000 7fd7677fe640 1 --2- 172.21.15.98:0/3023244093 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55f2d23d2060 0x55f2d23d4450 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b6496a8642e3125 server_cookie=365bde13b93328d7 in_seq=0 out_seq=0 2024-09-19T08:08:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.889+0000 7fd7477fe640 1 -- 172.21.15.98:0/3023244093 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f2d232a850 con 0x55f2d23d2060 2024-09-19T08:08:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.889+0000 7fd7477fe640 10 client.16938 handle_command_reply: tid=0 2024-09-19T08:08:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.889+0000 7fd770b73740 1 client.16938 shutdown 2024-09-19T08:08:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.889+0000 7fd770b73740 1 -- 172.21.15.98:0/3023244093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 msgr2=0x55f2d23c1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.889+0000 7fd770b73740 1 --2- 172.21.15.98:0/3023244093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23c1f50 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fd75800c9d0 tx=0x7fd7580046e0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.889+0000 7fd770b73740 1 -- 172.21.15.98:0/3023244093 shutdown_connections 2024-09-19T08:08:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.889+0000 7fd770b73740 1 --2- 172.21.15.98:0/3023244093 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55f2d23d2060 0x55f2d23d4450 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.889+0000 7fd770b73740 1 --2- 172.21.15.98:0/3023244093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23c2680 0x55f2d23bf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.889+0000 7fd770b73740 1 --2- 172.21.15.98:0/3023244093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23b7dc0 0x55f2d23bf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.889+0000 7fd7477fe640 0 client.16938 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.889+0000 7fd770b73740 1 --2- 172.21.15.98:0/3023244093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23c1f50 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.889+0000 7fd770b73740 1 -- 172.21.15.98:0/3023244093 >> 172.21.15.98:0/3023244093 conn(0x55f2d23b3a30 msgr2=0x55f2d23b9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.889+0000 7fd770b73740 1 -- 172.21.15.98:0/3023244093 shutdown_connections 2024-09-19T08:08:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.890+0000 7fd770b73740 1 -- 172.21.15.98:0/3023244093 wait complete. 2024-09-19T08:08:43.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.890+0000 7fd770b73740 20 client.16938 trim_cache size 0 max 16384 2024-09-19T08:08:43.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.906+0000 7fd770b73740 1 Processor -- start 2024-09-19T08:08:43.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.906+0000 7fd770b73740 1 -- start start 2024-09-19T08:08:43.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.906+0000 7fd770b73740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.906+0000 7fd770b73740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23b7dc0 0x55f2d23ba900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.906+0000 7fd770b73740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23c2680 0x55f2d23be1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.906+0000 7fd770b73740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd7681af9f0 con 0x55f2d23b74a0 2024-09-19T08:08:43.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.906+0000 7fd770b73740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd7681b1530 con 0x55f2d23c2680 2024-09-19T08:08:43.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.906+0000 7fd770b73740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd7681b00c0 con 0x55f2d23b7dc0 2024-09-19T08:08:43.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.906+0000 7fd767fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.906+0000 7fd767fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53442/0 (socket says 172.21.15.98:53442) 2024-09-19T08:08:43.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.907+0000 7fd767fff640 1 -- 172.21.15.98:0/879271677 learned_addr learned my addr 172.21.15.98:0/879271677 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:43.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.907+0000 7fd7677fe640 1 --2- 172.21.15.98:0/879271677 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23b7dc0 0x55f2d23ba900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.907+0000 7fd767fff640 1 -- 172.21.15.98:0/879271677 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23b7dc0 msgr2=0x55f2d23ba900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:43.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.907+0000 7fd767fff640 1 --2- 172.21.15.98:0/879271677 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23b7dc0 0x55f2d23ba900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.907+0000 7fd767fff640 1 -- 172.21.15.98:0/879271677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23c2680 msgr2=0x55f2d23be1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.907+0000 7fd767fff640 1 --2- 172.21.15.98:0/879271677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23c2680 0x55f2d23be1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.907+0000 7fd767fff640 1 -- 172.21.15.98:0/879271677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2d23c5380 con 0x55f2d23b74a0 2024-09-19T08:08:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.907+0000 7fd767fff640 1 --2- 172.21.15.98:0/879271677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23ce4b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fd75800c770 tx=0x7fd75800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4bc890e98d9103 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.907+0000 7fd74dffb640 1 -- 172.21.15.98:0/879271677 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd758004750 con 0x55f2d23b74a0 2024-09-19T08:08:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.907+0000 7fd74dffb640 1 -- 172.21.15.98:0/879271677 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd75802bba0 con 0x55f2d23b74a0 2024-09-19T08:08:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.907+0000 7fd74dffb640 1 -- 172.21.15.98:0/879271677 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd758034720 con 0x55f2d23b74a0 2024-09-19T08:08:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 -- 172.21.15.98:0/879271677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 msgr2=0x55f2d23ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 --2- 172.21.15.98:0/879271677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23ce4b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fd75800c770 tx=0x7fd75800ec60 comp rx=0 tx=0).stop 2024-09-19T08:08:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 -- 172.21.15.98:0/879271677 shutdown_connections 2024-09-19T08:08:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 --2- 172.21.15.98:0/879271677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23c2680 0x55f2d23be1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 --2- 172.21.15.98:0/879271677 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23b7dc0 0x55f2d23ba900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 --2- 172.21.15.98:0/879271677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23ce4b0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 -- 172.21.15.98:0/879271677 >> 172.21.15.98:0/879271677 conn(0x55f2d23b3a30 msgr2=0x55f2d23bdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:43.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 -- 172.21.15.98:0/879271677 shutdown_connections 2024-09-19T08:08:43.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 -- 172.21.15.98:0/879271677 wait complete. 2024-09-19T08:08:43.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 -- start start 2024-09-19T08:08:43.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 Processor -- start 2024-09-19T08:08:43.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23ca980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23b7dc0 0x55f2d23caec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23c2680 0x55f2d23c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd7681b0e60 con 0x55f2d23b74a0 2024-09-19T08:08:43.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd7681b0790 con 0x55f2d23c2680 2024-09-19T08:08:43.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.908+0000 7fd770b73740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd7681ad9d0 con 0x55f2d23b7dc0 2024-09-19T08:08:43.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd770b73740 10 client.0 _ll_register_callbacks cb 0x7fd76000a6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:08:43.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd767fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23ca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd767fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23ca980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53450/0 (socket says 172.21.15.98:53450) 2024-09-19T08:08:43.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd767fff640 1 -- 172.21.15.98:0/1041301177 learned_addr learned my addr 172.21.15.98:0/1041301177 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:08:43.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd7677fe640 1 --2- 172.21.15.98:0/1041301177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23b7dc0 0x55f2d23caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd76cd37640 1 --2- 172.21.15.98:0/1041301177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23c2680 0x55f2d23c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd767fff640 1 -- 172.21.15.98:0/1041301177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23b7dc0 msgr2=0x55f2d23caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:43.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd767fff640 1 --2- 172.21.15.98:0/1041301177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23b7dc0 0x55f2d23caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd767fff640 1 -- 172.21.15.98:0/1041301177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23c2680 msgr2=0x55f2d23c5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:08:43.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd767fff640 1 --2- 172.21.15.98:0/1041301177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23c2680 0x55f2d23c5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd767fff640 1 -- 172.21.15.98:0/1041301177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2d23b3860 con 0x55f2d23b74a0 2024-09-19T08:08:43.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd767fff640 1 --2- 172.21.15.98:0/1041301177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23ca980 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fd75800f010 tx=0x7fd75800c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb4113630dda56a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:08:43.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd7477fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:08:43.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd7477fe640 1 -- 172.21.15.98:0/1041301177 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd758038070 con 0x55f2d23b74a0 2024-09-19T08:08:43.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd7477fe640 1 -- 172.21.15.98:0/1041301177 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fd758005420 con 0x55f2d23b74a0 2024-09-19T08:08:43.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd770b73740 1 -- 172.21.15.98:0/1041301177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd75802bba0 con 0x55f2d23b74a0 2024-09-19T08:08:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd7477fe640 1 -- 172.21.15.98:0/1041301177 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd758043050 con 0x55f2d23b74a0 2024-09-19T08:08:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd770b73740 10 client.16944 fetch_fsmap learned FSMap version 32 2024-09-19T08:08:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.909+0000 7fd770b73740 1 -- 172.21.15.98:0/1041301177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f2d2444c70 con 0x55f2d23b74a0 2024-09-19T08:08:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.910+0000 7fd7477fe640 1 -- 172.21.15.98:0/1041301177 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd75802bba0 con 0x55f2d23b74a0 2024-09-19T08:08:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.910+0000 7fd7477fe640 1 -- 172.21.15.98:0/1041301177 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd758034b20 con 0x55f2d23b74a0 2024-09-19T08:08:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.910+0000 7fd770b73740 10 client.16944 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:08:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.910+0000 7fd770b73740 10 client.16944 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:08:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.910+0000 7fd770b73740 20 client.16944 populate_metadata read hostname 'smithi098' 2024-09-19T08:08:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.910+0000 7fd770b73740 1 --2- 172.21.15.98:0/1041301177 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55f2d2445320 0x55f2d2447710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:08:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.910+0000 7fd770b73740 4 client.16944 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:08:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.910+0000 7fd770b73740 1 -- 172.21.15.98:0/1041301177 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f2d2324d60 con 0x55f2d2445320 2024-09-19T08:08:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.910+0000 7fd7677fe640 1 --2- 172.21.15.98:0/1041301177 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55f2d2445320 0x55f2d2447710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:08:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.910+0000 7fd7477fe640 10 client.16944 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.910+0000 7fd7677fe640 1 --2- 172.21.15.98:0/1041301177 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55f2d2445320 0x55f2d2447710 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8f281a0ae71f881 server_cookie=7962b4bb9905f574 in_seq=0 out_seq=0 2024-09-19T08:08:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd7477fe640 1 -- 172.21.15.98:0/1041301177 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f2d2324d60 con 0x55f2d2445320 2024-09-19T08:08:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd7477fe640 10 client.16944 handle_command_reply: tid=0 2024-09-19T08:08:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd770b73740 1 client.16944 shutdown 2024-09-19T08:08:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd770b73740 1 -- 172.21.15.98:0/1041301177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 msgr2=0x55f2d23ca980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd770b73740 1 --2- 172.21.15.98:0/1041301177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23ca980 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fd75800f010 tx=0x7fd75800c3b0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd770b73740 1 -- 172.21.15.98:0/1041301177 shutdown_connections 2024-09-19T08:08:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd770b73740 1 --2- 172.21.15.98:0/1041301177 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55f2d2445320 0x55f2d2447710 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd770b73740 1 --2- 172.21.15.98:0/1041301177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2d23c2680 0x55f2d23c5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd770b73740 1 --2- 172.21.15.98:0/1041301177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d23b7dc0 0x55f2d23caec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.933 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:08:43.933 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-19T08:08:43.933 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-09-19T08:08:43.933 INFO:teuthology.orchestra.run.smithi098.stdout: "a1ba20d0-ccd9-4e1b-ba27-db4320c183b5": { 2024-09-19T08:08:43.933 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-09-19T08:08:43.933 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "a1ba20d0-ccd9-4e1b-ba27-db4320c183b5", 2024-09-19T08:08:43.933 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-09-19T08:08:43.933 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:08:43.933 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:08:43.933 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:08:43.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd770b73740 1 --2- 172.21.15.98:0/1041301177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2d23b74a0 0x55f2d23ca980 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd7477fe640 0 client.16944 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:08:43.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd770b73740 1 -- 172.21.15.98:0/1041301177 >> 172.21.15.98:0/1041301177 conn(0x55f2d23b3a30 msgr2=0x55f2d23b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:43.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd770b73740 1 -- 172.21.15.98:0/1041301177 shutdown_connections 2024-09-19T08:08:43.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd770b73740 1 -- 172.21.15.98:0/1041301177 wait complete. 2024-09-19T08:08:43.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd770b73740 20 client.16944 trim_cache size 0 max 16384 2024-09-19T08:08:43.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd76e79a640 1 -- 172.21.15.98:0/931327617 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd7500a33b0 msgr2=0x7fd7500a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:43.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd76e79a640 1 --2- 172.21.15.98:0/931327617 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd7500a33b0 0x7fd7500a5870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd760009fd0 tx=0x7fd760009d80 comp rx=0 tx=0).stop 2024-09-19T08:08:43.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd76e79a640 1 -- 172.21.15.98:0/931327617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd768102790 msgr2=0x7fd768074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:08:43.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd76e79a640 1 --2- 172.21.15.98:0/931327617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd768102790 0x7fd768074a30 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fd75c005ab0 tx=0x7fd75c0042f0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd76e79a640 1 -- 172.21.15.98:0/931327617 shutdown_connections 2024-09-19T08:08:43.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd76e79a640 1 --2- 172.21.15.98:0/931327617 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd7500a33b0 0x7fd7500a5870 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd76e79a640 1 --2- 172.21.15.98:0/931327617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd768106f10 0x7fd768074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd76e79a640 1 --2- 172.21.15.98:0/931327617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd768102790 0x7fd768074a30 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.911+0000 7fd76e79a640 1 --2- 172.21.15.98:0/931327617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd768101de0 0x7fd7680744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:08:43.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.912+0000 7fd76e79a640 1 -- 172.21.15.98:0/931327617 >> 172.21.15.98:0/931327617 conn(0x7fd7680fd8c0 msgr2=0x7fd7680fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:08:43.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.912+0000 7fd76e79a640 1 -- 172.21.15.98:0/931327617 shutdown_connections 2024-09-19T08:08:43.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:08:43.912+0000 7fd76e79a640 1 -- 172.21.15.98:0/931327617 wait complete. 2024-09-19T08:08:44.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:44 smithi098 ceph-mon[31333]: pgmap v174: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-09-19T08:08:44.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:44 smithi162 ceph-mon[35260]: pgmap v174: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-09-19T08:08:44.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:44 smithi139 ceph-mon[34874]: pgmap v174: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-09-19T08:08:44.800 INFO:tasks.cephfs.filesystem:scrub status for tag:a1ba20d0-ccd9-4e1b-ba27-db4320c183b5 - {'path': '/', 'tag': 'a1ba20d0-ccd9-4e1b-ba27-db4320c183b5', 'options': 'recursive,force'} 2024-09-19T08:08:46.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:46 smithi162 ceph-mon[35260]: pgmap v175: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:46.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:46 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:08:46.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:46 smithi139 ceph-mon[34874]: pgmap v175: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:46.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:46 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:08:46.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:46 smithi098 ceph-mon[31333]: pgmap v175: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:46.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:46 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:08:48.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:48 smithi162 ceph-mon[35260]: pgmap v176: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-09-19T08:08:48.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:48 smithi139 ceph-mon[34874]: pgmap v176: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-09-19T08:08:48.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:48 smithi098 ceph-mon[31333]: pgmap v176: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-09-19T08:08:50.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:50 smithi162 ceph-mon[35260]: pgmap v177: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-09-19T08:08:50.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:50 smithi139 ceph-mon[34874]: pgmap v177: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-09-19T08:08:50.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:50 smithi098 ceph-mon[31333]: pgmap v177: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-09-19T08:08:52.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:52 smithi162 ceph-mon[35260]: pgmap v178: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:08:52.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:52 smithi139 ceph-mon[34874]: pgmap v178: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:08:52.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:52 smithi098 ceph-mon[31333]: pgmap v178: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:08:54.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:54 smithi162 ceph-mon[35260]: pgmap v179: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:08:54.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:54 smithi139 ceph-mon[34874]: pgmap v179: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:08:54.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:54 smithi098 ceph-mon[31333]: pgmap v179: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:08:56.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:56 smithi162 ceph-mon[35260]: pgmap v180: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:56.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:56 smithi139 ceph-mon[34874]: pgmap v180: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:56.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:56 smithi098 ceph-mon[31333]: pgmap v180: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:08:58.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:08:58 smithi162 ceph-mon[35260]: pgmap v181: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-09-19T08:08:58.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:08:58 smithi139 ceph-mon[34874]: pgmap v181: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-09-19T08:08:58.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:08:58 smithi098 ceph-mon[31333]: pgmap v181: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-09-19T08:09:00.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:00 smithi162 ceph-mon[35260]: pgmap v182: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:09:00.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:00 smithi139 ceph-mon[34874]: pgmap v182: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:09:00.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:00 smithi098 ceph-mon[31333]: pgmap v182: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:09:02.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:02 smithi162 ceph-mon[35260]: pgmap v183: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:02.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:02 smithi139 ceph-mon[34874]: pgmap v183: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:02.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:02 smithi098 ceph-mon[31333]: pgmap v183: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:04.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:04 smithi162 ceph-mon[35260]: pgmap v184: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:09:04.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:04 smithi139 ceph-mon[34874]: pgmap v184: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:09:04.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:04 smithi098 ceph-mon[31333]: pgmap v184: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:09:06.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:06 smithi162 ceph-mon[35260]: pgmap v185: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:09:06.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:06 smithi139 ceph-mon[34874]: pgmap v185: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:09:06.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:06 smithi098 ceph-mon[31333]: pgmap v185: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:09:08.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:08 smithi162 ceph-mon[35260]: pgmap v186: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-09-19T08:09:08.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:08 smithi139 ceph-mon[34874]: pgmap v186: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-09-19T08:09:08.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:08 smithi098 ceph-mon[31333]: pgmap v186: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-09-19T08:09:10.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:10 smithi162 ceph-mon[35260]: pgmap v187: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-19T08:09:10.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:10 smithi139 ceph-mon[34874]: pgmap v187: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-19T08:09:10.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:10 smithi098 ceph-mon[31333]: pgmap v187: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-19T08:09:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:12 smithi162 ceph-mon[35260]: pgmap v188: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:09:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:12 smithi139 ceph-mon[34874]: pgmap v188: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:09:12.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:12 smithi098 ceph-mon[31333]: pgmap v188: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-09-19T08:09:14.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:14 smithi162 ceph-mon[35260]: pgmap v189: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:09:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:14 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:09:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:14 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:09:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:14 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:09:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:14 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:09:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:14 smithi139 ceph-mon[34874]: pgmap v189: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:09:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:14 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:09:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:14 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:09:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:14 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:09:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:14 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:09:14.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:14 smithi098 ceph-mon[31333]: pgmap v189: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-09-19T08:09:14.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:14 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:09:14.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:14 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:09:14.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:14 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:09:14.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:14 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:09:14.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:09:15.058 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:09:15.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.535+0000 7fc66fee1640 1 -- 172.21.15.98:0/3214916522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc668102570 msgr2=0x7fc6681029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:15.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.535+0000 7fc66fee1640 1 --2- 172.21.15.98:0/3214916522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc668102570 0x7fc6681029b0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fc65c009900 tx=0x7fc65c02a480 comp rx=0 tx=0).stop 2024-09-19T08:09:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.535+0000 7fc66fee1640 1 -- 172.21.15.98:0/3214916522 shutdown_connections 2024-09-19T08:09:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.535+0000 7fc66fee1640 1 --2- 172.21.15.98:0/3214916522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc668102ef0 0x7fc66810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.535+0000 7fc66fee1640 1 --2- 172.21.15.98:0/3214916522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc668102570 0x7fc6681029b0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.535+0000 7fc66fee1640 1 --2- 172.21.15.98:0/3214916522 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc668100e20 0x7fc668101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.535+0000 7fc66fee1640 1 -- 172.21.15.98:0/3214916522 >> 172.21.15.98:0/3214916522 conn(0x7fc6680fc7d0 msgr2=0x7fc6680febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.535+0000 7fc66fee1640 1 -- 172.21.15.98:0/3214916522 shutdown_connections 2024-09-19T08:09:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.535+0000 7fc66fee1640 1 -- 172.21.15.98:0/3214916522 wait complete. 2024-09-19T08:09:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.536+0000 7fc66fee1640 1 Processor -- start 2024-09-19T08:09:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.536+0000 7fc66fee1640 1 -- start start 2024-09-19T08:09:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.536+0000 7fc66fee1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc668100e20 0x7fc6680fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.536+0000 7fc66fee1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc668102570 0x7fc6680fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.536+0000 7fc66fee1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc668102ef0 0x7fc6680fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.536+0000 7fc66fee1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc668110f80 con 0x7fc668102570 2024-09-19T08:09:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.536+0000 7fc66fee1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc668110e00 con 0x7fc668100e20 2024-09-19T08:09:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.536+0000 7fc66fee1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc668111100 con 0x7fc668102ef0 2024-09-19T08:09:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.536+0000 7fc66d47c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc668102570 0x7fc6680fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.536+0000 7fc66d47c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc668102570 0x7fc6680fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36506/0 (socket says 172.21.15.98:36506) 2024-09-19T08:09:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.536+0000 7fc66d47c640 1 -- 172.21.15.98:0/1841405217 learned_addr learned my addr 172.21.15.98:0/1841405217 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.536+0000 7fc66dc7d640 1 --2- 172.21.15.98:0/1841405217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc668100e20 0x7fc6680fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.537+0000 7fc66e47e640 1 --2- 172.21.15.98:0/1841405217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc668102ef0 0x7fc6680fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.537+0000 7fc66d47c640 1 -- 172.21.15.98:0/1841405217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc668102ef0 msgr2=0x7fc6680fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.537+0000 7fc66d47c640 1 --2- 172.21.15.98:0/1841405217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc668102ef0 0x7fc6680fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.537+0000 7fc66d47c640 1 -- 172.21.15.98:0/1841405217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc668100e20 msgr2=0x7fc6680fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.537+0000 7fc66d47c640 1 --2- 172.21.15.98:0/1841405217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc668100e20 0x7fc6680fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.537+0000 7fc66d47c640 1 -- 172.21.15.98:0/1841405217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6681a6a20 con 0x7fc668102570 2024-09-19T08:09:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.537+0000 7fc66d47c640 1 --2- 172.21.15.98:0/1841405217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc668102570 0x7fc6680fd1c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fc65c006010 tx=0x7fc65c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.537+0000 7fc65affd640 1 -- 172.21.15.98:0/1841405217 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c03f070 con 0x7fc668102570 2024-09-19T08:09:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.537+0000 7fc65affd640 1 -- 172.21.15.98:0/1841405217 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc65c031c30 con 0x7fc668102570 2024-09-19T08:09:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.537+0000 7fc66fee1640 1 -- 172.21.15.98:0/1841405217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6681a6cb0 con 0x7fc668102570 2024-09-19T08:09:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.537+0000 7fc65affd640 1 -- 172.21.15.98:0/1841405217 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c035030 con 0x7fc668102570 2024-09-19T08:09:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.537+0000 7fc66fee1640 1 -- 172.21.15.98:0/1841405217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6681a7190 con 0x7fc668102570 2024-09-19T08:09:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.540+0000 7fc6722ba740 1 Processor -- start 2024-09-19T08:09:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.540+0000 7fc6722ba740 1 -- start start 2024-09-19T08:09:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.540+0000 7fc6722ba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b84a0 0x55a2873b8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.540+0000 7fc6722ba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873b8dc0 0x55a2873c3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.540+0000 7fc6722ba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873c3680 0x55a2873c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.540+0000 7fc6722ba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a28731e2e0 con 0x55a2873b8dc0 2024-09-19T08:09:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.540+0000 7fc6722ba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a2872c5ce0 con 0x55a2873c3680 2024-09-19T08:09:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.540+0000 7fc6722ba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a28731a2a0 con 0x55a2873b84a0 2024-09-19T08:09:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc66dc7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873b8dc0 0x55a2873c3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc66dc7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873b8dc0 0x55a2873c3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36518/0 (socket says 172.21.15.98:36518) 2024-09-19T08:09:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc66e47e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b84a0 0x55a2873b8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc66dc7d640 1 -- 172.21.15.98:0/285396126 learned_addr learned my addr 172.21.15.98:0/285396126 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:15.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc65affd640 1 -- 172.21.15.98:0/1841405217 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fc65c042070 con 0x7fc668102570 2024-09-19T08:09:15.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc66dc7d640 1 -- 172.21.15.98:0/285396126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b84a0 msgr2=0x55a2873b8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:15.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc66dc7d640 1 --2- 172.21.15.98:0/285396126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b84a0 0x55a2873b8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc66dc7d640 1 -- 172.21.15.98:0/285396126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873c3680 msgr2=0x55a2873c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:15.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc66dc7d640 1 --2- 172.21.15.98:0/285396126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873c3680 0x55a2873c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc66dc7d640 1 -- 172.21.15.98:0/285396126 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2872aa440 con 0x55a2873b8dc0 2024-09-19T08:09:15.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc66dc7d640 1 --2- 172.21.15.98:0/285396126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873b8dc0 0x55a2873c3140 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fc66400bcb0 tx=0x7fc66400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01abd0216756a68 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:15.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc653fff640 1 -- 172.21.15.98:0/285396126 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664015070 con 0x55a2873b8dc0 2024-09-19T08:09:15.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc653fff640 1 -- 172.21.15.98:0/285396126 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc66401a940 con 0x55a2873b8dc0 2024-09-19T08:09:15.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc653fff640 1 -- 172.21.15.98:0/285396126 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664011a40 con 0x55a2873b8dc0 2024-09-19T08:09:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.541+0000 7fc65affd640 1 --2- 172.21.15.98:0/1841405217 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc6540a3430 0x7fc6540a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.542+0000 7fc66e47e640 1 --2- 172.21.15.98:0/1841405217 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc6540a3430 0x7fc6540a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.542+0000 7fc65affd640 1 -- 172.21.15.98:0/1841405217 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fc65c0f05f0 con 0x7fc668102570 2024-09-19T08:09:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.542+0000 7fc6722ba740 1 -- 172.21.15.98:0/285396126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873b8dc0 msgr2=0x55a2873c3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.542+0000 7fc66e47e640 1 --2- 172.21.15.98:0/1841405217 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc6540a3430 0x7fc6540a58f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc660005fd0 tx=0x7fc6600096e0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.542+0000 7fc6722ba740 1 --2- 172.21.15.98:0/285396126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873b8dc0 0x55a2873c3140 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fc66400bcb0 tx=0x7fc66400f730 comp rx=0 tx=0).stop 2024-09-19T08:09:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.542+0000 7fc6722ba740 1 -- 172.21.15.98:0/285396126 shutdown_connections 2024-09-19T08:09:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.542+0000 7fc6722ba740 1 --2- 172.21.15.98:0/285396126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873c3680 0x55a2873c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.542+0000 7fc6722ba740 1 --2- 172.21.15.98:0/285396126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873b8dc0 0x55a2873c3140 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.542+0000 7fc6722ba740 1 --2- 172.21.15.98:0/285396126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b84a0 0x55a2873b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.542+0000 7fc6722ba740 1 -- 172.21.15.98:0/285396126 >> 172.21.15.98:0/285396126 conn(0x55a2873b4a30 msgr2=0x55a2873b4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.542+0000 7fc6722ba740 1 -- 172.21.15.98:0/285396126 shutdown_connections 2024-09-19T08:09:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.542+0000 7fc6722ba740 1 -- 172.21.15.98:0/285396126 wait complete. 2024-09-19T08:09:15.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.543+0000 7fc6722ba740 1 -- start start 2024-09-19T08:09:15.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.543+0000 7fc6722ba740 1 Processor -- start 2024-09-19T08:09:15.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc6722ba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b84a0 0x55a2873c2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc6722ba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b8dc0 0x55a2873c0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc6722ba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873c3680 0x55a2873c07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc6722ba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a2873c6250 con 0x55a2873c3680 2024-09-19T08:09:15.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc6722ba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a2872d9050 con 0x55a2873b84a0 2024-09-19T08:09:15.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc6722ba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a2873c63d0 con 0x55a2873b8dc0 2024-09-19T08:09:15.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc6722ba740 10 client.0 _ll_register_callbacks cb 0x55a287319160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:15.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc66e47e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873c3680 0x55a2873c07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc66e47e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873c3680 0x55a2873c07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36526/0 (socket says 172.21.15.98:36526) 2024-09-19T08:09:15.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc66e47e640 1 -- 172.21.15.98:0/2541313002 learned_addr learned my addr 172.21.15.98:0/2541313002 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:15.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc66d47c640 1 --2- 172.21.15.98:0/2541313002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b8dc0 0x55a2873c0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc66dc7d640 1 --2- 172.21.15.98:0/2541313002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b84a0 0x55a2873c2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc66e47e640 1 -- 172.21.15.98:0/2541313002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b8dc0 msgr2=0x55a2873c0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:15.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc66e47e640 1 --2- 172.21.15.98:0/2541313002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b8dc0 0x55a2873c0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc66e47e640 1 -- 172.21.15.98:0/2541313002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b84a0 msgr2=0x55a2873c2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:15.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc66e47e640 1 --2- 172.21.15.98:0/2541313002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b84a0 0x55a2873c2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc66e47e640 1 -- 172.21.15.98:0/2541313002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a28708f3c0 con 0x55a2873c3680 2024-09-19T08:09:15.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc66e47e640 1 --2- 172.21.15.98:0/2541313002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873c3680 0x55a2873c07a0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fc66001d870 tx=0x7fc660004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=579202670770c732 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:15.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.544+0000 7fc6517fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:15.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc6517fa640 1 -- 172.21.15.98:0/2541313002 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66001e820 con 0x55a2873c3680 2024-09-19T08:09:15.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc6517fa640 1 -- 172.21.15.98:0/2541313002 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fc66001e9c0 con 0x55a2873c3680 2024-09-19T08:09:15.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc6722ba740 1 -- 172.21.15.98:0/2541313002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc66401a940 con 0x55a2873c3680 2024-09-19T08:09:15.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc6517fa640 1 -- 172.21.15.98:0/2541313002 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc66001ebc0 con 0x55a2873c3680 2024-09-19T08:09:15.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc6722ba740 10 client.16956 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:15.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc6722ba740 1 -- 172.21.15.98:0/2541313002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a2873d1cc0 con 0x55a2873c3680 2024-09-19T08:09:15.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc6517fa640 1 -- 172.21.15.98:0/2541313002 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc66401a940 con 0x55a2873c3680 2024-09-19T08:09:15.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc6517fa640 1 -- 172.21.15.98:0/2541313002 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc66000f040 con 0x55a2873c3680 2024-09-19T08:09:15.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc6722ba740 10 client.16956 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:15.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc6722ba740 10 client.16956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:09:15.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc6722ba740 20 client.16956 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:15.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc6722ba740 1 --2- 172.21.15.98:0/2541313002 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55a2873d30d0 0x55a2873d54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc6722ba740 4 client.16956 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:09:15.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc6722ba740 1 -- 172.21.15.98:0/2541313002 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a28732b850 con 0x55a2873d30d0 2024-09-19T08:09:15.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.545+0000 7fc66dc7d640 1 --2- 172.21.15.98:0/2541313002 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55a2873d30d0 0x55a2873d54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.546+0000 7fc66dc7d640 1 --2- 172.21.15.98:0/2541313002 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55a2873d30d0 0x55a2873d54c0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=620503e16670eed3 server_cookie=c5a5139f9ac5a5d4 in_seq=0 out_seq=0 2024-09-19T08:09:15.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.546+0000 7fc6517fa640 10 client.16956 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:15.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.546+0000 7fc6517fa640 1 -- 172.21.15.98:0/2541313002 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a28732b850 con 0x55a2873d30d0 2024-09-19T08:09:15.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.546+0000 7fc6517fa640 10 client.16956 handle_command_reply: tid=0 2024-09-19T08:09:15.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.547+0000 7fc6722ba740 1 client.16956 shutdown 2024-09-19T08:09:15.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.547+0000 7fc6722ba740 1 -- 172.21.15.98:0/2541313002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873c3680 msgr2=0x55a2873c07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:15.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.547+0000 7fc6722ba740 1 --2- 172.21.15.98:0/2541313002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873c3680 0x55a2873c07a0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fc66001d870 tx=0x7fc660004a40 comp rx=0 tx=0).stop 2024-09-19T08:09:15.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.547+0000 7fc6722ba740 1 -- 172.21.15.98:0/2541313002 shutdown_connections 2024-09-19T08:09:15.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.547+0000 7fc6722ba740 1 --2- 172.21.15.98:0/2541313002 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55a2873d30d0 0x55a2873d54c0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.547+0000 7fc6722ba740 1 --2- 172.21.15.98:0/2541313002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873c3680 0x55a2873c07a0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.547+0000 7fc6517fa640 0 client.16956 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:15.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.547+0000 7fc6722ba740 1 --2- 172.21.15.98:0/2541313002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b8dc0 0x55a2873c0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.547+0000 7fc6722ba740 1 --2- 172.21.15.98:0/2541313002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b84a0 0x55a2873c2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.547+0000 7fc6722ba740 1 -- 172.21.15.98:0/2541313002 >> 172.21.15.98:0/2541313002 conn(0x55a2873b4a30 msgr2=0x55a2873ba710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:15.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.547+0000 7fc6722ba740 1 -- 172.21.15.98:0/2541313002 shutdown_connections 2024-09-19T08:09:15.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.547+0000 7fc6722ba740 1 -- 172.21.15.98:0/2541313002 wait complete. 2024-09-19T08:09:15.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.547+0000 7fc6722ba740 20 client.16956 trim_cache size 0 max 16384 2024-09-19T08:09:15.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.582+0000 7fc6722ba740 1 Processor -- start 2024-09-19T08:09:15.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.582+0000 7fc6722ba740 1 -- start start 2024-09-19T08:09:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc6722ba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b84a0 0x55a2873cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc6722ba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b8dc0 0x55a2873b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc6722ba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873c3680 0x55a2873dad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc6722ba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc6681af9f0 con 0x55a2873c3680 2024-09-19T08:09:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc6722ba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc6681b1530 con 0x55a2873b8dc0 2024-09-19T08:09:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc6722ba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6681b00c0 con 0x55a2873b84a0 2024-09-19T08:09:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc66e47e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873c3680 0x55a2873dad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc66e47e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873c3680 0x55a2873dad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36542/0 (socket says 172.21.15.98:36542) 2024-09-19T08:09:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc66e47e640 1 -- 172.21.15.98:0/2878824860 learned_addr learned my addr 172.21.15.98:0/2878824860 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc66dc7d640 1 --2- 172.21.15.98:0/2878824860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b84a0 0x55a2873cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc66d47c640 1 --2- 172.21.15.98:0/2878824860 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b8dc0 0x55a2873b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc66e47e640 1 -- 172.21.15.98:0/2878824860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b84a0 msgr2=0x55a2873cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:15.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc66e47e640 1 --2- 172.21.15.98:0/2878824860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b84a0 0x55a2873cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc66e47e640 1 -- 172.21.15.98:0/2878824860 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b8dc0 msgr2=0x55a2873b3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:15.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc66e47e640 1 --2- 172.21.15.98:0/2878824860 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b8dc0 0x55a2873b3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc66e47e640 1 -- 172.21.15.98:0/2878824860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2873d5830 con 0x55a2873c3680 2024-09-19T08:09:15.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.583+0000 7fc66e47e640 1 --2- 172.21.15.98:0/2878824860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873c3680 0x55a2873dad70 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc660004af0 tx=0x7fc660012cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dc1ed0fd66023fd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:15.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.584+0000 7fc6537fe640 1 -- 172.21.15.98:0/2878824860 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66001e820 con 0x55a2873c3680 2024-09-19T08:09:15.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.584+0000 7fc6537fe640 1 -- 172.21.15.98:0/2878824860 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc660037070 con 0x55a2873c3680 2024-09-19T08:09:15.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.584+0000 7fc6537fe640 1 -- 172.21.15.98:0/2878824860 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6600298e0 con 0x55a2873c3680 2024-09-19T08:09:15.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.584+0000 7fc6722ba740 1 -- 172.21.15.98:0/2878824860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873c3680 msgr2=0x55a2873dad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:15.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.584+0000 7fc6722ba740 1 --2- 172.21.15.98:0/2878824860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873c3680 0x55a2873dad70 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc660004af0 tx=0x7fc660012cd0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.584+0000 7fc6722ba740 1 -- 172.21.15.98:0/2878824860 shutdown_connections 2024-09-19T08:09:15.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.584+0000 7fc6722ba740 1 --2- 172.21.15.98:0/2878824860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873c3680 0x55a2873dad70 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.584+0000 7fc6722ba740 1 --2- 172.21.15.98:0/2878824860 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b8dc0 0x55a2873b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.584+0000 7fc6722ba740 1 --2- 172.21.15.98:0/2878824860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873b84a0 0x55a2873cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.584+0000 7fc6722ba740 1 -- 172.21.15.98:0/2878824860 >> 172.21.15.98:0/2878824860 conn(0x55a2873b4a30 msgr2=0x55a2873b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:15.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.584+0000 7fc6722ba740 1 -- 172.21.15.98:0/2878824860 shutdown_connections 2024-09-19T08:09:15.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.585+0000 7fc6722ba740 1 -- 172.21.15.98:0/2878824860 wait complete. 2024-09-19T08:09:15.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.585+0000 7fc6722ba740 1 -- start start 2024-09-19T08:09:15.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.585+0000 7fc6722ba740 1 Processor -- start 2024-09-19T08:09:15.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc6722ba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b84a0 0x55a2873c7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc6722ba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873b8dc0 0x55a2873c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc6722ba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873c3680 0x55a2873c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc6722ba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc6681b0e60 con 0x55a2873b8dc0 2024-09-19T08:09:15.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc6722ba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc6681b0790 con 0x55a2873b84a0 2024-09-19T08:09:15.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc6722ba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6681ad9d0 con 0x55a2873c3680 2024-09-19T08:09:15.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc6722ba740 10 client.0 _ll_register_callbacks cb 0x7fc66001d520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:15.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc66d47c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873b8dc0 0x55a2873c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc66d47c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873b8dc0 0x55a2873c7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36552/0 (socket says 172.21.15.98:36552) 2024-09-19T08:09:15.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc66d47c640 1 -- 172.21.15.98:0/845114101 learned_addr learned my addr 172.21.15.98:0/845114101 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:15.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc66e47e640 1 --2- 172.21.15.98:0/845114101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873c3680 0x55a2873c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc66dc7d640 1 --2- 172.21.15.98:0/845114101 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b84a0 0x55a2873c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc66d47c640 1 -- 172.21.15.98:0/845114101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873c3680 msgr2=0x55a2873c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:15.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc66d47c640 1 --2- 172.21.15.98:0/845114101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873c3680 0x55a2873c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc66d47c640 1 -- 172.21.15.98:0/845114101 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b84a0 msgr2=0x55a2873c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:15.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc66d47c640 1 --2- 172.21.15.98:0/845114101 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b84a0 0x55a2873c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc66d47c640 1 -- 172.21.15.98:0/845114101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a28743cea0 con 0x55a2873b8dc0 2024-09-19T08:09:15.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc66dc7d640 1 --2- 172.21.15.98:0/845114101 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b84a0 0x55a2873c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:09:15.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc66d47c640 1 --2- 172.21.15.98:0/845114101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873b8dc0 0x55a2873c7a50 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fc65c0b92d0 tx=0x7fc65c0bc260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed60057ec09517b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:15.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc6517fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:15.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc6517fa640 1 -- 172.21.15.98:0/845114101 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c049040 con 0x55a2873b8dc0 2024-09-19T08:09:15.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc6517fa640 1 -- 172.21.15.98:0/845114101 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fc65c0b8f70 con 0x55a2873b8dc0 2024-09-19T08:09:15.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.586+0000 7fc6722ba740 1 -- 172.21.15.98:0/845114101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc660037070 con 0x55a2873b8dc0 2024-09-19T08:09:15.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.587+0000 7fc6517fa640 1 -- 172.21.15.98:0/845114101 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc65c0bc900 con 0x55a2873b8dc0 2024-09-19T08:09:15.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.587+0000 7fc6722ba740 10 client.16962 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:15.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.587+0000 7fc6722ba740 1 -- 172.21.15.98:0/845114101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a28743e1b0 con 0x55a2873b8dc0 2024-09-19T08:09:15.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.587+0000 7fc6517fa640 1 -- 172.21.15.98:0/845114101 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc660037070 con 0x55a2873b8dc0 2024-09-19T08:09:15.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.587+0000 7fc6517fa640 1 -- 172.21.15.98:0/845114101 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc65c0bcb60 con 0x55a2873b8dc0 2024-09-19T08:09:15.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.587+0000 7fc6722ba740 10 client.16962 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:15.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.587+0000 7fc6722ba740 10 client.16962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:09:15.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.587+0000 7fc6722ba740 20 client.16962 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:15.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.587+0000 7fc6722ba740 1 --2- 172.21.15.98:0/845114101 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55a28743e930 0x55a287440d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:15.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.587+0000 7fc6722ba740 4 client.16962 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:09:15.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.587+0000 7fc6722ba740 1 -- 172.21.15.98:0/845114101 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a287325d60 con 0x55a28743e930 2024-09-19T08:09:15.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.587+0000 7fc66dc7d640 1 --2- 172.21.15.98:0/845114101 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55a28743e930 0x55a287440d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:15.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc66dc7d640 1 --2- 172.21.15.98:0/845114101 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55a28743e930 0x55a287440d50 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8291e6f867948f2 server_cookie=20f1813d1af33e6 in_seq=0 out_seq=0 2024-09-19T08:09:15.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6517fa640 10 client.16962 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:15.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6517fa640 1 -- 172.21.15.98:0/845114101 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a287325d60 con 0x55a28743e930 2024-09-19T08:09:15.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6517fa640 10 client.16962 handle_command_reply: tid=0 2024-09-19T08:09:15.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6722ba740 1 client.16962 shutdown 2024-09-19T08:09:15.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6722ba740 1 -- 172.21.15.98:0/845114101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873b8dc0 msgr2=0x55a2873c7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:15.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6722ba740 1 --2- 172.21.15.98:0/845114101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873b8dc0 0x55a2873c7a50 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fc65c0b92d0 tx=0x7fc65c0bc260 comp rx=0 tx=0).stop 2024-09-19T08:09:15.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6722ba740 1 -- 172.21.15.98:0/845114101 shutdown_connections 2024-09-19T08:09:15.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6722ba740 1 --2- 172.21.15.98:0/845114101 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55a28743e930 0x55a287440d50 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6722ba740 1 --2- 172.21.15.98:0/845114101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2873c3680 0x55a2873c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6722ba740 1 --2- 172.21.15.98:0/845114101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2873b8dc0 0x55a2873c7a50 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6722ba740 1 --2- 172.21.15.98:0/845114101 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a2873b84a0 0x55a2873c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6517fa640 0 client.16962 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:15.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6722ba740 1 -- 172.21.15.98:0/845114101 >> 172.21.15.98:0/845114101 conn(0x55a2873b4a30 msgr2=0x55a2873b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:15.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6722ba740 1 -- 172.21.15.98:0/845114101 shutdown_connections 2024-09-19T08:09:15.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6722ba740 1 -- 172.21.15.98:0/845114101 wait complete. 2024-09-19T08:09:15.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.588+0000 7fc6722ba740 20 client.16962 trim_cache size 0 max 16384 2024-09-19T08:09:15.594 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:09:15.594 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:09:15.594 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:09:15.594 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:09:15.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.589+0000 7fc66fee1640 1 -- 172.21.15.98:0/1841405217 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc6540a3430 msgr2=0x7fc6540a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:15.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.589+0000 7fc66fee1640 1 --2- 172.21.15.98:0/1841405217 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc6540a3430 0x7fc6540a58f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc660005fd0 tx=0x7fc6600096e0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.589+0000 7fc66fee1640 1 -- 172.21.15.98:0/1841405217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc668102570 msgr2=0x7fc6680fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:15.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.589+0000 7fc66fee1640 1 --2- 172.21.15.98:0/1841405217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc668102570 0x7fc6680fd1c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fc65c006010 tx=0x7fc65c003c10 comp rx=0 tx=0).stop 2024-09-19T08:09:15.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.589+0000 7fc66fee1640 1 -- 172.21.15.98:0/1841405217 shutdown_connections 2024-09-19T08:09:15.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.589+0000 7fc66fee1640 1 --2- 172.21.15.98:0/1841405217 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc6540a3430 0x7fc6540a58f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.589+0000 7fc66fee1640 1 --2- 172.21.15.98:0/1841405217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc668102ef0 0x7fc6680fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.589+0000 7fc66fee1640 1 --2- 172.21.15.98:0/1841405217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc668102570 0x7fc6680fd1c0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.589+0000 7fc66fee1640 1 --2- 172.21.15.98:0/1841405217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc668100e20 0x7fc6680fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:15.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.589+0000 7fc66fee1640 1 -- 172.21.15.98:0/1841405217 >> 172.21.15.98:0/1841405217 conn(0x7fc6680fc7d0 msgr2=0x7fc66806a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:15.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.589+0000 7fc66fee1640 1 -- 172.21.15.98:0/1841405217 shutdown_connections 2024-09-19T08:09:15.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:15.589+0000 7fc66fee1640 1 -- 172.21.15.98:0/1841405217 wait complete. 2024-09-19T08:09:16.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:16 smithi162 ceph-mon[35260]: pgmap v190: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-19T08:09:16.405 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:16 smithi098 ceph-mon[31333]: pgmap v190: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-19T08:09:16.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:16 smithi139 ceph-mon[34874]: pgmap v190: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-19T08:09:16.434 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:09:16.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:09:16.691 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:09:17.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.214+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/2901382662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf64106a80 msgr2=0x7fbf64106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:17.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.214+0000 7fbf6a5a6640 1 --2- 172.21.15.98:0/2901382662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf64106a80 0x7fbf64106e60 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fbf54009930 tx=0x7fbf5402a420 comp rx=0 tx=0).stop 2024-09-19T08:09:17.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.214+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/2901382662 shutdown_connections 2024-09-19T08:09:17.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.214+0000 7fbf6a5a6640 1 --2- 172.21.15.98:0/2901382662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbf64101420 0x7fbf6410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:17.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.214+0000 7fbf6a5a6640 1 --2- 172.21.15.98:0/2901382662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf64100a80 0x7fbf64100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:17.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.214+0000 7fbf6a5a6640 1 --2- 172.21.15.98:0/2901382662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf64106a80 0x7fbf64106e60 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:17.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.215+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/2901382662 >> 172.21.15.98:0/2901382662 conn(0x7fbf640fc7a0 msgr2=0x7fbf640febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:17.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.215+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/2901382662 shutdown_connections 2024-09-19T08:09:17.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.215+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/2901382662 wait complete. 2024-09-19T08:09:17.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.215+0000 7fbf6a5a6640 1 Processor -- start 2024-09-19T08:09:17.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.215+0000 7fbf6a5a6640 1 -- start start 2024-09-19T08:09:17.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf6a5a6640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbf64100a80 0x7fbf64199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:17.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf6a5a6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf64101420 0x7fbf64199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf6a5a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf64106a80 0x7fbf641a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf6a5a6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf64110e80 con 0x7fbf64106a80 2024-09-19T08:09:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf6a5a6640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbf64110d00 con 0x7fbf64100a80 2024-09-19T08:09:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf6a5a6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbf64111000 con 0x7fbf64101420 2024-09-19T08:09:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf68b43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf64106a80 0x7fbf641a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf68b43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf64106a80 0x7fbf641a4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38006/0 (socket says 172.21.15.98:38006) 2024-09-19T08:09:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf68b43640 1 -- 172.21.15.98:0/1187813316 learned_addr learned my addr 172.21.15.98:0/1187813316 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf63fff640 1 --2- 172.21.15.98:0/1187813316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbf64100a80 0x7fbf64199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf5bfff640 1 --2- 172.21.15.98:0/1187813316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf64101420 0x7fbf64199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf68b43640 1 -- 172.21.15.98:0/1187813316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf64101420 msgr2=0x7fbf64199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf68b43640 1 --2- 172.21.15.98:0/1187813316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf64101420 0x7fbf64199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf68b43640 1 -- 172.21.15.98:0/1187813316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbf64100a80 msgr2=0x7fbf64199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:17.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf68b43640 1 --2- 172.21.15.98:0/1187813316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbf64100a80 0x7fbf64199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:17.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf68b43640 1 -- 172.21.15.98:0/1187813316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf641a48d0 con 0x7fbf64106a80 2024-09-19T08:09:17.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf68b43640 1 --2- 172.21.15.98:0/1187813316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf64106a80 0x7fbf641a4390 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00eaa0 tx=0x7fbf5c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:17.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf61ffb640 1 -- 172.21.15.98:0/1187813316 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c00cdb0 con 0x7fbf64106a80 2024-09-19T08:09:17.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/1187813316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf641a4aa0 con 0x7fbf64106a80 2024-09-19T08:09:17.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf61ffb640 1 -- 172.21.15.98:0/1187813316 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf5c005020 con 0x7fbf64106a80 2024-09-19T08:09:17.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.216+0000 7fbf61ffb640 1 -- 172.21.15.98:0/1187813316 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c0116b0 con 0x7fbf64106a80 2024-09-19T08:09:17.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.217+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/1187813316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf641a4fb0 con 0x7fbf64106a80 2024-09-19T08:09:17.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.218+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/1187813316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf64100ee0 con 0x7fbf64106a80 2024-09-19T08:09:17.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.219+0000 7fbf61ffb640 1 -- 172.21.15.98:0/1187813316 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fbf5c005560 con 0x7fbf64106a80 2024-09-19T08:09:17.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.219+0000 7fbf61ffb640 1 --2- 172.21.15.98:0/1187813316 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbf500a3430 0x7fbf500a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:17.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.219+0000 7fbf63fff640 1 --2- 172.21.15.98:0/1187813316 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbf500a3430 0x7fbf500a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:17.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.219+0000 7fbf61ffb640 1 -- 172.21.15.98:0/1187813316 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fbf5c0cfe80 con 0x7fbf64106a80 2024-09-19T08:09:17.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.220+0000 7fbf63fff640 1 --2- 172.21.15.98:0/1187813316 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbf500a3430 0x7fbf500a58f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbf54009900 tx=0x7fbf540058d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:17.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.224+0000 7fbf61ffb640 1 -- 172.21.15.98:0/1187813316 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fbf5c016030 con 0x7fbf64106a80 2024-09-19T08:09:17.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.424+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/1187813316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf64109050 con 0x7fbf64106a80 2024-09-19T08:09:17.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.425+0000 7fbf61ffb640 1 -- 172.21.15.98:0/1187813316 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11983 (secure 0 0 0) 0x7fbf5c09c890 con 0x7fbf64106a80 2024-09-19T08:09:17.425 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:09:17.425 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"btime":"2024-09-19T08:08:26:985720+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:08:26.555658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:09:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2024-09-19T08:09:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.427+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/1187813316 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbf500a3430 msgr2=0x7fbf500a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.427+0000 7fbf6a5a6640 1 --2- 172.21.15.98:0/1187813316 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbf500a3430 0x7fbf500a58f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbf54009900 tx=0x7fbf540058d0 comp rx=0 tx=0).stop 2024-09-19T08:09:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.427+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/1187813316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf64106a80 msgr2=0x7fbf641a4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.427+0000 7fbf6a5a6640 1 --2- 172.21.15.98:0/1187813316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf64106a80 0x7fbf641a4390 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00eaa0 tx=0x7fbf5c00ef60 comp rx=0 tx=0).stop 2024-09-19T08:09:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.427+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/1187813316 shutdown_connections 2024-09-19T08:09:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.427+0000 7fbf6a5a6640 1 --2- 172.21.15.98:0/1187813316 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbf500a3430 0x7fbf500a58f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.427+0000 7fbf6a5a6640 1 --2- 172.21.15.98:0/1187813316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf64106a80 0x7fbf641a4390 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.427+0000 7fbf6a5a6640 1 --2- 172.21.15.98:0/1187813316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf64101420 0x7fbf64199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.427+0000 7fbf6a5a6640 1 --2- 172.21.15.98:0/1187813316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbf64100a80 0x7fbf64199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.427+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/1187813316 >> 172.21.15.98:0/1187813316 conn(0x7fbf640fc7a0 msgr2=0x7fbf640fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.428+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/1187813316 shutdown_connections 2024-09-19T08:09:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:17.428+0000 7fbf6a5a6640 1 -- 172.21.15.98:0/1187813316 wait complete. 2024-09-19T08:09:18.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:09:18.315 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:09:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:18 smithi162 ceph-mon[35260]: pgmap v191: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-19T08:09:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:18 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1187813316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:09:18.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:18 smithi139 ceph-mon[34874]: pgmap v191: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-19T08:09:18.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:18 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1187813316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:09:18.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:18 smithi098 ceph-mon[31333]: pgmap v191: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-19T08:09:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:18 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1187813316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:09:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.776+0000 7f584a2fe640 1 -- 172.21.15.98:0/3913614071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5844106a80 msgr2=0x7f5844106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.776+0000 7f584a2fe640 1 --2- 172.21.15.98:0/3913614071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5844106a80 0x7f5844106e60 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5834009930 tx=0x7f583402a420 comp rx=0 tx=0).stop 2024-09-19T08:09:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.776+0000 7f584a2fe640 1 -- 172.21.15.98:0/3913614071 shutdown_connections 2024-09-19T08:09:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.776+0000 7f584a2fe640 1 --2- 172.21.15.98:0/3913614071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5844101420 0x7f584410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.776+0000 7f584a2fe640 1 --2- 172.21.15.98:0/3913614071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5844100a80 0x7f5844100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.776+0000 7f584a2fe640 1 --2- 172.21.15.98:0/3913614071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5844106a80 0x7f5844106e60 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.776+0000 7f584a2fe640 1 -- 172.21.15.98:0/3913614071 >> 172.21.15.98:0/3913614071 conn(0x7f58440fc7a0 msgr2=0x7f58440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.776+0000 7f584a2fe640 1 -- 172.21.15.98:0/3913614071 shutdown_connections 2024-09-19T08:09:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.776+0000 7f584a2fe640 1 -- 172.21.15.98:0/3913614071 wait complete. 2024-09-19T08:09:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.776+0000 7f584a2fe640 1 Processor -- start 2024-09-19T08:09:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.776+0000 7f584a2fe640 1 -- start start 2024-09-19T08:09:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f584a2fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5844100a80 0x7f5844198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f584a2fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5844101420 0x7f58441989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f584a2fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5844106a80 0x7f584419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f584a2fe640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5844110ed0 con 0x7f5844106a80 2024-09-19T08:09:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f584a2fe640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5844110d50 con 0x7f5844100a80 2024-09-19T08:09:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f584a2fe640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5844111050 con 0x7f5844101420 2024-09-19T08:09:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f5843fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5844100a80 0x7f5844198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f584889b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5844106a80 0x7f584419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f5843fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5844100a80 0x7f5844198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:48782/0 (socket says 172.21.15.98:48782) 2024-09-19T08:09:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f5843fff640 1 -- 172.21.15.98:0/2238857699 learned_addr learned my addr 172.21.15.98:0/2238857699 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f583bfff640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5844101420 0x7f58441989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f5843fff640 1 -- 172.21.15.98:0/2238857699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5844101420 msgr2=0x7f58441989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f5843fff640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5844101420 0x7f58441989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f5843fff640 1 -- 172.21.15.98:0/2238857699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5844106a80 msgr2=0x7f584419cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f5843fff640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5844106a80 0x7f584419cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f5843fff640 1 -- 172.21.15.98:0/2238857699 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584419d430 con 0x7f5844100a80 2024-09-19T08:09:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f583bfff640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5844101420 0x7f58441989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:09:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f584889b640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5844106a80 0x7f584419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:09:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f5843fff640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5844100a80 0x7f5844198480 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f583402a940 tx=0x7f5834003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.777+0000 7f5841ffb640 1 -- 172.21.15.98:0/2238857699 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583403f070 con 0x7f5844100a80 2024-09-19T08:09:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.778+0000 7f584a2fe640 1 -- 172.21.15.98:0/2238857699 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f584419d6c0 con 0x7f5844100a80 2024-09-19T08:09:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.778+0000 7f5841ffb640 1 -- 172.21.15.98:0/2238857699 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5834031c90 con 0x7f5844100a80 2024-09-19T08:09:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.778+0000 7f5841ffb640 1 -- 172.21.15.98:0/2238857699 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5834035030 con 0x7f5844100a80 2024-09-19T08:09:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.778+0000 7f584a2fe640 1 -- 172.21.15.98:0/2238857699 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58441a5000 con 0x7f5844100a80 2024-09-19T08:09:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.780+0000 7f5841ffb640 1 -- 172.21.15.98:0/2238857699 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f5834042070 con 0x7f5844100a80 2024-09-19T08:09:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.780+0000 7f584c6d7740 1 Processor -- start 2024-09-19T08:09:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.780+0000 7f584c6d7740 1 -- start start 2024-09-19T08:09:18.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584c6d7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d1b40 0x55da981d0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584c6d7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d05d0 0x55da981d0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584c6d7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981db370 0x55da981dd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584c6d7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55da981362f0 con 0x55da981d1b40 2024-09-19T08:09:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584c6d7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55da980ddcf0 con 0x55da981db370 2024-09-19T08:09:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584c6d7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55da981322b0 con 0x55da981d05d0 2024-09-19T08:09:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584889b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d1b40 0x55da981d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584889b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d1b40 0x55da981d0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38030/0 (socket says 172.21.15.98:38030) 2024-09-19T08:09:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584889b640 1 -- 172.21.15.98:0/3991097576 learned_addr learned my addr 172.21.15.98:0/3991097576 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f5843fff640 1 --2- 172.21.15.98:0/3991097576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d05d0 0x55da981d0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f583bfff640 1 --2- 172.21.15.98:0/3991097576 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981db370 0x55da981dd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584889b640 1 -- 172.21.15.98:0/3991097576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d05d0 msgr2=0x55da981d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584889b640 1 --2- 172.21.15.98:0/3991097576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d05d0 0x55da981d0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584889b640 1 -- 172.21.15.98:0/3991097576 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981db370 msgr2=0x55da981dd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584889b640 1 --2- 172.21.15.98:0/3991097576 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981db370 0x55da981dd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584889b640 1 -- 172.21.15.98:0/3991097576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da980c23a0 con 0x55da981d1b40 2024-09-19T08:09:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f5841ffb640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58300a33b0 0x7f58300a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f584889b640 1 --2- 172.21.15.98:0/3991097576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d1b40 0x55da981d0090 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f583c010710 tx=0x7f583c010bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591fd8a50f12f41 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.781+0000 7f5843fff640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58300a33b0 0x7f58300a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.782+0000 7f5841ffb640 1 -- 172.21.15.98:0/2238857699 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f58340f0400 con 0x7f5844100a80 2024-09-19T08:09:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.782+0000 7f583a7fc640 1 -- 172.21.15.98:0/3991097576 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c00e8d0 con 0x55da981d1b40 2024-09-19T08:09:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.782+0000 7f583a7fc640 1 -- 172.21.15.98:0/3991097576 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f583c012450 con 0x55da981d1b40 2024-09-19T08:09:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.782+0000 7f583a7fc640 1 -- 172.21.15.98:0/3991097576 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c01c650 con 0x55da981d1b40 2024-09-19T08:09:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.782+0000 7f584c6d7740 1 -- 172.21.15.98:0/3991097576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d1b40 msgr2=0x55da981d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.782+0000 7f5843fff640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58300a33b0 0x7f58300a5870 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f58340edc30 tx=0x7f58340f3f50 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.782+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3991097576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d1b40 0x55da981d0090 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f583c010710 tx=0x7f583c010bd0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.782+0000 7f584c6d7740 1 -- 172.21.15.98:0/3991097576 shutdown_connections 2024-09-19T08:09:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.782+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3991097576 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981db370 0x55da981dd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.782+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3991097576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d05d0 0x55da981d0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.782+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3991097576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d1b40 0x55da981d0090 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.782+0000 7f584c6d7740 1 -- 172.21.15.98:0/3991097576 >> 172.21.15.98:0/3991097576 conn(0x55da98199310 msgr2=0x55da981ccc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.783+0000 7f584c6d7740 1 -- 172.21.15.98:0/3991097576 shutdown_connections 2024-09-19T08:09:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.783+0000 7f584c6d7740 1 -- 172.21.15.98:0/3991097576 wait complete. 2024-09-19T08:09:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.783+0000 7f584c6d7740 1 -- start start 2024-09-19T08:09:18.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.783+0000 7f584c6d7740 1 Processor -- start 2024-09-19T08:09:18.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.783+0000 7f584c6d7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981d05d0 0x55da981dad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.783+0000 7f584c6d7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d1b40 0x55da981d8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.783+0000 7f584c6d7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981db370 0x55da981d86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.783+0000 7f584c6d7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55da981de070 con 0x55da981d1b40 2024-09-19T08:09:18.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.783+0000 7f584c6d7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55da980f1060 con 0x55da981d05d0 2024-09-19T08:09:18.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.783+0000 7f584c6d7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55da981de1f0 con 0x55da981db370 2024-09-19T08:09:18.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f584c6d7740 10 client.0 _ll_register_callbacks cb 0x55da97e448c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:18.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f584889b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981d05d0 0x55da981dad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f583bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d1b40 0x55da981d8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f584889b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981d05d0 0x55da981dad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:48802/0 (socket says 172.21.15.98:48802) 2024-09-19T08:09:18.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f584889b640 1 -- 172.21.15.98:0/3727196151 learned_addr learned my addr 172.21.15.98:0/3727196151 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:18.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f584889b640 1 --2- 172.21.15.98:0/3727196151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981db370 0x55da981d86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f583bfff640 1 -- 172.21.15.98:0/3727196151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981db370 msgr2=0x55da981d86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:18.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f583bfff640 1 --2- 172.21.15.98:0/3727196151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981db370 0x55da981d86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f583bfff640 1 -- 172.21.15.98:0/3727196151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981d05d0 msgr2=0x55da981dad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:18.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f583bfff640 1 --2- 172.21.15.98:0/3727196151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981d05d0 0x55da981dad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f583bfff640 1 -- 172.21.15.98:0/3727196151 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da97ea73c0 con 0x55da981d1b40 2024-09-19T08:09:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f583bfff640 1 --2- 172.21.15.98:0/3727196151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d1b40 0x55da981d8060 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f58440fecb0 tx=0x7f582c012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1634910b516d358 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f58237fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f58237fe640 1 -- 172.21.15.98:0/3727196151 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582c010e00 con 0x55da981d1b40 2024-09-19T08:09:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f58237fe640 1 -- 172.21.15.98:0/3727196151 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f582c005020 con 0x55da981d1b40 2024-09-19T08:09:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.784+0000 7f584c6d7740 1 -- 172.21.15.98:0/3727196151 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f583c012450 con 0x55da981d1b40 2024-09-19T08:09:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.785+0000 7f58237fe640 1 -- 172.21.15.98:0/3727196151 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f582c005220 con 0x55da981d1b40 2024-09-19T08:09:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.785+0000 7f584c6d7740 10 client.16980 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.785+0000 7f584c6d7740 1 -- 172.21.15.98:0/3727196151 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55da981e9bb0 con 0x55da981d1b40 2024-09-19T08:09:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.785+0000 7f58237fe640 1 -- 172.21.15.98:0/3727196151 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f583c012450 con 0x55da981d1b40 2024-09-19T08:09:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.785+0000 7f58237fe640 1 -- 172.21.15.98:0/3727196151 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f582c01c490 con 0x55da981d1b40 2024-09-19T08:09:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.785+0000 7f584c6d7740 10 client.16980 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.785+0000 7f584c6d7740 10 client.16980 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:09:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.785+0000 7f584c6d7740 20 client.16980 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:18.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.785+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3727196151 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55da981eb070 0x55da981ed4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.785+0000 7f584c6d7740 4 client.16980 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:09:18.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.785+0000 7f584c6d7740 1 -- 172.21.15.98:0/3727196151 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da98143860 con 0x55da981eb070 2024-09-19T08:09:18.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.785+0000 7f584889b640 1 --2- 172.21.15.98:0/3727196151 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55da981eb070 0x55da981ed4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.786+0000 7f58237fe640 10 client.16980 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:09:18.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.786+0000 7f584889b640 1 --2- 172.21.15.98:0/3727196151 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55da981eb070 0x55da981ed4d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3801f44cb094c202 server_cookie=5740863cbd5964bb in_seq=0 out_seq=0 2024-09-19T08:09:18.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.786+0000 7f58237fe640 1 -- 172.21.15.98:0/3727196151 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55da98143860 con 0x55da981eb070 2024-09-19T08:09:18.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.786+0000 7f58237fe640 10 client.16980 handle_command_reply: tid=0 2024-09-19T08:09:18.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.787+0000 7f584c6d7740 1 client.16980 shutdown 2024-09-19T08:09:18.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.787+0000 7f584c6d7740 1 -- 172.21.15.98:0/3727196151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d1b40 msgr2=0x55da981d8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:18.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.787+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3727196151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d1b40 0x55da981d8060 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f58440fecb0 tx=0x7f582c012b60 comp rx=0 tx=0).stop 2024-09-19T08:09:18.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.787+0000 7f584c6d7740 1 -- 172.21.15.98:0/3727196151 shutdown_connections 2024-09-19T08:09:18.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.787+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3727196151 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55da981eb070 0x55da981ed4d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.787+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3727196151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981db370 0x55da981d86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.787+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3727196151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d1b40 0x55da981d8060 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.787+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3727196151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981d05d0 0x55da981dad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.787+0000 7f584c6d7740 1 -- 172.21.15.98:0/3727196151 >> 172.21.15.98:0/3727196151 conn(0x55da98199310 msgr2=0x55da981d2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:18.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.787+0000 7f58237fe640 0 client.16980 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:09:18.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.787+0000 7f584c6d7740 1 -- 172.21.15.98:0/3727196151 shutdown_connections 2024-09-19T08:09:18.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.787+0000 7f584c6d7740 1 -- 172.21.15.98:0/3727196151 wait complete. 2024-09-19T08:09:18.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.787+0000 7f584c6d7740 20 client.16980 trim_cache size 0 max 16384 2024-09-19T08:09:18.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.800+0000 7f584c6d7740 1 Processor -- start 2024-09-19T08:09:18.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.800+0000 7f584c6d7740 1 -- start start 2024-09-19T08:09:18.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584c6d7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d05d0 0x55da981c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584c6d7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d1b40 0x55da981d6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584c6d7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981db370 0x55da981cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584c6d7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f58441ad880 con 0x55da981d05d0 2024-09-19T08:09:18.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584c6d7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f58441af3c0 con 0x55da981db370 2024-09-19T08:09:18.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584c6d7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f58441adf50 con 0x55da981d1b40 2024-09-19T08:09:18.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584889b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d05d0 0x55da981c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584889b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d05d0 0x55da981c7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38038/0 (socket says 172.21.15.98:38038) 2024-09-19T08:09:18.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f583bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d1b40 0x55da981d6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584889b640 1 -- 172.21.15.98:0/630558643 learned_addr learned my addr 172.21.15.98:0/630558643 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:18.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584889b640 1 --2- 172.21.15.98:0/630558643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981db370 0x55da981cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584889b640 1 -- 172.21.15.98:0/630558643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d1b40 msgr2=0x55da981d6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:18.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584889b640 1 --2- 172.21.15.98:0/630558643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d1b40 0x55da981d6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584889b640 1 -- 172.21.15.98:0/630558643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981db370 msgr2=0x55da981cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:18.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584889b640 1 --2- 172.21.15.98:0/630558643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981db370 0x55da981cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584889b640 1 -- 172.21.15.98:0/630558643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da981d8a30 con 0x55da981d05d0 2024-09-19T08:09:18.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f583bfff640 1 --2- 172.21.15.98:0/630558643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d1b40 0x55da981d6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:09:18.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f584889b640 1 --2- 172.21.15.98:0/630558643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d05d0 0x55da981c7c60 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f583c012e60 tx=0x7f583c012e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a8b62c0000fef1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:18.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f5839ffb640 1 -- 172.21.15.98:0/630558643 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c031020 con 0x55da981d05d0 2024-09-19T08:09:18.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f5839ffb640 1 -- 172.21.15.98:0/630558643 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f583c0049a0 con 0x55da981d05d0 2024-09-19T08:09:18.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.801+0000 7f5839ffb640 1 -- 172.21.15.98:0/630558643 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c00eda0 con 0x55da981d05d0 2024-09-19T08:09:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.802+0000 7f584c6d7740 1 -- 172.21.15.98:0/630558643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d05d0 msgr2=0x55da981c7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.802+0000 7f584c6d7740 1 --2- 172.21.15.98:0/630558643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d05d0 0x55da981c7c60 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f583c012e60 tx=0x7f583c012e90 comp rx=0 tx=0).stop 2024-09-19T08:09:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.802+0000 7f584c6d7740 1 -- 172.21.15.98:0/630558643 shutdown_connections 2024-09-19T08:09:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.802+0000 7f584c6d7740 1 --2- 172.21.15.98:0/630558643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981db370 0x55da981cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.802+0000 7f584c6d7740 1 --2- 172.21.15.98:0/630558643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d1b40 0x55da981d6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.802+0000 7f584c6d7740 1 --2- 172.21.15.98:0/630558643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981d05d0 0x55da981c7c60 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.802+0000 7f584c6d7740 1 -- 172.21.15.98:0/630558643 >> 172.21.15.98:0/630558643 conn(0x55da98199310 msgr2=0x55da981cb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.802+0000 7f584c6d7740 1 -- 172.21.15.98:0/630558643 shutdown_connections 2024-09-19T08:09:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.802+0000 7f584c6d7740 1 -- 172.21.15.98:0/630558643 wait complete. 2024-09-19T08:09:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.802+0000 7f584c6d7740 1 -- start start 2024-09-19T08:09:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.803+0000 7f584c6d7740 1 Processor -- start 2024-09-19T08:09:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.803+0000 7f584c6d7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981d05d0 0x55da981e3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.803+0000 7f584c6d7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d1b40 0x55da981e3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.803+0000 7f584c6d7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981db370 0x55da981d4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.803+0000 7f584c6d7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f58441aecf0 con 0x55da981db370 2024-09-19T08:09:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.803+0000 7f584c6d7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f58441ae620 con 0x55da981d05d0 2024-09-19T08:09:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.803+0000 7f584c6d7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f58441ab860 con 0x55da981d1b40 2024-09-19T08:09:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.803+0000 7f584c6d7740 10 client.0 _ll_register_callbacks cb 0x7f582c00bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.803+0000 7f584889b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981db370 0x55da981d4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.803+0000 7f584889b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981db370 0x55da981d4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38052/0 (socket says 172.21.15.98:38052) 2024-09-19T08:09:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.803+0000 7f584889b640 1 -- 172.21.15.98:0/3975577674 learned_addr learned my addr 172.21.15.98:0/3975577674 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.803+0000 7f584889b640 1 --2- 172.21.15.98:0/3975577674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981d05d0 0x55da981e3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.803+0000 7f583bfff640 1 --2- 172.21.15.98:0/3975577674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d1b40 0x55da981e3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.803+0000 7f584889b640 1 -- 172.21.15.98:0/3975577674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d1b40 msgr2=0x55da981e3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f584889b640 1 --2- 172.21.15.98:0/3975577674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d1b40 0x55da981e3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f584889b640 1 -- 172.21.15.98:0/3975577674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981d05d0 msgr2=0x55da981e3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f584889b640 1 --2- 172.21.15.98:0/3975577674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981d05d0 0x55da981e3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f584889b640 1 -- 172.21.15.98:0/3975577674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da981d4fd0 con 0x55da981db370 2024-09-19T08:09:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f584889b640 1 --2- 172.21.15.98:0/3975577674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981db370 0x55da981d4890 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f583c00ede0 tx=0x7f583c01e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2ef2510b94e058 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f58237fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f58237fe640 1 -- 172.21.15.98:0/3975577674 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c031070 con 0x55da981db370 2024-09-19T08:09:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f58237fe640 1 -- 172.21.15.98:0/3975577674 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f583c012920 con 0x55da981db370 2024-09-19T08:09:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f584c6d7740 1 -- 172.21.15.98:0/3975577674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f583c0049a0 con 0x55da981db370 2024-09-19T08:09:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f58237fe640 1 -- 172.21.15.98:0/3975577674 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f583c012b20 con 0x55da981db370 2024-09-19T08:09:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f584c6d7740 10 client.16986 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f584c6d7740 1 -- 172.21.15.98:0/3975577674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55da98255770 con 0x55da981db370 2024-09-19T08:09:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f58237fe640 1 -- 172.21.15.98:0/3975577674 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f583c0049a0 con 0x55da981db370 2024-09-19T08:09:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f58237fe640 1 -- 172.21.15.98:0/3975577674 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f583c036040 con 0x55da981db370 2024-09-19T08:09:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f584c6d7740 10 client.16986 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f584c6d7740 10 client.16986 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:09:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f584c6d7740 20 client.16986 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3975577674 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55da98255c40 0x55da98258060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f584c6d7740 4 client.16986 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:09:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.804+0000 7f584c6d7740 1 -- 172.21.15.98:0/3975577674 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55da9813dd70 con 0x55da98255c40 2024-09-19T08:09:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.805+0000 7f583bfff640 1 --2- 172.21.15.98:0/3975577674 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55da98255c40 0x55da98258060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.805+0000 7f58237fe640 10 client.16986 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:09:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.805+0000 7f583bfff640 1 --2- 172.21.15.98:0/3975577674 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55da98255c40 0x55da98258060 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e97a968338784c7e server_cookie=cb466013c5852122 in_seq=0 out_seq=0 2024-09-19T08:09:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.805+0000 7f58237fe640 1 -- 172.21.15.98:0/3975577674 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55da9813dd70 con 0x55da98255c40 2024-09-19T08:09:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.805+0000 7f58237fe640 10 client.16986 handle_command_reply: tid=0 2024-09-19T08:09:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.805+0000 7f584c6d7740 1 client.16986 shutdown 2024-09-19T08:09:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.806+0000 7f584c6d7740 1 -- 172.21.15.98:0/3975577674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981db370 msgr2=0x55da981d4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:18.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.806+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3975577674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981db370 0x55da981d4890 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f583c00ede0 tx=0x7f583c01e040 comp rx=0 tx=0).stop 2024-09-19T08:09:18.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.806+0000 7f584c6d7740 1 -- 172.21.15.98:0/3975577674 shutdown_connections 2024-09-19T08:09:18.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.806+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3975577674 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55da98255c40 0x55da98258060 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.806+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3975577674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da981db370 0x55da981d4890 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.806+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3975577674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da981d1b40 0x55da981e3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.806+0000 7f58237fe640 0 client.16986 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:09:18.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.806+0000 7f584c6d7740 1 --2- 172.21.15.98:0/3975577674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55da981d05d0 0x55da981e3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.806+0000 7f584c6d7740 1 -- 172.21.15.98:0/3975577674 >> 172.21.15.98:0/3975577674 conn(0x55da98199310 msgr2=0x55da981da0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:18.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.806+0000 7f584c6d7740 1 -- 172.21.15.98:0/3975577674 shutdown_connections 2024-09-19T08:09:18.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.806+0000 7f584c6d7740 1 -- 172.21.15.98:0/3975577674 wait complete. 2024-09-19T08:09:18.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.806+0000 7f584c6d7740 20 client.16986 trim_cache size 0 max 16384 2024-09-19T08:09:18.813 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:09:18.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.807+0000 7f584a2fe640 1 -- 172.21.15.98:0/2238857699 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58300a33b0 msgr2=0x7f58300a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:18.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.807+0000 7f584a2fe640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58300a33b0 0x7f58300a5870 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f58340edc30 tx=0x7f58340f3f50 comp rx=0 tx=0).stop 2024-09-19T08:09:18.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.807+0000 7f584a2fe640 1 -- 172.21.15.98:0/2238857699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5844100a80 msgr2=0x7f5844198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:18.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.807+0000 7f584a2fe640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5844100a80 0x7f5844198480 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f583402a940 tx=0x7f5834003b90 comp rx=0 tx=0).stop 2024-09-19T08:09:18.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.807+0000 7f584a2fe640 1 -- 172.21.15.98:0/2238857699 shutdown_connections 2024-09-19T08:09:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.807+0000 7f584a2fe640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58300a33b0 0x7f58300a5870 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.807+0000 7f584a2fe640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5844106a80 0x7f584419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.807+0000 7f584a2fe640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5844101420 0x7f58441989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.807+0000 7f584a2fe640 1 --2- 172.21.15.98:0/2238857699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5844100a80 0x7f5844198480 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.807+0000 7f584a2fe640 1 -- 172.21.15.98:0/2238857699 >> 172.21.15.98:0/2238857699 conn(0x7f58440fc7a0 msgr2=0x7f584406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.807+0000 7f584a2fe640 1 -- 172.21.15.98:0/2238857699 shutdown_connections 2024-09-19T08:09:18.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:18.807+0000 7f584a2fe640 1 -- 172.21.15.98:0/2238857699 wait complete. 2024-09-19T08:09:19.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:09:19.720 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:09:20.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.235+0000 7faf04c3b640 1 -- 172.21.15.98:0/1324314865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00101de0 msgr2=0x7faf001021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:20.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.235+0000 7faf04c3b640 1 --2- 172.21.15.98:0/1324314865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00101de0 0x7faf001021c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7faef0009ca0 tx=0x7faef002a8c0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.235+0000 7faf04c3b640 1 -- 172.21.15.98:0/1324314865 shutdown_connections 2024-09-19T08:09:20.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.235+0000 7faf04c3b640 1 --2- 172.21.15.98:0/1324314865 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf00106f10 0x7faf0010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.235+0000 7faf04c3b640 1 --2- 172.21.15.98:0/1324314865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf00102790 0x7faf001067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.235+0000 7faf04c3b640 1 --2- 172.21.15.98:0/1324314865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00101de0 0x7faf001021c0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.235+0000 7faf04c3b640 1 -- 172.21.15.98:0/1324314865 >> 172.21.15.98:0/1324314865 conn(0x7faf000fd8c0 msgr2=0x7faf000ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:20.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.235+0000 7faf04c3b640 1 -- 172.21.15.98:0/1324314865 shutdown_connections 2024-09-19T08:09:20.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.235+0000 7faf04c3b640 1 -- 172.21.15.98:0/1324314865 wait complete. 2024-09-19T08:09:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.235+0000 7faf04c3b640 1 Processor -- start 2024-09-19T08:09:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.235+0000 7faf04c3b640 1 -- start start 2024-09-19T08:09:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faf04c3b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00101de0 0x7faf001984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faf04c3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf00102790 0x7faf00198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faf04c3b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf00106f10 0x7faf0019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faf04c3b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf00111f70 con 0x7faf00101de0 2024-09-19T08:09:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faf04c3b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7faf00111df0 con 0x7faf00106f10 2024-09-19T08:09:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faf04c3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf001120f0 con 0x7faf00102790 2024-09-19T08:09:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faefe59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00101de0 0x7faf001984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faefe59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00101de0 0x7faf001984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38080/0 (socket says 172.21.15.98:38080) 2024-09-19T08:09:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faefe59c640 1 -- 172.21.15.98:0/1355445532 learned_addr learned my addr 172.21.15.98:0/1355445532 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faefed9d640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf00106f10 0x7faf0019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faefdd9b640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf00102790 0x7faf00198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faefe59c640 1 -- 172.21.15.98:0/1355445532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf00102790 msgr2=0x7faf00198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faefe59c640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf00102790 0x7faf00198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faefe59c640 1 -- 172.21.15.98:0/1355445532 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf00106f10 msgr2=0x7faf0019cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faefe59c640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf00106f10 0x7faf0019cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faefe59c640 1 -- 172.21.15.98:0/1355445532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0019d480 con 0x7faf00101de0 2024-09-19T08:09:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faefed9d640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf00106f10 0x7faf0019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:09:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faefe59c640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00101de0 0x7faf001984d0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7faef002ade0 tx=0x7faef00319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.236+0000 7faeef7fe640 1 -- 172.21.15.98:0/1355445532 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef002fa20 con 0x7faf00101de0 2024-09-19T08:09:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.237+0000 7faeef7fe640 1 -- 172.21.15.98:0/1355445532 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faef002fbc0 con 0x7faf00101de0 2024-09-19T08:09:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.237+0000 7faf04c3b640 1 -- 172.21.15.98:0/1355445532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0019d710 con 0x7faf00101de0 2024-09-19T08:09:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.237+0000 7faeef7fe640 1 -- 172.21.15.98:0/1355445532 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef00466e0 con 0x7faf00101de0 2024-09-19T08:09:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.237+0000 7faf04c3b640 1 -- 172.21.15.98:0/1355445532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf001a5050 con 0x7faf00101de0 2024-09-19T08:09:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.239+0000 7faeef7fe640 1 -- 172.21.15.98:0/1355445532 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7faef0046940 con 0x7faf00101de0 2024-09-19T08:09:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.239+0000 7faf07014740 1 Processor -- start 2024-09-19T08:09:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.239+0000 7faf07014740 1 -- start start 2024-09-19T08:09:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faf07014740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366cb40 0x55ef1366b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faf07014740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1366b5d0 0x55ef1366ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faf07014740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1367a780 0x55ef1367cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faf07014740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ef135d12f0 con 0x55ef1366b5d0 2024-09-19T08:09:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faf07014740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ef13578cf0 con 0x55ef1367a780 2024-09-19T08:09:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faf07014740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ef135cd2b0 con 0x55ef1366cb40 2024-09-19T08:09:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faefed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366cb40 0x55ef1366b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faefe59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1366b5d0 0x55ef1366ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faefed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366cb40 0x55ef1366b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:54456/0 (socket says 172.21.15.98:54456) 2024-09-19T08:09:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faefed9d640 1 -- 172.21.15.98:0/3051600008 learned_addr learned my addr 172.21.15.98:0/3051600008 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faeef7fe640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7faee80a33b0 0x7faee80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faefdd9b640 1 --2- 172.21.15.98:0/3051600008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1367a780 0x55ef1367cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faefed9d640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7faee80a33b0 0x7faee80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faeef7fe640 1 -- 172.21.15.98:0/1355445532 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7faef00f1120 con 0x7faf00101de0 2024-09-19T08:09:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faefe59c640 1 -- 172.21.15.98:0/3051600008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366cb40 msgr2=0x55ef1366b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faefe59c640 1 --2- 172.21.15.98:0/3051600008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366cb40 0x55ef1366b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faefe59c640 1 -- 172.21.15.98:0/3051600008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1367a780 msgr2=0x55ef1367cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faefe59c640 1 --2- 172.21.15.98:0/3051600008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1367a780 0x55ef1367cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faefe59c640 1 -- 172.21.15.98:0/3051600008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef1355d3a0 con 0x55ef1366b5d0 2024-09-19T08:09:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.240+0000 7faefdd9b640 1 --2- 172.21.15.98:0/3051600008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1367a780 0x55ef1367cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:09:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.241+0000 7faefe59c640 1 --2- 172.21.15.98:0/3051600008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1366b5d0 0x55ef1366ba10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7faef00ee9b0 tx=0x7faef002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362f4a7cdcca3a03 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.241+0000 7faefed9d640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7faee80a33b0 0x7faee80a5870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faef8005fd0 tx=0x7faef8004730 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.241+0000 7faeecff9640 1 -- 172.21.15.98:0/3051600008 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef00bcea0 con 0x55ef1366b5d0 2024-09-19T08:09:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.241+0000 7faeecff9640 1 -- 172.21.15.98:0/3051600008 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faef0036030 con 0x55ef1366b5d0 2024-09-19T08:09:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.241+0000 7faeecff9640 1 -- 172.21.15.98:0/3051600008 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef0039e60 con 0x55ef1366b5d0 2024-09-19T08:09:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.241+0000 7faf07014740 1 -- 172.21.15.98:0/3051600008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1366b5d0 msgr2=0x55ef1366ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.241+0000 7faf07014740 1 --2- 172.21.15.98:0/3051600008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1366b5d0 0x55ef1366ba10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7faef00ee9b0 tx=0x7faef002a8c0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 -- 172.21.15.98:0/3051600008 shutdown_connections 2024-09-19T08:09:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 --2- 172.21.15.98:0/3051600008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1367a780 0x55ef1367cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 --2- 172.21.15.98:0/3051600008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1366b5d0 0x55ef1366ba10 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 --2- 172.21.15.98:0/3051600008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366cb40 0x55ef1366b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 -- 172.21.15.98:0/3051600008 >> 172.21.15.98:0/3051600008 conn(0x55ef13634310 msgr2=0x55ef13667c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 -- 172.21.15.98:0/3051600008 shutdown_connections 2024-09-19T08:09:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 -- 172.21.15.98:0/3051600008 wait complete. 2024-09-19T08:09:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 -- start start 2024-09-19T08:09:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 Processor -- start 2024-09-19T08:09:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366b5d0 0x55ef1367a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366cb40 0x55ef136774f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1367a780 0x55ef13677c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ef135d12f0 con 0x55ef1367a780 2024-09-19T08:09:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ef1358c060 con 0x55ef1366cb40 2024-09-19T08:09:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ef1367d240 con 0x55ef1366b5d0 2024-09-19T08:09:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.242+0000 7faf07014740 10 client.0 _ll_register_callbacks cb 0x55ef132df8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faefed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1367a780 0x55ef13677c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faefed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1367a780 0x55ef13677c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38104/0 (socket says 172.21.15.98:38104) 2024-09-19T08:09:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faefed9d640 1 -- 172.21.15.98:0/2074402029 learned_addr learned my addr 172.21.15.98:0/2074402029 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faefdd9b640 1 --2- 172.21.15.98:0/2074402029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366cb40 0x55ef136774f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faefe59c640 1 --2- 172.21.15.98:0/2074402029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366b5d0 0x55ef1367a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faefed9d640 1 -- 172.21.15.98:0/2074402029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366b5d0 msgr2=0x55ef1367a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faefed9d640 1 --2- 172.21.15.98:0/2074402029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366b5d0 0x55ef1367a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faefed9d640 1 -- 172.21.15.98:0/2074402029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366cb40 msgr2=0x55ef136774f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faefed9d640 1 --2- 172.21.15.98:0/2074402029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366cb40 0x55ef136774f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faefed9d640 1 -- 172.21.15.98:0/2074402029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef133423c0 con 0x55ef1367a780 2024-09-19T08:09:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faefed9d640 1 --2- 172.21.15.98:0/2074402029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1367a780 0x55ef13677c80 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7faef800a940 tx=0x7faef80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b885b64bc307eaf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faee5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faee5ffb640 1 -- 172.21.15.98:0/2074402029 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8025030 con 0x55ef1367a780 2024-09-19T08:09:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faee5ffb640 1 -- 172.21.15.98:0/2074402029 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7faef800ae00 con 0x55ef1367a780 2024-09-19T08:09:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faf07014740 1 -- 172.21.15.98:0/2074402029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faef0036030 con 0x55ef1367a780 2024-09-19T08:09:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faee5ffb640 1 -- 172.21.15.98:0/2074402029 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faef8010ae0 con 0x55ef1367a780 2024-09-19T08:09:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faf07014740 10 client.16998 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.243+0000 7faf07014740 1 -- 172.21.15.98:0/2074402029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ef13688b20 con 0x55ef1367a780 2024-09-19T08:09:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.244+0000 7faee5ffb640 1 -- 172.21.15.98:0/2074402029 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faef0036030 con 0x55ef1367a780 2024-09-19T08:09:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.244+0000 7faee5ffb640 1 -- 172.21.15.98:0/2074402029 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7faef801f040 con 0x55ef1367a780 2024-09-19T08:09:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.244+0000 7faf07014740 10 client.16998 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.244+0000 7faf07014740 10 client.16998 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:09:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.244+0000 7faf07014740 20 client.16998 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.244+0000 7faf07014740 1 --2- 172.21.15.98:0/2074402029 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55ef13689ff0 0x55ef1368c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.244+0000 7faf07014740 4 client.16998 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:09:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.244+0000 7faf07014740 1 -- 172.21.15.98:0/2074402029 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef135de860 con 0x55ef13689ff0 2024-09-19T08:09:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.244+0000 7faefe59c640 1 --2- 172.21.15.98:0/2074402029 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55ef13689ff0 0x55ef1368c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.245+0000 7faee5ffb640 10 client.16998 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:09:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.245+0000 7faefe59c640 1 --2- 172.21.15.98:0/2074402029 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55ef13689ff0 0x55ef1368c440 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ed32bd8907abaa0b server_cookie=60f638c2063959c8 in_seq=0 out_seq=0 2024-09-19T08:09:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.245+0000 7faee5ffb640 1 -- 172.21.15.98:0/2074402029 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ef135de860 con 0x55ef13689ff0 2024-09-19T08:09:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.245+0000 7faee5ffb640 10 client.16998 handle_command_reply: tid=0 2024-09-19T08:09:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.245+0000 7faf07014740 1 client.16998 shutdown 2024-09-19T08:09:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.245+0000 7faf07014740 1 -- 172.21.15.98:0/2074402029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1367a780 msgr2=0x55ef13677c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.245+0000 7faf07014740 1 --2- 172.21.15.98:0/2074402029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1367a780 0x55ef13677c80 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7faef800a940 tx=0x7faef80046c0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.246+0000 7faf07014740 1 -- 172.21.15.98:0/2074402029 shutdown_connections 2024-09-19T08:09:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.246+0000 7faf07014740 1 --2- 172.21.15.98:0/2074402029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1367a780 0x55ef13677c80 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.246+0000 7faf07014740 1 --2- 172.21.15.98:0/2074402029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366cb40 0x55ef136774f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.246+0000 7faf07014740 1 --2- 172.21.15.98:0/2074402029 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55ef13689ff0 0x55ef1368c440 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.246+0000 7faf07014740 1 --2- 172.21.15.98:0/2074402029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366b5d0 0x55ef1367a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.246+0000 7faee5ffb640 0 client.16998 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:09:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.246+0000 7faf07014740 1 -- 172.21.15.98:0/2074402029 >> 172.21.15.98:0/2074402029 conn(0x55ef13634310 msgr2=0x55ef1366d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.246+0000 7faf07014740 1 -- 172.21.15.98:0/2074402029 shutdown_connections 2024-09-19T08:09:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.246+0000 7faf07014740 1 -- 172.21.15.98:0/2074402029 wait complete. 2024-09-19T08:09:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.246+0000 7faf07014740 20 client.16998 trim_cache size 0 max 16384 2024-09-19T08:09:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.258+0000 7faf07014740 1 Processor -- start 2024-09-19T08:09:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.258+0000 7faf07014740 1 -- start start 2024-09-19T08:09:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faf07014740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366b5d0 0x55ef136764d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faf07014740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366cb40 0x55ef1367bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faf07014740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1367a780 0x55ef1367c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faf07014740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf001ab5b0 con 0x55ef1367a780 2024-09-19T08:09:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faf07014740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7faf001ad0f0 con 0x55ef1366cb40 2024-09-19T08:09:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faf07014740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf001abc80 con 0x55ef1366b5d0 2024-09-19T08:09:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faefed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1367a780 0x55ef1367c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faefed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1367a780 0x55ef1367c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38106/0 (socket says 172.21.15.98:38106) 2024-09-19T08:09:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faefed9d640 1 -- 172.21.15.98:0/467635390 learned_addr learned my addr 172.21.15.98:0/467635390 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faefe59c640 1 --2- 172.21.15.98:0/467635390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366b5d0 0x55ef136764d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faefdd9b640 1 --2- 172.21.15.98:0/467635390 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366cb40 0x55ef1367bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faefed9d640 1 -- 172.21.15.98:0/467635390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366b5d0 msgr2=0x55ef136764d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faefed9d640 1 --2- 172.21.15.98:0/467635390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366b5d0 0x55ef136764d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faefed9d640 1 -- 172.21.15.98:0/467635390 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366cb40 msgr2=0x55ef1367bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faefed9d640 1 --2- 172.21.15.98:0/467635390 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366cb40 0x55ef1367bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faefed9d640 1 -- 172.21.15.98:0/467635390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef13686290 con 0x55ef1367a780 2024-09-19T08:09:20.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faefed9d640 1 --2- 172.21.15.98:0/467635390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1367a780 0x55ef1367c0b0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7faef8005d80 tx=0x7faef800af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c7b5b0d1b5db46 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:20.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faee7fff640 1 -- 172.21.15.98:0/467635390 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8025070 con 0x55ef1367a780 2024-09-19T08:09:20.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faee7fff640 1 -- 172.21.15.98:0/467635390 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faef800b8a0 con 0x55ef1367a780 2024-09-19T08:09:20.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.259+0000 7faee7fff640 1 -- 172.21.15.98:0/467635390 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8021470 con 0x55ef1367a780 2024-09-19T08:09:20.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.260+0000 7faf07014740 1 -- 172.21.15.98:0/467635390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1367a780 msgr2=0x55ef1367c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:20.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.260+0000 7faf07014740 1 --2- 172.21.15.98:0/467635390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1367a780 0x55ef1367c0b0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7faef8005d80 tx=0x7faef800af90 comp rx=0 tx=0).stop 2024-09-19T08:09:20.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.260+0000 7faf07014740 1 -- 172.21.15.98:0/467635390 shutdown_connections 2024-09-19T08:09:20.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.260+0000 7faf07014740 1 --2- 172.21.15.98:0/467635390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1367a780 0x55ef1367c0b0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.260+0000 7faf07014740 1 --2- 172.21.15.98:0/467635390 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366cb40 0x55ef1367bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.260+0000 7faf07014740 1 --2- 172.21.15.98:0/467635390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1366b5d0 0x55ef136764d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.260+0000 7faf07014740 1 -- 172.21.15.98:0/467635390 >> 172.21.15.98:0/467635390 conn(0x55ef13634310 msgr2=0x55ef13675dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:20.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.260+0000 7faf07014740 1 -- 172.21.15.98:0/467635390 shutdown_connections 2024-09-19T08:09:20.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.260+0000 7faf07014740 1 -- 172.21.15.98:0/467635390 wait complete. 2024-09-19T08:09:20.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.260+0000 7faf07014740 1 -- start start 2024-09-19T08:09:20.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.260+0000 7faf07014740 1 Processor -- start 2024-09-19T08:09:20.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faf07014740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366b5d0 0x55ef13674690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faf07014740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1366cb40 0x55ef13674bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faf07014740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1367a780 0x55ef13675110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faf07014740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf001aca20 con 0x55ef1366cb40 2024-09-19T08:09:20.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faf07014740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7faf001ac350 con 0x55ef1366b5d0 2024-09-19T08:09:20.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faf07014740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf001a9590 con 0x55ef1367a780 2024-09-19T08:09:20.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faf07014740 10 client.0 _ll_register_callbacks cb 0x7faef8013ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:20.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faefdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1366cb40 0x55ef13674bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faefdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1366cb40 0x55ef13674bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38118/0 (socket says 172.21.15.98:38118) 2024-09-19T08:09:20.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faefdd9b640 1 -- 172.21.15.98:0/1568311870 learned_addr learned my addr 172.21.15.98:0/1568311870 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:20.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faefe59c640 1 --2- 172.21.15.98:0/1568311870 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366b5d0 0x55ef13674690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faefed9d640 1 --2- 172.21.15.98:0/1568311870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1367a780 0x55ef13675110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faefdd9b640 1 -- 172.21.15.98:0/1568311870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1367a780 msgr2=0x55ef13675110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:20.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faefdd9b640 1 --2- 172.21.15.98:0/1568311870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1367a780 0x55ef13675110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faefdd9b640 1 -- 172.21.15.98:0/1568311870 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366b5d0 msgr2=0x55ef13674690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:20.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faefdd9b640 1 --2- 172.21.15.98:0/1568311870 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366b5d0 0x55ef13674690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faefdd9b640 1 -- 172.21.15.98:0/1568311870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef13680a90 con 0x55ef1366cb40 2024-09-19T08:09:20.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faefdd9b640 1 --2- 172.21.15.98:0/1568311870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1366cb40 0x55ef13674bd0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x55ef1366c070 tx=0x7faef4012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b014c9f7095e8834 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:20.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faee5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:20.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faee5ffb640 1 -- 172.21.15.98:0/1568311870 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef4004230 con 0x55ef1366cb40 2024-09-19T08:09:20.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.261+0000 7faee5ffb640 1 -- 172.21.15.98:0/1568311870 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7faef40043d0 con 0x55ef1366cb40 2024-09-19T08:09:20.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.262+0000 7faf07014740 1 -- 172.21.15.98:0/1568311870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faef800b8a0 con 0x55ef1366cb40 2024-09-19T08:09:20.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.262+0000 7faee5ffb640 1 -- 172.21.15.98:0/1568311870 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faef4011970 con 0x55ef1366cb40 2024-09-19T08:09:20.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.262+0000 7faf07014740 10 client.17004 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:20.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.262+0000 7faf07014740 1 -- 172.21.15.98:0/1568311870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ef136f51d0 con 0x55ef1366cb40 2024-09-19T08:09:20.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.262+0000 7faee5ffb640 1 -- 172.21.15.98:0/1568311870 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faef402b070 con 0x55ef1366cb40 2024-09-19T08:09:20.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.262+0000 7faee5ffb640 1 -- 172.21.15.98:0/1568311870 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7faef4025050 con 0x55ef1366cb40 2024-09-19T08:09:20.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.262+0000 7faf07014740 10 client.17004 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:20.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.262+0000 7faf07014740 10 client.17004 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:09:20.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.262+0000 7faf07014740 20 client.17004 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:20.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.262+0000 7faf07014740 1 --2- 172.21.15.98:0/1568311870 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55ef136f5a00 0x55ef136f7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:20.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.262+0000 7faf07014740 4 client.17004 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:09:20.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.262+0000 7faf07014740 1 -- 172.21.15.98:0/1568311870 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ef135d8d70 con 0x55ef136f5a00 2024-09-19T08:09:20.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.262+0000 7faefe59c640 1 --2- 172.21.15.98:0/1568311870 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55ef136f5a00 0x55ef136f7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faefe59c640 1 --2- 172.21.15.98:0/1568311870 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55ef136f5a00 0x55ef136f7e20 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=80563d953a735b41 server_cookie=ac9572135c95b25b in_seq=0 out_seq=0 2024-09-19T08:09:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faee5ffb640 10 client.17004 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:09:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faee5ffb640 1 -- 172.21.15.98:0/1568311870 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ef135d8d70 con 0x55ef136f5a00 2024-09-19T08:09:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faee5ffb640 10 client.17004 handle_command_reply: tid=0 2024-09-19T08:09:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faf07014740 1 client.17004 shutdown 2024-09-19T08:09:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faf07014740 1 -- 172.21.15.98:0/1568311870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1366cb40 msgr2=0x55ef13674bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faf07014740 1 --2- 172.21.15.98:0/1568311870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1366cb40 0x55ef13674bd0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x55ef1366c070 tx=0x7faef4012b60 comp rx=0 tx=0).stop 2024-09-19T08:09:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faf07014740 1 -- 172.21.15.98:0/1568311870 shutdown_connections 2024-09-19T08:09:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faf07014740 1 --2- 172.21.15.98:0/1568311870 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55ef136f5a00 0x55ef136f7e20 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faee5ffb640 0 client.17004 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:09:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faf07014740 1 --2- 172.21.15.98:0/1568311870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef1367a780 0x55ef13675110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faf07014740 1 --2- 172.21.15.98:0/1568311870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef1366cb40 0x55ef13674bd0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faf07014740 1 --2- 172.21.15.98:0/1568311870 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ef1366b5d0 0x55ef13674690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faf07014740 1 -- 172.21.15.98:0/1568311870 >> 172.21.15.98:0/1568311870 conn(0x55ef13634310 msgr2=0x55ef136765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:20.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faf07014740 1 -- 172.21.15.98:0/1568311870 shutdown_connections 2024-09-19T08:09:20.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faf07014740 1 -- 172.21.15.98:0/1568311870 wait complete. 2024-09-19T08:09:20.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.263+0000 7faf07014740 20 client.17004 trim_cache size 0 max 16384 2024-09-19T08:09:20.269 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:09:20.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.264+0000 7faf04c3b640 1 -- 172.21.15.98:0/1355445532 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7faee80a33b0 msgr2=0x7faee80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:20.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.264+0000 7faf04c3b640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7faee80a33b0 0x7faee80a5870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faef8005fd0 tx=0x7faef8004730 comp rx=0 tx=0).stop 2024-09-19T08:09:20.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.264+0000 7faf04c3b640 1 -- 172.21.15.98:0/1355445532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00101de0 msgr2=0x7faf001984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:20.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.264+0000 7faf04c3b640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00101de0 0x7faf001984d0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7faef002ade0 tx=0x7faef00319f0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.264+0000 7faf04c3b640 1 -- 172.21.15.98:0/1355445532 shutdown_connections 2024-09-19T08:09:20.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.264+0000 7faf04c3b640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7faee80a33b0 0x7faee80a5870 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.264+0000 7faf04c3b640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf00106f10 0x7faf0019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.264+0000 7faf04c3b640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf00102790 0x7faf00198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.264+0000 7faf04c3b640 1 --2- 172.21.15.98:0/1355445532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00101de0 0x7faf001984d0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:20.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.264+0000 7faf04c3b640 1 -- 172.21.15.98:0/1355445532 >> 172.21.15.98:0/1355445532 conn(0x7faf000fd8c0 msgr2=0x7faf000fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:20.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.264+0000 7faf04c3b640 1 -- 172.21.15.98:0/1355445532 shutdown_connections 2024-09-19T08:09:20.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:20.264+0000 7faf04c3b640 1 -- 172.21.15.98:0/1355445532 wait complete. 2024-09-19T08:09:20.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:20 smithi162 ceph-mon[35260]: pgmap v192: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-19T08:09:20.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:20 smithi139 ceph-mon[34874]: pgmap v192: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-19T08:09:20.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:20 smithi098 ceph-mon[31333]: pgmap v192: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-19T08:09:20.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:09:21.192 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:09:21.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.684+0000 7fc2532d4640 1 -- 172.21.15.98:0/3051890668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24c101420 msgr2=0x7fc24c10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:21.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.684+0000 7fc2532d4640 1 --2- 172.21.15.98:0/3051890668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24c101420 0x7fc24c10d950 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fc244009cd0 tx=0x7fc24402a8e0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.684+0000 7fc2532d4640 1 -- 172.21.15.98:0/3051890668 shutdown_connections 2024-09-19T08:09:21.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.684+0000 7fc2532d4640 1 --2- 172.21.15.98:0/3051890668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24c101420 0x7fc24c10d950 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.684+0000 7fc2532d4640 1 --2- 172.21.15.98:0/3051890668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc24c100a80 0x7fc24c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.684+0000 7fc2532d4640 1 --2- 172.21.15.98:0/3051890668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc24c106a80 0x7fc24c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.684+0000 7fc2532d4640 1 -- 172.21.15.98:0/3051890668 >> 172.21.15.98:0/3051890668 conn(0x7fc24c0fc7a0 msgr2=0x7fc24c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:21.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.684+0000 7fc2532d4640 1 -- 172.21.15.98:0/3051890668 shutdown_connections 2024-09-19T08:09:21.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.684+0000 7fc2532d4640 1 -- 172.21.15.98:0/3051890668 wait complete. 2024-09-19T08:09:21.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.684+0000 7fc2532d4640 1 Processor -- start 2024-09-19T08:09:21.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.684+0000 7fc2532d4640 1 -- start start 2024-09-19T08:09:21.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc2532d4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc24c100a80 0x7fc24c1992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc2532d4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc24c101420 0x7fc24c1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc2532d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24c106a80 0x7fc24c1a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc2532d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c110e80 con 0x7fc24c106a80 2024-09-19T08:09:21.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc2532d4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c110d00 con 0x7fc24c100a80 2024-09-19T08:09:21.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc2532d4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c111000 con 0x7fc24c101420 2024-09-19T08:09:21.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc251871640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24c106a80 0x7fc24c1a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc251871640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24c106a80 0x7fc24c1a4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38140/0 (socket says 172.21.15.98:38140) 2024-09-19T08:09:21.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc251070640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc24c100a80 0x7fc24c1992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc251871640 1 -- 172.21.15.98:0/3169114409 learned_addr learned my addr 172.21.15.98:0/3169114409 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc25086f640 1 --2- 172.21.15.98:0/3169114409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc24c101420 0x7fc24c1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc251871640 1 -- 172.21.15.98:0/3169114409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc24c101420 msgr2=0x7fc24c1997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc251871640 1 --2- 172.21.15.98:0/3169114409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc24c101420 0x7fc24c1997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc251871640 1 -- 172.21.15.98:0/3169114409 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc24c100a80 msgr2=0x7fc24c1992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc251871640 1 --2- 172.21.15.98:0/3169114409 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc24c100a80 0x7fc24c1992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc251871640 1 -- 172.21.15.98:0/3169114409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c1a4870 con 0x7fc24c106a80 2024-09-19T08:09:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc251871640 1 --2- 172.21.15.98:0/3169114409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24c106a80 0x7fc24c1a4330 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fc244005cc0 tx=0x7fc244003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc2427fc640 1 -- 172.21.15.98:0/3169114409 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244004040 con 0x7fc24c106a80 2024-09-19T08:09:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc2427fc640 1 -- 172.21.15.98:0/3169114409 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2440046c0 con 0x7fc24c106a80 2024-09-19T08:09:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc2532d4640 1 -- 172.21.15.98:0/3169114409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24c1a4a40 con 0x7fc24c106a80 2024-09-19T08:09:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.685+0000 7fc2427fc640 1 -- 172.21.15.98:0/3169114409 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244046480 con 0x7fc24c106a80 2024-09-19T08:09:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.686+0000 7fc2532d4640 1 -- 172.21.15.98:0/3169114409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24c1a4f20 con 0x7fc24c106a80 2024-09-19T08:09:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.688+0000 7fc2427fc640 1 -- 172.21.15.98:0/3169114409 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fc2440305a0 con 0x7fc24c106a80 2024-09-19T08:09:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.688+0000 7fc2556ad740 1 Processor -- start 2024-09-19T08:09:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.688+0000 7fc2556ad740 1 -- start start 2024-09-19T08:09:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.688+0000 7fc2556ad740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b30b40 0x55a0f4b2f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.688+0000 7fc2556ad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b2fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.688+0000 7fc2556ad740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b3a370 0x55a0f4b3c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.688+0000 7fc2556ad740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a0f4a952f0 con 0x55a0f4b3a370 2024-09-19T08:09:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.688+0000 7fc2556ad740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a0f4a3ccf0 con 0x55a0f4b30b40 2024-09-19T08:09:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.688+0000 7fc2556ad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a0f4a912b0 con 0x55a0f4b2f5d0 2024-09-19T08:09:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.688+0000 7fc2427fc640 1 --2- 172.21.15.98:0/3169114409 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc2380a3430 0x7fc2380a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.688+0000 7fc25086f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b3a370 0x55a0f4b3c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.688+0000 7fc25086f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b3a370 0x55a0f4b3c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38150/0 (socket says 172.21.15.98:38150) 2024-09-19T08:09:21.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc25086f640 1 -- 172.21.15.98:0/3421994930 learned_addr learned my addr 172.21.15.98:0/3421994930 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc25086f640 1 --2- 172.21.15.98:0/3421994930 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b30b40 0x55a0f4b2f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc251871640 1 --2- 172.21.15.98:0/3169114409 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc2380a3430 0x7fc2380a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc251070640 1 --2- 172.21.15.98:0/3421994930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b2fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc25086f640 1 -- 172.21.15.98:0/3421994930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 msgr2=0x55a0f4b2fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc25086f640 1 --2- 172.21.15.98:0/3421994930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b2fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc25086f640 1 -- 172.21.15.98:0/3421994930 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b30b40 msgr2=0x55a0f4b2f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc2427fc640 1 -- 172.21.15.98:0/3169114409 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fc24402e030 con 0x7fc24c106a80 2024-09-19T08:09:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc25086f640 1 --2- 172.21.15.98:0/3421994930 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b30b40 0x55a0f4b2f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc25086f640 1 -- 172.21.15.98:0/3421994930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0f4a213a0 con 0x55a0f4b3a370 2024-09-19T08:09:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc25086f640 1 --2- 172.21.15.98:0/3421994930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b3a370 0x55a0f4b3c760 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0125c0 tx=0x7fc23c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=394bb241cc1a0b26 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc251871640 1 --2- 172.21.15.98:0/3169114409 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc2380a3430 0x7fc2380a58f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc2440eee00 tx=0x7fc24403d040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc2377fe640 1 -- 172.21.15.98:0/3421994930 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c0106b0 con 0x55a0f4b3a370 2024-09-19T08:09:21.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc2377fe640 1 -- 172.21.15.98:0/3421994930 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc23c010d30 con 0x55a0f4b3a370 2024-09-19T08:09:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.689+0000 7fc2377fe640 1 -- 172.21.15.98:0/3421994930 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c01dd90 con 0x55a0f4b3a370 2024-09-19T08:09:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.690+0000 7fc2556ad740 1 -- 172.21.15.98:0/3421994930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b3a370 msgr2=0x55a0f4b3c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.690+0000 7fc2556ad740 1 --2- 172.21.15.98:0/3421994930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b3a370 0x55a0f4b3c760 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0125c0 tx=0x7fc23c00b870 comp rx=0 tx=0).stop 2024-09-19T08:09:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.690+0000 7fc2556ad740 1 -- 172.21.15.98:0/3421994930 shutdown_connections 2024-09-19T08:09:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.690+0000 7fc2556ad740 1 --2- 172.21.15.98:0/3421994930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b3a370 0x55a0f4b3c760 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.690+0000 7fc2556ad740 1 --2- 172.21.15.98:0/3421994930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b2fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.690+0000 7fc2556ad740 1 --2- 172.21.15.98:0/3421994930 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b30b40 0x55a0f4b2f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.690+0000 7fc2556ad740 1 -- 172.21.15.98:0/3421994930 >> 172.21.15.98:0/3421994930 conn(0x55a0f4af8310 msgr2=0x55a0f4b2bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.690+0000 7fc2556ad740 1 -- 172.21.15.98:0/3421994930 shutdown_connections 2024-09-19T08:09:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.690+0000 7fc2556ad740 1 -- 172.21.15.98:0/3421994930 wait complete. 2024-09-19T08:09:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.690+0000 7fc2556ad740 1 -- start start 2024-09-19T08:09:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.690+0000 7fc2556ad740 1 Processor -- start 2024-09-19T08:09:21.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc2556ad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b39d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc2556ad740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b30b40 0x55a0f4b36fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc2556ad740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b3a370 0x55a0f4b375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc2556ad740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a0f4b3d070 con 0x55a0f4b3a370 2024-09-19T08:09:21.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc2556ad740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a0f4a50060 con 0x55a0f4b30b40 2024-09-19T08:09:21.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc2556ad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a0f4b3d1f0 con 0x55a0f4b2f5d0 2024-09-19T08:09:21.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc2556ad740 10 client.0 _ll_register_callbacks cb 0x55a0f47a38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:21.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc25086f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b3a370 0x55a0f4b375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc25086f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b3a370 0x55a0f4b375d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38158/0 (socket says 172.21.15.98:38158) 2024-09-19T08:09:21.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc25086f640 1 -- 172.21.15.98:0/3312998238 learned_addr learned my addr 172.21.15.98:0/3312998238 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:21.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc251070640 1 --2- 172.21.15.98:0/3312998238 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b30b40 0x55a0f4b36fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc25086f640 1 --2- 172.21.15.98:0/3312998238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b39d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc25086f640 1 -- 172.21.15.98:0/3312998238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 msgr2=0x55a0f4b39d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:21.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc25086f640 1 --2- 172.21.15.98:0/3312998238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b39d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc25086f640 1 -- 172.21.15.98:0/3312998238 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b30b40 msgr2=0x55a0f4b36fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:21.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc25086f640 1 --2- 172.21.15.98:0/3312998238 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b30b40 0x55a0f4b36fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc25086f640 1 -- 172.21.15.98:0/3312998238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0f48063c0 con 0x55a0f4b3a370 2024-09-19T08:09:21.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc25086f640 1 --2- 172.21.15.98:0/3312998238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b3a370 0x55a0f4b375d0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fc23c001eb0 tx=0x7fc23c01d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e18b8fd19d61c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:21.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.691+0000 7fc234ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:21.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc234ff9640 1 -- 172.21.15.98:0/3312998238 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c031060 con 0x55a0f4b3a370 2024-09-19T08:09:21.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc234ff9640 1 -- 172.21.15.98:0/3312998238 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fc23c005050 con 0x55a0f4b3a370 2024-09-19T08:09:21.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc2556ad740 1 -- 172.21.15.98:0/3312998238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc23c010d30 con 0x55a0f4b3a370 2024-09-19T08:09:21.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc234ff9640 1 -- 172.21.15.98:0/3312998238 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc23c023040 con 0x55a0f4b3a370 2024-09-19T08:09:21.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc2556ad740 10 client.17016 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:21.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc2556ad740 1 -- 172.21.15.98:0/3312998238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a0f4b48b00 con 0x55a0f4b3a370 2024-09-19T08:09:21.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc234ff9640 1 -- 172.21.15.98:0/3312998238 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc23c010d30 con 0x55a0f4b3a370 2024-09-19T08:09:21.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc234ff9640 1 -- 172.21.15.98:0/3312998238 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc23c029030 con 0x55a0f4b3a370 2024-09-19T08:09:21.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc2556ad740 10 client.17016 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:21.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc2556ad740 10 client.17016 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:09:21.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc2556ad740 20 client.17016 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:21.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc2556ad740 1 --2- 172.21.15.98:0/3312998238 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a0f4b49fc0 0x55a0f4b4c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc2556ad740 4 client.17016 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:09:21.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc2556ad740 1 -- 172.21.15.98:0/3312998238 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0f4aa2860 con 0x55a0f4b49fc0 2024-09-19T08:09:21.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.692+0000 7fc251070640 1 --2- 172.21.15.98:0/3312998238 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a0f4b49fc0 0x55a0f4b4c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.693+0000 7fc234ff9640 10 client.17016 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:09:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.693+0000 7fc251070640 1 --2- 172.21.15.98:0/3312998238 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a0f4b49fc0 0x55a0f4b4c3b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=60848a3b390dd14d server_cookie=55480d8785c19dbe in_seq=0 out_seq=0 2024-09-19T08:09:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.693+0000 7fc234ff9640 1 -- 172.21.15.98:0/3312998238 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a0f4aa2860 con 0x55a0f4b49fc0 2024-09-19T08:09:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.693+0000 7fc234ff9640 10 client.17016 handle_command_reply: tid=0 2024-09-19T08:09:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.693+0000 7fc2556ad740 1 client.17016 shutdown 2024-09-19T08:09:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.694+0000 7fc2556ad740 1 -- 172.21.15.98:0/3312998238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b3a370 msgr2=0x55a0f4b375d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.694+0000 7fc2556ad740 1 --2- 172.21.15.98:0/3312998238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b3a370 0x55a0f4b375d0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fc23c001eb0 tx=0x7fc23c01d8f0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.694+0000 7fc2556ad740 1 -- 172.21.15.98:0/3312998238 shutdown_connections 2024-09-19T08:09:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.694+0000 7fc2556ad740 1 --2- 172.21.15.98:0/3312998238 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a0f4b49fc0 0x55a0f4b4c3b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.694+0000 7fc2556ad740 1 --2- 172.21.15.98:0/3312998238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b3a370 0x55a0f4b375d0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.694+0000 7fc2556ad740 1 --2- 172.21.15.98:0/3312998238 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b30b40 0x55a0f4b36fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.694+0000 7fc2556ad740 1 --2- 172.21.15.98:0/3312998238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b39d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.694+0000 7fc234ff9640 0 client.17016 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:09:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.694+0000 7fc2556ad740 1 -- 172.21.15.98:0/3312998238 >> 172.21.15.98:0/3312998238 conn(0x55a0f4af8310 msgr2=0x55a0f4b31960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:21.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.694+0000 7fc2556ad740 1 -- 172.21.15.98:0/3312998238 shutdown_connections 2024-09-19T08:09:21.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.694+0000 7fc2556ad740 1 -- 172.21.15.98:0/3312998238 wait complete. 2024-09-19T08:09:21.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.694+0000 7fc2556ad740 20 client.17016 trim_cache size 0 max 16384 2024-09-19T08:09:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc2556ad740 1 Processor -- start 2024-09-19T08:09:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc2556ad740 1 -- start start 2024-09-19T08:09:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc2556ad740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b2be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc2556ad740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b30b40 0x55a0f4b46340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc2556ad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b3a370 0x55a0f4b368a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc2556ad740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1ad740 con 0x55a0f4b30b40 2024-09-19T08:09:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc2556ad740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1af280 con 0x55a0f4b2f5d0 2024-09-19T08:09:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc2556ad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1ade10 con 0x55a0f4b3a370 2024-09-19T08:09:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc251070640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b30b40 0x55a0f4b46340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc251070640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b30b40 0x55a0f4b46340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38164/0 (socket says 172.21.15.98:38164) 2024-09-19T08:09:21.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc251070640 1 -- 172.21.15.98:0/2122365632 learned_addr learned my addr 172.21.15.98:0/2122365632 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc251070640 1 -- 172.21.15.98:0/2122365632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b3a370 msgr2=0x55a0f4b368a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc251070640 1 --2- 172.21.15.98:0/2122365632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b3a370 0x55a0f4b368a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc251070640 1 -- 172.21.15.98:0/2122365632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b2f5d0 msgr2=0x55a0f4b2be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc251070640 1 --2- 172.21.15.98:0/2122365632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b2be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc251070640 1 -- 172.21.15.98:0/2122365632 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0f4b35fb0 con 0x55a0f4b30b40 2024-09-19T08:09:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.707+0000 7fc251070640 1 --2- 172.21.15.98:0/2122365632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b30b40 0x55a0f4b46340 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x55a0f4b38620 tx=0x7fc24800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d78877f3232ea4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.708+0000 7fc236ffd640 1 -- 172.21.15.98:0/2122365632 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc248003d80 con 0x55a0f4b30b40 2024-09-19T08:09:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.708+0000 7fc236ffd640 1 -- 172.21.15.98:0/2122365632 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc24801d900 con 0x55a0f4b30b40 2024-09-19T08:09:21.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.708+0000 7fc236ffd640 1 -- 172.21.15.98:0/2122365632 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc248011bc0 con 0x55a0f4b30b40 2024-09-19T08:09:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.708+0000 7fc2556ad740 1 -- 172.21.15.98:0/2122365632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b30b40 msgr2=0x55a0f4b46340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.708+0000 7fc2556ad740 1 --2- 172.21.15.98:0/2122365632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b30b40 0x55a0f4b46340 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x55a0f4b38620 tx=0x7fc24800be60 comp rx=0 tx=0).stop 2024-09-19T08:09:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.708+0000 7fc2556ad740 1 -- 172.21.15.98:0/2122365632 shutdown_connections 2024-09-19T08:09:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.708+0000 7fc2556ad740 1 --2- 172.21.15.98:0/2122365632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b3a370 0x55a0f4b368a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.708+0000 7fc2556ad740 1 --2- 172.21.15.98:0/2122365632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b30b40 0x55a0f4b46340 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.708+0000 7fc2556ad740 1 --2- 172.21.15.98:0/2122365632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b2be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.708+0000 7fc2556ad740 1 -- 172.21.15.98:0/2122365632 >> 172.21.15.98:0/2122365632 conn(0x55a0f4af8310 msgr2=0x55a0f4b2b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.708+0000 7fc2556ad740 1 -- 172.21.15.98:0/2122365632 shutdown_connections 2024-09-19T08:09:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.708+0000 7fc2556ad740 1 -- 172.21.15.98:0/2122365632 wait complete. 2024-09-19T08:09:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.709+0000 7fc2556ad740 1 -- start start 2024-09-19T08:09:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.709+0000 7fc2556ad740 1 Processor -- start 2024-09-19T08:09:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.709+0000 7fc2556ad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b42d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.709+0000 7fc2556ad740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b30b40 0x55a0f4b33e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.709+0000 7fc2556ad740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b3a370 0x55a0f4b343e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.709+0000 7fc2556ad740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1aebb0 con 0x55a0f4b30b40 2024-09-19T08:09:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.709+0000 7fc2556ad740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1ae4e0 con 0x55a0f4b3a370 2024-09-19T08:09:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.709+0000 7fc2556ad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1ab720 con 0x55a0f4b2f5d0 2024-09-19T08:09:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.709+0000 7fc2556ad740 10 client.0 _ll_register_callbacks cb 0x7fc23c0123f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.709+0000 7fc251070640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b30b40 0x55a0f4b33e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.709+0000 7fc251070640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b30b40 0x55a0f4b33e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38180/0 (socket says 172.21.15.98:38180) 2024-09-19T08:09:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.709+0000 7fc251070640 1 -- 172.21.15.98:0/4018729558 learned_addr learned my addr 172.21.15.98:0/4018729558 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.709+0000 7fc25086f640 1 --2- 172.21.15.98:0/4018729558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b42d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc25086f640 1 --2- 172.21.15.98:0/4018729558 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b3a370 0x55a0f4b343e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc251070640 1 -- 172.21.15.98:0/4018729558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 msgr2=0x55a0f4b42d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc251070640 1 --2- 172.21.15.98:0/4018729558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b42d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc251070640 1 -- 172.21.15.98:0/4018729558 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b3a370 msgr2=0x55a0f4b343e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc251070640 1 --2- 172.21.15.98:0/4018729558 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b3a370 0x55a0f4b343e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc251070640 1 -- 172.21.15.98:0/4018729558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0f4b34c10 con 0x55a0f4b30b40 2024-09-19T08:09:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc251070640 1 --2- 172.21.15.98:0/4018729558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b30b40 0x55a0f4b33e40 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc24800b6b0 tx=0x7fc24800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28ce0b993ba960c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc234ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc234ff9640 1 -- 172.21.15.98:0/4018729558 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24801dcb0 con 0x55a0f4b30b40 2024-09-19T08:09:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc234ff9640 1 -- 172.21.15.98:0/4018729558 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fc248003b80 con 0x55a0f4b30b40 2024-09-19T08:09:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc2556ad740 1 -- 172.21.15.98:0/4018729558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc24801d900 con 0x55a0f4b30b40 2024-09-19T08:09:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc234ff9640 1 -- 172.21.15.98:0/4018729558 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc248003d80 con 0x55a0f4b30b40 2024-09-19T08:09:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc2556ad740 10 client.17022 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc2556ad740 1 -- 172.21.15.98:0/4018729558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a0f4bb4bc0 con 0x55a0f4b30b40 2024-09-19T08:09:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc234ff9640 1 -- 172.21.15.98:0/4018729558 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc24801d900 con 0x55a0f4b30b40 2024-09-19T08:09:21.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.710+0000 7fc234ff9640 1 -- 172.21.15.98:0/4018729558 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc24802e070 con 0x55a0f4b30b40 2024-09-19T08:09:21.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.711+0000 7fc2556ad740 10 client.17022 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:21.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.711+0000 7fc2556ad740 10 client.17022 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:09:21.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.711+0000 7fc2556ad740 20 client.17022 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:21.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.711+0000 7fc2556ad740 1 --2- 172.21.15.98:0/4018729558 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a0f4bb52c0 0x55a0f4bb76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:21.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.711+0000 7fc2556ad740 4 client.17022 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:09:21.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.711+0000 7fc2556ad740 1 -- 172.21.15.98:0/4018729558 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a0f4a9cd70 con 0x55a0f4bb52c0 2024-09-19T08:09:21.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.711+0000 7fc25086f640 1 --2- 172.21.15.98:0/4018729558 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a0f4bb52c0 0x55a0f4bb76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:21.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.711+0000 7fc25086f640 1 --2- 172.21.15.98:0/4018729558 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a0f4bb52c0 0x55a0f4bb76e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=11abb3cef8a53a9d server_cookie=70e6c5deb9f7c9de in_seq=0 out_seq=0 2024-09-19T08:09:21.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.711+0000 7fc234ff9640 10 client.17022 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:09:21.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.711+0000 7fc234ff9640 1 -- 172.21.15.98:0/4018729558 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a0f4a9cd70 con 0x55a0f4bb52c0 2024-09-19T08:09:21.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.711+0000 7fc234ff9640 10 client.17022 handle_command_reply: tid=0 2024-09-19T08:09:21.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.711+0000 7fc2556ad740 1 client.17022 shutdown 2024-09-19T08:09:21.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.711+0000 7fc2556ad740 1 -- 172.21.15.98:0/4018729558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b30b40 msgr2=0x55a0f4b33e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:21.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.711+0000 7fc2556ad740 1 --2- 172.21.15.98:0/4018729558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b30b40 0x55a0f4b33e40 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc24800b6b0 tx=0x7fc24800b8c0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2556ad740 1 -- 172.21.15.98:0/4018729558 shutdown_connections 2024-09-19T08:09:21.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2556ad740 1 --2- 172.21.15.98:0/4018729558 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55a0f4bb52c0 0x55a0f4bb76e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2556ad740 1 --2- 172.21.15.98:0/4018729558 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a0f4b3a370 0x55a0f4b343e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2556ad740 1 --2- 172.21.15.98:0/4018729558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0f4b30b40 0x55a0f4b33e40 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2556ad740 1 --2- 172.21.15.98:0/4018729558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a0f4b2f5d0 0x55a0f4b42d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2556ad740 1 -- 172.21.15.98:0/4018729558 >> 172.21.15.98:0/4018729558 conn(0x55a0f4af8310 msgr2=0x55a0f4b31110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:21.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc234ff9640 0 client.17022 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:09:21.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2556ad740 1 -- 172.21.15.98:0/4018729558 shutdown_connections 2024-09-19T08:09:21.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2556ad740 1 -- 172.21.15.98:0/4018729558 wait complete. 2024-09-19T08:09:21.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2556ad740 20 client.17022 trim_cache size 0 max 16384 2024-09-19T08:09:21.718 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:09:21.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2532d4640 1 -- 172.21.15.98:0/3169114409 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc2380a3430 msgr2=0x7fc2380a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:21.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2532d4640 1 --2- 172.21.15.98:0/3169114409 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc2380a3430 0x7fc2380a58f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc2440eee00 tx=0x7fc24403d040 comp rx=0 tx=0).stop 2024-09-19T08:09:21.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2532d4640 1 -- 172.21.15.98:0/3169114409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24c106a80 msgr2=0x7fc24c1a4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:21.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2532d4640 1 --2- 172.21.15.98:0/3169114409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24c106a80 0x7fc24c1a4330 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fc244005cc0 tx=0x7fc244003970 comp rx=0 tx=0).stop 2024-09-19T08:09:21.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2532d4640 1 -- 172.21.15.98:0/3169114409 shutdown_connections 2024-09-19T08:09:21.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2532d4640 1 --2- 172.21.15.98:0/3169114409 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc2380a3430 0x7fc2380a58f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2532d4640 1 --2- 172.21.15.98:0/3169114409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24c106a80 0x7fc24c1a4330 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2532d4640 1 --2- 172.21.15.98:0/3169114409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc24c101420 0x7fc24c1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.712+0000 7fc2532d4640 1 --2- 172.21.15.98:0/3169114409 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc24c100a80 0x7fc24c1992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:21.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.713+0000 7fc2532d4640 1 -- 172.21.15.98:0/3169114409 >> 172.21.15.98:0/3169114409 conn(0x7fc24c0fc7a0 msgr2=0x7fc24c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:21.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.713+0000 7fc2532d4640 1 -- 172.21.15.98:0/3169114409 shutdown_connections 2024-09-19T08:09:21.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:21.713+0000 7fc2532d4640 1 -- 172.21.15.98:0/3169114409 wait complete. 2024-09-19T08:09:22.366 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:22 smithi098 ceph-mon[31333]: pgmap v193: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-19T08:09:22.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:22 smithi162 ceph-mon[35260]: pgmap v193: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-19T08:09:22.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:09:22.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:22 smithi139 ceph-mon[34874]: pgmap v193: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-19T08:09:22.637 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:09:23.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.134+0000 7f22dcc03640 1 -- 172.21.15.98:0/3936874731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d8106a80 msgr2=0x7f22d8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:23.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.134+0000 7f22dcc03640 1 --2- 172.21.15.98:0/3936874731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d8106a80 0x7f22d8106e60 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009930 tx=0x7f22c802a420 comp rx=0 tx=0).stop 2024-09-19T08:09:23.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.134+0000 7f22dcc03640 1 -- 172.21.15.98:0/3936874731 shutdown_connections 2024-09-19T08:09:23.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.134+0000 7f22dcc03640 1 --2- 172.21.15.98:0/3936874731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22d8101420 0x7f22d810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.134+0000 7f22dcc03640 1 --2- 172.21.15.98:0/3936874731 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f22d8100a80 0x7f22d8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.134+0000 7f22dcc03640 1 --2- 172.21.15.98:0/3936874731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d8106a80 0x7f22d8106e60 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.134+0000 7f22dcc03640 1 -- 172.21.15.98:0/3936874731 >> 172.21.15.98:0/3936874731 conn(0x7f22d80fc7a0 msgr2=0x7f22d80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:23.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.134+0000 7f22dcc03640 1 -- 172.21.15.98:0/3936874731 shutdown_connections 2024-09-19T08:09:23.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.134+0000 7f22dcc03640 1 -- 172.21.15.98:0/3936874731 wait complete. 2024-09-19T08:09:23.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22dcc03640 1 Processor -- start 2024-09-19T08:09:23.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22dcc03640 1 -- start start 2024-09-19T08:09:23.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22dcc03640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22d8100a80 0x7f22d81984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22dcc03640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f22d8101420 0x7f22d8198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22dcc03640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d8106a80 0x7f22d819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22dcc03640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f22d8111fa0 con 0x7f22d8106a80 2024-09-19T08:09:23.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22dcc03640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f22d8111e20 con 0x7f22d8101420 2024-09-19T08:09:23.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22dcc03640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22d8112120 con 0x7f22d8100a80 2024-09-19T08:09:23.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22d6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d8106a80 0x7f22d819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22d6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d8106a80 0x7f22d819cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38198/0 (socket says 172.21.15.98:38198) 2024-09-19T08:09:23.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22d6d9d640 1 -- 172.21.15.98:0/2036611301 learned_addr learned my addr 172.21.15.98:0/2036611301 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:23.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22cffff640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f22d8101420 0x7f22d8198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22d659c640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22d8100a80 0x7f22d81984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22d6d9d640 1 -- 172.21.15.98:0/2036611301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22d8100a80 msgr2=0x7f22d81984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:23.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22d6d9d640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22d8100a80 0x7f22d81984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.135+0000 7f22d6d9d640 1 -- 172.21.15.98:0/2036611301 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f22d8101420 msgr2=0x7f22d8198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.136+0000 7f22d6d9d640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f22d8101420 0x7f22d8198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.136+0000 7f22d6d9d640 1 -- 172.21.15.98:0/2036611301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d819d490 con 0x7f22d8106a80 2024-09-19T08:09:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.136+0000 7f22d659c640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22d8100a80 0x7f22d81984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:09:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.136+0000 7f22d6d9d640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d8106a80 0x7f22d819cdb0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f22d000cad0 tx=0x7f22d000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.136+0000 7f22cf7fe640 1 -- 172.21.15.98:0/2036611301 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d000ad60 con 0x7f22d8106a80 2024-09-19T08:09:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.136+0000 7f22cf7fe640 1 -- 172.21.15.98:0/2036611301 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22d0005020 con 0x7f22d8106a80 2024-09-19T08:09:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.136+0000 7f22dcc03640 1 -- 172.21.15.98:0/2036611301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d819d780 con 0x7f22d8106a80 2024-09-19T08:09:23.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.136+0000 7f22cf7fe640 1 -- 172.21.15.98:0/2036611301 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d000f6b0 con 0x7f22d8106a80 2024-09-19T08:09:23.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.136+0000 7f22dcc03640 1 -- 172.21.15.98:0/2036611301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d81a5060 con 0x7f22d8106a80 2024-09-19T08:09:23.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.138+0000 7f22cf7fe640 1 -- 172.21.15.98:0/2036611301 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f22d0005560 con 0x7f22d8106a80 2024-09-19T08:09:23.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.138+0000 7f22defdc740 1 Processor -- start 2024-09-19T08:09:23.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22defdc740 1 -- start start 2024-09-19T08:09:23.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22cf7fe640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f22c40a33b0 0x7f22c40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22defdc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c370b40 0x564b1c36f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22defdc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 0x564b1c36fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22defdc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c37e780 0x564b1c380b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22defdc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564b1c2d52f0 con 0x564b1c370b40 2024-09-19T08:09:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22defdc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564b1c27ccf0 con 0x564b1c36f5d0 2024-09-19T08:09:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22defdc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564b1c2d12b0 con 0x564b1c37e780 2024-09-19T08:09:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22cffff640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f22c40a33b0 0x7f22c40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22cf7fe640 1 -- 172.21.15.98:0/2036611301 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f22d00cfd80 con 0x7f22d8106a80 2024-09-19T08:09:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22d659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c370b40 0x564b1c36f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22d659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c370b40 0x564b1c36f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38208/0 (socket says 172.21.15.98:38208) 2024-09-19T08:09:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22d659c640 1 -- 172.21.15.98:0/148368898 learned_addr learned my addr 172.21.15.98:0/148368898 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:23.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22d6d9d640 1 --2- 172.21.15.98:0/148368898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c37e780 0x564b1c380b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22cffff640 1 --2- 172.21.15.98:0/148368898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 0x564b1c36fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22d659c640 1 -- 172.21.15.98:0/148368898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c37e780 msgr2=0x564b1c380b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:23.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22d659c640 1 --2- 172.21.15.98:0/148368898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c37e780 0x564b1c380b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22d659c640 1 -- 172.21.15.98:0/148368898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 msgr2=0x564b1c36fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:23.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22d659c640 1 --2- 172.21.15.98:0/148368898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 0x564b1c36fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.139+0000 7f22d659c640 1 -- 172.21.15.98:0/148368898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b1c2613a0 con 0x564b1c370b40 2024-09-19T08:09:23.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.140+0000 7f22cffff640 1 --2- 172.21.15.98:0/148368898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 0x564b1c36fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:09:23.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.140+0000 7f22cffff640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f22c40a33b0 0x7f22c40a5870 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f22d80fed80 tx=0x7f22c00156d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:23.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.140+0000 7f22d659c640 1 --2- 172.21.15.98:0/148368898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c370b40 0x564b1c36f090 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f22c8005ab0 tx=0x7f22c8009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde71e97a97a7684 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:23.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.140+0000 7f22ccff9640 1 -- 172.21.15.98:0/148368898 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c803f070 con 0x564b1c370b40 2024-09-19T08:09:23.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.140+0000 7f22ccff9640 1 -- 172.21.15.98:0/148368898 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22c80317b0 con 0x564b1c370b40 2024-09-19T08:09:23.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.140+0000 7f22ccff9640 1 -- 172.21.15.98:0/148368898 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c8035070 con 0x564b1c370b40 2024-09-19T08:09:23.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 -- 172.21.15.98:0/148368898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c370b40 msgr2=0x564b1c36f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:23.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 --2- 172.21.15.98:0/148368898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c370b40 0x564b1c36f090 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f22c8005ab0 tx=0x7f22c8009680 comp rx=0 tx=0).stop 2024-09-19T08:09:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 -- 172.21.15.98:0/148368898 shutdown_connections 2024-09-19T08:09:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 --2- 172.21.15.98:0/148368898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c37e780 0x564b1c380b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 --2- 172.21.15.98:0/148368898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 0x564b1c36fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 --2- 172.21.15.98:0/148368898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c370b40 0x564b1c36f090 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 -- 172.21.15.98:0/148368898 >> 172.21.15.98:0/148368898 conn(0x564b1c338310 msgr2=0x564b1c36bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 -- 172.21.15.98:0/148368898 shutdown_connections 2024-09-19T08:09:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 -- 172.21.15.98:0/148368898 wait complete. 2024-09-19T08:09:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 -- start start 2024-09-19T08:09:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 Processor -- start 2024-09-19T08:09:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c36f5d0 0x564b1c37e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c370b40 0x564b1c37b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c37e780 0x564b1c37b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564b1c381420 con 0x564b1c36f5d0 2024-09-19T08:09:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564b1c290060 con 0x564b1c370b40 2024-09-19T08:09:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.141+0000 7f22defdc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564b1c3815a0 con 0x564b1c37e780 2024-09-19T08:09:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22defdc740 10 client.0 _ll_register_callbacks cb 0x564b1bfe38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:23.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22d659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c36f5d0 0x564b1c37e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22d659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c36f5d0 0x564b1c37e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38218/0 (socket says 172.21.15.98:38218) 2024-09-19T08:09:23.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22d659c640 1 -- 172.21.15.98:0/3051485787 learned_addr learned my addr 172.21.15.98:0/3051485787 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:23.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22cffff640 1 --2- 172.21.15.98:0/3051485787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c370b40 0x564b1c37b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22d659c640 1 -- 172.21.15.98:0/3051485787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c37e780 msgr2=0x564b1c37b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:23.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22d659c640 1 --2- 172.21.15.98:0/3051485787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c37e780 0x564b1c37b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22d659c640 1 -- 172.21.15.98:0/3051485787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c370b40 msgr2=0x564b1c37b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:23.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22d659c640 1 --2- 172.21.15.98:0/3051485787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c370b40 0x564b1c37b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22d659c640 1 -- 172.21.15.98:0/3051485787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b1c0463c0 con 0x564b1c36f5d0 2024-09-19T08:09:23.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22d659c640 1 --2- 172.21.15.98:0/3051485787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c36f5d0 0x564b1c37e0a0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f22c8005270 tx=0x7f22c80052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a505a67f981b05 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:23.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:23.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22bdffb640 1 -- 172.21.15.98:0/3051485787 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c803f050 con 0x564b1c36f5d0 2024-09-19T08:09:23.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22bdffb640 1 -- 172.21.15.98:0/3051485787 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f22c8004240 con 0x564b1c36f5d0 2024-09-19T08:09:23.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22defdc740 1 -- 172.21.15.98:0/3051485787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22c80317b0 con 0x564b1c36f5d0 2024-09-19T08:09:23.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22bdffb640 1 -- 172.21.15.98:0/3051485787 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f22c8035040 con 0x564b1c36f5d0 2024-09-19T08:09:23.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22defdc740 10 client.17034 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:23.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.142+0000 7f22defdc740 1 -- 172.21.15.98:0/3051485787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564b1c38ce30 con 0x564b1c36f5d0 2024-09-19T08:09:23.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.143+0000 7f22bdffb640 1 -- 172.21.15.98:0/3051485787 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22c80317b0 con 0x564b1c36f5d0 2024-09-19T08:09:23.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.143+0000 7f22bdffb640 1 -- 172.21.15.98:0/3051485787 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f22c80046c0 con 0x564b1c36f5d0 2024-09-19T08:09:23.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.143+0000 7f22defdc740 10 client.17034 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:23.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.143+0000 7f22defdc740 10 client.17034 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:09:23.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.143+0000 7f22defdc740 20 client.17034 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:23.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.143+0000 7f22defdc740 1 --2- 172.21.15.98:0/3051485787 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x564b1c38e180 0x564b1c390570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.143+0000 7f22defdc740 4 client.17034 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:09:23.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.143+0000 7f22defdc740 1 -- 172.21.15.98:0/3051485787 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b1c2e2860 con 0x564b1c38e180 2024-09-19T08:09:23.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.143+0000 7f22cffff640 1 --2- 172.21.15.98:0/3051485787 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x564b1c38e180 0x564b1c390570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.143+0000 7f22bdffb640 10 client.17034 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:09:23.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.143+0000 7f22cffff640 1 --2- 172.21.15.98:0/3051485787 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x564b1c38e180 0x564b1c390570 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6eb6dc7c13b0e9a7 server_cookie=5f18614576d416bb in_seq=0 out_seq=0 2024-09-19T08:09:23.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.144+0000 7f22bdffb640 1 -- 172.21.15.98:0/3051485787 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564b1c2e2860 con 0x564b1c38e180 2024-09-19T08:09:23.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.144+0000 7f22bdffb640 10 client.17034 handle_command_reply: tid=0 2024-09-19T08:09:23.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.144+0000 7f22defdc740 1 client.17034 shutdown 2024-09-19T08:09:23.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.144+0000 7f22defdc740 1 -- 172.21.15.98:0/3051485787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c36f5d0 msgr2=0x564b1c37e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:23.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.144+0000 7f22defdc740 1 --2- 172.21.15.98:0/3051485787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c36f5d0 0x564b1c37e0a0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f22c8005270 tx=0x7f22c80052d0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.144+0000 7f22defdc740 1 -- 172.21.15.98:0/3051485787 shutdown_connections 2024-09-19T08:09:23.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.144+0000 7f22defdc740 1 --2- 172.21.15.98:0/3051485787 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x564b1c38e180 0x564b1c390570 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.144+0000 7f22bdffb640 0 client.17034 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:09:23.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.144+0000 7f22defdc740 1 --2- 172.21.15.98:0/3051485787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c37e780 0x564b1c37b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.144+0000 7f22defdc740 1 --2- 172.21.15.98:0/3051485787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c370b40 0x564b1c37b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.144+0000 7f22defdc740 1 --2- 172.21.15.98:0/3051485787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c36f5d0 0x564b1c37e0a0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.144+0000 7f22defdc740 1 -- 172.21.15.98:0/3051485787 >> 172.21.15.98:0/3051485787 conn(0x564b1c338310 msgr2=0x564b1c371960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:23.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.145+0000 7f22defdc740 1 -- 172.21.15.98:0/3051485787 shutdown_connections 2024-09-19T08:09:23.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.145+0000 7f22defdc740 1 -- 172.21.15.98:0/3051485787 wait complete. 2024-09-19T08:09:23.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.145+0000 7f22defdc740 20 client.17034 trim_cache size 0 max 16384 2024-09-19T08:09:23.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.157+0000 7f22defdc740 1 Processor -- start 2024-09-19T08:09:23.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.157+0000 7f22defdc740 1 -- start start 2024-09-19T08:09:23.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22defdc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 0x564b1c38a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22defdc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c370b40 0x564b1c395bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22defdc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c37e780 0x564b1c396110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22defdc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81ab5e0 con 0x564b1c370b40 2024-09-19T08:09:23.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22defdc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81ad120 con 0x564b1c36f5d0 2024-09-19T08:09:23.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22defdc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81abcb0 con 0x564b1c37e780 2024-09-19T08:09:23.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22cffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c370b40 0x564b1c395bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22cffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c370b40 0x564b1c395bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38234/0 (socket says 172.21.15.98:38234) 2024-09-19T08:09:23.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22cffff640 1 -- 172.21.15.98:0/2132201240 learned_addr learned my addr 172.21.15.98:0/2132201240 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:23.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22d659c640 1 --2- 172.21.15.98:0/2132201240 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 0x564b1c38a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22d6d9d640 1 --2- 172.21.15.98:0/2132201240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c37e780 0x564b1c396110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22cffff640 1 -- 172.21.15.98:0/2132201240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c37e780 msgr2=0x564b1c396110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:23.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22cffff640 1 --2- 172.21.15.98:0/2132201240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c37e780 0x564b1c396110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22cffff640 1 -- 172.21.15.98:0/2132201240 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 msgr2=0x564b1c38a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:23.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22cffff640 1 --2- 172.21.15.98:0/2132201240 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 0x564b1c38a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22cffff640 1 -- 172.21.15.98:0/2132201240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b1c381590 con 0x564b1c370b40 2024-09-19T08:09:23.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22cffff640 1 --2- 172.21.15.98:0/2132201240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c370b40 0x564b1c395bd0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x564b1c36cac0 tx=0x7f22c001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e80afef0b3c85cd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:23.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22bffff640 1 -- 172.21.15.98:0/2132201240 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c0034070 con 0x564b1c370b40 2024-09-19T08:09:23.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22bffff640 1 -- 172.21.15.98:0/2132201240 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22c002fb00 con 0x564b1c370b40 2024-09-19T08:09:23.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.158+0000 7f22bffff640 1 -- 172.21.15.98:0/2132201240 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c0021040 con 0x564b1c370b40 2024-09-19T08:09:23.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.159+0000 7f22defdc740 1 -- 172.21.15.98:0/2132201240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c370b40 msgr2=0x564b1c395bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:23.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.159+0000 7f22defdc740 1 --2- 172.21.15.98:0/2132201240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c370b40 0x564b1c395bd0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x564b1c36cac0 tx=0x7f22c001c000 comp rx=0 tx=0).stop 2024-09-19T08:09:23.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.159+0000 7f22defdc740 1 -- 172.21.15.98:0/2132201240 shutdown_connections 2024-09-19T08:09:23.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.159+0000 7f22defdc740 1 --2- 172.21.15.98:0/2132201240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c37e780 0x564b1c396110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.159+0000 7f22defdc740 1 --2- 172.21.15.98:0/2132201240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c370b40 0x564b1c395bd0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.159+0000 7f22defdc740 1 --2- 172.21.15.98:0/2132201240 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 0x564b1c38a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.159+0000 7f22defdc740 1 -- 172.21.15.98:0/2132201240 >> 172.21.15.98:0/2132201240 conn(0x564b1c338310 msgr2=0x564b1c37a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:23.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.159+0000 7f22defdc740 1 -- 172.21.15.98:0/2132201240 shutdown_connections 2024-09-19T08:09:23.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.159+0000 7f22defdc740 1 -- 172.21.15.98:0/2132201240 wait complete. 2024-09-19T08:09:23.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22defdc740 1 -- start start 2024-09-19T08:09:23.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22defdc740 1 Processor -- start 2024-09-19T08:09:23.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22defdc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 0x564b1c386e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22defdc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c370b40 0x564b1c377f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22defdc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c37e780 0x564b1c3784a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22defdc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81aca50 con 0x564b1c37e780 2024-09-19T08:09:23.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22defdc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81ac380 con 0x564b1c36f5d0 2024-09-19T08:09:23.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22defdc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81a95c0 con 0x564b1c370b40 2024-09-19T08:09:23.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22defdc740 10 client.0 _ll_register_callbacks cb 0x7f22d0098330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:23.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22d6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c37e780 0x564b1c3784a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22d6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c37e780 0x564b1c3784a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38236/0 (socket says 172.21.15.98:38236) 2024-09-19T08:09:23.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22d6d9d640 1 -- 172.21.15.98:0/765309187 learned_addr learned my addr 172.21.15.98:0/765309187 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:23.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22cffff640 1 --2- 172.21.15.98:0/765309187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c370b40 0x564b1c377f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22d659c640 1 --2- 172.21.15.98:0/765309187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 0x564b1c386e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22d6d9d640 1 -- 172.21.15.98:0/765309187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c370b40 msgr2=0x564b1c377f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:23.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22d6d9d640 1 --2- 172.21.15.98:0/765309187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c370b40 0x564b1c377f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22d6d9d640 1 -- 172.21.15.98:0/765309187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 msgr2=0x564b1c386e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:23.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22d6d9d640 1 --2- 172.21.15.98:0/765309187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 0x564b1c386e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.160+0000 7f22d6d9d640 1 -- 172.21.15.98:0/765309187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b1c378b40 con 0x564b1c37e780 2024-09-19T08:09:23.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22d6d9d640 1 --2- 172.21.15.98:0/765309187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c37e780 0x564b1c3784a0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f22d009bd90 tx=0x7f22d009c150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=983522813006a59b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:23.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:23.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22bdffb640 1 -- 172.21.15.98:0/765309187 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d001d040 con 0x564b1c37e780 2024-09-19T08:09:23.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22bdffb640 1 -- 172.21.15.98:0/765309187 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f22d00a00e0 con 0x564b1c37e780 2024-09-19T08:09:23.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22defdc740 1 -- 172.21.15.98:0/765309187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22c002fb00 con 0x564b1c37e780 2024-09-19T08:09:23.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22bdffb640 1 -- 172.21.15.98:0/765309187 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f22d00b5050 con 0x564b1c37e780 2024-09-19T08:09:23.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22defdc740 10 client.17040 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:23.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22defdc740 1 -- 172.21.15.98:0/765309187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564b1c3f8a90 con 0x564b1c37e780 2024-09-19T08:09:23.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22bdffb640 1 -- 172.21.15.98:0/765309187 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22c002fb00 con 0x564b1c37e780 2024-09-19T08:09:23.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22bdffb640 1 -- 172.21.15.98:0/765309187 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f22d009b570 con 0x564b1c37e780 2024-09-19T08:09:23.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22defdc740 10 client.17040 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:23.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22defdc740 10 client.17040 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:09:23.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22defdc740 20 client.17040 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:23.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22defdc740 1 --2- 172.21.15.98:0/765309187 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x564b1c38e180 0x564b1c3fb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:23.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22defdc740 4 client.17040 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:09:23.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22defdc740 1 -- 172.21.15.98:0/765309187 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564b1c2dcd70 con 0x564b1c38e180 2024-09-19T08:09:23.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.161+0000 7f22d659c640 1 --2- 172.21.15.98:0/765309187 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x564b1c38e180 0x564b1c3fb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:23.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22d659c640 1 --2- 172.21.15.98:0/765309187 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x564b1c38e180 0x564b1c3fb220 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=95cadffe9d466042 server_cookie=9efdbdd1cd7dc355 in_seq=0 out_seq=0 2024-09-19T08:09:23.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22bdffb640 10 client.17040 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:09:23.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22bdffb640 1 -- 172.21.15.98:0/765309187 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564b1c2dcd70 con 0x564b1c38e180 2024-09-19T08:09:23.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22bdffb640 10 client.17040 handle_command_reply: tid=0 2024-09-19T08:09:23.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22defdc740 1 client.17040 shutdown 2024-09-19T08:09:23.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22defdc740 1 -- 172.21.15.98:0/765309187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c37e780 msgr2=0x564b1c3784a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:23.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22defdc740 1 --2- 172.21.15.98:0/765309187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c37e780 0x564b1c3784a0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f22d009bd90 tx=0x7f22d009c150 comp rx=0 tx=0).stop 2024-09-19T08:09:23.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22defdc740 1 -- 172.21.15.98:0/765309187 shutdown_connections 2024-09-19T08:09:23.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22defdc740 1 --2- 172.21.15.98:0/765309187 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x564b1c38e180 0x564b1c3fb220 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22defdc740 1 --2- 172.21.15.98:0/765309187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b1c37e780 0x564b1c3784a0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22defdc740 1 --2- 172.21.15.98:0/765309187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b1c370b40 0x564b1c377f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22defdc740 1 --2- 172.21.15.98:0/765309187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b1c36f5d0 0x564b1c386e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22defdc740 1 -- 172.21.15.98:0/765309187 >> 172.21.15.98:0/765309187 conn(0x564b1c338310 msgr2=0x564b1c36a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:23.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22bdffb640 0 client.17040 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:09:23.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22defdc740 1 -- 172.21.15.98:0/765309187 shutdown_connections 2024-09-19T08:09:23.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22defdc740 1 -- 172.21.15.98:0/765309187 wait complete. 2024-09-19T08:09:23.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.162+0000 7f22defdc740 20 client.17040 trim_cache size 0 max 16384 2024-09-19T08:09:23.184 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:09:23.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.163+0000 7f22dcc03640 1 -- 172.21.15.98:0/2036611301 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f22c40a33b0 msgr2=0x7f22c40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:23.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.163+0000 7f22dcc03640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f22c40a33b0 0x7f22c40a5870 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f22d80fed80 tx=0x7f22c00156d0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.163+0000 7f22dcc03640 1 -- 172.21.15.98:0/2036611301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d8106a80 msgr2=0x7f22d819cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:23.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.163+0000 7f22dcc03640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d8106a80 0x7f22d819cdb0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f22d000cad0 tx=0x7f22d000ced0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.163+0000 7f22dcc03640 1 -- 172.21.15.98:0/2036611301 shutdown_connections 2024-09-19T08:09:23.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.163+0000 7f22dcc03640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f22c40a33b0 0x7f22c40a5870 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.163+0000 7f22dcc03640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22d8106a80 0x7f22d819cdb0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.163+0000 7f22dcc03640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f22d8101420 0x7f22d8198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.163+0000 7f22dcc03640 1 --2- 172.21.15.98:0/2036611301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22d8100a80 0x7f22d81984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:23.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.163+0000 7f22dcc03640 1 -- 172.21.15.98:0/2036611301 >> 172.21.15.98:0/2036611301 conn(0x7f22d80fc7a0 msgr2=0x7f22d80fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:23.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.163+0000 7f22dcc03640 1 -- 172.21.15.98:0/2036611301 shutdown_connections 2024-09-19T08:09:23.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:23.163+0000 7f22dcc03640 1 -- 172.21.15.98:0/2036611301 wait complete. 2024-09-19T08:09:23.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:09:24.083 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:09:24.359 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:24 smithi098 ceph-mon[31333]: pgmap v194: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-09-19T08:09:24.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:24 smithi162 ceph-mon[35260]: pgmap v194: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-09-19T08:09:24.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:24 smithi139 ceph-mon[34874]: pgmap v194: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-09-19T08:09:24.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.561+0000 7f3dc29a8640 1 -- 172.21.15.98:0/3469430736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102d40 msgr2=0x7f3dbc103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:24.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.561+0000 7f3dc29a8640 1 --2- 172.21.15.98:0/3469430736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102d40 0x7f3dbc103120 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f3da4009900 tx=0x7f3da402a480 comp rx=0 tx=0).stop 2024-09-19T08:09:24.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.561+0000 7f3dc29a8640 1 -- 172.21.15.98:0/3469430736 shutdown_connections 2024-09-19T08:09:24.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.561+0000 7f3dc29a8640 1 --2- 172.21.15.98:0/3469430736 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3dbc10c170 0x7f3dbc10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.561+0000 7f3dc29a8640 1 --2- 172.21.15.98:0/3469430736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dbc103660 0x7f3dbc10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.561+0000 7f3dc29a8640 1 --2- 172.21.15.98:0/3469430736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102d40 0x7f3dbc103120 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.561+0000 7f3dc29a8640 1 -- 172.21.15.98:0/3469430736 >> 172.21.15.98:0/3469430736 conn(0x7f3dbc0fc7a0 msgr2=0x7f3dbc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:24.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.561+0000 7f3dc29a8640 1 -- 172.21.15.98:0/3469430736 shutdown_connections 2024-09-19T08:09:24.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.561+0000 7f3dc29a8640 1 -- 172.21.15.98:0/3469430736 wait complete. 2024-09-19T08:09:24.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.561+0000 7f3dc29a8640 1 Processor -- start 2024-09-19T08:09:24.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.561+0000 7f3dc29a8640 1 -- start start 2024-09-19T08:09:24.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dc29a8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102d40 0x7f3dbc198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dc29a8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3dbc103660 0x7f3dbc198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dc29a8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dbc10c170 0x7f3dbc19cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dc29a8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc111f70 con 0x7f3dbc102d40 2024-09-19T08:09:24.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dc29a8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc111df0 con 0x7f3dbc103660 2024-09-19T08:09:24.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dc29a8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1120f0 con 0x7f3dbc10c170 2024-09-19T08:09:24.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dbbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102d40 0x7f3dbc198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dbbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102d40 0x7f3dbc198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38254/0 (socket says 172.21.15.98:38254) 2024-09-19T08:09:24.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dbbfff640 1 -- 172.21.15.98:0/1759635250 learned_addr learned my addr 172.21.15.98:0/1759635250 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:24.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dbb7fe640 1 --2- 172.21.15.98:0/1759635250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3dbc103660 0x7f3dbc198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dc0f45640 1 --2- 172.21.15.98:0/1759635250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dbc10c170 0x7f3dbc19cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dbbfff640 1 -- 172.21.15.98:0/1759635250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dbc10c170 msgr2=0x7f3dbc19cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:24.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dbbfff640 1 --2- 172.21.15.98:0/1759635250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dbc10c170 0x7f3dbc19cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dbbfff640 1 -- 172.21.15.98:0/1759635250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3dbc103660 msgr2=0x7f3dbc198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:24.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dbbfff640 1 --2- 172.21.15.98:0/1759635250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3dbc103660 0x7f3dbc198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dbbfff640 1 -- 172.21.15.98:0/1759635250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dbc19d3e0 con 0x7f3dbc102d40 2024-09-19T08:09:24.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.562+0000 7f3dbbfff640 1 --2- 172.21.15.98:0/1759635250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102d40 0x7f3dbc198430 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f3da4006010 tx=0x7f3da40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:24.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.563+0000 7f3db97fa640 1 -- 172.21.15.98:0/1759635250 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da403f030 con 0x7f3dbc102d40 2024-09-19T08:09:24.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.563+0000 7f3dc29a8640 1 -- 172.21.15.98:0/1759635250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dbc19d670 con 0x7f3dbc102d40 2024-09-19T08:09:24.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.563+0000 7f3db97fa640 1 -- 172.21.15.98:0/1759635250 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da4003ce0 con 0x7f3dbc102d40 2024-09-19T08:09:24.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.563+0000 7f3db97fa640 1 -- 172.21.15.98:0/1759635250 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da4035070 con 0x7f3dbc102d40 2024-09-19T08:09:24.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.563+0000 7f3dc29a8640 1 -- 172.21.15.98:0/1759635250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dbc1008d0 con 0x7f3dbc102d40 2024-09-19T08:09:24.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.565+0000 7f3db97fa640 1 -- 172.21.15.98:0/1759635250 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f3da4042030 con 0x7f3dbc102d40 2024-09-19T08:09:24.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.565+0000 7f3dc4d81740 1 Processor -- start 2024-09-19T08:09:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.565+0000 7f3dc4d81740 1 -- start start 2024-09-19T08:09:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dc4d81740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a3b40 0x55bfa53a2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dc4d81740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53a25d0 0x55bfa53a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dc4d81740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53ad370 0x55bfa53af760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dc4d81740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bfa53082f0 con 0x55bfa53ad370 2024-09-19T08:09:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dc4d81740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bfa52afcf0 con 0x55bfa53a3b40 2024-09-19T08:09:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dc4d81740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bfa53042b0 con 0x55bfa53a25d0 2024-09-19T08:09:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3db97fa640 1 --2- 172.21.15.98:0/1759635250 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3da80a33b0 0x7f3da80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dbb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53ad370 0x55bfa53af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dbb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53ad370 0x55bfa53af760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38262/0 (socket says 172.21.15.98:38262) 2024-09-19T08:09:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dbb7fe640 1 -- 172.21.15.98:0/2310395791 learned_addr learned my addr 172.21.15.98:0/2310395791 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dc0f45640 1 --2- 172.21.15.98:0/1759635250 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3da80a33b0 0x7f3da80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dbbfff640 1 --2- 172.21.15.98:0/2310395791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53a25d0 0x55bfa53a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dc0f45640 1 --2- 172.21.15.98:0/2310395791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a3b40 0x55bfa53a2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3db97fa640 1 -- 172.21.15.98:0/1759635250 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f3da40f0270 con 0x7f3dbc102d40 2024-09-19T08:09:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dbb7fe640 1 -- 172.21.15.98:0/2310395791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53a25d0 msgr2=0x55bfa53a2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dbb7fe640 1 --2- 172.21.15.98:0/2310395791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53a25d0 0x55bfa53a2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dbb7fe640 1 -- 172.21.15.98:0/2310395791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a3b40 msgr2=0x55bfa53a2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dbb7fe640 1 --2- 172.21.15.98:0/2310395791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a3b40 0x55bfa53a2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dbb7fe640 1 -- 172.21.15.98:0/2310395791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfa52943a0 con 0x55bfa53ad370 2024-09-19T08:09:24.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dc0f45640 1 --2- 172.21.15.98:0/1759635250 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3da80a33b0 0x7f3da80a5870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009fd0 tx=0x7f3db4009da0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:24.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3dbb7fe640 1 --2- 172.21.15.98:0/2310395791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53ad370 0x55bfa53af760 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f3dac00b730 tx=0x7f3dac00bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b62b011daa0374b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:24.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3db27fc640 1 -- 172.21.15.98:0/2310395791 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac003dd0 con 0x55bfa53ad370 2024-09-19T08:09:24.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3db27fc640 1 -- 172.21.15.98:0/2310395791 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dac015070 con 0x55bfa53ad370 2024-09-19T08:09:24.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.566+0000 7f3db27fc640 1 -- 172.21.15.98:0/2310395791 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac004e00 con 0x55bfa53ad370 2024-09-19T08:09:24.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.567+0000 7f3dc4d81740 1 -- 172.21.15.98:0/2310395791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53ad370 msgr2=0x55bfa53af760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:24.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.567+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/2310395791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53ad370 0x55bfa53af760 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f3dac00b730 tx=0x7f3dac00bbf0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.567+0000 7f3dc4d81740 1 -- 172.21.15.98:0/2310395791 shutdown_connections 2024-09-19T08:09:24.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.567+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/2310395791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53ad370 0x55bfa53af760 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.567+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/2310395791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53a25d0 0x55bfa53a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.567+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/2310395791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a3b40 0x55bfa53a2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.567+0000 7f3dc4d81740 1 -- 172.21.15.98:0/2310395791 >> 172.21.15.98:0/2310395791 conn(0x55bfa536b310 msgr2=0x55bfa539ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.567+0000 7f3dc4d81740 1 -- 172.21.15.98:0/2310395791 shutdown_connections 2024-09-19T08:09:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.567+0000 7f3dc4d81740 1 -- 172.21.15.98:0/2310395791 wait complete. 2024-09-19T08:09:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.567+0000 7f3dc4d81740 1 -- start start 2024-09-19T08:09:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc4d81740 1 Processor -- start 2024-09-19T08:09:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc4d81740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a25d0 0x55bfa53acd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc4d81740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53a3b40 0x55bfa53aa040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc4d81740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53ad370 0x55bfa53aa650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc4d81740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bfa53b0070 con 0x55bfa53ad370 2024-09-19T08:09:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc4d81740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bfa52c3060 con 0x55bfa53a25d0 2024-09-19T08:09:24.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc4d81740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bfa53b01f0 con 0x55bfa53a3b40 2024-09-19T08:09:24.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc4d81740 10 client.0 _ll_register_callbacks cb 0x55bfa50168c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:24.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc0f45640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53ad370 0x55bfa53aa650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc0f45640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53ad370 0x55bfa53aa650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38270/0 (socket says 172.21.15.98:38270) 2024-09-19T08:09:24.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc0f45640 1 -- 172.21.15.98:0/3270556870 learned_addr learned my addr 172.21.15.98:0/3270556870 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:24.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dbbfff640 1 --2- 172.21.15.98:0/3270556870 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a25d0 0x55bfa53acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dbb7fe640 1 --2- 172.21.15.98:0/3270556870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53a3b40 0x55bfa53aa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc0f45640 1 -- 172.21.15.98:0/3270556870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53a3b40 msgr2=0x55bfa53aa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:24.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc0f45640 1 --2- 172.21.15.98:0/3270556870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53a3b40 0x55bfa53aa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc0f45640 1 -- 172.21.15.98:0/3270556870 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a25d0 msgr2=0x55bfa53acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:24.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc0f45640 1 --2- 172.21.15.98:0/3270556870 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a25d0 0x55bfa53acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.568+0000 7f3dc0f45640 1 -- 172.21.15.98:0/3270556870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfa50793c0 con 0x55bfa53ad370 2024-09-19T08:09:24.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3dc0f45640 1 --2- 172.21.15.98:0/3270556870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53ad370 0x55bfa53aa650 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3db4003e40 tx=0x7f3db4009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a6a1f3dd0cdf6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:24.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3d9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:24.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3d9b7fe640 1 -- 172.21.15.98:0/3270556870 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db4012250 con 0x55bfa53ad370 2024-09-19T08:09:24.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3d9b7fe640 1 -- 172.21.15.98:0/3270556870 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f3db40123f0 con 0x55bfa53ad370 2024-09-19T08:09:24.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3270556870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dac015070 con 0x55bfa53ad370 2024-09-19T08:09:24.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3d9b7fe640 1 -- 172.21.15.98:0/3270556870 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3db40125f0 con 0x55bfa53ad370 2024-09-19T08:09:24.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3dc4d81740 10 client.17052 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:24.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3270556870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bfa53bbb40 con 0x55bfa53ad370 2024-09-19T08:09:24.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3d9b7fe640 1 -- 172.21.15.98:0/3270556870 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dac015070 con 0x55bfa53ad370 2024-09-19T08:09:24.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3d9b7fe640 1 -- 172.21.15.98:0/3270556870 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3db4029070 con 0x55bfa53ad370 2024-09-19T08:09:24.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3dc4d81740 10 client.17052 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:24.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3dc4d81740 10 client.17052 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:09:24.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3dc4d81740 20 client.17052 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:24.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/3270556870 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55bfa53bd000 0x55bfa53bf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3dc4d81740 4 client.17052 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:09:24.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.569+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3270556870 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfa5315860 con 0x55bfa53bd000 2024-09-19T08:09:24.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.570+0000 7f3dbbfff640 1 --2- 172.21.15.98:0/3270556870 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55bfa53bd000 0x55bfa53bf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.570+0000 7f3d9b7fe640 10 client.17052 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:24.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.570+0000 7f3dbbfff640 1 --2- 172.21.15.98:0/3270556870 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55bfa53bd000 0x55bfa53bf470 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8378ed43626030c4 server_cookie=82cd57a963392390 in_seq=0 out_seq=0 2024-09-19T08:09:24.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3d9b7fe640 1 -- 172.21.15.98:0/3270556870 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bfa5315860 con 0x55bfa53bd000 2024-09-19T08:09:24.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3d9b7fe640 10 client.17052 handle_command_reply: tid=0 2024-09-19T08:09:24.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3dc4d81740 1 client.17052 shutdown 2024-09-19T08:09:24.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3270556870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53ad370 msgr2=0x55bfa53aa650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:24.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/3270556870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53ad370 0x55bfa53aa650 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3db4003e40 tx=0x7f3db4009600 comp rx=0 tx=0).stop 2024-09-19T08:09:24.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3270556870 shutdown_connections 2024-09-19T08:09:24.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/3270556870 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55bfa53bd000 0x55bfa53bf470 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/3270556870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53ad370 0x55bfa53aa650 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/3270556870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53a3b40 0x55bfa53aa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/3270556870 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a25d0 0x55bfa53acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3d9b7fe640 0 client.17052 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:24.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3270556870 >> 172.21.15.98:0/3270556870 conn(0x55bfa536b310 msgr2=0x55bfa53a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:24.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3270556870 shutdown_connections 2024-09-19T08:09:24.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3270556870 wait complete. 2024-09-19T08:09:24.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.571+0000 7f3dc4d81740 20 client.17052 trim_cache size 0 max 16384 2024-09-19T08:09:24.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.584+0000 7f3dc4d81740 1 Processor -- start 2024-09-19T08:09:24.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.584+0000 7f3dc4d81740 1 -- start start 2024-09-19T08:09:24.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.584+0000 7f3dc4d81740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a25d0 0x55bfa53b9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.584+0000 7f3dc4d81740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53a3b40 0x55bfa53a5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.584+0000 7f3dc4d81740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53ad370 0x55bfa53a5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.584+0000 7f3dc4d81740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1ab550 con 0x55bfa53a3b40 2024-09-19T08:09:24.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.584+0000 7f3dc4d81740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1ad090 con 0x55bfa53a25d0 2024-09-19T08:09:24.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.584+0000 7f3dc4d81740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1abc20 con 0x55bfa53ad370 2024-09-19T08:09:24.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3dbb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53a3b40 0x55bfa53a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3dbb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53a3b40 0x55bfa53a5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38286/0 (socket says 172.21.15.98:38286) 2024-09-19T08:09:24.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3dbb7fe640 1 -- 172.21.15.98:0/385365023 learned_addr learned my addr 172.21.15.98:0/385365023 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:24.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3dc0f45640 1 --2- 172.21.15.98:0/385365023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53ad370 0x55bfa53a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3dbbfff640 1 --2- 172.21.15.98:0/385365023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a25d0 0x55bfa53b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3dbb7fe640 1 -- 172.21.15.98:0/385365023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53ad370 msgr2=0x55bfa53a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:24.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3dbb7fe640 1 --2- 172.21.15.98:0/385365023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53ad370 0x55bfa53a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3dbb7fe640 1 -- 172.21.15.98:0/385365023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a25d0 msgr2=0x55bfa53b9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:24.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3dbb7fe640 1 --2- 172.21.15.98:0/385365023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a25d0 0x55bfa53b9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3dbb7fe640 1 -- 172.21.15.98:0/385365023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfa539fdd0 con 0x55bfa53a3b40 2024-09-19T08:09:24.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3dc0f45640 1 --2- 172.21.15.98:0/385365023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53ad370 0x55bfa53a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:09:24.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3dbbfff640 1 --2- 172.21.15.98:0/385365023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a25d0 0x55bfa53b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:09:24.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3dbb7fe640 1 --2- 172.21.15.98:0/385365023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53a3b40 0x55bfa53a5120 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x55bfa53a3070 tx=0x7f3dac0042b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=973f002cf54f7ab1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:24.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3db1ffb640 1 -- 172.21.15.98:0/385365023 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac003bd0 con 0x55bfa53a3b40 2024-09-19T08:09:24.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3db1ffb640 1 -- 172.21.15.98:0/385365023 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dac028070 con 0x55bfa53a3b40 2024-09-19T08:09:24.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.585+0000 7f3db1ffb640 1 -- 172.21.15.98:0/385365023 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac019420 con 0x55bfa53a3b40 2024-09-19T08:09:24.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.586+0000 7f3dc4d81740 1 -- 172.21.15.98:0/385365023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53a3b40 msgr2=0x55bfa53a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:24.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.586+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/385365023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53a3b40 0x55bfa53a5120 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x55bfa53a3070 tx=0x7f3dac0042b0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.586+0000 7f3dc4d81740 1 -- 172.21.15.98:0/385365023 shutdown_connections 2024-09-19T08:09:24.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.586+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/385365023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53ad370 0x55bfa53a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.586+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/385365023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53a3b40 0x55bfa53a5120 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.586+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/385365023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a25d0 0x55bfa53b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.586+0000 7f3dc4d81740 1 -- 172.21.15.98:0/385365023 >> 172.21.15.98:0/385365023 conn(0x55bfa536b310 msgr2=0x55bfa53aa000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:24.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.586+0000 7f3dc4d81740 1 -- 172.21.15.98:0/385365023 shutdown_connections 2024-09-19T08:09:24.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.586+0000 7f3dc4d81740 1 -- 172.21.15.98:0/385365023 wait complete. 2024-09-19T08:09:24.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.586+0000 7f3dc4d81740 1 -- start start 2024-09-19T08:09:24.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dc4d81740 1 Processor -- start 2024-09-19T08:09:24.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dc4d81740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53a25d0 0x55bfa53a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dc4d81740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a3b40 0x55bfa53b5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dc4d81740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53ad370 0x55bfa53b59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dc4d81740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1ac9c0 con 0x55bfa53a25d0 2024-09-19T08:09:24.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dc4d81740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1ac2f0 con 0x55bfa53a3b40 2024-09-19T08:09:24.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dc4d81740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1a9530 con 0x55bfa53ad370 2024-09-19T08:09:24.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dc4d81740 10 client.0 _ll_register_callbacks cb 0x7f3db40128c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:24.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dbbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53a25d0 0x55bfa53a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dbbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53a25d0 0x55bfa53a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38294/0 (socket says 172.21.15.98:38294) 2024-09-19T08:09:24.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dbbfff640 1 -- 172.21.15.98:0/3888144062 learned_addr learned my addr 172.21.15.98:0/3888144062 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:24.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dbb7fe640 1 --2- 172.21.15.98:0/3888144062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a3b40 0x55bfa53b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dc0f45640 1 --2- 172.21.15.98:0/3888144062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53ad370 0x55bfa53b59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dbbfff640 1 -- 172.21.15.98:0/3888144062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53ad370 msgr2=0x55bfa53b59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:24.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dbbfff640 1 --2- 172.21.15.98:0/3888144062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53ad370 0x55bfa53b59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dbbfff640 1 -- 172.21.15.98:0/3888144062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a3b40 msgr2=0x55bfa53b5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:24.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dbbfff640 1 --2- 172.21.15.98:0/3888144062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a3b40 0x55bfa53b5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dbbfff640 1 -- 172.21.15.98:0/3888144062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfa53b60b0 con 0x55bfa53a25d0 2024-09-19T08:09:24.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dc0f45640 1 --2- 172.21.15.98:0/3888144062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53ad370 0x55bfa53b59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:09:24.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3dbbfff640 1 --2- 172.21.15.98:0/3888144062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53a25d0 0x55bfa53a7a70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f3da40b90c0 tx=0x7f3da408c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0a15c22a0cd0bd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:24.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.587+0000 7f3d9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:24.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3d9b7fe640 1 -- 172.21.15.98:0/3888144062 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da408b060 con 0x55bfa53a25d0 2024-09-19T08:09:24.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3d9b7fe640 1 -- 172.21.15.98:0/3888144062 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f3da408b200 con 0x55bfa53a25d0 2024-09-19T08:09:24.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3888144062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dac028070 con 0x55bfa53a25d0 2024-09-19T08:09:24.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3d9b7fe640 1 -- 172.21.15.98:0/3888144062 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3da4049040 con 0x55bfa53a25d0 2024-09-19T08:09:24.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3dc4d81740 10 client.17058 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:24.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3888144062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bfa5427e20 con 0x55bfa53a25d0 2024-09-19T08:09:24.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3d9b7fe640 1 -- 172.21.15.98:0/3888144062 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dac028070 con 0x55bfa53a25d0 2024-09-19T08:09:24.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3d9b7fe640 1 -- 172.21.15.98:0/3888144062 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3da40edc70 con 0x55bfa53a25d0 2024-09-19T08:09:24.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3dc4d81740 10 client.17058 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:24.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3dc4d81740 10 client.17058 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:09:24.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3dc4d81740 20 client.17058 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:24.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/3888144062 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55bfa54284a0 0x55bfa542a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:24.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3dc4d81740 4 client.17058 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:09:24.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3888144062 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bfa530fd70 con 0x55bfa54284a0 2024-09-19T08:09:24.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.588+0000 7f3dbb7fe640 1 --2- 172.21.15.98:0/3888144062 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55bfa54284a0 0x55bfa542a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:24.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3d9b7fe640 10 client.17058 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:24.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3dbb7fe640 1 --2- 172.21.15.98:0/3888144062 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55bfa54284a0 0x55bfa542a8c0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fa25462f61cd8c8 server_cookie=161c6b2eef1060eb in_seq=0 out_seq=0 2024-09-19T08:09:24.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3d9b7fe640 1 -- 172.21.15.98:0/3888144062 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bfa530fd70 con 0x55bfa54284a0 2024-09-19T08:09:24.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3d9b7fe640 10 client.17058 handle_command_reply: tid=0 2024-09-19T08:09:24.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3dc4d81740 1 client.17058 shutdown 2024-09-19T08:09:24.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3888144062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53a25d0 msgr2=0x55bfa53a7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:24.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/3888144062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53a25d0 0x55bfa53a7a70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f3da40b90c0 tx=0x7f3da408c3d0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3888144062 shutdown_connections 2024-09-19T08:09:24.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/3888144062 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55bfa54284a0 0x55bfa542a8c0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/3888144062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bfa53ad370 0x55bfa53b59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/3888144062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bfa53a3b40 0x55bfa53b5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3d9b7fe640 0 client.17058 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:24.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3dc4d81740 1 --2- 172.21.15.98:0/3888144062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfa53a25d0 0x55bfa53a7a70 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3888144062 >> 172.21.15.98:0/3888144062 conn(0x55bfa536b310 msgr2=0x55bfa53ab8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:24.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3888144062 shutdown_connections 2024-09-19T08:09:24.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3dc4d81740 1 -- 172.21.15.98:0/3888144062 wait complete. 2024-09-19T08:09:24.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.589+0000 7f3dc4d81740 20 client.17058 trim_cache size 0 max 16384 2024-09-19T08:09:24.601 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:09:24.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.590+0000 7f3dc29a8640 1 -- 172.21.15.98:0/1759635250 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3da80a33b0 msgr2=0x7f3da80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:24.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.590+0000 7f3dc29a8640 1 --2- 172.21.15.98:0/1759635250 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3da80a33b0 0x7f3da80a5870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009fd0 tx=0x7f3db4009da0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.590+0000 7f3dc29a8640 1 -- 172.21.15.98:0/1759635250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102d40 msgr2=0x7f3dbc198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:24.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.590+0000 7f3dc29a8640 1 --2- 172.21.15.98:0/1759635250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102d40 0x7f3dbc198430 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f3da4006010 tx=0x7f3da40042f0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.590+0000 7f3dc29a8640 1 -- 172.21.15.98:0/1759635250 shutdown_connections 2024-09-19T08:09:24.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.590+0000 7f3dc29a8640 1 --2- 172.21.15.98:0/1759635250 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3da80a33b0 0x7f3da80a5870 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.590+0000 7f3dc29a8640 1 --2- 172.21.15.98:0/1759635250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dbc10c170 0x7f3dbc19cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.590+0000 7f3dc29a8640 1 --2- 172.21.15.98:0/1759635250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3dbc103660 0x7f3dbc198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.590+0000 7f3dc29a8640 1 --2- 172.21.15.98:0/1759635250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dbc102d40 0x7f3dbc198430 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:24.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.590+0000 7f3dc29a8640 1 -- 172.21.15.98:0/1759635250 >> 172.21.15.98:0/1759635250 conn(0x7f3dbc0fc7a0 msgr2=0x7f3dbc0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:24.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.590+0000 7f3dc29a8640 1 -- 172.21.15.98:0/1759635250 shutdown_connections 2024-09-19T08:09:24.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:24.590+0000 7f3dc29a8640 1 -- 172.21.15.98:0/1759635250 wait complete. 2024-09-19T08:09:26.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:26 smithi162 ceph-mon[35260]: pgmap v195: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:26.421 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:09:26.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:09:26.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:26 smithi139 ceph-mon[34874]: pgmap v195: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:26.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:26 smithi098 ceph-mon[31333]: pgmap v195: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:26.677 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:09:27.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.133+0000 7fee6e064640 1 -- 172.21.15.98:0/2229153868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee68106a80 msgr2=0x7fee68106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:27.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.133+0000 7fee6e064640 1 --2- 172.21.15.98:0/2229153868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee68106a80 0x7fee68106e60 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fee58009930 tx=0x7fee5802a420 comp rx=0 tx=0).stop 2024-09-19T08:09:27.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.133+0000 7fee6e064640 1 -- 172.21.15.98:0/2229153868 shutdown_connections 2024-09-19T08:09:27.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.133+0000 7fee6e064640 1 --2- 172.21.15.98:0/2229153868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee68101420 0x7fee6810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.133+0000 7fee6e064640 1 --2- 172.21.15.98:0/2229153868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee68100a80 0x7fee68100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.133+0000 7fee6e064640 1 --2- 172.21.15.98:0/2229153868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee68106a80 0x7fee68106e60 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.133+0000 7fee6e064640 1 -- 172.21.15.98:0/2229153868 >> 172.21.15.98:0/2229153868 conn(0x7fee680fc7a0 msgr2=0x7fee680febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:27.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.133+0000 7fee6e064640 1 -- 172.21.15.98:0/2229153868 shutdown_connections 2024-09-19T08:09:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.133+0000 7fee6e064640 1 -- 172.21.15.98:0/2229153868 wait complete. 2024-09-19T08:09:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.133+0000 7fee6e064640 1 Processor -- start 2024-09-19T08:09:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.133+0000 7fee6e064640 1 -- start start 2024-09-19T08:09:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee6e064640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee68100a80 0x7fee68198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee6e064640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee68101420 0x7fee681989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee6e064640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee68106a80 0x7fee6819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee6e064640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fee68111fa0 con 0x7fee68101420 2024-09-19T08:09:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee6e064640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fee68111e20 con 0x7fee68100a80 2024-09-19T08:09:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee6e064640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fee68112120 con 0x7fee68106a80 2024-09-19T08:09:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee66ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee68101420 0x7fee681989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee66ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee68101420 0x7fee681989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46538/0 (socket says 172.21.15.98:46538) 2024-09-19T08:09:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee66ffd640 1 -- 172.21.15.98:0/3917886146 learned_addr learned my addr 172.21.15.98:0/3917886146 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:27.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee67fff640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee68106a80 0x7fee6819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee677fe640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee68100a80 0x7fee68198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee66ffd640 1 -- 172.21.15.98:0/3917886146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee68106a80 msgr2=0x7fee6819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee66ffd640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee68106a80 0x7fee6819cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee66ffd640 1 -- 172.21.15.98:0/3917886146 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee68100a80 msgr2=0x7fee68198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee66ffd640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee68100a80 0x7fee68198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee66ffd640 1 -- 172.21.15.98:0/3917886146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6819d430 con 0x7fee68101420 2024-09-19T08:09:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee67fff640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee68106a80 0x7fee6819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:09:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee66ffd640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee68101420 0x7fee681989c0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fee5c00c890 tx=0x7fee5c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee64ff9640 1 -- 172.21.15.98:0/3917886146 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c004220 con 0x7fee68101420 2024-09-19T08:09:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.134+0000 7fee6e064640 1 -- 172.21.15.98:0/3917886146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee6819d720 con 0x7fee68101420 2024-09-19T08:09:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.135+0000 7fee64ff9640 1 -- 172.21.15.98:0/3917886146 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee5c0043c0 con 0x7fee68101420 2024-09-19T08:09:27.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.135+0000 7fee64ff9640 1 -- 172.21.15.98:0/3917886146 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c004a00 con 0x7fee68101420 2024-09-19T08:09:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.135+0000 7fee6e064640 1 -- 172.21.15.98:0/3917886146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee681a5000 con 0x7fee68101420 2024-09-19T08:09:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.137+0000 7fee7043d740 1 Processor -- start 2024-09-19T08:09:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.137+0000 7fee7043d740 1 -- start start 2024-09-19T08:09:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee64ff9640 1 -- 172.21.15.98:0/3917886146 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fee5c0079b0 con 0x7fee68101420 2024-09-19T08:09:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee7043d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 0x55d17828df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee7043d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 0x55d178298800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee7043d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d17829b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee7043d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d1781f3280 con 0x55d178298d40 2024-09-19T08:09:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee7043d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d17819ab10 con 0x55d17828db60 2024-09-19T08:09:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee7043d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d1781ef240 con 0x55d17828e480 2024-09-19T08:09:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee66ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d17829b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee66ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d17829b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46554/0 (socket says 172.21.15.98:46554) 2024-09-19T08:09:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee66ffd640 1 -- 172.21.15.98:0/3263796746 learned_addr learned my addr 172.21.15.98:0/3263796746 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee67fff640 1 --2- 172.21.15.98:0/3263796746 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 0x55d17828df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee677fe640 1 --2- 172.21.15.98:0/3263796746 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 0x55d178298800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee66ffd640 1 -- 172.21.15.98:0/3263796746 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 msgr2=0x55d178298800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee66ffd640 1 --2- 172.21.15.98:0/3263796746 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 0x55d178298800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee66ffd640 1 -- 172.21.15.98:0/3263796746 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 msgr2=0x55d17828df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee66ffd640 1 --2- 172.21.15.98:0/3263796746 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 0x55d17828df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee66ffd640 1 -- 172.21.15.98:0/3263796746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1781a71f0 con 0x55d178298d40 2024-09-19T08:09:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee64ff9640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fee500a33b0 0x7fee500a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee66ffd640 1 --2- 172.21.15.98:0/3263796746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d17829b130 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fee5c0cd190 tx=0x7fee5c0d85e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b4a417ec2fd047c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.138+0000 7fee67fff640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fee500a33b0 0x7fee500a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.139+0000 7fee64ff9640 1 -- 172.21.15.98:0/3917886146 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fee5c0cfaa0 con 0x7fee68101420 2024-09-19T08:09:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.139+0000 7fee4dffb640 1 -- 172.21.15.98:0/3263796746 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c01d070 con 0x55d178298d40 2024-09-19T08:09:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.139+0000 7fee4dffb640 1 -- 172.21.15.98:0/3263796746 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee5c09c4b0 con 0x55d178298d40 2024-09-19T08:09:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.139+0000 7fee4dffb640 1 -- 172.21.15.98:0/3263796746 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c0d5050 con 0x55d178298d40 2024-09-19T08:09:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.139+0000 7fee7043d740 1 -- 172.21.15.98:0/3263796746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 msgr2=0x55d17829b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.139+0000 7fee7043d740 1 --2- 172.21.15.98:0/3263796746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d17829b130 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fee5c0cd190 tx=0x7fee5c0d85e0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.139+0000 7fee67fff640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fee500a33b0 0x7fee500a5870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fee6000c610 tx=0x7fee6000e040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.139+0000 7fee7043d740 1 -- 172.21.15.98:0/3263796746 shutdown_connections 2024-09-19T08:09:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.139+0000 7fee7043d740 1 --2- 172.21.15.98:0/3263796746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d17829b130 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.139+0000 7fee7043d740 1 --2- 172.21.15.98:0/3263796746 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 0x55d178298800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.139+0000 7fee7043d740 1 --2- 172.21.15.98:0/3263796746 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 0x55d17828df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.140+0000 7fee7043d740 1 -- 172.21.15.98:0/3263796746 >> 172.21.15.98:0/3263796746 conn(0x55d17828a0f0 msgr2=0x55d17828a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.140+0000 7fee7043d740 1 -- 172.21.15.98:0/3263796746 shutdown_connections 2024-09-19T08:09:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.140+0000 7fee7043d740 1 -- 172.21.15.98:0/3263796746 wait complete. 2024-09-19T08:09:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.140+0000 7fee7043d740 1 -- start start 2024-09-19T08:09:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.140+0000 7fee7043d740 1 Processor -- start 2024-09-19T08:09:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.140+0000 7fee7043d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 0x55d17829d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.140+0000 7fee7043d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 0x55d17829d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.140+0000 7fee7043d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d1782a1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.140+0000 7fee7043d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d17829b910 con 0x55d178298d40 2024-09-19T08:09:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.140+0000 7fee7043d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d1781adf10 con 0x55d17828db60 2024-09-19T08:09:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.140+0000 7fee7043d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d17829ba90 con 0x55d17828e480 2024-09-19T08:09:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.140+0000 7fee7043d740 10 client.0 _ll_register_callbacks cb 0x55d178176a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.140+0000 7fee67fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d1782a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee67fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d1782a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46562/0 (socket says 172.21.15.98:46562) 2024-09-19T08:09:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee67fff640 1 -- 172.21.15.98:0/1612214488 learned_addr learned my addr 172.21.15.98:0/1612214488 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee677fe640 1 --2- 172.21.15.98:0/1612214488 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 0x55d17829d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee66ffd640 1 --2- 172.21.15.98:0/1612214488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 0x55d17829d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee67fff640 1 -- 172.21.15.98:0/1612214488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 msgr2=0x55d17829d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee67fff640 1 --2- 172.21.15.98:0/1612214488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 0x55d17829d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee67fff640 1 -- 172.21.15.98:0/1612214488 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 msgr2=0x55d17829d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee67fff640 1 --2- 172.21.15.98:0/1612214488 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 0x55d17829d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee67fff640 1 -- 172.21.15.98:0/1612214488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d177f646a0 con 0x55d178298d40 2024-09-19T08:09:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee66ffd640 1 --2- 172.21.15.98:0/1612214488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 0x55d17829d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:09:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee67fff640 1 --2- 172.21.15.98:0/1612214488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d1782a1780 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fee60007da0 tx=0x7fee60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b81244b08e70f685 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee46ffd640 1 -- 172.21.15.98:0/1612214488 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6001e040 con 0x55d178298d40 2024-09-19T08:09:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee7043d740 1 -- 172.21.15.98:0/1612214488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee5c09c4b0 con 0x55d178298d40 2024-09-19T08:09:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee46ffd640 1 -- 172.21.15.98:0/1612214488 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fee60022420 con 0x55d178298d40 2024-09-19T08:09:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee46ffd640 1 -- 172.21.15.98:0/1612214488 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fee60022620 con 0x55d178298d40 2024-09-19T08:09:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee7043d740 10 client.17070 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee7043d740 1 -- 172.21.15.98:0/1612214488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d1782ab930 con 0x55d178298d40 2024-09-19T08:09:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.141+0000 7fee46ffd640 1 -- 172.21.15.98:0/1612214488 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee5c09c4b0 con 0x55d178298d40 2024-09-19T08:09:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.142+0000 7fee46ffd640 1 -- 172.21.15.98:0/1612214488 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fee6002c920 con 0x55d178298d40 2024-09-19T08:09:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.142+0000 7fee7043d740 10 client.17070 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.142+0000 7fee7043d740 10 client.17070 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:09:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.142+0000 7fee7043d740 20 client.17070 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.142+0000 7fee7043d740 1 --2- 172.21.15.98:0/1612214488 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d1782acd90 0x55d1782af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.142+0000 7fee7043d740 4 client.17070 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:09:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.142+0000 7fee7043d740 1 -- 172.21.15.98:0/1612214488 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1782007f0 con 0x55d1782acd90 2024-09-19T08:09:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.142+0000 7fee677fe640 1 --2- 172.21.15.98:0/1612214488 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d1782acd90 0x55d1782af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.142+0000 7fee46ffd640 10 client.17070 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.142+0000 7fee677fe640 1 --2- 172.21.15.98:0/1612214488 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d1782acd90 0x55d1782af1c0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2331f974bf1aa8e server_cookie=55c450213fc1acc2 in_seq=0 out_seq=0 2024-09-19T08:09:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.143+0000 7fee46ffd640 1 -- 172.21.15.98:0/1612214488 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d1782007f0 con 0x55d1782acd90 2024-09-19T08:09:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.143+0000 7fee46ffd640 10 client.17070 handle_command_reply: tid=0 2024-09-19T08:09:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.143+0000 7fee7043d740 1 client.17070 shutdown 2024-09-19T08:09:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.143+0000 7fee7043d740 1 -- 172.21.15.98:0/1612214488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 msgr2=0x55d1782a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.143+0000 7fee7043d740 1 --2- 172.21.15.98:0/1612214488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d1782a1780 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fee60007da0 tx=0x7fee60004a40 comp rx=0 tx=0).stop 2024-09-19T08:09:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.143+0000 7fee7043d740 1 -- 172.21.15.98:0/1612214488 shutdown_connections 2024-09-19T08:09:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.143+0000 7fee7043d740 1 --2- 172.21.15.98:0/1612214488 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d1782acd90 0x55d1782af1c0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.143+0000 7fee7043d740 1 --2- 172.21.15.98:0/1612214488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d1782a1780 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.143+0000 7fee7043d740 1 --2- 172.21.15.98:0/1612214488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 0x55d17829d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.143+0000 7fee46ffd640 0 client.17070 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.143+0000 7fee7043d740 1 --2- 172.21.15.98:0/1612214488 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 0x55d17829d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.143+0000 7fee7043d740 1 -- 172.21.15.98:0/1612214488 >> 172.21.15.98:0/1612214488 conn(0x55d17828a0f0 msgr2=0x55d17828fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:27.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.143+0000 7fee7043d740 1 -- 172.21.15.98:0/1612214488 shutdown_connections 2024-09-19T08:09:27.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.143+0000 7fee7043d740 1 -- 172.21.15.98:0/1612214488 wait complete. 2024-09-19T08:09:27.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.144+0000 7fee7043d740 20 client.17070 trim_cache size 0 max 16384 2024-09-19T08:09:27.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.161+0000 7fee7043d740 1 Processor -- start 2024-09-19T08:09:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.161+0000 7fee7043d740 1 -- start start 2024-09-19T08:09:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.161+0000 7fee7043d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828db60 0x55d1782a1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.161+0000 7fee7043d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d17828e480 0x55d1782b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee7043d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d178298d40 0x55d1782b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee7043d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fee681ab690 con 0x55d17828e480 2024-09-19T08:09:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee7043d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fee681ad1d0 con 0x55d178298d40 2024-09-19T08:09:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee7043d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fee681abd60 con 0x55d17828db60 2024-09-19T08:09:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee66ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d17828e480 0x55d1782b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee66ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d17828e480 0x55d1782b4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46572/0 (socket says 172.21.15.98:46572) 2024-09-19T08:09:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee66ffd640 1 -- 172.21.15.98:0/2879951638 learned_addr learned my addr 172.21.15.98:0/2879951638 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee67fff640 1 --2- 172.21.15.98:0/2879951638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d178298d40 0x55d1782b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee677fe640 1 --2- 172.21.15.98:0/2879951638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828db60 0x55d1782a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee66ffd640 1 -- 172.21.15.98:0/2879951638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828db60 msgr2=0x55d1782a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee66ffd640 1 --2- 172.21.15.98:0/2879951638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828db60 0x55d1782a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee66ffd640 1 -- 172.21.15.98:0/2879951638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d178298d40 msgr2=0x55d1782b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee66ffd640 1 --2- 172.21.15.98:0/2879951638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d178298d40 0x55d1782b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee66ffd640 1 -- 172.21.15.98:0/2879951638 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d178289f20 con 0x55d17828e480 2024-09-19T08:09:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee66ffd640 1 --2- 172.21.15.98:0/2879951638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d17828e480 0x55d1782b4a90 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fee5c09bab0 tx=0x7fee5c0054a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed57956c6ccb7c74 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee4d7fa640 1 -- 172.21.15.98:0/2879951638 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c01d040 con 0x55d17828e480 2024-09-19T08:09:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee4d7fa640 1 -- 172.21.15.98:0/2879951638 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee5c09c4b0 con 0x55d17828e480 2024-09-19T08:09:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.162+0000 7fee4d7fa640 1 -- 172.21.15.98:0/2879951638 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c09b710 con 0x55d17828e480 2024-09-19T08:09:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.163+0000 7fee7043d740 1 -- 172.21.15.98:0/2879951638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d17828e480 msgr2=0x55d1782b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.163+0000 7fee7043d740 1 --2- 172.21.15.98:0/2879951638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d17828e480 0x55d1782b4a90 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fee5c09bab0 tx=0x7fee5c0054a0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.163+0000 7fee7043d740 1 -- 172.21.15.98:0/2879951638 shutdown_connections 2024-09-19T08:09:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.163+0000 7fee7043d740 1 --2- 172.21.15.98:0/2879951638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d178298d40 0x55d1782b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.163+0000 7fee7043d740 1 --2- 172.21.15.98:0/2879951638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d17828e480 0x55d1782b4a90 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.163+0000 7fee7043d740 1 --2- 172.21.15.98:0/2879951638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828db60 0x55d1782a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.163+0000 7fee7043d740 1 -- 172.21.15.98:0/2879951638 >> 172.21.15.98:0/2879951638 conn(0x55d17828a0f0 msgr2=0x55d1782952f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.163+0000 7fee7043d740 1 -- 172.21.15.98:0/2879951638 shutdown_connections 2024-09-19T08:09:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.163+0000 7fee7043d740 1 -- 172.21.15.98:0/2879951638 wait complete. 2024-09-19T08:09:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.163+0000 7fee7043d740 1 -- start start 2024-09-19T08:09:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee7043d740 1 Processor -- start 2024-09-19T08:09:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee7043d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 0x55d1782a8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee7043d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 0x55d1782a8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee7043d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d1782a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee7043d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fee681acb00 con 0x55d178298d40 2024-09-19T08:09:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee7043d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fee681ac430 con 0x55d17828db60 2024-09-19T08:09:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee7043d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fee681a9690 con 0x55d17828e480 2024-09-19T08:09:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee7043d740 10 client.0 _ll_register_callbacks cb 0x7fee6001b040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee67fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d1782a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee67fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d1782a59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46584/0 (socket says 172.21.15.98:46584) 2024-09-19T08:09:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee67fff640 1 -- 172.21.15.98:0/2298425412 learned_addr learned my addr 172.21.15.98:0/2298425412 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee677fe640 1 --2- 172.21.15.98:0/2298425412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 0x55d1782a8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee66ffd640 1 --2- 172.21.15.98:0/2298425412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 0x55d1782a8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee67fff640 1 -- 172.21.15.98:0/2298425412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 msgr2=0x55d1782a8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee67fff640 1 --2- 172.21.15.98:0/2298425412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 0x55d1782a8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee67fff640 1 -- 172.21.15.98:0/2298425412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 msgr2=0x55d1782a8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee67fff640 1 --2- 172.21.15.98:0/2298425412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 0x55d1782a8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee67fff640 1 -- 172.21.15.98:0/2298425412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d178294de0 con 0x55d178298d40 2024-09-19T08:09:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.164+0000 7fee66ffd640 1 --2- 172.21.15.98:0/2298425412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 0x55d1782a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:09:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee67fff640 1 --2- 172.21.15.98:0/2298425412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d1782a59f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fee60007d70 tx=0x7fee60018d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf6a8cdcdae93e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee46ffd640 1 -- 172.21.15.98:0/2298425412 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6003b070 con 0x55d178298d40 2024-09-19T08:09:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee46ffd640 1 -- 172.21.15.98:0/2298425412 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fee600227c0 con 0x55d178298d40 2024-09-19T08:09:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee7043d740 1 -- 172.21.15.98:0/2298425412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee5c09c4b0 con 0x55d178298d40 2024-09-19T08:09:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee46ffd640 1 -- 172.21.15.98:0/2298425412 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fee60047050 con 0x55d178298d40 2024-09-19T08:09:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee7043d740 10 client.17076 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee7043d740 1 -- 172.21.15.98:0/2298425412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d1783181d0 con 0x55d178298d40 2024-09-19T08:09:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee46ffd640 1 -- 172.21.15.98:0/2298425412 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee5c09c4b0 con 0x55d178298d40 2024-09-19T08:09:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee46ffd640 1 -- 172.21.15.98:0/2298425412 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fee600216a0 con 0x55d178298d40 2024-09-19T08:09:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee7043d740 10 client.17076 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee7043d740 10 client.17076 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:09:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee7043d740 20 client.17076 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee7043d740 1 --2- 172.21.15.98:0/2298425412 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d178318950 0x55d17831ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee7043d740 4 client.17076 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:09:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.165+0000 7fee7043d740 1 -- 172.21.15.98:0/2298425412 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d1781fad00 con 0x55d178318950 2024-09-19T08:09:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.166+0000 7fee677fe640 1 --2- 172.21.15.98:0/2298425412 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d178318950 0x55d17831ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.166+0000 7fee46ffd640 10 client.17076 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.166+0000 7fee677fe640 1 --2- 172.21.15.98:0/2298425412 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d178318950 0x55d17831ad70 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3d974a8239065d2 server_cookie=95772d31b0960b50 in_seq=0 out_seq=0 2024-09-19T08:09:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.167+0000 7fee46ffd640 1 -- 172.21.15.98:0/2298425412 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d1781fad00 con 0x55d178318950 2024-09-19T08:09:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.167+0000 7fee46ffd640 10 client.17076 handle_command_reply: tid=0 2024-09-19T08:09:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.167+0000 7fee7043d740 1 client.17076 shutdown 2024-09-19T08:09:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.167+0000 7fee7043d740 1 -- 172.21.15.98:0/2298425412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 msgr2=0x55d1782a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.167+0000 7fee7043d740 1 --2- 172.21.15.98:0/2298425412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d1782a59f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fee60007d70 tx=0x7fee60018d40 comp rx=0 tx=0).stop 2024-09-19T08:09:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.167+0000 7fee7043d740 1 -- 172.21.15.98:0/2298425412 shutdown_connections 2024-09-19T08:09:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.167+0000 7fee7043d740 1 --2- 172.21.15.98:0/2298425412 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d178318950 0x55d17831ad70 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.167+0000 7fee7043d740 1 --2- 172.21.15.98:0/2298425412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d178298d40 0x55d1782a59f0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.167+0000 7fee7043d740 1 --2- 172.21.15.98:0/2298425412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d17828e480 0x55d1782a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.167+0000 7fee7043d740 1 --2- 172.21.15.98:0/2298425412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d17828db60 0x55d1782a8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.167+0000 7fee7043d740 1 -- 172.21.15.98:0/2298425412 >> 172.21.15.98:0/2298425412 conn(0x55d17828a0f0 msgr2=0x55d17828f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.168+0000 7fee46ffd640 0 client.17076 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.168+0000 7fee7043d740 1 -- 172.21.15.98:0/2298425412 shutdown_connections 2024-09-19T08:09:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.168+0000 7fee7043d740 1 -- 172.21.15.98:0/2298425412 wait complete. 2024-09-19T08:09:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.168+0000 7fee7043d740 20 client.17076 trim_cache size 0 max 16384 2024-09-19T08:09:27.178 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:09:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:09:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "a2ef10f3-53d8-4d3d-a763-e62742b8063e", 2024-09-19T08:09:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:09:27.179 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:09:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.168+0000 7fee6e064640 1 -- 172.21.15.98:0/3917886146 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fee500a33b0 msgr2=0x7fee500a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.168+0000 7fee6e064640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fee500a33b0 0x7fee500a5870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fee6000c610 tx=0x7fee6000e040 comp rx=0 tx=0).stop 2024-09-19T08:09:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.168+0000 7fee6e064640 1 -- 172.21.15.98:0/3917886146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee68101420 msgr2=0x7fee681989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.168+0000 7fee6e064640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee68101420 0x7fee681989c0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fee5c00c890 tx=0x7fee5c00cd50 comp rx=0 tx=0).stop 2024-09-19T08:09:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.168+0000 7fee6e064640 1 -- 172.21.15.98:0/3917886146 shutdown_connections 2024-09-19T08:09:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.168+0000 7fee6e064640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fee500a33b0 0x7fee500a5870 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.168+0000 7fee6e064640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee68106a80 0x7fee6819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.168+0000 7fee6e064640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee68101420 0x7fee681989c0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.168+0000 7fee6e064640 1 --2- 172.21.15.98:0/3917886146 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee68100a80 0x7fee68198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.168+0000 7fee6e064640 1 -- 172.21.15.98:0/3917886146 >> 172.21.15.98:0/3917886146 conn(0x7fee680fc7a0 msgr2=0x7fee680fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.169+0000 7fee6e064640 1 -- 172.21.15.98:0/3917886146 shutdown_connections 2024-09-19T08:09:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:27.169+0000 7fee6e064640 1 -- 172.21.15.98:0/3917886146 wait complete. 2024-09-19T08:09:28.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:09:28.304 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:09:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:28 smithi162 ceph-mon[35260]: pgmap v196: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:09:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:28 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:09:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:28 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:09:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:28 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:09:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:28 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:09:28.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:28 smithi098 ceph-mon[31333]: pgmap v196: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:09:28.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:28 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:09:28.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:28 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:09:28.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:28 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:09:28.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:28 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:09:28.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:28 smithi139 ceph-mon[34874]: pgmap v196: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:09:28.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:28 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:09:28.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:28 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:09:28.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:28 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:09:28.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:28 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:09:28.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.755+0000 7fe6ab5ef640 1 -- 172.21.15.98:0/173373753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a4112ba0 msgr2=0x7fe6a4114f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:28.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.755+0000 7fe6ab5ef640 1 --2- 172.21.15.98:0/173373753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a4112ba0 0x7fe6a4114f90 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009ca0 tx=0x7fe69c02a950 comp rx=0 tx=0).stop 2024-09-19T08:09:28.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.755+0000 7fe6ab5ef640 1 -- 172.21.15.98:0/173373753 shutdown_connections 2024-09-19T08:09:28.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.755+0000 7fe6ab5ef640 1 --2- 172.21.15.98:0/173373753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a4112ba0 0x7fe6a4114f90 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.755+0000 7fe6ab5ef640 1 --2- 172.21.15.98:0/173373753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6a4074eb0 0x7fe6a4112660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.755+0000 7fe6ab5ef640 1 --2- 172.21.15.98:0/173373753 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6a4074590 0x7fe6a4074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.755+0000 7fe6ab5ef640 1 -- 172.21.15.98:0/173373753 >> 172.21.15.98:0/173373753 conn(0x7fe6a40fd8f0 msgr2=0x7fe6a40ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:28.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.755+0000 7fe6ab5ef640 1 -- 172.21.15.98:0/173373753 shutdown_connections 2024-09-19T08:09:28.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.755+0000 7fe6ab5ef640 1 -- 172.21.15.98:0/173373753 wait complete. 2024-09-19T08:09:28.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.755+0000 7fe6ab5ef640 1 Processor -- start 2024-09-19T08:09:28.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.755+0000 7fe6ab5ef640 1 -- start start 2024-09-19T08:09:28.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.755+0000 7fe6ab5ef640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a4074590 0x7fe6a41a4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.755+0000 7fe6ab5ef640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6a4074eb0 0x7fe6a41a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6ab5ef640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6a4112ba0 0x7fe6a419e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6ab5ef640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a41185f0 con 0x7fe6a4074590 2024-09-19T08:09:28.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6ab5ef640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a4118470 con 0x7fe6a4112ba0 2024-09-19T08:09:28.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6ab5ef640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a4118770 con 0x7fe6a4074eb0 2024-09-19T08:09:28.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6a938b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a4074590 0x7fe6a41a4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6a938b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a4074590 0x7fe6a41a4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46616/0 (socket says 172.21.15.98:46616) 2024-09-19T08:09:28.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6a938b640 1 -- 172.21.15.98:0/1917321824 learned_addr learned my addr 172.21.15.98:0/1917321824 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:28.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6a8b8a640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6a4074eb0 0x7fe6a41a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6a9b8c640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6a4112ba0 0x7fe6a419e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6a938b640 1 -- 172.21.15.98:0/1917321824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6a4074eb0 msgr2=0x7fe6a41a4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:28.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6a938b640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6a4074eb0 0x7fe6a41a4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6a938b640 1 -- 172.21.15.98:0/1917321824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6a4112ba0 msgr2=0x7fe6a419e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:28.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6a938b640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6a4112ba0 0x7fe6a419e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6a938b640 1 -- 172.21.15.98:0/1917321824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a419f030 con 0x7fe6a4074590 2024-09-19T08:09:28.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6a8b8a640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6a4074eb0 0x7fe6a41a4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:09:28.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6a938b640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a4074590 0x7fe6a41a4740 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000b4a0 tx=0x7fe6a000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:28.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6967fc640 1 -- 172.21.15.98:0/1917321824 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0014070 con 0x7fe6a4074590 2024-09-19T08:09:28.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6ab5ef640 1 -- 172.21.15.98:0/1917321824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6a419f320 con 0x7fe6a4074590 2024-09-19T08:09:28.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6967fc640 1 -- 172.21.15.98:0/1917321824 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6a0010af0 con 0x7fe6a4074590 2024-09-19T08:09:28.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6967fc640 1 -- 172.21.15.98:0/1917321824 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0017d10 con 0x7fe6a4074590 2024-09-19T08:09:28.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.756+0000 7fe6ab5ef640 1 -- 172.21.15.98:0/1917321824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6a41ab610 con 0x7fe6a4074590 2024-09-19T08:09:28.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.759+0000 7fe6967fc640 1 -- 172.21.15.98:0/1917321824 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fe6a0017070 con 0x7fe6a4074590 2024-09-19T08:09:28.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.759+0000 7fe6ad9c8740 1 Processor -- start 2024-09-19T08:09:28.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.759+0000 7fe6ad9c8740 1 -- start start 2024-09-19T08:09:28.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.759+0000 7fe6ad9c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x559134080880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.759+0000 7fe6ad9c8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559134080dc0 0x55913408b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.759+0000 7fe6ad9c8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55913408b680 0x55913408da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.759+0000 7fe6ad9c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559133fe62e0 con 0x5591340804a0 2024-09-19T08:09:28.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.759+0000 7fe6ad9c8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559133f8dce0 con 0x559134080dc0 2024-09-19T08:09:28.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.759+0000 7fe6ad9c8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559133fe22a0 con 0x55913408b680 2024-09-19T08:09:28.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6a9b8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x559134080880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6a9b8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x559134080880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46628/0 (socket says 172.21.15.98:46628) 2024-09-19T08:09:28.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6a9b8c640 1 -- 172.21.15.98:0/357802726 learned_addr learned my addr 172.21.15.98:0/357802726 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:28.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6967fc640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe6900a33b0 0x7fe6900a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6a8b8a640 1 --2- 172.21.15.98:0/357802726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55913408b680 0x55913408da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6a938b640 1 --2- 172.21.15.98:0/357802726 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559134080dc0 0x55913408b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6a9b8c640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe6900a33b0 0x7fe6900a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6a9b8c640 1 -- 172.21.15.98:0/357802726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55913408b680 msgr2=0x55913408da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:28.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6a9b8c640 1 --2- 172.21.15.98:0/357802726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55913408b680 0x55913408da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6a9b8c640 1 -- 172.21.15.98:0/357802726 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559134080dc0 msgr2=0x55913408b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:28.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6a9b8c640 1 --2- 172.21.15.98:0/357802726 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559134080dc0 0x55913408b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6a9b8c640 1 -- 172.21.15.98:0/357802726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559133f72440 con 0x5591340804a0 2024-09-19T08:09:28.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6a938b640 1 --2- 172.21.15.98:0/357802726 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559134080dc0 0x55913408b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:09:28.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6967fc640 1 -- 172.21.15.98:0/1917321824 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fe6a00cf060 con 0x7fe6a4074590 2024-09-19T08:09:28.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe6a9b8c640 1 --2- 172.21.15.98:0/357802726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x559134080880 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fe69c035000 tx=0x7fe69c02fd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe67b2209a56392 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:28.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe68f7fe640 1 -- 172.21.15.98:0/357802726 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c034040 con 0x5591340804a0 2024-09-19T08:09:28.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe68f7fe640 1 -- 172.21.15.98:0/357802726 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe69c04e070 con 0x5591340804a0 2024-09-19T08:09:28.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.760+0000 7fe68f7fe640 1 -- 172.21.15.98:0/357802726 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c049750 con 0x5591340804a0 2024-09-19T08:09:28.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.761+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/357802726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 msgr2=0x559134080880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:28.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.761+0000 7fe6a9b8c640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe6900a33b0 0x7fe6900a5870 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe69c04d000 tx=0x7fe69c04b000 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:28.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.761+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/357802726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x559134080880 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fe69c035000 tx=0x7fe69c02fd50 comp rx=0 tx=0).stop 2024-09-19T08:09:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.761+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/357802726 shutdown_connections 2024-09-19T08:09:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.761+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/357802726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55913408b680 0x55913408da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.761+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/357802726 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559134080dc0 0x55913408b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.761+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/357802726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x559134080880 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.761+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/357802726 >> 172.21.15.98:0/357802726 conn(0x55913407ca30 msgr2=0x55913407ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.761+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/357802726 shutdown_connections 2024-09-19T08:09:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.761+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/357802726 wait complete. 2024-09-19T08:09:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.761+0000 7fe6ad9c8740 1 -- start start 2024-09-19T08:09:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6ad9c8740 1 Processor -- start 2024-09-19T08:09:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6ad9c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x55913408afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6ad9c8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559134080dc0 0x559134088280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6ad9c8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55913408b680 0x5591340887e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6ad9c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55913408e250 con 0x5591340804a0 2024-09-19T08:09:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6ad9c8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559133fa1050 con 0x55913408b680 2024-09-19T08:09:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6ad9c8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55913408e3d0 con 0x559134080dc0 2024-09-19T08:09:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6ad9c8740 10 client.0 _ll_register_callbacks cb 0x559133fe1160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6a938b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x55913408afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6a938b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x55913408afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46634/0 (socket says 172.21.15.98:46634) 2024-09-19T08:09:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6a938b640 1 -- 172.21.15.98:0/656884910 learned_addr learned my addr 172.21.15.98:0/656884910 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6a8b8a640 1 --2- 172.21.15.98:0/656884910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559134080dc0 0x559134088280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6a938b640 1 -- 172.21.15.98:0/656884910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559134080dc0 msgr2=0x559134088280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6a938b640 1 --2- 172.21.15.98:0/656884910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559134080dc0 0x559134088280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6a938b640 1 -- 172.21.15.98:0/656884910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55913408b680 msgr2=0x5591340887e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6a9b8c640 1 --2- 172.21.15.98:0/656884910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55913408b680 0x5591340887e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6a938b640 1 --2- 172.21.15.98:0/656884910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55913408b680 0x5591340887e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.762+0000 7fe6a938b640 1 -- 172.21.15.98:0/656884910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559133d573c0 con 0x5591340804a0 2024-09-19T08:09:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe6a938b640 1 --2- 172.21.15.98:0/656884910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x55913408afd0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0097e50 tx=0x7fe6a009b250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c3f7347227e92 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe68cff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:09:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe68cff9640 1 -- 172.21.15.98:0/656884910 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a001e040 con 0x5591340804a0 2024-09-19T08:09:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe68cff9640 1 -- 172.21.15.98:0/656884910 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fe6a00cca60 con 0x5591340804a0 2024-09-19T08:09:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/656884910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe69c04e070 con 0x5591340804a0 2024-09-19T08:09:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe68cff9640 1 -- 172.21.15.98:0/656884910 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe6a009bb80 con 0x5591340804a0 2024-09-19T08:09:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe6ad9c8740 10 client.17088 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/656884910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559134099cb0 con 0x5591340804a0 2024-09-19T08:09:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe68cff9640 1 -- 172.21.15.98:0/656884910 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe69c04e070 con 0x5591340804a0 2024-09-19T08:09:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe68cff9640 1 -- 172.21.15.98:0/656884910 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe6a0097af0 con 0x5591340804a0 2024-09-19T08:09:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe6ad9c8740 10 client.17088 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe6ad9c8740 10 client.17088 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:09:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe6ad9c8740 20 client.17088 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/656884910 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55913409b1c0 0x55913409d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe6ad9c8740 4 client.17088 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:09:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/656884910 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559133ff3850 con 0x55913409b1c0 2024-09-19T08:09:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe6a8b8a640 1 --2- 172.21.15.98:0/656884910 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55913409b1c0 0x55913409d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe68cff9640 10 client.17088 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.763+0000 7fe6a8b8a640 1 --2- 172.21.15.98:0/656884910 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55913409b1c0 0x55913409d610 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbed86b9fe9d816f server_cookie=d693a07411cb2a1f in_seq=0 out_seq=0 2024-09-19T08:09:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe68cff9640 1 -- 172.21.15.98:0/656884910 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559133ff3850 con 0x55913409b1c0 2024-09-19T08:09:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe68cff9640 10 client.17088 handle_command_reply: tid=0 2024-09-19T08:09:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe6ad9c8740 1 client.17088 shutdown 2024-09-19T08:09:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/656884910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 msgr2=0x55913408afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/656884910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x55913408afd0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0097e50 tx=0x7fe6a009b250 comp rx=0 tx=0).stop 2024-09-19T08:09:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/656884910 shutdown_connections 2024-09-19T08:09:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/656884910 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55913409b1c0 0x55913409d610 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/656884910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55913408b680 0x5591340887e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/656884910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559134080dc0 0x559134088280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/656884910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x55913408afd0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe68cff9640 0 client.17088 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/656884910 >> 172.21.15.98:0/656884910 conn(0x55913407ca30 msgr2=0x55913407bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/656884910 shutdown_connections 2024-09-19T08:09:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/656884910 wait complete. 2024-09-19T08:09:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.764+0000 7fe6ad9c8740 20 client.17088 trim_cache size 0 max 16384 2024-09-19T08:09:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6ad9c8740 1 Processor -- start 2024-09-19T08:09:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6ad9c8740 1 -- start start 2024-09-19T08:09:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6ad9c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x5591340974e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6ad9c8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559134080dc0 0x559134086b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6ad9c8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55913408b680 0x5591340a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6ad9c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a41b3f20 con 0x5591340804a0 2024-09-19T08:09:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6ad9c8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a41b5a60 con 0x559134080dc0 2024-09-19T08:09:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6ad9c8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a41b45f0 con 0x55913408b680 2024-09-19T08:09:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6a938b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x5591340974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6a938b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x5591340974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46640/0 (socket says 172.21.15.98:46640) 2024-09-19T08:09:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6a938b640 1 -- 172.21.15.98:0/1142404343 learned_addr learned my addr 172.21.15.98:0/1142404343 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6a8b8a640 1 --2- 172.21.15.98:0/1142404343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559134080dc0 0x559134086b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6a9b8c640 1 --2- 172.21.15.98:0/1142404343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55913408b680 0x5591340a2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6a938b640 1 -- 172.21.15.98:0/1142404343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55913408b680 msgr2=0x5591340a2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6a938b640 1 --2- 172.21.15.98:0/1142404343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55913408b680 0x5591340a2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6a938b640 1 -- 172.21.15.98:0/1142404343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559134080dc0 msgr2=0x559134086b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6a938b640 1 --2- 172.21.15.98:0/1142404343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559134080dc0 0x559134086b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.781+0000 7fe6a938b640 1 -- 172.21.15.98:0/1142404343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559134087230 con 0x5591340804a0 2024-09-19T08:09:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.782+0000 7fe6a938b640 1 --2- 172.21.15.98:0/1142404343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x5591340974e0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fe6a00ccde0 tx=0x7fe6a0017730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5783c48db9acdb6a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.782+0000 7fe68effd640 1 -- 172.21.15.98:0/1142404343 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a001e070 con 0x5591340804a0 2024-09-19T08:09:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.782+0000 7fe68effd640 1 -- 172.21.15.98:0/1142404343 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6a00a40f0 con 0x5591340804a0 2024-09-19T08:09:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.782+0000 7fe68effd640 1 -- 172.21.15.98:0/1142404343 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a009ab30 con 0x5591340804a0 2024-09-19T08:09:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.782+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/1142404343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 msgr2=0x5591340974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.782+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/1142404343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x5591340974e0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fe6a00ccde0 tx=0x7fe6a0017730 comp rx=0 tx=0).stop 2024-09-19T08:09:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.782+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/1142404343 shutdown_connections 2024-09-19T08:09:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.782+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/1142404343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55913408b680 0x5591340a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.782+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/1142404343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559134080dc0 0x559134086b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.782+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/1142404343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5591340804a0 0x5591340974e0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.782+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/1142404343 >> 172.21.15.98:0/1142404343 conn(0x55913407ca30 msgr2=0x5591340838c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/1142404343 shutdown_connections 2024-09-19T08:09:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/1142404343 wait complete. 2024-09-19T08:09:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6ad9c8740 1 -- start start 2024-09-19T08:09:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6ad9c8740 1 Processor -- start 2024-09-19T08:09:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6ad9c8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5591340804a0 0x559134093f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6ad9c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559134080dc0 0x55913408f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6ad9c8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55913408b680 0x55913408fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6ad9c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a41b5390 con 0x559134080dc0 2024-09-19T08:09:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6ad9c8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a41b4cc0 con 0x55913408b680 2024-09-19T08:09:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6ad9c8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a41b1f00 con 0x5591340804a0 2024-09-19T08:09:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6ad9c8740 10 client.0 _ll_register_callbacks cb 0x7fe6a00970c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:09:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6a8b8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559134080dc0 0x55913408f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6a938b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5591340804a0 0x559134093f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6a8b8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559134080dc0 0x55913408f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46642/0 (socket says 172.21.15.98:46642) 2024-09-19T08:09:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6a8b8a640 1 -- 172.21.15.98:0/195519250 learned_addr learned my addr 172.21.15.98:0/195519250 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:09:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.783+0000 7fe6a9b8c640 1 --2- 172.21.15.98:0/195519250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55913408b680 0x55913408fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.784+0000 7fe6a938b640 1 -- 172.21.15.98:0/195519250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55913408b680 msgr2=0x55913408fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.784+0000 7fe6a938b640 1 --2- 172.21.15.98:0/195519250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55913408b680 0x55913408fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.784+0000 7fe6a938b640 1 -- 172.21.15.98:0/195519250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559134080dc0 msgr2=0x55913408f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:09:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.784+0000 7fe6a938b640 1 --2- 172.21.15.98:0/195519250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559134080dc0 0x55913408f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.784+0000 7fe6a938b640 1 -- 172.21.15.98:0/195519250 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559134080bf0 con 0x5591340804a0 2024-09-19T08:09:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.784+0000 7fe6a938b640 1 --2- 172.21.15.98:0/195519250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5591340804a0 0x559134093f60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe6a00978e0 tx=0x7fe6a00047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2ddb33bc6d09a12 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:09:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.784+0000 7fe68cff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-09-19T08:09:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.784+0000 7fe68cff9640 1 -- 172.21.15.98:0/195519250 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a001e070 con 0x5591340804a0 2024-09-19T08:09:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.784+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/195519250 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6a00a40f0 con 0x5591340804a0 2024-09-19T08:09:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.784+0000 7fe68cff9640 1 -- 172.21.15.98:0/195519250 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fe6a009aad0 con 0x5591340804a0 2024-09-19T08:09:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.784+0000 7fe68cff9640 1 -- 172.21.15.98:0/195519250 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe6a009f830 con 0x5591340804a0 2024-09-19T08:09:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.784+0000 7fe6ad9c8740 10 client.26128 fetch_fsmap learned FSMap version 32 2024-09-19T08:09:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.784+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/195519250 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559134088b50 con 0x5591340804a0 2024-09-19T08:09:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.784+0000 7fe68cff9640 1 -- 172.21.15.98:0/195519250 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6a00a40f0 con 0x5591340804a0 2024-09-19T08:09:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.785+0000 7fe68cff9640 1 -- 172.21.15.98:0/195519250 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe6a00a09c0 con 0x5591340804a0 2024-09-19T08:09:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.785+0000 7fe6ad9c8740 10 client.26128 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:09:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.785+0000 7fe6ad9c8740 10 client.26128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:09:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.785+0000 7fe6ad9c8740 20 client.26128 populate_metadata read hostname 'smithi098' 2024-09-19T08:09:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.785+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/195519250 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x559134106200 0x5591341085f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:09:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.785+0000 7fe6ad9c8740 4 client.26128 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:09:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.785+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/195519250 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559133fedd60 con 0x559134106200 2024-09-19T08:09:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.785+0000 7fe6a8b8a640 1 --2- 172.21.15.98:0/195519250 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x559134106200 0x5591341085f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:09:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.785+0000 7fe68cff9640 10 client.26128 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.785+0000 7fe6a8b8a640 1 --2- 172.21.15.98:0/195519250 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x559134106200 0x5591341085f0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c371a287a520074 server_cookie=b95f3be1512fc75 in_seq=0 out_seq=0 2024-09-19T08:09:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.785+0000 7fe68cff9640 1 -- 172.21.15.98:0/195519250 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559133fedd60 con 0x559134106200 2024-09-19T08:09:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.785+0000 7fe68cff9640 10 client.26128 handle_command_reply: tid=0 2024-09-19T08:09:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.785+0000 7fe6ad9c8740 1 client.26128 shutdown 2024-09-19T08:09:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/195519250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5591340804a0 msgr2=0x559134093f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/195519250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5591340804a0 0x559134093f60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe6a00978e0 tx=0x7fe6a00047d0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/195519250 shutdown_connections 2024-09-19T08:09:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/195519250 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x559134106200 0x5591341085f0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/195519250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55913408b680 0x55913408fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/195519250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559134080dc0 0x55913408f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe68cff9640 0 client.26128 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:09:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ad9c8740 1 --2- 172.21.15.98:0/195519250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5591340804a0 0x559134093f60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/195519250 >> 172.21.15.98:0/195519250 conn(0x55913407ca30 msgr2=0x559134081f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/195519250 shutdown_connections 2024-09-19T08:09:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ad9c8740 1 -- 172.21.15.98:0/195519250 wait complete. 2024-09-19T08:09:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ad9c8740 20 client.26128 trim_cache size 0 max 16384 2024-09-19T08:09:28.809 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:09:28.809 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-19T08:09:28.809 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-09-19T08:09:28.810 INFO:teuthology.orchestra.run.smithi098.stdout: "a2ef10f3-53d8-4d3d-a763-e62742b8063e": { 2024-09-19T08:09:28.810 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-09-19T08:09:28.810 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "a2ef10f3-53d8-4d3d-a763-e62742b8063e", 2024-09-19T08:09:28.810 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-09-19T08:09:28.810 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:09:28.810 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:09:28.810 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:09:28.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ab5ef640 1 -- 172.21.15.98:0/1917321824 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe6900a33b0 msgr2=0x7fe6900a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:28.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ab5ef640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe6900a33b0 0x7fe6900a5870 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe69c04d000 tx=0x7fe69c04b000 comp rx=0 tx=0).stop 2024-09-19T08:09:28.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ab5ef640 1 -- 172.21.15.98:0/1917321824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a4074590 msgr2=0x7fe6a41a4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:09:28.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ab5ef640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a4074590 0x7fe6a41a4740 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000b4a0 tx=0x7fe6a000b960 comp rx=0 tx=0).stop 2024-09-19T08:09:28.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ab5ef640 1 -- 172.21.15.98:0/1917321824 shutdown_connections 2024-09-19T08:09:28.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ab5ef640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe6900a33b0 0x7fe6900a5870 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ab5ef640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6a4112ba0 0x7fe6a419e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ab5ef640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6a4074eb0 0x7fe6a41a4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ab5ef640 1 --2- 172.21.15.98:0/1917321824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a4074590 0x7fe6a41a4740 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:09:28.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ab5ef640 1 -- 172.21.15.98:0/1917321824 >> 172.21.15.98:0/1917321824 conn(0x7fe6a40fd8f0 msgr2=0x7fe6a40fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:09:28.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ab5ef640 1 -- 172.21.15.98:0/1917321824 shutdown_connections 2024-09-19T08:09:28.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:09:28.786+0000 7fe6ab5ef640 1 -- 172.21.15.98:0/1917321824 wait complete. 2024-09-19T08:09:29.574 INFO:tasks.cephfs.filesystem:scrub status for tag:a2ef10f3-53d8-4d3d-a763-e62742b8063e - {'path': '/', 'tag': 'a2ef10f3-53d8-4d3d-a763-e62742b8063e', 'options': 'recursive,force'} 2024-09-19T08:09:30.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:30 smithi162 ceph-mon[35260]: pgmap v197: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-09-19T08:09:30.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:30 smithi139 ceph-mon[34874]: pgmap v197: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-09-19T08:09:30.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:30 smithi098 ceph-mon[31333]: pgmap v197: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-09-19T08:09:31.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:31 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:09:31.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:31 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:09:31.619 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:31 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:09:32.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:32 smithi139 ceph-mon[34874]: pgmap v198: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:32.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:32 smithi098 ceph-mon[31333]: pgmap v198: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:32.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:32 smithi162 ceph-mon[35260]: pgmap v198: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:34.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:34 smithi139 ceph-mon[34874]: pgmap v199: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:09:34.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:34 smithi098 ceph-mon[31333]: pgmap v199: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:09:34.619 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:34 smithi162 ceph-mon[35260]: pgmap v199: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:09:36.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:36 smithi139 ceph-mon[34874]: pgmap v200: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:09:36.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:36 smithi098 ceph-mon[31333]: pgmap v200: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:09:36.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:36 smithi162 ceph-mon[35260]: pgmap v200: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:09:38.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:38 smithi139 ceph-mon[34874]: pgmap v201: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-09-19T08:09:38.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:38 smithi098 ceph-mon[31333]: pgmap v201: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-09-19T08:09:38.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:38 smithi162 ceph-mon[35260]: pgmap v201: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-09-19T08:09:40.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:40 smithi139 ceph-mon[34874]: pgmap v202: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-09-19T08:09:40.636 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:40 smithi162 ceph-mon[35260]: pgmap v202: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-09-19T08:09:40.636 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:40 smithi098 ceph-mon[31333]: pgmap v202: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-09-19T08:09:43.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:42 smithi162 ceph-mon[35260]: pgmap v203: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:43.130 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:42 smithi139 ceph-mon[34874]: pgmap v203: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:43.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:42 smithi098 ceph-mon[31333]: pgmap v203: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:45.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:44 smithi162 ceph-mon[35260]: pgmap v204: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-09-19T08:09:45.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:44 smithi139 ceph-mon[34874]: pgmap v204: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-09-19T08:09:45.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:44 smithi098 ceph-mon[31333]: pgmap v204: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-09-19T08:09:47.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:46 smithi162 ceph-mon[35260]: pgmap v205: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:47.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:46 smithi139 ceph-mon[34874]: pgmap v205: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:47.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:46 smithi098 ceph-mon[31333]: pgmap v205: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:49.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:48 smithi162 ceph-mon[35260]: pgmap v206: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:09:49.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:48 smithi139 ceph-mon[34874]: pgmap v206: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:09:49.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:48 smithi098 ceph-mon[31333]: pgmap v206: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:09:51.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:50 smithi162 ceph-mon[35260]: pgmap v207: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:09:51.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:50 smithi139 ceph-mon[34874]: pgmap v207: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:09:51.185 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:50 smithi098 ceph-mon[31333]: pgmap v207: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:09:53.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:52 smithi162 ceph-mon[35260]: pgmap v208: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:53.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:52 smithi139 ceph-mon[34874]: pgmap v208: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:53.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:52 smithi098 ceph-mon[31333]: pgmap v208: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:55.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:54 smithi162 ceph-mon[35260]: pgmap v209: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:09:55.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:54 smithi139 ceph-mon[34874]: pgmap v209: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:09:55.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:54 smithi098 ceph-mon[31333]: pgmap v209: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:09:57.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:56 smithi162 ceph-mon[35260]: pgmap v210: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:57.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:56 smithi139 ceph-mon[34874]: pgmap v210: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:57.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:56 smithi098 ceph-mon[31333]: pgmap v210: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:09:59.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:09:58 smithi162 ceph-mon[35260]: pgmap v211: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:09:59.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:09:58 smithi139 ceph-mon[34874]: pgmap v211: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:09:59.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:09:58 smithi098 ceph-mon[31333]: pgmap v211: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:09:59.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:09:59.838 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:00.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.317+0000 7fae23492640 1 -- 172.21.15.98:0/1489367998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae1c102570 msgr2=0x7fae1c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:00.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.317+0000 7fae23492640 1 --2- 172.21.15.98:0/1489367998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae1c102570 0x7fae1c1029b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fae0c009900 tx=0x7fae0c02a480 comp rx=0 tx=0).stop 2024-09-19T08:10:00.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 -- 172.21.15.98:0/1489367998 shutdown_connections 2024-09-19T08:10:00.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 --2- 172.21.15.98:0/1489367998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae1c102ef0 0x7fae1c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 --2- 172.21.15.98:0/1489367998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae1c102570 0x7fae1c1029b0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 --2- 172.21.15.98:0/1489367998 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae1c100e20 0x7fae1c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 -- 172.21.15.98:0/1489367998 >> 172.21.15.98:0/1489367998 conn(0x7fae1c0fc7d0 msgr2=0x7fae1c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:00.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 -- 172.21.15.98:0/1489367998 shutdown_connections 2024-09-19T08:10:00.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 -- 172.21.15.98:0/1489367998 wait complete. 2024-09-19T08:10:00.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 Processor -- start 2024-09-19T08:10:00.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 -- start start 2024-09-19T08:10:00.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae1c100e20 0x7fae1c199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae1c102570 0x7fae1c199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae1c102ef0 0x7fae1c1a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c110f80 con 0x7fae1c100e20 2024-09-19T08:10:00.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c110e00 con 0x7fae1c102570 2024-09-19T08:10:00.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.318+0000 7fae23492640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c111100 con 0x7fae1c102ef0 2024-09-19T08:10:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae2122e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae1c100e20 0x7fae1c199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae2122e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae1c100e20 0x7fae1c199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34602/0 (socket says 172.21.15.98:34602) 2024-09-19T08:10:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae2122e640 1 -- 172.21.15.98:0/3664733141 learned_addr learned my addr 172.21.15.98:0/3664733141 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae21a2f640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae1c102ef0 0x7fae1c1a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae20a2d640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae1c102570 0x7fae1c199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae2122e640 1 -- 172.21.15.98:0/3664733141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae1c102ef0 msgr2=0x7fae1c1a4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae2122e640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae1c102ef0 0x7fae1c1a4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae2122e640 1 -- 172.21.15.98:0/3664733141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae1c102570 msgr2=0x7fae1c199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae2122e640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae1c102570 0x7fae1c199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae2122e640 1 -- 172.21.15.98:0/3664733141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae1c1a48d0 con 0x7fae1c100e20 2024-09-19T08:10:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae20a2d640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae1c102570 0x7fae1c199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae2122e640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae1c100e20 0x7fae1c199310 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fae1800d950 tx=0x7fae1800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae127fc640 1 -- 172.21.15.98:0/3664733141 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae18017070 con 0x7fae1c100e20 2024-09-19T08:10:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae23492640 1 -- 172.21.15.98:0/3664733141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae1c1a4aa0 con 0x7fae1c100e20 2024-09-19T08:10:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae127fc640 1 -- 172.21.15.98:0/3664733141 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae18009c30 con 0x7fae1c100e20 2024-09-19T08:10:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae127fc640 1 -- 172.21.15.98:0/3664733141 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1800f6b0 con 0x7fae1c100e20 2024-09-19T08:10:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.319+0000 7fae23492640 1 -- 172.21.15.98:0/3664733141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae1c1a4fe0 con 0x7fae1c100e20 2024-09-19T08:10:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.322+0000 7fae127fc640 1 -- 172.21.15.98:0/3664733141 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fae180109d0 con 0x7fae1c100e20 2024-09-19T08:10:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.322+0000 7fae2586b740 1 Processor -- start 2024-09-19T08:10:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.322+0000 7fae2586b740 1 -- start start 2024-09-19T08:10:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.322+0000 7fae2586b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a64a0 0x5574c51a6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.322+0000 7fae2586b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51a6dc0 0x5574c51b1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.322+0000 7fae2586b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51b1680 0x5574c51b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.322+0000 7fae2586b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5574c510c2e0 con 0x5574c51a64a0 2024-09-19T08:10:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.322+0000 7fae2586b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5574c50b3ce0 con 0x5574c51b1680 2024-09-19T08:10:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.322+0000 7fae2586b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5574c51082a0 con 0x5574c51a6dc0 2024-09-19T08:10:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.322+0000 7fae127fc640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae080a33b0 0x7fae080a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae21a2f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a64a0 0x5574c51a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae21a2f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a64a0 0x5574c51a6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34614/0 (socket says 172.21.15.98:34614) 2024-09-19T08:10:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae21a2f640 1 -- 172.21.15.98:0/2126028209 learned_addr learned my addr 172.21.15.98:0/2126028209 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae21a2f640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae080a33b0 0x7fae080a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae127fc640 1 -- 172.21.15.98:0/3664733141 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fae180cfa10 con 0x7fae1c100e20 2024-09-19T08:10:00.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae20a2d640 1 --2- 172.21.15.98:0/2126028209 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51b1680 0x5574c51b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae2122e640 1 --2- 172.21.15.98:0/2126028209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51a6dc0 0x5574c51b1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae21a2f640 1 -- 172.21.15.98:0/2126028209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51a6dc0 msgr2=0x5574c51b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:00.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae21a2f640 1 --2- 172.21.15.98:0/2126028209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51a6dc0 0x5574c51b1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae21a2f640 1 -- 172.21.15.98:0/2126028209 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51b1680 msgr2=0x5574c51b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:00.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae21a2f640 1 --2- 172.21.15.98:0/2126028209 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51b1680 0x5574c51b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae21a2f640 1 -- 172.21.15.98:0/2126028209 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574c5098440 con 0x5574c51a64a0 2024-09-19T08:10:00.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae2122e640 1 --2- 172.21.15.98:0/2126028209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51a6dc0 0x5574c51b1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:00.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae21a2f640 1 --2- 172.21.15.98:0/2126028209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a64a0 0x5574c51a6880 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fae14012c50 tx=0x7fae14010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2989c6f482a34e15 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:00.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae21a2f640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae080a33b0 0x7fae080a5870 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fae14010c40 tx=0x7fae14017000 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:00.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.323+0000 7fae077fe640 1 -- 172.21.15.98:0/2126028209 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae14012e00 con 0x5574c51a64a0 2024-09-19T08:10:00.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.324+0000 7fae077fe640 1 -- 172.21.15.98:0/2126028209 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae14009c00 con 0x5574c51a64a0 2024-09-19T08:10:00.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.324+0000 7fae077fe640 1 -- 172.21.15.98:0/2126028209 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae14005420 con 0x5574c51a64a0 2024-09-19T08:10:00.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.324+0000 7fae2586b740 1 -- 172.21.15.98:0/2126028209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a64a0 msgr2=0x5574c51a6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:00.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.324+0000 7fae2586b740 1 --2- 172.21.15.98:0/2126028209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a64a0 0x5574c51a6880 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fae14012c50 tx=0x7fae14010740 comp rx=0 tx=0).stop 2024-09-19T08:10:00.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.324+0000 7fae2586b740 1 -- 172.21.15.98:0/2126028209 shutdown_connections 2024-09-19T08:10:00.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.324+0000 7fae2586b740 1 --2- 172.21.15.98:0/2126028209 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51b1680 0x5574c51b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.324+0000 7fae2586b740 1 --2- 172.21.15.98:0/2126028209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51a6dc0 0x5574c51b1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.324+0000 7fae2586b740 1 --2- 172.21.15.98:0/2126028209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a64a0 0x5574c51a6880 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.324+0000 7fae2586b740 1 -- 172.21.15.98:0/2126028209 >> 172.21.15.98:0/2126028209 conn(0x5574c51a2a30 msgr2=0x5574c51a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:00.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.324+0000 7fae2586b740 1 -- 172.21.15.98:0/2126028209 shutdown_connections 2024-09-19T08:10:00.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.324+0000 7fae2586b740 1 -- 172.21.15.98:0/2126028209 wait complete. 2024-09-19T08:10:00.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.325+0000 7fae2586b740 1 -- start start 2024-09-19T08:10:00.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.325+0000 7fae2586b740 1 Processor -- start 2024-09-19T08:10:00.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.325+0000 7fae2586b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a64a0 0x5574c51b0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.325+0000 7fae2586b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a6dc0 0x5574c51ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.325+0000 7fae2586b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51b1680 0x5574c51ae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.325+0000 7fae2586b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5574c51b4250 con 0x5574c51a64a0 2024-09-19T08:10:00.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.325+0000 7fae2586b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5574c50c7050 con 0x5574c51a6dc0 2024-09-19T08:10:00.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.325+0000 7fae2586b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5574c51b43d0 con 0x5574c51b1680 2024-09-19T08:10:00.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.325+0000 7fae2586b740 10 client.0 _ll_register_callbacks cb 0x5574c5107160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:00.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.325+0000 7fae2122e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a64a0 0x5574c51b0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.325+0000 7fae2122e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a64a0 0x5574c51b0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34626/0 (socket says 172.21.15.98:34626) 2024-09-19T08:10:00.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.325+0000 7fae2122e640 1 -- 172.21.15.98:0/1171486468 learned_addr learned my addr 172.21.15.98:0/1171486468 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:00.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae20a2d640 1 --2- 172.21.15.98:0/1171486468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a6dc0 0x5574c51ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2122e640 1 -- 172.21.15.98:0/1171486468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51b1680 msgr2=0x5574c51ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:00.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2122e640 1 --2- 172.21.15.98:0/1171486468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51b1680 0x5574c51ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2122e640 1 -- 172.21.15.98:0/1171486468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a6dc0 msgr2=0x5574c51ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:00.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2122e640 1 --2- 172.21.15.98:0/1171486468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a6dc0 0x5574c51ae2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2122e640 1 -- 172.21.15.98:0/1171486468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574c4e7d3c0 con 0x5574c51a64a0 2024-09-19T08:10:00.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae20a2d640 1 --2- 172.21.15.98:0/1171486468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a6dc0 0x5574c51ae2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:00.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2122e640 1 --2- 172.21.15.98:0/1171486468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a64a0 0x5574c51b0ff0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fae18009610 tx=0x7fae1809b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c54f88e2bc12368 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae04ff9640 1 -- 172.21.15.98:0/1171486468 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae18020040 con 0x5574c51a64a0 2024-09-19T08:10:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae04ff9640 1 -- 172.21.15.98:0/1171486468 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fae1809b170 con 0x5574c51a64a0 2024-09-19T08:10:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2586b740 1 -- 172.21.15.98:0/1171486468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5574c5119850 con 0x5574c51a64a0 2024-09-19T08:10:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae04ff9640 1 -- 172.21.15.98:0/1171486468 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fae1809bd20 con 0x5574c51a64a0 2024-09-19T08:10:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2586b740 10 client.17106 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2586b740 1 -- 172.21.15.98:0/1171486468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5574c51bfc20 con 0x5574c51a64a0 2024-09-19T08:10:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae04ff9640 1 -- 172.21.15.98:0/1171486468 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5574c5119850 con 0x5574c51a64a0 2024-09-19T08:10:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae04ff9640 1 -- 172.21.15.98:0/1171486468 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fae18010460 con 0x5574c51a64a0 2024-09-19T08:10:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2586b740 10 client.17106 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2586b740 10 client.17106 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2586b740 20 client.17106 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2586b740 1 --2- 172.21.15.98:0/1171486468 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5574c51c1080 0x5574c51c3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2586b740 4 client.17106 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.326+0000 7fae2586b740 1 -- 172.21.15.98:0/1171486468 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574c5113d60 con 0x5574c51c1080 2024-09-19T08:10:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.327+0000 7fae20a2d640 1 --2- 172.21.15.98:0/1171486468 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5574c51c1080 0x5574c51c3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.327+0000 7fae20a2d640 1 --2- 172.21.15.98:0/1171486468 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5574c51c1080 0x5574c51c3490 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48712911d8bb2476 server_cookie=7ce3150c4d6bc6d in_seq=0 out_seq=0 2024-09-19T08:10:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.327+0000 7fae04ff9640 10 client.17106 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae04ff9640 1 -- 172.21.15.98:0/1171486468 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5574c5113d60 con 0x5574c51c1080 2024-09-19T08:10:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae04ff9640 10 client.17106 handle_command_reply: tid=0 2024-09-19T08:10:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae2586b740 1 client.17106 shutdown 2024-09-19T08:10:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae2586b740 1 -- 172.21.15.98:0/1171486468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a64a0 msgr2=0x5574c51b0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae2586b740 1 --2- 172.21.15.98:0/1171486468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a64a0 0x5574c51b0ff0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fae18009610 tx=0x7fae1809b740 comp rx=0 tx=0).stop 2024-09-19T08:10:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae2586b740 1 -- 172.21.15.98:0/1171486468 shutdown_connections 2024-09-19T08:10:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae2586b740 1 --2- 172.21.15.98:0/1171486468 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5574c51c1080 0x5574c51c3490 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae2586b740 1 --2- 172.21.15.98:0/1171486468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51b1680 0x5574c51ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae2586b740 1 --2- 172.21.15.98:0/1171486468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a6dc0 0x5574c51ae2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae04ff9640 0 client.17106 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae2586b740 1 --2- 172.21.15.98:0/1171486468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a64a0 0x5574c51b0ff0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae2586b740 1 -- 172.21.15.98:0/1171486468 >> 172.21.15.98:0/1171486468 conn(0x5574c51a2a30 msgr2=0x5574c51a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae2586b740 1 -- 172.21.15.98:0/1171486468 shutdown_connections 2024-09-19T08:10:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae2586b740 1 -- 172.21.15.98:0/1171486468 wait complete. 2024-09-19T08:10:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.328+0000 7fae2586b740 20 client.17106 trim_cache size 0 max 16384 2024-09-19T08:10:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.345+0000 7fae2586b740 1 Processor -- start 2024-09-19T08:10:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.345+0000 7fae2586b740 1 -- start start 2024-09-19T08:10:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae2586b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a64a0 0x5574c51bd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae2586b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a6dc0 0x5574c51c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae2586b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51b1680 0x5574c51c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae2586b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c1ad8a0 con 0x5574c51a6dc0 2024-09-19T08:10:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae2586b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c1af3e0 con 0x5574c51a64a0 2024-09-19T08:10:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae2586b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c1adf70 con 0x5574c51b1680 2024-09-19T08:10:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae20a2d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a6dc0 0x5574c51c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae20a2d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a6dc0 0x5574c51c8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34628/0 (socket says 172.21.15.98:34628) 2024-09-19T08:10:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae20a2d640 1 -- 172.21.15.98:0/3627813588 learned_addr learned my addr 172.21.15.98:0/3627813588 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae21a2f640 1 --2- 172.21.15.98:0/3627813588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51b1680 0x5574c51c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae2122e640 1 --2- 172.21.15.98:0/3627813588 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a64a0 0x5574c51bd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae20a2d640 1 -- 172.21.15.98:0/3627813588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51b1680 msgr2=0x5574c51c92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae20a2d640 1 --2- 172.21.15.98:0/3627813588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51b1680 0x5574c51c92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae20a2d640 1 -- 172.21.15.98:0/3627813588 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a64a0 msgr2=0x5574c51bd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae20a2d640 1 --2- 172.21.15.98:0/3627813588 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a64a0 0x5574c51bd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae20a2d640 1 -- 172.21.15.98:0/3627813588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574c51ad250 con 0x5574c51a6dc0 2024-09-19T08:10:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae20a2d640 1 --2- 172.21.15.98:0/3627813588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a6dc0 0x5574c51c8d60 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fae0c009230 tx=0x7fae0c0045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db454535d4987f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.346+0000 7fae06ffd640 1 -- 172.21.15.98:0/3627813588 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c04b070 con 0x5574c51a6dc0 2024-09-19T08:10:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.347+0000 7fae06ffd640 1 -- 172.21.15.98:0/3627813588 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae0c0055e0 con 0x5574c51a6dc0 2024-09-19T08:10:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.347+0000 7fae06ffd640 1 -- 172.21.15.98:0/3627813588 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c035040 con 0x5574c51a6dc0 2024-09-19T08:10:00.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.347+0000 7fae2586b740 1 -- 172.21.15.98:0/3627813588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a6dc0 msgr2=0x5574c51c8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:00.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.347+0000 7fae2586b740 1 --2- 172.21.15.98:0/3627813588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a6dc0 0x5574c51c8d60 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fae0c009230 tx=0x7fae0c0045b0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.347+0000 7fae2586b740 1 -- 172.21.15.98:0/3627813588 shutdown_connections 2024-09-19T08:10:00.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.347+0000 7fae2586b740 1 --2- 172.21.15.98:0/3627813588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51b1680 0x5574c51c92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.347+0000 7fae2586b740 1 --2- 172.21.15.98:0/3627813588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a6dc0 0x5574c51c8d60 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.347+0000 7fae2586b740 1 --2- 172.21.15.98:0/3627813588 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a64a0 0x5574c51bd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.347+0000 7fae2586b740 1 -- 172.21.15.98:0/3627813588 >> 172.21.15.98:0/3627813588 conn(0x5574c51a2a30 msgr2=0x5574c51a8710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:00.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.347+0000 7fae2586b740 1 -- 172.21.15.98:0/3627813588 shutdown_connections 2024-09-19T08:10:00.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.347+0000 7fae2586b740 1 -- 172.21.15.98:0/3627813588 wait complete. 2024-09-19T08:10:00.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.348+0000 7fae2586b740 1 -- start start 2024-09-19T08:10:00.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.348+0000 7fae2586b740 1 Processor -- start 2024-09-19T08:10:00.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.348+0000 7fae2586b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a64a0 0x5574c51ab340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.348+0000 7fae2586b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a6dc0 0x5574c51ab880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.348+0000 7fae2586b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51b1680 0x5574c51abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.348+0000 7fae2586b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c1aed10 con 0x5574c51a6dc0 2024-09-19T08:10:00.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.348+0000 7fae2586b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c1ae640 con 0x5574c51a64a0 2024-09-19T08:10:00.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.348+0000 7fae2586b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c1ab880 con 0x5574c51b1680 2024-09-19T08:10:00.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.348+0000 7fae2586b740 10 client.0 _ll_register_callbacks cb 0x7fae1800df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:00.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.348+0000 7fae20a2d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a6dc0 0x5574c51ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.349+0000 7fae20a2d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a6dc0 0x5574c51ab880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34630/0 (socket says 172.21.15.98:34630) 2024-09-19T08:10:00.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.349+0000 7fae20a2d640 1 -- 172.21.15.98:0/1578448317 learned_addr learned my addr 172.21.15.98:0/1578448317 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:00.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.349+0000 7fae2122e640 1 --2- 172.21.15.98:0/1578448317 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a64a0 0x5574c51ab340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.349+0000 7fae21a2f640 1 --2- 172.21.15.98:0/1578448317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51b1680 0x5574c51abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.349+0000 7fae20a2d640 1 -- 172.21.15.98:0/1578448317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51b1680 msgr2=0x5574c51abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.349+0000 7fae20a2d640 1 --2- 172.21.15.98:0/1578448317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51b1680 0x5574c51abdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.349+0000 7fae20a2d640 1 -- 172.21.15.98:0/1578448317 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a64a0 msgr2=0x5574c51ab340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.349+0000 7fae20a2d640 1 --2- 172.21.15.98:0/1578448317 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a64a0 0x5574c51ab340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.349+0000 7fae20a2d640 1 -- 172.21.15.98:0/1578448317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574c522ade0 con 0x5574c51a6dc0 2024-09-19T08:10:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.349+0000 7fae20a2d640 1 --2- 172.21.15.98:0/1578448317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a6dc0 0x5574c51ab880 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fae0c049040 tx=0x7fae0c02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6545de698703f71f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.349+0000 7fae04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.349+0000 7fae04ff9640 1 -- 172.21.15.98:0/1578448317 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c054060 con 0x5574c51a6dc0 2024-09-19T08:10:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.349+0000 7fae2586b740 1 -- 172.21.15.98:0/1578448317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae0c0055e0 con 0x5574c51a6dc0 2024-09-19T08:10:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.350+0000 7fae04ff9640 1 -- 172.21.15.98:0/1578448317 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fae0c004d70 con 0x5574c51a6dc0 2024-09-19T08:10:00.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.350+0000 7fae04ff9640 1 -- 172.21.15.98:0/1578448317 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fae0c035050 con 0x5574c51a6dc0 2024-09-19T08:10:00.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.350+0000 7fae2586b740 10 client.17112 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:00.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.350+0000 7fae2586b740 1 -- 172.21.15.98:0/1578448317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5574c522c0d0 con 0x5574c51a6dc0 2024-09-19T08:10:00.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.350+0000 7fae04ff9640 1 -- 172.21.15.98:0/1578448317 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae0c0055e0 con 0x5574c51a6dc0 2024-09-19T08:10:00.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.350+0000 7fae04ff9640 1 -- 172.21.15.98:0/1578448317 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fae0c003760 con 0x5574c51a6dc0 2024-09-19T08:10:00.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.350+0000 7fae2586b740 10 client.17112 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:00.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.350+0000 7fae2586b740 10 client.17112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:00.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.350+0000 7fae2586b740 20 client.17112 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:00.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.350+0000 7fae2586b740 1 --2- 172.21.15.98:0/1578448317 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5574c522c780 0x5574c522eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:00.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.350+0000 7fae2586b740 4 client.17112 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:10:00.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.350+0000 7fae2586b740 1 -- 172.21.15.98:0/1578448317 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5574c5090210 con 0x5574c522c780 2024-09-19T08:10:00.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae2122e640 1 --2- 172.21.15.98:0/1578448317 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5574c522c780 0x5574c522eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:00.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae2122e640 1 --2- 172.21.15.98:0/1578448317 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5574c522c780 0x5574c522eba0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68884be6f63ed93e server_cookie=8db711f6251e7004 in_seq=0 out_seq=0 2024-09-19T08:10:00.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae04ff9640 10 client.17112 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:00.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae04ff9640 1 -- 172.21.15.98:0/1578448317 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5574c5090210 con 0x5574c522c780 2024-09-19T08:10:00.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae04ff9640 10 client.17112 handle_command_reply: tid=0 2024-09-19T08:10:00.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae2586b740 1 client.17112 shutdown 2024-09-19T08:10:00.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae2586b740 1 -- 172.21.15.98:0/1578448317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a6dc0 msgr2=0x5574c51ab880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:00.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae2586b740 1 --2- 172.21.15.98:0/1578448317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a6dc0 0x5574c51ab880 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fae0c049040 tx=0x7fae0c02a480 comp rx=0 tx=0).stop 2024-09-19T08:10:00.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae2586b740 1 -- 172.21.15.98:0/1578448317 shutdown_connections 2024-09-19T08:10:00.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae2586b740 1 --2- 172.21.15.98:0/1578448317 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5574c522c780 0x5574c522eba0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae2586b740 1 --2- 172.21.15.98:0/1578448317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574c51b1680 0x5574c51abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae2586b740 1 --2- 172.21.15.98:0/1578448317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574c51a6dc0 0x5574c51ab880 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae2586b740 1 --2- 172.21.15.98:0/1578448317 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5574c51a64a0 0x5574c51ab340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae04ff9640 0 client.17112 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:00.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.351+0000 7fae2586b740 1 -- 172.21.15.98:0/1578448317 >> 172.21.15.98:0/1578448317 conn(0x5574c51a2a30 msgr2=0x5574c51a8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:00.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae2586b740 1 -- 172.21.15.98:0/1578448317 shutdown_connections 2024-09-19T08:10:00.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae2586b740 1 -- 172.21.15.98:0/1578448317 wait complete. 2024-09-19T08:10:00.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae2586b740 20 client.17112 trim_cache size 0 max 16384 2024-09-19T08:10:00.372 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:10:00.372 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:10:00.372 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:10:00.373 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:10:00.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae23492640 1 -- 172.21.15.98:0/3664733141 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae080a33b0 msgr2=0x7fae080a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:00.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae23492640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae080a33b0 0x7fae080a5870 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fae14010c40 tx=0x7fae14017000 comp rx=0 tx=0).stop 2024-09-19T08:10:00.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae23492640 1 -- 172.21.15.98:0/3664733141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae1c100e20 msgr2=0x7fae1c199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:00.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae23492640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae1c100e20 0x7fae1c199310 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fae1800d950 tx=0x7fae1800de10 comp rx=0 tx=0).stop 2024-09-19T08:10:00.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae23492640 1 -- 172.21.15.98:0/3664733141 shutdown_connections 2024-09-19T08:10:00.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae23492640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae080a33b0 0x7fae080a5870 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae23492640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae1c102ef0 0x7fae1c1a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae23492640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae1c102570 0x7fae1c199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae23492640 1 --2- 172.21.15.98:0/3664733141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae1c100e20 0x7fae1c199310 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:00.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae23492640 1 -- 172.21.15.98:0/3664733141 >> 172.21.15.98:0/3664733141 conn(0x7fae1c0fc7d0 msgr2=0x7fae1c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:00.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae23492640 1 -- 172.21.15.98:0/3664733141 shutdown_connections 2024-09-19T08:10:00.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:00.352+0000 7fae23492640 1 -- 172.21.15.98:0/3664733141 wait complete. 2024-09-19T08:10:01.118 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:00 smithi098 ceph-mon[31333]: pgmap v212: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:10:01.118 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:00 smithi098 ceph-mon[31333]: overall HEALTH_OK 2024-09-19T08:10:01.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:00 smithi162 ceph-mon[35260]: pgmap v212: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:10:01.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:00 smithi162 ceph-mon[35260]: overall HEALTH_OK 2024-09-19T08:10:01.147 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:10:01.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:10:01.196 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:00 smithi139 ceph-mon[34874]: pgmap v212: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:10:01.197 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:00 smithi139 ceph-mon[34874]: overall HEALTH_OK 2024-09-19T08:10:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.939+0000 7f8365464640 1 -- 172.21.15.98:0/52613965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8360100a80 msgr2=0x7f8360100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.939+0000 7f8365464640 1 --2- 172.21.15.98:0/52613965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8360100a80 0x7f8360100ee0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f8348009930 tx=0x7f834802a420 comp rx=0 tx=0).stop 2024-09-19T08:10:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.939+0000 7f8365464640 1 -- 172.21.15.98:0/52613965 shutdown_connections 2024-09-19T08:10:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.939+0000 7f8365464640 1 --2- 172.21.15.98:0/52613965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8360101420 0x7f836010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.939+0000 7f8365464640 1 --2- 172.21.15.98:0/52613965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8360100a80 0x7f8360100ee0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.939+0000 7f8365464640 1 --2- 172.21.15.98:0/52613965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8360106a80 0x7f8360106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.939+0000 7f8365464640 1 -- 172.21.15.98:0/52613965 >> 172.21.15.98:0/52613965 conn(0x7f83600fc7a0 msgr2=0x7f83600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.939+0000 7f8365464640 1 -- 172.21.15.98:0/52613965 shutdown_connections 2024-09-19T08:10:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.939+0000 7f8365464640 1 -- 172.21.15.98:0/52613965 wait complete. 2024-09-19T08:10:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.939+0000 7f8365464640 1 Processor -- start 2024-09-19T08:10:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f8365464640 1 -- start start 2024-09-19T08:10:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f8365464640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8360100a80 0x7f83601992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f8365464640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8360101420 0x7f8360199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:01.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f8365464640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8360106a80 0x7f83601a4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:01.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f8365464640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8360110e80 con 0x7f8360101420 2024-09-19T08:10:01.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f8365464640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8360110d00 con 0x7f8360100a80 2024-09-19T08:10:01.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f8365464640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8360111000 con 0x7f8360106a80 2024-09-19T08:10:01.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f835e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8360101420 0x7f8360199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:01.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f835e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8360101420 0x7f8360199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34656/0 (socket says 172.21.15.98:34656) 2024-09-19T08:10:01.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f835e7fc640 1 -- 172.21.15.98:0/424679054 learned_addr learned my addr 172.21.15.98:0/424679054 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:01.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f835f7fe640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8360106a80 0x7f83601a4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:01.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f835effd640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8360100a80 0x7f83601992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:01.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f835e7fc640 1 -- 172.21.15.98:0/424679054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8360106a80 msgr2=0x7f83601a4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:01.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f835e7fc640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8360106a80 0x7f83601a4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:01.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f835e7fc640 1 -- 172.21.15.98:0/424679054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8360100a80 msgr2=0x7f83601992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f835e7fc640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8360100a80 0x7f83601992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f835e7fc640 1 -- 172.21.15.98:0/424679054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83601a48b0 con 0x7f8360101420 2024-09-19T08:10:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.940+0000 7f835f7fe640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8360106a80 0x7f83601a4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.941+0000 7f835e7fc640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8360101420 0x7f8360199830 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f8348009a50 tx=0x7f8348004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.941+0000 7f8357fff640 1 -- 172.21.15.98:0/424679054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834803f030 con 0x7f8360101420 2024-09-19T08:10:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.941+0000 7f8365464640 1 -- 172.21.15.98:0/424679054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83601a4a80 con 0x7f8360101420 2024-09-19T08:10:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.941+0000 7f8357fff640 1 -- 172.21.15.98:0/424679054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8348003ce0 con 0x7f8360101420 2024-09-19T08:10:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.941+0000 7f8357fff640 1 -- 172.21.15.98:0/424679054 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8348035070 con 0x7f8360101420 2024-09-19T08:10:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.941+0000 7f8365464640 1 -- 172.21.15.98:0/424679054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83601a4ee0 con 0x7f8360101420 2024-09-19T08:10:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.942+0000 7f8365464640 1 -- 172.21.15.98:0/424679054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8360100ee0 con 0x7f8360101420 2024-09-19T08:10:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.943+0000 7f8357fff640 1 -- 172.21.15.98:0/424679054 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f8348042030 con 0x7f8360101420 2024-09-19T08:10:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.943+0000 7f8357fff640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f834c0a33b0 0x7f834c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.944+0000 7f835effd640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f834c0a33b0 0x7f834c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.944+0000 7f8357fff640 1 -- 172.21.15.98:0/424679054 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f83480f07f0 con 0x7f8360101420 2024-09-19T08:10:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.944+0000 7f835effd640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f834c0a33b0 0x7f834c0a5870 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f835800a300 tx=0x7f835800a290 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:01.947+0000 7f8357fff640 1 -- 172.21.15.98:0/424679054 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f83480bd1a0 con 0x7f8360101420 2024-09-19T08:10:02.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:02.150+0000 7f8365464640 1 -- 172.21.15.98:0/424679054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8360109010 con 0x7f8360101420 2024-09-19T08:10:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:02.151+0000 7f8357fff640 1 -- 172.21.15.98:0/424679054 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11983 (secure 0 0 0) 0x7f83480bcaa0 con 0x7f8360101420 2024-09-19T08:10:02.151 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:10:02.152 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"btime":"2024-09-19T08:08:26:985720+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:08:26.555658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:10:02.152 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2024-09-19T08:10:02.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:02.153+0000 7f8365464640 1 -- 172.21.15.98:0/424679054 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f834c0a33b0 msgr2=0x7f834c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:02.153+0000 7f8365464640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f834c0a33b0 0x7f834c0a5870 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f835800a300 tx=0x7f835800a290 comp rx=0 tx=0).stop 2024-09-19T08:10:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:02.153+0000 7f8365464640 1 -- 172.21.15.98:0/424679054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8360101420 msgr2=0x7f8360199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:02.153+0000 7f8365464640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8360101420 0x7f8360199830 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f8348009a50 tx=0x7f8348004270 comp rx=0 tx=0).stop 2024-09-19T08:10:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:02.153+0000 7f8365464640 1 -- 172.21.15.98:0/424679054 shutdown_connections 2024-09-19T08:10:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:02.153+0000 7f8365464640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f834c0a33b0 0x7f834c0a5870 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:02.153+0000 7f8365464640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8360106a80 0x7f83601a4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:02.154+0000 7f8365464640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8360101420 0x7f8360199830 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:02.154+0000 7f8365464640 1 --2- 172.21.15.98:0/424679054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8360100a80 0x7f83601992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:02.154+0000 7f8365464640 1 -- 172.21.15.98:0/424679054 >> 172.21.15.98:0/424679054 conn(0x7f83600fc7a0 msgr2=0x7f83600fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:02.154+0000 7f8365464640 1 -- 172.21.15.98:0/424679054 shutdown_connections 2024-09-19T08:10:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:02.154+0000 7f8365464640 1 -- 172.21.15.98:0/424679054 wait complete. 2024-09-19T08:10:02.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:10:03.060 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:03.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:02 smithi162 ceph-mon[35260]: pgmap v213: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:03.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:02 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/424679054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:10:03.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:02 smithi139 ceph-mon[34874]: pgmap v213: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:03.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:02 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/424679054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:10:03.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:02 smithi098 ceph-mon[31333]: pgmap v213: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:03.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:02 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/424679054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:10:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.482+0000 7ff687849640 1 -- 172.21.15.98:0/1100470821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6800745a0 msgr2=0x7ff680074980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.482+0000 7ff687849640 1 --2- 172.21.15.98:0/1100470821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6800745a0 0x7ff680074980 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7ff670009ca0 tx=0x7ff67002a8c0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.482+0000 7ff687849640 1 -- 172.21.15.98:0/1100470821 shutdown_connections 2024-09-19T08:10:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.482+0000 7ff687849640 1 --2- 172.21.15.98:0/1100470821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff68010e470 0x7ff6801108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.482+0000 7ff687849640 1 --2- 172.21.15.98:0/1100470821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff680074f50 0x7ff68010df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.482+0000 7ff687849640 1 --2- 172.21.15.98:0/1100470821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6800745a0 0x7ff680074980 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.482+0000 7ff687849640 1 -- 172.21.15.98:0/1100470821 >> 172.21.15.98:0/1100470821 conn(0x7ff6800fd8d0 msgr2=0x7ff6800ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.482+0000 7ff687849640 1 -- 172.21.15.98:0/1100470821 shutdown_connections 2024-09-19T08:10:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.482+0000 7ff687849640 1 -- 172.21.15.98:0/1100470821 wait complete. 2024-09-19T08:10:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.482+0000 7ff687849640 1 Processor -- start 2024-09-19T08:10:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.482+0000 7ff687849640 1 -- start start 2024-09-19T08:10:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff687849640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6800745a0 0x7ff68019a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff687849640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff680074f50 0x7ff68019ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff687849640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff68010e470 0x7ff68019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff687849640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6801141e0 con 0x7ff680074f50 2024-09-19T08:10:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff687849640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff680114060 con 0x7ff6800745a0 2024-09-19T08:10:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff687849640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff680114360 con 0x7ff68010e470 2024-09-19T08:10:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff684de4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff680074f50 0x7ff68019ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff684de4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff680074f50 0x7ff68019ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34680/0 (socket says 172.21.15.98:34680) 2024-09-19T08:10:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff684de4640 1 -- 172.21.15.98:0/2005444585 learned_addr learned my addr 172.21.15.98:0/2005444585 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff685de6640 1 --2- 172.21.15.98:0/2005444585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff68010e470 0x7ff68019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff6855e5640 1 --2- 172.21.15.98:0/2005444585 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6800745a0 0x7ff68019a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff684de4640 1 -- 172.21.15.98:0/2005444585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff68010e470 msgr2=0x7ff68019eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff684de4640 1 --2- 172.21.15.98:0/2005444585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff68010e470 0x7ff68019eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff684de4640 1 -- 172.21.15.98:0/2005444585 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6800745a0 msgr2=0x7ff68019a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff684de4640 1 --2- 172.21.15.98:0/2005444585 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6800745a0 0x7ff68019a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff684de4640 1 -- 172.21.15.98:0/2005444585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff68019f590 con 0x7ff680074f50 2024-09-19T08:10:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff684de4640 1 --2- 172.21.15.98:0/2005444585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff680074f50 0x7ff68019ab20 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7ff67400c890 tx=0x7ff67400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.483+0000 7ff67e7fc640 1 -- 172.21.15.98:0/2005444585 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff674004220 con 0x7ff680074f50 2024-09-19T08:10:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.484+0000 7ff67e7fc640 1 -- 172.21.15.98:0/2005444585 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6740043c0 con 0x7ff680074f50 2024-09-19T08:10:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.484+0000 7ff687849640 1 -- 172.21.15.98:0/2005444585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff68019f880 con 0x7ff680074f50 2024-09-19T08:10:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.484+0000 7ff67e7fc640 1 -- 172.21.15.98:0/2005444585 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6740049e0 con 0x7ff680074f50 2024-09-19T08:10:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.484+0000 7ff687849640 1 -- 172.21.15.98:0/2005444585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6801a7160 con 0x7ff680074f50 2024-09-19T08:10:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.486+0000 7ff67e7fc640 1 -- 172.21.15.98:0/2005444585 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7ff6740079b0 con 0x7ff680074f50 2024-09-19T08:10:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.486+0000 7ff689c22740 1 Processor -- start 2024-09-19T08:10:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.486+0000 7ff689c22740 1 -- start start 2024-09-19T08:10:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff689c22740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 0x558594d80090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff689c22740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d80a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff689c22740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 0x558594d8d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff689c22740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558594ce62f0 con 0x558594d805d0 2024-09-19T08:10:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff689c22740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558594c8dcf0 con 0x558594d8b370 2024-09-19T08:10:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff689c22740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558594ce22b0 con 0x558594d81b40 2024-09-19T08:10:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff6855e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d80a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff6855e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d80a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34696/0 (socket says 172.21.15.98:34696) 2024-09-19T08:10:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff685de6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 0x558594d80090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff6855e5640 1 -- 172.21.15.98:0/1036326455 learned_addr learned my addr 172.21.15.98:0/1036326455 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff67e7fc640 1 --2- 172.21.15.98:0/2005444585 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff6680a33b0 0x7ff6680a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff684de4640 1 --2- 172.21.15.98:0/1036326455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 0x558594d8d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff685de6640 1 --2- 172.21.15.98:0/2005444585 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff6680a33b0 0x7ff6680a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff6855e5640 1 -- 172.21.15.98:0/1036326455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 msgr2=0x558594d80090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff6855e5640 1 --2- 172.21.15.98:0/1036326455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 0x558594d80090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff6855e5640 1 -- 172.21.15.98:0/1036326455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 msgr2=0x558594d8d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff6855e5640 1 --2- 172.21.15.98:0/1036326455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 0x558594d8d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff6855e5640 1 -- 172.21.15.98:0/1036326455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558594c723a0 con 0x558594d805d0 2024-09-19T08:10:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff684de4640 1 --2- 172.21.15.98:0/1036326455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 0x558594d8d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff67e7fc640 1 -- 172.21.15.98:0/2005444585 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff6740cfca0 con 0x7ff680074f50 2024-09-19T08:10:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.487+0000 7ff6855e5640 1 --2- 172.21.15.98:0/1036326455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d80a10 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7ff670009ca0 tx=0x7ff67002f450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b59732e6de805e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.488+0000 7ff685de6640 1 --2- 172.21.15.98:0/2005444585 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff6680a33b0 0x7ff6680a5870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff6780136c0 tx=0x7ff678004730 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.488+0000 7ff6677fe640 1 -- 172.21.15.98:0/1036326455 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff670049070 con 0x558594d805d0 2024-09-19T08:10:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.488+0000 7ff6677fe640 1 -- 172.21.15.98:0/1036326455 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff670036040 con 0x558594d805d0 2024-09-19T08:10:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.488+0000 7ff6677fe640 1 -- 172.21.15.98:0/1036326455 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67002fdc0 con 0x558594d805d0 2024-09-19T08:10:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.488+0000 7ff689c22740 1 -- 172.21.15.98:0/1036326455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 msgr2=0x558594d80a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.488+0000 7ff689c22740 1 --2- 172.21.15.98:0/1036326455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d80a10 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7ff670009ca0 tx=0x7ff67002f450 comp rx=0 tx=0).stop 2024-09-19T08:10:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.488+0000 7ff689c22740 1 -- 172.21.15.98:0/1036326455 shutdown_connections 2024-09-19T08:10:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.488+0000 7ff689c22740 1 --2- 172.21.15.98:0/1036326455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 0x558594d8d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.488+0000 7ff689c22740 1 --2- 172.21.15.98:0/1036326455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d80a10 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.488+0000 7ff689c22740 1 --2- 172.21.15.98:0/1036326455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 0x558594d80090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.488+0000 7ff689c22740 1 -- 172.21.15.98:0/1036326455 >> 172.21.15.98:0/1036326455 conn(0x558594d49310 msgr2=0x558594d7cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.488+0000 7ff689c22740 1 -- 172.21.15.98:0/1036326455 shutdown_connections 2024-09-19T08:10:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.488+0000 7ff689c22740 1 -- 172.21.15.98:0/1036326455 wait complete. 2024-09-19T08:10:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.489+0000 7ff689c22740 1 -- start start 2024-09-19T08:10:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.489+0000 7ff689c22740 1 Processor -- start 2024-09-19T08:10:03.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.489+0000 7ff689c22740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d805d0 0x558594d8ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.489+0000 7ff689c22740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d81b40 0x558594d88010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.489+0000 7ff689c22740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d8b370 0x558594d88600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.489+0000 7ff689c22740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558594d8de90 con 0x558594d81b40 2024-09-19T08:10:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.489+0000 7ff689c22740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558594ca1060 con 0x558594d805d0 2024-09-19T08:10:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.489+0000 7ff689c22740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558594d8e010 con 0x558594d8b370 2024-09-19T08:10:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.489+0000 7ff689c22740 10 client.0 _ll_register_callbacks cb 0x5585949f48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.489+0000 7ff684de4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d81b40 0x558594d88010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff684de4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d81b40 0x558594d88010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34708/0 (socket says 172.21.15.98:34708) 2024-09-19T08:10:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff684de4640 1 -- 172.21.15.98:0/801995773 learned_addr learned my addr 172.21.15.98:0/801995773 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff6855e5640 1 --2- 172.21.15.98:0/801995773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d805d0 0x558594d8ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff685de6640 1 --2- 172.21.15.98:0/801995773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d8b370 0x558594d88600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff684de4640 1 -- 172.21.15.98:0/801995773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d8b370 msgr2=0x558594d88600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff684de4640 1 --2- 172.21.15.98:0/801995773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d8b370 0x558594d88600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff684de4640 1 -- 172.21.15.98:0/801995773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d805d0 msgr2=0x558594d8ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff684de4640 1 --2- 172.21.15.98:0/801995773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d805d0 0x558594d8ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff684de4640 1 -- 172.21.15.98:0/801995773 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558594a573c0 con 0x558594d81b40 2024-09-19T08:10:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff685de6640 1 --2- 172.21.15.98:0/801995773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d8b370 0x558594d88600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff684de4640 1 --2- 172.21.15.98:0/801995773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d81b40 0x558594d88010 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7ff6740cd480 tx=0x7ff674005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a9095aaea866aa server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff664ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff664ff9640 1 -- 172.21.15.98:0/801995773 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67401d070 con 0x558594d81b40 2024-09-19T08:10:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff664ff9640 1 -- 172.21.15.98:0/801995773 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff674096bd0 con 0x558594d81b40 2024-09-19T08:10:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff689c22740 1 -- 172.21.15.98:0/801995773 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff670036040 con 0x558594d81b40 2024-09-19T08:10:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff664ff9640 1 -- 172.21.15.98:0/801995773 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff6740bc070 con 0x558594d81b40 2024-09-19T08:10:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff689c22740 10 client.17130 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff689c22740 1 -- 172.21.15.98:0/801995773 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558594d99990 con 0x558594d81b40 2024-09-19T08:10:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.490+0000 7ff664ff9640 1 -- 172.21.15.98:0/801995773 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff670036040 con 0x558594d81b40 2024-09-19T08:10:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.491+0000 7ff664ff9640 1 -- 172.21.15.98:0/801995773 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff6740a4790 con 0x558594d81b40 2024-09-19T08:10:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.491+0000 7ff689c22740 10 client.17130 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.491+0000 7ff689c22740 10 client.17130 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:10:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.491+0000 7ff689c22740 20 client.17130 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.491+0000 7ff689c22740 1 --2- 172.21.15.98:0/801995773 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x558594d9aea0 0x558594d9d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.491+0000 7ff689c22740 4 client.17130 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.491+0000 7ff689c22740 1 -- 172.21.15.98:0/801995773 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558594cf3860 con 0x558594d9aea0 2024-09-19T08:10:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.491+0000 7ff6855e5640 1 --2- 172.21.15.98:0/801995773 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x558594d9aea0 0x558594d9d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.492+0000 7ff664ff9640 10 client.17130 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:10:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.491+0000 7ff6855e5640 1 --2- 172.21.15.98:0/801995773 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x558594d9aea0 0x558594d9d290 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6497f7a357a7ad8b server_cookie=1253e3f3b9e614fc in_seq=0 out_seq=0 2024-09-19T08:10:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.492+0000 7ff664ff9640 1 -- 172.21.15.98:0/801995773 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558594cf3860 con 0x558594d9aea0 2024-09-19T08:10:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.492+0000 7ff664ff9640 10 client.17130 handle_command_reply: tid=0 2024-09-19T08:10:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.492+0000 7ff689c22740 1 client.17130 shutdown 2024-09-19T08:10:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.492+0000 7ff689c22740 1 -- 172.21.15.98:0/801995773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d81b40 msgr2=0x558594d88010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.492+0000 7ff689c22740 1 --2- 172.21.15.98:0/801995773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d81b40 0x558594d88010 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7ff6740cd480 tx=0x7ff674005300 comp rx=0 tx=0).stop 2024-09-19T08:10:03.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.492+0000 7ff689c22740 1 -- 172.21.15.98:0/801995773 shutdown_connections 2024-09-19T08:10:03.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.492+0000 7ff689c22740 1 --2- 172.21.15.98:0/801995773 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x558594d9aea0 0x558594d9d290 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.492+0000 7ff689c22740 1 --2- 172.21.15.98:0/801995773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d8b370 0x558594d88600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.492+0000 7ff689c22740 1 --2- 172.21.15.98:0/801995773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d81b40 0x558594d88010 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.492+0000 7ff689c22740 1 --2- 172.21.15.98:0/801995773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d805d0 0x558594d8ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.492+0000 7ff689c22740 1 -- 172.21.15.98:0/801995773 >> 172.21.15.98:0/801995773 conn(0x558594d49310 msgr2=0x558594d82960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:03.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.493+0000 7ff664ff9640 0 client.17130 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:10:03.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.493+0000 7ff689c22740 1 -- 172.21.15.98:0/801995773 shutdown_connections 2024-09-19T08:10:03.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.493+0000 7ff689c22740 1 -- 172.21.15.98:0/801995773 wait complete. 2024-09-19T08:10:03.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.493+0000 7ff689c22740 20 client.17130 trim_cache size 0 max 16384 2024-09-19T08:10:03.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.505+0000 7ff689c22740 1 Processor -- start 2024-09-19T08:10:03.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.505+0000 7ff689c22740 1 -- start start 2024-09-19T08:10:03.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff689c22740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d97130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff689c22740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 0x558594d89f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff689c22740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 0x558594da2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff689c22740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6801af8c0 con 0x558594d805d0 2024-09-19T08:10:03.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff689c22740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff6801b1400 con 0x558594d8b370 2024-09-19T08:10:03.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff689c22740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff6801aff90 con 0x558594d81b40 2024-09-19T08:10:03.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff6855e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d97130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff6855e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d97130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34724/0 (socket says 172.21.15.98:34724) 2024-09-19T08:10:03.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff6855e5640 1 -- 172.21.15.98:0/2507149525 learned_addr learned my addr 172.21.15.98:0/2507149525 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:03.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff6855e5640 1 -- 172.21.15.98:0/2507149525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 msgr2=0x558594d89f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:03.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff684de4640 1 --2- 172.21.15.98:0/2507149525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 0x558594d89f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff685de6640 1 --2- 172.21.15.98:0/2507149525 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 0x558594da2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff6855e5640 1 --2- 172.21.15.98:0/2507149525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 0x558594d89f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff6855e5640 1 -- 172.21.15.98:0/2507149525 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 msgr2=0x558594da2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:03.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff6855e5640 1 --2- 172.21.15.98:0/2507149525 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 0x558594da2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff6855e5640 1 -- 172.21.15.98:0/2507149525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558594d77a30 con 0x558594d805d0 2024-09-19T08:10:03.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff6855e5640 1 --2- 172.21.15.98:0/2507149525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d97130 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7ff670009ca0 tx=0x7ff67002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d840986590b56c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:03.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff666ffd640 1 -- 172.21.15.98:0/2507149525 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff670049040 con 0x558594d805d0 2024-09-19T08:10:03.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff666ffd640 1 -- 172.21.15.98:0/2507149525 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff670055070 con 0x558594d805d0 2024-09-19T08:10:03.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.506+0000 7ff666ffd640 1 -- 172.21.15.98:0/2507149525 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67002f8f0 con 0x558594d805d0 2024-09-19T08:10:03.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.507+0000 7ff689c22740 1 -- 172.21.15.98:0/2507149525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 msgr2=0x558594d97130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:03.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.507+0000 7ff689c22740 1 --2- 172.21.15.98:0/2507149525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d97130 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7ff670009ca0 tx=0x7ff67002a8c0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.507+0000 7ff689c22740 1 -- 172.21.15.98:0/2507149525 shutdown_connections 2024-09-19T08:10:03.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.507+0000 7ff689c22740 1 --2- 172.21.15.98:0/2507149525 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 0x558594da2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.507+0000 7ff689c22740 1 --2- 172.21.15.98:0/2507149525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 0x558594d89f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.507+0000 7ff689c22740 1 --2- 172.21.15.98:0/2507149525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d97130 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.507+0000 7ff689c22740 1 -- 172.21.15.98:0/2507149525 >> 172.21.15.98:0/2507149525 conn(0x558594d49310 msgr2=0x558594d89870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:03.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.507+0000 7ff689c22740 1 -- 172.21.15.98:0/2507149525 shutdown_connections 2024-09-19T08:10:03.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.507+0000 7ff689c22740 1 -- 172.21.15.98:0/2507149525 wait complete. 2024-09-19T08:10:03.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.507+0000 7ff689c22740 1 -- start start 2024-09-19T08:10:03.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.507+0000 7ff689c22740 1 Processor -- start 2024-09-19T08:10:03.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff689c22740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d932a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff689c22740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 0x558594d937e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff689c22740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 0x558594d90270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff689c22740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6801b0d30 con 0x558594d805d0 2024-09-19T08:10:03.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff689c22740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff6801b0660 con 0x558594d8b370 2024-09-19T08:10:03.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff689c22740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff6801ad8a0 con 0x558594d81b40 2024-09-19T08:10:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff689c22740 10 client.0 _ll_register_callbacks cb 0x7ff6740058c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff6855e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d932a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff6855e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d932a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34730/0 (socket says 172.21.15.98:34730) 2024-09-19T08:10:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff6855e5640 1 -- 172.21.15.98:0/2924730023 learned_addr learned my addr 172.21.15.98:0/2924730023 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff684de4640 1 --2- 172.21.15.98:0/2924730023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 0x558594d937e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff685de6640 1 --2- 172.21.15.98:0/2924730023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 0x558594d90270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff6855e5640 1 -- 172.21.15.98:0/2924730023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 msgr2=0x558594d937e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff6855e5640 1 --2- 172.21.15.98:0/2924730023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 0x558594d937e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff6855e5640 1 -- 172.21.15.98:0/2924730023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 msgr2=0x558594d90270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff6855e5640 1 --2- 172.21.15.98:0/2924730023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 0x558594d90270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff6855e5640 1 -- 172.21.15.98:0/2924730023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558594d907b0 con 0x558594d805d0 2024-09-19T08:10:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff685de6640 1 --2- 172.21.15.98:0/2924730023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 0x558594d90270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff6855e5640 1 --2- 172.21.15.98:0/2924730023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d932a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7ff670009860 tx=0x7ff67002f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ea024428cf1614 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:03.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.508+0000 7ff664ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:03.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.509+0000 7ff664ff9640 1 -- 172.21.15.98:0/2924730023 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff670049070 con 0x558594d805d0 2024-09-19T08:10:03.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.509+0000 7ff664ff9640 1 -- 172.21.15.98:0/2924730023 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7ff670031870 con 0x558594d805d0 2024-09-19T08:10:03.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.509+0000 7ff689c22740 1 -- 172.21.15.98:0/2924730023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff670055070 con 0x558594d805d0 2024-09-19T08:10:03.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.509+0000 7ff664ff9640 1 -- 172.21.15.98:0/2924730023 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff670003970 con 0x558594d805d0 2024-09-19T08:10:03.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.509+0000 7ff689c22740 10 client.17136 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:03.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.509+0000 7ff689c22740 1 -- 172.21.15.98:0/2924730023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558594e06170 con 0x558594d805d0 2024-09-19T08:10:03.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.509+0000 7ff664ff9640 1 -- 172.21.15.98:0/2924730023 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff670055070 con 0x558594d805d0 2024-09-19T08:10:03.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.509+0000 7ff664ff9640 1 -- 172.21.15.98:0/2924730023 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff6700040d0 con 0x558594d805d0 2024-09-19T08:10:03.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.509+0000 7ff689c22740 10 client.17136 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:03.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.509+0000 7ff689c22740 10 client.17136 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:10:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.509+0000 7ff689c22740 20 client.17136 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.509+0000 7ff689c22740 1 --2- 172.21.15.98:0/2924730023 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x558594e06550 0x558594e08940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.509+0000 7ff689c22740 4 client.17136 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:10:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.509+0000 7ff689c22740 1 -- 172.21.15.98:0/2924730023 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558594cedd70 con 0x558594e06550 2024-09-19T08:10:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff684de4640 1 --2- 172.21.15.98:0/2924730023 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x558594e06550 0x558594e08940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff664ff9640 10 client.17136 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:10:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff684de4640 1 --2- 172.21.15.98:0/2924730023 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x558594e06550 0x558594e08940 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ae923fc64cebc323 server_cookie=97e537e2b3ddb7f8 in_seq=0 out_seq=0 2024-09-19T08:10:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff664ff9640 1 -- 172.21.15.98:0/2924730023 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558594cedd70 con 0x558594e06550 2024-09-19T08:10:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff664ff9640 10 client.17136 handle_command_reply: tid=0 2024-09-19T08:10:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff689c22740 1 client.17136 shutdown 2024-09-19T08:10:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff689c22740 1 -- 172.21.15.98:0/2924730023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 msgr2=0x558594d932a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff689c22740 1 --2- 172.21.15.98:0/2924730023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d932a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7ff670009860 tx=0x7ff67002f5d0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff689c22740 1 -- 172.21.15.98:0/2924730023 shutdown_connections 2024-09-19T08:10:03.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff689c22740 1 --2- 172.21.15.98:0/2924730023 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x558594e06550 0x558594e08940 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff689c22740 1 --2- 172.21.15.98:0/2924730023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558594d8b370 0x558594d90270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff689c22740 1 --2- 172.21.15.98:0/2924730023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558594d81b40 0x558594d937e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff689c22740 1 --2- 172.21.15.98:0/2924730023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558594d805d0 0x558594d932a0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff689c22740 1 -- 172.21.15.98:0/2924730023 >> 172.21.15.98:0/2924730023 conn(0x558594d49310 msgr2=0x558594d8a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:03.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.510+0000 7ff664ff9640 0 client.17136 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:10:03.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff689c22740 1 -- 172.21.15.98:0/2924730023 shutdown_connections 2024-09-19T08:10:03.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff689c22740 1 -- 172.21.15.98:0/2924730023 wait complete. 2024-09-19T08:10:03.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff689c22740 20 client.17136 trim_cache size 0 max 16384 2024-09-19T08:10:03.518 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:10:03.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff687849640 1 -- 172.21.15.98:0/2005444585 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff6680a33b0 msgr2=0x7ff6680a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff687849640 1 --2- 172.21.15.98:0/2005444585 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff6680a33b0 0x7ff6680a5870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff6780136c0 tx=0x7ff678004730 comp rx=0 tx=0).stop 2024-09-19T08:10:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff687849640 1 -- 172.21.15.98:0/2005444585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff680074f50 msgr2=0x7ff68019ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff687849640 1 --2- 172.21.15.98:0/2005444585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff680074f50 0x7ff68019ab20 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7ff67400c890 tx=0x7ff67400cd50 comp rx=0 tx=0).stop 2024-09-19T08:10:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff687849640 1 -- 172.21.15.98:0/2005444585 shutdown_connections 2024-09-19T08:10:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff687849640 1 --2- 172.21.15.98:0/2005444585 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff6680a33b0 0x7ff6680a5870 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff687849640 1 --2- 172.21.15.98:0/2005444585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff68010e470 0x7ff68019eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff687849640 1 --2- 172.21.15.98:0/2005444585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff680074f50 0x7ff68019ab20 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff687849640 1 --2- 172.21.15.98:0/2005444585 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6800745a0 0x7ff68019a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff687849640 1 -- 172.21.15.98:0/2005444585 >> 172.21.15.98:0/2005444585 conn(0x7ff6800fd8d0 msgr2=0x7ff6800fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff687849640 1 -- 172.21.15.98:0/2005444585 shutdown_connections 2024-09-19T08:10:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:03.511+0000 7ff687849640 1 -- 172.21.15.98:0/2005444585 wait complete. 2024-09-19T08:10:04.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:10:04.473 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:04.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.927+0000 7f14198fa640 1 -- 172.21.15.98:0/2153971563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f141410c170 msgr2=0x7f141410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:04.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.927+0000 7f14198fa640 1 --2- 172.21.15.98:0/2153971563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f141410c170 0x7f141410e610 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f140c009ca0 tx=0x7f140c02a950 comp rx=0 tx=0).stop 2024-09-19T08:10:04.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.927+0000 7f14198fa640 1 -- 172.21.15.98:0/2153971563 shutdown_connections 2024-09-19T08:10:04.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.927+0000 7f14198fa640 1 --2- 172.21.15.98:0/2153971563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f141410c170 0x7f141410e610 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.927+0000 7f14198fa640 1 --2- 172.21.15.98:0/2153971563 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1414103660 0x7f141410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.927+0000 7f14198fa640 1 --2- 172.21.15.98:0/2153971563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1414102d40 0x7f1414103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.927+0000 7f14198fa640 1 -- 172.21.15.98:0/2153971563 >> 172.21.15.98:0/2153971563 conn(0x7f14140fc7a0 msgr2=0x7f14140febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:04.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.927+0000 7f14198fa640 1 -- 172.21.15.98:0/2153971563 shutdown_connections 2024-09-19T08:10:04.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.927+0000 7f14198fa640 1 -- 172.21.15.98:0/2153971563 wait complete. 2024-09-19T08:10:04.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.928+0000 7f14198fa640 1 Processor -- start 2024-09-19T08:10:04.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.928+0000 7f14198fa640 1 -- start start 2024-09-19T08:10:04.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.928+0000 7f14198fa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1414102d40 0x7f14141984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.928+0000 7f14198fa640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1414103660 0x7f14141989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.928+0000 7f14198fa640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141410c170 0x7f141419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.928+0000 7f14198fa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1414111fb0 con 0x7f1414102d40 2024-09-19T08:10:04.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.928+0000 7f14198fa640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1414111e30 con 0x7f141410c170 2024-09-19T08:10:04.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.928+0000 7f14198fa640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1414112130 con 0x7f1414103660 2024-09-19T08:10:04.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.928+0000 7f1412ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1414102d40 0x7f14141984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.928+0000 7f1412ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1414102d40 0x7f14141984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34738/0 (socket says 172.21.15.98:34738) 2024-09-19T08:10:04.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.928+0000 7f1412ffd640 1 -- 172.21.15.98:0/3420379556 learned_addr learned my addr 172.21.15.98:0/3420379556 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:04.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.928+0000 7f14127fc640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1414103660 0x7f14141989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.928+0000 7f14137fe640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141410c170 0x7f141419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.929+0000 7f1412ffd640 1 -- 172.21.15.98:0/3420379556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1414103660 msgr2=0x7f14141989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:04.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.929+0000 7f1412ffd640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1414103660 0x7f14141989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.929+0000 7f1412ffd640 1 -- 172.21.15.98:0/3420379556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141410c170 msgr2=0x7f141419cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:04.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.929+0000 7f1412ffd640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141410c170 0x7f141419cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.929+0000 7f1412ffd640 1 -- 172.21.15.98:0/3420379556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141419d460 con 0x7f1414102d40 2024-09-19T08:10:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.929+0000 7f14137fe640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141410c170 0x7f141419cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.929+0000 7f1412ffd640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1414102d40 0x7f14141984b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f140400b4a0 tx=0x7f140400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.929+0000 7f14188f8640 1 -- 172.21.15.98:0/3420379556 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1404014070 con 0x7f1414102d40 2024-09-19T08:10:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.929+0000 7f14188f8640 1 -- 172.21.15.98:0/3420379556 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1404010af0 con 0x7f1414102d40 2024-09-19T08:10:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.929+0000 7f14188f8640 1 -- 172.21.15.98:0/3420379556 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1404017d10 con 0x7f1414102d40 2024-09-19T08:10:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.929+0000 7f14198fa640 1 -- 172.21.15.98:0/3420379556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141419d750 con 0x7f1414102d40 2024-09-19T08:10:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.929+0000 7f14198fa640 1 -- 172.21.15.98:0/3420379556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14141a5030 con 0x7f1414102d40 2024-09-19T08:10:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.931+0000 7f14188f8640 1 -- 172.21.15.98:0/3420379556 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f1404017070 con 0x7f1414102d40 2024-09-19T08:10:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.931+0000 7f141bcd3740 1 Processor -- start 2024-09-19T08:10:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f141bcd3740 1 -- start start 2024-09-19T08:10:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f14188f8640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f13fc0a3430 0x7f13fc0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f141bcd3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 0x5622633cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f141bcd3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633cda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f141bcd3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633d8370 0x5622633da760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f141bcd3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5622633332f0 con 0x5622633cd5d0 2024-09-19T08:10:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f141bcd3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5622632dacf0 con 0x5622633d8370 2024-09-19T08:10:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f141bcd3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56226332f2b0 con 0x5622633ceb40 2024-09-19T08:10:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f14137fe640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f13fc0a3430 0x7f13fc0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f14127fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f14127fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34746/0 (socket says 172.21.15.98:34746) 2024-09-19T08:10:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f14127fc640 1 -- 172.21.15.98:0/3667985083 learned_addr learned my addr 172.21.15.98:0/3667985083 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f14188f8640 1 -- 172.21.15.98:0/3420379556 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f14040ceed0 con 0x7f1414102d40 2024-09-19T08:10:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f1412ffd640 1 --2- 172.21.15.98:0/3667985083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 0x5622633cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f14137fe640 1 --2- 172.21.15.98:0/3667985083 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633d8370 0x5622633da760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f14127fc640 1 -- 172.21.15.98:0/3667985083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 msgr2=0x5622633cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f14127fc640 1 --2- 172.21.15.98:0/3667985083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 0x5622633cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f14127fc640 1 -- 172.21.15.98:0/3667985083 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633d8370 msgr2=0x5622633da760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f14127fc640 1 --2- 172.21.15.98:0/3667985083 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633d8370 0x5622633da760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f14127fc640 1 -- 172.21.15.98:0/3667985083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622632bf3a0 con 0x5622633cd5d0 2024-09-19T08:10:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.932+0000 7f14137fe640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f13fc0a3430 0x7f13fc0a58f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f140c005cc0 tx=0x7f140c02fc00 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.933+0000 7f14127fc640 1 --2- 172.21.15.98:0/3667985083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633cda10 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f140800abe0 tx=0x7f140800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74f18d176932a47 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.933+0000 7f13f9ffb640 1 -- 172.21.15.98:0/3667985083 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140801d070 con 0x5622633cd5d0 2024-09-19T08:10:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.933+0000 7f13f9ffb640 1 -- 172.21.15.98:0/3667985083 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1408018b20 con 0x5622633cd5d0 2024-09-19T08:10:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.933+0000 7f13f9ffb640 1 -- 172.21.15.98:0/3667985083 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1408010ab0 con 0x5622633cd5d0 2024-09-19T08:10:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.933+0000 7f141bcd3740 1 -- 172.21.15.98:0/3667985083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 msgr2=0x5622633cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.933+0000 7f141bcd3740 1 --2- 172.21.15.98:0/3667985083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633cda10 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f140800abe0 tx=0x7f140800f790 comp rx=0 tx=0).stop 2024-09-19T08:10:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.933+0000 7f141bcd3740 1 -- 172.21.15.98:0/3667985083 shutdown_connections 2024-09-19T08:10:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.933+0000 7f141bcd3740 1 --2- 172.21.15.98:0/3667985083 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633d8370 0x5622633da760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.933+0000 7f141bcd3740 1 --2- 172.21.15.98:0/3667985083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633cda10 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.933+0000 7f141bcd3740 1 --2- 172.21.15.98:0/3667985083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 0x5622633cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.933+0000 7f141bcd3740 1 -- 172.21.15.98:0/3667985083 >> 172.21.15.98:0/3667985083 conn(0x562263396310 msgr2=0x5622633c9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.933+0000 7f141bcd3740 1 -- 172.21.15.98:0/3667985083 shutdown_connections 2024-09-19T08:10:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.933+0000 7f141bcd3740 1 -- 172.21.15.98:0/3667985083 wait complete. 2024-09-19T08:10:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.934+0000 7f141bcd3740 1 -- start start 2024-09-19T08:10:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.934+0000 7f141bcd3740 1 Processor -- start 2024-09-19T08:10:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.934+0000 7f141bcd3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633d7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.934+0000 7f141bcd3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633ceb40 0x5622633d5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.934+0000 7f141bcd3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633d8370 0x5622633d55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.934+0000 7f141bcd3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5622633db070 con 0x5622633cd5d0 2024-09-19T08:10:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.934+0000 7f141bcd3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5622632ee060 con 0x5622633ceb40 2024-09-19T08:10:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.934+0000 7f141bcd3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5622633db1f0 con 0x5622633d8370 2024-09-19T08:10:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.934+0000 7f141bcd3740 10 client.0 _ll_register_callbacks cb 0x5622630418c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.934+0000 7f1412ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633d7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.934+0000 7f1412ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633d7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34748/0 (socket says 172.21.15.98:34748) 2024-09-19T08:10:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.934+0000 7f1412ffd640 1 -- 172.21.15.98:0/2389528392 learned_addr learned my addr 172.21.15.98:0/2389528392 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.934+0000 7f14137fe640 1 --2- 172.21.15.98:0/2389528392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633d8370 0x5622633d55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f14127fc640 1 --2- 172.21.15.98:0/2389528392 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633ceb40 0x5622633d5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f1412ffd640 1 -- 172.21.15.98:0/2389528392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633d8370 msgr2=0x5622633d55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f1412ffd640 1 --2- 172.21.15.98:0/2389528392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633d8370 0x5622633d55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f1412ffd640 1 -- 172.21.15.98:0/2389528392 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633ceb40 msgr2=0x5622633d5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f1412ffd640 1 --2- 172.21.15.98:0/2389528392 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633ceb40 0x5622633d5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f1412ffd640 1 -- 172.21.15.98:0/2389528392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622630a43c0 con 0x5622633cd5d0 2024-09-19T08:10:04.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f1412ffd640 1 --2- 172.21.15.98:0/2389528392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633d7d50 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f14040cc7c0 tx=0x7f14040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99a82cc379b4030 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:04.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f13f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:04.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f13f2ffd640 1 -- 172.21.15.98:0/2389528392 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140401e070 con 0x5622633cd5d0 2024-09-19T08:10:04.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f13f2ffd640 1 -- 172.21.15.98:0/2389528392 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f1404010ca0 con 0x5622633cd5d0 2024-09-19T08:10:04.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f141bcd3740 1 -- 172.21.15.98:0/2389528392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1408018b20 con 0x5622633cd5d0 2024-09-19T08:10:04.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f13f2ffd640 1 -- 172.21.15.98:0/2389528392 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f14040b2920 con 0x5622633cd5d0 2024-09-19T08:10:04.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f141bcd3740 10 client.17148 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:04.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f141bcd3740 1 -- 172.21.15.98:0/2389528392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5622633eaf00 con 0x5622633cd5d0 2024-09-19T08:10:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.935+0000 7f13f2ffd640 1 -- 172.21.15.98:0/2389528392 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1408018b20 con 0x5622633cd5d0 2024-09-19T08:10:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.936+0000 7f13f2ffd640 1 -- 172.21.15.98:0/2389528392 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1404097960 con 0x5622633cd5d0 2024-09-19T08:10:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.936+0000 7f141bcd3740 10 client.17148 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.936+0000 7f141bcd3740 10 client.17148 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:10:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.936+0000 7f141bcd3740 20 client.17148 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.936+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2389528392 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5622633ec310 0x5622633ee720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.936+0000 7f141bcd3740 4 client.17148 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.936+0000 7f141bcd3740 1 -- 172.21.15.98:0/2389528392 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562263340860 con 0x5622633ec310 2024-09-19T08:10:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.936+0000 7f14127fc640 1 --2- 172.21.15.98:0/2389528392 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5622633ec310 0x5622633ee720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.936+0000 7f14127fc640 1 --2- 172.21.15.98:0/2389528392 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5622633ec310 0x5622633ee720 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e1dc5c320ce3f644 server_cookie=1489bc33eb1f59c0 in_seq=0 out_seq=0 2024-09-19T08:10:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.936+0000 7f13f2ffd640 10 client.17148 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:10:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.937+0000 7f13f2ffd640 1 -- 172.21.15.98:0/2389528392 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562263340860 con 0x5622633ec310 2024-09-19T08:10:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.937+0000 7f13f2ffd640 10 client.17148 handle_command_reply: tid=0 2024-09-19T08:10:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.937+0000 7f141bcd3740 1 client.17148 shutdown 2024-09-19T08:10:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.937+0000 7f141bcd3740 1 -- 172.21.15.98:0/2389528392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 msgr2=0x5622633d7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.937+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2389528392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633d7d50 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f14040cc7c0 tx=0x7f14040046c0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.937+0000 7f141bcd3740 1 -- 172.21.15.98:0/2389528392 shutdown_connections 2024-09-19T08:10:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.937+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2389528392 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5622633ec310 0x5622633ee720 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.937+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2389528392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633d8370 0x5622633d55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.937+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2389528392 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633ceb40 0x5622633d5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.937+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2389528392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633d7d50 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.937+0000 7f13f2ffd640 0 client.17148 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:10:04.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.937+0000 7f141bcd3740 1 -- 172.21.15.98:0/2389528392 >> 172.21.15.98:0/2389528392 conn(0x562263396310 msgr2=0x5622633cf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:04.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.937+0000 7f141bcd3740 1 -- 172.21.15.98:0/2389528392 shutdown_connections 2024-09-19T08:10:04.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.938+0000 7f141bcd3740 1 -- 172.21.15.98:0/2389528392 wait complete. 2024-09-19T08:10:04.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.938+0000 7f141bcd3740 20 client.17148 trim_cache size 0 max 16384 2024-09-19T08:10:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.950+0000 7f141bcd3740 1 Processor -- start 2024-09-19T08:10:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.950+0000 7f141bcd3740 1 -- start start 2024-09-19T08:10:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.950+0000 7f141bcd3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633cd5d0 0x5622633c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.950+0000 7f141bcd3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 0x5622633d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.950+0000 7f141bcd3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633d8370 0x5622633d0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.950+0000 7f141bcd3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14141ab640 con 0x5622633d8370 2024-09-19T08:10:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.950+0000 7f141bcd3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f14141ad180 con 0x5622633cd5d0 2024-09-19T08:10:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.950+0000 7f141bcd3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f14141abd10 con 0x5622633ceb40 2024-09-19T08:10:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.951+0000 7f14137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633d8370 0x5622633d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.951+0000 7f14137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633d8370 0x5622633d0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34752/0 (socket says 172.21.15.98:34752) 2024-09-19T08:10:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.951+0000 7f14137fe640 1 -- 172.21.15.98:0/2046262327 learned_addr learned my addr 172.21.15.98:0/2046262327 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.951+0000 7f14127fc640 1 --2- 172.21.15.98:0/2046262327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 0x5622633d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.951+0000 7f1412ffd640 1 --2- 172.21.15.98:0/2046262327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633cd5d0 0x5622633c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.951+0000 7f14137fe640 1 -- 172.21.15.98:0/2046262327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 msgr2=0x5622633d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.951+0000 7f14137fe640 1 --2- 172.21.15.98:0/2046262327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 0x5622633d0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.951+0000 7f14137fe640 1 -- 172.21.15.98:0/2046262327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633cd5d0 msgr2=0x5622633c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.951+0000 7f14137fe640 1 --2- 172.21.15.98:0/2046262327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633cd5d0 0x5622633c8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.951+0000 7f14137fe640 1 -- 172.21.15.98:0/2046262327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622633eeda0 con 0x5622633d8370 2024-09-19T08:10:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.951+0000 7f14137fe640 1 --2- 172.21.15.98:0/2046262327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633d8370 0x5622633d0660 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f140c009ca0 tx=0x7f140c005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df3626ab3b7944e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:04.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.951+0000 7f13f97fa640 1 -- 172.21.15.98:0/2046262327 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c034040 con 0x5622633d8370 2024-09-19T08:10:04.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.951+0000 7f13f97fa640 1 -- 172.21.15.98:0/2046262327 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f140c003cf0 con 0x5622633d8370 2024-09-19T08:10:04.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.951+0000 7f13f97fa640 1 -- 172.21.15.98:0/2046262327 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c03d9a0 con 0x5622633d8370 2024-09-19T08:10:04.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.952+0000 7f141bcd3740 1 -- 172.21.15.98:0/2046262327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633d8370 msgr2=0x5622633d0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:04.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.952+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2046262327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633d8370 0x5622633d0660 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f140c009ca0 tx=0x7f140c005c30 comp rx=0 tx=0).stop 2024-09-19T08:10:04.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.952+0000 7f141bcd3740 1 -- 172.21.15.98:0/2046262327 shutdown_connections 2024-09-19T08:10:04.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.952+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2046262327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633d8370 0x5622633d0660 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.952+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2046262327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 0x5622633d0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.952+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2046262327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633cd5d0 0x5622633c8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.952+0000 7f141bcd3740 1 -- 172.21.15.98:0/2046262327 >> 172.21.15.98:0/2046262327 conn(0x562263396310 msgr2=0x5622633c84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:04.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.952+0000 7f141bcd3740 1 -- 172.21.15.98:0/2046262327 shutdown_connections 2024-09-19T08:10:04.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.952+0000 7f141bcd3740 1 -- 172.21.15.98:0/2046262327 wait complete. 2024-09-19T08:10:04.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.952+0000 7f141bcd3740 1 -- start start 2024-09-19T08:10:04.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.952+0000 7f141bcd3740 1 Processor -- start 2024-09-19T08:10:04.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f141bcd3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633d20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f141bcd3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 0x5622633d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f141bcd3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633d8370 0x5622633d2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f141bcd3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14141acab0 con 0x5622633cd5d0 2024-09-19T08:10:04.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f141bcd3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f14141ac3e0 con 0x5622633d8370 2024-09-19T08:10:04.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f141bcd3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f14141a9620 con 0x5622633ceb40 2024-09-19T08:10:04.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f141bcd3740 10 client.0 _ll_register_callbacks cb 0x7f1404097610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:04.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f1412ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633d20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f1412ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633d20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34760/0 (socket says 172.21.15.98:34760) 2024-09-19T08:10:04.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f14127fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 0x5622633d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f1412ffd640 1 -- 172.21.15.98:0/2909328191 learned_addr learned my addr 172.21.15.98:0/2909328191 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:04.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f14137fe640 1 --2- 172.21.15.98:0/2909328191 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633d8370 0x5622633d2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f1412ffd640 1 -- 172.21.15.98:0/2909328191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 msgr2=0x5622633d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:04.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f1412ffd640 1 --2- 172.21.15.98:0/2909328191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 0x5622633d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f1412ffd640 1 -- 172.21.15.98:0/2909328191 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633d8370 msgr2=0x5622633d2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:04.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f1412ffd640 1 --2- 172.21.15.98:0/2909328191 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633d8370 0x5622633d2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f1412ffd640 1 -- 172.21.15.98:0/2909328191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562263456010 con 0x5622633cd5d0 2024-09-19T08:10:04.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f1412ffd640 1 --2- 172.21.15.98:0/2909328191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633d20a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f1404097750 tx=0x7f140409b150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23698c2e3f3eb6eb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:04.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f13f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:04.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.953+0000 7f13f2ffd640 1 -- 172.21.15.98:0/2909328191 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140401e070 con 0x5622633cd5d0 2024-09-19T08:10:04.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.954+0000 7f13f2ffd640 1 -- 172.21.15.98:0/2909328191 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f140409a740 con 0x5622633cd5d0 2024-09-19T08:10:04.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.954+0000 7f141bcd3740 1 -- 172.21.15.98:0/2909328191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f140c003cf0 con 0x5622633cd5d0 2024-09-19T08:10:04.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.954+0000 7f13f2ffd640 1 -- 172.21.15.98:0/2909328191 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1404009230 con 0x5622633cd5d0 2024-09-19T08:10:04.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.954+0000 7f141bcd3740 10 client.17154 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:04.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.954+0000 7f141bcd3740 1 -- 172.21.15.98:0/2909328191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562263457310 con 0x5622633cd5d0 2024-09-19T08:10:04.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.954+0000 7f13f2ffd640 1 -- 172.21.15.98:0/2909328191 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f140c003cf0 con 0x5622633cd5d0 2024-09-19T08:10:04.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.954+0000 7f13f2ffd640 1 -- 172.21.15.98:0/2909328191 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1404010610 con 0x5622633cd5d0 2024-09-19T08:10:04.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.954+0000 7f141bcd3740 10 client.17154 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:04.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.954+0000 7f141bcd3740 10 client.17154 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:10:04.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.954+0000 7f141bcd3740 20 client.17154 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:04.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.954+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2909328191 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x562263457aa0 0x562263459ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:04.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.954+0000 7f141bcd3740 4 client.17154 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:10:04.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.954+0000 7f141bcd3740 1 -- 172.21.15.98:0/2909328191 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56226333ad70 con 0x562263457aa0 2024-09-19T08:10:04.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.954+0000 7f14127fc640 1 --2- 172.21.15.98:0/2909328191 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x562263457aa0 0x562263459ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:04.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f14127fc640 1 --2- 172.21.15.98:0/2909328191 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x562263457aa0 0x562263459ec0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d3db265f61ff5fe8 server_cookie=fc40144734ea480d in_seq=0 out_seq=0 2024-09-19T08:10:04.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f13f2ffd640 10 client.17154 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:10:04.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f13f2ffd640 1 -- 172.21.15.98:0/2909328191 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56226333ad70 con 0x562263457aa0 2024-09-19T08:10:04.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f13f2ffd640 10 client.17154 handle_command_reply: tid=0 2024-09-19T08:10:04.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f141bcd3740 1 client.17154 shutdown 2024-09-19T08:10:04.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f141bcd3740 1 -- 172.21.15.98:0/2909328191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 msgr2=0x5622633d20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:04.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2909328191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633d20a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f1404097750 tx=0x7f140409b150 comp rx=0 tx=0).stop 2024-09-19T08:10:04.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f141bcd3740 1 -- 172.21.15.98:0/2909328191 shutdown_connections 2024-09-19T08:10:04.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2909328191 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622633d8370 0x5622633d2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2909328191 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x562263457aa0 0x562263459ec0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2909328191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622633ceb40 0x5622633d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f141bcd3740 1 --2- 172.21.15.98:0/2909328191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622633cd5d0 0x5622633d20a0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f13f2ffd640 0 client.17154 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:10:04.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f141bcd3740 1 -- 172.21.15.98:0/2909328191 >> 172.21.15.98:0/2909328191 conn(0x562263396310 msgr2=0x5622633c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:04.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f141bcd3740 1 -- 172.21.15.98:0/2909328191 shutdown_connections 2024-09-19T08:10:04.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f141bcd3740 1 -- 172.21.15.98:0/2909328191 wait complete. 2024-09-19T08:10:04.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.955+0000 7f141bcd3740 20 client.17154 trim_cache size 0 max 16384 2024-09-19T08:10:04.961 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:10:04.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.956+0000 7f14198fa640 1 -- 172.21.15.98:0/3420379556 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f13fc0a3430 msgr2=0x7f13fc0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:04.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.956+0000 7f14198fa640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f13fc0a3430 0x7f13fc0a58f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f140c005cc0 tx=0x7f140c02fc00 comp rx=0 tx=0).stop 2024-09-19T08:10:04.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.956+0000 7f14198fa640 1 -- 172.21.15.98:0/3420379556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1414102d40 msgr2=0x7f14141984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:04.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.956+0000 7f14198fa640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1414102d40 0x7f14141984b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f140400b4a0 tx=0x7f140400b960 comp rx=0 tx=0).stop 2024-09-19T08:10:04.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.956+0000 7f14198fa640 1 -- 172.21.15.98:0/3420379556 shutdown_connections 2024-09-19T08:10:04.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.956+0000 7f14198fa640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f13fc0a3430 0x7f13fc0a58f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.956+0000 7f14198fa640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141410c170 0x7f141419cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.956+0000 7f14198fa640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1414103660 0x7f14141989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.956+0000 7f14198fa640 1 --2- 172.21.15.98:0/3420379556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1414102d40 0x7f14141984b0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:04.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.956+0000 7f14198fa640 1 -- 172.21.15.98:0/3420379556 >> 172.21.15.98:0/3420379556 conn(0x7f14140fc7a0 msgr2=0x7f14140fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:04.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.956+0000 7f14198fa640 1 -- 172.21.15.98:0/3420379556 shutdown_connections 2024-09-19T08:10:04.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:04.956+0000 7f14198fa640 1 -- 172.21.15.98:0/3420379556 wait complete. 2024-09-19T08:10:05.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:04 smithi162 ceph-mon[35260]: pgmap v214: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:10:05.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:04 smithi139 ceph-mon[34874]: pgmap v214: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:10:05.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:04 smithi098 ceph-mon[31333]: pgmap v214: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:10:05.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:10:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:06.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.356+0000 7f8eefa25640 1 -- 172.21.15.98:0/85657287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee8100b30 msgr2=0x7f8ee8100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:06.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.356+0000 7f8eefa25640 1 --2- 172.21.15.98:0/85657287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee8100b30 0x7f8ee8100f10 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4009900 tx=0x7f8ee402a470 comp rx=0 tx=0).stop 2024-09-19T08:10:06.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 -- 172.21.15.98:0/85657287 shutdown_connections 2024-09-19T08:10:06.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 --2- 172.21.15.98:0/85657287 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ee8105f40 0x7f8ee81063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 --2- 172.21.15.98:0/85657287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ee8101450 0x7f8ee8105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 --2- 172.21.15.98:0/85657287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee8100b30 0x7f8ee8100f10 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 -- 172.21.15.98:0/85657287 >> 172.21.15.98:0/85657287 conn(0x7f8ee80fc780 msgr2=0x7f8ee80feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:06.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 -- 172.21.15.98:0/85657287 shutdown_connections 2024-09-19T08:10:06.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 -- 172.21.15.98:0/85657287 wait complete. 2024-09-19T08:10:06.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 Processor -- start 2024-09-19T08:10:06.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 -- start start 2024-09-19T08:10:06.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ee8100b30 0x7f8ee81a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee8101450 0x7f8ee81a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ee8105f40 0x7f8ee819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee81185f0 con 0x7f8ee8101450 2024-09-19T08:10:06.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee8118470 con 0x7f8ee8100b30 2024-09-19T08:10:06.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.357+0000 7f8eefa25640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee8118770 con 0x7f8ee8105f40 2024-09-19T08:10:06.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8eecfc0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee8101450 0x7f8ee81a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8eecfc0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee8101450 0x7f8ee81a4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40640/0 (socket says 172.21.15.98:40640) 2024-09-19T08:10:06.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8eecfc0640 1 -- 172.21.15.98:0/303953834 learned_addr learned my addr 172.21.15.98:0/303953834 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:06.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8eed7c1640 1 --2- 172.21.15.98:0/303953834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ee8100b30 0x7f8ee81a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8eedfc2640 1 --2- 172.21.15.98:0/303953834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ee8105f40 0x7f8ee819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8eecfc0640 1 -- 172.21.15.98:0/303953834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ee8105f40 msgr2=0x7f8ee819e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:06.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8eecfc0640 1 --2- 172.21.15.98:0/303953834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ee8105f40 0x7f8ee819e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8eecfc0640 1 -- 172.21.15.98:0/303953834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ee8100b30 msgr2=0x7f8ee81a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:06.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8eecfc0640 1 --2- 172.21.15.98:0/303953834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ee8100b30 0x7f8ee81a47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8eecfc0640 1 -- 172.21.15.98:0/303953834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee819f0d0 con 0x7f8ee8101450 2024-09-19T08:10:06.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8eecfc0640 1 --2- 172.21.15.98:0/303953834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee8101450 0x7f8ee81a4d20 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f8ed800c970 tx=0x7f8ed800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:06.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8ede7fc640 1 -- 172.21.15.98:0/303953834 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8007bb0 con 0x7f8ee8101450 2024-09-19T08:10:06.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8ede7fc640 1 -- 172.21.15.98:0/303953834 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ed8007d50 con 0x7f8ee8101450 2024-09-19T08:10:06.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8eefa25640 1 -- 172.21.15.98:0/303953834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee819f3c0 con 0x7f8ee8101450 2024-09-19T08:10:06.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8ede7fc640 1 -- 172.21.15.98:0/303953834 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed800f450 con 0x7f8ee8101450 2024-09-19T08:10:06.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.358+0000 7f8eefa25640 1 -- 172.21.15.98:0/303953834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee81ab6b0 con 0x7f8ee8101450 2024-09-19T08:10:06.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.360+0000 7f8ede7fc640 1 -- 172.21.15.98:0/303953834 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f8ed8004f60 con 0x7f8ee8101450 2024-09-19T08:10:06.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8ef1dfe740 1 Processor -- start 2024-09-19T08:10:06.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8ef1dfe740 1 -- start start 2024-09-19T08:10:06.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8ef1dfe740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e79b40 0x55f037e78090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8ef1dfe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e78a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8ef1dfe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e83370 0x55f037e85760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8ede7fc640 1 --2- 172.21.15.98:0/303953834 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8ed40a33b0 0x7f8ed40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8ef1dfe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f037dde2f0 con 0x55f037e785d0 2024-09-19T08:10:06.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8ef1dfe740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f037d85cf0 con 0x55f037e83370 2024-09-19T08:10:06.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8ef1dfe740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f037dda2b0 con 0x55f037e79b40 2024-09-19T08:10:06.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8eed7c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e78a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8eed7c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e78a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40644/0 (socket says 172.21.15.98:40644) 2024-09-19T08:10:06.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8eed7c1640 1 -- 172.21.15.98:0/69369093 learned_addr learned my addr 172.21.15.98:0/69369093 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:06.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8eedfc2640 1 --2- 172.21.15.98:0/303953834 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8ed40a33b0 0x7f8ed40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8eedfc2640 1 --2- 172.21.15.98:0/69369093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e79b40 0x55f037e78090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.361+0000 7f8ede7fc640 1 -- 172.21.15.98:0/303953834 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f8ed80cfab0 con 0x7f8ee8101450 2024-09-19T08:10:06.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.362+0000 7f8eecfc0640 1 --2- 172.21.15.98:0/69369093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e83370 0x55f037e85760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.362+0000 7f8eed7c1640 1 -- 172.21.15.98:0/69369093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e79b40 msgr2=0x55f037e78090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:06.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.362+0000 7f8eed7c1640 1 --2- 172.21.15.98:0/69369093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e79b40 0x55f037e78090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.362+0000 7f8eed7c1640 1 -- 172.21.15.98:0/69369093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e83370 msgr2=0x55f037e85760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:06.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.362+0000 7f8eed7c1640 1 --2- 172.21.15.98:0/69369093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e83370 0x55f037e85760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.362+0000 7f8eed7c1640 1 -- 172.21.15.98:0/69369093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f037d6a3a0 con 0x55f037e785d0 2024-09-19T08:10:06.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.362+0000 7f8eecfc0640 1 --2- 172.21.15.98:0/69369093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e83370 0x55f037e85760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:06.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.362+0000 7f8eedfc2640 1 --2- 172.21.15.98:0/303953834 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8ed40a33b0 0x7f8ed40a5870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0009fd0 tx=0x7f8ee0009540 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:06.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.362+0000 7f8eed7c1640 1 --2- 172.21.15.98:0/69369093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e78a10 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f8ee402a990 tx=0x7f8ee4031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da59a80cff1198a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:06.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.362+0000 7f8ed37fe640 1 -- 172.21.15.98:0/69369093 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee403f070 con 0x55f037e785d0 2024-09-19T08:10:06.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.362+0000 7f8ed37fe640 1 -- 172.21.15.98:0/69369093 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ee4031a20 con 0x55f037e785d0 2024-09-19T08:10:06.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.362+0000 7f8ed37fe640 1 -- 172.21.15.98:0/69369093 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4035070 con 0x55f037e785d0 2024-09-19T08:10:06.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.363+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/69369093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 msgr2=0x55f037e78a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.363+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/69369093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e78a10 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f8ee402a990 tx=0x7f8ee4031630 comp rx=0 tx=0).stop 2024-09-19T08:10:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.363+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/69369093 shutdown_connections 2024-09-19T08:10:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.363+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/69369093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e83370 0x55f037e85760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.363+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/69369093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e78a10 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.363+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/69369093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e79b40 0x55f037e78090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.363+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/69369093 >> 172.21.15.98:0/69369093 conn(0x55f037e41310 msgr2=0x55f037e74c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.363+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/69369093 shutdown_connections 2024-09-19T08:10:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.363+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/69369093 wait complete. 2024-09-19T08:10:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.363+0000 7f8ef1dfe740 1 -- start start 2024-09-19T08:10:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.363+0000 7f8ef1dfe740 1 Processor -- start 2024-09-19T08:10:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8ef1dfe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e82d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8ef1dfe740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e79b40 0x55f037e7ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8ef1dfe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e83370 0x55f037e805d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8ef1dfe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f037e86010 con 0x55f037e785d0 2024-09-19T08:10:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8ef1dfe740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f037d99060 con 0x55f037e83370 2024-09-19T08:10:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8ef1dfe740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f037e86190 con 0x55f037e79b40 2024-09-19T08:10:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8ef1dfe740 10 client.0 _ll_register_callbacks cb 0x55f037aec8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8eed7c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e82d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8eed7c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e82d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40654/0 (socket says 172.21.15.98:40654) 2024-09-19T08:10:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8eed7c1640 1 -- 172.21.15.98:0/763283518 learned_addr learned my addr 172.21.15.98:0/763283518 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8eecfc0640 1 --2- 172.21.15.98:0/763283518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e79b40 0x55f037e7ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8eed7c1640 1 -- 172.21.15.98:0/763283518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e79b40 msgr2=0x55f037e7ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8eed7c1640 1 --2- 172.21.15.98:0/763283518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e79b40 0x55f037e7ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8eed7c1640 1 -- 172.21.15.98:0/763283518 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e83370 msgr2=0x55f037e805d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8eed7c1640 1 --2- 172.21.15.98:0/763283518 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e83370 0x55f037e805d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8eed7c1640 1 -- 172.21.15.98:0/763283518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f037b4f3c0 con 0x55f037e785d0 2024-09-19T08:10:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8eed7c1640 1 --2- 172.21.15.98:0/763283518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e82d30 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4003cd0 tx=0x7f8ee402ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2349d25c79fccbbc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.364+0000 7f8ed0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.365+0000 7f8ed0ff9640 1 -- 172.21.15.98:0/763283518 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4035070 con 0x55f037e785d0 2024-09-19T08:10:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.365+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/763283518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ee4031a20 con 0x55f037e785d0 2024-09-19T08:10:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.365+0000 7f8ed0ff9640 1 -- 172.21.15.98:0/763283518 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f8ee4004410 con 0x55f037e785d0 2024-09-19T08:10:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.365+0000 7f8ed0ff9640 1 -- 172.21.15.98:0/763283518 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8ee4055050 con 0x55f037e785d0 2024-09-19T08:10:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.365+0000 7f8ef1dfe740 10 client.17166 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.365+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/763283518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f037e919e0 con 0x55f037e785d0 2024-09-19T08:10:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.365+0000 7f8ed0ff9640 1 -- 172.21.15.98:0/763283518 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ee4031a20 con 0x55f037e785d0 2024-09-19T08:10:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.365+0000 7f8ed0ff9640 1 -- 172.21.15.98:0/763283518 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8ee40436d0 con 0x55f037e785d0 2024-09-19T08:10:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.365+0000 7f8ef1dfe740 10 client.17166 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.365+0000 7f8ef1dfe740 10 client.17166 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:10:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.365+0000 7f8ef1dfe740 20 client.17166 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.365+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/763283518 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55f037e92d30 0x55f037e95120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.365+0000 7f8ef1dfe740 4 client.17166 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.365+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/763283518 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f037deb860 con 0x55f037e92d30 2024-09-19T08:10:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.366+0000 7f8eecfc0640 1 --2- 172.21.15.98:0/763283518 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55f037e92d30 0x55f037e95120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.366+0000 7f8eecfc0640 1 --2- 172.21.15.98:0/763283518 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55f037e92d30 0x55f037e95120 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f3d0ef2fe4ce801b server_cookie=a9b6fe81b3249135 in_seq=0 out_seq=0 2024-09-19T08:10:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.366+0000 7f8ed0ff9640 10 client.17166 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:10:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ed0ff9640 1 -- 172.21.15.98:0/763283518 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f037deb860 con 0x55f037e92d30 2024-09-19T08:10:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ed0ff9640 10 client.17166 handle_command_reply: tid=0 2024-09-19T08:10:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ef1dfe740 1 client.17166 shutdown 2024-09-19T08:10:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/763283518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 msgr2=0x55f037e82d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/763283518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e82d30 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4003cd0 tx=0x7f8ee402ab80 comp rx=0 tx=0).stop 2024-09-19T08:10:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/763283518 shutdown_connections 2024-09-19T08:10:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/763283518 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55f037e92d30 0x55f037e95120 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/763283518 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e83370 0x55f037e805d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/763283518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e79b40 0x55f037e7ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ed0ff9640 0 client.17166 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:10:06.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/763283518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e82d30 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/763283518 >> 172.21.15.98:0/763283518 conn(0x55f037e41310 msgr2=0x55f037e7a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:06.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/763283518 shutdown_connections 2024-09-19T08:10:06.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/763283518 wait complete. 2024-09-19T08:10:06.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.367+0000 7f8ef1dfe740 20 client.17166 trim_cache size 0 max 16384 2024-09-19T08:10:06.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.380+0000 7f8ef1dfe740 1 Processor -- start 2024-09-19T08:10:06.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.380+0000 7f8ef1dfe740 1 -- start start 2024-09-19T08:10:06.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.380+0000 7f8ef1dfe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e8f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.380+0000 7f8ef1dfe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e79b40 0x55f037e77ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.380+0000 7f8ef1dfe740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e83370 0x55f037e7efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.380+0000 7f8ef1dfe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee81b3e10 con 0x55f037e785d0 2024-09-19T08:10:06.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.380+0000 7f8ef1dfe740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee81b5950 con 0x55f037e79b40 2024-09-19T08:10:06.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.380+0000 7f8ef1dfe740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee81b44e0 con 0x55f037e83370 2024-09-19T08:10:06.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.380+0000 7f8eed7c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e8f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.380+0000 7f8eed7c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e8f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40666/0 (socket says 172.21.15.98:40666) 2024-09-19T08:10:06.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.380+0000 7f8eed7c1640 1 -- 172.21.15.98:0/3697278053 learned_addr learned my addr 172.21.15.98:0/3697278053 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:06.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.381+0000 7f8eedfc2640 1 --2- 172.21.15.98:0/3697278053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e83370 0x55f037e7efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.381+0000 7f8eed7c1640 1 -- 172.21.15.98:0/3697278053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e83370 msgr2=0x55f037e7efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:06.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.381+0000 7f8eecfc0640 1 --2- 172.21.15.98:0/3697278053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e79b40 0x55f037e77ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.381+0000 7f8eed7c1640 1 --2- 172.21.15.98:0/3697278053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e83370 0x55f037e7efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.381+0000 7f8eed7c1640 1 -- 172.21.15.98:0/3697278053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e79b40 msgr2=0x55f037e77ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.381+0000 7f8eed7c1640 1 --2- 172.21.15.98:0/3697278053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e79b40 0x55f037e77ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.381+0000 7f8eed7c1640 1 -- 172.21.15.98:0/3697278053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f037e7e530 con 0x55f037e785d0 2024-09-19T08:10:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.381+0000 7f8eed7c1640 1 --2- 172.21.15.98:0/3697278053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e8f2b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4031550 tx=0x7f8ee4005930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebb01d213ffacb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.381+0000 7f8ed2ffd640 1 -- 172.21.15.98:0/3697278053 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4035030 con 0x55f037e785d0 2024-09-19T08:10:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.381+0000 7f8ed2ffd640 1 -- 172.21.15.98:0/3697278053 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ee404c070 con 0x55f037e785d0 2024-09-19T08:10:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.381+0000 7f8ed2ffd640 1 -- 172.21.15.98:0/3697278053 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee403f040 con 0x55f037e785d0 2024-09-19T08:10:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/3697278053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 msgr2=0x55f037e8f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/3697278053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e8f2b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4031550 tx=0x7f8ee4005930 comp rx=0 tx=0).stop 2024-09-19T08:10:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/3697278053 shutdown_connections 2024-09-19T08:10:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/3697278053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e83370 0x55f037e7efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/3697278053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e79b40 0x55f037e77ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/3697278053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e785d0 0x55f037e8f2b0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/3697278053 >> 172.21.15.98:0/3697278053 conn(0x55f037e41310 msgr2=0x55f037e7e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/3697278053 shutdown_connections 2024-09-19T08:10:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/3697278053 wait complete. 2024-09-19T08:10:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 -- start start 2024-09-19T08:10:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 Processor -- start 2024-09-19T08:10:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e785d0 0x55f037e8bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e79b40 0x55f037e9a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e83370 0x55f037e9a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee81b5280 con 0x55f037e83370 2024-09-19T08:10:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.382+0000 7f8ef1dfe740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee81b4bb0 con 0x55f037e79b40 2024-09-19T08:10:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8ef1dfe740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee81b1df0 con 0x55f037e785d0 2024-09-19T08:10:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8ef1dfe740 10 client.0 _ll_register_callbacks cb 0x7f8ee000a6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8eedfc2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e83370 0x55f037e9a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8eedfc2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e83370 0x55f037e9a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40672/0 (socket says 172.21.15.98:40672) 2024-09-19T08:10:06.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8eedfc2640 1 -- 172.21.15.98:0/2560773070 learned_addr learned my addr 172.21.15.98:0/2560773070 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:06.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8eed7c1640 1 --2- 172.21.15.98:0/2560773070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e785d0 0x55f037e8bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8eecfc0640 1 --2- 172.21.15.98:0/2560773070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e79b40 0x55f037e9a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8eedfc2640 1 -- 172.21.15.98:0/2560773070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e785d0 msgr2=0x55f037e8bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:06.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8eedfc2640 1 --2- 172.21.15.98:0/2560773070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e785d0 0x55f037e8bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8eedfc2640 1 -- 172.21.15.98:0/2560773070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e79b40 msgr2=0x55f037e9a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:06.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8eedfc2640 1 --2- 172.21.15.98:0/2560773070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e79b40 0x55f037e9a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8eedfc2640 1 -- 172.21.15.98:0/2560773070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f037e9aed0 con 0x55f037e83370 2024-09-19T08:10:06.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8eecfc0640 1 --2- 172.21.15.98:0/2560773070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e79b40 0x55f037e9a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8eedfc2640 1 --2- 172.21.15.98:0/2560773070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e83370 0x55f037e9a6d0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0007090 tx=0x7f8ee0009d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b30c212cf7129c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8ed0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8ed0ff9640 1 -- 172.21.15.98:0/2560773070 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0029060 con 0x55f037e83370 2024-09-19T08:10:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8ed0ff9640 1 -- 172.21.15.98:0/2560773070 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7f8ee0024400 con 0x55f037e83370 2024-09-19T08:10:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/2560773070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ee404c070 con 0x55f037e83370 2024-09-19T08:10:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.383+0000 7f8ed0ff9640 1 -- 172.21.15.98:0/2560773070 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8ee0013040 con 0x55f037e83370 2024-09-19T08:10:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.384+0000 7f8ef1dfe740 10 client.17172 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.384+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/2560773070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f037efd850 con 0x55f037e83370 2024-09-19T08:10:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.384+0000 7f8ed0ff9640 1 -- 172.21.15.98:0/2560773070 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ee404c070 con 0x55f037e83370 2024-09-19T08:10:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.384+0000 7f8ed0ff9640 1 -- 172.21.15.98:0/2560773070 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8ee002c460 con 0x55f037e83370 2024-09-19T08:10:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.384+0000 7f8ef1dfe740 10 client.17172 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.384+0000 7f8ef1dfe740 10 client.17172 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:10:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.384+0000 7f8ef1dfe740 20 client.17172 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.384+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/2560773070 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55f037efdfd0 0x55f037f003f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.384+0000 7f8ef1dfe740 4 client.17172 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:10:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.384+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/2560773070 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f037de5d70 con 0x55f037efdfd0 2024-09-19T08:10:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.384+0000 7f8eed7c1640 1 --2- 172.21.15.98:0/2560773070 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55f037efdfd0 0x55f037f003f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8eed7c1640 1 --2- 172.21.15.98:0/2560773070 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55f037efdfd0 0x55f037f003f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=576f13a71d2b838e server_cookie=d0cab8d5973214df in_seq=0 out_seq=0 2024-09-19T08:10:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ed0ff9640 10 client.17172 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:10:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ed0ff9640 1 -- 172.21.15.98:0/2560773070 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f037de5d70 con 0x55f037efdfd0 2024-09-19T08:10:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ed0ff9640 10 client.17172 handle_command_reply: tid=0 2024-09-19T08:10:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ef1dfe740 1 client.17172 shutdown 2024-09-19T08:10:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/2560773070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e83370 msgr2=0x55f037e9a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/2560773070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e83370 0x55f037e9a6d0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0007090 tx=0x7f8ee0009d80 comp rx=0 tx=0).stop 2024-09-19T08:10:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/2560773070 shutdown_connections 2024-09-19T08:10:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/2560773070 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55f037efdfd0 0x55f037f003f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/2560773070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f037e83370 0x55f037e9a6d0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/2560773070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f037e79b40 0x55f037e9a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ef1dfe740 1 --2- 172.21.15.98:0/2560773070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f037e785d0 0x55f037e8bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ed0ff9640 0 client.17172 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:10:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/2560773070 >> 172.21.15.98:0/2560773070 conn(0x55f037e41310 msgr2=0x55f037e734e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/2560773070 shutdown_connections 2024-09-19T08:10:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ef1dfe740 1 -- 172.21.15.98:0/2560773070 wait complete. 2024-09-19T08:10:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.385+0000 7f8ef1dfe740 20 client.17172 trim_cache size 0 max 16384 2024-09-19T08:10:06.391 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:10:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.386+0000 7f8eefa25640 1 -- 172.21.15.98:0/303953834 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8ed40a33b0 msgr2=0x7f8ed40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.386+0000 7f8eefa25640 1 --2- 172.21.15.98:0/303953834 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8ed40a33b0 0x7f8ed40a5870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0009fd0 tx=0x7f8ee0009540 comp rx=0 tx=0).stop 2024-09-19T08:10:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.386+0000 7f8eefa25640 1 -- 172.21.15.98:0/303953834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee8101450 msgr2=0x7f8ee81a4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.386+0000 7f8eefa25640 1 --2- 172.21.15.98:0/303953834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee8101450 0x7f8ee81a4d20 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f8ed800c970 tx=0x7f8ed800ce30 comp rx=0 tx=0).stop 2024-09-19T08:10:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.386+0000 7f8eefa25640 1 -- 172.21.15.98:0/303953834 shutdown_connections 2024-09-19T08:10:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.386+0000 7f8eefa25640 1 --2- 172.21.15.98:0/303953834 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f8ed40a33b0 0x7f8ed40a5870 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.386+0000 7f8eefa25640 1 --2- 172.21.15.98:0/303953834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ee8105f40 0x7f8ee819e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.386+0000 7f8eefa25640 1 --2- 172.21.15.98:0/303953834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee8101450 0x7f8ee81a4d20 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.386+0000 7f8eefa25640 1 --2- 172.21.15.98:0/303953834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ee8100b30 0x7f8ee81a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:06.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.386+0000 7f8eefa25640 1 -- 172.21.15.98:0/303953834 >> 172.21.15.98:0/303953834 conn(0x7f8ee80fc780 msgr2=0x7f8ee80fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:06.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.386+0000 7f8eefa25640 1 -- 172.21.15.98:0/303953834 shutdown_connections 2024-09-19T08:10:06.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:06.386+0000 7f8eefa25640 1 -- 172.21.15.98:0/303953834 wait complete. 2024-09-19T08:10:06.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:06 smithi098 ceph-mon[31333]: pgmap v215: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:10:07.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:10:07.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:06 smithi162 ceph-mon[35260]: pgmap v215: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:10:07.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:06 smithi139 ceph-mon[34874]: pgmap v215: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:10:07.327 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:07.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.841+0000 7fa5d469c640 1 -- 172.21.15.98:0/1386647726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5cc106f10 msgr2=0x7fa5cc10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:07.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.841+0000 7fa5d469c640 1 --2- 172.21.15.98:0/1386647726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5cc106f10 0x7fa5cc10ea90 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009ca0 tx=0x7fa5c402a8c0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.841+0000 7fa5d469c640 1 -- 172.21.15.98:0/1386647726 shutdown_connections 2024-09-19T08:10:07.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.841+0000 7fa5d469c640 1 --2- 172.21.15.98:0/1386647726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5cc106f10 0x7fa5cc10ea90 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.841+0000 7fa5d469c640 1 --2- 172.21.15.98:0/1386647726 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5cc102790 0x7fa5cc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.841+0000 7fa5d469c640 1 --2- 172.21.15.98:0/1386647726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc101de0 0x7fa5cc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.841+0000 7fa5d469c640 1 -- 172.21.15.98:0/1386647726 >> 172.21.15.98:0/1386647726 conn(0x7fa5cc0fd8c0 msgr2=0x7fa5cc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:07.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.841+0000 7fa5d469c640 1 -- 172.21.15.98:0/1386647726 shutdown_connections 2024-09-19T08:10:07.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.841+0000 7fa5d469c640 1 -- 172.21.15.98:0/1386647726 wait complete. 2024-09-19T08:10:07.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d469c640 1 Processor -- start 2024-09-19T08:10:07.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d469c640 1 -- start start 2024-09-19T08:10:07.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d469c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5cc101de0 0x7fa5cc1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d469c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5cc102790 0x7fa5cc198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d469c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc106f10 0x7fa5cc19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d469c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc112010 con 0x7fa5cc101de0 2024-09-19T08:10:07.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d469c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc111e90 con 0x7fa5cc102790 2024-09-19T08:10:07.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d469c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc112190 con 0x7fa5cc106f10 2024-09-19T08:10:07.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d2438640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5cc101de0 0x7fa5cc1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d2438640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5cc101de0 0x7fa5cc1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40702/0 (socket says 172.21.15.98:40702) 2024-09-19T08:10:07.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d2438640 1 -- 172.21.15.98:0/2714482710 learned_addr learned my addr 172.21.15.98:0/2714482710 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:07.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d1c37640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5cc102790 0x7fa5cc198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d2c39640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc106f10 0x7fa5cc19cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d2438640 1 -- 172.21.15.98:0/2714482710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc106f10 msgr2=0x7fa5cc19cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:07.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d2438640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc106f10 0x7fa5cc19cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d2438640 1 -- 172.21.15.98:0/2714482710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5cc102790 msgr2=0x7fa5cc198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:07.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d2438640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5cc102790 0x7fa5cc198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d2438640 1 -- 172.21.15.98:0/2714482710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5cc19d5a0 con 0x7fa5cc101de0 2024-09-19T08:10:07.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d2c39640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc106f10 0x7fa5cc19cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:07.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.842+0000 7fa5d1c37640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5cc102790 0x7fa5cc198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:07.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.843+0000 7fa5d2438640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5cc101de0 0x7fa5cc1985f0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800b4a0 tx=0x7fa5c800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:07.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.843+0000 7fa5c37fe640 1 -- 172.21.15.98:0/2714482710 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8014070 con 0x7fa5cc101de0 2024-09-19T08:10:07.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.843+0000 7fa5c37fe640 1 -- 172.21.15.98:0/2714482710 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5c8010af0 con 0x7fa5cc101de0 2024-09-19T08:10:07.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.843+0000 7fa5d469c640 1 -- 172.21.15.98:0/2714482710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5cc19d890 con 0x7fa5cc101de0 2024-09-19T08:10:07.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.843+0000 7fa5c37fe640 1 -- 172.21.15.98:0/2714482710 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8017d10 con 0x7fa5cc101de0 2024-09-19T08:10:07.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.843+0000 7fa5d469c640 1 -- 172.21.15.98:0/2714482710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5cc1a5170 con 0x7fa5cc101de0 2024-09-19T08:10:07.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.845+0000 7fa5c37fe640 1 -- 172.21.15.98:0/2714482710 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fa5c8017070 con 0x7fa5cc101de0 2024-09-19T08:10:07.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.845+0000 7fa5d6a75740 1 Processor -- start 2024-09-19T08:10:07.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.845+0000 7fa5d6a75740 1 -- start start 2024-09-19T08:10:07.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d6a75740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66db40 0x55dbea66c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d6a75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66c5d0 0x55dbea66ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d6a75740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea677370 0x55dbea679760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d6a75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dbea5d22f0 con 0x55dbea66c5d0 2024-09-19T08:10:07.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d6a75740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dbea579cf0 con 0x55dbea677370 2024-09-19T08:10:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d6a75740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dbea5ce2b0 con 0x55dbea66db40 2024-09-19T08:10:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d2438640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66c5d0 0x55dbea66ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d2438640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66c5d0 0x55dbea66ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40718/0 (socket says 172.21.15.98:40718) 2024-09-19T08:10:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d2438640 1 -- 172.21.15.98:0/3329679126 learned_addr learned my addr 172.21.15.98:0/3329679126 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d2c39640 1 --2- 172.21.15.98:0/3329679126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66db40 0x55dbea66c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5c37fe640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa5b80a33b0 0x7fa5b80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d1c37640 1 --2- 172.21.15.98:0/3329679126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea677370 0x55dbea679760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d2c39640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa5b80a33b0 0x7fa5b80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5c37fe640 1 -- 172.21.15.98:0/2714482710 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fa5c80cf0c0 con 0x7fa5cc101de0 2024-09-19T08:10:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d2438640 1 -- 172.21.15.98:0/3329679126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66db40 msgr2=0x55dbea66c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d2438640 1 --2- 172.21.15.98:0/3329679126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66db40 0x55dbea66c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d2438640 1 -- 172.21.15.98:0/3329679126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea677370 msgr2=0x55dbea679760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d2438640 1 --2- 172.21.15.98:0/3329679126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea677370 0x55dbea679760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.846+0000 7fa5d2438640 1 -- 172.21.15.98:0/3329679126 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbea55e3a0 con 0x55dbea66c5d0 2024-09-19T08:10:07.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.847+0000 7fa5d2438640 1 --2- 172.21.15.98:0/3329679126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66c5d0 0x55dbea66ca10 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80978e0 tx=0x7fa5c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67c9218fec8d532 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:07.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.847+0000 7fa5d2c39640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa5b80a33b0 0x7fa5b80a5870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009bf0 tx=0x7fa5c402a8c0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:07.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.847+0000 7fa5c0ff9640 1 -- 172.21.15.98:0/3329679126 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c801e070 con 0x55dbea66c5d0 2024-09-19T08:10:07.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.847+0000 7fa5c0ff9640 1 -- 172.21.15.98:0/3329679126 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5c80a2bb0 con 0x55dbea66c5d0 2024-09-19T08:10:07.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.847+0000 7fa5c0ff9640 1 -- 172.21.15.98:0/3329679126 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8010ca0 con 0x55dbea66c5d0 2024-09-19T08:10:07.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.847+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3329679126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66c5d0 msgr2=0x55dbea66ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:07.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.847+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/3329679126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66c5d0 0x55dbea66ca10 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80978e0 tx=0x7fa5c80046c0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.847+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3329679126 shutdown_connections 2024-09-19T08:10:07.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.847+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/3329679126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea677370 0x55dbea679760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.847+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/3329679126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66c5d0 0x55dbea66ca10 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.847+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/3329679126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66db40 0x55dbea66c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3329679126 >> 172.21.15.98:0/3329679126 conn(0x55dbea635310 msgr2=0x55dbea668c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3329679126 shutdown_connections 2024-09-19T08:10:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3329679126 wait complete. 2024-09-19T08:10:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d6a75740 1 -- start start 2024-09-19T08:10:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d6a75740 1 Processor -- start 2024-09-19T08:10:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d6a75740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea66c5d0 0x55dbea676dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d6a75740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66db40 0x55dbea674080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d6a75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea677370 0x55dbea674810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d6a75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dbea67a070 con 0x55dbea677370 2024-09-19T08:10:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d6a75740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dbea58d060 con 0x55dbea66c5d0 2024-09-19T08:10:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d6a75740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dbea67a1f0 con 0x55dbea66db40 2024-09-19T08:10:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d6a75740 10 client.0 _ll_register_callbacks cb 0x55dbea2e08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d2c39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea677370 0x55dbea674810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d2c39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea677370 0x55dbea674810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40734/0 (socket says 172.21.15.98:40734) 2024-09-19T08:10:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d2c39640 1 -- 172.21.15.98:0/3784862758 learned_addr learned my addr 172.21.15.98:0/3784862758 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.848+0000 7fa5d2438640 1 --2- 172.21.15.98:0/3784862758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea66c5d0 0x55dbea676dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5d1c37640 1 --2- 172.21.15.98:0/3784862758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66db40 0x55dbea674080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5d2c39640 1 -- 172.21.15.98:0/3784862758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66db40 msgr2=0x55dbea674080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5d2c39640 1 --2- 172.21.15.98:0/3784862758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66db40 0x55dbea674080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5d2c39640 1 -- 172.21.15.98:0/3784862758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea66c5d0 msgr2=0x55dbea676dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5d2c39640 1 --2- 172.21.15.98:0/3784862758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea66c5d0 0x55dbea676dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5d2c39640 1 -- 172.21.15.98:0/3784862758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbea3433c0 con 0x55dbea677370 2024-09-19T08:10:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5d2c39640 1 --2- 172.21.15.98:0/3784862758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea677370 0x55dbea674810 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fa5c402aeb0 tx=0x7fa5c40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9150b0fdcbe78406 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5b5ffb640 1 -- 172.21.15.98:0/3784862758 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c404f020 con 0x55dbea677370 2024-09-19T08:10:07.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5b5ffb640 1 -- 172.21.15.98:0/3784862758 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fa5c4004040 con 0x55dbea677370 2024-09-19T08:10:07.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3784862758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dbea5df860 con 0x55dbea677370 2024-09-19T08:10:07.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5b5ffb640 1 -- 172.21.15.98:0/3784862758 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa5c405a050 con 0x55dbea677370 2024-09-19T08:10:07.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5d6a75740 10 client.17184 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:07.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3784862758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dbea685b90 con 0x55dbea677370 2024-09-19T08:10:07.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.849+0000 7fa5b5ffb640 1 -- 172.21.15.98:0/3784862758 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55dbea5df860 con 0x55dbea677370 2024-09-19T08:10:07.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.850+0000 7fa5b5ffb640 1 -- 172.21.15.98:0/3784862758 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa5c404a070 con 0x55dbea677370 2024-09-19T08:10:07.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.850+0000 7fa5d6a75740 10 client.17184 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:07.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.850+0000 7fa5d6a75740 10 client.17184 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:10:07.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.850+0000 7fa5d6a75740 20 client.17184 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:07.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.850+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/3784862758 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55dbea6870a0 0x55dbea689500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.850+0000 7fa5d6a75740 4 client.17184 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:07.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.850+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3784862758 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbea5d9d70 con 0x55dbea6870a0 2024-09-19T08:10:07.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.850+0000 7fa5d2438640 1 --2- 172.21.15.98:0/3784862758 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55dbea6870a0 0x55dbea689500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.850+0000 7fa5d2438640 1 --2- 172.21.15.98:0/3784862758 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55dbea6870a0 0x55dbea689500 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5e6f93ec64f42f54 server_cookie=3c624fd0c343c6a7 in_seq=0 out_seq=0 2024-09-19T08:10:07.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.850+0000 7fa5b5ffb640 10 client.17184 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:10:07.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.851+0000 7fa5b5ffb640 1 -- 172.21.15.98:0/3784862758 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dbea5d9d70 con 0x55dbea6870a0 2024-09-19T08:10:07.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.851+0000 7fa5b5ffb640 10 client.17184 handle_command_reply: tid=0 2024-09-19T08:10:07.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.851+0000 7fa5d6a75740 1 client.17184 shutdown 2024-09-19T08:10:07.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.851+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3784862758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea677370 msgr2=0x55dbea674810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:07.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.851+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/3784862758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea677370 0x55dbea674810 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fa5c402aeb0 tx=0x7fa5c40095c0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.851+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3784862758 shutdown_connections 2024-09-19T08:10:07.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.851+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/3784862758 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55dbea6870a0 0x55dbea689500 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.851+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/3784862758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea677370 0x55dbea674810 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.851+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/3784862758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66db40 0x55dbea674080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.851+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/3784862758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea66c5d0 0x55dbea676dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.851+0000 7fa5b5ffb640 0 client.17184 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:10:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.851+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3784862758 >> 172.21.15.98:0/3784862758 conn(0x55dbea635310 msgr2=0x55dbea66e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.851+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3784862758 shutdown_connections 2024-09-19T08:10:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.851+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3784862758 wait complete. 2024-09-19T08:10:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.852+0000 7fa5d6a75740 20 client.17184 trim_cache size 0 max 16384 2024-09-19T08:10:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d6a75740 1 Processor -- start 2024-09-19T08:10:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d6a75740 1 -- start start 2024-09-19T08:10:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d6a75740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66c5d0 0x55dbea673940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d6a75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66db40 0x55dbea668700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d6a75740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea677370 0x55dbea668c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d6a75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1ada00 con 0x55dbea66db40 2024-09-19T08:10:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d6a75740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1af540 con 0x55dbea677370 2024-09-19T08:10:07.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d6a75740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1ae0d0 con 0x55dbea66c5d0 2024-09-19T08:10:07.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d1c37640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66db40 0x55dbea668700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d1c37640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66db40 0x55dbea668700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40748/0 (socket says 172.21.15.98:40748) 2024-09-19T08:10:07.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d1c37640 1 -- 172.21.15.98:0/3029913740 learned_addr learned my addr 172.21.15.98:0/3029913740 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:07.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d2438640 1 --2- 172.21.15.98:0/3029913740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66c5d0 0x55dbea673940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d2c39640 1 --2- 172.21.15.98:0/3029913740 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea677370 0x55dbea668c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d1c37640 1 -- 172.21.15.98:0/3029913740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66c5d0 msgr2=0x55dbea673940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:07.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d1c37640 1 --2- 172.21.15.98:0/3029913740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66c5d0 0x55dbea673940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d1c37640 1 -- 172.21.15.98:0/3029913740 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea677370 msgr2=0x55dbea668c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:07.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d1c37640 1 --2- 172.21.15.98:0/3029913740 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea677370 0x55dbea668c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d1c37640 1 -- 172.21.15.98:0/3029913740 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbea669180 con 0x55dbea66db40 2024-09-19T08:10:07.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d2c39640 1 --2- 172.21.15.98:0/3029913740 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea677370 0x55dbea668c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:07.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.865+0000 7fa5d1c37640 1 --2- 172.21.15.98:0/3029913740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66db40 0x55dbea668700 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc007920 tx=0x7fa5bc007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed85996a86239e0f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:07.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.866+0000 7fa5b7fff640 1 -- 172.21.15.98:0/3029913740 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc019070 con 0x55dbea66db40 2024-09-19T08:10:07.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.866+0000 7fa5b7fff640 1 -- 172.21.15.98:0/3029913740 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5bc013c30 con 0x55dbea66db40 2024-09-19T08:10:07.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.866+0000 7fa5b7fff640 1 -- 172.21.15.98:0/3029913740 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc014610 con 0x55dbea66db40 2024-09-19T08:10:07.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.866+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3029913740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66db40 msgr2=0x55dbea668700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:07.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.866+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/3029913740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66db40 0x55dbea668700 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc007920 tx=0x7fa5bc007de0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.866+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3029913740 shutdown_connections 2024-09-19T08:10:07.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.866+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/3029913740 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea677370 0x55dbea668c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.866+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/3029913740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66db40 0x55dbea668700 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.866+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/3029913740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea66c5d0 0x55dbea673940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.866+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3029913740 >> 172.21.15.98:0/3029913740 conn(0x55dbea635310 msgr2=0x55dbea672950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.866+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3029913740 shutdown_connections 2024-09-19T08:10:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.866+0000 7fa5d6a75740 1 -- 172.21.15.98:0/3029913740 wait complete. 2024-09-19T08:10:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d6a75740 1 -- start start 2024-09-19T08:10:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d6a75740 1 Processor -- start 2024-09-19T08:10:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d6a75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66c5d0 0x55dbea67f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d6a75740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea66db40 0x55dbea67f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d6a75740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea677370 0x55dbea6706b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d6a75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1aee70 con 0x55dbea66c5d0 2024-09-19T08:10:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d6a75740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1ae7a0 con 0x55dbea66db40 2024-09-19T08:10:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d6a75740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1ab9e0 con 0x55dbea677370 2024-09-19T08:10:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d6a75740 10 client.0 _ll_register_callbacks cb 0x7fa5c4009940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d2438640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66c5d0 0x55dbea67f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d2438640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66c5d0 0x55dbea67f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40752/0 (socket says 172.21.15.98:40752) 2024-09-19T08:10:07.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d2438640 1 -- 172.21.15.98:0/2421508804 learned_addr learned my addr 172.21.15.98:0/2421508804 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:07.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d2c39640 1 --2- 172.21.15.98:0/2421508804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea677370 0x55dbea6706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d2438640 1 -- 172.21.15.98:0/2421508804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea677370 msgr2=0x55dbea6706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:07.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d2438640 1 --2- 172.21.15.98:0/2421508804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea677370 0x55dbea6706b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d2438640 1 -- 172.21.15.98:0/2421508804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea66db40 msgr2=0x55dbea67f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:07.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d1c37640 1 --2- 172.21.15.98:0/2421508804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea66db40 0x55dbea67f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d2438640 1 --2- 172.21.15.98:0/2421508804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea66db40 0x55dbea67f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d2438640 1 -- 172.21.15.98:0/2421508804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbea670bf0 con 0x55dbea66c5d0 2024-09-19T08:10:07.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d1c37640 1 --2- 172.21.15.98:0/2421508804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea66db40 0x55dbea67f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:07.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5d2438640 1 --2- 172.21.15.98:0/2421508804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66c5d0 0x55dbea67f480 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8099730 tx=0x7fa5c8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de662fdc94260cf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:07.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.867+0000 7fa5b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:07.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5b5ffb640 1 -- 172.21.15.98:0/2421508804 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c80d6070 con 0x55dbea66c5d0 2024-09-19T08:10:07.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5b5ffb640 1 -- 172.21.15.98:0/2421508804 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fa5c8010ca0 con 0x55dbea66c5d0 2024-09-19T08:10:07.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5d6a75740 1 -- 172.21.15.98:0/2421508804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5bc013c30 con 0x55dbea66c5d0 2024-09-19T08:10:07.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5b5ffb640 1 -- 172.21.15.98:0/2421508804 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa5c801e040 con 0x55dbea66c5d0 2024-09-19T08:10:07.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5d6a75740 10 client.17190 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:07.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5d6a75740 1 -- 172.21.15.98:0/2421508804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dbea6f15a0 con 0x55dbea66c5d0 2024-09-19T08:10:07.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5b5ffb640 1 -- 172.21.15.98:0/2421508804 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5bc013c30 con 0x55dbea66c5d0 2024-09-19T08:10:07.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5b5ffb640 1 -- 172.21.15.98:0/2421508804 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa5c80cca60 con 0x55dbea66c5d0 2024-09-19T08:10:07.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5d6a75740 10 client.17190 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:07.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5d6a75740 10 client.17190 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:10:07.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5d6a75740 20 client.17190 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:07.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/2421508804 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55dbea6f1880 0x55dbea6f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:07.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5d6a75740 4 client.17190 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:10:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5d6a75740 1 -- 172.21.15.98:0/2421508804 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dbea556170 con 0x55dbea6f1880 2024-09-19T08:10:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.868+0000 7fa5d1c37640 1 --2- 172.21.15.98:0/2421508804 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55dbea6f1880 0x55dbea6f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5b5ffb640 10 client.17190 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:10:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5d1c37640 1 --2- 172.21.15.98:0/2421508804 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55dbea6f1880 0x55dbea6f3c70 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3bdf58c318fc73ba server_cookie=39b522f6df413ee4 in_seq=0 out_seq=0 2024-09-19T08:10:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5b5ffb640 1 -- 172.21.15.98:0/2421508804 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dbea556170 con 0x55dbea6f1880 2024-09-19T08:10:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5b5ffb640 10 client.17190 handle_command_reply: tid=0 2024-09-19T08:10:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5d6a75740 1 client.17190 shutdown 2024-09-19T08:10:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5d6a75740 1 -- 172.21.15.98:0/2421508804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66c5d0 msgr2=0x55dbea67f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/2421508804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66c5d0 0x55dbea67f480 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8099730 tx=0x7fa5c8000bc0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5d6a75740 1 -- 172.21.15.98:0/2421508804 shutdown_connections 2024-09-19T08:10:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/2421508804 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55dbea6f1880 0x55dbea6f3c70 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/2421508804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbea677370 0x55dbea6706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/2421508804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbea66db40 0x55dbea67f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5d6a75740 1 --2- 172.21.15.98:0/2421508804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbea66c5d0 0x55dbea67f480 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5b5ffb640 0 client.17190 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:10:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5d6a75740 1 -- 172.21.15.98:0/2421508804 >> 172.21.15.98:0/2421508804 conn(0x55dbea635310 msgr2=0x55dbea66f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5d6a75740 1 -- 172.21.15.98:0/2421508804 shutdown_connections 2024-09-19T08:10:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5d6a75740 1 -- 172.21.15.98:0/2421508804 wait complete. 2024-09-19T08:10:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.869+0000 7fa5d6a75740 20 client.17190 trim_cache size 0 max 16384 2024-09-19T08:10:07.887 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:10:07.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.870+0000 7fa5d469c640 1 -- 172.21.15.98:0/2714482710 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa5b80a33b0 msgr2=0x7fa5b80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:07.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.870+0000 7fa5d469c640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa5b80a33b0 0x7fa5b80a5870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009bf0 tx=0x7fa5c402a8c0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.870+0000 7fa5d469c640 1 -- 172.21.15.98:0/2714482710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5cc101de0 msgr2=0x7fa5cc1985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:07.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.870+0000 7fa5d469c640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5cc101de0 0x7fa5cc1985f0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800b4a0 tx=0x7fa5c800b960 comp rx=0 tx=0).stop 2024-09-19T08:10:07.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.870+0000 7fa5d469c640 1 -- 172.21.15.98:0/2714482710 shutdown_connections 2024-09-19T08:10:07.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.870+0000 7fa5d469c640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa5b80a33b0 0x7fa5b80a5870 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.870+0000 7fa5d469c640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc106f10 0x7fa5cc19cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.870+0000 7fa5d469c640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5cc102790 0x7fa5cc198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.870+0000 7fa5d469c640 1 --2- 172.21.15.98:0/2714482710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5cc101de0 0x7fa5cc1985f0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:07.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.870+0000 7fa5d469c640 1 -- 172.21.15.98:0/2714482710 >> 172.21.15.98:0/2714482710 conn(0x7fa5cc0fd8c0 msgr2=0x7fa5cc0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:07.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.870+0000 7fa5d469c640 1 -- 172.21.15.98:0/2714482710 shutdown_connections 2024-09-19T08:10:07.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:07.870+0000 7fa5d469c640 1 -- 172.21.15.98:0/2714482710 wait complete. 2024-09-19T08:10:08.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:10:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:08.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:08 smithi098 ceph-mon[31333]: pgmap v216: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-09-19T08:10:09.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:08 smithi162 ceph-mon[35260]: pgmap v216: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-09-19T08:10:09.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:08 smithi139 ceph-mon[34874]: pgmap v216: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-09-19T08:10:09.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.376+0000 7fcdd9e66640 1 -- 172.21.15.98:0/781688691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd4106f10 msgr2=0x7fcdd410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:09.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.376+0000 7fcdd9e66640 1 --2- 172.21.15.98:0/781688691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd4106f10 0x7fcdd410ea90 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009ca0 tx=0x7fcdcc02a8c0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.376+0000 7fcdd9e66640 1 -- 172.21.15.98:0/781688691 shutdown_connections 2024-09-19T08:10:09.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.376+0000 7fcdd9e66640 1 --2- 172.21.15.98:0/781688691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd4106f10 0x7fcdd410ea90 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.376+0000 7fcdd9e66640 1 --2- 172.21.15.98:0/781688691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdd4102790 0x7fcdd41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.376+0000 7fcdd9e66640 1 --2- 172.21.15.98:0/781688691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd4101de0 0x7fcdd41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.376+0000 7fcdd9e66640 1 -- 172.21.15.98:0/781688691 >> 172.21.15.98:0/781688691 conn(0x7fcdd40fd8c0 msgr2=0x7fcdd40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:09.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.376+0000 7fcdd9e66640 1 -- 172.21.15.98:0/781688691 shutdown_connections 2024-09-19T08:10:09.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.376+0000 7fcdd9e66640 1 -- 172.21.15.98:0/781688691 wait complete. 2024-09-19T08:10:09.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.377+0000 7fcdd9e66640 1 Processor -- start 2024-09-19T08:10:09.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.377+0000 7fcdd9e66640 1 -- start start 2024-09-19T08:10:09.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.377+0000 7fcdd9e66640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd4101de0 0x7fcdd4198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.377+0000 7fcdd9e66640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdd4102790 0x7fcdd41989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.377+0000 7fcdd9e66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd4106f10 0x7fcdd419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.377+0000 7fcdd9e66640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd4111f90 con 0x7fcdd4106f10 2024-09-19T08:10:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.377+0000 7fcdd9e66640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd4111e10 con 0x7fcdd4102790 2024-09-19T08:10:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.377+0000 7fcdd9e66640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd4112110 con 0x7fcdd4101de0 2024-09-19T08:10:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.377+0000 7fcdd3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd4106f10 0x7fcdd419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.377+0000 7fcdd3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd4106f10 0x7fcdd419cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40766/0 (socket says 172.21.15.98:40766) 2024-09-19T08:10:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.377+0000 7fcdd3fff640 1 -- 172.21.15.98:0/3619877557 learned_addr learned my addr 172.21.15.98:0/3619877557 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.377+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdd4102790 0x7fcdd41989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.378+0000 7fcdd37fe640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd4101de0 0x7fcdd4198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.378+0000 7fcdd3fff640 1 -- 172.21.15.98:0/3619877557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd4101de0 msgr2=0x7fcdd4198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:09.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.378+0000 7fcdd3fff640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd4101de0 0x7fcdd4198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.378+0000 7fcdd3fff640 1 -- 172.21.15.98:0/3619877557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdd4102790 msgr2=0x7fcdd41989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:09.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.378+0000 7fcdd3fff640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdd4102790 0x7fcdd41989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.378+0000 7fcdd3fff640 1 -- 172.21.15.98:0/3619877557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd419d420 con 0x7fcdd4106f10 2024-09-19T08:10:09.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.378+0000 7fcdd37fe640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd4101de0 0x7fcdd4198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:09.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.378+0000 7fcdd3fff640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd4106f10 0x7fcdd419cd40 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc031a20 tx=0x7fcdcc031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:09.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.378+0000 7fcdd0ff9640 1 -- 172.21.15.98:0/3619877557 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc003c30 con 0x7fcdd4106f10 2024-09-19T08:10:09.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.378+0000 7fcdd0ff9640 1 -- 172.21.15.98:0/3619877557 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdcc003dd0 con 0x7fcdd4106f10 2024-09-19T08:10:09.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.378+0000 7fcdd9e66640 1 -- 172.21.15.98:0/3619877557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd419d6b0 con 0x7fcdd4106f10 2024-09-19T08:10:09.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.378+0000 7fcdd0ff9640 1 -- 172.21.15.98:0/3619877557 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc02fcf0 con 0x7fcdd4106f10 2024-09-19T08:10:09.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.378+0000 7fcdd9e66640 1 -- 172.21.15.98:0/3619877557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd41a4ff0 con 0x7fcdd4106f10 2024-09-19T08:10:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.380+0000 7fcdd0ff9640 1 -- 172.21.15.98:0/3619877557 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fcdcc004050 con 0x7fcdd4106f10 2024-09-19T08:10:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.380+0000 7fcddc23f740 1 Processor -- start 2024-09-19T08:10:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcddc23f740 1 -- start start 2024-09-19T08:10:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcddc23f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fcbb40 0x5653c9fca090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcddc23f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fca5d0 0x5653c9fcaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcddc23f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fdbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcddc23f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5653c9f302f0 con 0x5653c9fd9780 2024-09-19T08:10:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcddc23f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5653c9ed7cf0 con 0x5653c9fcbb40 2024-09-19T08:10:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcddc23f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5653c9f2c2b0 con 0x5653c9fca5d0 2024-09-19T08:10:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fdbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fdbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40774/0 (socket says 172.21.15.98:40774) 2024-09-19T08:10:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/656543623 learned_addr learned my addr 172.21.15.98:0/656543623 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/656543623 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fcbb40 0x5653c9fca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcdd0ff9640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcdbc0a3430 0x7fcdbc0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/656543623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fca5d0 msgr2=0x5653c9fcaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/656543623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fca5d0 0x5653c9fcaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/656543623 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fcbb40 msgr2=0x5653c9fca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:09.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcdd3fff640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcdbc0a3430 0x7fcdbc0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/656543623 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fcbb40 0x5653c9fca090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/656543623 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653c9ebc3a0 con 0x5653c9fd9780 2024-09-19T08:10:09.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcdd0ff9640 1 -- 172.21.15.98:0/3619877557 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fcdcc0f0ff0 con 0x7fcdd4106f10 2024-09-19T08:10:09.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.381+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/656543623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fdbb70 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800ae00 tx=0x7fcdc800a2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d4b49991e76c00 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:09.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.382+0000 7fcdb9ffb640 1 -- 172.21.15.98:0/656543623 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc801b070 con 0x5653c9fd9780 2024-09-19T08:10:09.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.382+0000 7fcdb9ffb640 1 -- 172.21.15.98:0/656543623 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdc8011bf0 con 0x5653c9fd9780 2024-09-19T08:10:09.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.382+0000 7fcdb9ffb640 1 -- 172.21.15.98:0/656543623 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8014650 con 0x5653c9fd9780 2024-09-19T08:10:09.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.382+0000 7fcddc23f740 1 -- 172.21.15.98:0/656543623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 msgr2=0x5653c9fdbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:09.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.382+0000 7fcddc23f740 1 --2- 172.21.15.98:0/656543623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fdbb70 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800ae00 tx=0x7fcdc800a2b0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.382+0000 7fcdd3fff640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcdbc0a3430 0x7fcdbc0a58f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc0bd280 tx=0x7fcdcc08f420 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:09.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.383+0000 7fcddc23f740 1 -- 172.21.15.98:0/656543623 shutdown_connections 2024-09-19T08:10:09.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.383+0000 7fcddc23f740 1 --2- 172.21.15.98:0/656543623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fdbb70 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.383+0000 7fcddc23f740 1 --2- 172.21.15.98:0/656543623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fca5d0 0x5653c9fcaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.383+0000 7fcddc23f740 1 --2- 172.21.15.98:0/656543623 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fcbb40 0x5653c9fca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.383+0000 7fcddc23f740 1 -- 172.21.15.98:0/656543623 >> 172.21.15.98:0/656543623 conn(0x5653c9f93310 msgr2=0x5653c9fc6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:09.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.383+0000 7fcddc23f740 1 -- 172.21.15.98:0/656543623 shutdown_connections 2024-09-19T08:10:09.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.383+0000 7fcddc23f740 1 -- 172.21.15.98:0/656543623 wait complete. 2024-09-19T08:10:09.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.383+0000 7fcddc23f740 1 -- start start 2024-09-19T08:10:09.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.383+0000 7fcddc23f740 1 Processor -- start 2024-09-19T08:10:09.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.383+0000 7fcddc23f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 0x5653c9fd91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcddc23f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fcbb40 0x5653c9fd6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcddc23f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcddc23f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5653c9fdc480 con 0x5653c9fd9780 2024-09-19T08:10:09.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcddc23f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5653c9eeb060 con 0x5653c9fca5d0 2024-09-19T08:10:09.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcddc23f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5653c9fdc600 con 0x5653c9fcbb40 2024-09-19T08:10:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcddc23f740 10 client.0 _ll_register_callbacks cb 0x5653c9c3e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40780/0 (socket says 172.21.15.98:40780) 2024-09-19T08:10:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/871550040 learned_addr learned my addr 172.21.15.98:0/871550040 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/871550040 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 0x5653c9fd91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdd37fe640 1 --2- 172.21.15.98:0/871550040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fcbb40 0x5653c9fd6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/871550040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fcbb40 msgr2=0x5653c9fd6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/871550040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fcbb40 0x5653c9fd6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/871550040 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 msgr2=0x5653c9fd91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/871550040 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 0x5653c9fd91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/871550040 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653c9ca13c0 con 0x5653c9fd9780 2024-09-19T08:10:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/871550040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd6c10 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800afd0 tx=0x7fcdc800a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f584af89e1900f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdb2ffd640 1 -- 172.21.15.98:0/871550040 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc802a060 con 0x5653c9fd9780 2024-09-19T08:10:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdb2ffd640 1 -- 172.21.15.98:0/871550040 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fcdc8025620 con 0x5653c9fd9780 2024-09-19T08:10:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcddc23f740 1 -- 172.21.15.98:0/871550040 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdc8011bf0 con 0x5653c9fd9780 2024-09-19T08:10:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcdb2ffd640 1 -- 172.21.15.98:0/871550040 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcdc801b070 con 0x5653c9fd9780 2024-09-19T08:10:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcddc23f740 10 client.17202 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.384+0000 7fcddc23f740 1 -- 172.21.15.98:0/871550040 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5653c9fe7f90 con 0x5653c9fd9780 2024-09-19T08:10:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.385+0000 7fcdb2ffd640 1 -- 172.21.15.98:0/871550040 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdc8011bf0 con 0x5653c9fd9780 2024-09-19T08:10:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.385+0000 7fcdb2ffd640 1 -- 172.21.15.98:0/871550040 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcdc8015040 con 0x5653c9fd9780 2024-09-19T08:10:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.385+0000 7fcddc23f740 10 client.17202 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.385+0000 7fcddc23f740 10 client.17202 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.385+0000 7fcddc23f740 20 client.17202 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.385+0000 7fcddc23f740 1 --2- 172.21.15.98:0/871550040 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5653c9fe9340 0x5653c9feb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.385+0000 7fcddc23f740 4 client.17202 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.385+0000 7fcddc23f740 1 -- 172.21.15.98:0/871550040 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653c9f3d860 con 0x5653c9fe9340 2024-09-19T08:10:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.385+0000 7fcdd37fe640 1 --2- 172.21.15.98:0/871550040 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5653c9fe9340 0x5653c9feb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.386+0000 7fcdd37fe640 1 --2- 172.21.15.98:0/871550040 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5653c9fe9340 0x5653c9feb730 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bed20b52a4858feb server_cookie=74b1070fd1cafa91 in_seq=0 out_seq=0 2024-09-19T08:10:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.386+0000 7fcdb2ffd640 10 client.17202 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.386+0000 7fcdb2ffd640 1 -- 172.21.15.98:0/871550040 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5653c9f3d860 con 0x5653c9fe9340 2024-09-19T08:10:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.386+0000 7fcdb2ffd640 10 client.17202 handle_command_reply: tid=0 2024-09-19T08:10:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.386+0000 7fcddc23f740 1 client.17202 shutdown 2024-09-19T08:10:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.387+0000 7fcddc23f740 1 -- 172.21.15.98:0/871550040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 msgr2=0x5653c9fd6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.387+0000 7fcddc23f740 1 --2- 172.21.15.98:0/871550040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd6c10 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800afd0 tx=0x7fcdc800a1f0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.387+0000 7fcddc23f740 1 -- 172.21.15.98:0/871550040 shutdown_connections 2024-09-19T08:10:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.387+0000 7fcddc23f740 1 --2- 172.21.15.98:0/871550040 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5653c9fe9340 0x5653c9feb730 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.387+0000 7fcdb2ffd640 0 client.17202 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.387+0000 7fcddc23f740 1 --2- 172.21.15.98:0/871550040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd6c10 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.387+0000 7fcddc23f740 1 --2- 172.21.15.98:0/871550040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fcbb40 0x5653c9fd6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.387+0000 7fcddc23f740 1 --2- 172.21.15.98:0/871550040 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 0x5653c9fd91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.387+0000 7fcddc23f740 1 -- 172.21.15.98:0/871550040 >> 172.21.15.98:0/871550040 conn(0x5653c9f93310 msgr2=0x5653c9fcc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.387+0000 7fcddc23f740 1 -- 172.21.15.98:0/871550040 shutdown_connections 2024-09-19T08:10:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.387+0000 7fcddc23f740 1 -- 172.21.15.98:0/871550040 wait complete. 2024-09-19T08:10:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.387+0000 7fcddc23f740 20 client.17202 trim_cache size 0 max 16384 2024-09-19T08:10:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.399+0000 7fcddc23f740 1 Processor -- start 2024-09-19T08:10:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.399+0000 7fcddc23f740 1 -- start start 2024-09-19T08:10:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcddc23f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 0x5653c9fc6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcddc23f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fcbb40 0x5653c9fd5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcddc23f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcddc23f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd41ab600 con 0x5653c9fd9780 2024-09-19T08:10:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcddc23f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd41ad140 con 0x5653c9fca5d0 2024-09-19T08:10:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcddc23f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd41abcd0 con 0x5653c9fcbb40 2024-09-19T08:10:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40784/0 (socket says 172.21.15.98:40784) 2024-09-19T08:10:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/106151847 learned_addr learned my addr 172.21.15.98:0/106151847 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/106151847 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 0x5653c9fc6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/106151847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fcbb40 msgr2=0x5653c9fd5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/106151847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fcbb40 0x5653c9fd5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/106151847 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 msgr2=0x5653c9fc6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/106151847 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 0x5653c9fc6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/106151847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653c9febaa0 con 0x5653c9fd9780 2024-09-19T08:10:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/106151847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd5460 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8014cc0 tx=0x7fcdc800a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dacd47afcb0939cd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcdb97fa640 1 -- 172.21.15.98:0/106151847 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc802a070 con 0x5653c9fd9780 2024-09-19T08:10:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcdb97fa640 1 -- 172.21.15.98:0/106151847 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdc800a5c0 con 0x5653c9fd9780 2024-09-19T08:10:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.400+0000 7fcdb97fa640 1 -- 172.21.15.98:0/106151847 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc802f6e0 con 0x5653c9fd9780 2024-09-19T08:10:09.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.401+0000 7fcddc23f740 1 -- 172.21.15.98:0/106151847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 msgr2=0x5653c9fd5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:09.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.401+0000 7fcddc23f740 1 --2- 172.21.15.98:0/106151847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd5460 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8014cc0 tx=0x7fcdc800a4c0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.401+0000 7fcddc23f740 1 -- 172.21.15.98:0/106151847 shutdown_connections 2024-09-19T08:10:09.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.401+0000 7fcddc23f740 1 --2- 172.21.15.98:0/106151847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd5460 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.401+0000 7fcddc23f740 1 --2- 172.21.15.98:0/106151847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fcbb40 0x5653c9fd5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.401+0000 7fcddc23f740 1 --2- 172.21.15.98:0/106151847 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 0x5653c9fc6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.401+0000 7fcddc23f740 1 -- 172.21.15.98:0/106151847 >> 172.21.15.98:0/106151847 conn(0x5653c9f93310 msgr2=0x5653c9fd4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:09.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.401+0000 7fcddc23f740 1 -- 172.21.15.98:0/106151847 shutdown_connections 2024-09-19T08:10:09.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.401+0000 7fcddc23f740 1 -- 172.21.15.98:0/106151847 wait complete. 2024-09-19T08:10:09.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.401+0000 7fcddc23f740 1 -- start start 2024-09-19T08:10:09.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcddc23f740 1 Processor -- start 2024-09-19T08:10:09.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcddc23f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 0x5653c9fe1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcddc23f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fcbb40 0x5653c9fe1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcddc23f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcddc23f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd41aca70 con 0x5653c9fd9780 2024-09-19T08:10:09.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcddc23f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd41ac3a0 con 0x5653c9fca5d0 2024-09-19T08:10:09.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcddc23f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd41a95e0 con 0x5653c9fcbb40 2024-09-19T08:10:09.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcddc23f740 10 client.0 _ll_register_callbacks cb 0x7fcdc8019070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:09.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40794/0 (socket says 172.21.15.98:40794) 2024-09-19T08:10:09.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/2094995641 learned_addr learned my addr 172.21.15.98:0/2094995641 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:09.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcdd37fe640 1 --2- 172.21.15.98:0/2094995641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fcbb40 0x5653c9fe1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/2094995641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 0x5653c9fe1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/2094995641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fcbb40 msgr2=0x5653c9fe1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/2094995641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fcbb40 0x5653c9fe1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/2094995641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 msgr2=0x5653c9fe1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/2094995641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 0x5653c9fe1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcdd2ffd640 1 -- 172.21.15.98:0/2094995641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653c9fd9540 con 0x5653c9fd9780 2024-09-19T08:10:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcdd2ffd640 1 --2- 172.21.15.98:0/2094995641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd2c90 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8000be0 tx=0x7fcdc80145b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a996c6adad42f1cc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.402+0000 7fcdb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.403+0000 7fcdb2ffd640 1 -- 172.21.15.98:0/2094995641 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc803f070 con 0x5653c9fd9780 2024-09-19T08:10:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.403+0000 7fcdb2ffd640 1 -- 172.21.15.98:0/2094995641 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8477+0+0 (secure 0 0 0) 0x7fcdc8016be0 con 0x5653c9fd9780 2024-09-19T08:10:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.403+0000 7fcddc23f740 1 -- 172.21.15.98:0/2094995641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdc800a5c0 con 0x5653c9fd9780 2024-09-19T08:10:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.403+0000 7fcdb2ffd640 1 -- 172.21.15.98:0/2094995641 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcdc802a070 con 0x5653c9fd9780 2024-09-19T08:10:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.403+0000 7fcddc23f740 10 client.17208 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.403+0000 7fcddc23f740 1 -- 172.21.15.98:0/2094995641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5653ca053560 con 0x5653c9fd9780 2024-09-19T08:10:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.403+0000 7fcdb2ffd640 1 -- 172.21.15.98:0/2094995641 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdc800a5c0 con 0x5653c9fd9780 2024-09-19T08:10:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.403+0000 7fcdb2ffd640 1 -- 172.21.15.98:0/2094995641 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcdc80454d0 con 0x5653c9fd9780 2024-09-19T08:10:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.403+0000 7fcddc23f740 10 client.17208 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.403+0000 7fcddc23f740 10 client.17208 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.403+0000 7fcddc23f740 20 client.17208 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.403+0000 7fcddc23f740 1 --2- 172.21.15.98:0/2094995641 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5653c9fe9340 0x5653ca055a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.403+0000 7fcddc23f740 4 client.17208 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:10:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.403+0000 7fcddc23f740 1 -- 172.21.15.98:0/2094995641 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5653c9f37d70 con 0x5653c9fe9340 2024-09-19T08:10:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.404+0000 7fcdd37fe640 1 --2- 172.21.15.98:0/2094995641 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5653c9fe9340 0x5653ca055a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.404+0000 7fcdb2ffd640 10 client.17208 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.404+0000 7fcdd37fe640 1 --2- 172.21.15.98:0/2094995641 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5653c9fe9340 0x5653ca055a40 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14728f3e886aba31 server_cookie=713524a56b1f04b5 in_seq=0 out_seq=0 2024-09-19T08:10:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.404+0000 7fcdb2ffd640 1 -- 172.21.15.98:0/2094995641 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5653c9f37d70 con 0x5653c9fe9340 2024-09-19T08:10:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.404+0000 7fcdb2ffd640 10 client.17208 handle_command_reply: tid=0 2024-09-19T08:10:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.404+0000 7fcddc23f740 1 client.17208 shutdown 2024-09-19T08:10:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcddc23f740 1 -- 172.21.15.98:0/2094995641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 msgr2=0x5653c9fd2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcddc23f740 1 --2- 172.21.15.98:0/2094995641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd2c90 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8000be0 tx=0x7fcdc80145b0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcddc23f740 1 -- 172.21.15.98:0/2094995641 shutdown_connections 2024-09-19T08:10:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcddc23f740 1 --2- 172.21.15.98:0/2094995641 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5653c9fe9340 0x5653ca055a40 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcddc23f740 1 --2- 172.21.15.98:0/2094995641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5653c9fd9780 0x5653c9fd2c90 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcddc23f740 1 --2- 172.21.15.98:0/2094995641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5653c9fcbb40 0x5653c9fe1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcddc23f740 1 --2- 172.21.15.98:0/2094995641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5653c9fca5d0 0x5653c9fe1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcddc23f740 1 -- 172.21.15.98:0/2094995641 >> 172.21.15.98:0/2094995641 conn(0x5653c9f93310 msgr2=0x5653c9fd5570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcdb2ffd640 0 client.17208 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcddc23f740 1 -- 172.21.15.98:0/2094995641 shutdown_connections 2024-09-19T08:10:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcddc23f740 1 -- 172.21.15.98:0/2094995641 wait complete. 2024-09-19T08:10:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcddc23f740 20 client.17208 trim_cache size 0 max 16384 2024-09-19T08:10:09.411 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:10:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcdd9e66640 1 -- 172.21.15.98:0/3619877557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcdbc0a3430 msgr2=0x7fcdbc0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcdd9e66640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcdbc0a3430 0x7fcdbc0a58f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc0bd280 tx=0x7fcdcc08f420 comp rx=0 tx=0).stop 2024-09-19T08:10:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcdd9e66640 1 -- 172.21.15.98:0/3619877557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd4106f10 msgr2=0x7fcdd419cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcdd9e66640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd4106f10 0x7fcdd419cd40 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc031a20 tx=0x7fcdcc031a50 comp rx=0 tx=0).stop 2024-09-19T08:10:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.405+0000 7fcdd9e66640 1 -- 172.21.15.98:0/3619877557 shutdown_connections 2024-09-19T08:10:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.406+0000 7fcdd9e66640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcdbc0a3430 0x7fcdbc0a58f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.406+0000 7fcdd9e66640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd4106f10 0x7fcdd419cd40 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.406+0000 7fcdd9e66640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdd4102790 0x7fcdd41989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.406+0000 7fcdd9e66640 1 --2- 172.21.15.98:0/3619877557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd4101de0 0x7fcdd4198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.406+0000 7fcdd9e66640 1 -- 172.21.15.98:0/3619877557 >> 172.21.15.98:0/3619877557 conn(0x7fcdd40fd8c0 msgr2=0x7fcdd40fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.406+0000 7fcdd9e66640 1 -- 172.21.15.98:0/3619877557 shutdown_connections 2024-09-19T08:10:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:09.406+0000 7fcdd9e66640 1 -- 172.21.15.98:0/3619877557 wait complete. 2024-09-19T08:10:11.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:10 smithi162 ceph-mon[35260]: pgmap v217: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:10:11.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:10 smithi139 ceph-mon[34874]: pgmap v217: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:10:11.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:10 smithi098 ceph-mon[31333]: pgmap v217: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:10:11.246 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:10:11.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:10:11.500 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:12.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.014+0000 7fdc021e3640 1 -- 172.21.15.98:0/2289974958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc100a80 msgr2=0x7fdbfc100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:12.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.014+0000 7fdc021e3640 1 --2- 172.21.15.98:0/2289974958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc100a80 0x7fdbfc100ee0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009930 tx=0x7fdbec02a420 comp rx=0 tx=0).stop 2024-09-19T08:10:12.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.014+0000 7fdc021e3640 1 -- 172.21.15.98:0/2289974958 shutdown_connections 2024-09-19T08:10:12.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.014+0000 7fdc021e3640 1 --2- 172.21.15.98:0/2289974958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbfc101420 0x7fdbfc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.014+0000 7fdc021e3640 1 --2- 172.21.15.98:0/2289974958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc100a80 0x7fdbfc100ee0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.014+0000 7fdc021e3640 1 --2- 172.21.15.98:0/2289974958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbfc106a80 0x7fdbfc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.014+0000 7fdc021e3640 1 -- 172.21.15.98:0/2289974958 >> 172.21.15.98:0/2289974958 conn(0x7fdbfc0fc7a0 msgr2=0x7fdbfc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:12.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.014+0000 7fdc021e3640 1 -- 172.21.15.98:0/2289974958 shutdown_connections 2024-09-19T08:10:12.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.014+0000 7fdc021e3640 1 -- 172.21.15.98:0/2289974958 wait complete. 2024-09-19T08:10:12.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.014+0000 7fdc021e3640 1 Processor -- start 2024-09-19T08:10:12.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdc021e3640 1 -- start start 2024-09-19T08:10:12.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdc021e3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbfc100a80 0x7fdbfc197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdc021e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc101420 0x7fdbfc197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdc021e3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbfc106a80 0x7fdbfc19bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdc021e3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc110e80 con 0x7fdbfc101420 2024-09-19T08:10:12.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdc021e3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc110d00 con 0x7fdbfc106a80 2024-09-19T08:10:12.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdc021e3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc111000 con 0x7fdbfc100a80 2024-09-19T08:10:12.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdbfaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc101420 0x7fdbfc197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdbfaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc101420 0x7fdbfc197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40820/0 (socket says 172.21.15.98:40820) 2024-09-19T08:10:12.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdbfaffd640 1 -- 172.21.15.98:0/3632107204 learned_addr learned my addr 172.21.15.98:0/3632107204 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:12.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdbfb7fe640 1 --2- 172.21.15.98:0/3632107204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbfc100a80 0x7fdbfc197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdbfbfff640 1 --2- 172.21.15.98:0/3632107204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbfc106a80 0x7fdbfc19bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdbfaffd640 1 -- 172.21.15.98:0/3632107204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbfc100a80 msgr2=0x7fdbfc197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:12.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdbfaffd640 1 --2- 172.21.15.98:0/3632107204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbfc100a80 0x7fdbfc197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdbfaffd640 1 -- 172.21.15.98:0/3632107204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbfc106a80 msgr2=0x7fdbfc19bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:12.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdbfaffd640 1 --2- 172.21.15.98:0/3632107204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbfc106a80 0x7fdbfc19bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.015+0000 7fdbfaffd640 1 -- 172.21.15.98:0/3632107204 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbfc19c2b0 con 0x7fdbfc101420 2024-09-19T08:10:12.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.016+0000 7fdbfaffd640 1 --2- 172.21.15.98:0/3632107204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc101420 0x7fdbfc197840 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009a50 tx=0x7fdbec004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:12.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.016+0000 7fdbf8ff9640 1 -- 172.21.15.98:0/3632107204 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec03f030 con 0x7fdbfc101420 2024-09-19T08:10:12.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.016+0000 7fdc021e3640 1 -- 172.21.15.98:0/3632107204 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbfc19c540 con 0x7fdbfc101420 2024-09-19T08:10:12.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.016+0000 7fdbf8ff9640 1 -- 172.21.15.98:0/3632107204 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbec003ad0 con 0x7fdbfc101420 2024-09-19T08:10:12.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.016+0000 7fdbf8ff9640 1 -- 172.21.15.98:0/3632107204 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec035070 con 0x7fdbfc101420 2024-09-19T08:10:12.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.016+0000 7fdc021e3640 1 -- 172.21.15.98:0/3632107204 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbfc1a3e80 con 0x7fdbfc101420 2024-09-19T08:10:12.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.019+0000 7fdbf8ff9640 1 -- 172.21.15.98:0/3632107204 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fdbec042030 con 0x7fdbfc101420 2024-09-19T08:10:12.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.019+0000 7fdc045bc740 1 Processor -- start 2024-09-19T08:10:12.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.019+0000 7fdc045bc740 1 -- start start 2024-09-19T08:10:12.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.019+0000 7fdc045bc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa569a0 0x561c1aa56d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.019+0000 7fdc045bc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa572c0 0x561c1aa63a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.019+0000 7fdc045bc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa63f50 0x561c1aa66340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.019+0000 7fdc045bc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561c1a9bc280 con 0x561c1aa63f50 2024-09-19T08:10:12.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.019+0000 7fdc045bc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561c1a963b10 con 0x561c1aa569a0 2024-09-19T08:10:12.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.019+0000 7fdc045bc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561c1a9b8240 con 0x561c1aa572c0 2024-09-19T08:10:12.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.019+0000 7fdbfaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa63f50 0x561c1aa66340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.019+0000 7fdbfbfff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa569a0 0x561c1aa56d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.019+0000 7fdbfaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa63f50 0x561c1aa66340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40828/0 (socket says 172.21.15.98:40828) 2024-09-19T08:10:12.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.019+0000 7fdbfaffd640 1 -- 172.21.15.98:0/1998576800 learned_addr learned my addr 172.21.15.98:0/1998576800 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:12.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.019+0000 7fdbfb7fe640 1 --2- 172.21.15.98:0/1998576800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa572c0 0x561c1aa63a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.020+0000 7fdbf8ff9640 1 --2- 172.21.15.98:0/3632107204 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdbe80a33b0 0x7fdbe80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.020+0000 7fdbfaffd640 1 -- 172.21.15.98:0/1998576800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa572c0 msgr2=0x561c1aa63a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:12.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.020+0000 7fdbfaffd640 1 --2- 172.21.15.98:0/1998576800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa572c0 0x561c1aa63a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.020+0000 7fdbfaffd640 1 -- 172.21.15.98:0/1998576800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa569a0 msgr2=0x561c1aa56d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:12.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.020+0000 7fdbfaffd640 1 --2- 172.21.15.98:0/1998576800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa569a0 0x561c1aa56d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.020+0000 7fdbfbfff640 1 --2- 172.21.15.98:0/3632107204 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdbe80a33b0 0x7fdbe80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.020+0000 7fdbfaffd640 1 -- 172.21.15.98:0/1998576800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c1a9701f0 con 0x561c1aa63f50 2024-09-19T08:10:12.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.020+0000 7fdbf8ff9640 1 -- 172.21.15.98:0/3632107204 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8565+0+0 (secure 0 0 0) 0x7fdbec0f0530 con 0x7fdbfc101420 2024-09-19T08:10:12.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.020+0000 7fdbfb7fe640 1 --2- 172.21.15.98:0/1998576800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa572c0 0x561c1aa63a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:10:12.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.020+0000 7fdbfaffd640 1 --2- 172.21.15.98:0/1998576800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa63f50 0x561c1aa66340 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0edbc0 tx=0x7fdbec009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e47aaaae50d01ccb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:12.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.020+0000 7fdbfbfff640 1 --2- 172.21.15.98:0/3632107204 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdbe80a33b0 0x7fdbe80a5870 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4008e80 tx=0x7fdbf4004a40 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:12.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.020+0000 7fdbf1ffb640 1 -- 172.21.15.98:0/1998576800 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec049020 con 0x561c1aa63f50 2024-09-19T08:10:12.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.020+0000 7fdbf1ffb640 1 -- 172.21.15.98:0/1998576800 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbec0bcee0 con 0x561c1aa63f50 2024-09-19T08:10:12.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.020+0000 7fdbf1ffb640 1 -- 172.21.15.98:0/1998576800 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec0c0b90 con 0x561c1aa63f50 2024-09-19T08:10:12.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.021+0000 7fdc045bc740 1 -- 172.21.15.98:0/1998576800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa63f50 msgr2=0x561c1aa66340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:12.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.021+0000 7fdc045bc740 1 --2- 172.21.15.98:0/1998576800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa63f50 0x561c1aa66340 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0edbc0 tx=0x7fdbec009680 comp rx=0 tx=0).stop 2024-09-19T08:10:12.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.021+0000 7fdc045bc740 1 -- 172.21.15.98:0/1998576800 shutdown_connections 2024-09-19T08:10:12.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.021+0000 7fdc045bc740 1 --2- 172.21.15.98:0/1998576800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa63f50 0x561c1aa66340 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.021+0000 7fdc045bc740 1 --2- 172.21.15.98:0/1998576800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa572c0 0x561c1aa63a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.021+0000 7fdc045bc740 1 --2- 172.21.15.98:0/1998576800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa569a0 0x561c1aa56d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.021+0000 7fdc045bc740 1 -- 172.21.15.98:0/1998576800 >> 172.21.15.98:0/1998576800 conn(0x561c1aa530f0 msgr2=0x561c1aa534d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:12.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.021+0000 7fdc045bc740 1 -- 172.21.15.98:0/1998576800 shutdown_connections 2024-09-19T08:10:12.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.021+0000 7fdc045bc740 1 -- 172.21.15.98:0/1998576800 wait complete. 2024-09-19T08:10:12.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.021+0000 7fdc045bc740 1 -- start start 2024-09-19T08:10:12.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdc045bc740 1 Processor -- start 2024-09-19T08:10:12.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdc045bc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa569a0 0x561c1aa60e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdc045bc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa572c0 0x561c1aa61370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdc045bc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa63f50 0x561c1aa618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdc045bc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561c1aa668e0 con 0x561c1aa569a0 2024-09-19T08:10:12.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdc045bc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561c1a976f10 con 0x561c1aa63f50 2024-09-19T08:10:12.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdc045bc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561c1aa66a60 con 0x561c1aa572c0 2024-09-19T08:10:12.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdc045bc740 10 client.0 _ll_register_callbacks cb 0x561c1a93fa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:12.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdbfb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa569a0 0x561c1aa60e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdbfb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa569a0 0x561c1aa60e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40844/0 (socket says 172.21.15.98:40844) 2024-09-19T08:10:12.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdbfb7fe640 1 -- 172.21.15.98:0/1350128378 learned_addr learned my addr 172.21.15.98:0/1350128378 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:12.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdbfaffd640 1 --2- 172.21.15.98:0/1350128378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa572c0 0x561c1aa61370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdbfbfff640 1 --2- 172.21.15.98:0/1350128378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa63f50 0x561c1aa618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdbfb7fe640 1 -- 172.21.15.98:0/1350128378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa572c0 msgr2=0x561c1aa61370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:12.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdbfb7fe640 1 --2- 172.21.15.98:0/1350128378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa572c0 0x561c1aa61370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdbfb7fe640 1 -- 172.21.15.98:0/1350128378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa63f50 msgr2=0x561c1aa618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:12.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdbfb7fe640 1 --2- 172.21.15.98:0/1350128378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa63f50 0x561c1aa618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdbfb7fe640 1 -- 172.21.15.98:0/1350128378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c1a72d6a0 con 0x561c1aa569a0 2024-09-19T08:10:12.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdbfb7fe640 1 --2- 172.21.15.98:0/1350128378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa569a0 0x561c1aa60e30 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4010a50 tx=0x7fdbe4010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e4b598450b4cdc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:12.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.022+0000 7fdbdaffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:12.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.023+0000 7fdbdaffd640 1 -- 172.21.15.98:0/1350128378 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe401e060 con 0x561c1aa569a0 2024-09-19T08:10:12.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.023+0000 7fdbdaffd640 1 -- 172.21.15.98:0/1350128378 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8565+0+0 (secure 0 0 0) 0x7fdbe400fb10 con 0x561c1aa569a0 2024-09-19T08:10:12.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.023+0000 7fdc045bc740 1 -- 172.21.15.98:0/1350128378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbec0bcee0 con 0x561c1aa569a0 2024-09-19T08:10:12.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.023+0000 7fdbdaffd640 1 -- 172.21.15.98:0/1350128378 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdbe400fd10 con 0x561c1aa569a0 2024-09-19T08:10:12.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.023+0000 7fdc045bc740 10 client.17220 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:12.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.023+0000 7fdc045bc740 1 -- 172.21.15.98:0/1350128378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561c1aa724c0 con 0x561c1aa569a0 2024-09-19T08:10:12.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.023+0000 7fdbdaffd640 1 -- 172.21.15.98:0/1350128378 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbec0bcee0 con 0x561c1aa569a0 2024-09-19T08:10:12.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.023+0000 7fdbdaffd640 1 -- 172.21.15.98:0/1350128378 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fdbe400e590 con 0x561c1aa569a0 2024-09-19T08:10:12.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.023+0000 7fdc045bc740 10 client.17220 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:12.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.023+0000 7fdc045bc740 10 client.17220 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:12.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.023+0000 7fdc045bc740 20 client.17220 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:12.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.023+0000 7fdc045bc740 1 --2- 172.21.15.98:0/1350128378 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c1aa73980 0x561c1aa75d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.023+0000 7fdc045bc740 4 client.17220 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:12.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.023+0000 7fdc045bc740 1 -- 172.21.15.98:0/1350128378 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c1a9c97f0 con 0x561c1aa73980 2024-09-19T08:10:12.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.024+0000 7fdbfaffd640 1 --2- 172.21.15.98:0/1350128378 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c1aa73980 0x561c1aa75d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.024+0000 7fdbfaffd640 1 --2- 172.21.15.98:0/1350128378 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c1aa73980 0x561c1aa75d70 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96e48d92928650d server_cookie=744f40bd2d0b2d2e in_seq=0 out_seq=0 2024-09-19T08:10:12.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.024+0000 7fdbdaffd640 10 client.17220 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:12.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdbdaffd640 1 -- 172.21.15.98:0/1350128378 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561c1a9c97f0 con 0x561c1aa73980 2024-09-19T08:10:12.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdbdaffd640 10 client.17220 handle_command_reply: tid=0 2024-09-19T08:10:12.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdc045bc740 1 client.17220 shutdown 2024-09-19T08:10:12.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdc045bc740 1 -- 172.21.15.98:0/1350128378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa569a0 msgr2=0x561c1aa60e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:12.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdc045bc740 1 --2- 172.21.15.98:0/1350128378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa569a0 0x561c1aa60e30 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4010a50 tx=0x7fdbe4010f10 comp rx=0 tx=0).stop 2024-09-19T08:10:12.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdc045bc740 1 -- 172.21.15.98:0/1350128378 shutdown_connections 2024-09-19T08:10:12.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdc045bc740 1 --2- 172.21.15.98:0/1350128378 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c1aa73980 0x561c1aa75d70 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdc045bc740 1 --2- 172.21.15.98:0/1350128378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa63f50 0x561c1aa618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdc045bc740 1 --2- 172.21.15.98:0/1350128378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa572c0 0x561c1aa61370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdc045bc740 1 --2- 172.21.15.98:0/1350128378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa569a0 0x561c1aa60e30 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdc045bc740 1 -- 172.21.15.98:0/1350128378 >> 172.21.15.98:0/1350128378 conn(0x561c1aa530f0 msgr2=0x561c1aa52270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:12.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdbdaffd640 0 client.17220 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:12.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdc045bc740 1 -- 172.21.15.98:0/1350128378 shutdown_connections 2024-09-19T08:10:12.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdc045bc740 1 -- 172.21.15.98:0/1350128378 wait complete. 2024-09-19T08:10:12.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.025+0000 7fdc045bc740 20 client.17220 trim_cache size 0 max 16384 2024-09-19T08:10:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.042+0000 7fdc045bc740 1 Processor -- start 2024-09-19T08:10:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.042+0000 7fdc045bc740 1 -- start start 2024-09-19T08:10:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.042+0000 7fdc045bc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa569a0 0x561c1aa7b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.042+0000 7fdc045bc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa572c0 0x561c1aa7bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.042+0000 7fdc045bc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa63f50 0x561c1aa60df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.042+0000 7fdc045bc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1ab690 con 0x561c1aa569a0 2024-09-19T08:10:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.042+0000 7fdc045bc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1ad1d0 con 0x561c1aa572c0 2024-09-19T08:10:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.042+0000 7fdc045bc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1abd60 con 0x561c1aa63f50 2024-09-19T08:10:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.042+0000 7fdbfb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa569a0 0x561c1aa7b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.042+0000 7fdbfb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa569a0 0x561c1aa7b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40854/0 (socket says 172.21.15.98:40854) 2024-09-19T08:10:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.042+0000 7fdbfb7fe640 1 -- 172.21.15.98:0/422910192 learned_addr learned my addr 172.21.15.98:0/422910192 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.043+0000 7fdbfaffd640 1 --2- 172.21.15.98:0/422910192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa572c0 0x561c1aa7bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.043+0000 7fdbfb7fe640 1 -- 172.21.15.98:0/422910192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa63f50 msgr2=0x561c1aa60df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.043+0000 7fdbfb7fe640 1 --2- 172.21.15.98:0/422910192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa63f50 0x561c1aa60df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.043+0000 7fdbfb7fe640 1 -- 172.21.15.98:0/422910192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa572c0 msgr2=0x561c1aa7bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.043+0000 7fdbfb7fe640 1 --2- 172.21.15.98:0/422910192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa572c0 0x561c1aa7bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.043+0000 7fdbfb7fe640 1 -- 172.21.15.98:0/422910192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c1aa621b0 con 0x561c1aa569a0 2024-09-19T08:10:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.043+0000 7fdbfaffd640 1 --2- 172.21.15.98:0/422910192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa572c0 0x561c1aa7bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.043+0000 7fdbfb7fe640 1 --2- 172.21.15.98:0/422910192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa569a0 0x561c1aa7b5f0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fdbe400f870 tx=0x7fdbe4005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da66de01c56b8ff server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.043+0000 7fdbf17fa640 1 -- 172.21.15.98:0/422910192 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe4030070 con 0x561c1aa569a0 2024-09-19T08:10:12.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.043+0000 7fdbf17fa640 1 -- 172.21.15.98:0/422910192 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbe400fb10 con 0x561c1aa569a0 2024-09-19T08:10:12.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.043+0000 7fdbf17fa640 1 -- 172.21.15.98:0/422910192 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe401a930 con 0x561c1aa569a0 2024-09-19T08:10:12.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.044+0000 7fdc045bc740 1 -- 172.21.15.98:0/422910192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa569a0 msgr2=0x561c1aa7b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:12.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.044+0000 7fdc045bc740 1 --2- 172.21.15.98:0/422910192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa569a0 0x561c1aa7b5f0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fdbe400f870 tx=0x7fdbe4005670 comp rx=0 tx=0).stop 2024-09-19T08:10:12.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.044+0000 7fdc045bc740 1 -- 172.21.15.98:0/422910192 shutdown_connections 2024-09-19T08:10:12.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.044+0000 7fdc045bc740 1 --2- 172.21.15.98:0/422910192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa63f50 0x561c1aa60df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.044+0000 7fdc045bc740 1 --2- 172.21.15.98:0/422910192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa572c0 0x561c1aa7bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.044+0000 7fdc045bc740 1 --2- 172.21.15.98:0/422910192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa569a0 0x561c1aa7b5f0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.044+0000 7fdc045bc740 1 -- 172.21.15.98:0/422910192 >> 172.21.15.98:0/422910192 conn(0x561c1aa530f0 msgr2=0x561c1aa5ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:12.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.044+0000 7fdc045bc740 1 -- 172.21.15.98:0/422910192 shutdown_connections 2024-09-19T08:10:12.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.044+0000 7fdc045bc740 1 -- 172.21.15.98:0/422910192 wait complete. 2024-09-19T08:10:12.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.044+0000 7fdc045bc740 1 -- start start 2024-09-19T08:10:12.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.044+0000 7fdc045bc740 1 Processor -- start 2024-09-19T08:10:12.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdc045bc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa569a0 0x561c1aa6c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdc045bc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa572c0 0x561c1aa67680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdc045bc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa63f50 0x561c1aa67be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdc045bc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1acb00 con 0x561c1aa572c0 2024-09-19T08:10:12.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdc045bc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1ac430 con 0x561c1aa569a0 2024-09-19T08:10:12.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdc045bc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1a9690 con 0x561c1aa63f50 2024-09-19T08:10:12.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdc045bc740 10 client.0 _ll_register_callbacks cb 0x7fdbe400a7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:12.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbfaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa572c0 0x561c1aa67680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbfaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa572c0 0x561c1aa67680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40860/0 (socket says 172.21.15.98:40860) 2024-09-19T08:10:12.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbfaffd640 1 -- 172.21.15.98:0/707961247 learned_addr learned my addr 172.21.15.98:0/707961247 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:12.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbfb7fe640 1 --2- 172.21.15.98:0/707961247 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa569a0 0x561c1aa6c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbfbfff640 1 --2- 172.21.15.98:0/707961247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa63f50 0x561c1aa67be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbfaffd640 1 -- 172.21.15.98:0/707961247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa63f50 msgr2=0x561c1aa67be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:12.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbfaffd640 1 --2- 172.21.15.98:0/707961247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa63f50 0x561c1aa67be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbfaffd640 1 -- 172.21.15.98:0/707961247 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa569a0 msgr2=0x561c1aa6c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbfaffd640 1 --2- 172.21.15.98:0/707961247 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa569a0 0x561c1aa6c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbfaffd640 1 -- 172.21.15.98:0/707961247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c1aa68410 con 0x561c1aa572c0 2024-09-19T08:10:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbfbfff640 1 --2- 172.21.15.98:0/707961247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa63f50 0x561c1aa67be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbfaffd640 1 --2- 172.21.15.98:0/707961247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa572c0 0x561c1aa67680 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0b8c40 tx=0x7fdbec03b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895bc22128df58fa server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbdaffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbdaffd640 1 -- 172.21.15.98:0/707961247 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec049040 con 0x561c1aa572c0 2024-09-19T08:10:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdbdaffd640 1 -- 172.21.15.98:0/707961247 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8565+0+0 (secure 0 0 0) 0x7fdbec003c80 con 0x561c1aa572c0 2024-09-19T08:10:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.045+0000 7fdc045bc740 1 -- 172.21.15.98:0/707961247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbe400fb10 con 0x561c1aa572c0 2024-09-19T08:10:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.046+0000 7fdbdaffd640 1 -- 172.21.15.98:0/707961247 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdbec0bc5b0 con 0x561c1aa572c0 2024-09-19T08:10:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.046+0000 7fdc045bc740 10 client.17226 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.046+0000 7fdc045bc740 1 -- 172.21.15.98:0/707961247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561c1aade600 con 0x561c1aa572c0 2024-09-19T08:10:12.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.046+0000 7fdbdaffd640 1 -- 172.21.15.98:0/707961247 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbe400fb10 con 0x561c1aa572c0 2024-09-19T08:10:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.046+0000 7fdbdaffd640 1 -- 172.21.15.98:0/707961247 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fdbec0eded0 con 0x561c1aa572c0 2024-09-19T08:10:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.046+0000 7fdc045bc740 10 client.17226 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.046+0000 7fdc045bc740 10 client.17226 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.046+0000 7fdc045bc740 20 client.17226 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.046+0000 7fdc045bc740 1 --2- 172.21.15.98:0/707961247 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c1aade9a0 0x561c1aae0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.046+0000 7fdc045bc740 4 client.17226 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:10:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.046+0000 7fdc045bc740 1 -- 172.21.15.98:0/707961247 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561c1a9c3d00 con 0x561c1aade9a0 2024-09-19T08:10:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.046+0000 7fdbfb7fe640 1 --2- 172.21.15.98:0/707961247 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c1aade9a0 0x561c1aae0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.047+0000 7fdbdaffd640 10 client.17226 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.047+0000 7fdbfb7fe640 1 --2- 172.21.15.98:0/707961247 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c1aade9a0 0x561c1aae0dc0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfe4dd19954f098b server_cookie=8775633db9b97ec4 in_seq=0 out_seq=0 2024-09-19T08:10:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdbdaffd640 1 -- 172.21.15.98:0/707961247 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561c1a9c3d00 con 0x561c1aade9a0 2024-09-19T08:10:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdbdaffd640 10 client.17226 handle_command_reply: tid=0 2024-09-19T08:10:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdc045bc740 1 client.17226 shutdown 2024-09-19T08:10:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdc045bc740 1 -- 172.21.15.98:0/707961247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa572c0 msgr2=0x561c1aa67680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdc045bc740 1 --2- 172.21.15.98:0/707961247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa572c0 0x561c1aa67680 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0b8c40 tx=0x7fdbec03b720 comp rx=0 tx=0).stop 2024-09-19T08:10:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdc045bc740 1 -- 172.21.15.98:0/707961247 shutdown_connections 2024-09-19T08:10:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdc045bc740 1 --2- 172.21.15.98:0/707961247 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c1aade9a0 0x561c1aae0dc0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdc045bc740 1 --2- 172.21.15.98:0/707961247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c1aa63f50 0x561c1aa67be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdc045bc740 1 --2- 172.21.15.98:0/707961247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c1aa572c0 0x561c1aa67680 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdc045bc740 1 --2- 172.21.15.98:0/707961247 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c1aa569a0 0x561c1aa6c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdc045bc740 1 -- 172.21.15.98:0/707961247 >> 172.21.15.98:0/707961247 conn(0x561c1aa530f0 msgr2=0x561c1aa51a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:12.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdbdaffd640 0 client.17226 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:12.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdc045bc740 1 -- 172.21.15.98:0/707961247 shutdown_connections 2024-09-19T08:10:12.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdc045bc740 1 -- 172.21.15.98:0/707961247 wait complete. 2024-09-19T08:10:12.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.048+0000 7fdc045bc740 20 client.17226 trim_cache size 0 max 16384 2024-09-19T08:10:12.054 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:10:12.054 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:10:12.054 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "7a61df69-57e0-416d-9015-e501969895b5", 2024-09-19T08:10:12.054 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:10:12.055 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:10:12.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.049+0000 7fdc021e3640 1 -- 172.21.15.98:0/3632107204 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdbe80a33b0 msgr2=0x7fdbe80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:12.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.049+0000 7fdc021e3640 1 --2- 172.21.15.98:0/3632107204 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdbe80a33b0 0x7fdbe80a5870 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4008e80 tx=0x7fdbf4004a40 comp rx=0 tx=0).stop 2024-09-19T08:10:12.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.049+0000 7fdc021e3640 1 -- 172.21.15.98:0/3632107204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc101420 msgr2=0x7fdbfc197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:12.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.049+0000 7fdc021e3640 1 --2- 172.21.15.98:0/3632107204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc101420 0x7fdbfc197840 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009a50 tx=0x7fdbec004270 comp rx=0 tx=0).stop 2024-09-19T08:10:12.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.049+0000 7fdc021e3640 1 -- 172.21.15.98:0/3632107204 shutdown_connections 2024-09-19T08:10:12.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.049+0000 7fdc021e3640 1 --2- 172.21.15.98:0/3632107204 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdbe80a33b0 0x7fdbe80a5870 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.049+0000 7fdc021e3640 1 --2- 172.21.15.98:0/3632107204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbfc106a80 0x7fdbfc19bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.049+0000 7fdc021e3640 1 --2- 172.21.15.98:0/3632107204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc101420 0x7fdbfc197840 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.049+0000 7fdc021e3640 1 --2- 172.21.15.98:0/3632107204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbfc100a80 0x7fdbfc197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:12.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.049+0000 7fdc021e3640 1 -- 172.21.15.98:0/3632107204 >> 172.21.15.98:0/3632107204 conn(0x7fdbfc0fc7a0 msgr2=0x7fdbfc0fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:12.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.049+0000 7fdc021e3640 1 -- 172.21.15.98:0/3632107204 shutdown_connections 2024-09-19T08:10:12.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:12.049+0000 7fdc021e3640 1 -- 172.21.15.98:0/3632107204 wait complete. 2024-09-19T08:10:12.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:11 smithi139 ceph-mon[34874]: osdmap e82: 12 total, 12 up, 12 in 2024-09-19T08:10:12.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:11 smithi139 ceph-mon[34874]: pgmap v219: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-09-19T08:10:12.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:11 smithi098 ceph-mon[31333]: osdmap e82: 12 total, 12 up, 12 in 2024-09-19T08:10:12.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:11 smithi098 ceph-mon[31333]: pgmap v219: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-09-19T08:10:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:11 smithi162 ceph-mon[35260]: osdmap e82: 12 total, 12 up, 12 in 2024-09-19T08:10:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:11 smithi162 ceph-mon[35260]: pgmap v219: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-09-19T08:10:12.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:10:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:12.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:12 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:10:12.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:12 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:10:12.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:12 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:10:12.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:12 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:10:13.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:12 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:10:13.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:12 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:10:13.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:12 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:10:13.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:12 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:10:13.271 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:12 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:10:13.271 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:12 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:10:13.271 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:12 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:10:13.271 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:12 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:10:13.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.370+0000 7fadfcbd9640 1 -- 172.21.15.98:0/2132432920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf8101420 msgr2=0x7fadf810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:13.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.370+0000 7fadfcbd9640 1 --2- 172.21.15.98:0/2132432920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf8101420 0x7fadf810d950 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009cd0 tx=0x7fadf002a8e0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.370+0000 7fadfcbd9640 1 -- 172.21.15.98:0/2132432920 shutdown_connections 2024-09-19T08:10:13.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.370+0000 7fadfcbd9640 1 --2- 172.21.15.98:0/2132432920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf8101420 0x7fadf810d950 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.370+0000 7fadfcbd9640 1 --2- 172.21.15.98:0/2132432920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fadf8100a80 0x7fadf8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.370+0000 7fadfcbd9640 1 --2- 172.21.15.98:0/2132432920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadf8106a80 0x7fadf8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.370+0000 7fadfcbd9640 1 -- 172.21.15.98:0/2132432920 >> 172.21.15.98:0/2132432920 conn(0x7fadf80fc7a0 msgr2=0x7fadf80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.370+0000 7fadfcbd9640 1 -- 172.21.15.98:0/2132432920 shutdown_connections 2024-09-19T08:10:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.370+0000 7fadfcbd9640 1 -- 172.21.15.98:0/2132432920 wait complete. 2024-09-19T08:10:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.371+0000 7fadfcbd9640 1 Processor -- start 2024-09-19T08:10:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.371+0000 7fadfcbd9640 1 -- start start 2024-09-19T08:10:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.371+0000 7fadfcbd9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadf8100a80 0x7fadf8198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.371+0000 7fadfcbd9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf8101420 0x7fadf81988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.371+0000 7fadfcbd9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fadf8106a80 0x7fadf819cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.371+0000 7fadfcbd9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fadf8110e80 con 0x7fadf8101420 2024-09-19T08:10:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.371+0000 7fadfcbd9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fadf8110d00 con 0x7fadf8106a80 2024-09-19T08:10:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.371+0000 7fadfcbd9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fadf8111000 con 0x7fadf8100a80 2024-09-19T08:10:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.371+0000 7fadf5d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf8101420 0x7fadf81988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.371+0000 7fadf5d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf8101420 0x7fadf81988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40876/0 (socket says 172.21.15.98:40876) 2024-09-19T08:10:13.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.371+0000 7fadf5d9b640 1 -- 172.21.15.98:0/2714928229 learned_addr learned my addr 172.21.15.98:0/2714928229 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:13.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.371+0000 7fadf659c640 1 --2- 172.21.15.98:0/2714928229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadf8100a80 0x7fadf8198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.372+0000 7fadf6d9d640 1 --2- 172.21.15.98:0/2714928229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fadf8106a80 0x7fadf819cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.372+0000 7fadf5d9b640 1 -- 172.21.15.98:0/2714928229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadf8100a80 msgr2=0x7fadf8198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:13.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.372+0000 7fadf5d9b640 1 --2- 172.21.15.98:0/2714928229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadf8100a80 0x7fadf8198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.372+0000 7fadf5d9b640 1 -- 172.21.15.98:0/2714928229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fadf8106a80 msgr2=0x7fadf819cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:13.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.372+0000 7fadf5d9b640 1 --2- 172.21.15.98:0/2714928229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fadf8106a80 0x7fadf819cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.372+0000 7fadf5d9b640 1 -- 172.21.15.98:0/2714928229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf819d330 con 0x7fadf8101420 2024-09-19T08:10:13.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.372+0000 7fadf5d9b640 1 --2- 172.21.15.98:0/2714928229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf8101420 0x7fadf81988c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fade000c890 tx=0x7fade000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:13.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.372+0000 7fadef7fe640 1 -- 172.21.15.98:0/2714928229 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade0004220 con 0x7fadf8101420 2024-09-19T08:10:13.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.372+0000 7fadef7fe640 1 -- 172.21.15.98:0/2714928229 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fade00043c0 con 0x7fadf8101420 2024-09-19T08:10:13.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.372+0000 7fadfcbd9640 1 -- 172.21.15.98:0/2714928229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadf819d620 con 0x7fadf8101420 2024-09-19T08:10:13.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.372+0000 7fadef7fe640 1 -- 172.21.15.98:0/2714928229 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade0004a00 con 0x7fadf8101420 2024-09-19T08:10:13.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.372+0000 7fadfcbd9640 1 -- 172.21.15.98:0/2714928229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadf81a4f00 con 0x7fadf8101420 2024-09-19T08:10:13.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.374+0000 7fadef7fe640 1 -- 172.21.15.98:0/2714928229 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fade00079b0 con 0x7fadf8101420 2024-09-19T08:10:13.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.375+0000 7fadfefb2740 1 Processor -- start 2024-09-19T08:10:13.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.375+0000 7fadfefb2740 1 -- start start 2024-09-19T08:10:13.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.375+0000 7fadfefb2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd74a0 0x55ca80bd7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.375+0000 7fadfefb2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80bd7dc0 0x55ca80be2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.375+0000 7fadfefb2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80be2680 0x55ca80be4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.375+0000 7fadfefb2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ca80b3d2e0 con 0x55ca80be2680 2024-09-19T08:10:13.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.375+0000 7fadfefb2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ca80ae4ce0 con 0x55ca80bd7dc0 2024-09-19T08:10:13.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.375+0000 7fadfefb2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ca80b392a0 con 0x55ca80bd74a0 2024-09-19T08:10:13.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.375+0000 7fadf6d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd74a0 0x55ca80bd7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.375+0000 7fadf5d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80be2680 0x55ca80be4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.375+0000 7fadf659c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80bd7dc0 0x55ca80be2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.375+0000 7fadf6d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd74a0 0x55ca80bd7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:40040/0 (socket says 172.21.15.98:40040) 2024-09-19T08:10:13.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.375+0000 7fadf6d9d640 1 -- 172.21.15.98:0/1683171430 learned_addr learned my addr 172.21.15.98:0/1683171430 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:13.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.375+0000 7fadef7fe640 1 --2- 172.21.15.98:0/2714928229 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fade40a33b0 0x7fade40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.376+0000 7fadf6d9d640 1 --2- 172.21.15.98:0/2714928229 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fade40a33b0 0x7fade40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.376+0000 7fadef7fe640 1 -- 172.21.15.98:0/2714928229 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8565+0+0 (secure 0 0 0) 0x7fade00cfa20 con 0x7fadf8101420 2024-09-19T08:10:13.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.376+0000 7fadf5d9b640 1 -- 172.21.15.98:0/1683171430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd74a0 msgr2=0x55ca80bd7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:13.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.376+0000 7fadf5d9b640 1 --2- 172.21.15.98:0/1683171430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd74a0 0x55ca80bd7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.376+0000 7fadf5d9b640 1 -- 172.21.15.98:0/1683171430 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80bd7dc0 msgr2=0x55ca80be2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:13.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.376+0000 7fadf5d9b640 1 --2- 172.21.15.98:0/1683171430 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80bd7dc0 0x55ca80be2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.376+0000 7fadf5d9b640 1 -- 172.21.15.98:0/1683171430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca80ac9440 con 0x55ca80be2680 2024-09-19T08:10:13.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.376+0000 7fadf659c640 1 --2- 172.21.15.98:0/1683171430 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80bd7dc0 0x55ca80be2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:10:13.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.376+0000 7fadf6d9d640 1 --2- 172.21.15.98:0/1683171430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd74a0 0x55ca80bd7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:10:13.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.376+0000 7fadf5d9b640 1 --2- 172.21.15.98:0/1683171430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80be2680 0x55ca80be4a70 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fade00cd140 tx=0x7fade009bbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d0190084f28ad6c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:13.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.376+0000 7fadecff9640 1 -- 172.21.15.98:0/1683171430 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade001d050 con 0x55ca80be2680 2024-09-19T08:10:13.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.376+0000 7fadecff9640 1 -- 172.21.15.98:0/1683171430 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fade009c3d0 con 0x55ca80be2680 2024-09-19T08:10:13.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.376+0000 7fadecff9640 1 -- 172.21.15.98:0/1683171430 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade0098340 con 0x55ca80be2680 2024-09-19T08:10:13.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.376+0000 7fadf6d9d640 1 --2- 172.21.15.98:0/2714928229 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fade40a33b0 0x7fade40a5870 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fadf002ae00 tx=0x7fadf002a8e0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:13.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.377+0000 7fadfefb2740 1 -- 172.21.15.98:0/1683171430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80be2680 msgr2=0x55ca80be4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:13.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.377+0000 7fadfefb2740 1 --2- 172.21.15.98:0/1683171430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80be2680 0x55ca80be4a70 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fade00cd140 tx=0x7fade009bbd0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.377+0000 7fadfefb2740 1 -- 172.21.15.98:0/1683171430 shutdown_connections 2024-09-19T08:10:13.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.377+0000 7fadfefb2740 1 --2- 172.21.15.98:0/1683171430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80be2680 0x55ca80be4a70 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.377+0000 7fadfefb2740 1 --2- 172.21.15.98:0/1683171430 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80bd7dc0 0x55ca80be2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.377+0000 7fadfefb2740 1 --2- 172.21.15.98:0/1683171430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd74a0 0x55ca80bd7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.377+0000 7fadfefb2740 1 -- 172.21.15.98:0/1683171430 >> 172.21.15.98:0/1683171430 conn(0x55ca80bd3a30 msgr2=0x55ca80bd3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.377+0000 7fadfefb2740 1 -- 172.21.15.98:0/1683171430 shutdown_connections 2024-09-19T08:10:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.377+0000 7fadfefb2740 1 -- 172.21.15.98:0/1683171430 wait complete. 2024-09-19T08:10:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.377+0000 7fadfefb2740 1 -- start start 2024-09-19T08:10:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.377+0000 7fadfefb2740 1 Processor -- start 2024-09-19T08:10:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadfefb2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd74a0 0x55ca80be2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadfefb2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd7dc0 0x55ca80bdf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadfefb2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80be2680 0x55ca80bdf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadfefb2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ca80be5250 con 0x55ca80bd74a0 2024-09-19T08:10:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadfefb2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ca80af8050 con 0x55ca80be2680 2024-09-19T08:10:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadfefb2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ca80be53d0 con 0x55ca80bd7dc0 2024-09-19T08:10:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadfefb2740 10 client.0 _ll_register_callbacks cb 0x55ca80b38160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadf659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd74a0 0x55ca80be2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadf659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd74a0 0x55ca80be2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40892/0 (socket says 172.21.15.98:40892) 2024-09-19T08:10:13.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadf659c640 1 -- 172.21.15.98:0/254630569 learned_addr learned my addr 172.21.15.98:0/254630569 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:13.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadf5d9b640 1 --2- 172.21.15.98:0/254630569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd7dc0 0x55ca80bdf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadf6d9d640 1 --2- 172.21.15.98:0/254630569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80be2680 0x55ca80bdf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadf659c640 1 -- 172.21.15.98:0/254630569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd7dc0 msgr2=0x55ca80bdf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:13.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadf659c640 1 --2- 172.21.15.98:0/254630569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd7dc0 0x55ca80bdf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadf659c640 1 -- 172.21.15.98:0/254630569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80be2680 msgr2=0x55ca80bdf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:13.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadf659c640 1 --2- 172.21.15.98:0/254630569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80be2680 0x55ca80bdf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadf659c640 1 -- 172.21.15.98:0/254630569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca808ae3c0 con 0x55ca80bd74a0 2024-09-19T08:10:13.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadf6d9d640 1 --2- 172.21.15.98:0/254630569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80be2680 0x55ca80bdf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:13.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadf659c640 1 --2- 172.21.15.98:0/254630569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd74a0 0x55ca80be2010 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fade8010a50 tx=0x7fade8010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea4c1dc2ad546eb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:13.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadddffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:13.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadddffb640 1 -- 172.21.15.98:0/254630569 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade801e060 con 0x55ca80bd74a0 2024-09-19T08:10:13.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadddffb640 1 -- 172.21.15.98:0/254630569 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8565+0+0 (secure 0 0 0) 0x7fade800fb10 con 0x55ca80bd74a0 2024-09-19T08:10:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.378+0000 7fadfefb2740 1 -- 172.21.15.98:0/254630569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fade009c3d0 con 0x55ca80bd74a0 2024-09-19T08:10:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.379+0000 7fadddffb640 1 -- 172.21.15.98:0/254630569 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fade800fd10 con 0x55ca80bd74a0 2024-09-19T08:10:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.379+0000 7fadfefb2740 10 client.17238 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.379+0000 7fadfefb2740 1 -- 172.21.15.98:0/254630569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ca80bf0ca0 con 0x55ca80bd74a0 2024-09-19T08:10:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.379+0000 7fadddffb640 1 -- 172.21.15.98:0/254630569 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fade009c3d0 con 0x55ca80bd74a0 2024-09-19T08:10:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.379+0000 7fadddffb640 1 -- 172.21.15.98:0/254630569 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fade800e570 con 0x55ca80bd74a0 2024-09-19T08:10:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.379+0000 7fadfefb2740 10 client.17238 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.379+0000 7fadfefb2740 10 client.17238 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.379+0000 7fadfefb2740 20 client.17238 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.379+0000 7fadfefb2740 1 --2- 172.21.15.98:0/254630569 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ca80bf2160 0x55ca80bf4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.379+0000 7fadfefb2740 4 client.17238 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.379+0000 7fadfefb2740 1 -- 172.21.15.98:0/254630569 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca80b4a850 con 0x55ca80bf2160 2024-09-19T08:10:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.379+0000 7fadf5d9b640 1 --2- 172.21.15.98:0/254630569 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ca80bf2160 0x55ca80bf4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.380+0000 7fadddffb640 10 client.17238 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.380+0000 7fadf5d9b640 1 --2- 172.21.15.98:0/254630569 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ca80bf2160 0x55ca80bf4550 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9daa53fdbc39aeb1 server_cookie=bf2bd7dab2244797 in_seq=0 out_seq=0 2024-09-19T08:10:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.380+0000 7fadddffb640 1 -- 172.21.15.98:0/254630569 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ca80b4a850 con 0x55ca80bf2160 2024-09-19T08:10:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.380+0000 7fadddffb640 10 client.17238 handle_command_reply: tid=0 2024-09-19T08:10:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.380+0000 7fadfefb2740 1 client.17238 shutdown 2024-09-19T08:10:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.380+0000 7fadfefb2740 1 -- 172.21.15.98:0/254630569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd74a0 msgr2=0x55ca80be2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.380+0000 7fadfefb2740 1 --2- 172.21.15.98:0/254630569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd74a0 0x55ca80be2010 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fade8010a50 tx=0x7fade8010f10 comp rx=0 tx=0).stop 2024-09-19T08:10:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.380+0000 7fadfefb2740 1 -- 172.21.15.98:0/254630569 shutdown_connections 2024-09-19T08:10:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.380+0000 7fadfefb2740 1 --2- 172.21.15.98:0/254630569 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ca80bf2160 0x55ca80bf4550 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.380+0000 7fadfefb2740 1 --2- 172.21.15.98:0/254630569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80be2680 0x55ca80bdf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.380+0000 7fadfefb2740 1 --2- 172.21.15.98:0/254630569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd7dc0 0x55ca80bdf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.380+0000 7fadddffb640 0 client.17238 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.380+0000 7fadfefb2740 1 --2- 172.21.15.98:0/254630569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd74a0 0x55ca80be2010 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.380+0000 7fadfefb2740 1 -- 172.21.15.98:0/254630569 >> 172.21.15.98:0/254630569 conn(0x55ca80bd3a30 msgr2=0x55ca80bd9710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:13.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.381+0000 7fadfefb2740 1 -- 172.21.15.98:0/254630569 shutdown_connections 2024-09-19T08:10:13.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.381+0000 7fadfefb2740 1 -- 172.21.15.98:0/254630569 wait complete. 2024-09-19T08:10:13.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.381+0000 7fadfefb2740 20 client.17238 trim_cache size 0 max 16384 2024-09-19T08:10:13.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.397+0000 7fadfefb2740 1 Processor -- start 2024-09-19T08:10:13.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.397+0000 7fadfefb2740 1 -- start start 2024-09-19T08:10:13.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadfefb2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd74a0 0x55ca80bee4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadfefb2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd7dc0 0x55ca80bdeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadfefb2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80be2680 0x55ca80bda7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadfefb2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fadf81ad9a0 con 0x55ca80bd74a0 2024-09-19T08:10:13.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadfefb2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fadf81af4e0 con 0x55ca80be2680 2024-09-19T08:10:13.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadfefb2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fadf81ae070 con 0x55ca80bd7dc0 2024-09-19T08:10:13.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadf659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd74a0 0x55ca80bee4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadf659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd74a0 0x55ca80bee4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40894/0 (socket says 172.21.15.98:40894) 2024-09-19T08:10:13.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadf659c640 1 -- 172.21.15.98:0/1232433182 learned_addr learned my addr 172.21.15.98:0/1232433182 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:13.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadf659c640 1 -- 172.21.15.98:0/1232433182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd7dc0 msgr2=0x55ca80bdeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:13.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadf659c640 1 --2- 172.21.15.98:0/1232433182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd7dc0 0x55ca80bdeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadf659c640 1 -- 172.21.15.98:0/1232433182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80be2680 msgr2=0x55ca80bda7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:13.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadf659c640 1 --2- 172.21.15.98:0/1232433182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80be2680 0x55ca80bda7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadf659c640 1 -- 172.21.15.98:0/1232433182 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca80bd72d0 con 0x55ca80bd74a0 2024-09-19T08:10:13.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7fadf659c640 1 --2- 172.21.15.98:0/1232433182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd74a0 0x55ca80bee4b0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fade800f010 tx=0x7fade8005ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d91b9aa1e224ac5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:13.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7faddffff640 1 -- 172.21.15.98:0/1232433182 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8030070 con 0x55ca80bd74a0 2024-09-19T08:10:13.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7faddffff640 1 -- 172.21.15.98:0/1232433182 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fade800fb10 con 0x55ca80bd74a0 2024-09-19T08:10:13.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.398+0000 7faddffff640 1 -- 172.21.15.98:0/1232433182 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade801a870 con 0x55ca80bd74a0 2024-09-19T08:10:13.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.399+0000 7fadfefb2740 1 -- 172.21.15.98:0/1232433182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd74a0 msgr2=0x55ca80bee4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:13.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.399+0000 7fadfefb2740 1 --2- 172.21.15.98:0/1232433182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd74a0 0x55ca80bee4b0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fade800f010 tx=0x7fade8005ec0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.399+0000 7fadfefb2740 1 -- 172.21.15.98:0/1232433182 shutdown_connections 2024-09-19T08:10:13.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.399+0000 7fadfefb2740 1 --2- 172.21.15.98:0/1232433182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80be2680 0x55ca80bda7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.399+0000 7fadfefb2740 1 --2- 172.21.15.98:0/1232433182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80bd7dc0 0x55ca80bdeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.399+0000 7fadfefb2740 1 --2- 172.21.15.98:0/1232433182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd74a0 0x55ca80bee4b0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.399+0000 7fadfefb2740 1 -- 172.21.15.98:0/1232433182 >> 172.21.15.98:0/1232433182 conn(0x55ca80bd3a30 msgr2=0x55ca80bda090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:13.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.399+0000 7fadfefb2740 1 -- 172.21.15.98:0/1232433182 shutdown_connections 2024-09-19T08:10:13.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.399+0000 7fadfefb2740 1 -- 172.21.15.98:0/1232433182 wait complete. 2024-09-19T08:10:13.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.399+0000 7fadfefb2740 1 -- start start 2024-09-19T08:10:13.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadfefb2740 1 Processor -- start 2024-09-19T08:10:13.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadfefb2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80bd74a0 0x55ca80bead60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadfefb2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd7dc0 0x55ca80be60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadfefb2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80be2680 0x55ca80be6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadfefb2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fadf81aee10 con 0x55ca80bd7dc0 2024-09-19T08:10:13.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadfefb2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fadf81ae740 con 0x55ca80bd74a0 2024-09-19T08:10:13.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadfefb2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fadf81ab9a0 con 0x55ca80be2680 2024-09-19T08:10:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadfefb2740 10 client.0 _ll_register_callbacks cb 0x7fade800a7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadf5d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd7dc0 0x55ca80be60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadf5d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd7dc0 0x55ca80be60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40902/0 (socket says 172.21.15.98:40902) 2024-09-19T08:10:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadf5d9b640 1 -- 172.21.15.98:0/2175200801 learned_addr learned my addr 172.21.15.98:0/2175200801 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadf6d9d640 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80be2680 0x55ca80be6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadf659c640 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80bd74a0 0x55ca80bead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadf5d9b640 1 -- 172.21.15.98:0/2175200801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80be2680 msgr2=0x55ca80be6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadf5d9b640 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80be2680 0x55ca80be6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadf5d9b640 1 -- 172.21.15.98:0/2175200801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80bd74a0 msgr2=0x55ca80bead60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadf5d9b640 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80bd74a0 0x55ca80bead60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadf5d9b640 1 -- 172.21.15.98:0/2175200801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca80be6e50 con 0x55ca80bd7dc0 2024-09-19T08:10:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadf659c640 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80bd74a0 0x55ca80bead60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadf6d9d640 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80be2680 0x55ca80be6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadf5d9b640 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd7dc0 0x55ca80be60e0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fade00058c0 tx=0x7fade00cfc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd299bff78dda594 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.400+0000 7fadddffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadddffb640 1 -- 172.21.15.98:0/2175200801 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade001d050 con 0x55ca80bd7dc0 2024-09-19T08:10:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadddffb640 1 -- 172.21.15.98:0/2175200801 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8565+0+0 (secure 0 0 0) 0x7fade0098340 con 0x55ca80bd7dc0 2024-09-19T08:10:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadfefb2740 1 -- 172.21.15.98:0/2175200801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fade800fb10 con 0x55ca80bd7dc0 2024-09-19T08:10:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadddffb640 1 -- 172.21.15.98:0/2175200801 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fade009c3d0 con 0x55ca80bd7dc0 2024-09-19T08:10:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadfefb2740 10 client.17244 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadfefb2740 1 -- 172.21.15.98:0/2175200801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ca80c5cdd0 con 0x55ca80bd7dc0 2024-09-19T08:10:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadddffb640 1 -- 172.21.15.98:0/2175200801 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fade800fb10 con 0x55ca80bd7dc0 2024-09-19T08:10:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadddffb640 1 -- 172.21.15.98:0/2175200801 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fade009b120 con 0x55ca80bd7dc0 2024-09-19T08:10:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadfefb2740 10 client.17244 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadfefb2740 10 client.17244 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadfefb2740 20 client.17244 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadfefb2740 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ca80c5d0b0 0x55ca80c5f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadfefb2740 4 client.17244 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:10:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadfefb2740 1 -- 172.21.15.98:0/2175200801 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ca80b44d60 con 0x55ca80c5d0b0 2024-09-19T08:10:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.401+0000 7fadf659c640 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ca80c5d0b0 0x55ca80c5f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.402+0000 7fadf659c640 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ca80c5d0b0 0x55ca80c5f4a0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28d9ff775cbeac3f server_cookie=821004c0cb123219 in_seq=0 out_seq=0 2024-09-19T08:10:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.402+0000 7fadddffb640 10 client.17244 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.402+0000 7fadddffb640 1 -- 172.21.15.98:0/2175200801 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ca80b44d60 con 0x55ca80c5d0b0 2024-09-19T08:10:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.402+0000 7fadddffb640 10 client.17244 handle_command_reply: tid=0 2024-09-19T08:10:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.402+0000 7fadfefb2740 1 client.17244 shutdown 2024-09-19T08:10:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.402+0000 7fadfefb2740 1 -- 172.21.15.98:0/2175200801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd7dc0 msgr2=0x55ca80be60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.402+0000 7fadfefb2740 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd7dc0 0x55ca80be60e0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fade00058c0 tx=0x7fade00cfc20 comp rx=0 tx=0).stop 2024-09-19T08:10:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.402+0000 7fadfefb2740 1 -- 172.21.15.98:0/2175200801 shutdown_connections 2024-09-19T08:10:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.402+0000 7fadfefb2740 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ca80c5d0b0 0x55ca80c5f4a0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.402+0000 7fadfefb2740 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca80be2680 0x55ca80be6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.402+0000 7fadfefb2740 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca80bd7dc0 0x55ca80be60e0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.402+0000 7fadfefb2740 1 --2- 172.21.15.98:0/2175200801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ca80bd74a0 0x55ca80bead60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.402+0000 7fadfefb2740 1 -- 172.21.15.98:0/2175200801 >> 172.21.15.98:0/2175200801 conn(0x55ca80bd3a30 msgr2=0x55ca80bda8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.402+0000 7fadddffb640 0 client.17244 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfefb2740 1 -- 172.21.15.98:0/2175200801 shutdown_connections 2024-09-19T08:10:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfefb2740 1 -- 172.21.15.98:0/2175200801 wait complete. 2024-09-19T08:10:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfefb2740 20 client.17244 trim_cache size 0 max 16384 2024-09-19T08:10:13.408 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:10:13.408 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-19T08:10:13.409 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-09-19T08:10:13.409 INFO:teuthology.orchestra.run.smithi098.stdout: "7a61df69-57e0-416d-9015-e501969895b5": { 2024-09-19T08:10:13.409 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-09-19T08:10:13.409 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "7a61df69-57e0-416d-9015-e501969895b5", 2024-09-19T08:10:13.409 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-09-19T08:10:13.409 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:10:13.409 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:10:13.409 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:10:13.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfcbd9640 1 -- 172.21.15.98:0/2714928229 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fade40a33b0 msgr2=0x7fade40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:13.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfcbd9640 1 --2- 172.21.15.98:0/2714928229 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fade40a33b0 0x7fade40a5870 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fadf002ae00 tx=0x7fadf002a8e0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfcbd9640 1 -- 172.21.15.98:0/2714928229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf8101420 msgr2=0x7fadf81988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:13.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfcbd9640 1 --2- 172.21.15.98:0/2714928229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf8101420 0x7fadf81988c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fade000c890 tx=0x7fade000cc90 comp rx=0 tx=0).stop 2024-09-19T08:10:13.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfcbd9640 1 -- 172.21.15.98:0/2714928229 shutdown_connections 2024-09-19T08:10:13.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfcbd9640 1 --2- 172.21.15.98:0/2714928229 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fade40a33b0 0x7fade40a5870 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfcbd9640 1 --2- 172.21.15.98:0/2714928229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fadf8106a80 0x7fadf819cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfcbd9640 1 --2- 172.21.15.98:0/2714928229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf8101420 0x7fadf81988c0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfcbd9640 1 --2- 172.21.15.98:0/2714928229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadf8100a80 0x7fadf8198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:13.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfcbd9640 1 -- 172.21.15.98:0/2714928229 >> 172.21.15.98:0/2714928229 conn(0x7fadf80fc7a0 msgr2=0x7fadf8104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:13.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfcbd9640 1 -- 172.21.15.98:0/2714928229 shutdown_connections 2024-09-19T08:10:13.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:13.403+0000 7fadfcbd9640 1 -- 172.21.15.98:0/2714928229 wait complete. 2024-09-19T08:10:14.129 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:13 smithi098 ceph-mon[31333]: pgmap v220: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:10:14.158 INFO:tasks.cephfs.filesystem:scrub status for tag:7a61df69-57e0-416d-9015-e501969895b5 - {'path': '/', 'tag': '7a61df69-57e0-416d-9015-e501969895b5', 'options': 'recursive,force'} 2024-09-19T08:10:14.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:13 smithi139 ceph-mon[34874]: pgmap v220: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:10:14.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:13 smithi162 ceph-mon[35260]: pgmap v220: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:10:15.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:14 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:10:15.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:14 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:10:15.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:14 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:10:15.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:14 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:10:15.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:14 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:10:15.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:14 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:10:15.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:14 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:10:15.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:14 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:10:15.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:14 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:10:15.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:14 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:10:15.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:14 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:10:15.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:14 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:10:16.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:15 smithi139 ceph-mon[34874]: pgmap v221: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:10:16.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:15 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:10:16.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:15 smithi139 ceph-mon[34874]: osdmap e83: 12 total, 12 up, 12 in 2024-09-19T08:10:16.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:15 smithi098 ceph-mon[31333]: pgmap v221: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:10:16.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:15 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:10:16.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:15 smithi098 ceph-mon[31333]: osdmap e83: 12 total, 12 up, 12 in 2024-09-19T08:10:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:15 smithi162 ceph-mon[35260]: pgmap v221: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:10:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:15 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:10:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:15 smithi162 ceph-mon[35260]: osdmap e83: 12 total, 12 up, 12 in 2024-09-19T08:10:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:17 smithi162 ceph-mon[35260]: pgmap v223: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-19T08:10:18.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:17 smithi139 ceph-mon[34874]: pgmap v223: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-19T08:10:18.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:17 smithi098 ceph-mon[31333]: pgmap v223: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-19T08:10:20.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:19 smithi162 ceph-mon[35260]: pgmap v224: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 255 B/s wr, 13 op/s 2024-09-19T08:10:20.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:19 smithi139 ceph-mon[34874]: pgmap v224: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 255 B/s wr, 13 op/s 2024-09-19T08:10:20.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:19 smithi098 ceph-mon[31333]: pgmap v224: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 255 B/s wr, 13 op/s 2024-09-19T08:10:22.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:21 smithi162 ceph-mon[35260]: pgmap v225: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:10:22.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:21 smithi139 ceph-mon[34874]: pgmap v225: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:10:22.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:21 smithi098 ceph-mon[31333]: pgmap v225: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:10:24.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:23 smithi162 ceph-mon[35260]: pgmap v226: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:10:24.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:23 smithi139 ceph-mon[34874]: pgmap v226: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:10:24.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:23 smithi098 ceph-mon[31333]: pgmap v226: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:10:26.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:25 smithi162 ceph-mon[35260]: pgmap v227: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:10:26.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:25 smithi139 ceph-mon[34874]: pgmap v227: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:10:26.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:25 smithi098 ceph-mon[31333]: pgmap v227: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:10:28.234 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:27 smithi098 ceph-mon[31333]: pgmap v228: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 92 B/s wr, 12 op/s 2024-09-19T08:10:28.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:27 smithi162 ceph-mon[35260]: pgmap v228: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 92 B/s wr, 12 op/s 2024-09-19T08:10:28.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:27 smithi139 ceph-mon[34874]: pgmap v228: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 92 B/s wr, 12 op/s 2024-09-19T08:10:30.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:29 smithi162 ceph-mon[35260]: pgmap v229: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:10:30.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:29 smithi139 ceph-mon[34874]: pgmap v229: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:10:30.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:29 smithi098 ceph-mon[31333]: pgmap v229: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:10:32.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:31 smithi162 ceph-mon[35260]: pgmap v230: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:32.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:31 smithi139 ceph-mon[34874]: pgmap v230: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:32.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:31 smithi098 ceph-mon[31333]: pgmap v230: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:34.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:33 smithi162 ceph-mon[35260]: pgmap v231: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:10:34.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:33 smithi139 ceph-mon[34874]: pgmap v231: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:10:34.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:33 smithi098 ceph-mon[31333]: pgmap v231: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:10:36.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:35 smithi162 ceph-mon[35260]: pgmap v232: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:36.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:35 smithi139 ceph-mon[34874]: pgmap v232: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:36.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:35 smithi098 ceph-mon[31333]: pgmap v232: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:38.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:37 smithi162 ceph-mon[35260]: pgmap v233: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:10:38.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:37 smithi139 ceph-mon[34874]: pgmap v233: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:10:38.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:37 smithi098 ceph-mon[31333]: pgmap v233: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:10:40.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:39 smithi162 ceph-mon[35260]: pgmap v234: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:10:40.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:39 smithi139 ceph-mon[34874]: pgmap v234: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:10:40.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:39 smithi098 ceph-mon[31333]: pgmap v234: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:10:42.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:41 smithi162 ceph-mon[35260]: pgmap v235: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:42.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:41 smithi139 ceph-mon[34874]: pgmap v235: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:42.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:41 smithi098 ceph-mon[31333]: pgmap v235: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:44.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:10:44.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:43 smithi162 ceph-mon[35260]: pgmap v236: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:10:44.422 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:44.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:43 smithi139 ceph-mon[34874]: pgmap v236: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:10:44.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:43 smithi098 ceph-mon[31333]: pgmap v236: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:10:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.949+0000 7f6d0396d640 1 -- 172.21.15.98:0/645649627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc106f00 msgr2=0x7f6cfc10ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.949+0000 7f6d0396d640 1 --2- 172.21.15.98:0/645649627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc106f00 0x7f6cfc10ea80 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009ca0 tx=0x7f6cf402a8c0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.949+0000 7f6d0396d640 1 -- 172.21.15.98:0/645649627 shutdown_connections 2024-09-19T08:10:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.949+0000 7f6d0396d640 1 --2- 172.21.15.98:0/645649627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc106f00 0x7f6cfc10ea80 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.949+0000 7f6d0396d640 1 --2- 172.21.15.98:0/645649627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cfc102780 0x7f6cfc1067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.949+0000 7f6d0396d640 1 --2- 172.21.15.98:0/645649627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cfc101dd0 0x7f6cfc1021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.949+0000 7f6d0396d640 1 -- 172.21.15.98:0/645649627 >> 172.21.15.98:0/645649627 conn(0x7f6cfc0fd890 msgr2=0x7f6cfc0ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.949+0000 7f6d0396d640 1 -- 172.21.15.98:0/645649627 shutdown_connections 2024-09-19T08:10:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.949+0000 7f6d0396d640 1 -- 172.21.15.98:0/645649627 wait complete. 2024-09-19T08:10:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.950+0000 7f6d0396d640 1 Processor -- start 2024-09-19T08:10:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.950+0000 7f6d0396d640 1 -- start start 2024-09-19T08:10:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.950+0000 7f6d0396d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cfc101dd0 0x7f6cfc198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.950+0000 7f6d0396d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc102780 0x7f6cfc198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.950+0000 7f6d0396d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cfc106f00 0x7f6cfc19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.950+0000 7f6d0396d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc111fd0 con 0x7f6cfc102780 2024-09-19T08:10:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.950+0000 7f6d0396d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc111e50 con 0x7f6cfc101dd0 2024-09-19T08:10:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.950+0000 7f6d0396d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc112150 con 0x7f6cfc106f00 2024-09-19T08:10:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.950+0000 7f6d00f08640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc102780 0x7f6cfc198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.950+0000 7f6d00f08640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc102780 0x7f6cfc198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57774/0 (socket says 172.21.15.98:57774) 2024-09-19T08:10:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.950+0000 7f6d00f08640 1 -- 172.21.15.98:0/2549091911 learned_addr learned my addr 172.21.15.98:0/2549091911 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.951+0000 7f6d01f0a640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cfc106f00 0x7f6cfc19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.951+0000 7f6d01709640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cfc101dd0 0x7f6cfc198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.951+0000 7f6d00f08640 1 -- 172.21.15.98:0/2549091911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cfc106f00 msgr2=0x7f6cfc19cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.951+0000 7f6d00f08640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cfc106f00 0x7f6cfc19cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.951+0000 7f6d00f08640 1 -- 172.21.15.98:0/2549091911 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cfc101dd0 msgr2=0x7f6cfc198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.951+0000 7f6d00f08640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cfc101dd0 0x7f6cfc198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.951+0000 7f6d00f08640 1 -- 172.21.15.98:0/2549091911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cfc19d3f0 con 0x7f6cfc102780 2024-09-19T08:10:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.951+0000 7f6d01f0a640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cfc106f00 0x7f6cfc19cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.951+0000 7f6d00f08640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc102780 0x7f6cfc198980 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00c890 tx=0x7f6cec00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.951+0000 7f6cf27fc640 1 -- 172.21.15.98:0/2549091911 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec004220 con 0x7f6cfc102780 2024-09-19T08:10:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.951+0000 7f6d0396d640 1 -- 172.21.15.98:0/2549091911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cfc19d6e0 con 0x7f6cfc102780 2024-09-19T08:10:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.951+0000 7f6cf27fc640 1 -- 172.21.15.98:0/2549091911 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cec0043c0 con 0x7f6cfc102780 2024-09-19T08:10:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.951+0000 7f6cf27fc640 1 -- 172.21.15.98:0/2549091911 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec004a00 con 0x7f6cfc102780 2024-09-19T08:10:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.951+0000 7f6d0396d640 1 -- 172.21.15.98:0/2549091911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cfc1a4fc0 con 0x7f6cfc102780 2024-09-19T08:10:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.953+0000 7f6cf27fc640 1 -- 172.21.15.98:0/2549091911 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f6cec0079b0 con 0x7f6cfc102780 2024-09-19T08:10:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6d05d46740 1 Processor -- start 2024-09-19T08:10:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6d05d46740 1 -- start start 2024-09-19T08:10:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6d05d46740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d90595380 0x561d90595760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6d05d46740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 0x561d905a4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6d05d46740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d905a4a90 0x561d905a6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6d05d46740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561d904fb2e0 con 0x561d90595380 2024-09-19T08:10:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6d05d46740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561d904a2ce0 con 0x561d905a4a90 2024-09-19T08:10:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6d05d46740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561d904f72a0 con 0x561d90595ca0 2024-09-19T08:10:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6cf27fc640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6ce80a3430 0x7f6ce80a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6d01f0a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d90595380 0x561d90595760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6d01f0a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d90595380 0x561d90595760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57788/0 (socket says 172.21.15.98:57788) 2024-09-19T08:10:44.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6d01709640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 0x561d905a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6d01f0a640 1 -- 172.21.15.98:0/3579861282 learned_addr learned my addr 172.21.15.98:0/3579861282 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:44.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6cf27fc640 1 -- 172.21.15.98:0/2549091911 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f6cec0cfa50 con 0x7f6cfc102780 2024-09-19T08:10:44.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6d01f0a640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6ce80a3430 0x7f6ce80a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.954+0000 7f6d00f08640 1 --2- 172.21.15.98:0/3579861282 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d905a4a90 0x561d905a6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.955+0000 7f6d01709640 1 -- 172.21.15.98:0/3579861282 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d905a4a90 msgr2=0x561d905a6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:44.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.955+0000 7f6d01709640 1 --2- 172.21.15.98:0/3579861282 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d905a4a90 0x561d905a6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.955+0000 7f6d01709640 1 -- 172.21.15.98:0/3579861282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d90595380 msgr2=0x561d90595760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:44.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.955+0000 7f6d01709640 1 --2- 172.21.15.98:0/3579861282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d90595380 0x561d90595760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.955+0000 7f6d01709640 1 -- 172.21.15.98:0/3579861282 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d90487440 con 0x561d90595ca0 2024-09-19T08:10:44.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.955+0000 7f6d01f0a640 1 --2- 172.21.15.98:0/3579861282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d90595380 0x561d90595760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:10:44.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.955+0000 7f6d00f08640 1 --2- 172.21.15.98:0/3579861282 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d905a4a90 0x561d905a6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:44.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.955+0000 7f6d01709640 1 --2- 172.21.15.98:0/3579861282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 0x561d905a4550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800b760 tx=0x7f6cf800bc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87e921d215b493a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:44.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.955+0000 7f6d01f0a640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6ce80a3430 0x7f6ce80a58f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4035000 tx=0x7f6cf402a8c0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:44.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.955+0000 7f6ce77fe640 1 -- 172.21.15.98:0/3579861282 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf800d950 con 0x561d90595ca0 2024-09-19T08:10:44.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.955+0000 7f6ce77fe640 1 -- 172.21.15.98:0/3579861282 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf8019070 con 0x561d90595ca0 2024-09-19T08:10:44.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.955+0000 7f6ce77fe640 1 -- 172.21.15.98:0/3579861282 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8014e10 con 0x561d90595ca0 2024-09-19T08:10:44.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.956+0000 7f6d05d46740 1 -- 172.21.15.98:0/3579861282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 msgr2=0x561d905a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:44.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.956+0000 7f6d05d46740 1 --2- 172.21.15.98:0/3579861282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 0x561d905a4550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800b760 tx=0x7f6cf800bc20 comp rx=0 tx=0).stop 2024-09-19T08:10:44.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.956+0000 7f6d05d46740 1 -- 172.21.15.98:0/3579861282 shutdown_connections 2024-09-19T08:10:44.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.956+0000 7f6d05d46740 1 --2- 172.21.15.98:0/3579861282 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d905a4a90 0x561d905a6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.956+0000 7f6d05d46740 1 --2- 172.21.15.98:0/3579861282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 0x561d905a4550 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.956+0000 7f6d05d46740 1 --2- 172.21.15.98:0/3579861282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d90595380 0x561d90595760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.956+0000 7f6d05d46740 1 -- 172.21.15.98:0/3579861282 >> 172.21.15.98:0/3579861282 conn(0x561d90591a30 msgr2=0x561d90591e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:44.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.956+0000 7f6d05d46740 1 -- 172.21.15.98:0/3579861282 shutdown_connections 2024-09-19T08:10:44.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.956+0000 7f6d05d46740 1 -- 172.21.15.98:0/3579861282 wait complete. 2024-09-19T08:10:44.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.956+0000 7f6d05d46740 1 -- start start 2024-09-19T08:10:44.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.956+0000 7f6d05d46740 1 Processor -- start 2024-09-19T08:10:44.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d05d46740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595380 0x561d905a4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d05d46740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d90595ca0 0x561d905a1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d05d46740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d905a4a90 0x561d905a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d05d46740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561d905a7660 con 0x561d905a4a90 2024-09-19T08:10:44.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d05d46740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561d904b6050 con 0x561d90595ca0 2024-09-19T08:10:44.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d05d46740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561d905a77e0 con 0x561d90595380 2024-09-19T08:10:44.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d05d46740 10 client.0 _ll_register_callbacks cb 0x561d904f6160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:44.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d01f0a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d905a4a90 0x561d905a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d01709640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595380 0x561d905a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d01709640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595380 0x561d905a4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:53776/0 (socket says 172.21.15.98:53776) 2024-09-19T08:10:44.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d01709640 1 -- 172.21.15.98:0/975309041 learned_addr learned my addr 172.21.15.98:0/975309041 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:44.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d00f08640 1 --2- 172.21.15.98:0/975309041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d90595ca0 0x561d905a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d01f0a640 1 -- 172.21.15.98:0/975309041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595380 msgr2=0x561d905a4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:44.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d01f0a640 1 --2- 172.21.15.98:0/975309041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595380 0x561d905a4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d01f0a640 1 -- 172.21.15.98:0/975309041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d90595ca0 msgr2=0x561d905a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:44.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d01f0a640 1 --2- 172.21.15.98:0/975309041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d90595ca0 0x561d905a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d01f0a640 1 -- 172.21.15.98:0/975309041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d9026c3c0 con 0x561d905a4a90 2024-09-19T08:10:44.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6d01f0a640 1 --2- 172.21.15.98:0/975309041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d905a4a90 0x561d905a1b90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009ca0 tx=0x7f6cf40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d13fd9ea2a5555 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:44.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.957+0000 7f6ce4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:44.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6ce4ff9640 1 -- 172.21.15.98:0/975309041 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4045070 con 0x561d905a4a90 2024-09-19T08:10:44.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6d05d46740 1 -- 172.21.15.98:0/975309041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cf8019070 con 0x561d905a4a90 2024-09-19T08:10:44.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6ce4ff9640 1 -- 172.21.15.98:0/975309041 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f6cf40307c0 con 0x561d905a4a90 2024-09-19T08:10:44.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6ce4ff9640 1 -- 172.21.15.98:0/975309041 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6cf40309c0 con 0x561d905a4a90 2024-09-19T08:10:44.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6d05d46740 10 client.17256 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:44.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6d05d46740 1 -- 172.21.15.98:0/975309041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561d905b2f90 con 0x561d905a4a90 2024-09-19T08:10:44.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6ce4ff9640 1 -- 172.21.15.98:0/975309041 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf8019070 con 0x561d905a4a90 2024-09-19T08:10:44.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6ce4ff9640 1 -- 172.21.15.98:0/975309041 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6cf403ecb0 con 0x561d905a4a90 2024-09-19T08:10:44.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6d05d46740 10 client.17256 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:44.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6d05d46740 10 client.17256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:44.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6d05d46740 20 client.17256 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:44.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6d05d46740 1 --2- 172.21.15.98:0/975309041 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561d905b43f0 0x561d905b6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6d05d46740 4 client.17256 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:44.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6d05d46740 1 -- 172.21.15.98:0/975309041 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d90508850 con 0x561d905b43f0 2024-09-19T08:10:44.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.958+0000 7f6d01709640 1 --2- 172.21.15.98:0/975309041 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561d905b43f0 0x561d905b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.959+0000 7f6d01709640 1 --2- 172.21.15.98:0/975309041 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561d905b43f0 0x561d905b6820 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4bf67bc916d2ec3 server_cookie=8d2a603bfd80e1b0 in_seq=0 out_seq=0 2024-09-19T08:10:44.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.959+0000 7f6ce4ff9640 10 client.17256 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6ce4ff9640 1 -- 172.21.15.98:0/975309041 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561d90508850 con 0x561d905b43f0 2024-09-19T08:10:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6ce4ff9640 10 client.17256 handle_command_reply: tid=0 2024-09-19T08:10:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6d05d46740 1 client.17256 shutdown 2024-09-19T08:10:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6d05d46740 1 -- 172.21.15.98:0/975309041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d905a4a90 msgr2=0x561d905a1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6d05d46740 1 --2- 172.21.15.98:0/975309041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d905a4a90 0x561d905a1b90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009ca0 tx=0x7f6cf40095c0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6d05d46740 1 -- 172.21.15.98:0/975309041 shutdown_connections 2024-09-19T08:10:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6d05d46740 1 --2- 172.21.15.98:0/975309041 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561d905b43f0 0x561d905b6820 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6d05d46740 1 --2- 172.21.15.98:0/975309041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d905a4a90 0x561d905a1b90 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6ce4ff9640 0 client.17256 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6d05d46740 1 --2- 172.21.15.98:0/975309041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d90595ca0 0x561d905a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6d05d46740 1 --2- 172.21.15.98:0/975309041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595380 0x561d905a4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6d05d46740 1 -- 172.21.15.98:0/975309041 >> 172.21.15.98:0/975309041 conn(0x561d90591a30 msgr2=0x561d90597790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6d05d46740 1 -- 172.21.15.98:0/975309041 shutdown_connections 2024-09-19T08:10:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6d05d46740 1 -- 172.21.15.98:0/975309041 wait complete. 2024-09-19T08:10:44.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.960+0000 7f6d05d46740 20 client.17256 trim_cache size 0 max 16384 2024-09-19T08:10:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.977+0000 7f6d05d46740 1 Processor -- start 2024-09-19T08:10:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.977+0000 7f6d05d46740 1 -- start start 2024-09-19T08:10:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.977+0000 7f6d05d46740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d90595380 0x561d905b0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.977+0000 7f6d05d46740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 0x561d905bc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.977+0000 7f6d05d46740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d905a4a90 0x561d905bc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.977+0000 7f6d05d46740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc1ab700 con 0x561d90595380 2024-09-19T08:10:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.977+0000 7f6d05d46740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc1ad240 con 0x561d905a4a90 2024-09-19T08:10:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.977+0000 7f6d05d46740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc1abdd0 con 0x561d90595ca0 2024-09-19T08:10:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.977+0000 7f6d01709640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d90595380 0x561d905b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.977+0000 7f6d01709640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d90595380 0x561d905b0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57808/0 (socket says 172.21.15.98:57808) 2024-09-19T08:10:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.977+0000 7f6d01709640 1 -- 172.21.15.98:0/3550484605 learned_addr learned my addr 172.21.15.98:0/3550484605 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.978+0000 7f6d00f08640 1 --2- 172.21.15.98:0/3550484605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 0x561d905bc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.978+0000 7f6d01f0a640 1 --2- 172.21.15.98:0/3550484605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d905a4a90 0x561d905bc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.978+0000 7f6d01709640 1 -- 172.21.15.98:0/3550484605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 msgr2=0x561d905bc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.978+0000 7f6d01709640 1 --2- 172.21.15.98:0/3550484605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 0x561d905bc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.978+0000 7f6d01709640 1 -- 172.21.15.98:0/3550484605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d905a4a90 msgr2=0x561d905bc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.978+0000 7f6d01709640 1 --2- 172.21.15.98:0/3550484605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d905a4a90 0x561d905bc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.978+0000 7f6d01709640 1 -- 172.21.15.98:0/3550484605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d905b6ea0 con 0x561d90595380 2024-09-19T08:10:44.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.978+0000 7f6d01f0a640 1 --2- 172.21.15.98:0/3550484605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d905a4a90 0x561d905bc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:44.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.978+0000 7f6d01709640 1 --2- 172.21.15.98:0/3550484605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d90595380 0x561d905b0900 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8004810 tx=0x7f6cf8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c866232631a95c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:44.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.978+0000 7f6ce6ffd640 1 -- 172.21.15.98:0/3550484605 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf802c070 con 0x561d90595380 2024-09-19T08:10:44.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.978+0000 7f6ce6ffd640 1 -- 172.21.15.98:0/3550484605 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf8019070 con 0x561d90595380 2024-09-19T08:10:44.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.978+0000 7f6ce6ffd640 1 -- 172.21.15.98:0/3550484605 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8027630 con 0x561d90595380 2024-09-19T08:10:44.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.979+0000 7f6d05d46740 1 -- 172.21.15.98:0/3550484605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d90595380 msgr2=0x561d905b0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:44.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.979+0000 7f6d05d46740 1 --2- 172.21.15.98:0/3550484605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d90595380 0x561d905b0900 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8004810 tx=0x7f6cf8008040 comp rx=0 tx=0).stop 2024-09-19T08:10:44.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.979+0000 7f6d05d46740 1 -- 172.21.15.98:0/3550484605 shutdown_connections 2024-09-19T08:10:44.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.979+0000 7f6d05d46740 1 --2- 172.21.15.98:0/3550484605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d905a4a90 0x561d905bc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.979+0000 7f6d05d46740 1 --2- 172.21.15.98:0/3550484605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 0x561d905bc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.979+0000 7f6d05d46740 1 --2- 172.21.15.98:0/3550484605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d90595380 0x561d905b0900 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.979+0000 7f6d05d46740 1 -- 172.21.15.98:0/3550484605 >> 172.21.15.98:0/3550484605 conn(0x561d90591a30 msgr2=0x561d9059cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:44.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.979+0000 7f6d05d46740 1 -- 172.21.15.98:0/3550484605 shutdown_connections 2024-09-19T08:10:44.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.979+0000 7f6d05d46740 1 -- 172.21.15.98:0/3550484605 wait complete. 2024-09-19T08:10:44.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.979+0000 7f6d05d46740 1 -- start start 2024-09-19T08:10:44.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.979+0000 7f6d05d46740 1 Processor -- start 2024-09-19T08:10:44.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d05d46740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d90595380 0x561d905ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d05d46740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 0x561d905a85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d05d46740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d905a4a90 0x561d905a8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d05d46740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc1acb70 con 0x561d905a4a90 2024-09-19T08:10:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d05d46740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc1ac4a0 con 0x561d90595380 2024-09-19T08:10:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d05d46740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc1a9700 con 0x561d90595ca0 2024-09-19T08:10:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d05d46740 10 client.0 _ll_register_callbacks cb 0x7f6cf402f3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d01f0a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d905a4a90 0x561d905a8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d01f0a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d905a4a90 0x561d905a8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57816/0 (socket says 172.21.15.98:57816) 2024-09-19T08:10:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d01f0a640 1 -- 172.21.15.98:0/3478595474 learned_addr learned my addr 172.21.15.98:0/3478595474 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d01709640 1 --2- 172.21.15.98:0/3478595474 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d90595380 0x561d905ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d00f08640 1 --2- 172.21.15.98:0/3478595474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 0x561d905a85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d01f0a640 1 -- 172.21.15.98:0/3478595474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 msgr2=0x561d905a85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d01f0a640 1 --2- 172.21.15.98:0/3478595474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 0x561d905a85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d01f0a640 1 -- 172.21.15.98:0/3478595474 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d90595380 msgr2=0x561d905ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d01f0a640 1 --2- 172.21.15.98:0/3478595474 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d90595380 0x561d905ad2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d01f0a640 1 -- 172.21.15.98:0/3478595474 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d905a93f0 con 0x561d905a4a90 2024-09-19T08:10:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6d01f0a640 1 --2- 172.21.15.98:0/3478595474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d905a4a90 0x561d905a8bc0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4056530 tx=0x7f6cf40569d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb2cf5cd1cca0317 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.980+0000 7f6ce4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.981+0000 7f6ce4ff9640 1 -- 172.21.15.98:0/3478595474 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4060030 con 0x561d905a4a90 2024-09-19T08:10:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.981+0000 7f6d05d46740 1 -- 172.21.15.98:0/3478595474 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cf8019070 con 0x561d905a4a90 2024-09-19T08:10:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.981+0000 7f6ce4ff9640 1 -- 172.21.15.98:0/3478595474 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f6cf40560d0 con 0x561d905a4a90 2024-09-19T08:10:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.981+0000 7f6ce4ff9640 1 -- 172.21.15.98:0/3478595474 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6cf4045040 con 0x561d905a4a90 2024-09-19T08:10:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.981+0000 7f6d05d46740 10 client.17262 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.981+0000 7f6d05d46740 1 -- 172.21.15.98:0/3478595474 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561d9061f120 con 0x561d905a4a90 2024-09-19T08:10:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.981+0000 7f6ce4ff9640 1 -- 172.21.15.98:0/3478595474 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf8019070 con 0x561d905a4a90 2024-09-19T08:10:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.981+0000 7f6ce4ff9640 1 -- 172.21.15.98:0/3478595474 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6cf403e430 con 0x561d905a4a90 2024-09-19T08:10:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.981+0000 7f6d05d46740 10 client.17262 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.981+0000 7f6d05d46740 10 client.17262 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.981+0000 7f6d05d46740 20 client.17262 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.981+0000 7f6d05d46740 1 --2- 172.21.15.98:0/3478595474 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561d9061f830 0x561d90621c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.981+0000 7f6d05d46740 4 client.17262 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:10:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.981+0000 7f6d05d46740 1 -- 172.21.15.98:0/3478595474 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561d90502d60 con 0x561d9061f830 2024-09-19T08:10:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6d01709640 1 --2- 172.21.15.98:0/3478595474 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561d9061f830 0x561d90621c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6ce4ff9640 10 client.17262 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6d01709640 1 --2- 172.21.15.98:0/3478595474 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561d9061f830 0x561d90621c50 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ca7a224706fe7df server_cookie=4faf1c79a431bff4 in_seq=0 out_seq=0 2024-09-19T08:10:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6ce4ff9640 1 -- 172.21.15.98:0/3478595474 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561d90502d60 con 0x561d9061f830 2024-09-19T08:10:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6ce4ff9640 10 client.17262 handle_command_reply: tid=0 2024-09-19T08:10:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6d05d46740 1 client.17262 shutdown 2024-09-19T08:10:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6d05d46740 1 -- 172.21.15.98:0/3478595474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d905a4a90 msgr2=0x561d905a8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6d05d46740 1 --2- 172.21.15.98:0/3478595474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d905a4a90 0x561d905a8bc0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4056530 tx=0x7f6cf40569d0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6d05d46740 1 -- 172.21.15.98:0/3478595474 shutdown_connections 2024-09-19T08:10:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6d05d46740 1 --2- 172.21.15.98:0/3478595474 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561d9061f830 0x561d90621c50 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6d05d46740 1 --2- 172.21.15.98:0/3478595474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d905a4a90 0x561d905a8bc0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6d05d46740 1 --2- 172.21.15.98:0/3478595474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d90595ca0 0x561d905a85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6ce4ff9640 0 client.17262 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6d05d46740 1 --2- 172.21.15.98:0/3478595474 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d90595380 0x561d905ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6d05d46740 1 -- 172.21.15.98:0/3478595474 >> 172.21.15.98:0/3478595474 conn(0x561d90591a30 msgr2=0x561d90590350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6d05d46740 1 -- 172.21.15.98:0/3478595474 shutdown_connections 2024-09-19T08:10:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.982+0000 7f6d05d46740 1 -- 172.21.15.98:0/3478595474 wait complete. 2024-09-19T08:10:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.983+0000 7f6d05d46740 20 client.17262 trim_cache size 0 max 16384 2024-09-19T08:10:44.991 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:10:44.991 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:10:44.991 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:10:44.991 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:10:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.983+0000 7f6d0396d640 1 -- 172.21.15.98:0/2549091911 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6ce80a3430 msgr2=0x7f6ce80a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.983+0000 7f6d0396d640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6ce80a3430 0x7f6ce80a58f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4035000 tx=0x7f6cf402a8c0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.983+0000 7f6d0396d640 1 -- 172.21.15.98:0/2549091911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc102780 msgr2=0x7f6cfc198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.983+0000 7f6d0396d640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc102780 0x7f6cfc198980 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00c890 tx=0x7f6cec00cd50 comp rx=0 tx=0).stop 2024-09-19T08:10:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.983+0000 7f6d0396d640 1 -- 172.21.15.98:0/2549091911 shutdown_connections 2024-09-19T08:10:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.983+0000 7f6d0396d640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6ce80a3430 0x7f6ce80a58f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.983+0000 7f6d0396d640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cfc106f00 0x7f6cfc19cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.983+0000 7f6d0396d640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc102780 0x7f6cfc198980 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.983+0000 7f6d0396d640 1 --2- 172.21.15.98:0/2549091911 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cfc101dd0 0x7f6cfc198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.983+0000 7f6d0396d640 1 -- 172.21.15.98:0/2549091911 >> 172.21.15.98:0/2549091911 conn(0x7f6cfc0fd890 msgr2=0x7f6cfc0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.983+0000 7f6d0396d640 1 -- 172.21.15.98:0/2549091911 shutdown_connections 2024-09-19T08:10:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:44.983+0000 7f6d0396d640 1 -- 172.21.15.98:0/2549091911 wait complete. 2024-09-19T08:10:45.750 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:10:45.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:10:46.007 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:46.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:45 smithi098 ceph-mon[31333]: pgmap v237: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:46.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:45 smithi162 ceph-mon[35260]: pgmap v237: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:46.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:45 smithi139 ceph-mon[34874]: pgmap v237: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:10:46.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.455+0000 7fcabd702640 1 -- 172.21.15.98:0/1972611786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab8102ef0 msgr2=0x7fcab810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:46.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.455+0000 7fcabd702640 1 --2- 172.21.15.98:0/1972611786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab8102ef0 0x7fcab810da40 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009900 tx=0x7fcab002a480 comp rx=0 tx=0).stop 2024-09-19T08:10:46.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.456+0000 7fcabd702640 1 -- 172.21.15.98:0/1972611786 shutdown_connections 2024-09-19T08:10:46.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.456+0000 7fcabd702640 1 --2- 172.21.15.98:0/1972611786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab8102ef0 0x7fcab810da40 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:46.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.456+0000 7fcabd702640 1 --2- 172.21.15.98:0/1972611786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab8102570 0x7fcab81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:46.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.456+0000 7fcabd702640 1 --2- 172.21.15.98:0/1972611786 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcab8100e20 0x7fcab8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:46.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.456+0000 7fcabd702640 1 -- 172.21.15.98:0/1972611786 >> 172.21.15.98:0/1972611786 conn(0x7fcab80fc7d0 msgr2=0x7fcab80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:46.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.456+0000 7fcabd702640 1 -- 172.21.15.98:0/1972611786 shutdown_connections 2024-09-19T08:10:46.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.456+0000 7fcabd702640 1 -- 172.21.15.98:0/1972611786 wait complete. 2024-09-19T08:10:46.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.456+0000 7fcabd702640 1 Processor -- start 2024-09-19T08:10:46.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.456+0000 7fcabd702640 1 -- start start 2024-09-19T08:10:46.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.456+0000 7fcabd702640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcab8100e20 0x7fcab8198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:46.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.456+0000 7fcabd702640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab8102570 0x7fcab8198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:46.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.456+0000 7fcabd702640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab8102ef0 0x7fcab819cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:46.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcabd702640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcab81120a0 con 0x7fcab8102ef0 2024-09-19T08:10:46.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcabd702640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcab8111f20 con 0x7fcab8100e20 2024-09-19T08:10:46.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcabd702640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcab8112220 con 0x7fcab8102570 2024-09-19T08:10:46.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcab77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab8102ef0 0x7fcab819cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:46.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcab77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab8102ef0 0x7fcab819cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43458/0 (socket says 172.21.15.98:43458) 2024-09-19T08:10:46.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcab77fe640 1 -- 172.21.15.98:0/2114271716 learned_addr learned my addr 172.21.15.98:0/2114271716 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:46.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcab67fc640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab8102570 0x7fcab8198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:46.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcab6ffd640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcab8100e20 0x7fcab8198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:46.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcab77fe640 1 -- 172.21.15.98:0/2114271716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab8102570 msgr2=0x7fcab8198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:46.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcab77fe640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab8102570 0x7fcab8198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:46.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcab77fe640 1 -- 172.21.15.98:0/2114271716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcab8100e20 msgr2=0x7fcab8198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:46.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcab77fe640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcab8100e20 0x7fcab8198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:46.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcab77fe640 1 -- 172.21.15.98:0/2114271716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab819d3e0 con 0x7fcab8102ef0 2024-09-19T08:10:46.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcab6ffd640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcab8100e20 0x7fcab8198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:46.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcab77fe640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab8102ef0 0x7fcab819cd00 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fcab0005ab0 tx=0x7fcab0003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:46.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fca9ffff640 1 -- 172.21.15.98:0/2114271716 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab003f070 con 0x7fcab8102ef0 2024-09-19T08:10:46.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcabd702640 1 -- 172.21.15.98:0/2114271716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcab819d670 con 0x7fcab8102ef0 2024-09-19T08:10:46.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fca9ffff640 1 -- 172.21.15.98:0/2114271716 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcab0031e30 con 0x7fcab8102ef0 2024-09-19T08:10:46.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fca9ffff640 1 -- 172.21.15.98:0/2114271716 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab0035030 con 0x7fcab8102ef0 2024-09-19T08:10:46.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.457+0000 7fcabd702640 1 -- 172.21.15.98:0/2114271716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcab81a4fb0 con 0x7fcab8102ef0 2024-09-19T08:10:46.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.458+0000 7fcabd702640 1 -- 172.21.15.98:0/2114271716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcab8101200 con 0x7fcab8102ef0 2024-09-19T08:10:46.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.460+0000 7fca9ffff640 1 -- 172.21.15.98:0/2114271716 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fcab0042070 con 0x7fcab8102ef0 2024-09-19T08:10:46.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.461+0000 7fca9ffff640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcaa00a33b0 0x7fcaa00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:46.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.461+0000 7fcab6ffd640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcaa00a33b0 0x7fcaa00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:46.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.461+0000 7fca9ffff640 1 -- 172.21.15.98:0/2114271716 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7fcab00f0d60 con 0x7fcab8102ef0 2024-09-19T08:10:46.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.461+0000 7fcab6ffd640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcaa00a33b0 0x7fcaa00a5870 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009fa0 tx=0x7fcaa8009570 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:46.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.464+0000 7fca9ffff640 1 -- 172.21.15.98:0/2114271716 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fcab00bd6e0 con 0x7fcab8102ef0 2024-09-19T08:10:46.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.665+0000 7fcabd702640 1 -- 172.21.15.98:0/2114271716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcab81a51e0 con 0x7fcab8102ef0 2024-09-19T08:10:46.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.665+0000 7fca9ffff640 1 -- 172.21.15.98:0/2114271716 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11983 (secure 0 0 0) 0x7fcab0003e20 con 0x7fcab8102ef0 2024-09-19T08:10:46.666 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:10:46.666 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"btime":"2024-09-19T08:08:26:985720+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:08:26.555658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:10:46.666 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2024-09-19T08:10:46.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.668+0000 7fcabd702640 1 -- 172.21.15.98:0/2114271716 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcaa00a33b0 msgr2=0x7fcaa00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:46.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.668+0000 7fcabd702640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcaa00a33b0 0x7fcaa00a5870 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009fa0 tx=0x7fcaa8009570 comp rx=0 tx=0).stop 2024-09-19T08:10:46.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.668+0000 7fcabd702640 1 -- 172.21.15.98:0/2114271716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab8102ef0 msgr2=0x7fcab819cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:46.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.668+0000 7fcabd702640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab8102ef0 0x7fcab819cd00 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fcab0005ab0 tx=0x7fcab0003c30 comp rx=0 tx=0).stop 2024-09-19T08:10:46.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.668+0000 7fcabd702640 1 -- 172.21.15.98:0/2114271716 shutdown_connections 2024-09-19T08:10:46.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.668+0000 7fcabd702640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcaa00a33b0 0x7fcaa00a5870 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:46.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.668+0000 7fcabd702640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab8102ef0 0x7fcab819cd00 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:46.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.668+0000 7fcabd702640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab8102570 0x7fcab8198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:46.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.668+0000 7fcabd702640 1 --2- 172.21.15.98:0/2114271716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcab8100e20 0x7fcab8198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:46.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.668+0000 7fcabd702640 1 -- 172.21.15.98:0/2114271716 >> 172.21.15.98:0/2114271716 conn(0x7fcab80fc7d0 msgr2=0x7fcab8106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:46.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.668+0000 7fcabd702640 1 -- 172.21.15.98:0/2114271716 shutdown_connections 2024-09-19T08:10:46.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:46.668+0000 7fcabd702640 1 -- 172.21.15.98:0/2114271716 wait complete. 2024-09-19T08:10:47.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:47 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2114271716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:10:47.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:10:47.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:47 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2114271716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:10:47.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:47 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2114271716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:10:47.578 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.015+0000 7f4d9bfb7640 1 -- 172.21.15.98:0/1273749502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d94100eb0 msgr2=0x7f4d941067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.015+0000 7f4d9bfb7640 1 --2- 172.21.15.98:0/1273749502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d94100eb0 0x7f4d941067a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f4d84009900 tx=0x7f4d8402a410 comp rx=0 tx=0).stop 2024-09-19T08:10:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.015+0000 7f4d9bfb7640 1 -- 172.21.15.98:0/1273749502 shutdown_connections 2024-09-19T08:10:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.015+0000 7f4d9bfb7640 1 --2- 172.21.15.98:0/1273749502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d94106ed0 0x7f4d9410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.015+0000 7f4d9bfb7640 1 --2- 172.21.15.98:0/1273749502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d94100eb0 0x7f4d941067a0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.015+0000 7f4d9bfb7640 1 --2- 172.21.15.98:0/1273749502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d94100500 0x7f4d941008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.015+0000 7f4d9bfb7640 1 -- 172.21.15.98:0/1273749502 >> 172.21.15.98:0/1273749502 conn(0x7f4d940fc5c0 msgr2=0x7f4d940fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.015+0000 7f4d9bfb7640 1 -- 172.21.15.98:0/1273749502 shutdown_connections 2024-09-19T08:10:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.015+0000 7f4d9bfb7640 1 -- 172.21.15.98:0/1273749502 wait complete. 2024-09-19T08:10:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.016+0000 7f4d9bfb7640 1 Processor -- start 2024-09-19T08:10:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.016+0000 7f4d9bfb7640 1 -- start start 2024-09-19T08:10:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.016+0000 7f4d9bfb7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d94100500 0x7f4d941984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.016+0000 7f4d9bfb7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d94100eb0 0x7f4d94198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.016+0000 7f4d9bfb7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d94106ed0 0x7f4d9419cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.016+0000 7f4d9bfb7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d94111fc0 con 0x7f4d94100eb0 2024-09-19T08:10:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.016+0000 7f4d9bfb7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4d94111e40 con 0x7f4d94100500 2024-09-19T08:10:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.016+0000 7f4d9bfb7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4d94112140 con 0x7f4d94106ed0 2024-09-19T08:10:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.016+0000 7f4d99552640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d94100eb0 0x7f4d94198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.016+0000 7f4d99552640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d94100eb0 0x7f4d94198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43476/0 (socket says 172.21.15.98:43476) 2024-09-19T08:10:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.016+0000 7f4d99552640 1 -- 172.21.15.98:0/116018283 learned_addr learned my addr 172.21.15.98:0/116018283 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.016+0000 7f4d9a554640 1 --2- 172.21.15.98:0/116018283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d94106ed0 0x7f4d9419cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.016+0000 7f4d99d53640 1 --2- 172.21.15.98:0/116018283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d94100500 0x7f4d941984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.017+0000 7f4d99552640 1 -- 172.21.15.98:0/116018283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d94106ed0 msgr2=0x7f4d9419cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.017+0000 7f4d99552640 1 --2- 172.21.15.98:0/116018283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d94106ed0 0x7f4d9419cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.017+0000 7f4d99552640 1 -- 172.21.15.98:0/116018283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d94100500 msgr2=0x7f4d941984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.017+0000 7f4d99552640 1 --2- 172.21.15.98:0/116018283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d94100500 0x7f4d941984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.017+0000 7f4d99552640 1 -- 172.21.15.98:0/116018283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d9419d470 con 0x7f4d94100eb0 2024-09-19T08:10:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.017+0000 7f4d99552640 1 --2- 172.21.15.98:0/116018283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d94100eb0 0x7f4d94198a00 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4d84006010 tx=0x7f4d840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.017+0000 7f4d8affd640 1 -- 172.21.15.98:0/116018283 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8403f030 con 0x7f4d94100eb0 2024-09-19T08:10:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.017+0000 7f4d8affd640 1 -- 172.21.15.98:0/116018283 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d84003ce0 con 0x7f4d94100eb0 2024-09-19T08:10:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.017+0000 7f4d9bfb7640 1 -- 172.21.15.98:0/116018283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d9419d700 con 0x7f4d94100eb0 2024-09-19T08:10:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.017+0000 7f4d8affd640 1 -- 172.21.15.98:0/116018283 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84035070 con 0x7f4d94100eb0 2024-09-19T08:10:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.017+0000 7f4d9bfb7640 1 -- 172.21.15.98:0/116018283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d941a5040 con 0x7f4d94100eb0 2024-09-19T08:10:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.020+0000 7f4d9e390740 1 Processor -- start 2024-09-19T08:10:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.020+0000 7f4d9e390740 1 -- start start 2024-09-19T08:10:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.020+0000 7f4d9e390740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbeb40 0x55ac60bbd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.020+0000 7f4d9e390740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bbd5d0 0x55ac60bbda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.020+0000 7f4d9e390740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bc8370 0x55ac60bca760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.020+0000 7f4d9e390740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ac60b232f0 con 0x55ac60bbeb40 2024-09-19T08:10:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.020+0000 7f4d9e390740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ac60acacf0 con 0x55ac60bc8370 2024-09-19T08:10:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.020+0000 7f4d9e390740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ac60b1f2b0 con 0x55ac60bbd5d0 2024-09-19T08:10:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.020+0000 7f4d9a554640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbeb40 0x55ac60bbd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.020+0000 7f4d9a554640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbeb40 0x55ac60bbd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43486/0 (socket says 172.21.15.98:43486) 2024-09-19T08:10:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.020+0000 7f4d9a554640 1 -- 172.21.15.98:0/1165186168 learned_addr learned my addr 172.21.15.98:0/1165186168 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.020+0000 7f4d8affd640 1 -- 172.21.15.98:0/116018283 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f4d84042030 con 0x7f4d94100eb0 2024-09-19T08:10:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.021+0000 7f4d99d53640 1 --2- 172.21.15.98:0/1165186168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bbd5d0 0x55ac60bbda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.021+0000 7f4d9a554640 1 -- 172.21.15.98:0/1165186168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bbd5d0 msgr2=0x55ac60bbda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.021+0000 7f4d9a554640 1 --2- 172.21.15.98:0/1165186168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bbd5d0 0x55ac60bbda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.021+0000 7f4d9a554640 1 -- 172.21.15.98:0/1165186168 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bc8370 msgr2=0x55ac60bca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.021+0000 7f4d9a554640 1 --2- 172.21.15.98:0/1165186168 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bc8370 0x55ac60bca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.021+0000 7f4d9a554640 1 -- 172.21.15.98:0/1165186168 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac60aaf3a0 con 0x55ac60bbeb40 2024-09-19T08:10:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.021+0000 7f4d9a554640 1 --2- 172.21.15.98:0/1165186168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbeb40 0x55ac60bbd090 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c00a430 tx=0x7f4d8c009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d704c7b99b77db server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.021+0000 7f4d7ffff640 1 -- 172.21.15.98:0/1165186168 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c013790 con 0x55ac60bbeb40 2024-09-19T08:10:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.021+0000 7f4d7ffff640 1 -- 172.21.15.98:0/1165186168 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d8c019070 con 0x55ac60bbeb40 2024-09-19T08:10:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.021+0000 7f4d7ffff640 1 -- 172.21.15.98:0/1165186168 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c014d80 con 0x55ac60bbeb40 2024-09-19T08:10:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.021+0000 7f4d8affd640 1 --2- 172.21.15.98:0/116018283 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4d800a33b0 0x7f4d800a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.021+0000 7f4d99d53640 1 --2- 172.21.15.98:0/116018283 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4d800a33b0 0x7f4d800a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.021+0000 7f4d8affd640 1 -- 172.21.15.98:0/116018283 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f4d840f0300 con 0x7f4d94100eb0 2024-09-19T08:10:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.022+0000 7f4d9e390740 1 -- 172.21.15.98:0/1165186168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbeb40 msgr2=0x55ac60bbd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.022+0000 7f4d99d53640 1 --2- 172.21.15.98:0/116018283 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4d800a33b0 0x7f4d800a5870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55ac60bbaac0 tx=0x7f4d90005f90 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.022+0000 7f4d9e390740 1 --2- 172.21.15.98:0/1165186168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbeb40 0x55ac60bbd090 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c00a430 tx=0x7f4d8c009650 comp rx=0 tx=0).stop 2024-09-19T08:10:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.022+0000 7f4d9e390740 1 -- 172.21.15.98:0/1165186168 shutdown_connections 2024-09-19T08:10:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.022+0000 7f4d9e390740 1 --2- 172.21.15.98:0/1165186168 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bc8370 0x55ac60bca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.022+0000 7f4d9e390740 1 --2- 172.21.15.98:0/1165186168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bbd5d0 0x55ac60bbda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.022+0000 7f4d9e390740 1 --2- 172.21.15.98:0/1165186168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbeb40 0x55ac60bbd090 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.022+0000 7f4d9e390740 1 -- 172.21.15.98:0/1165186168 >> 172.21.15.98:0/1165186168 conn(0x55ac60b86310 msgr2=0x55ac60bb9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.022+0000 7f4d9e390740 1 -- 172.21.15.98:0/1165186168 shutdown_connections 2024-09-19T08:10:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.022+0000 7f4d9e390740 1 -- 172.21.15.98:0/1165186168 wait complete. 2024-09-19T08:10:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.022+0000 7f4d9e390740 1 -- start start 2024-09-19T08:10:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.022+0000 7f4d9e390740 1 Processor -- start 2024-09-19T08:10:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d9e390740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbd5d0 0x55ac60bc7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d9e390740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bbeb40 0x55ac60bc5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d9e390740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bc8370 0x55ac60bc5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d9e390740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ac60bcb070 con 0x55ac60bbd5d0 2024-09-19T08:10:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d9e390740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ac60ade060 con 0x55ac60bbeb40 2024-09-19T08:10:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d9e390740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ac60bcb1f0 con 0x55ac60bc8370 2024-09-19T08:10:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d9e390740 10 client.0 _ll_register_callbacks cb 0x55ac608318c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d99d53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbd5d0 0x55ac60bc7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d99d53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbd5d0 0x55ac60bc7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43492/0 (socket says 172.21.15.98:43492) 2024-09-19T08:10:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d99d53640 1 -- 172.21.15.98:0/3386963048 learned_addr learned my addr 172.21.15.98:0/3386963048 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d9a554640 1 --2- 172.21.15.98:0/3386963048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bc8370 0x55ac60bc5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d99552640 1 --2- 172.21.15.98:0/3386963048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bbeb40 0x55ac60bc5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d99d53640 1 -- 172.21.15.98:0/3386963048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bc8370 msgr2=0x55ac60bc5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d99d53640 1 --2- 172.21.15.98:0/3386963048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bc8370 0x55ac60bc5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d99d53640 1 -- 172.21.15.98:0/3386963048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bbeb40 msgr2=0x55ac60bc5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d99d53640 1 --2- 172.21.15.98:0/3386963048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bbeb40 0x55ac60bc5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.023+0000 7f4d99d53640 1 -- 172.21.15.98:0/3386963048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac608943c0 con 0x55ac60bbd5d0 2024-09-19T08:10:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d99d53640 1 --2- 172.21.15.98:0/3386963048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbd5d0 0x55ac60bc7dd0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f4d90011c30 tx=0x7f4d900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5783e18fc8c8c27 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d7d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d7d7fa640 1 -- 172.21.15.98:0/3386963048 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d90005220 con 0x55ac60bbd5d0 2024-09-19T08:10:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d7d7fa640 1 -- 172.21.15.98:0/3386963048 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f4d900053c0 con 0x55ac60bbd5d0 2024-09-19T08:10:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d9e390740 1 -- 172.21.15.98:0/3386963048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d8c019070 con 0x55ac60bbd5d0 2024-09-19T08:10:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d7d7fa640 1 -- 172.21.15.98:0/3386963048 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4d900055c0 con 0x55ac60bbd5d0 2024-09-19T08:10:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d9e390740 10 client.17280 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d9e390740 1 -- 172.21.15.98:0/3386963048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ac60bd6bf0 con 0x55ac60bbd5d0 2024-09-19T08:10:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d7d7fa640 1 -- 172.21.15.98:0/3386963048 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d8c019070 con 0x55ac60bbd5d0 2024-09-19T08:10:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d7d7fa640 1 -- 172.21.15.98:0/3386963048 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4d9001e9b0 con 0x55ac60bbd5d0 2024-09-19T08:10:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d9e390740 10 client.17280 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d9e390740 10 client.17280 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:10:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d9e390740 20 client.17280 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d9e390740 1 --2- 172.21.15.98:0/3386963048 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55ac60bd80b0 0x55ac60bda4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d9e390740 4 client.17280 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.024+0000 7f4d9e390740 1 -- 172.21.15.98:0/3386963048 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac60b30860 con 0x55ac60bd80b0 2024-09-19T08:10:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.025+0000 7f4d99552640 1 --2- 172.21.15.98:0/3386963048 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55ac60bd80b0 0x55ac60bda4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.025+0000 7f4d99552640 1 --2- 172.21.15.98:0/3386963048 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55ac60bd80b0 0x55ac60bda4a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=feb8e25abd49d6b9 server_cookie=9b66a11ef01fdae8 in_seq=0 out_seq=0 2024-09-19T08:10:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.025+0000 7f4d7d7fa640 10 client.17280 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:10:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d7d7fa640 1 -- 172.21.15.98:0/3386963048 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ac60b30860 con 0x55ac60bd80b0 2024-09-19T08:10:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d7d7fa640 10 client.17280 handle_command_reply: tid=0 2024-09-19T08:10:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d9e390740 1 client.17280 shutdown 2024-09-19T08:10:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d9e390740 1 -- 172.21.15.98:0/3386963048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbd5d0 msgr2=0x55ac60bc7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d9e390740 1 --2- 172.21.15.98:0/3386963048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbd5d0 0x55ac60bc7dd0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f4d90011c30 tx=0x7f4d900046c0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d9e390740 1 -- 172.21.15.98:0/3386963048 shutdown_connections 2024-09-19T08:10:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d9e390740 1 --2- 172.21.15.98:0/3386963048 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55ac60bd80b0 0x55ac60bda4a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d9e390740 1 --2- 172.21.15.98:0/3386963048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bc8370 0x55ac60bc5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d9e390740 1 --2- 172.21.15.98:0/3386963048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bbeb40 0x55ac60bc5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d7d7fa640 0 client.17280 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:10:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d9e390740 1 --2- 172.21.15.98:0/3386963048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbd5d0 0x55ac60bc7dd0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d9e390740 1 -- 172.21.15.98:0/3386963048 >> 172.21.15.98:0/3386963048 conn(0x55ac60b86310 msgr2=0x55ac60bbf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d9e390740 1 -- 172.21.15.98:0/3386963048 shutdown_connections 2024-09-19T08:10:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d9e390740 1 -- 172.21.15.98:0/3386963048 wait complete. 2024-09-19T08:10:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.026+0000 7f4d9e390740 20 client.17280 trim_cache size 0 max 16384 2024-09-19T08:10:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d9e390740 1 Processor -- start 2024-09-19T08:10:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d9e390740 1 -- start start 2024-09-19T08:10:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d9e390740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbd5d0 0x55ac60bc5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d9e390740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bbeb40 0x55ac60bb4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d9e390740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bc8370 0x55ac60bdfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d9e390740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d941ad8d0 con 0x55ac60bbd5d0 2024-09-19T08:10:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d9e390740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4d941af410 con 0x55ac60bc8370 2024-09-19T08:10:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d9e390740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4d941adfa0 con 0x55ac60bbeb40 2024-09-19T08:10:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d99d53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbd5d0 0x55ac60bc5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d99d53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbd5d0 0x55ac60bc5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43494/0 (socket says 172.21.15.98:43494) 2024-09-19T08:10:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d99d53640 1 -- 172.21.15.98:0/3373030319 learned_addr learned my addr 172.21.15.98:0/3373030319 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d99552640 1 --2- 172.21.15.98:0/3373030319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bbeb40 0x55ac60bb4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d9a554640 1 --2- 172.21.15.98:0/3373030319 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bc8370 0x55ac60bdfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d99d53640 1 -- 172.21.15.98:0/3373030319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bbeb40 msgr2=0x55ac60bb4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d99d53640 1 --2- 172.21.15.98:0/3373030319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bbeb40 0x55ac60bb4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d99d53640 1 -- 172.21.15.98:0/3373030319 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bc8370 msgr2=0x55ac60bdfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d99d53640 1 --2- 172.21.15.98:0/3373030319 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bc8370 0x55ac60bdfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d99d53640 1 -- 172.21.15.98:0/3373030319 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac60bc4030 con 0x55ac60bbd5d0 2024-09-19T08:10:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.039+0000 7f4d99d53640 1 --2- 172.21.15.98:0/3373030319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbd5d0 0x55ac60bc5050 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f4d9001f010 tx=0x7f4d90009a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab4cf398e4e87221 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.040+0000 7f4d7f7fe640 1 -- 172.21.15.98:0/3373030319 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9003e070 con 0x55ac60bbd5d0 2024-09-19T08:10:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.040+0000 7f4d7f7fe640 1 -- 172.21.15.98:0/3373030319 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d900117f0 con 0x55ac60bbd5d0 2024-09-19T08:10:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.040+0000 7f4d7f7fe640 1 -- 172.21.15.98:0/3373030319 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d90035040 con 0x55ac60bbd5d0 2024-09-19T08:10:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.040+0000 7f4d9e390740 1 -- 172.21.15.98:0/3373030319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbd5d0 msgr2=0x55ac60bc5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.040+0000 7f4d9e390740 1 --2- 172.21.15.98:0/3373030319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbd5d0 0x55ac60bc5050 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f4d9001f010 tx=0x7f4d90009a80 comp rx=0 tx=0).stop 2024-09-19T08:10:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.040+0000 7f4d9e390740 1 -- 172.21.15.98:0/3373030319 shutdown_connections 2024-09-19T08:10:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.040+0000 7f4d9e390740 1 --2- 172.21.15.98:0/3373030319 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bc8370 0x55ac60bdfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.040+0000 7f4d9e390740 1 --2- 172.21.15.98:0/3373030319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bbeb40 0x55ac60bb4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.040+0000 7f4d9e390740 1 --2- 172.21.15.98:0/3373030319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbd5d0 0x55ac60bc5050 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.040+0000 7f4d9e390740 1 -- 172.21.15.98:0/3373030319 >> 172.21.15.98:0/3373030319 conn(0x55ac60b86310 msgr2=0x55ac60bc4940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:48.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.040+0000 7f4d9e390740 1 -- 172.21.15.98:0/3373030319 shutdown_connections 2024-09-19T08:10:48.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.040+0000 7f4d9e390740 1 -- 172.21.15.98:0/3373030319 wait complete. 2024-09-19T08:10:48.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.041+0000 7f4d9e390740 1 -- start start 2024-09-19T08:10:48.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.041+0000 7f4d9e390740 1 Processor -- start 2024-09-19T08:10:48.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.041+0000 7f4d9e390740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bbd5d0 0x55ac60bd0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.041+0000 7f4d9e390740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbeb40 0x55ac60bcbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.041+0000 7f4d9e390740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bc8370 0x55ac60bcc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.041+0000 7f4d9e390740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d941aed40 con 0x55ac60bbeb40 2024-09-19T08:10:48.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.041+0000 7f4d9e390740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4d941ae670 con 0x55ac60bbd5d0 2024-09-19T08:10:48.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.041+0000 7f4d9e390740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4d941ab8d0 con 0x55ac60bc8370 2024-09-19T08:10:48.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.041+0000 7f4d9e390740 10 client.0 _ll_register_callbacks cb 0x7f4d90009950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:48.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.041+0000 7f4d99552640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbeb40 0x55ac60bcbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.041+0000 7f4d99552640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbeb40 0x55ac60bcbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43510/0 (socket says 172.21.15.98:43510) 2024-09-19T08:10:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.041+0000 7f4d99552640 1 -- 172.21.15.98:0/4019929518 learned_addr learned my addr 172.21.15.98:0/4019929518 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.041+0000 7f4d9a554640 1 --2- 172.21.15.98:0/4019929518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bc8370 0x55ac60bcc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d99d53640 1 --2- 172.21.15.98:0/4019929518 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bbd5d0 0x55ac60bd0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d99552640 1 -- 172.21.15.98:0/4019929518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bc8370 msgr2=0x55ac60bcc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d99552640 1 --2- 172.21.15.98:0/4019929518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bc8370 0x55ac60bcc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d99552640 1 -- 172.21.15.98:0/4019929518 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bbd5d0 msgr2=0x55ac60bd0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d99552640 1 --2- 172.21.15.98:0/4019929518 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bbd5d0 0x55ac60bd0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d99552640 1 -- 172.21.15.98:0/4019929518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac60bccda0 con 0x55ac60bbeb40 2024-09-19T08:10:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d99552640 1 --2- 172.21.15.98:0/4019929518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbeb40 0x55ac60bcbff0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f4d840bc630 tx=0x7f4d840f0500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55949fe3cac9424a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d7d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d7d7fa640 1 -- 172.21.15.98:0/4019929518 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d840d9070 con 0x55ac60bbeb40 2024-09-19T08:10:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d7d7fa640 1 -- 172.21.15.98:0/4019929518 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f4d840edcd0 con 0x55ac60bbeb40 2024-09-19T08:10:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d9e390740 1 -- 172.21.15.98:0/4019929518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ac60b2ad70 con 0x55ac60bbeb40 2024-09-19T08:10:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d7d7fa640 1 -- 172.21.15.98:0/4019929518 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4d84049040 con 0x55ac60bbeb40 2024-09-19T08:10:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d9e390740 10 client.17286 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d9e390740 1 -- 172.21.15.98:0/4019929518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ac60c42da0 con 0x55ac60bbeb40 2024-09-19T08:10:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d7d7fa640 1 -- 172.21.15.98:0/4019929518 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ac60b2ad70 con 0x55ac60bbeb40 2024-09-19T08:10:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.042+0000 7f4d7d7fa640 1 -- 172.21.15.98:0/4019929518 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4d840b8d10 con 0x55ac60bbeb40 2024-09-19T08:10:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.043+0000 7f4d9e390740 10 client.17286 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.043+0000 7f4d9e390740 10 client.17286 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:10:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.043+0000 7f4d9e390740 20 client.17286 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.043+0000 7f4d9e390740 1 --2- 172.21.15.98:0/4019929518 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55ac60c434f0 0x55ac60c45910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.043+0000 7f4d9e390740 4 client.17286 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:10:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.043+0000 7f4d9e390740 1 -- 172.21.15.98:0/4019929518 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ac60aa7170 con 0x55ac60c434f0 2024-09-19T08:10:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.043+0000 7f4d99d53640 1 --2- 172.21.15.98:0/4019929518 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55ac60c434f0 0x55ac60c45910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.043+0000 7f4d99d53640 1 --2- 172.21.15.98:0/4019929518 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55ac60c434f0 0x55ac60c45910 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6f895b2e075e9ff3 server_cookie=5bb2deb189aa9877 in_seq=0 out_seq=0 2024-09-19T08:10:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.043+0000 7f4d7d7fa640 10 client.17286 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:10:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.043+0000 7f4d7d7fa640 1 -- 172.21.15.98:0/4019929518 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ac60aa7170 con 0x55ac60c434f0 2024-09-19T08:10:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.043+0000 7f4d7d7fa640 10 client.17286 handle_command_reply: tid=0 2024-09-19T08:10:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.043+0000 7f4d9e390740 1 client.17286 shutdown 2024-09-19T08:10:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.043+0000 7f4d9e390740 1 -- 172.21.15.98:0/4019929518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbeb40 msgr2=0x55ac60bcbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.043+0000 7f4d9e390740 1 --2- 172.21.15.98:0/4019929518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbeb40 0x55ac60bcbff0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f4d840bc630 tx=0x7f4d840f0500 comp rx=0 tx=0).stop 2024-09-19T08:10:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9e390740 1 -- 172.21.15.98:0/4019929518 shutdown_connections 2024-09-19T08:10:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9e390740 1 --2- 172.21.15.98:0/4019929518 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55ac60c434f0 0x55ac60c45910 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9e390740 1 --2- 172.21.15.98:0/4019929518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac60bc8370 0x55ac60bcc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9e390740 1 --2- 172.21.15.98:0/4019929518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ac60bbeb40 0x55ac60bcbff0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9e390740 1 --2- 172.21.15.98:0/4019929518 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ac60bbd5d0 0x55ac60bd0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d7d7fa640 0 client.17286 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:10:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9e390740 1 -- 172.21.15.98:0/4019929518 >> 172.21.15.98:0/4019929518 conn(0x55ac60b86310 msgr2=0x55ac60bb84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9e390740 1 -- 172.21.15.98:0/4019929518 shutdown_connections 2024-09-19T08:10:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9e390740 1 -- 172.21.15.98:0/4019929518 wait complete. 2024-09-19T08:10:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9e390740 20 client.17286 trim_cache size 0 max 16384 2024-09-19T08:10:48.050 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:10:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9bfb7640 1 -- 172.21.15.98:0/116018283 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4d800a33b0 msgr2=0x7f4d800a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9bfb7640 1 --2- 172.21.15.98:0/116018283 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4d800a33b0 0x7f4d800a5870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55ac60bbaac0 tx=0x7f4d90005f90 comp rx=0 tx=0).stop 2024-09-19T08:10:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9bfb7640 1 -- 172.21.15.98:0/116018283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d94100eb0 msgr2=0x7f4d94198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9bfb7640 1 --2- 172.21.15.98:0/116018283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d94100eb0 0x7f4d94198a00 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4d84006010 tx=0x7f4d840042f0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9bfb7640 1 -- 172.21.15.98:0/116018283 shutdown_connections 2024-09-19T08:10:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9bfb7640 1 --2- 172.21.15.98:0/116018283 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4d800a33b0 0x7f4d800a5870 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9bfb7640 1 --2- 172.21.15.98:0/116018283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d94106ed0 0x7f4d9419cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9bfb7640 1 --2- 172.21.15.98:0/116018283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d94100eb0 0x7f4d94198a00 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9bfb7640 1 --2- 172.21.15.98:0/116018283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d94100500 0x7f4d941984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9bfb7640 1 -- 172.21.15.98:0/116018283 >> 172.21.15.98:0/116018283 conn(0x7f4d940fc5c0 msgr2=0x7f4d94105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9bfb7640 1 -- 172.21.15.98:0/116018283 shutdown_connections 2024-09-19T08:10:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:48.044+0000 7f4d9bfb7640 1 -- 172.21.15.98:0/116018283 wait complete. 2024-09-19T08:10:48.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:48 smithi162 ceph-mon[35260]: pgmap v238: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2024-09-19T08:10:48.386 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:48 smithi098 ceph-mon[31333]: pgmap v238: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2024-09-19T08:10:48.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:48 smithi139 ceph-mon[34874]: pgmap v238: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2024-09-19T08:10:48.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:10:49.063 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:49.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.514+0000 7f3ba39c3640 1 -- 172.21.15.98:0/3936373486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c102790 msgr2=0x7f3b9c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:49.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.514+0000 7f3ba39c3640 1 --2- 172.21.15.98:0/3936373486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c102790 0x7f3b9c1067e0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c009900 tx=0x7f3b8c02a410 comp rx=0 tx=0).stop 2024-09-19T08:10:49.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.514+0000 7f3ba39c3640 1 -- 172.21.15.98:0/3936373486 shutdown_connections 2024-09-19T08:10:49.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.514+0000 7f3ba39c3640 1 --2- 172.21.15.98:0/3936373486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b9c106f10 0x7f3b9c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.514+0000 7f3ba39c3640 1 --2- 172.21.15.98:0/3936373486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c102790 0x7f3b9c1067e0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.514+0000 7f3ba39c3640 1 --2- 172.21.15.98:0/3936373486 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3b9c101de0 0x7f3b9c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.514+0000 7f3ba39c3640 1 -- 172.21.15.98:0/3936373486 >> 172.21.15.98:0/3936373486 conn(0x7f3b9c0fd8c0 msgr2=0x7f3b9c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:49.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.514+0000 7f3ba39c3640 1 -- 172.21.15.98:0/3936373486 shutdown_connections 2024-09-19T08:10:49.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.514+0000 7f3ba39c3640 1 -- 172.21.15.98:0/3936373486 wait complete. 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.515+0000 7f3ba39c3640 1 Processor -- start 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.515+0000 7f3ba39c3640 1 -- start start 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.515+0000 7f3ba39c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c101de0 0x7f3b9c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.515+0000 7f3ba39c3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b9c102790 0x7f3b9c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.515+0000 7f3ba39c3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3b9c106f10 0x7f3b9c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.515+0000 7f3ba39c3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c111fb0 con 0x7f3b9c101de0 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.515+0000 7f3ba39c3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c111e30 con 0x7f3b9c106f10 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.515+0000 7f3ba39c3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c112130 con 0x7f3b9c102790 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.515+0000 7f3ba175f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c101de0 0x7f3b9c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.515+0000 7f3ba175f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c101de0 0x7f3b9c198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43532/0 (socket says 172.21.15.98:43532) 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.515+0000 7f3ba175f640 1 -- 172.21.15.98:0/1841161376 learned_addr learned my addr 172.21.15.98:0/1841161376 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.515+0000 7f3ba0f5e640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b9c102790 0x7f3b9c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.515+0000 7f3ba1f60640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3b9c106f10 0x7f3b9c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.516+0000 7f3ba175f640 1 -- 172.21.15.98:0/1841161376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b9c102790 msgr2=0x7f3b9c1989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.516+0000 7f3ba175f640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b9c102790 0x7f3b9c1989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.516+0000 7f3ba175f640 1 -- 172.21.15.98:0/1841161376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3b9c106f10 msgr2=0x7f3b9c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.516+0000 7f3ba175f640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3b9c106f10 0x7f3b9c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.516+0000 7f3ba175f640 1 -- 172.21.15.98:0/1841161376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b9c19d440 con 0x7f3b9c101de0 2024-09-19T08:10:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.516+0000 7f3ba1f60640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3b9c106f10 0x7f3b9c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.516+0000 7f3ba175f640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c101de0 0x7f3b9c198490 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3b9800b4a0 tx=0x7f3b9800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.516+0000 7f3b927fc640 1 -- 172.21.15.98:0/1841161376 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98014070 con 0x7f3b9c101de0 2024-09-19T08:10:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.516+0000 7f3b927fc640 1 -- 172.21.15.98:0/1841161376 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b98010af0 con 0x7f3b9c101de0 2024-09-19T08:10:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.516+0000 7f3ba39c3640 1 -- 172.21.15.98:0/1841161376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b9c19d730 con 0x7f3b9c101de0 2024-09-19T08:10:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.516+0000 7f3b927fc640 1 -- 172.21.15.98:0/1841161376 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98017d10 con 0x7f3b9c101de0 2024-09-19T08:10:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.516+0000 7f3ba39c3640 1 -- 172.21.15.98:0/1841161376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b9c1a5010 con 0x7f3b9c101de0 2024-09-19T08:10:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.518+0000 7f3b927fc640 1 -- 172.21.15.98:0/1841161376 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f3b98017070 con 0x7f3b9c101de0 2024-09-19T08:10:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.518+0000 7f3ba5d9c740 1 Processor -- start 2024-09-19T08:10:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba5d9c740 1 -- start start 2024-09-19T08:10:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba5d9c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b584090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba5d9c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b5845d0 0x55622b584a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3b927fc640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3b880a33b0 0x7f3b880a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba5d9c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b58f370 0x55622b591760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba5d9c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55622b4ea2f0 con 0x55622b585b40 2024-09-19T08:10:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba5d9c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55622b491cf0 con 0x55622b58f370 2024-09-19T08:10:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba5d9c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55622b4e62b0 con 0x55622b5845d0 2024-09-19T08:10:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba1f60640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b584090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba1f60640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b584090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43536/0 (socket says 172.21.15.98:43536) 2024-09-19T08:10:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba1f60640 1 -- 172.21.15.98:0/1633714990 learned_addr learned my addr 172.21.15.98:0/1633714990 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba0f5e640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3b880a33b0 0x7f3b880a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3b927fc640 1 -- 172.21.15.98:0/1841161376 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f3b980cefe0 con 0x7f3b9c101de0 2024-09-19T08:10:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba175f640 1 --2- 172.21.15.98:0/1633714990 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b5845d0 0x55622b584a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba1f60640 1 --2- 172.21.15.98:0/1633714990 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b58f370 0x55622b591760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba1f60640 1 -- 172.21.15.98:0/1633714990 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b5845d0 msgr2=0x55622b584a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba1f60640 1 --2- 172.21.15.98:0/1633714990 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b5845d0 0x55622b584a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba1f60640 1 -- 172.21.15.98:0/1633714990 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b58f370 msgr2=0x55622b591760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba1f60640 1 --2- 172.21.15.98:0/1633714990 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b58f370 0x55622b591760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba1f60640 1 -- 172.21.15.98:0/1633714990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55622b4763a0 con 0x55622b585b40 2024-09-19T08:10:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.519+0000 7f3ba0f5e640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3b880a33b0 0x7f3b880a5870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c009230 tx=0x7f3b8c0058d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.520+0000 7f3ba1f60640 1 --2- 172.21.15.98:0/1633714990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b584090 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x55622b585070 tx=0x7f3b940099a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4035bf9c15312 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.520+0000 7f3b877fe640 1 -- 172.21.15.98:0/1633714990 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9401a020 con 0x55622b585b40 2024-09-19T08:10:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.520+0000 7f3b877fe640 1 -- 172.21.15.98:0/1633714990 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b94012d70 con 0x55622b585b40 2024-09-19T08:10:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.520+0000 7f3b877fe640 1 -- 172.21.15.98:0/1633714990 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b94010920 con 0x55622b585b40 2024-09-19T08:10:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.520+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/1633714990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 msgr2=0x55622b584090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:49.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.520+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/1633714990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b584090 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x55622b585070 tx=0x7f3b940099a0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/1633714990 shutdown_connections 2024-09-19T08:10:49.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/1633714990 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b58f370 0x55622b591760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/1633714990 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b5845d0 0x55622b584a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/1633714990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b584090 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/1633714990 >> 172.21.15.98:0/1633714990 conn(0x55622b54d310 msgr2=0x55622b580c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:49.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/1633714990 shutdown_connections 2024-09-19T08:10:49.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/1633714990 wait complete. 2024-09-19T08:10:49.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 -- start start 2024-09-19T08:10:49.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 Processor -- start 2024-09-19T08:10:49.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b5845d0 0x55622b58ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b58bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b58f370 0x55622b58c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55622b592070 con 0x55622b585b40 2024-09-19T08:10:49.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55622b4a5060 con 0x55622b58f370 2024-09-19T08:10:49.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55622b5921f0 con 0x55622b5845d0 2024-09-19T08:10:49.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba5d9c740 10 client.0 _ll_register_callbacks cb 0x55622b1f88c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:49.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba0f5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b58bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba0f5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b58bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43540/0 (socket says 172.21.15.98:43540) 2024-09-19T08:10:49.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.521+0000 7f3ba0f5e640 1 -- 172.21.15.98:0/2372042229 learned_addr learned my addr 172.21.15.98:0/2372042229 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:49.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3ba175f640 1 --2- 172.21.15.98:0/2372042229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b5845d0 0x55622b58ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3ba0f5e640 1 -- 172.21.15.98:0/2372042229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b5845d0 msgr2=0x55622b58ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:49.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3ba0f5e640 1 --2- 172.21.15.98:0/2372042229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b5845d0 0x55622b58ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3ba0f5e640 1 -- 172.21.15.98:0/2372042229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b58f370 msgr2=0x55622b58c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:49.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3ba0f5e640 1 --2- 172.21.15.98:0/2372042229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b58f370 0x55622b58c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3ba0f5e640 1 -- 172.21.15.98:0/2372042229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55622b25b3c0 con 0x55622b585b40 2024-09-19T08:10:49.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3ba0f5e640 1 --2- 172.21.15.98:0/2372042229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b58bfd0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c009900 tx=0x7f3b8c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ebede2b2ef06a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:49.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3b84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:49.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3b84ff9640 1 -- 172.21.15.98:0/2372042229 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c04d020 con 0x55622b585b40 2024-09-19T08:10:49.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3b84ff9640 1 -- 172.21.15.98:0/2372042229 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f3b8c0035f0 con 0x55622b585b40 2024-09-19T08:10:49.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2372042229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b94012d70 con 0x55622b585b40 2024-09-19T08:10:49.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3b84ff9640 1 -- 172.21.15.98:0/2372042229 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3b8c058050 con 0x55622b585b40 2024-09-19T08:10:49.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3ba5d9c740 10 client.17298 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:49.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2372042229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55622b59dac0 con 0x55622b585b40 2024-09-19T08:10:49.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.522+0000 7f3b84ff9640 1 -- 172.21.15.98:0/2372042229 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b94012d70 con 0x55622b585b40 2024-09-19T08:10:49.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.523+0000 7f3b84ff9640 1 -- 172.21.15.98:0/2372042229 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3b8c0498e0 con 0x55622b585b40 2024-09-19T08:10:49.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.523+0000 7f3ba5d9c740 10 client.17298 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:49.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.523+0000 7f3ba5d9c740 10 client.17298 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:10:49.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.523+0000 7f3ba5d9c740 20 client.17298 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:49.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.523+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/2372042229 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55622b59efd0 0x55622b5a1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.523+0000 7f3ba5d9c740 4 client.17298 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:49.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.523+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2372042229 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55622b4f7860 con 0x55622b59efd0 2024-09-19T08:10:49.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.523+0000 7f3ba175f640 1 --2- 172.21.15.98:0/2372042229 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55622b59efd0 0x55622b5a1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.523+0000 7f3b84ff9640 10 client.17298 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:10:49.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.523+0000 7f3ba175f640 1 --2- 172.21.15.98:0/2372042229 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55622b59efd0 0x55622b5a1430 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bf1449cf9da59751 server_cookie=c199f8b8c0cba5e8 in_seq=0 out_seq=0 2024-09-19T08:10:49.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3b84ff9640 1 -- 172.21.15.98:0/2372042229 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55622b4f7860 con 0x55622b59efd0 2024-09-19T08:10:49.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3b84ff9640 10 client.17298 handle_command_reply: tid=0 2024-09-19T08:10:49.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3ba5d9c740 1 client.17298 shutdown 2024-09-19T08:10:49.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2372042229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 msgr2=0x55622b58bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:49.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/2372042229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b58bfd0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c009900 tx=0x7f3b8c009260 comp rx=0 tx=0).stop 2024-09-19T08:10:49.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2372042229 shutdown_connections 2024-09-19T08:10:49.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/2372042229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b58f370 0x55622b58c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/2372042229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b58bfd0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/2372042229 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55622b59efd0 0x55622b5a1430 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/2372042229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b5845d0 0x55622b58ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3b84ff9640 0 client.17298 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:10:49.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2372042229 >> 172.21.15.98:0/2372042229 conn(0x55622b54d310 msgr2=0x55622b586960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:49.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2372042229 shutdown_connections 2024-09-19T08:10:49.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2372042229 wait complete. 2024-09-19T08:10:49.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.524+0000 7f3ba5d9c740 20 client.17298 trim_cache size 0 max 16384 2024-09-19T08:10:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba5d9c740 1 Processor -- start 2024-09-19T08:10:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba5d9c740 1 -- start start 2024-09-19T08:10:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba5d9c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b5845d0 0x55622b58b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba5d9c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b587120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba5d9c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b58f370 0x55622b587660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba5d9c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1ad930 con 0x55622b585b40 2024-09-19T08:10:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba5d9c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1af470 con 0x55622b5845d0 2024-09-19T08:10:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba5d9c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1ae000 con 0x55622b58f370 2024-09-19T08:10:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba0f5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b587120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba0f5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b587120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43556/0 (socket says 172.21.15.98:43556) 2024-09-19T08:10:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba0f5e640 1 -- 172.21.15.98:0/793903144 learned_addr learned my addr 172.21.15.98:0/793903144 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba1f60640 1 --2- 172.21.15.98:0/793903144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b58f370 0x55622b587660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba0f5e640 1 -- 172.21.15.98:0/793903144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b58f370 msgr2=0x55622b587660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba175f640 1 --2- 172.21.15.98:0/793903144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b5845d0 0x55622b58b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba0f5e640 1 --2- 172.21.15.98:0/793903144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b58f370 0x55622b587660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba0f5e640 1 -- 172.21.15.98:0/793903144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b5845d0 msgr2=0x55622b58b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba0f5e640 1 --2- 172.21.15.98:0/793903144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b5845d0 0x55622b58b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba0f5e640 1 -- 172.21.15.98:0/793903144 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55622b58af80 con 0x55622b585b40 2024-09-19T08:10:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.539+0000 7f3ba175f640 1 --2- 172.21.15.98:0/793903144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b5845d0 0x55622b58b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.540+0000 7f3ba0f5e640 1 --2- 172.21.15.98:0/793903144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b587120 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c049ef0 tx=0x7f3b8c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a24759ab6b9b2de server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.540+0000 7f3b86ffd640 1 -- 172.21.15.98:0/793903144 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c04d040 con 0x55622b585b40 2024-09-19T08:10:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.540+0000 7f3b86ffd640 1 -- 172.21.15.98:0/793903144 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b8c049960 con 0x55622b585b40 2024-09-19T08:10:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.540+0000 7f3b86ffd640 1 -- 172.21.15.98:0/793903144 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c035040 con 0x55622b585b40 2024-09-19T08:10:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.540+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/793903144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 msgr2=0x55622b587120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.540+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/793903144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b587120 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c049ef0 tx=0x7f3b8c005a20 comp rx=0 tx=0).stop 2024-09-19T08:10:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.540+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/793903144 shutdown_connections 2024-09-19T08:10:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.540+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/793903144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b58f370 0x55622b587660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.540+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/793903144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b587120 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.540+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/793903144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b5845d0 0x55622b58b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.540+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/793903144 >> 172.21.15.98:0/793903144 conn(0x55622b54d310 msgr2=0x55622b58e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.540+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/793903144 shutdown_connections 2024-09-19T08:10:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.541+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/793903144 wait complete. 2024-09-19T08:10:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.541+0000 7f3ba5d9c740 1 -- start start 2024-09-19T08:10:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.541+0000 7f3ba5d9c740 1 Processor -- start 2024-09-19T08:10:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.541+0000 7f3ba5d9c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b5845d0 0x55622b597480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.541+0000 7f3ba5d9c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b5979c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.541+0000 7f3ba5d9c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b58f370 0x55622b594450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.541+0000 7f3ba5d9c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1aeda0 con 0x55622b585b40 2024-09-19T08:10:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.541+0000 7f3ba5d9c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1ae6d0 con 0x55622b5845d0 2024-09-19T08:10:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.541+0000 7f3ba5d9c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1ab930 con 0x55622b58f370 2024-09-19T08:10:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.541+0000 7f3ba5d9c740 10 client.0 _ll_register_callbacks cb 0x7f3b8c040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.541+0000 7f3ba0f5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b5979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.541+0000 7f3ba0f5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b5979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43562/0 (socket says 172.21.15.98:43562) 2024-09-19T08:10:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.541+0000 7f3ba0f5e640 1 -- 172.21.15.98:0/2390840711 learned_addr learned my addr 172.21.15.98:0/2390840711 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3ba1f60640 1 --2- 172.21.15.98:0/2390840711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b58f370 0x55622b594450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3ba0f5e640 1 -- 172.21.15.98:0/2390840711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b58f370 msgr2=0x55622b594450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3ba175f640 1 --2- 172.21.15.98:0/2390840711 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b5845d0 0x55622b597480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3ba0f5e640 1 --2- 172.21.15.98:0/2390840711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b58f370 0x55622b594450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3ba0f5e640 1 -- 172.21.15.98:0/2390840711 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b5845d0 msgr2=0x55622b597480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3ba0f5e640 1 --2- 172.21.15.98:0/2390840711 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b5845d0 0x55622b597480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3ba0f5e640 1 -- 172.21.15.98:0/2390840711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55622b594990 con 0x55622b585b40 2024-09-19T08:10:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3ba1f60640 1 --2- 172.21.15.98:0/2390840711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b58f370 0x55622b594450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3ba0f5e640 1 --2- 172.21.15.98:0/2390840711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b5979c0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c005940 tx=0x7f3b8c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f2020ead02b0fd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3b84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3b84ff9640 1 -- 172.21.15.98:0/2390840711 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c06b020 con 0x55622b585b40 2024-09-19T08:10:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3b84ff9640 1 -- 172.21.15.98:0/2390840711 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f3b8c03c8d0 con 0x55622b585b40 2024-09-19T08:10:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2390840711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b8c049960 con 0x55622b585b40 2024-09-19T08:10:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3b84ff9640 1 -- 172.21.15.98:0/2390840711 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3b8c076050 con 0x55622b585b40 2024-09-19T08:10:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3ba5d9c740 10 client.17304 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2390840711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55622b60e880 con 0x55622b585b40 2024-09-19T08:10:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.542+0000 7f3b84ff9640 1 -- 172.21.15.98:0/2390840711 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b8c049960 con 0x55622b585b40 2024-09-19T08:10:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.543+0000 7f3b84ff9640 1 -- 172.21.15.98:0/2390840711 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3b8c003cd0 con 0x55622b585b40 2024-09-19T08:10:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.543+0000 7f3ba5d9c740 10 client.17304 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.543+0000 7f3ba5d9c740 10 client.17304 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:10:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.543+0000 7f3ba5d9c740 20 client.17304 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.543+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/2390840711 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55622b60ef40 0x55622b611360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.543+0000 7f3ba5d9c740 4 client.17304 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:10:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.543+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2390840711 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55622b4f1d70 con 0x55622b60ef40 2024-09-19T08:10:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.543+0000 7f3ba175f640 1 --2- 172.21.15.98:0/2390840711 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55622b60ef40 0x55622b611360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.543+0000 7f3ba175f640 1 --2- 172.21.15.98:0/2390840711 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55622b60ef40 0x55622b611360 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=30ba6d1b7bb951bb server_cookie=62f4c13fd9c4d6 in_seq=0 out_seq=0 2024-09-19T08:10:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.543+0000 7f3b84ff9640 10 client.17304 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:10:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.543+0000 7f3b84ff9640 1 -- 172.21.15.98:0/2390840711 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55622b4f1d70 con 0x55622b60ef40 2024-09-19T08:10:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.543+0000 7f3b84ff9640 10 client.17304 handle_command_reply: tid=0 2024-09-19T08:10:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3ba5d9c740 1 client.17304 shutdown 2024-09-19T08:10:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2390840711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 msgr2=0x55622b5979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/2390840711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b5979c0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c005940 tx=0x7f3b8c009260 comp rx=0 tx=0).stop 2024-09-19T08:10:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2390840711 shutdown_connections 2024-09-19T08:10:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/2390840711 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x55622b60ef40 0x55622b611360 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/2390840711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55622b58f370 0x55622b594450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/2390840711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55622b585b40 0x55622b5979c0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3ba5d9c740 1 --2- 172.21.15.98:0/2390840711 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55622b5845d0 0x55622b597480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2390840711 >> 172.21.15.98:0/2390840711 conn(0x55622b54d310 msgr2=0x55622b58e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3b84ff9640 0 client.17304 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:10:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2390840711 shutdown_connections 2024-09-19T08:10:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3ba5d9c740 1 -- 172.21.15.98:0/2390840711 wait complete. 2024-09-19T08:10:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3ba5d9c740 20 client.17304 trim_cache size 0 max 16384 2024-09-19T08:10:49.551 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:10:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3ba39c3640 1 -- 172.21.15.98:0/1841161376 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3b880a33b0 msgr2=0x7f3b880a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.544+0000 7f3ba39c3640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3b880a33b0 0x7f3b880a5870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c009230 tx=0x7f3b8c0058d0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.545+0000 7f3ba39c3640 1 -- 172.21.15.98:0/1841161376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c101de0 msgr2=0x7f3b9c198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.545+0000 7f3ba39c3640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c101de0 0x7f3b9c198490 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3b9800b4a0 tx=0x7f3b9800b960 comp rx=0 tx=0).stop 2024-09-19T08:10:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.545+0000 7f3ba39c3640 1 -- 172.21.15.98:0/1841161376 shutdown_connections 2024-09-19T08:10:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.545+0000 7f3ba39c3640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3b880a33b0 0x7f3b880a5870 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.545+0000 7f3ba39c3640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3b9c106f10 0x7f3b9c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.545+0000 7f3ba39c3640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b9c102790 0x7f3b9c1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.545+0000 7f3ba39c3640 1 --2- 172.21.15.98:0/1841161376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b9c101de0 0x7f3b9c198490 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.545+0000 7f3ba39c3640 1 -- 172.21.15.98:0/1841161376 >> 172.21.15.98:0/1841161376 conn(0x7f3b9c0fd8c0 msgr2=0x7f3b9c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.545+0000 7f3ba39c3640 1 -- 172.21.15.98:0/1841161376 shutdown_connections 2024-09-19T08:10:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:49.545+0000 7f3ba39c3640 1 -- 172.21.15.98:0/1841161376 wait complete. 2024-09-19T08:10:50.266 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:50 smithi098 ceph-mon[31333]: pgmap v239: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-09-19T08:10:50.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:10:50.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:50 smithi162 ceph-mon[35260]: pgmap v239: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-09-19T08:10:50.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:50 smithi139 ceph-mon[34874]: pgmap v239: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-09-19T08:10:50.511 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.958+0000 7ffa15285640 1 -- 172.21.15.98:0/2031294212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa10102d50 msgr2=0x7ffa10103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.958+0000 7ffa15285640 1 --2- 172.21.15.98:0/2031294212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa10102d50 0x7ffa10103130 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009900 tx=0x7ffa0002a480 comp rx=0 tx=0).stop 2024-09-19T08:10:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.958+0000 7ffa15285640 1 -- 172.21.15.98:0/2031294212 shutdown_connections 2024-09-19T08:10:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.958+0000 7ffa15285640 1 --2- 172.21.15.98:0/2031294212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa1010c180 0x7ffa1010e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.958+0000 7ffa15285640 1 --2- 172.21.15.98:0/2031294212 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffa10103670 0x7ffa1010bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.958+0000 7ffa15285640 1 --2- 172.21.15.98:0/2031294212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa10102d50 0x7ffa10103130 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.958+0000 7ffa15285640 1 -- 172.21.15.98:0/2031294212 >> 172.21.15.98:0/2031294212 conn(0x7ffa100fc7d0 msgr2=0x7ffa100febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.958+0000 7ffa15285640 1 -- 172.21.15.98:0/2031294212 shutdown_connections 2024-09-19T08:10:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.958+0000 7ffa15285640 1 -- 172.21.15.98:0/2031294212 wait complete. 2024-09-19T08:10:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.959+0000 7ffa15285640 1 Processor -- start 2024-09-19T08:10:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.959+0000 7ffa15285640 1 -- start start 2024-09-19T08:10:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.959+0000 7ffa15285640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa10102d50 0x7ffa10198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.959+0000 7ffa15285640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa10103670 0x7ffa10198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.959+0000 7ffa15285640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffa1010c180 0x7ffa1019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.959+0000 7ffa15285640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa10111f80 con 0x7ffa10103670 2024-09-19T08:10:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.959+0000 7ffa15285640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ffa10111e00 con 0x7ffa1010c180 2024-09-19T08:10:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.959+0000 7ffa15285640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffa10112100 con 0x7ffa10102d50 2024-09-19T08:10:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.959+0000 7ffa0e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa10103670 0x7ffa10198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ffa0e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa10103670 0x7ffa10198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43588/0 (socket says 172.21.15.98:43588) 2024-09-19T08:10:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ffa0e7fc640 1 -- 172.21.15.98:0/515026437 learned_addr learned my addr 172.21.15.98:0/515026437 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ffa0effd640 1 --2- 172.21.15.98:0/515026437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa10102d50 0x7ffa10198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ffa0f7fe640 1 --2- 172.21.15.98:0/515026437 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffa1010c180 0x7ffa1019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ffa0e7fc640 1 -- 172.21.15.98:0/515026437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa10102d50 msgr2=0x7ffa10198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/515026437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa10102d50 0x7ffa10198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ffa0e7fc640 1 -- 172.21.15.98:0/515026437 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffa1010c180 msgr2=0x7ffa1019cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/515026437 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffa1010c180 0x7ffa1019cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ffa0e7fc640 1 -- 172.21.15.98:0/515026437 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1019d4b0 con 0x7ffa10103670 2024-09-19T08:10:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/515026437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa10103670 0x7ffa10198a40 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400d950 tx=0x7ffa0400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ff9f7fff640 1 -- 172.21.15.98:0/515026437 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa04017030 con 0x7ffa10103670 2024-09-19T08:10:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ffa15285640 1 -- 172.21.15.98:0/515026437 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa1019d7a0 con 0x7ffa10103670 2024-09-19T08:10:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ff9f7fff640 1 -- 172.21.15.98:0/515026437 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa04004020 con 0x7ffa10103670 2024-09-19T08:10:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ff9f7fff640 1 -- 172.21.15.98:0/515026437 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa040047a0 con 0x7ffa10103670 2024-09-19T08:10:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.960+0000 7ffa15285640 1 -- 172.21.15.98:0/515026437 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa101a5080 con 0x7ffa10103670 2024-09-19T08:10:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.962+0000 7ff9f7fff640 1 -- 172.21.15.98:0/515026437 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7ffa040041d0 con 0x7ffa10103670 2024-09-19T08:10:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ffa1765e740 1 Processor -- start 2024-09-19T08:10:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ffa1765e740 1 -- start start 2024-09-19T08:10:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ffa1765e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 0x55dabcb57090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ffa1765e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb575d0 0x55dabcb57a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ffa1765e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb62370 0x55dabcb64760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ffa1765e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dabcabd2f0 con 0x55dabcb62370 2024-09-19T08:10:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ffa1765e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dabca64cf0 con 0x55dabcb58b40 2024-09-19T08:10:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ffa1765e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dabcab92b0 con 0x55dabcb575d0 2024-09-19T08:10:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ff9f7fff640 1 --2- 172.21.15.98:0/515026437 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff9f80a33b0 0x7ff9f80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ffa0e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb62370 0x55dabcb64760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ffa0f7fe640 1 --2- 172.21.15.98:0/515026437 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff9f80a33b0 0x7ff9f80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ffa0e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb62370 0x55dabcb64760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43592/0 (socket says 172.21.15.98:43592) 2024-09-19T08:10:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ffa0e7fc640 1 -- 172.21.15.98:0/3840444655 learned_addr learned my addr 172.21.15.98:0/3840444655 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ffa0f7fe640 1 --2- 172.21.15.98:0/3840444655 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 0x55dabcb57090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ffa0effd640 1 --2- 172.21.15.98:0/3840444655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb575d0 0x55dabcb57a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.963+0000 7ff9f7fff640 1 -- 172.21.15.98:0/515026437 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7ffa040cf5f0 con 0x7ffa10103670 2024-09-19T08:10:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.964+0000 7ffa0e7fc640 1 -- 172.21.15.98:0/3840444655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb575d0 msgr2=0x55dabcb57a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.964+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/3840444655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb575d0 0x55dabcb57a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.964+0000 7ffa0e7fc640 1 -- 172.21.15.98:0/3840444655 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 msgr2=0x55dabcb57090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.964+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/3840444655 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 0x55dabcb57090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.964+0000 7ffa0e7fc640 1 -- 172.21.15.98:0/3840444655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dabca493a0 con 0x55dabcb62370 2024-09-19T08:10:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.964+0000 7ffa0f7fe640 1 --2- 172.21.15.98:0/515026437 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff9f80a33b0 0x7ff9f80a5870 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffa08005cc0 tx=0x7ffa0800ad50 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.964+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/3840444655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb62370 0x55dabcb64760 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7ffa0409b860 tx=0x7ffa040c66a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef3c781b98f491fb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.964+0000 7ff9f57fa640 1 -- 172.21.15.98:0/3840444655 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0409b6a0 con 0x55dabcb62370 2024-09-19T08:10:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.964+0000 7ff9f57fa640 1 -- 172.21.15.98:0/3840444655 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa04084bd0 con 0x55dabcb62370 2024-09-19T08:10:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.964+0000 7ff9f57fa640 1 -- 172.21.15.98:0/3840444655 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0409b230 con 0x55dabcb62370 2024-09-19T08:10:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 -- 172.21.15.98:0/3840444655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb62370 msgr2=0x55dabcb64760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 --2- 172.21.15.98:0/3840444655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb62370 0x55dabcb64760 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7ffa0409b860 tx=0x7ffa040c66a0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 -- 172.21.15.98:0/3840444655 shutdown_connections 2024-09-19T08:10:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 --2- 172.21.15.98:0/3840444655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb62370 0x55dabcb64760 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 --2- 172.21.15.98:0/3840444655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb575d0 0x55dabcb57a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 --2- 172.21.15.98:0/3840444655 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 0x55dabcb57090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 -- 172.21.15.98:0/3840444655 >> 172.21.15.98:0/3840444655 conn(0x55dabcb20310 msgr2=0x55dabcb53c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 -- 172.21.15.98:0/3840444655 shutdown_connections 2024-09-19T08:10:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 -- 172.21.15.98:0/3840444655 wait complete. 2024-09-19T08:10:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 -- start start 2024-09-19T08:10:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 Processor -- start 2024-09-19T08:10:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb575d0 0x55dabcb61db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 0x55dabcb5f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 0x55dabcb5f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dabcb65010 con 0x55dabcb575d0 2024-09-19T08:10:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dabca78060 con 0x55dabcb58b40 2024-09-19T08:10:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dabcb65190 con 0x55dabcb62370 2024-09-19T08:10:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa1765e740 10 client.0 _ll_register_callbacks cb 0x55dabc7cb8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa0effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb575d0 0x55dabcb61db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa0effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb575d0 0x55dabcb61db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43600/0 (socket says 172.21.15.98:43600) 2024-09-19T08:10:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.965+0000 7ffa0effd640 1 -- 172.21.15.98:0/2464478791 learned_addr learned my addr 172.21.15.98:0/2464478791 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/2464478791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 0x55dabcb5f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ffa0f7fe640 1 --2- 172.21.15.98:0/2464478791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 0x55dabcb5f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ffa0effd640 1 -- 172.21.15.98:0/2464478791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 msgr2=0x55dabcb5f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ffa0effd640 1 --2- 172.21.15.98:0/2464478791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 0x55dabcb5f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ffa0effd640 1 -- 172.21.15.98:0/2464478791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 msgr2=0x55dabcb5f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ffa0effd640 1 --2- 172.21.15.98:0/2464478791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 0x55dabcb5f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ffa0effd640 1 -- 172.21.15.98:0/2464478791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dabc82e3c0 con 0x55dabcb575d0 2024-09-19T08:10:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/2464478791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 0x55dabcb5f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ffa0effd640 1 --2- 172.21.15.98:0/2464478791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb575d0 0x55dabcb61db0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009900 tx=0x7ffa000319b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c0285d9e34e215 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ff9f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ff9f2ff5640 1 -- 172.21.15.98:0/2464478791 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa00035040 con 0x55dabcb575d0 2024-09-19T08:10:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ff9f2ff5640 1 -- 172.21.15.98:0/2464478791 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7ffa00004300 con 0x55dabcb575d0 2024-09-19T08:10:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ffa1765e740 1 -- 172.21.15.98:0/2464478791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa04084bd0 con 0x55dabcb575d0 2024-09-19T08:10:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ff9f2ff5640 1 -- 172.21.15.98:0/2464478791 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffa00005020 con 0x55dabcb575d0 2024-09-19T08:10:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ffa1765e740 10 client.17316 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ffa1765e740 1 -- 172.21.15.98:0/2464478791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dabcb70b00 con 0x55dabcb575d0 2024-09-19T08:10:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.966+0000 7ff9f2ff5640 1 -- 172.21.15.98:0/2464478791 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa04084bd0 con 0x55dabcb575d0 2024-09-19T08:10:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.967+0000 7ff9f2ff5640 1 -- 172.21.15.98:0/2464478791 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ffa000035f0 con 0x55dabcb575d0 2024-09-19T08:10:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.967+0000 7ffa1765e740 10 client.17316 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.967+0000 7ffa1765e740 10 client.17316 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:10:50.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.967+0000 7ffa1765e740 20 client.17316 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:50.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.967+0000 7ffa1765e740 1 --2- 172.21.15.98:0/2464478791 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55dabcb71fc0 0x55dabcb743f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.967+0000 7ffa1765e740 4 client.17316 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:50.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.967+0000 7ffa1765e740 1 -- 172.21.15.98:0/2464478791 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dabcaca860 con 0x55dabcb71fc0 2024-09-19T08:10:50.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.967+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/2464478791 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55dabcb71fc0 0x55dabcb743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.967+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/2464478791 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55dabcb71fc0 0x55dabcb743f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1f53a594dbe97ddb server_cookie=21fea5a1225de9b5 in_seq=0 out_seq=0 2024-09-19T08:10:50.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.967+0000 7ff9f2ff5640 10 client.17316 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:10:50.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.968+0000 7ff9f2ff5640 1 -- 172.21.15.98:0/2464478791 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dabcaca860 con 0x55dabcb71fc0 2024-09-19T08:10:50.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.968+0000 7ff9f2ff5640 10 client.17316 handle_command_reply: tid=0 2024-09-19T08:10:50.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.968+0000 7ffa1765e740 1 client.17316 shutdown 2024-09-19T08:10:50.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.968+0000 7ffa1765e740 1 -- 172.21.15.98:0/2464478791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb575d0 msgr2=0x55dabcb61db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:50.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.968+0000 7ffa1765e740 1 --2- 172.21.15.98:0/2464478791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb575d0 0x55dabcb61db0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009900 tx=0x7ffa000319b0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.968+0000 7ffa1765e740 1 -- 172.21.15.98:0/2464478791 shutdown_connections 2024-09-19T08:10:50.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.968+0000 7ffa1765e740 1 --2- 172.21.15.98:0/2464478791 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55dabcb71fc0 0x55dabcb743f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.968+0000 7ffa1765e740 1 --2- 172.21.15.98:0/2464478791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 0x55dabcb5f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.968+0000 7ffa1765e740 1 --2- 172.21.15.98:0/2464478791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 0x55dabcb5f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.968+0000 7ffa1765e740 1 --2- 172.21.15.98:0/2464478791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb575d0 0x55dabcb61db0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.968+0000 7ff9f2ff5640 0 client.17316 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:10:50.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.968+0000 7ffa1765e740 1 -- 172.21.15.98:0/2464478791 >> 172.21.15.98:0/2464478791 conn(0x55dabcb20310 msgr2=0x55dabcb59990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:50.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.968+0000 7ffa1765e740 1 -- 172.21.15.98:0/2464478791 shutdown_connections 2024-09-19T08:10:50.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.968+0000 7ffa1765e740 1 -- 172.21.15.98:0/2464478791 wait complete. 2024-09-19T08:10:50.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.969+0000 7ffa1765e740 20 client.17316 trim_cache size 0 max 16384 2024-09-19T08:10:50.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.981+0000 7ffa1765e740 1 Processor -- start 2024-09-19T08:10:50.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.981+0000 7ffa1765e740 1 -- start start 2024-09-19T08:10:50.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.981+0000 7ffa1765e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb575d0 0x55dabcb64760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.981+0000 7ffa1765e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb58b40 0x55dabcb53700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.981+0000 7ffa1765e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 0x55dabcb53c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.981+0000 7ffa1765e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa101ad8e0 con 0x55dabcb58b40 2024-09-19T08:10:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.981+0000 7ffa1765e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ffa101af420 con 0x55dabcb575d0 2024-09-19T08:10:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.981+0000 7ffa1765e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffa101adfb0 con 0x55dabcb62370 2024-09-19T08:10:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ffa0e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb58b40 0x55dabcb53700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ffa0e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb58b40 0x55dabcb53700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43604/0 (socket says 172.21.15.98:43604) 2024-09-19T08:10:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ffa0e7fc640 1 -- 172.21.15.98:0/1167481762 learned_addr learned my addr 172.21.15.98:0/1167481762 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ffa0effd640 1 --2- 172.21.15.98:0/1167481762 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb575d0 0x55dabcb64760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ffa0f7fe640 1 --2- 172.21.15.98:0/1167481762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 0x55dabcb53c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ffa0e7fc640 1 -- 172.21.15.98:0/1167481762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 msgr2=0x55dabcb53c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/1167481762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 0x55dabcb53c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ffa0e7fc640 1 -- 172.21.15.98:0/1167481762 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb575d0 msgr2=0x55dabcb64760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/1167481762 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb575d0 0x55dabcb64760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ffa0e7fc640 1 -- 172.21.15.98:0/1167481762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dabcb54960 con 0x55dabcb58b40 2024-09-19T08:10:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ffa0effd640 1 --2- 172.21.15.98:0/1167481762 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb575d0 0x55dabcb64760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/1167481762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb58b40 0x55dabcb53700 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7ffa040ccdf0 tx=0x7ffa0400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d04d2a47b4cbcd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ff9f4ff9640 1 -- 172.21.15.98:0/1167481762 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0409b410 con 0x55dabcb58b40 2024-09-19T08:10:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ff9f4ff9640 1 -- 172.21.15.98:0/1167481762 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa04014040 con 0x55dabcb58b40 2024-09-19T08:10:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.982+0000 7ff9f4ff9640 1 -- 172.21.15.98:0/1167481762 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0409fc80 con 0x55dabcb58b40 2024-09-19T08:10:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.983+0000 7ffa1765e740 1 -- 172.21.15.98:0/1167481762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb58b40 msgr2=0x55dabcb53700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.983+0000 7ffa1765e740 1 --2- 172.21.15.98:0/1167481762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb58b40 0x55dabcb53700 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7ffa040ccdf0 tx=0x7ffa0400bf90 comp rx=0 tx=0).stop 2024-09-19T08:10:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.983+0000 7ffa1765e740 1 -- 172.21.15.98:0/1167481762 shutdown_connections 2024-09-19T08:10:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.983+0000 7ffa1765e740 1 --2- 172.21.15.98:0/1167481762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 0x55dabcb53c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.983+0000 7ffa1765e740 1 --2- 172.21.15.98:0/1167481762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb58b40 0x55dabcb53700 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.983+0000 7ffa1765e740 1 --2- 172.21.15.98:0/1167481762 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb575d0 0x55dabcb64760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.983+0000 7ffa1765e740 1 -- 172.21.15.98:0/1167481762 >> 172.21.15.98:0/1167481762 conn(0x55dabcb20310 msgr2=0x55dabcb4ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.983+0000 7ffa1765e740 1 -- 172.21.15.98:0/1167481762 shutdown_connections 2024-09-19T08:10:50.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.983+0000 7ffa1765e740 1 -- 172.21.15.98:0/1167481762 wait complete. 2024-09-19T08:10:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.983+0000 7ffa1765e740 1 -- start start 2024-09-19T08:10:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.983+0000 7ffa1765e740 1 Processor -- start 2024-09-19T08:10:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa1765e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb575d0 0x55dabcb6ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa1765e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 0x55dabcb5bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa1765e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 0x55dabcb5c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa1765e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa101aed50 con 0x55dabcb575d0 2024-09-19T08:10:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa1765e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ffa101ae680 con 0x55dabcb58b40 2024-09-19T08:10:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa1765e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffa101ab8e0 con 0x55dabcb62370 2024-09-19T08:10:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa1765e740 10 client.0 _ll_register_callbacks cb 0x7ffa00031640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa0effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb575d0 0x55dabcb6ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa0effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb575d0 0x55dabcb6ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43614/0 (socket says 172.21.15.98:43614) 2024-09-19T08:10:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa0e7fc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 0x55dabcb5bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa0effd640 1 -- 172.21.15.98:0/1382115857 learned_addr learned my addr 172.21.15.98:0/1382115857 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:50.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa0f7fe640 1 --2- 172.21.15.98:0/1382115857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 0x55dabcb5c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa0effd640 1 -- 172.21.15.98:0/1382115857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 msgr2=0x55dabcb5c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:50.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa0effd640 1 --2- 172.21.15.98:0/1382115857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 0x55dabcb5c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa0effd640 1 -- 172.21.15.98:0/1382115857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 msgr2=0x55dabcb5bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:50.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa0effd640 1 --2- 172.21.15.98:0/1382115857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 0x55dabcb5bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:50.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa0effd640 1 -- 172.21.15.98:0/1382115857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dabcb5cb30 con 0x55dabcb575d0 2024-09-19T08:10:50.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/1382115857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 0x55dabcb5bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:10:50.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ffa0effd640 1 --2- 172.21.15.98:0/1382115857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb575d0 0x55dabcb6ac00 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7ffa00000c00 tx=0x7ffa00004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7847dfd98c89155 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:50.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ff9f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:50.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.984+0000 7ff9f2ff5640 1 -- 172.21.15.98:0/1382115857 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa00035070 con 0x55dabcb575d0 2024-09-19T08:10:50.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.985+0000 7ff9f2ff5640 1 -- 172.21.15.98:0/1382115857 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7ffa00003e30 con 0x55dabcb575d0 2024-09-19T08:10:50.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.985+0000 7ffa1765e740 1 -- 172.21.15.98:0/1382115857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa04014040 con 0x55dabcb575d0 2024-09-19T08:10:50.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.985+0000 7ff9f2ff5640 1 -- 172.21.15.98:0/1382115857 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffa00053040 con 0x55dabcb575d0 2024-09-19T08:10:50.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.985+0000 7ffa1765e740 10 client.17322 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:50.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.985+0000 7ffa1765e740 1 -- 172.21.15.98:0/1382115857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dabcbe10a0 con 0x55dabcb575d0 2024-09-19T08:10:50.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.985+0000 7ff9f2ff5640 1 -- 172.21.15.98:0/1382115857 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa04014040 con 0x55dabcb575d0 2024-09-19T08:10:50.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.985+0000 7ff9f2ff5640 1 -- 172.21.15.98:0/1382115857 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ffa00056070 con 0x55dabcb575d0 2024-09-19T08:10:50.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.985+0000 7ffa1765e740 10 client.17322 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.985+0000 7ffa1765e740 10 client.17322 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:10:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.985+0000 7ffa1765e740 20 client.17322 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.985+0000 7ffa1765e740 1 --2- 172.21.15.98:0/1382115857 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55dabcbe1870 0x55dabcbe3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.985+0000 7ffa1765e740 4 client.17322 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:10:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.985+0000 7ffa1765e740 1 -- 172.21.15.98:0/1382115857 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dabcac4d70 con 0x55dabcbe1870 2024-09-19T08:10:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.985+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/1382115857 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55dabcbe1870 0x55dabcbe3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ffa0e7fc640 1 --2- 172.21.15.98:0/1382115857 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55dabcbe1870 0x55dabcbe3c90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e8a65404dd165793 server_cookie=28a1028322d46fd7 in_seq=0 out_seq=0 2024-09-19T08:10:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ff9f2ff5640 10 client.17322 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:10:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ff9f2ff5640 1 -- 172.21.15.98:0/1382115857 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dabcac4d70 con 0x55dabcbe1870 2024-09-19T08:10:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ff9f2ff5640 10 client.17322 handle_command_reply: tid=0 2024-09-19T08:10:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ffa1765e740 1 client.17322 shutdown 2024-09-19T08:10:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ffa1765e740 1 -- 172.21.15.98:0/1382115857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb575d0 msgr2=0x55dabcb6ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ffa1765e740 1 --2- 172.21.15.98:0/1382115857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb575d0 0x55dabcb6ac00 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7ffa00000c00 tx=0x7ffa00004f20 comp rx=0 tx=0).stop 2024-09-19T08:10:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ffa1765e740 1 -- 172.21.15.98:0/1382115857 shutdown_connections 2024-09-19T08:10:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ffa1765e740 1 --2- 172.21.15.98:0/1382115857 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55dabcbe1870 0x55dabcbe3c90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ffa1765e740 1 --2- 172.21.15.98:0/1382115857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dabcb62370 0x55dabcb5c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ffa1765e740 1 --2- 172.21.15.98:0/1382115857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dabcb58b40 0x55dabcb5bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ffa1765e740 1 --2- 172.21.15.98:0/1382115857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dabcb575d0 0x55dabcb6ac00 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ffa1765e740 1 -- 172.21.15.98:0/1382115857 >> 172.21.15.98:0/1382115857 conn(0x55dabcb20310 msgr2=0x55dabcb53700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ff9f2ff5640 0 client.17322 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:10:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ffa1765e740 1 -- 172.21.15.98:0/1382115857 shutdown_connections 2024-09-19T08:10:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ffa1765e740 1 -- 172.21.15.98:0/1382115857 wait complete. 2024-09-19T08:10:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.986+0000 7ffa1765e740 20 client.17322 trim_cache size 0 max 16384 2024-09-19T08:10:51.001 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:10:51.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.987+0000 7ffa15285640 1 -- 172.21.15.98:0/515026437 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff9f80a33b0 msgr2=0x7ff9f80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:51.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.987+0000 7ffa15285640 1 --2- 172.21.15.98:0/515026437 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff9f80a33b0 0x7ff9f80a5870 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffa08005cc0 tx=0x7ffa0800ad50 comp rx=0 tx=0).stop 2024-09-19T08:10:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.987+0000 7ffa15285640 1 -- 172.21.15.98:0/515026437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa10103670 msgr2=0x7ffa10198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.987+0000 7ffa15285640 1 --2- 172.21.15.98:0/515026437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa10103670 0x7ffa10198a40 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400d950 tx=0x7ffa0400de10 comp rx=0 tx=0).stop 2024-09-19T08:10:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.987+0000 7ffa15285640 1 -- 172.21.15.98:0/515026437 shutdown_connections 2024-09-19T08:10:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.987+0000 7ffa15285640 1 --2- 172.21.15.98:0/515026437 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff9f80a33b0 0x7ff9f80a5870 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.987+0000 7ffa15285640 1 --2- 172.21.15.98:0/515026437 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffa1010c180 0x7ffa1019cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.987+0000 7ffa15285640 1 --2- 172.21.15.98:0/515026437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa10103670 0x7ffa10198a40 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.987+0000 7ffa15285640 1 --2- 172.21.15.98:0/515026437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa10102d50 0x7ffa10198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.987+0000 7ffa15285640 1 -- 172.21.15.98:0/515026437 >> 172.21.15.98:0/515026437 conn(0x7ffa100fc7d0 msgr2=0x7ffa100fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.987+0000 7ffa15285640 1 -- 172.21.15.98:0/515026437 shutdown_connections 2024-09-19T08:10:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:50.987+0000 7ffa15285640 1 -- 172.21.15.98:0/515026437 wait complete. 2024-09-19T08:10:51.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:10:51.934 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:52.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:52 smithi098 ceph-mon[31333]: pgmap v240: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-19T08:10:52.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:52 smithi162 ceph-mon[35260]: pgmap v240: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-19T08:10:52.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:52 smithi139 ceph-mon[34874]: pgmap v240: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-19T08:10:52.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.427+0000 7f871a8e9640 1 -- 172.21.15.98:0/4273525863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8714102790 msgr2=0x7f87141067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:52.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.427+0000 7f871a8e9640 1 --2- 172.21.15.98:0/4273525863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8714102790 0x7f87141067e0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009900 tx=0x7f86fc02a410 comp rx=0 tx=0).stop 2024-09-19T08:10:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.427+0000 7f871a8e9640 1 -- 172.21.15.98:0/4273525863 shutdown_connections 2024-09-19T08:10:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.427+0000 7f871a8e9640 1 --2- 172.21.15.98:0/4273525863 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8714106f10 0x7f871410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.427+0000 7f871a8e9640 1 --2- 172.21.15.98:0/4273525863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8714102790 0x7f87141067e0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.427+0000 7f871a8e9640 1 --2- 172.21.15.98:0/4273525863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8714101de0 0x7f87141021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.427+0000 7f871a8e9640 1 -- 172.21.15.98:0/4273525863 >> 172.21.15.98:0/4273525863 conn(0x7f87140fd8c0 msgr2=0x7f87140ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.427+0000 7f871a8e9640 1 -- 172.21.15.98:0/4273525863 shutdown_connections 2024-09-19T08:10:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.427+0000 7f871a8e9640 1 -- 172.21.15.98:0/4273525863 wait complete. 2024-09-19T08:10:52.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.428+0000 7f871a8e9640 1 Processor -- start 2024-09-19T08:10:52.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.428+0000 7f871a8e9640 1 -- start start 2024-09-19T08:10:52.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.428+0000 7f871a8e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8714101de0 0x7f87140744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.428+0000 7f871a8e9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8714102790 0x7f8714074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.428+0000 7f871a8e9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8714106f10 0x7f8714074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.428+0000 7f871a8e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8714111fb0 con 0x7f8714101de0 2024-09-19T08:10:52.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.428+0000 7f871a8e9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8714111e30 con 0x7f8714106f10 2024-09-19T08:10:52.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.428+0000 7f871a8e9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8714112130 con 0x7f8714102790 2024-09-19T08:10:52.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.428+0000 7f8713fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8714101de0 0x7f87140744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.428+0000 7f8713fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8714101de0 0x7f87140744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43628/0 (socket says 172.21.15.98:43628) 2024-09-19T08:10:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.428+0000 7f8713fff640 1 -- 172.21.15.98:0/235604260 learned_addr learned my addr 172.21.15.98:0/235604260 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.429+0000 7f870bfff640 1 --2- 172.21.15.98:0/235604260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8714102790 0x7f8714074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.429+0000 7f8713fff640 1 -- 172.21.15.98:0/235604260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8714102790 msgr2=0x7f8714074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.429+0000 7f8713fff640 1 --2- 172.21.15.98:0/235604260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8714102790 0x7f8714074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.429+0000 7f8713fff640 1 -- 172.21.15.98:0/235604260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8714106f10 msgr2=0x7f8714074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.429+0000 7f8713fff640 1 --2- 172.21.15.98:0/235604260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8714106f10 0x7f8714074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.429+0000 7f8713fff640 1 -- 172.21.15.98:0/235604260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87141a6950 con 0x7f8714101de0 2024-09-19T08:10:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.429+0000 7f870bfff640 1 --2- 172.21.15.98:0/235604260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8714102790 0x7f8714074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.429+0000 7f8713fff640 1 --2- 172.21.15.98:0/235604260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8714101de0 0x7f87140744f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f870400d770 tx=0x7f870400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.429+0000 7f8711ffb640 1 -- 172.21.15.98:0/235604260 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704017070 con 0x7f8714101de0 2024-09-19T08:10:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.429+0000 7f8711ffb640 1 -- 172.21.15.98:0/235604260 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8704005500 con 0x7f8714101de0 2024-09-19T08:10:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.429+0000 7f871a8e9640 1 -- 172.21.15.98:0/235604260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87141a6c40 con 0x7f8714101de0 2024-09-19T08:10:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.429+0000 7f8711ffb640 1 -- 172.21.15.98:0/235604260 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704004e00 con 0x7f8714101de0 2024-09-19T08:10:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.429+0000 7f871a8e9640 1 -- 172.21.15.98:0/235604260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87141a7130 con 0x7f8714101de0 2024-09-19T08:10:52.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.431+0000 7f8711ffb640 1 -- 172.21.15.98:0/235604260 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f8704004fa0 con 0x7f8714101de0 2024-09-19T08:10:52.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f871ccc2740 1 Processor -- start 2024-09-19T08:10:52.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f871ccc2740 1 -- start start 2024-09-19T08:10:52.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f871ccc2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d9315db40 0x556d9315c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f871ccc2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315c5d0 0x556d9315ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f871ccc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d93167370 0x556d93169760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f871ccc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556d930c22f0 con 0x556d93167370 2024-09-19T08:10:52.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f871ccc2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556d93069cf0 con 0x556d9315db40 2024-09-19T08:10:52.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f871ccc2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556d930be2b0 con 0x556d9315c5d0 2024-09-19T08:10:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f870bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d93167370 0x556d93169760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f870bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d93167370 0x556d93169760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43630/0 (socket says 172.21.15.98:43630) 2024-09-19T08:10:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f870bfff640 1 -- 172.21.15.98:0/3703868676 learned_addr learned my addr 172.21.15.98:0/3703868676 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f8711ffb640 1 --2- 172.21.15.98:0/235604260 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f87000a33b0 0x7f87000a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f8713fff640 1 --2- 172.21.15.98:0/3703868676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315c5d0 0x556d9315ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f8718e86640 1 --2- 172.21.15.98:0/3703868676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d9315db40 0x556d9315c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.432+0000 7f8718e86640 1 --2- 172.21.15.98:0/235604260 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f87000a33b0 0x7f87000a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.433+0000 7f870bfff640 1 -- 172.21.15.98:0/3703868676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315c5d0 msgr2=0x556d9315ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.433+0000 7f8711ffb640 1 -- 172.21.15.98:0/235604260 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f87040cf810 con 0x7f8714101de0 2024-09-19T08:10:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.433+0000 7f870bfff640 1 --2- 172.21.15.98:0/3703868676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315c5d0 0x556d9315ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.433+0000 7f870bfff640 1 -- 172.21.15.98:0/3703868676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d9315db40 msgr2=0x556d9315c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.433+0000 7f870bfff640 1 --2- 172.21.15.98:0/3703868676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d9315db40 0x556d9315c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.433+0000 7f870bfff640 1 -- 172.21.15.98:0/3703868676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d9304e3a0 con 0x556d93167370 2024-09-19T08:10:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.433+0000 7f8718e86640 1 --2- 172.21.15.98:0/3703868676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d9315db40 0x556d9315c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.433+0000 7f8713fff640 1 --2- 172.21.15.98:0/3703868676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315c5d0 0x556d9315ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.433+0000 7f870bfff640 1 --2- 172.21.15.98:0/3703868676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d93167370 0x556d93169760 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f86fc005c40 tx=0x7f86fc032cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e334d6a1b9d24a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.433+0000 7f8718e86640 1 --2- 172.21.15.98:0/235604260 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f87000a33b0 0x7f87000a5870 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f870c008fd0 tx=0x7f870c006ce0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.433+0000 7f870a7fc640 1 -- 172.21.15.98:0/3703868676 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc03f070 con 0x556d93167370 2024-09-19T08:10:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.433+0000 7f870a7fc640 1 -- 172.21.15.98:0/3703868676 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86fc0317b0 con 0x556d93167370 2024-09-19T08:10:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.433+0000 7f870a7fc640 1 -- 172.21.15.98:0/3703868676 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc035070 con 0x556d93167370 2024-09-19T08:10:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.434+0000 7f871ccc2740 1 -- 172.21.15.98:0/3703868676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d93167370 msgr2=0x556d93169760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.434+0000 7f871ccc2740 1 --2- 172.21.15.98:0/3703868676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d93167370 0x556d93169760 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f86fc005c40 tx=0x7f86fc032cb0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.434+0000 7f871ccc2740 1 -- 172.21.15.98:0/3703868676 shutdown_connections 2024-09-19T08:10:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.434+0000 7f871ccc2740 1 --2- 172.21.15.98:0/3703868676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d93167370 0x556d93169760 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.434+0000 7f871ccc2740 1 --2- 172.21.15.98:0/3703868676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315c5d0 0x556d9315ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.434+0000 7f871ccc2740 1 --2- 172.21.15.98:0/3703868676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d9315db40 0x556d9315c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.434+0000 7f871ccc2740 1 -- 172.21.15.98:0/3703868676 >> 172.21.15.98:0/3703868676 conn(0x556d93125310 msgr2=0x556d93158c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.434+0000 7f871ccc2740 1 -- 172.21.15.98:0/3703868676 shutdown_connections 2024-09-19T08:10:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.434+0000 7f871ccc2740 1 -- 172.21.15.98:0/3703868676 wait complete. 2024-09-19T08:10:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.434+0000 7f871ccc2740 1 -- start start 2024-09-19T08:10:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.434+0000 7f871ccc2740 1 Processor -- start 2024-09-19T08:10:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f871ccc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d9315c5d0 0x556d93166cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f871ccc2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 0x556d93163f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f871ccc2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d93167370 0x556d931644d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f871ccc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556d9316a010 con 0x556d9315c5d0 2024-09-19T08:10:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f871ccc2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556d9307d060 con 0x556d93167370 2024-09-19T08:10:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f871ccc2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556d9316a190 con 0x556d9315db40 2024-09-19T08:10:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f871ccc2740 10 client.0 _ll_register_callbacks cb 0x556d92dd08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f8713fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d9315c5d0 0x556d93166cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f8713fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d9315c5d0 0x556d93166cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43638/0 (socket says 172.21.15.98:43638) 2024-09-19T08:10:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f8713fff640 1 -- 172.21.15.98:0/1833491657 learned_addr learned my addr 172.21.15.98:0/1833491657 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f870bfff640 1 --2- 172.21.15.98:0/1833491657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 0x556d93163f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f8718e86640 1 --2- 172.21.15.98:0/1833491657 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d93167370 0x556d931644d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f8713fff640 1 -- 172.21.15.98:0/1833491657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 msgr2=0x556d93163f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f8713fff640 1 --2- 172.21.15.98:0/1833491657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 0x556d93163f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f8713fff640 1 -- 172.21.15.98:0/1833491657 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d93167370 msgr2=0x556d931644d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f8713fff640 1 --2- 172.21.15.98:0/1833491657 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d93167370 0x556d931644d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f8713fff640 1 -- 172.21.15.98:0/1833491657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d92e333c0 con 0x556d9315c5d0 2024-09-19T08:10:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f8713fff640 1 --2- 172.21.15.98:0/1833491657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d9315c5d0 0x556d93166cc0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f870409ba80 tx=0x7f870406b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50b780e93c3dcc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f86f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f86f37fe640 1 -- 172.21.15.98:0/1833491657 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704020040 con 0x556d9315c5d0 2024-09-19T08:10:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f86f37fe640 1 -- 172.21.15.98:0/1833491657 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f87040cd1e0 con 0x556d9315c5d0 2024-09-19T08:10:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.435+0000 7f871ccc2740 1 -- 172.21.15.98:0/1833491657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86fc0317b0 con 0x556d9315c5d0 2024-09-19T08:10:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.436+0000 7f86f37fe640 1 -- 172.21.15.98:0/1833491657 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8704066940 con 0x556d9315c5d0 2024-09-19T08:10:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.436+0000 7f871ccc2740 10 client.17334 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.436+0000 7f871ccc2740 1 -- 172.21.15.98:0/1833491657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556d93175aa0 con 0x556d9315c5d0 2024-09-19T08:10:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.436+0000 7f86f37fe640 1 -- 172.21.15.98:0/1833491657 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86fc0317b0 con 0x556d9315c5d0 2024-09-19T08:10:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.436+0000 7f86f37fe640 1 -- 172.21.15.98:0/1833491657 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8704098160 con 0x556d9315c5d0 2024-09-19T08:10:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.436+0000 7f871ccc2740 10 client.17334 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.436+0000 7f871ccc2740 10 client.17334 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:10:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.436+0000 7f871ccc2740 20 client.17334 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.436+0000 7f871ccc2740 1 --2- 172.21.15.98:0/1833491657 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x556d93176f60 0x556d93179350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.436+0000 7f871ccc2740 4 client.17334 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.436+0000 7f871ccc2740 1 -- 172.21.15.98:0/1833491657 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d930cf860 con 0x556d93176f60 2024-09-19T08:10:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.436+0000 7f870bfff640 1 --2- 172.21.15.98:0/1833491657 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x556d93176f60 0x556d93179350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.437+0000 7f86f37fe640 10 client.17334 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:10:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.437+0000 7f870bfff640 1 --2- 172.21.15.98:0/1833491657 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x556d93176f60 0x556d93179350 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=67adda7fef125b98 server_cookie=88ebed534ea66afb in_seq=0 out_seq=0 2024-09-19T08:10:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.437+0000 7f86f37fe640 1 -- 172.21.15.98:0/1833491657 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556d930cf860 con 0x556d93176f60 2024-09-19T08:10:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.437+0000 7f86f37fe640 10 client.17334 handle_command_reply: tid=0 2024-09-19T08:10:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.437+0000 7f871ccc2740 1 client.17334 shutdown 2024-09-19T08:10:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.438+0000 7f871ccc2740 1 -- 172.21.15.98:0/1833491657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d9315c5d0 msgr2=0x556d93166cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.438+0000 7f871ccc2740 1 --2- 172.21.15.98:0/1833491657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d9315c5d0 0x556d93166cc0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f870409ba80 tx=0x7f870406b710 comp rx=0 tx=0).stop 2024-09-19T08:10:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.438+0000 7f871ccc2740 1 -- 172.21.15.98:0/1833491657 shutdown_connections 2024-09-19T08:10:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.438+0000 7f871ccc2740 1 --2- 172.21.15.98:0/1833491657 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x556d93176f60 0x556d93179350 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.438+0000 7f871ccc2740 1 --2- 172.21.15.98:0/1833491657 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d93167370 0x556d931644d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.438+0000 7f871ccc2740 1 --2- 172.21.15.98:0/1833491657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 0x556d93163f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.438+0000 7f871ccc2740 1 --2- 172.21.15.98:0/1833491657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d9315c5d0 0x556d93166cc0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.438+0000 7f871ccc2740 1 -- 172.21.15.98:0/1833491657 >> 172.21.15.98:0/1833491657 conn(0x556d93125310 msgr2=0x556d9315e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:52.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.438+0000 7f86f37fe640 0 client.17334 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:10:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.438+0000 7f871ccc2740 1 -- 172.21.15.98:0/1833491657 shutdown_connections 2024-09-19T08:10:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.438+0000 7f871ccc2740 1 -- 172.21.15.98:0/1833491657 wait complete. 2024-09-19T08:10:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.438+0000 7f871ccc2740 20 client.17334 trim_cache size 0 max 16384 2024-09-19T08:10:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f871ccc2740 1 Processor -- start 2024-09-19T08:10:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f871ccc2740 1 -- start start 2024-09-19T08:10:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f871ccc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d9315c5d0 0x556d931732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f871ccc2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 0x556d9317eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f871ccc2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d93167370 0x556d9317f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f871ccc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87141afa60 con 0x556d9315c5d0 2024-09-19T08:10:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f871ccc2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f87141b15a0 con 0x556d93167370 2024-09-19T08:10:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f871ccc2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f87141b0130 con 0x556d9315db40 2024-09-19T08:10:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f8713fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d9315c5d0 0x556d931732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f8713fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d9315c5d0 0x556d931732e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43640/0 (socket says 172.21.15.98:43640) 2024-09-19T08:10:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f8713fff640 1 -- 172.21.15.98:0/3593464130 learned_addr learned my addr 172.21.15.98:0/3593464130 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f870bfff640 1 --2- 172.21.15.98:0/3593464130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 0x556d9317eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f8713fff640 1 -- 172.21.15.98:0/3593464130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 msgr2=0x556d9317eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f8713fff640 1 --2- 172.21.15.98:0/3593464130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 0x556d9317eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f8713fff640 1 -- 172.21.15.98:0/3593464130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d93167370 msgr2=0x556d9317f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f8713fff640 1 --2- 172.21.15.98:0/3593464130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d93167370 0x556d9317f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f8713fff640 1 -- 172.21.15.98:0/3593464130 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d93159960 con 0x556d9315c5d0 2024-09-19T08:10:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f8713fff640 1 --2- 172.21.15.98:0/3593464130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d9315c5d0 0x556d931732e0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f870400d740 tx=0x7f8704009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1399607126da597 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f8709ffb640 1 -- 172.21.15.98:0/3593464130 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704020030 con 0x556d9315c5d0 2024-09-19T08:10:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f8709ffb640 1 -- 172.21.15.98:0/3593464130 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f870409c1f0 con 0x556d9315c5d0 2024-09-19T08:10:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.451+0000 7f8709ffb640 1 -- 172.21.15.98:0/3593464130 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87040cd340 con 0x556d9315c5d0 2024-09-19T08:10:52.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.452+0000 7f871ccc2740 1 -- 172.21.15.98:0/3593464130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d9315c5d0 msgr2=0x556d931732e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:52.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.452+0000 7f871ccc2740 1 --2- 172.21.15.98:0/3593464130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d9315c5d0 0x556d931732e0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f870400d740 tx=0x7f8704009c30 comp rx=0 tx=0).stop 2024-09-19T08:10:52.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.452+0000 7f871ccc2740 1 -- 172.21.15.98:0/3593464130 shutdown_connections 2024-09-19T08:10:52.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.452+0000 7f871ccc2740 1 --2- 172.21.15.98:0/3593464130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d93167370 0x556d9317f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.452+0000 7f871ccc2740 1 --2- 172.21.15.98:0/3593464130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 0x556d9317eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.452+0000 7f871ccc2740 1 --2- 172.21.15.98:0/3593464130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d9315c5d0 0x556d931732e0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.452+0000 7f871ccc2740 1 -- 172.21.15.98:0/3593464130 >> 172.21.15.98:0/3593464130 conn(0x556d93125310 msgr2=0x556d93162840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:52.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.452+0000 7f871ccc2740 1 -- 172.21.15.98:0/3593464130 shutdown_connections 2024-09-19T08:10:52.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.452+0000 7f871ccc2740 1 -- 172.21.15.98:0/3593464130 wait complete. 2024-09-19T08:10:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.452+0000 7f871ccc2740 1 -- start start 2024-09-19T08:10:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f871ccc2740 1 Processor -- start 2024-09-19T08:10:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f871ccc2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d9315c5d0 0x556d9316fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f871ccc2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 0x556d9316ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f871ccc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d93167370 0x556d9316b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f871ccc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87141b0ed0 con 0x556d93167370 2024-09-19T08:10:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f871ccc2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f87141b0800 con 0x556d9315c5d0 2024-09-19T08:10:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f871ccc2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f87141ada60 con 0x556d9315db40 2024-09-19T08:10:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f871ccc2740 10 client.0 _ll_register_callbacks cb 0x7f870400de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f8718e86640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d93167370 0x556d9316b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f8718e86640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d93167370 0x556d9316b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43648/0 (socket says 172.21.15.98:43648) 2024-09-19T08:10:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f8718e86640 1 -- 172.21.15.98:0/357492161 learned_addr learned my addr 172.21.15.98:0/357492161 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f870bfff640 1 --2- 172.21.15.98:0/357492161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 0x556d9316ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f8713fff640 1 --2- 172.21.15.98:0/357492161 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d9315c5d0 0x556d9316fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f870bfff640 1 -- 172.21.15.98:0/357492161 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d9315c5d0 msgr2=0x556d9316fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f870bfff640 1 --2- 172.21.15.98:0/357492161 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d9315c5d0 0x556d9316fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f870bfff640 1 -- 172.21.15.98:0/357492161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d93167370 msgr2=0x556d9316b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f870bfff640 1 --2- 172.21.15.98:0/357492161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d93167370 0x556d9316b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f870bfff640 1 -- 172.21.15.98:0/357492161 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d9316b8e0 con 0x556d9315db40 2024-09-19T08:10:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.453+0000 7f8718e86640 1 --2- 172.21.15.98:0/357492161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d93167370 0x556d9316b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:10:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.454+0000 7f870bfff640 1 --2- 172.21.15.98:0/357492161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 0x556d9316ad30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0096c0 tx=0x7f86fc0315b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87a86c1866021f47 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.454+0000 7f86f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-09-19T08:10:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.454+0000 7f86f37fe640 1 -- 172.21.15.98:0/357492161 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc03f070 con 0x556d9315db40 2024-09-19T08:10:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.454+0000 7f871ccc2740 1 -- 172.21.15.98:0/357492161 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f870409c1f0 con 0x556d9315db40 2024-09-19T08:10:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.454+0000 7f86f37fe640 1 -- 172.21.15.98:0/357492161 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f86fc005540 con 0x556d9315db40 2024-09-19T08:10:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.454+0000 7f86f37fe640 1 -- 172.21.15.98:0/357492161 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f86fc05b050 con 0x556d9315db40 2024-09-19T08:10:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.454+0000 7f871ccc2740 10 client.26296 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.454+0000 7f871ccc2740 1 -- 172.21.15.98:0/357492161 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556d931e1960 con 0x556d9315db40 2024-09-19T08:10:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.454+0000 7f86f37fe640 1 -- 172.21.15.98:0/357492161 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f870409c1f0 con 0x556d9315db40 2024-09-19T08:10:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.454+0000 7f86f37fe640 1 -- 172.21.15.98:0/357492161 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f86fc003b10 con 0x556d9315db40 2024-09-19T08:10:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.454+0000 7f871ccc2740 10 client.26296 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.454+0000 7f871ccc2740 10 client.26296 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:10:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.454+0000 7f871ccc2740 20 client.26296 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.455+0000 7f871ccc2740 1 --2- 172.21.15.98:0/357492161 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x556d931e1cc0 0x556d931e40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.455+0000 7f871ccc2740 4 client.26296 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:10:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.455+0000 7f871ccc2740 1 -- 172.21.15.98:0/357492161 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556d930c9d70 con 0x556d931e1cc0 2024-09-19T08:10:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.455+0000 7f8713fff640 1 --2- 172.21.15.98:0/357492161 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x556d931e1cc0 0x556d931e40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.455+0000 7f86f37fe640 10 client.26296 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:10:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.455+0000 7f8713fff640 1 --2- 172.21.15.98:0/357492161 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x556d931e1cc0 0x556d931e40b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1a51d1aef9f3863c server_cookie=def860198cfc8ab8 in_seq=0 out_seq=0 2024-09-19T08:10:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.455+0000 7f86f37fe640 1 -- 172.21.15.98:0/357492161 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556d930c9d70 con 0x556d931e1cc0 2024-09-19T08:10:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.455+0000 7f86f37fe640 10 client.26296 handle_command_reply: tid=0 2024-09-19T08:10:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.455+0000 7f871ccc2740 1 client.26296 shutdown 2024-09-19T08:10:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871ccc2740 1 -- 172.21.15.98:0/357492161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 msgr2=0x556d9316ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871ccc2740 1 --2- 172.21.15.98:0/357492161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 0x556d9316ad30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0096c0 tx=0x7f86fc0315b0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871ccc2740 1 -- 172.21.15.98:0/357492161 shutdown_connections 2024-09-19T08:10:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871ccc2740 1 --2- 172.21.15.98:0/357492161 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x556d931e1cc0 0x556d931e40b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871ccc2740 1 --2- 172.21.15.98:0/357492161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d93167370 0x556d9316b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871ccc2740 1 --2- 172.21.15.98:0/357492161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d9315db40 0x556d9316ad30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f86f37fe640 0 client.26296 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:10:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871ccc2740 1 --2- 172.21.15.98:0/357492161 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d9315c5d0 0x556d9316fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871ccc2740 1 -- 172.21.15.98:0/357492161 >> 172.21.15.98:0/357492161 conn(0x556d93125310 msgr2=0x556d931574e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871ccc2740 1 -- 172.21.15.98:0/357492161 shutdown_connections 2024-09-19T08:10:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871ccc2740 1 -- 172.21.15.98:0/357492161 wait complete. 2024-09-19T08:10:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871ccc2740 20 client.26296 trim_cache size 0 max 16384 2024-09-19T08:10:52.463 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:10:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871a8e9640 1 -- 172.21.15.98:0/235604260 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f87000a33b0 msgr2=0x7f87000a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871a8e9640 1 --2- 172.21.15.98:0/235604260 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f87000a33b0 0x7f87000a5870 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f870c008fd0 tx=0x7f870c006ce0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871a8e9640 1 -- 172.21.15.98:0/235604260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8714101de0 msgr2=0x7f87140744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871a8e9640 1 --2- 172.21.15.98:0/235604260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8714101de0 0x7f87140744f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f870400d770 tx=0x7f870400dc30 comp rx=0 tx=0).stop 2024-09-19T08:10:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871a8e9640 1 -- 172.21.15.98:0/235604260 shutdown_connections 2024-09-19T08:10:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871a8e9640 1 --2- 172.21.15.98:0/235604260 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f87000a33b0 0x7f87000a5870 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871a8e9640 1 --2- 172.21.15.98:0/235604260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8714106f10 0x7f8714074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871a8e9640 1 --2- 172.21.15.98:0/235604260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8714102790 0x7f8714074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871a8e9640 1 --2- 172.21.15.98:0/235604260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8714101de0 0x7f87140744f0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871a8e9640 1 -- 172.21.15.98:0/235604260 >> 172.21.15.98:0/235604260 conn(0x7f87140fd8c0 msgr2=0x7f87140fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871a8e9640 1 -- 172.21.15.98:0/235604260 shutdown_connections 2024-09-19T08:10:52.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:52.456+0000 7f871a8e9640 1 -- 172.21.15.98:0/235604260 wait complete. 2024-09-19T08:10:53.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:10:53.409 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:53.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.911+0000 7fbc3e0a2640 1 -- 172.21.15.98:0/662372407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc38108300 msgr2=0x7fbc3810ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:53.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.911+0000 7fbc3e0a2640 1 --2- 172.21.15.98:0/662372407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc38108300 0x7fbc3810ec70 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009cd0 tx=0x7fbc3002a8f0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.911+0000 7fbc3e0a2640 1 -- 172.21.15.98:0/662372407 shutdown_connections 2024-09-19T08:10:53.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.911+0000 7fbc3e0a2640 1 --2- 172.21.15.98:0/662372407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc38108300 0x7fbc3810ec70 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.911+0000 7fbc3e0a2640 1 --2- 172.21.15.98:0/662372407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc38103780 0x7fbc38107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.911+0000 7fbc3e0a2640 1 --2- 172.21.15.98:0/662372407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc38102dd0 0x7fbc381031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.911+0000 7fbc3e0a2640 1 -- 172.21.15.98:0/662372407 >> 172.21.15.98:0/662372407 conn(0x7fbc380fc7d0 msgr2=0x7fbc380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.911+0000 7fbc3e0a2640 1 -- 172.21.15.98:0/662372407 shutdown_connections 2024-09-19T08:10:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.911+0000 7fbc3e0a2640 1 -- 172.21.15.98:0/662372407 wait complete. 2024-09-19T08:10:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.912+0000 7fbc3e0a2640 1 Processor -- start 2024-09-19T08:10:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.912+0000 7fbc3e0a2640 1 -- start start 2024-09-19T08:10:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc3e0a2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc38102dd0 0x7fbc381985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc3e0a2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc38103780 0x7fbc38198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc3e0a2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc38108300 0x7fbc3819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc3e0a2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc381120f0 con 0x7fbc38108300 2024-09-19T08:10:53.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc3e0a2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbc38111f70 con 0x7fbc38102dd0 2024-09-19T08:10:53.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc3e0a2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbc38112270 con 0x7fbc38103780 2024-09-19T08:10:53.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc37fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc38108300 0x7fbc3819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc37fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc38108300 0x7fbc3819ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43680/0 (socket says 172.21.15.98:43680) 2024-09-19T08:10:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc37fff640 1 -- 172.21.15.98:0/3480069929 learned_addr learned my addr 172.21.15.98:0/3480069929 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc377fe640 1 --2- 172.21.15.98:0/3480069929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc38102dd0 0x7fbc381985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc2effd640 1 --2- 172.21.15.98:0/3480069929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc38103780 0x7fbc38198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc37fff640 1 -- 172.21.15.98:0/3480069929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc38103780 msgr2=0x7fbc38198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc37fff640 1 --2- 172.21.15.98:0/3480069929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc38103780 0x7fbc38198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc37fff640 1 -- 172.21.15.98:0/3480069929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc38102dd0 msgr2=0x7fbc381985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc37fff640 1 --2- 172.21.15.98:0/3480069929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc38102dd0 0x7fbc381985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc37fff640 1 -- 172.21.15.98:0/3480069929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3819d570 con 0x7fbc38108300 2024-09-19T08:10:53.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc37fff640 1 --2- 172.21.15.98:0/3480069929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc38108300 0x7fbc3819ce90 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009ca0 tx=0x7fbc300039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:53.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.913+0000 7fbc357fa640 1 -- 172.21.15.98:0/3480069929 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc30004040 con 0x7fbc38108300 2024-09-19T08:10:53.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.914+0000 7fbc357fa640 1 -- 172.21.15.98:0/3480069929 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc30040070 con 0x7fbc38108300 2024-09-19T08:10:53.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.914+0000 7fbc3e0a2640 1 -- 172.21.15.98:0/3480069929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc3819d800 con 0x7fbc38108300 2024-09-19T08:10:53.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.914+0000 7fbc357fa640 1 -- 172.21.15.98:0/3480069929 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc30031d40 con 0x7fbc38108300 2024-09-19T08:10:53.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.914+0000 7fbc3e0a2640 1 -- 172.21.15.98:0/3480069929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc381a5140 con 0x7fbc38108300 2024-09-19T08:10:53.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.916+0000 7fbc357fa640 1 -- 172.21.15.98:0/3480069929 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fbc30031440 con 0x7fbc38108300 2024-09-19T08:10:53.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.916+0000 7fbc4047b740 1 Processor -- start 2024-09-19T08:10:53.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.916+0000 7fbc4047b740 1 -- start start 2024-09-19T08:10:53.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.916+0000 7fbc4047b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafdb40 0x55d04cafc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.916+0000 7fbc4047b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cafc5d0 0x55d04cafca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.916+0000 7fbc4047b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cb07370 0x55d04cb09760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.916+0000 7fbc4047b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d04ca622f0 con 0x55d04cafdb40 2024-09-19T08:10:53.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.916+0000 7fbc4047b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d04ca09cf0 con 0x55d04cafc5d0 2024-09-19T08:10:53.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.916+0000 7fbc4047b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d04ca5e2b0 con 0x55d04cb07370 2024-09-19T08:10:53.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc357fa640 1 --2- 172.21.15.98:0/3480069929 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbc240a3430 0x7fbc240a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc2effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafdb40 0x55d04cafc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc2effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafdb40 0x55d04cafc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43686/0 (socket says 172.21.15.98:43686) 2024-09-19T08:10:53.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc2effd640 1 -- 172.21.15.98:0/1490730917 learned_addr learned my addr 172.21.15.98:0/1490730917 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:53.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc377fe640 1 --2- 172.21.15.98:0/1490730917 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cafc5d0 0x55d04cafca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc357fa640 1 -- 172.21.15.98:0/3480069929 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7fbc300f0f80 con 0x7fbc38108300 2024-09-19T08:10:53.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc37fff640 1 --2- 172.21.15.98:0/3480069929 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbc240a3430 0x7fbc240a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc2effd640 1 --2- 172.21.15.98:0/1490730917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cb07370 0x55d04cb09760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc377fe640 1 -- 172.21.15.98:0/1490730917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cb07370 msgr2=0x55d04cb09760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc377fe640 1 --2- 172.21.15.98:0/1490730917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cb07370 0x55d04cb09760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc377fe640 1 -- 172.21.15.98:0/1490730917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafdb40 msgr2=0x55d04cafc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc377fe640 1 --2- 172.21.15.98:0/1490730917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafdb40 0x55d04cafc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc377fe640 1 -- 172.21.15.98:0/1490730917 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d04c9ee3a0 con 0x55d04cafc5d0 2024-09-19T08:10:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc2effd640 1 --2- 172.21.15.98:0/1490730917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafdb40 0x55d04cafc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:10:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc37fff640 1 --2- 172.21.15.98:0/3480069929 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbc240a3430 0x7fbc240a58f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbc300c5ab0 tx=0x7fbc3002a8f0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.917+0000 7fbc377fe640 1 --2- 172.21.15.98:0/1490730917 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cafc5d0 0x55d04cafca10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800ab40 tx=0x7fbc2800f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa901c87ac45aa77 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.918+0000 7fbc2dffb640 1 -- 172.21.15.98:0/1490730917 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc28004020 con 0x55d04cafc5d0 2024-09-19T08:10:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.918+0000 7fbc2dffb640 1 -- 172.21.15.98:0/1490730917 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc280041c0 con 0x55d04cafc5d0 2024-09-19T08:10:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.918+0000 7fbc2dffb640 1 -- 172.21.15.98:0/1490730917 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc280187c0 con 0x55d04cafc5d0 2024-09-19T08:10:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.918+0000 7fbc4047b740 1 -- 172.21.15.98:0/1490730917 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cafc5d0 msgr2=0x55d04cafca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.918+0000 7fbc4047b740 1 --2- 172.21.15.98:0/1490730917 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cafc5d0 0x55d04cafca10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800ab40 tx=0x7fbc2800f3c0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.918+0000 7fbc4047b740 1 -- 172.21.15.98:0/1490730917 shutdown_connections 2024-09-19T08:10:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.918+0000 7fbc4047b740 1 --2- 172.21.15.98:0/1490730917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cb07370 0x55d04cb09760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.918+0000 7fbc4047b740 1 --2- 172.21.15.98:0/1490730917 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cafc5d0 0x55d04cafca10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.918+0000 7fbc4047b740 1 --2- 172.21.15.98:0/1490730917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafdb40 0x55d04cafc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.918+0000 7fbc4047b740 1 -- 172.21.15.98:0/1490730917 >> 172.21.15.98:0/1490730917 conn(0x55d04cac5310 msgr2=0x55d04caf8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.918+0000 7fbc4047b740 1 -- 172.21.15.98:0/1490730917 shutdown_connections 2024-09-19T08:10:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.918+0000 7fbc4047b740 1 -- 172.21.15.98:0/1490730917 wait complete. 2024-09-19T08:10:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.919+0000 7fbc4047b740 1 -- start start 2024-09-19T08:10:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.919+0000 7fbc4047b740 1 Processor -- start 2024-09-19T08:10:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.919+0000 7fbc4047b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafc5d0 0x55d04cb06d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.919+0000 7fbc4047b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafdb40 0x55d04cb03fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.919+0000 7fbc4047b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cb07370 0x55d04cb04570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.919+0000 7fbc4047b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d04cb0a070 con 0x55d04cafdb40 2024-09-19T08:10:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.919+0000 7fbc4047b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d04ca1d060 con 0x55d04cb07370 2024-09-19T08:10:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.919+0000 7fbc4047b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d04cb0a1f0 con 0x55d04cafc5d0 2024-09-19T08:10:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.919+0000 7fbc377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafdb40 0x55d04cb03fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.919+0000 7fbc4047b740 10 client.0 _ll_register_callbacks cb 0x55d04c7708c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.919+0000 7fbc377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafdb40 0x55d04cb03fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43690/0 (socket says 172.21.15.98:43690) 2024-09-19T08:10:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.919+0000 7fbc377fe640 1 -- 172.21.15.98:0/49420370 learned_addr learned my addr 172.21.15.98:0/49420370 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.919+0000 7fbc2effd640 1 --2- 172.21.15.98:0/49420370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cb07370 0x55d04cb04570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc2effd640 1 --2- 172.21.15.98:0/49420370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafc5d0 0x55d04cb06d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc377fe640 1 -- 172.21.15.98:0/49420370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafc5d0 msgr2=0x55d04cb06d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc377fe640 1 --2- 172.21.15.98:0/49420370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafc5d0 0x55d04cb06d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc377fe640 1 -- 172.21.15.98:0/49420370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cb07370 msgr2=0x55d04cb04570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc377fe640 1 --2- 172.21.15.98:0/49420370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cb07370 0x55d04cb04570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc377fe640 1 -- 172.21.15.98:0/49420370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d04c7d33c0 con 0x55d04cafdb40 2024-09-19T08:10:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc2effd640 1 --2- 172.21.15.98:0/49420370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafc5d0 0x55d04cb06d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc377fe640 1 --2- 172.21.15.98:0/49420370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafdb40 0x55d04cb03fd0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fbc2801a040 tx=0x7fbc28003dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5514beceb5a472b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc16ffd640 1 -- 172.21.15.98:0/49420370 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc28027020 con 0x55d04cafdb40 2024-09-19T08:10:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc16ffd640 1 -- 172.21.15.98:0/49420370 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7fbc28003f70 con 0x55d04cafdb40 2024-09-19T08:10:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc4047b740 1 -- 172.21.15.98:0/49420370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc280041c0 con 0x55d04cafdb40 2024-09-19T08:10:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc16ffd640 1 -- 172.21.15.98:0/49420370 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbc28032050 con 0x55d04cafdb40 2024-09-19T08:10:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc4047b740 10 client.17352 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc4047b740 1 -- 172.21.15.98:0/49420370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d04cb15ac0 con 0x55d04cafdb40 2024-09-19T08:10:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.920+0000 7fbc16ffd640 1 -- 172.21.15.98:0/49420370 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc280041c0 con 0x55d04cafdb40 2024-09-19T08:10:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.921+0000 7fbc16ffd640 1 -- 172.21.15.98:0/49420370 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fbc28004a10 con 0x55d04cafdb40 2024-09-19T08:10:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.921+0000 7fbc4047b740 10 client.17352 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.921+0000 7fbc4047b740 10 client.17352 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.921+0000 7fbc4047b740 20 client.17352 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.921+0000 7fbc4047b740 1 --2- 172.21.15.98:0/49420370 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d04cb16f20 0x55d04cb19310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.921+0000 7fbc4047b740 4 client.17352 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.921+0000 7fbc4047b740 1 -- 172.21.15.98:0/49420370 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d04ca6f860 con 0x55d04cb16f20 2024-09-19T08:10:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.921+0000 7fbc2effd640 1 --2- 172.21.15.98:0/49420370 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d04cb16f20 0x55d04cb19310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.921+0000 7fbc2effd640 1 --2- 172.21.15.98:0/49420370 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d04cb16f20 0x55d04cb19310 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93c1004978f214d9 server_cookie=1b104c00a99ad8b4 in_seq=0 out_seq=0 2024-09-19T08:10:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.921+0000 7fbc16ffd640 10 client.17352 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.922+0000 7fbc16ffd640 1 -- 172.21.15.98:0/49420370 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d04ca6f860 con 0x55d04cb16f20 2024-09-19T08:10:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.922+0000 7fbc16ffd640 10 client.17352 handle_command_reply: tid=0 2024-09-19T08:10:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.922+0000 7fbc4047b740 1 client.17352 shutdown 2024-09-19T08:10:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.922+0000 7fbc4047b740 1 -- 172.21.15.98:0/49420370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafdb40 msgr2=0x55d04cb03fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.922+0000 7fbc4047b740 1 --2- 172.21.15.98:0/49420370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafdb40 0x55d04cb03fd0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fbc2801a040 tx=0x7fbc28003dd0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.922+0000 7fbc4047b740 1 -- 172.21.15.98:0/49420370 shutdown_connections 2024-09-19T08:10:53.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.922+0000 7fbc4047b740 1 --2- 172.21.15.98:0/49420370 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d04cb16f20 0x55d04cb19310 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.922+0000 7fbc4047b740 1 --2- 172.21.15.98:0/49420370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cb07370 0x55d04cb04570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.922+0000 7fbc4047b740 1 --2- 172.21.15.98:0/49420370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafdb40 0x55d04cb03fd0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.922+0000 7fbc4047b740 1 --2- 172.21.15.98:0/49420370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafc5d0 0x55d04cb06d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.922+0000 7fbc4047b740 1 -- 172.21.15.98:0/49420370 >> 172.21.15.98:0/49420370 conn(0x55d04cac5310 msgr2=0x55d04cafe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.922+0000 7fbc16ffd640 0 client.17352 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.923+0000 7fbc4047b740 1 -- 172.21.15.98:0/49420370 shutdown_connections 2024-09-19T08:10:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.923+0000 7fbc4047b740 1 -- 172.21.15.98:0/49420370 wait complete. 2024-09-19T08:10:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.923+0000 7fbc4047b740 20 client.17352 trim_cache size 0 max 16384 2024-09-19T08:10:53.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.935+0000 7fbc4047b740 1 Processor -- start 2024-09-19T08:10:53.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.935+0000 7fbc4047b740 1 -- start start 2024-09-19T08:10:53.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.935+0000 7fbc4047b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafc5d0 0x55d04cb13310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.935+0000 7fbc4047b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafdb40 0x55d04cb18f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.935+0000 7fbc4047b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cb07370 0x55d04cb0f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.935+0000 7fbc4047b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc381ad820 con 0x55d04cafc5d0 2024-09-19T08:10:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.935+0000 7fbc4047b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbc381af360 con 0x55d04cb07370 2024-09-19T08:10:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.935+0000 7fbc4047b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbc381adef0 con 0x55d04cafdb40 2024-09-19T08:10:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.936+0000 7fbc2effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafc5d0 0x55d04cb13310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.936+0000 7fbc2effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafc5d0 0x55d04cb13310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43698/0 (socket says 172.21.15.98:43698) 2024-09-19T08:10:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.936+0000 7fbc2effd640 1 -- 172.21.15.98:0/1662106316 learned_addr learned my addr 172.21.15.98:0/1662106316 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.936+0000 7fbc377fe640 1 --2- 172.21.15.98:0/1662106316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafdb40 0x55d04cb18f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.936+0000 7fbc2effd640 1 -- 172.21.15.98:0/1662106316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafdb40 msgr2=0x55d04cb18f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.936+0000 7fbc2effd640 1 --2- 172.21.15.98:0/1662106316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafdb40 0x55d04cb18f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.936+0000 7fbc2effd640 1 -- 172.21.15.98:0/1662106316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cb07370 msgr2=0x55d04cb0f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.936+0000 7fbc2effd640 1 --2- 172.21.15.98:0/1662106316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cb07370 0x55d04cb0f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.936+0000 7fbc2effd640 1 -- 172.21.15.98:0/1662106316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d04caf7310 con 0x55d04cafc5d0 2024-09-19T08:10:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.936+0000 7fbc2effd640 1 --2- 172.21.15.98:0/1662106316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafc5d0 0x55d04cb13310 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x55d04cafd070 tx=0x7fbc2000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7d0dab30278395 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.936+0000 7fbc2d7fa640 1 -- 172.21.15.98:0/1662106316 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20000e60 con 0x55d04cafc5d0 2024-09-19T08:10:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.936+0000 7fbc2d7fa640 1 -- 172.21.15.98:0/1662106316 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc20010040 con 0x55d04cafc5d0 2024-09-19T08:10:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.936+0000 7fbc2d7fa640 1 -- 172.21.15.98:0/1662106316 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc200184e0 con 0x55d04cafc5d0 2024-09-19T08:10:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.937+0000 7fbc4047b740 1 -- 172.21.15.98:0/1662106316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafc5d0 msgr2=0x55d04cb13310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.937+0000 7fbc4047b740 1 --2- 172.21.15.98:0/1662106316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafc5d0 0x55d04cb13310 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x55d04cafd070 tx=0x7fbc2000bf90 comp rx=0 tx=0).stop 2024-09-19T08:10:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.937+0000 7fbc4047b740 1 -- 172.21.15.98:0/1662106316 shutdown_connections 2024-09-19T08:10:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.937+0000 7fbc4047b740 1 --2- 172.21.15.98:0/1662106316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cb07370 0x55d04cb0f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.937+0000 7fbc4047b740 1 --2- 172.21.15.98:0/1662106316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafdb40 0x55d04cb18f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.937+0000 7fbc4047b740 1 --2- 172.21.15.98:0/1662106316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cafc5d0 0x55d04cb13310 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.937+0000 7fbc4047b740 1 -- 172.21.15.98:0/1662106316 >> 172.21.15.98:0/1662106316 conn(0x55d04cac5310 msgr2=0x55d04caff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.937+0000 7fbc4047b740 1 -- 172.21.15.98:0/1662106316 shutdown_connections 2024-09-19T08:10:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.937+0000 7fbc4047b740 1 -- 172.21.15.98:0/1662106316 wait complete. 2024-09-19T08:10:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.937+0000 7fbc4047b740 1 -- start start 2024-09-19T08:10:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.937+0000 7fbc4047b740 1 Processor -- start 2024-09-19T08:10:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc4047b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cafc5d0 0x55d04cb1ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc4047b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafdb40 0x55d04cb1dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc4047b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cb07370 0x55d04cb1e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc4047b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc381aec90 con 0x55d04cb07370 2024-09-19T08:10:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc4047b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbc381ae5c0 con 0x55d04cafc5d0 2024-09-19T08:10:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc4047b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbc381ab820 con 0x55d04cafdb40 2024-09-19T08:10:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc4047b740 10 client.0 _ll_register_callbacks cb 0x7fbc2000b260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc2effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cb07370 0x55d04cb1e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc2effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cb07370 0x55d04cb1e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43712/0 (socket says 172.21.15.98:43712) 2024-09-19T08:10:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc2effd640 1 -- 172.21.15.98:0/1422356257 learned_addr learned my addr 172.21.15.98:0/1422356257 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc2effd640 1 --2- 172.21.15.98:0/1422356257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cafc5d0 0x55d04cb1ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc377fe640 1 --2- 172.21.15.98:0/1422356257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafdb40 0x55d04cb1dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc2effd640 1 -- 172.21.15.98:0/1422356257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafdb40 msgr2=0x55d04cb1dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc2effd640 1 --2- 172.21.15.98:0/1422356257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafdb40 0x55d04cb1dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc2effd640 1 -- 172.21.15.98:0/1422356257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cafc5d0 msgr2=0x55d04cb1ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc2effd640 1 --2- 172.21.15.98:0/1422356257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cafc5d0 0x55d04cb1ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc2effd640 1 -- 172.21.15.98:0/1422356257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d04cb1e830 con 0x55d04cb07370 2024-09-19T08:10:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc2effd640 1 --2- 172.21.15.98:0/1422356257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cb07370 0x55d04cb1e000 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fbc2001d400 tx=0x7fbc2001f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3067f49530795b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc16ffd640 1 -- 172.21.15.98:0/1422356257 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2001ab50 con 0x55d04cb07370 2024-09-19T08:10:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.938+0000 7fbc16ffd640 1 -- 172.21.15.98:0/1422356257 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7fbc20018460 con 0x55d04cb07370 2024-09-19T08:10:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.939+0000 7fbc4047b740 1 -- 172.21.15.98:0/1422356257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc20010040 con 0x55d04cb07370 2024-09-19T08:10:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.939+0000 7fbc16ffd640 1 -- 172.21.15.98:0/1422356257 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbc20018660 con 0x55d04cb07370 2024-09-19T08:10:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.939+0000 7fbc4047b740 10 client.17358 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.939+0000 7fbc4047b740 1 -- 172.21.15.98:0/1422356257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d04cb85820 con 0x55d04cb07370 2024-09-19T08:10:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.939+0000 7fbc16ffd640 1 -- 172.21.15.98:0/1422356257 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc20010040 con 0x55d04cb07370 2024-09-19T08:10:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.939+0000 7fbc16ffd640 1 -- 172.21.15.98:0/1422356257 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fbc2002d750 con 0x55d04cb07370 2024-09-19T08:10:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.939+0000 7fbc4047b740 10 client.17358 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.939+0000 7fbc4047b740 10 client.17358 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.939+0000 7fbc4047b740 20 client.17358 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.939+0000 7fbc4047b740 1 --2- 172.21.15.98:0/1422356257 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d04cb85f10 0x55d04cb88330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.939+0000 7fbc4047b740 4 client.17358 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:10:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.939+0000 7fbc4047b740 1 -- 172.21.15.98:0/1422356257 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d04ca69d70 con 0x55d04cb85f10 2024-09-19T08:10:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.939+0000 7fbc377fe640 1 --2- 172.21.15.98:0/1422356257 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d04cb85f10 0x55d04cb88330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc377fe640 1 --2- 172.21.15.98:0/1422356257 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d04cb85f10 0x55d04cb88330 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3a256ab5f49c6d server_cookie=5d66fe1c33cf7025 in_seq=0 out_seq=0 2024-09-19T08:10:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc16ffd640 10 client.17358 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc16ffd640 1 -- 172.21.15.98:0/1422356257 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d04ca69d70 con 0x55d04cb85f10 2024-09-19T08:10:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc16ffd640 10 client.17358 handle_command_reply: tid=0 2024-09-19T08:10:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc4047b740 1 client.17358 shutdown 2024-09-19T08:10:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc4047b740 1 -- 172.21.15.98:0/1422356257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cb07370 msgr2=0x55d04cb1e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc4047b740 1 --2- 172.21.15.98:0/1422356257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cb07370 0x55d04cb1e000 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fbc2001d400 tx=0x7fbc2001f040 comp rx=0 tx=0).stop 2024-09-19T08:10:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc4047b740 1 -- 172.21.15.98:0/1422356257 shutdown_connections 2024-09-19T08:10:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc4047b740 1 --2- 172.21.15.98:0/1422356257 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55d04cb85f10 0x55d04cb88330 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc4047b740 1 --2- 172.21.15.98:0/1422356257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d04cb07370 0x55d04cb1e000 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc4047b740 1 --2- 172.21.15.98:0/1422356257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04cafdb40 0x55d04cb1dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc4047b740 1 --2- 172.21.15.98:0/1422356257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d04cafc5d0 0x55d04cb1ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc16ffd640 0 client.17358 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc4047b740 1 -- 172.21.15.98:0/1422356257 >> 172.21.15.98:0/1422356257 conn(0x55d04cac5310 msgr2=0x55d04cafe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc4047b740 1 -- 172.21.15.98:0/1422356257 shutdown_connections 2024-09-19T08:10:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc4047b740 1 -- 172.21.15.98:0/1422356257 wait complete. 2024-09-19T08:10:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.940+0000 7fbc4047b740 20 client.17358 trim_cache size 0 max 16384 2024-09-19T08:10:53.947 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:10:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.941+0000 7fbc3e0a2640 1 -- 172.21.15.98:0/3480069929 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbc240a3430 msgr2=0x7fbc240a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.941+0000 7fbc3e0a2640 1 --2- 172.21.15.98:0/3480069929 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbc240a3430 0x7fbc240a58f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbc300c5ab0 tx=0x7fbc3002a8f0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.941+0000 7fbc3e0a2640 1 -- 172.21.15.98:0/3480069929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc38108300 msgr2=0x7fbc3819ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.941+0000 7fbc3e0a2640 1 --2- 172.21.15.98:0/3480069929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc38108300 0x7fbc3819ce90 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009ca0 tx=0x7fbc300039f0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.941+0000 7fbc3e0a2640 1 -- 172.21.15.98:0/3480069929 shutdown_connections 2024-09-19T08:10:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.941+0000 7fbc3e0a2640 1 --2- 172.21.15.98:0/3480069929 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbc240a3430 0x7fbc240a58f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.941+0000 7fbc3e0a2640 1 --2- 172.21.15.98:0/3480069929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc38108300 0x7fbc3819ce90 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.941+0000 7fbc3e0a2640 1 --2- 172.21.15.98:0/3480069929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc38103780 0x7fbc38198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.941+0000 7fbc3e0a2640 1 --2- 172.21.15.98:0/3480069929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc38102dd0 0x7fbc381985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.941+0000 7fbc3e0a2640 1 -- 172.21.15.98:0/3480069929 >> 172.21.15.98:0/3480069929 conn(0x7fbc380fc7d0 msgr2=0x7fbc38105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.941+0000 7fbc3e0a2640 1 -- 172.21.15.98:0/3480069929 shutdown_connections 2024-09-19T08:10:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:53.941+0000 7fbc3e0a2640 1 -- 172.21.15.98:0/3480069929 wait complete. 2024-09-19T08:10:54.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:54 smithi098 ceph-mon[31333]: pgmap v241: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:10:54.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:54 smithi162 ceph-mon[35260]: pgmap v241: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:10:54.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:54 smithi139 ceph-mon[34874]: pgmap v241: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:10:55.666 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:10:55.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:10:55.930 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:56.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:56 smithi098 ceph-mon[31333]: pgmap v242: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-19T08:10:56.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:56 smithi162 ceph-mon[35260]: pgmap v242: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-19T08:10:56.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:56 smithi139 ceph-mon[34874]: pgmap v242: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-19T08:10:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.423+0000 7f5b8d103640 1 -- 172.21.15.98:0/400891334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b88102790 msgr2=0x7f5b881067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.423+0000 7f5b8d103640 1 --2- 172.21.15.98:0/400891334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b88102790 0x7f5b881067e0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009900 tx=0x7f5b7002a410 comp rx=0 tx=0).stop 2024-09-19T08:10:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.423+0000 7f5b8d103640 1 -- 172.21.15.98:0/400891334 shutdown_connections 2024-09-19T08:10:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.423+0000 7f5b8d103640 1 --2- 172.21.15.98:0/400891334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b88106f10 0x7f5b8810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.423+0000 7f5b8d103640 1 --2- 172.21.15.98:0/400891334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b88102790 0x7f5b881067e0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.423+0000 7f5b8d103640 1 --2- 172.21.15.98:0/400891334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5b88101de0 0x7f5b881021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.423+0000 7f5b8d103640 1 -- 172.21.15.98:0/400891334 >> 172.21.15.98:0/400891334 conn(0x7f5b880fd8c0 msgr2=0x7f5b880ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.423+0000 7f5b8d103640 1 -- 172.21.15.98:0/400891334 shutdown_connections 2024-09-19T08:10:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.423+0000 7f5b8d103640 1 -- 172.21.15.98:0/400891334 wait complete. 2024-09-19T08:10:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.424+0000 7f5b8d103640 1 Processor -- start 2024-09-19T08:10:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.424+0000 7f5b8d103640 1 -- start start 2024-09-19T08:10:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b8d103640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b88101de0 0x7f5b881985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b8d103640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b88102790 0x7f5b88198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b8d103640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5b88106f10 0x7f5b8819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b8d103640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5b88112010 con 0x7f5b88101de0 2024-09-19T08:10:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b8d103640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5b88111e90 con 0x7f5b88106f10 2024-09-19T08:10:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b8d103640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5b88112190 con 0x7f5b88102790 2024-09-19T08:10:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b86d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b88101de0 0x7f5b881985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b86d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b88101de0 0x7f5b881985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50756/0 (socket says 172.21.15.98:50756) 2024-09-19T08:10:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b86d9d640 1 -- 172.21.15.98:0/2321550169 learned_addr learned my addr 172.21.15.98:0/2321550169 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b8759e640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5b88106f10 0x7f5b8819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b8659c640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b88102790 0x7f5b88198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b86d9d640 1 -- 172.21.15.98:0/2321550169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b88102790 msgr2=0x7f5b88198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b86d9d640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b88102790 0x7f5b88198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b86d9d640 1 -- 172.21.15.98:0/2321550169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5b88106f10 msgr2=0x7f5b8819ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b86d9d640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5b88106f10 0x7f5b8819ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b86d9d640 1 -- 172.21.15.98:0/2321550169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8819d560 con 0x7f5b88101de0 2024-09-19T08:10:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b8659c640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b88102790 0x7f5b88198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b8759e640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5b88106f10 0x7f5b8819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:10:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b86d9d640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b88101de0 0x7f5b881985b0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800ba50 tx=0x7f5b7800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b7ffff640 1 -- 172.21.15.98:0/2321550169 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b78019070 con 0x7f5b88101de0 2024-09-19T08:10:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b8d103640 1 -- 172.21.15.98:0/2321550169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b8819d850 con 0x7f5b88101de0 2024-09-19T08:10:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b7ffff640 1 -- 172.21.15.98:0/2321550169 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b7800cbe0 con 0x7f5b88101de0 2024-09-19T08:10:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b7ffff640 1 -- 172.21.15.98:0/2321550169 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b780076e0 con 0x7f5b88101de0 2024-09-19T08:10:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.425+0000 7f5b8d103640 1 -- 172.21.15.98:0/2321550169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b881a5130 con 0x7f5b88101de0 2024-09-19T08:10:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.428+0000 7f5b7ffff640 1 -- 172.21.15.98:0/2321550169 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f5b7801c070 con 0x7f5b88101de0 2024-09-19T08:10:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.428+0000 7f5b8f4dc740 1 Processor -- start 2024-09-19T08:10:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.428+0000 7f5b8f4dc740 1 -- start start 2024-09-19T08:10:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.428+0000 7f5b8f4dc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717acaa40 0x561717acae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.428+0000 7f5b8f4dc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 0x561717ad9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.428+0000 7f5b8f4dc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717ada150 0x561717adc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.428+0000 7f5b8f4dc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561717a30280 con 0x561717acaa40 2024-09-19T08:10:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.428+0000 7f5b8f4dc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5617179d7b10 con 0x561717acb360 2024-09-19T08:10:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.428+0000 7f5b8f4dc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561717a2c240 con 0x561717ada150 2024-09-19T08:10:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.428+0000 7f5b8759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717acaa40 0x561717acae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.428+0000 7f5b8759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717acaa40 0x561717acae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50764/0 (socket says 172.21.15.98:50764) 2024-09-19T08:10:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.428+0000 7f5b8759e640 1 -- 172.21.15.98:0/2595415579 learned_addr learned my addr 172.21.15.98:0/2595415579 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b7ffff640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5b740a33b0 0x7f5b740a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b8659c640 1 --2- 172.21.15.98:0/2595415579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717ada150 0x561717adc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b86d9d640 1 --2- 172.21.15.98:0/2595415579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 0x561717ad9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b8759e640 1 -- 172.21.15.98:0/2595415579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717ada150 msgr2=0x561717adc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b8759e640 1 --2- 172.21.15.98:0/2595415579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717ada150 0x561717adc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b8759e640 1 -- 172.21.15.98:0/2595415579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 msgr2=0x561717ad9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b8759e640 1 --2- 172.21.15.98:0/2595415579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 0x561717ad9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b8759e640 1 -- 172.21.15.98:0/2595415579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617179e41f0 con 0x561717acaa40 2024-09-19T08:10:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b86d9d640 1 --2- 172.21.15.98:0/2595415579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 0x561717ad9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b7ffff640 1 -- 172.21.15.98:0/2321550169 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f5b780cf270 con 0x7f5b88101de0 2024-09-19T08:10:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b8759e640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5b740a33b0 0x7f5b740a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b8759e640 1 --2- 172.21.15.98:0/2595415579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717acaa40 0x561717acae20 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f5b80007940 tx=0x7f5b80007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73da7c514ac76324 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b7d7fa640 1 -- 172.21.15.98:0/2595415579 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b80015e60 con 0x561717acaa40 2024-09-19T08:10:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b7d7fa640 1 -- 172.21.15.98:0/2595415579 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b80019070 con 0x561717acaa40 2024-09-19T08:10:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.429+0000 7f5b7d7fa640 1 -- 172.21.15.98:0/2595415579 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8000d760 con 0x561717acaa40 2024-09-19T08:10:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.430+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/2595415579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717acaa40 msgr2=0x561717acae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.430+0000 7f5b8759e640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5b740a33b0 0x7f5b740a5870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5b8001e000 tx=0x7f5b8001f000 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.430+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/2595415579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717acaa40 0x561717acae20 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f5b80007940 tx=0x7f5b80007e00 comp rx=0 tx=0).stop 2024-09-19T08:10:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.430+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/2595415579 shutdown_connections 2024-09-19T08:10:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.430+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/2595415579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717ada150 0x561717adc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.430+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/2595415579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 0x561717ad9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.430+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/2595415579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717acaa40 0x561717acae20 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.430+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/2595415579 >> 172.21.15.98:0/2595415579 conn(0x561717ac70f0 msgr2=0x561717ac74d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.430+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/2595415579 shutdown_connections 2024-09-19T08:10:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.430+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/2595415579 wait complete. 2024-09-19T08:10:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.430+0000 7f5b8f4dc740 1 -- start start 2024-09-19T08:10:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8f4dc740 1 Processor -- start 2024-09-19T08:10:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8f4dc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acaa40 0x561717ad70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8f4dc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acb360 0x561717ad7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8f4dc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717ad7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8f4dc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561717adcd20 con 0x561717ada150 2024-09-19T08:10:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8f4dc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5617179eaf10 con 0x561717acaa40 2024-09-19T08:10:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8f4dc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561717adcea0 con 0x561717acb360 2024-09-19T08:10:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8f4dc740 10 client.0 _ll_register_callbacks cb 0x5617179b3a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717ad7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717ad7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50776/0 (socket says 172.21.15.98:50776) 2024-09-19T08:10:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8759e640 1 -- 172.21.15.98:0/1097506873 learned_addr learned my addr 172.21.15.98:0/1097506873 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b86d9d640 1 --2- 172.21.15.98:0/1097506873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acaa40 0x561717ad70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8659c640 1 --2- 172.21.15.98:0/1097506873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acb360 0x561717ad7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8759e640 1 -- 172.21.15.98:0/1097506873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acb360 msgr2=0x561717ad7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8759e640 1 --2- 172.21.15.98:0/1097506873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acb360 0x561717ad7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8759e640 1 -- 172.21.15.98:0/1097506873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acaa40 msgr2=0x561717ad70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8759e640 1 --2- 172.21.15.98:0/1097506873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acaa40 0x561717ad70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8759e640 1 -- 172.21.15.98:0/1097506873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617177a16a0 con 0x561717ada150 2024-09-19T08:10:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b86d9d640 1 --2- 172.21.15.98:0/1097506873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acaa40 0x561717ad70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:10:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b8759e640 1 --2- 172.21.15.98:0/1097506873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717ad7b50 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f5b80014e50 tx=0x7f5b80014590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e6a9112932e4c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.431+0000 7f5b6e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.432+0000 7f5b6e7fc640 1 -- 172.21.15.98:0/1097506873 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b80015e60 con 0x561717ada150 2024-09-19T08:10:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.432+0000 7f5b6e7fc640 1 -- 172.21.15.98:0/1097506873 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f5b8001f3f0 con 0x561717ada150 2024-09-19T08:10:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.432+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/1097506873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b80019070 con 0x561717ada150 2024-09-19T08:10:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.432+0000 7f5b6e7fc640 1 -- 172.21.15.98:0/1097506873 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5b80038050 con 0x561717ada150 2024-09-19T08:10:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.432+0000 7f5b8f4dc740 10 client.17370 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.432+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/1097506873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561717ae8970 con 0x561717ada150 2024-09-19T08:10:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.432+0000 7f5b6e7fc640 1 -- 172.21.15.98:0/1097506873 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b80019070 con 0x561717ada150 2024-09-19T08:10:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.432+0000 7f5b6e7fc640 1 -- 172.21.15.98:0/1097506873 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5b80026040 con 0x561717ada150 2024-09-19T08:10:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.432+0000 7f5b8f4dc740 10 client.17370 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.432+0000 7f5b8f4dc740 10 client.17370 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.432+0000 7f5b8f4dc740 20 client.17370 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.432+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/1097506873 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561717ae9d20 0x561717aec110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.432+0000 7f5b8f4dc740 4 client.17370 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.432+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/1097506873 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561717a3d7f0 con 0x561717ae9d20 2024-09-19T08:10:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.433+0000 7f5b86d9d640 1 --2- 172.21.15.98:0/1097506873 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561717ae9d20 0x561717aec110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.433+0000 7f5b6e7fc640 10 client.17370 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.433+0000 7f5b86d9d640 1 --2- 172.21.15.98:0/1097506873 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561717ae9d20 0x561717aec110 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da8167a1617309ce server_cookie=bd6aabcc56c21e19 in_seq=0 out_seq=0 2024-09-19T08:10:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b6e7fc640 1 -- 172.21.15.98:0/1097506873 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561717a3d7f0 con 0x561717ae9d20 2024-09-19T08:10:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b6e7fc640 10 client.17370 handle_command_reply: tid=0 2024-09-19T08:10:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b8f4dc740 1 client.17370 shutdown 2024-09-19T08:10:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/1097506873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 msgr2=0x561717ad7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/1097506873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717ad7b50 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f5b80014e50 tx=0x7f5b80014590 comp rx=0 tx=0).stop 2024-09-19T08:10:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/1097506873 shutdown_connections 2024-09-19T08:10:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/1097506873 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561717ae9d20 0x561717aec110 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/1097506873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717ad7b50 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/1097506873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acb360 0x561717ad7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/1097506873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acaa40 0x561717ad70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/1097506873 >> 172.21.15.98:0/1097506873 conn(0x561717ac70f0 msgr2=0x561717acce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b6e7fc640 0 client.17370 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/1097506873 shutdown_connections 2024-09-19T08:10:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/1097506873 wait complete. 2024-09-19T08:10:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.434+0000 7f5b8f4dc740 20 client.17370 trim_cache size 0 max 16384 2024-09-19T08:10:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.451+0000 7f5b8f4dc740 1 Processor -- start 2024-09-19T08:10:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.451+0000 7f5b8f4dc740 1 -- start start 2024-09-19T08:10:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.451+0000 7f5b8f4dc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acaa40 0x561717aec190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.451+0000 7f5b8f4dc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 0x561717ad8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.451+0000 7f5b8f4dc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717af1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.451+0000 7f5b8f4dc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5b881ab7d0 con 0x561717ada150 2024-09-19T08:10:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.451+0000 7f5b8f4dc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5b881ad310 con 0x561717acb360 2024-09-19T08:10:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.451+0000 7f5b8f4dc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5b881abea0 con 0x561717acaa40 2024-09-19T08:10:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.451+0000 7f5b8759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717af1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.451+0000 7f5b8759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717af1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50782/0 (socket says 172.21.15.98:50782) 2024-09-19T08:10:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.451+0000 7f5b8759e640 1 -- 172.21.15.98:0/947912350 learned_addr learned my addr 172.21.15.98:0/947912350 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.452+0000 7f5b8759e640 1 -- 172.21.15.98:0/947912350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acaa40 msgr2=0x561717aec190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.452+0000 7f5b86d9d640 1 --2- 172.21.15.98:0/947912350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acaa40 0x561717aec190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.452+0000 7f5b8759e640 1 --2- 172.21.15.98:0/947912350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acaa40 0x561717aec190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.452+0000 7f5b8759e640 1 -- 172.21.15.98:0/947912350 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 msgr2=0x561717ad8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.452+0000 7f5b8759e640 1 --2- 172.21.15.98:0/947912350 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 0x561717ad8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.452+0000 7f5b8759e640 1 -- 172.21.15.98:0/947912350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561717ae5fc0 con 0x561717ada150 2024-09-19T08:10:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.452+0000 7f5b8759e640 1 --2- 172.21.15.98:0/947912350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717af1850 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5b80004810 tx=0x7f5b80014600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c98c123cefd5dbdc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.452+0000 7f5b7cff9640 1 -- 172.21.15.98:0/947912350 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b80015e60 con 0x561717ada150 2024-09-19T08:10:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.452+0000 7f5b7cff9640 1 -- 172.21.15.98:0/947912350 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b80019040 con 0x561717ada150 2024-09-19T08:10:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.452+0000 7f5b7cff9640 1 -- 172.21.15.98:0/947912350 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b80027de0 con 0x561717ada150 2024-09-19T08:10:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.453+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/947912350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 msgr2=0x561717af1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.453+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/947912350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717af1850 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5b80004810 tx=0x7f5b80014600 comp rx=0 tx=0).stop 2024-09-19T08:10:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.453+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/947912350 shutdown_connections 2024-09-19T08:10:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.453+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/947912350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717af1850 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.453+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/947912350 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 0x561717ad8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.453+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/947912350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acaa40 0x561717aec190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.453+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/947912350 >> 172.21.15.98:0/947912350 conn(0x561717ac70f0 msgr2=0x561717ad8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:56.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.453+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/947912350 shutdown_connections 2024-09-19T08:10:56.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.453+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/947912350 wait complete. 2024-09-19T08:10:56.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.453+0000 7f5b8f4dc740 1 -- start start 2024-09-19T08:10:56.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.453+0000 7f5b8f4dc740 1 Processor -- start 2024-09-19T08:10:56.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.453+0000 7f5b8f4dc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acaa40 0x561717ae26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.453+0000 7f5b8f4dc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 0x561717adda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.453+0000 7f5b8f4dc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717addf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8f4dc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5b881acc40 con 0x561717ada150 2024-09-19T08:10:56.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8f4dc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5b881ac570 con 0x561717acb360 2024-09-19T08:10:56.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8f4dc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5b881a97d0 con 0x561717acaa40 2024-09-19T08:10:56.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8f4dc740 10 client.0 _ll_register_callbacks cb 0x7f5b80007770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:56.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717addf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717addf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50796/0 (socket says 172.21.15.98:50796) 2024-09-19T08:10:56.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8759e640 1 -- 172.21.15.98:0/212905054 learned_addr learned my addr 172.21.15.98:0/212905054 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:56.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b86d9d640 1 --2- 172.21.15.98:0/212905054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acaa40 0x561717ae26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8659c640 1 --2- 172.21.15.98:0/212905054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 0x561717adda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8759e640 1 -- 172.21.15.98:0/212905054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acaa40 msgr2=0x561717ae26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:56.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8759e640 1 --2- 172.21.15.98:0/212905054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acaa40 0x561717ae26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8759e640 1 -- 172.21.15.98:0/212905054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 msgr2=0x561717adda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:56.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8759e640 1 --2- 172.21.15.98:0/212905054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 0x561717adda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8759e640 1 -- 172.21.15.98:0/212905054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561717ade5d0 con 0x561717ada150 2024-09-19T08:10:56.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8759e640 1 --2- 172.21.15.98:0/212905054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717addf60 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5b8000d530 tx=0x7f5b80007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7b300e4337177a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:56.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b6e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:56.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b6e7fc640 1 -- 172.21.15.98:0/212905054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b80015e60 con 0x561717ada150 2024-09-19T08:10:56.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b6e7fc640 1 -- 172.21.15.98:0/212905054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7f5b8003c430 con 0x561717ada150 2024-09-19T08:10:56.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/212905054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b80019040 con 0x561717ada150 2024-09-19T08:10:56.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b6e7fc640 1 -- 172.21.15.98:0/212905054 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5b8004c050 con 0x561717ada150 2024-09-19T08:10:56.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8f4dc740 10 client.17376 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:56.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.454+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/212905054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561717b546d0 con 0x561717ada150 2024-09-19T08:10:56.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.455+0000 7f5b6e7fc640 1 -- 172.21.15.98:0/212905054 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b80019040 con 0x561717ada150 2024-09-19T08:10:56.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.455+0000 7f5b6e7fc640 1 -- 172.21.15.98:0/212905054 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5b80026040 con 0x561717ada150 2024-09-19T08:10:56.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.455+0000 7f5b8f4dc740 10 client.17376 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:56.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.455+0000 7f5b8f4dc740 10 client.17376 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:56.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.455+0000 7f5b8f4dc740 20 client.17376 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:56.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.455+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/212905054 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561717b54d20 0x561717b57110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:56.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.455+0000 7f5b8f4dc740 4 client.17376 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:10:56.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.455+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/212905054 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561717a37d00 con 0x561717b54d20 2024-09-19T08:10:56.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.455+0000 7f5b86d9d640 1 --2- 172.21.15.98:0/212905054 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561717b54d20 0x561717b57110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:56.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.455+0000 7f5b86d9d640 1 --2- 172.21.15.98:0/212905054 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561717b54d20 0x561717b57110 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d674d3ec463ab455 server_cookie=9ad0090834c4604d in_seq=0 out_seq=0 2024-09-19T08:10:56.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.455+0000 7f5b6e7fc640 10 client.17376 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:56.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.456+0000 7f5b6e7fc640 1 -- 172.21.15.98:0/212905054 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561717a37d00 con 0x561717b54d20 2024-09-19T08:10:56.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.456+0000 7f5b6e7fc640 10 client.17376 handle_command_reply: tid=0 2024-09-19T08:10:56.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.456+0000 7f5b8f4dc740 1 client.17376 shutdown 2024-09-19T08:10:56.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/212905054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 msgr2=0x561717addf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:56.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/212905054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717addf60 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5b8000d530 tx=0x7f5b80007e00 comp rx=0 tx=0).stop 2024-09-19T08:10:56.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/212905054 shutdown_connections 2024-09-19T08:10:56.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/212905054 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561717b54d20 0x561717b57110 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/212905054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561717ada150 0x561717addf60 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/212905054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561717acb360 0x561717adda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8f4dc740 1 --2- 172.21.15.98:0/212905054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561717acaa40 0x561717ae26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/212905054 >> 172.21.15.98:0/212905054 conn(0x561717ac70f0 msgr2=0x561717ae57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:56.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b6e7fc640 0 client.17376 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:56.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/212905054 shutdown_connections 2024-09-19T08:10:56.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8f4dc740 1 -- 172.21.15.98:0/212905054 wait complete. 2024-09-19T08:10:56.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8f4dc740 20 client.17376 trim_cache size 0 max 16384 2024-09-19T08:10:56.463 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:10:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:10:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "686755c7-a8b3-4d69-9a41-581440edc602", 2024-09-19T08:10:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:10:56.464 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:10:56.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8d103640 1 -- 172.21.15.98:0/2321550169 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5b740a33b0 msgr2=0x7f5b740a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:56.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8d103640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5b740a33b0 0x7f5b740a5870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5b8001e000 tx=0x7f5b8001f000 comp rx=0 tx=0).stop 2024-09-19T08:10:56.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8d103640 1 -- 172.21.15.98:0/2321550169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b88101de0 msgr2=0x7f5b881985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:56.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.457+0000 7f5b8d103640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b88101de0 0x7f5b881985b0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800ba50 tx=0x7f5b7800bf10 comp rx=0 tx=0).stop 2024-09-19T08:10:56.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.458+0000 7f5b8d103640 1 -- 172.21.15.98:0/2321550169 shutdown_connections 2024-09-19T08:10:56.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.458+0000 7f5b8d103640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f5b740a33b0 0x7f5b740a5870 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.458+0000 7f5b8d103640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5b88106f10 0x7f5b8819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.458+0000 7f5b8d103640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b88102790 0x7f5b88198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.458+0000 7f5b8d103640 1 --2- 172.21.15.98:0/2321550169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b88101de0 0x7f5b881985b0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:56.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.458+0000 7f5b8d103640 1 -- 172.21.15.98:0/2321550169 >> 172.21.15.98:0/2321550169 conn(0x7f5b880fd8c0 msgr2=0x7f5b880fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:56.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.458+0000 7f5b8d103640 1 -- 172.21.15.98:0/2321550169 shutdown_connections 2024-09-19T08:10:56.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:56.458+0000 7f5b8d103640 1 -- 172.21.15.98:0/2321550169 wait complete. 2024-09-19T08:10:57.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:10:57.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:57 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:10:57.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:57 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:10:57.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:57 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:10:57.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:57 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:10:57.394 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:10:57.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:57 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:10:57.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:57 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:10:57.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:57 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:10:57.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:57 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:10:57.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:57 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:10:57.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:57 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:10:57.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:57 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:10:57.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:57 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:10:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.841+0000 7fd8c4825640 1 -- 172.21.15.98:0/1410039513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c0102790 msgr2=0x7fd8c01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.841+0000 7fd8c4825640 1 --2- 172.21.15.98:0/1410039513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c0102790 0x7fd8c01067e0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009900 tx=0x7fd8b402a410 comp rx=0 tx=0).stop 2024-09-19T08:10:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.841+0000 7fd8c4825640 1 -- 172.21.15.98:0/1410039513 shutdown_connections 2024-09-19T08:10:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.841+0000 7fd8c4825640 1 --2- 172.21.15.98:0/1410039513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8c0106f10 0x7fd8c010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.841+0000 7fd8c4825640 1 --2- 172.21.15.98:0/1410039513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c0102790 0x7fd8c01067e0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.841+0000 7fd8c4825640 1 --2- 172.21.15.98:0/1410039513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8c0101de0 0x7fd8c01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.841+0000 7fd8c4825640 1 -- 172.21.15.98:0/1410039513 >> 172.21.15.98:0/1410039513 conn(0x7fd8c00fd8c0 msgr2=0x7fd8c00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.841+0000 7fd8c4825640 1 -- 172.21.15.98:0/1410039513 shutdown_connections 2024-09-19T08:10:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.841+0000 7fd8c4825640 1 -- 172.21.15.98:0/1410039513 wait complete. 2024-09-19T08:10:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.842+0000 7fd8c4825640 1 Processor -- start 2024-09-19T08:10:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.842+0000 7fd8c4825640 1 -- start start 2024-09-19T08:10:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.842+0000 7fd8c4825640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8c0101de0 0x7fd8c0198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.842+0000 7fd8c4825640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8c0102790 0x7fd8c0198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.842+0000 7fd8c4825640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c0106f10 0x7fd8c019ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.842+0000 7fd8c4825640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0111f90 con 0x7fd8c0106f10 2024-09-19T08:10:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.842+0000 7fd8c4825640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0111e10 con 0x7fd8c0102790 2024-09-19T08:10:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.842+0000 7fd8c4825640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0112110 con 0x7fd8c0101de0 2024-09-19T08:10:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.842+0000 7fd8bed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c0106f10 0x7fd8c019ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.842+0000 7fd8bed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c0106f10 0x7fd8c019ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50802/0 (socket says 172.21.15.98:50802) 2024-09-19T08:10:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.842+0000 7fd8bed9d640 1 -- 172.21.15.98:0/2692170989 learned_addr learned my addr 172.21.15.98:0/2692170989 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.842+0000 7fd8be59c640 1 --2- 172.21.15.98:0/2692170989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8c0101de0 0x7fd8c0198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.843+0000 7fd8bed9d640 1 -- 172.21.15.98:0/2692170989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8c0101de0 msgr2=0x7fd8c0198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.843+0000 7fd8bed9d640 1 --2- 172.21.15.98:0/2692170989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8c0101de0 0x7fd8c0198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.843+0000 7fd8bed9d640 1 -- 172.21.15.98:0/2692170989 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8c0102790 msgr2=0x7fd8c0198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.843+0000 7fd8bed9d640 1 --2- 172.21.15.98:0/2692170989 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8c0102790 0x7fd8c0198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.843+0000 7fd8bed9d640 1 -- 172.21.15.98:0/2692170989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c019d3d0 con 0x7fd8c0106f10 2024-09-19T08:10:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.843+0000 7fd8bed9d640 1 --2- 172.21.15.98:0/2692170989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c0106f10 0x7fd8c019ccf0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800b970 tx=0x7fd8b800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.843+0000 7fd8af7fe640 1 -- 172.21.15.98:0/2692170989 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8019070 con 0x7fd8c0106f10 2024-09-19T08:10:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.843+0000 7fd8af7fe640 1 -- 172.21.15.98:0/2692170989 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8b800cb40 con 0x7fd8c0106f10 2024-09-19T08:10:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.843+0000 7fd8c4825640 1 -- 172.21.15.98:0/2692170989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c019d6c0 con 0x7fd8c0106f10 2024-09-19T08:10:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.843+0000 7fd8af7fe640 1 -- 172.21.15.98:0/2692170989 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8007650 con 0x7fd8c0106f10 2024-09-19T08:10:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.843+0000 7fd8c4825640 1 -- 172.21.15.98:0/2692170989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c01a4fa0 con 0x7fd8c0106f10 2024-09-19T08:10:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.845+0000 7fd8af7fe640 1 -- 172.21.15.98:0/2692170989 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fd8b801c070 con 0x7fd8c0106f10 2024-09-19T08:10:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8c6bfe740 1 Processor -- start 2024-09-19T08:10:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8c6bfe740 1 -- start start 2024-09-19T08:10:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8c6bfe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cec4a0 0x55afe6cec880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8c6bfe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 0x55afe6cf7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8c6bfe740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cf7680 0x55afe6cf9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8c6bfe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55afe6c522e0 con 0x55afe6cec4a0 2024-09-19T08:10:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8c6bfe740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55afe6bf9ce0 con 0x55afe6cecdc0 2024-09-19T08:10:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8c6bfe740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55afe6c4e2a0 con 0x55afe6cf7680 2024-09-19T08:10:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8af7fe640 1 --2- 172.21.15.98:0/2692170989 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd8a80a33b0 0x7fd8a80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8bdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cec4a0 0x55afe6cec880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8bdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cec4a0 0x55afe6cec880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50806/0 (socket says 172.21.15.98:50806) 2024-09-19T08:10:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8bdd9b640 1 -- 172.21.15.98:0/4096348020 learned_addr learned my addr 172.21.15.98:0/4096348020 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8bed9d640 1 --2- 172.21.15.98:0/2692170989 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd8a80a33b0 0x7fd8a80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8be59c640 1 --2- 172.21.15.98:0/4096348020 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 0x55afe6cf7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.846+0000 7fd8af7fe640 1 -- 172.21.15.98:0/2692170989 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7fd8b80cf220 con 0x7fd8c0106f10 2024-09-19T08:10:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.847+0000 7fd8bdd9b640 1 -- 172.21.15.98:0/4096348020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cf7680 msgr2=0x55afe6cf9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.847+0000 7fd8bdd9b640 1 --2- 172.21.15.98:0/4096348020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cf7680 0x55afe6cf9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.847+0000 7fd8bdd9b640 1 -- 172.21.15.98:0/4096348020 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 msgr2=0x55afe6cf7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.847+0000 7fd8bdd9b640 1 --2- 172.21.15.98:0/4096348020 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 0x55afe6cf7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.847+0000 7fd8bdd9b640 1 -- 172.21.15.98:0/4096348020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afe6bde440 con 0x55afe6cec4a0 2024-09-19T08:10:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.847+0000 7fd8bed9d640 1 --2- 172.21.15.98:0/2692170989 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd8a80a33b0 0x7fd8a80a5870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8097a70 tx=0x7fd8b806d390 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.847+0000 7fd8bdd9b640 1 --2- 172.21.15.98:0/4096348020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cec4a0 0x55afe6cec880 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fd8b402a930 tx=0x7fd8b4032c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebde40a8e97fa8e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.847+0000 7fd8acff9640 1 -- 172.21.15.98:0/4096348020 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4040070 con 0x55afe6cec4a0 2024-09-19T08:10:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.847+0000 7fd8acff9640 1 -- 172.21.15.98:0/4096348020 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8b4032e20 con 0x55afe6cec4a0 2024-09-19T08:10:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.847+0000 7fd8acff9640 1 -- 172.21.15.98:0/4096348020 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4035040 con 0x55afe6cec4a0 2024-09-19T08:10:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.848+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/4096348020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cec4a0 msgr2=0x55afe6cec880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.848+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/4096348020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cec4a0 0x55afe6cec880 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fd8b402a930 tx=0x7fd8b4032c30 comp rx=0 tx=0).stop 2024-09-19T08:10:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.848+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/4096348020 shutdown_connections 2024-09-19T08:10:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.848+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/4096348020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cf7680 0x55afe6cf9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.848+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/4096348020 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 0x55afe6cf7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.848+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/4096348020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cec4a0 0x55afe6cec880 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.848+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/4096348020 >> 172.21.15.98:0/4096348020 conn(0x55afe6ce8a30 msgr2=0x55afe6ce8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.848+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/4096348020 shutdown_connections 2024-09-19T08:10:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.848+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/4096348020 wait complete. 2024-09-19T08:10:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.848+0000 7fd8c6bfe740 1 -- start start 2024-09-19T08:10:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.848+0000 7fd8c6bfe740 1 Processor -- start 2024-09-19T08:10:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.848+0000 7fd8c6bfe740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cec4a0 0x55afe6cf7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8c6bfe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 0x55afe6cf42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8c6bfe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cf7680 0x55afe6cf48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8c6bfe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55afe6cfa250 con 0x55afe6cf7680 2024-09-19T08:10:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8c6bfe740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55afe6c0d050 con 0x55afe6cecdc0 2024-09-19T08:10:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8c6bfe740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55afe6cfa3d0 con 0x55afe6cec4a0 2024-09-19T08:10:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8c6bfe740 10 client.0 _ll_register_callbacks cb 0x55afe6c4d160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8bdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cf7680 0x55afe6cf48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8bdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cf7680 0x55afe6cf48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50822/0 (socket says 172.21.15.98:50822) 2024-09-19T08:10:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8bdd9b640 1 -- 172.21.15.98:0/2975126563 learned_addr learned my addr 172.21.15.98:0/2975126563 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8bdd9b640 1 --2- 172.21.15.98:0/2975126563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cec4a0 0x55afe6cf7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8be59c640 1 --2- 172.21.15.98:0/2975126563 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 0x55afe6cf42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8bdd9b640 1 -- 172.21.15.98:0/2975126563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cec4a0 msgr2=0x55afe6cf7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8bdd9b640 1 --2- 172.21.15.98:0/2975126563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cec4a0 0x55afe6cf7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8bdd9b640 1 -- 172.21.15.98:0/2975126563 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 msgr2=0x55afe6cf42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8bdd9b640 1 --2- 172.21.15.98:0/2975126563 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 0x55afe6cf42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8bdd9b640 1 -- 172.21.15.98:0/2975126563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afe69c33c0 con 0x55afe6cf7680 2024-09-19T08:10:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8bdd9b640 1 --2- 172.21.15.98:0/2975126563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cf7680 0x55afe6cf48e0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4003970 tx=0x7fd8b4003870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a78e6b08534b44 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8a5ffb640 1 -- 172.21.15.98:0/2975126563 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4035030 con 0x55afe6cf7680 2024-09-19T08:10:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8a5ffb640 1 -- 172.21.15.98:0/2975126563 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7fd8b40315b0 con 0x55afe6cf7680 2024-09-19T08:10:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2975126563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8b4032e20 con 0x55afe6cf7680 2024-09-19T08:10:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8a5ffb640 1 -- 172.21.15.98:0/2975126563 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd8b405a050 con 0x55afe6cf7680 2024-09-19T08:10:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8c6bfe740 10 client.17388 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.849+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2975126563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55afe6d05c70 con 0x55afe6cf7680 2024-09-19T08:10:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.850+0000 7fd8a5ffb640 1 -- 172.21.15.98:0/2975126563 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8b4032e20 con 0x55afe6cf7680 2024-09-19T08:10:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.850+0000 7fd8a5ffb640 1 -- 172.21.15.98:0/2975126563 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd8b403c420 con 0x55afe6cf7680 2024-09-19T08:10:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.850+0000 7fd8c6bfe740 10 client.17388 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.850+0000 7fd8c6bfe740 10 client.17388 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.850+0000 7fd8c6bfe740 20 client.17388 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.850+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/2975126563 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55afe6d06fc0 0x55afe6d093b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.850+0000 7fd8c6bfe740 4 client.17388 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:10:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.850+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2975126563 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afe6c5f850 con 0x55afe6d06fc0 2024-09-19T08:10:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.850+0000 7fd8be59c640 1 --2- 172.21.15.98:0/2975126563 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55afe6d06fc0 0x55afe6d093b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.851+0000 7fd8a5ffb640 10 client.17388 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.851+0000 7fd8be59c640 1 --2- 172.21.15.98:0/2975126563 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55afe6d06fc0 0x55afe6d093b0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14933a41c95ef3d6 server_cookie=2cf9e86f622ef02a in_seq=0 out_seq=0 2024-09-19T08:10:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.851+0000 7fd8a5ffb640 1 -- 172.21.15.98:0/2975126563 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55afe6c5f850 con 0x55afe6d06fc0 2024-09-19T08:10:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.851+0000 7fd8a5ffb640 10 client.17388 handle_command_reply: tid=0 2024-09-19T08:10:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.851+0000 7fd8c6bfe740 1 client.17388 shutdown 2024-09-19T08:10:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.852+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2975126563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cf7680 msgr2=0x55afe6cf48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.852+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/2975126563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cf7680 0x55afe6cf48e0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4003970 tx=0x7fd8b4003870 comp rx=0 tx=0).stop 2024-09-19T08:10:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.852+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2975126563 shutdown_connections 2024-09-19T08:10:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.852+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/2975126563 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55afe6d06fc0 0x55afe6d093b0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.852+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/2975126563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cf7680 0x55afe6cf48e0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.852+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/2975126563 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 0x55afe6cf42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.852+0000 7fd8a5ffb640 0 client.17388 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.852+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/2975126563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cec4a0 0x55afe6cf7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.852+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2975126563 >> 172.21.15.98:0/2975126563 conn(0x55afe6ce8a30 msgr2=0x55afe6cee760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.852+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2975126563 shutdown_connections 2024-09-19T08:10:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.852+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2975126563 wait complete. 2024-09-19T08:10:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.852+0000 7fd8c6bfe740 20 client.17388 trim_cache size 0 max 16384 2024-09-19T08:10:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.870+0000 7fd8c6bfe740 1 Processor -- start 2024-09-19T08:10:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.870+0000 7fd8c6bfe740 1 -- start start 2024-09-19T08:10:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8c6bfe740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cec4a0 0x55afe6cf3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8c6bfe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cecdc0 0x55afe6cf6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8c6bfe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cf7680 0x55afe6d0c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8c6bfe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01ad760 con 0x55afe6cecdc0 2024-09-19T08:10:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8c6bfe740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01af2a0 con 0x55afe6cf7680 2024-09-19T08:10:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8c6bfe740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01ade30 con 0x55afe6cec4a0 2024-09-19T08:10:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8be59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cecdc0 0x55afe6cf6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8be59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cecdc0 0x55afe6cf6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50834/0 (socket says 172.21.15.98:50834) 2024-09-19T08:10:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8be59c640 1 -- 172.21.15.98:0/4021242572 learned_addr learned my addr 172.21.15.98:0/4021242572 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8bdd9b640 1 --2- 172.21.15.98:0/4021242572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cec4a0 0x55afe6cf3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8be59c640 1 -- 172.21.15.98:0/4021242572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cec4a0 msgr2=0x55afe6cf3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8be59c640 1 --2- 172.21.15.98:0/4021242572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cec4a0 0x55afe6cf3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8be59c640 1 -- 172.21.15.98:0/4021242572 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cf7680 msgr2=0x55afe6d0c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8be59c640 1 --2- 172.21.15.98:0/4021242572 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cf7680 0x55afe6d0c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8be59c640 1 -- 172.21.15.98:0/4021242572 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afe6cf32a0 con 0x55afe6cecdc0 2024-09-19T08:10:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8be59c640 1 --2- 172.21.15.98:0/4021242572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cecdc0 0x55afe6cf6260 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x55afe6cf5930 tx=0x7fd8b0005d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab273c188d2de10 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8a7fff640 1 -- 172.21.15.98:0/4021242572 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0004220 con 0x55afe6cecdc0 2024-09-19T08:10:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8a7fff640 1 -- 172.21.15.98:0/4021242572 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8b00043c0 con 0x55afe6cecdc0 2024-09-19T08:10:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.871+0000 7fd8a7fff640 1 -- 172.21.15.98:0/4021242572 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b001b710 con 0x55afe6cecdc0 2024-09-19T08:10:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.872+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/4021242572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cecdc0 msgr2=0x55afe6cf6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.872+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/4021242572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cecdc0 0x55afe6cf6260 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x55afe6cf5930 tx=0x7fd8b0005d60 comp rx=0 tx=0).stop 2024-09-19T08:10:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.872+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/4021242572 shutdown_connections 2024-09-19T08:10:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.872+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/4021242572 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cf7680 0x55afe6d0c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.872+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/4021242572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cecdc0 0x55afe6cf6260 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.872+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/4021242572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cec4a0 0x55afe6cf3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.872+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/4021242572 >> 172.21.15.98:0/4021242572 conn(0x55afe6ce8a30 msgr2=0x55afe6cf5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.872+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/4021242572 shutdown_connections 2024-09-19T08:10:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.872+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/4021242572 wait complete. 2024-09-19T08:10:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.872+0000 7fd8c6bfe740 1 -- start start 2024-09-19T08:10:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8c6bfe740 1 Processor -- start 2024-09-19T08:10:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8c6bfe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cec4a0 0x55afe6d0c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8c6bfe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 0x55afe6d0c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8c6bfe740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cf7680 0x55afe6d0cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8c6bfe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01aebd0 con 0x55afe6cec4a0 2024-09-19T08:10:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8c6bfe740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01ae500 con 0x55afe6cecdc0 2024-09-19T08:10:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8c6bfe740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01ab760 con 0x55afe6cf7680 2024-09-19T08:10:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8c6bfe740 10 client.0 _ll_register_callbacks cb 0x7fd8b4005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:10:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8bdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cec4a0 0x55afe6d0c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8bdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cec4a0 0x55afe6d0c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50846/0 (socket says 172.21.15.98:50846) 2024-09-19T08:10:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8bdd9b640 1 -- 172.21.15.98:0/2519385369 learned_addr learned my addr 172.21.15.98:0/2519385369 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:10:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8be59c640 1 --2- 172.21.15.98:0/2519385369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 0x55afe6d0c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8bdd9b640 1 -- 172.21.15.98:0/2519385369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cf7680 msgr2=0x55afe6d0cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8bdd9b640 1 --2- 172.21.15.98:0/2519385369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cf7680 0x55afe6d0cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8bdd9b640 1 -- 172.21.15.98:0/2519385369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 msgr2=0x55afe6d0c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:10:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8bdd9b640 1 --2- 172.21.15.98:0/2519385369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 0x55afe6d0c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8bdd9b640 1 -- 172.21.15.98:0/2519385369 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afe6cf73b0 con 0x55afe6cec4a0 2024-09-19T08:10:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8bdd9b640 1 --2- 172.21.15.98:0/2519385369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cec4a0 0x55afe6d0c1c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4001eb0 tx=0x7fd8b4003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b56ac56a13e61c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:10:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.873+0000 7fd8a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:10:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8a5ffb640 1 -- 172.21.15.98:0/2519385369 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4063060 con 0x55afe6cec4a0 2024-09-19T08:10:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2519385369 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8b00043c0 con 0x55afe6cec4a0 2024-09-19T08:10:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8a5ffb640 1 -- 172.21.15.98:0/2519385369 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8521+0+0 (secure 0 0 0) 0x7fd8b403ca60 con 0x55afe6cec4a0 2024-09-19T08:10:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8a5ffb640 1 -- 172.21.15.98:0/2519385369 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd8b4035040 con 0x55afe6cec4a0 2024-09-19T08:10:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8c6bfe740 10 client.17394 fetch_fsmap learned FSMap version 32 2024-09-19T08:10:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2519385369 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55afe6d75ee0 con 0x55afe6cec4a0 2024-09-19T08:10:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8a5ffb640 1 -- 172.21.15.98:0/2519385369 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8b00043c0 con 0x55afe6cec4a0 2024-09-19T08:10:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8a5ffb640 1 -- 172.21.15.98:0/2519385369 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd8b40048c0 con 0x55afe6cec4a0 2024-09-19T08:10:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8c6bfe740 10 client.17394 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:10:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8c6bfe740 10 client.17394 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:10:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8c6bfe740 20 client.17394 populate_metadata read hostname 'smithi098' 2024-09-19T08:10:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/2519385369 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55afe6d76540 0x55afe6d78930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:10:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8c6bfe740 4 client.17394 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:10:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2519385369 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55afe6c59d60 con 0x55afe6d76540 2024-09-19T08:10:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.874+0000 7fd8be59c640 1 --2- 172.21.15.98:0/2519385369 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55afe6d76540 0x55afe6d78930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:10:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8be59c640 1 --2- 172.21.15.98:0/2519385369 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55afe6d76540 0x55afe6d78930 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be6fd06c43fc9565 server_cookie=426915b4f87d300a in_seq=0 out_seq=0 2024-09-19T08:10:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8a5ffb640 10 client.17394 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8a5ffb640 1 -- 172.21.15.98:0/2519385369 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55afe6c59d60 con 0x55afe6d76540 2024-09-19T08:10:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8a5ffb640 10 client.17394 handle_command_reply: tid=0 2024-09-19T08:10:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8c6bfe740 1 client.17394 shutdown 2024-09-19T08:10:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2519385369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cec4a0 msgr2=0x55afe6d0c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/2519385369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cec4a0 0x55afe6d0c1c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4001eb0 tx=0x7fd8b4003f50 comp rx=0 tx=0).stop 2024-09-19T08:10:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2519385369 shutdown_connections 2024-09-19T08:10:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/2519385369 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55afe6d76540 0x55afe6d78930 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/2519385369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afe6cf7680 0x55afe6d0cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/2519385369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55afe6cecdc0 0x55afe6d0c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8c6bfe740 1 --2- 172.21.15.98:0/2519385369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe6cec4a0 0x55afe6d0c1c0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2519385369 >> 172.21.15.98:0/2519385369 conn(0x55afe6ce8a30 msgr2=0x55afe6ce7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8a5ffb640 0 client.17394 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:10:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2519385369 shutdown_connections 2024-09-19T08:10:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8c6bfe740 1 -- 172.21.15.98:0/2519385369 wait complete. 2024-09-19T08:10:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.875+0000 7fd8c6bfe740 20 client.17394 trim_cache size 0 max 16384 2024-09-19T08:10:57.885 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:10:57.885 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-19T08:10:57.885 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-09-19T08:10:57.885 INFO:teuthology.orchestra.run.smithi098.stdout: "686755c7-a8b3-4d69-9a41-581440edc602": { 2024-09-19T08:10:57.885 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-09-19T08:10:57.885 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "686755c7-a8b3-4d69-9a41-581440edc602", 2024-09-19T08:10:57.885 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-09-19T08:10:57.885 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:10:57.885 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:10:57.885 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:10:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.876+0000 7fd8c4825640 1 -- 172.21.15.98:0/2692170989 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd8a80a33b0 msgr2=0x7fd8a80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.876+0000 7fd8c4825640 1 --2- 172.21.15.98:0/2692170989 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd8a80a33b0 0x7fd8a80a5870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8097a70 tx=0x7fd8b806d390 comp rx=0 tx=0).stop 2024-09-19T08:10:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.876+0000 7fd8c4825640 1 -- 172.21.15.98:0/2692170989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c0106f10 msgr2=0x7fd8c019ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:10:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.876+0000 7fd8c4825640 1 --2- 172.21.15.98:0/2692170989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c0106f10 0x7fd8c019ccf0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800b970 tx=0x7fd8b800be30 comp rx=0 tx=0).stop 2024-09-19T08:10:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.876+0000 7fd8c4825640 1 -- 172.21.15.98:0/2692170989 shutdown_connections 2024-09-19T08:10:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.876+0000 7fd8c4825640 1 --2- 172.21.15.98:0/2692170989 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd8a80a33b0 0x7fd8a80a5870 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.876+0000 7fd8c4825640 1 --2- 172.21.15.98:0/2692170989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c0106f10 0x7fd8c019ccf0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.876+0000 7fd8c4825640 1 --2- 172.21.15.98:0/2692170989 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8c0102790 0x7fd8c0198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.876+0000 7fd8c4825640 1 --2- 172.21.15.98:0/2692170989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8c0101de0 0x7fd8c0198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:10:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.876+0000 7fd8c4825640 1 -- 172.21.15.98:0/2692170989 >> 172.21.15.98:0/2692170989 conn(0x7fd8c00fd8c0 msgr2=0x7fd8c00fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:10:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.876+0000 7fd8c4825640 1 -- 172.21.15.98:0/2692170989 shutdown_connections 2024-09-19T08:10:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:10:57.876+0000 7fd8c4825640 1 -- 172.21.15.98:0/2692170989 wait complete. 2024-09-19T08:10:58.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:10:58 smithi098 ceph-mon[31333]: pgmap v243: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:10:58.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:10:58 smithi162 ceph-mon[35260]: pgmap v243: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:10:58.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:10:58 smithi139 ceph-mon[34874]: pgmap v243: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:10:58.550 INFO:tasks.cephfs.filesystem:scrub status for tag:686755c7-a8b3-4d69-9a41-581440edc602 - {'path': '/', 'tag': '686755c7-a8b3-4d69-9a41-581440edc602', 'options': 'recursive,force'} 2024-09-19T08:11:00.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:00 smithi098 ceph-mon[31333]: pgmap v244: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-09-19T08:11:00.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:00 smithi162 ceph-mon[35260]: pgmap v244: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-09-19T08:11:00.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:00 smithi139 ceph-mon[34874]: pgmap v244: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-09-19T08:11:01.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:01 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:11:01.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:01 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:11:01.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:01 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:11:02.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:02 smithi162 ceph-mon[35260]: pgmap v245: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:11:02.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:02 smithi139 ceph-mon[34874]: pgmap v245: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:11:02.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:02 smithi098 ceph-mon[31333]: pgmap v245: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:11:04.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:04 smithi162 ceph-mon[35260]: pgmap v246: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:11:04.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:04 smithi139 ceph-mon[34874]: pgmap v246: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:11:04.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:04 smithi098 ceph-mon[31333]: pgmap v246: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:11:06.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:06 smithi162 ceph-mon[35260]: pgmap v247: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:11:06.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:06 smithi139 ceph-mon[34874]: pgmap v247: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:11:06.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:06 smithi098 ceph-mon[31333]: pgmap v247: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:11:08.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:08 smithi162 ceph-mon[35260]: pgmap v248: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:11:08.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:08 smithi139 ceph-mon[34874]: pgmap v248: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:11:08.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:08 smithi098 ceph-mon[31333]: pgmap v248: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:11:10.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:10 smithi162 ceph-mon[35260]: pgmap v249: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:11:10.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:10 smithi139 ceph-mon[34874]: pgmap v249: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:11:10.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:10 smithi098 ceph-mon[31333]: pgmap v249: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:11:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:12 smithi162 ceph-mon[35260]: pgmap v250: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:11:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:12 smithi162 ceph-mon[35260]: osdmap e84: 12 total, 12 up, 12 in 2024-09-19T08:11:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:12 smithi139 ceph-mon[34874]: pgmap v250: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:11:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:12 smithi139 ceph-mon[34874]: osdmap e84: 12 total, 12 up, 12 in 2024-09-19T08:11:12.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:12 smithi098 ceph-mon[31333]: pgmap v250: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:11:12.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:12 smithi098 ceph-mon[31333]: osdmap e84: 12 total, 12 up, 12 in 2024-09-19T08:11:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:14 smithi162 ceph-mon[35260]: pgmap v252: 97 pgs: 97 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:11:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:14 smithi139 ceph-mon[34874]: pgmap v252: 97 pgs: 97 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:11:14.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:14 smithi098 ceph-mon[31333]: pgmap v252: 97 pgs: 97 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:11:15.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:15 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:11:15.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:15 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:11:15.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:15 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:11:15.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:15 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:11:15.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:15 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:11:15.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:15 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:11:15.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:15 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:11:15.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:15 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:11:15.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:15 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:11:15.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:15 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:11:15.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:15 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:11:15.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:15 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:11:17.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:16 smithi162 ceph-mon[35260]: pgmap v253: 97 pgs: 97 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:11:17.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:16 smithi162 ceph-mon[35260]: osdmap e85: 12 total, 12 up, 12 in 2024-09-19T08:11:17.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:16 smithi139 ceph-mon[34874]: pgmap v253: 97 pgs: 97 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:11:17.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:16 smithi139 ceph-mon[34874]: osdmap e85: 12 total, 12 up, 12 in 2024-09-19T08:11:17.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:16 smithi098 ceph-mon[31333]: pgmap v253: 97 pgs: 97 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:11:17.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:16 smithi098 ceph-mon[31333]: osdmap e85: 12 total, 12 up, 12 in 2024-09-19T08:11:19.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:18 smithi162 ceph-mon[35260]: pgmap v255: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-19T08:11:19.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:18 smithi139 ceph-mon[34874]: pgmap v255: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-19T08:11:19.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:18 smithi098 ceph-mon[31333]: pgmap v255: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-19T08:11:21.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:20 smithi162 ceph-mon[35260]: pgmap v256: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-09-19T08:11:21.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:20 smithi139 ceph-mon[34874]: pgmap v256: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-09-19T08:11:21.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:20 smithi098 ceph-mon[31333]: pgmap v256: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-09-19T08:11:23.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:22 smithi162 ceph-mon[35260]: pgmap v257: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 207 B/s wr, 10 op/s 2024-09-19T08:11:23.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:22 smithi139 ceph-mon[34874]: pgmap v257: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 207 B/s wr, 10 op/s 2024-09-19T08:11:23.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:22 smithi098 ceph-mon[31333]: pgmap v257: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 207 B/s wr, 10 op/s 2024-09-19T08:11:25.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:24 smithi162 ceph-mon[35260]: pgmap v258: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:11:25.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:24 smithi139 ceph-mon[34874]: pgmap v258: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:11:25.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:24 smithi098 ceph-mon[31333]: pgmap v258: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:11:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:26 smithi162 ceph-mon[35260]: pgmap v259: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:11:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:26 smithi139 ceph-mon[34874]: pgmap v259: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:11:27.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:26 smithi098 ceph-mon[31333]: pgmap v259: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:11:28.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:11:28.811 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:11:29.119 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:28 smithi098 ceph-mon[31333]: pgmap v260: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 92 B/s wr, 12 op/s 2024-09-19T08:11:29.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:28 smithi162 ceph-mon[35260]: pgmap v260: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 92 B/s wr, 12 op/s 2024-09-19T08:11:29.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:28 smithi139 ceph-mon[34874]: pgmap v260: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 92 B/s wr, 12 op/s 2024-09-19T08:11:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.429+0000 7f7f1cd01640 1 -- 172.21.15.98:0/1573528472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f18108290 msgr2=0x7f7f1810ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.429+0000 7f7f1cd01640 1 --2- 172.21.15.98:0/1573528472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f18108290 0x7f7f1810ec30 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f7f10009cd0 tx=0x7f7f1002a8e0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.429+0000 7f7f1cd01640 1 -- 172.21.15.98:0/1573528472 shutdown_connections 2024-09-19T08:11:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.429+0000 7f7f1cd01640 1 --2- 172.21.15.98:0/1573528472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f18108290 0x7f7f1810ec30 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.429+0000 7f7f1cd01640 1 --2- 172.21.15.98:0/1573528472 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f18103740 0x7f7f18107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.429+0000 7f7f1cd01640 1 --2- 172.21.15.98:0/1573528472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f18102d90 0x7f7f18103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.429+0000 7f7f1cd01640 1 -- 172.21.15.98:0/1573528472 >> 172.21.15.98:0/1573528472 conn(0x7f7f180fc7d0 msgr2=0x7f7f180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.429+0000 7f7f1cd01640 1 -- 172.21.15.98:0/1573528472 shutdown_connections 2024-09-19T08:11:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.429+0000 7f7f1cd01640 1 -- 172.21.15.98:0/1573528472 wait complete. 2024-09-19T08:11:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.430+0000 7f7f1cd01640 1 Processor -- start 2024-09-19T08:11:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.430+0000 7f7f1cd01640 1 -- start start 2024-09-19T08:11:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f1cd01640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f18102d90 0x7f7f181a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f1cd01640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f18103740 0x7f7f181a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f1cd01640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f18108290 0x7f7f1819a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f1cd01640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f18112090 con 0x7f7f18102d90 2024-09-19T08:11:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f1cd01640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7f18111f10 con 0x7f7f18108290 2024-09-19T08:11:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f1cd01640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f18112210 con 0x7f7f18103740 2024-09-19T08:11:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f15d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f18103740 0x7f7f181a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f1659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f18102d90 0x7f7f181a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f15d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f18103740 0x7f7f181a0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:51532/0 (socket says 172.21.15.98:51532) 2024-09-19T08:11:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f15d9b640 1 -- 172.21.15.98:0/3821191745 learned_addr learned my addr 172.21.15.98:0/3821191745 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f1659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f18102d90 0x7f7f181a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53706/0 (socket says 172.21.15.98:53706) 2024-09-19T08:11:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f16d9d640 1 --2- 172.21.15.98:0/3821191745 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f18108290 0x7f7f1819a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f1659c640 1 -- 172.21.15.98:0/3821191745 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f18103740 msgr2=0x7f7f181a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f1659c640 1 --2- 172.21.15.98:0/3821191745 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f18103740 0x7f7f181a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f1659c640 1 -- 172.21.15.98:0/3821191745 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f18108290 msgr2=0x7f7f1819a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f1659c640 1 --2- 172.21.15.98:0/3821191745 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f18108290 0x7f7f1819a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f1659c640 1 -- 172.21.15.98:0/3821191745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1819acd0 con 0x7f7f18102d90 2024-09-19T08:11:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.431+0000 7f7f15d9b640 1 --2- 172.21.15.98:0/3821191745 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f18103740 0x7f7f181a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:11:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.432+0000 7f7f1659c640 1 --2- 172.21.15.98:0/3821191745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f18102d90 0x7f7f181a03e0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800d720 tx=0x7f7f0800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.432+0000 7f7f077fe640 1 -- 172.21.15.98:0/3821191745 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08017070 con 0x7f7f18102d90 2024-09-19T08:11:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.432+0000 7f7f077fe640 1 -- 172.21.15.98:0/3821191745 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f0800de30 con 0x7f7f18102d90 2024-09-19T08:11:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.432+0000 7f7f1cd01640 1 -- 172.21.15.98:0/3821191745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f1819afc0 con 0x7f7f18102d90 2024-09-19T08:11:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.432+0000 7f7f077fe640 1 -- 172.21.15.98:0/3821191745 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f080047a0 con 0x7f7f18102d90 2024-09-19T08:11:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.432+0000 7f7f1cd01640 1 -- 172.21.15.98:0/3821191745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f181a72b0 con 0x7f7f18102d90 2024-09-19T08:11:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.435+0000 7f7f1f0da740 1 Processor -- start 2024-09-19T08:11:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.435+0000 7f7f1f0da740 1 -- start start 2024-09-19T08:11:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.435+0000 7f7f077fe640 1 -- 172.21.15.98:0/3821191745 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f7f080099d0 con 0x7f7f18102d90 2024-09-19T08:11:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.435+0000 7f7f1f0da740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c84a0 0x55770b2c8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.435+0000 7f7f1f0da740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c8dc0 0x55770b2d3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.435+0000 7f7f1f0da740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2d3680 0x55770b2d5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.435+0000 7f7f1f0da740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55770b22e2e0 con 0x55770b2c8dc0 2024-09-19T08:11:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.435+0000 7f7f1f0da740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55770b1d5ce0 con 0x55770b2d3680 2024-09-19T08:11:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.435+0000 7f7f1f0da740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55770b22a2a0 con 0x55770b2c84a0 2024-09-19T08:11:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.435+0000 7f7f16d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c84a0 0x55770b2c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.435+0000 7f7f1659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c8dc0 0x55770b2d3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.436+0000 7f7f16d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c84a0 0x55770b2c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:51548/0 (socket says 172.21.15.98:51548) 2024-09-19T08:11:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.436+0000 7f7f15d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2d3680 0x55770b2d5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.436+0000 7f7f16d9d640 1 -- 172.21.15.98:0/728425074 learned_addr learned my addr 172.21.15.98:0/728425074 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.436+0000 7f7f16d9d640 1 -- 172.21.15.98:0/728425074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2d3680 msgr2=0x55770b2d5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.436+0000 7f7f16d9d640 1 --2- 172.21.15.98:0/728425074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2d3680 0x55770b2d5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.436+0000 7f7f16d9d640 1 -- 172.21.15.98:0/728425074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c8dc0 msgr2=0x55770b2d3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.436+0000 7f7f16d9d640 1 --2- 172.21.15.98:0/728425074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c8dc0 0x55770b2d3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.436+0000 7f7f16d9d640 1 -- 172.21.15.98:0/728425074 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55770b1ba440 con 0x55770b2c84a0 2024-09-19T08:11:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.436+0000 7f7f16d9d640 1 --2- 172.21.15.98:0/728425074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c84a0 0x55770b2c8880 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7f10004ab0 tx=0x7f7f10009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2704912d1d9ca954 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.437+0000 7f7f04ff9640 1 -- 172.21.15.98:0/728425074 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10048070 con 0x55770b2c84a0 2024-09-19T08:11:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.437+0000 7f7f04ff9640 1 -- 172.21.15.98:0/728425074 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f10036070 con 0x55770b2c84a0 2024-09-19T08:11:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.437+0000 7f7f04ff9640 1 -- 172.21.15.98:0/728425074 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10004580 con 0x55770b2c84a0 2024-09-19T08:11:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.437+0000 7f7f077fe640 1 --2- 172.21.15.98:0/3821191745 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7f000a33b0 0x7f7f000a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.437+0000 7f7f1659c640 1 --2- 172.21.15.98:0/3821191745 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7f000a33b0 0x7f7f000a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.437+0000 7f7f077fe640 1 -- 172.21.15.98:0/3821191745 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f7f080cf930 con 0x7f7f18102d90 2024-09-19T08:11:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.438+0000 7f7f1f0da740 1 -- 172.21.15.98:0/728425074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c84a0 msgr2=0x55770b2c8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.438+0000 7f7f1659c640 1 --2- 172.21.15.98:0/3821191745 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7f000a33b0 0x7f7f000a5870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7f080cd130 tx=0x7f7f080d2f90 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.438+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/728425074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c84a0 0x55770b2c8880 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7f10004ab0 tx=0x7f7f10009a20 comp rx=0 tx=0).stop 2024-09-19T08:11:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.438+0000 7f7f1f0da740 1 -- 172.21.15.98:0/728425074 shutdown_connections 2024-09-19T08:11:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.438+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/728425074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2d3680 0x55770b2d5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.438+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/728425074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c8dc0 0x55770b2d3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.438+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/728425074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c84a0 0x55770b2c8880 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.438+0000 7f7f1f0da740 1 -- 172.21.15.98:0/728425074 >> 172.21.15.98:0/728425074 conn(0x55770b2c4a30 msgr2=0x55770b2c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.438+0000 7f7f1f0da740 1 -- 172.21.15.98:0/728425074 shutdown_connections 2024-09-19T08:11:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.438+0000 7f7f1f0da740 1 -- 172.21.15.98:0/728425074 wait complete. 2024-09-19T08:11:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.438+0000 7f7f1f0da740 1 -- start start 2024-09-19T08:11:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.438+0000 7f7f1f0da740 1 Processor -- start 2024-09-19T08:11:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f1f0da740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c84a0 0x55770b2d2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f1f0da740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c8dc0 0x55770b2d0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f1f0da740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2d3680 0x55770b2d07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f1f0da740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55770b2d62f0 con 0x55770b2c84a0 2024-09-19T08:11:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f1f0da740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55770b1e9050 con 0x55770b2d3680 2024-09-19T08:11:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f1f0da740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55770b2d6470 con 0x55770b2c8dc0 2024-09-19T08:11:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f1f0da740 10 client.0 _ll_register_callbacks cb 0x55770b229160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f15d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c8dc0 0x55770b2d0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f16d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c84a0 0x55770b2d2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f15d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c8dc0 0x55770b2d0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:51554/0 (socket says 172.21.15.98:51554) 2024-09-19T08:11:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f15d9b640 1 -- 172.21.15.98:0/2482998793 learned_addr learned my addr 172.21.15.98:0/2482998793 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f16d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c84a0 0x55770b2d2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53720/0 (socket says 172.21.15.98:53720) 2024-09-19T08:11:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f15d9b640 1 -- 172.21.15.98:0/2482998793 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2d3680 msgr2=0x55770b2d07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f15d9b640 1 --2- 172.21.15.98:0/2482998793 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2d3680 0x55770b2d07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f15d9b640 1 -- 172.21.15.98:0/2482998793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c84a0 msgr2=0x55770b2d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f15d9b640 1 --2- 172.21.15.98:0/2482998793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c84a0 0x55770b2d2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.439+0000 7f7f15d9b640 1 -- 172.21.15.98:0/2482998793 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55770af9f3c0 con 0x55770b2c8dc0 2024-09-19T08:11:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.440+0000 7f7f15d9b640 1 --2- 172.21.15.98:0/2482998793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c8dc0 0x55770b2d0270 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005fd0 tx=0x7f7f0c0039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb13891892d0fe42 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.440+0000 7f7efdffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-09-19T08:11:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.440+0000 7f7efdffb640 1 -- 172.21.15.98:0/2482998793 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c011710 con 0x55770b2c8dc0 2024-09-19T08:11:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.440+0000 7f7f1f0da740 1 -- 172.21.15.98:0/2482998793 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f10036070 con 0x55770b2c8dc0 2024-09-19T08:11:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.440+0000 7f7efdffb640 1 -- 172.21.15.98:0/2482998793 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f7f0c0118b0 con 0x55770b2c8dc0 2024-09-19T08:11:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.440+0000 7f7efdffb640 1 -- 172.21.15.98:0/2482998793 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7f0c011ab0 con 0x55770b2c8dc0 2024-09-19T08:11:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.440+0000 7f7f1f0da740 10 client.26338 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.440+0000 7f7f1f0da740 1 -- 172.21.15.98:0/2482998793 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55770b2e1d80 con 0x55770b2c8dc0 2024-09-19T08:11:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.440+0000 7f7efdffb640 1 -- 172.21.15.98:0/2482998793 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f10036070 con 0x55770b2c8dc0 2024-09-19T08:11:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.440+0000 7f7efdffb640 1 -- 172.21.15.98:0/2482998793 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7f0c010440 con 0x55770b2c8dc0 2024-09-19T08:11:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.441+0000 7f7f1f0da740 10 client.26338 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.441+0000 7f7f1f0da740 10 client.26338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:11:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.441+0000 7f7f1f0da740 20 client.26338 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.441+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/2482998793 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55770b2e3600 0x55770b2e5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.441+0000 7f7f1f0da740 4 client.26338 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:11:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.441+0000 7f7f1f0da740 1 -- 172.21.15.98:0/2482998793 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55770b23b850 con 0x55770b2e3600 2024-09-19T08:11:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.441+0000 7f7f16d9d640 1 --2- 172.21.15.98:0/2482998793 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55770b2e3600 0x55770b2e5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.441+0000 7f7efdffb640 10 client.26338 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.441+0000 7f7f16d9d640 1 --2- 172.21.15.98:0/2482998793 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55770b2e3600 0x55770b2e5a80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6d1ae92e4f51c55 server_cookie=3f4b76e5e9a8d4b5 in_seq=0 out_seq=0 2024-09-19T08:11:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.442+0000 7f7efdffb640 1 -- 172.21.15.98:0/2482998793 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55770b23b850 con 0x55770b2e3600 2024-09-19T08:11:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.442+0000 7f7efdffb640 10 client.26338 handle_command_reply: tid=0 2024-09-19T08:11:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.442+0000 7f7f1f0da740 1 client.26338 shutdown 2024-09-19T08:11:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.442+0000 7f7f1f0da740 1 -- 172.21.15.98:0/2482998793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c8dc0 msgr2=0x55770b2d0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.442+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/2482998793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c8dc0 0x55770b2d0270 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005fd0 tx=0x7f7f0c0039c0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.442+0000 7f7f1f0da740 1 -- 172.21.15.98:0/2482998793 shutdown_connections 2024-09-19T08:11:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.442+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/2482998793 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55770b2e3600 0x55770b2e5a80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.442+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/2482998793 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2d3680 0x55770b2d07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.442+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/2482998793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c8dc0 0x55770b2d0270 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.442+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/2482998793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c84a0 0x55770b2d2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.442+0000 7f7f1f0da740 1 -- 172.21.15.98:0/2482998793 >> 172.21.15.98:0/2482998793 conn(0x55770b2c4a30 msgr2=0x55770b2ca760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.442+0000 7f7efdffb640 0 client.26338 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.442+0000 7f7f1f0da740 1 -- 172.21.15.98:0/2482998793 shutdown_connections 2024-09-19T08:11:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.442+0000 7f7f1f0da740 1 -- 172.21.15.98:0/2482998793 wait complete. 2024-09-19T08:11:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.443+0000 7f7f1f0da740 20 client.26338 trim_cache size 0 max 16384 2024-09-19T08:11:29.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f1f0da740 1 Processor -- start 2024-09-19T08:11:29.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f1f0da740 1 -- start start 2024-09-19T08:11:29.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f1f0da740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c84a0 0x55770b2ceb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f1f0da740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c8dc0 0x55770b2d21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f1f0da740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2d3680 0x55770b2eb350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f1f0da740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f181afaa0 con 0x55770b2c84a0 2024-09-19T08:11:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f1f0da740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7f181b15e0 con 0x55770b2d3680 2024-09-19T08:11:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f1f0da740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f181b0170 con 0x55770b2c8dc0 2024-09-19T08:11:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f16d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c84a0 0x55770b2ceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f16d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c84a0 0x55770b2ceb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53722/0 (socket says 172.21.15.98:53722) 2024-09-19T08:11:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f16d9d640 1 -- 172.21.15.98:0/3089349351 learned_addr learned my addr 172.21.15.98:0/3089349351 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f15d9b640 1 --2- 172.21.15.98:0/3089349351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c8dc0 0x55770b2d21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f16d9d640 1 --2- 172.21.15.98:0/3089349351 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2d3680 0x55770b2eb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f16d9d640 1 -- 172.21.15.98:0/3089349351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c8dc0 msgr2=0x55770b2d21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f16d9d640 1 --2- 172.21.15.98:0/3089349351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c8dc0 0x55770b2d21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f16d9d640 1 -- 172.21.15.98:0/3089349351 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2d3680 msgr2=0x55770b2eb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f16d9d640 1 --2- 172.21.15.98:0/3089349351 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2d3680 0x55770b2eb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.460+0000 7f7f16d9d640 1 -- 172.21.15.98:0/3089349351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55770b2cf220 con 0x55770b2c84a0 2024-09-19T08:11:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.461+0000 7f7f15d9b640 1 --2- 172.21.15.98:0/3089349351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c8dc0 0x55770b2d21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:11:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.461+0000 7f7f16d9d640 1 --2- 172.21.15.98:0/3089349351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c84a0 0x55770b2ceb40 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f7f10009960 tx=0x7f7f100095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52fbd803dc57d8e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.461+0000 7f7efffff640 1 -- 172.21.15.98:0/3089349351 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10056070 con 0x55770b2c84a0 2024-09-19T08:11:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.461+0000 7f7efffff640 1 -- 172.21.15.98:0/3089349351 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f10036070 con 0x55770b2c84a0 2024-09-19T08:11:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.461+0000 7f7efffff640 1 -- 172.21.15.98:0/3089349351 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10048040 con 0x55770b2c84a0 2024-09-19T08:11:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 -- 172.21.15.98:0/3089349351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c84a0 msgr2=0x55770b2ceb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/3089349351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c84a0 0x55770b2ceb40 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f7f10009960 tx=0x7f7f100095c0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 -- 172.21.15.98:0/3089349351 shutdown_connections 2024-09-19T08:11:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/3089349351 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2d3680 0x55770b2eb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/3089349351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c8dc0 0x55770b2d21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/3089349351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2c84a0 0x55770b2ceb40 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 -- 172.21.15.98:0/3089349351 >> 172.21.15.98:0/3089349351 conn(0x55770b2c4a30 msgr2=0x55770b2d1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 -- 172.21.15.98:0/3089349351 shutdown_connections 2024-09-19T08:11:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 -- 172.21.15.98:0/3089349351 wait complete. 2024-09-19T08:11:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 -- start start 2024-09-19T08:11:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 Processor -- start 2024-09-19T08:11:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c84a0 0x55770b2db7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2c8dc0 0x55770b2dbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2d3680 0x55770b2d8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f181b0f10 con 0x55770b2d3680 2024-09-19T08:11:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7f181b0840 con 0x55770b2c8dc0 2024-09-19T08:11:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f181adaa0 con 0x55770b2c84a0 2024-09-19T08:11:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.462+0000 7f7f1f0da740 10 client.0 _ll_register_callbacks cb 0x7f7f0c00bc90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7f16d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2d3680 0x55770b2d8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7f16d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2d3680 0x55770b2d8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53734/0 (socket says 172.21.15.98:53734) 2024-09-19T08:11:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7f16d9d640 1 -- 172.21.15.98:0/385083714 learned_addr learned my addr 172.21.15.98:0/385083714 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7f16d9d640 1 --2- 172.21.15.98:0/385083714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c84a0 0x55770b2db7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7f15d9b640 1 --2- 172.21.15.98:0/385083714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2c8dc0 0x55770b2dbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7f16d9d640 1 -- 172.21.15.98:0/385083714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c84a0 msgr2=0x55770b2db7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7f16d9d640 1 --2- 172.21.15.98:0/385083714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c84a0 0x55770b2db7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7f16d9d640 1 -- 172.21.15.98:0/385083714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2c8dc0 msgr2=0x55770b2dbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7f16d9d640 1 --2- 172.21.15.98:0/385083714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2c8dc0 0x55770b2dbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7f16d9d640 1 -- 172.21.15.98:0/385083714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55770b2d3330 con 0x55770b2d3680 2024-09-19T08:11:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7f15d9b640 1 --2- 172.21.15.98:0/385083714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2c8dc0 0x55770b2dbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7f16d9d640 1 --2- 172.21.15.98:0/385083714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2d3680 0x55770b2d8c10 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f7f10005e30 tx=0x7f7f10005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea3816ec6830e6b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7efdffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:11:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7efdffb640 1 -- 172.21.15.98:0/385083714 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10048070 con 0x55770b2d3680 2024-09-19T08:11:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7efdffb640 1 -- 172.21.15.98:0/385083714 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f7f100043d0 con 0x55770b2d3680 2024-09-19T08:11:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7f1f0da740 1 -- 172.21.15.98:0/385083714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f10036070 con 0x55770b2d3680 2024-09-19T08:11:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.463+0000 7f7efdffb640 1 -- 172.21.15.98:0/385083714 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7f10056040 con 0x55770b2d3680 2024-09-19T08:11:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.464+0000 7f7f1f0da740 10 client.17412 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.464+0000 7f7f1f0da740 1 -- 172.21.15.98:0/385083714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55770b34dbe0 con 0x55770b2d3680 2024-09-19T08:11:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.464+0000 7f7efdffb640 1 -- 172.21.15.98:0/385083714 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f10036070 con 0x55770b2d3680 2024-09-19T08:11:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.464+0000 7f7efdffb640 1 -- 172.21.15.98:0/385083714 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7f10069030 con 0x55770b2d3680 2024-09-19T08:11:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.464+0000 7f7f1f0da740 10 client.17412 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.464+0000 7f7f1f0da740 10 client.17412 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:11:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.464+0000 7f7f1f0da740 20 client.17412 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.464+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/385083714 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55770b34de60 0x55770b350250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.464+0000 7f7f1f0da740 4 client.17412 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:11:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.464+0000 7f7f1f0da740 1 -- 172.21.15.98:0/385083714 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55770b235d60 con 0x55770b34de60 2024-09-19T08:11:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.464+0000 7f7f15d9b640 1 --2- 172.21.15.98:0/385083714 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55770b34de60 0x55770b350250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7f15d9b640 1 --2- 172.21.15.98:0/385083714 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55770b34de60 0x55770b350250 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b37c7e6d16b117ce server_cookie=29cf6a583960739f in_seq=0 out_seq=0 2024-09-19T08:11:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7efdffb640 10 client.17412 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7efdffb640 1 -- 172.21.15.98:0/385083714 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55770b235d60 con 0x55770b34de60 2024-09-19T08:11:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7efdffb640 10 client.17412 handle_command_reply: tid=0 2024-09-19T08:11:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7f1f0da740 1 client.17412 shutdown 2024-09-19T08:11:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7f1f0da740 1 -- 172.21.15.98:0/385083714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2d3680 msgr2=0x55770b2d8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/385083714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2d3680 0x55770b2d8c10 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f7f10005e30 tx=0x7f7f10005d80 comp rx=0 tx=0).stop 2024-09-19T08:11:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7f1f0da740 1 -- 172.21.15.98:0/385083714 shutdown_connections 2024-09-19T08:11:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/385083714 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55770b34de60 0x55770b350250 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7efdffb640 0 client.17412 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/385083714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55770b2d3680 0x55770b2d8c10 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/385083714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55770b2c8dc0 0x55770b2dbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7f1f0da740 1 --2- 172.21.15.98:0/385083714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55770b2c84a0 0x55770b2db7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7f1f0da740 1 -- 172.21.15.98:0/385083714 >> 172.21.15.98:0/385083714 conn(0x55770b2c4a30 msgr2=0x55770b2d22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7f1f0da740 1 -- 172.21.15.98:0/385083714 shutdown_connections 2024-09-19T08:11:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7f1f0da740 1 -- 172.21.15.98:0/385083714 wait complete. 2024-09-19T08:11:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.465+0000 7f7f1f0da740 20 client.17412 trim_cache size 0 max 16384 2024-09-19T08:11:29.473 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:11:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:11:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:11:29.473 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:11:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.466+0000 7f7f1cd01640 1 -- 172.21.15.98:0/3821191745 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7f000a33b0 msgr2=0x7f7f000a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.466+0000 7f7f1cd01640 1 --2- 172.21.15.98:0/3821191745 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7f000a33b0 0x7f7f000a5870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7f080cd130 tx=0x7f7f080d2f90 comp rx=0 tx=0).stop 2024-09-19T08:11:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.466+0000 7f7f1cd01640 1 -- 172.21.15.98:0/3821191745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f18102d90 msgr2=0x7f7f181a03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.466+0000 7f7f1cd01640 1 --2- 172.21.15.98:0/3821191745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f18102d90 0x7f7f181a03e0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800d720 tx=0x7f7f0800dbe0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.466+0000 7f7f1cd01640 1 -- 172.21.15.98:0/3821191745 shutdown_connections 2024-09-19T08:11:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.466+0000 7f7f1cd01640 1 --2- 172.21.15.98:0/3821191745 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7f000a33b0 0x7f7f000a5870 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.466+0000 7f7f1cd01640 1 --2- 172.21.15.98:0/3821191745 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f18108290 0x7f7f1819a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.466+0000 7f7f1cd01640 1 --2- 172.21.15.98:0/3821191745 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f18103740 0x7f7f181a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.466+0000 7f7f1cd01640 1 --2- 172.21.15.98:0/3821191745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f18102d90 0x7f7f181a03e0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.466+0000 7f7f1cd01640 1 -- 172.21.15.98:0/3821191745 >> 172.21.15.98:0/3821191745 conn(0x7f7f180fc7d0 msgr2=0x7f7f1810d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.466+0000 7f7f1cd01640 1 -- 172.21.15.98:0/3821191745 shutdown_connections 2024-09-19T08:11:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:29.466+0000 7f7f1cd01640 1 -- 172.21.15.98:0/3821191745 wait complete. 2024-09-19T08:11:30.292 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:11:30.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:11:30.541 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:11:30.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:30 smithi098 ceph-mon[31333]: pgmap v261: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:11:31.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.009+0000 7fe473355640 1 -- 172.21.15.98:0/3196561553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe46c102d30 msgr2=0x7fe46c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:31.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.009+0000 7fe473355640 1 --2- 172.21.15.98:0/3196561553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe46c102d30 0x7fe46c103110 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009cd0 tx=0x7fe45c02a8e0 comp rx=0 tx=0).stop 2024-09-19T08:11:31.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.009+0000 7fe473355640 1 -- 172.21.15.98:0/3196561553 shutdown_connections 2024-09-19T08:11:31.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.009+0000 7fe473355640 1 --2- 172.21.15.98:0/3196561553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe46c1082a0 0x7fe46c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:31.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.009+0000 7fe473355640 1 --2- 172.21.15.98:0/3196561553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe46c1036e0 0x7fe46c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:31.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.009+0000 7fe473355640 1 --2- 172.21.15.98:0/3196561553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe46c102d30 0x7fe46c103110 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:31.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.009+0000 7fe473355640 1 -- 172.21.15.98:0/3196561553 >> 172.21.15.98:0/3196561553 conn(0x7fe46c0fc7d0 msgr2=0x7fe46c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:31.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.009+0000 7fe473355640 1 -- 172.21.15.98:0/3196561553 shutdown_connections 2024-09-19T08:11:31.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.009+0000 7fe473355640 1 -- 172.21.15.98:0/3196561553 wait complete. 2024-09-19T08:11:31.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.009+0000 7fe473355640 1 Processor -- start 2024-09-19T08:11:31.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.009+0000 7fe473355640 1 -- start start 2024-09-19T08:11:31.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe473355640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe46c102d30 0x7fe46c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:31.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe473355640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe46c1036e0 0x7fe46c198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:31.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe473355640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe46c1082a0 0x7fe46c19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:31.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe473355640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe46c1120a0 con 0x7fe46c1036e0 2024-09-19T08:11:31.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe473355640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe46c111f20 con 0x7fe46c102d30 2024-09-19T08:11:31.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe473355640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe46c112220 con 0x7fe46c1082a0 2024-09-19T08:11:31.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe4708f0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe46c1036e0 0x7fe46c198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:31.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe4708f0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe46c1036e0 0x7fe46c198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53748/0 (socket says 172.21.15.98:53748) 2024-09-19T08:11:31.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe4708f0640 1 -- 172.21.15.98:0/3745702978 learned_addr learned my addr 172.21.15.98:0/3745702978 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:31.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe4710f1640 1 --2- 172.21.15.98:0/3745702978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe46c102d30 0x7fe46c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:31.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe4718f2640 1 --2- 172.21.15.98:0/3745702978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe46c1082a0 0x7fe46c19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:31.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe4708f0640 1 -- 172.21.15.98:0/3745702978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe46c1082a0 msgr2=0x7fe46c19ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:31.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe4708f0640 1 --2- 172.21.15.98:0/3745702978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe46c1082a0 0x7fe46c19ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:31.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe4708f0640 1 -- 172.21.15.98:0/3745702978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe46c102d30 msgr2=0x7fe46c1985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:31.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe4708f0640 1 --2- 172.21.15.98:0/3745702978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe46c102d30 0x7fe46c1985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:31.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe4708f0640 1 -- 172.21.15.98:0/3745702978 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe46c19d590 con 0x7fe46c1036e0 2024-09-19T08:11:31.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe4708f0640 1 --2- 172.21.15.98:0/3745702978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe46c1036e0 0x7fe46c198b20 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fe46000d750 tx=0x7fe46000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:31.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe46a7fc640 1 -- 172.21.15.98:0/3745702978 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe460017070 con 0x7fe46c1036e0 2024-09-19T08:11:31.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe46a7fc640 1 -- 172.21.15.98:0/3745702978 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe460005500 con 0x7fe46c1036e0 2024-09-19T08:11:31.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe46a7fc640 1 -- 172.21.15.98:0/3745702978 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe460004e00 con 0x7fe46c1036e0 2024-09-19T08:11:31.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.010+0000 7fe473355640 1 -- 172.21.15.98:0/3745702978 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe46c19d880 con 0x7fe46c1036e0 2024-09-19T08:11:31.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.011+0000 7fe473355640 1 -- 172.21.15.98:0/3745702978 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe46c1a5160 con 0x7fe46c1036e0 2024-09-19T08:11:31.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.011+0000 7fe473355640 1 -- 172.21.15.98:0/3745702978 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe46c06a850 con 0x7fe46c1036e0 2024-09-19T08:11:31.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.013+0000 7fe46a7fc640 1 -- 172.21.15.98:0/3745702978 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fe460004fa0 con 0x7fe46c1036e0 2024-09-19T08:11:31.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.014+0000 7fe46a7fc640 1 --2- 172.21.15.98:0/3745702978 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe4540a33b0 0x7fe4540a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:31.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.014+0000 7fe4710f1640 1 --2- 172.21.15.98:0/3745702978 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe4540a33b0 0x7fe4540a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:31.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.014+0000 7fe46a7fc640 1 -- 172.21.15.98:0/3745702978 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fe4600cfaa0 con 0x7fe46c1036e0 2024-09-19T08:11:31.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.014+0000 7fe4710f1640 1 --2- 172.21.15.98:0/3745702978 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe4540a33b0 0x7fe4540a5870 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe45c006010 tx=0x7fe45c009360 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:31.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.017+0000 7fe46a7fc640 1 -- 172.21.15.98:0/3745702978 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fe46009c480 con 0x7fe46c1036e0 2024-09-19T08:11:31.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:30 smithi162 ceph-mon[35260]: pgmap v261: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:11:31.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:30 smithi139 ceph-mon[34874]: pgmap v261: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:11:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.219+0000 7fe473355640 1 -- 172.21.15.98:0/3745702978 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe46c10a2b0 con 0x7fe46c1036e0 2024-09-19T08:11:31.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.219+0000 7fe46a7fc640 1 -- 172.21.15.98:0/3745702978 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11983 (secure 0 0 0) 0x7fe46009bd80 con 0x7fe46c1036e0 2024-09-19T08:11:31.219 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:11:31.220 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"btime":"2024-09-19T08:08:26:985720+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:08:26.555658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:11:31.220 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2024-09-19T08:11:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.221+0000 7fe473355640 1 -- 172.21.15.98:0/3745702978 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe4540a33b0 msgr2=0x7fe4540a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.221+0000 7fe473355640 1 --2- 172.21.15.98:0/3745702978 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe4540a33b0 0x7fe4540a5870 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe45c006010 tx=0x7fe45c009360 comp rx=0 tx=0).stop 2024-09-19T08:11:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.221+0000 7fe473355640 1 -- 172.21.15.98:0/3745702978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe46c1036e0 msgr2=0x7fe46c198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.221+0000 7fe473355640 1 --2- 172.21.15.98:0/3745702978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe46c1036e0 0x7fe46c198b20 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fe46000d750 tx=0x7fe46000dc10 comp rx=0 tx=0).stop 2024-09-19T08:11:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.221+0000 7fe473355640 1 -- 172.21.15.98:0/3745702978 shutdown_connections 2024-09-19T08:11:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.221+0000 7fe473355640 1 --2- 172.21.15.98:0/3745702978 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe4540a33b0 0x7fe4540a5870 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.221+0000 7fe473355640 1 --2- 172.21.15.98:0/3745702978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe46c1082a0 0x7fe46c19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.221+0000 7fe473355640 1 --2- 172.21.15.98:0/3745702978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe46c1036e0 0x7fe46c198b20 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.221+0000 7fe473355640 1 --2- 172.21.15.98:0/3745702978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe46c102d30 0x7fe46c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.221+0000 7fe473355640 1 -- 172.21.15.98:0/3745702978 >> 172.21.15.98:0/3745702978 conn(0x7fe46c0fc7d0 msgr2=0x7fe46c105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.222+0000 7fe473355640 1 -- 172.21.15.98:0/3745702978 shutdown_connections 2024-09-19T08:11:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:31.222+0000 7fe473355640 1 -- 172.21.15.98:0/3745702978 wait complete. 2024-09-19T08:11:31.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:11:32.176 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:11:32.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:31 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3745702978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:11:32.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:31 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3745702978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:11:32.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:31 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3745702978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:11:32.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.643+0000 7f21ac2df640 1 -- 172.21.15.98:0/855344267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21a4102790 msgr2=0x7f21a41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:32.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.643+0000 7f21ac2df640 1 --2- 172.21.15.98:0/855344267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21a4102790 0x7f21a41067e0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f21a0009900 tx=0x7f21a002a410 comp rx=0 tx=0).stop 2024-09-19T08:11:32.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.643+0000 7f21ac2df640 1 -- 172.21.15.98:0/855344267 shutdown_connections 2024-09-19T08:11:32.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.643+0000 7f21ac2df640 1 --2- 172.21.15.98:0/855344267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21a4106f10 0x7f21a410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.643+0000 7f21ac2df640 1 --2- 172.21.15.98:0/855344267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21a4102790 0x7f21a41067e0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.643+0000 7f21ac2df640 1 --2- 172.21.15.98:0/855344267 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21a4101de0 0x7f21a41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.643+0000 7f21ac2df640 1 -- 172.21.15.98:0/855344267 >> 172.21.15.98:0/855344267 conn(0x7f21a40fd8c0 msgr2=0x7f21a40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.643+0000 7f21ac2df640 1 -- 172.21.15.98:0/855344267 shutdown_connections 2024-09-19T08:11:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.643+0000 7f21ac2df640 1 -- 172.21.15.98:0/855344267 wait complete. 2024-09-19T08:11:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21ac2df640 1 Processor -- start 2024-09-19T08:11:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21ac2df640 1 -- start start 2024-09-19T08:11:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21ac2df640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21a4101de0 0x7f21a41a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21ac2df640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21a4102790 0x7f21a41a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21ac2df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21a4106f10 0x7f21a419a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21ac2df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21a4112010 con 0x7f21a4106f10 2024-09-19T08:11:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21ac2df640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f21a4111e90 con 0x7f21a4102790 2024-09-19T08:11:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21ac2df640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f21a4112190 con 0x7f21a4101de0 2024-09-19T08:11:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21aa87c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21a4106f10 0x7f21a419a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21aa87c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21a4106f10 0x7f21a419a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53766/0 (socket says 172.21.15.98:53766) 2024-09-19T08:11:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21aa87c640 1 -- 172.21.15.98:0/1880626628 learned_addr learned my addr 172.21.15.98:0/1880626628 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21a987a640 1 --2- 172.21.15.98:0/1880626628 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21a4102790 0x7f21a41a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21aa07b640 1 --2- 172.21.15.98:0/1880626628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21a4101de0 0x7f21a41a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21aa87c640 1 -- 172.21.15.98:0/1880626628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21a4101de0 msgr2=0x7f21a41a0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21aa87c640 1 --2- 172.21.15.98:0/1880626628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21a4101de0 0x7f21a41a0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21aa87c640 1 -- 172.21.15.98:0/1880626628 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21a4102790 msgr2=0x7f21a41a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21aa87c640 1 --2- 172.21.15.98:0/1880626628 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21a4102790 0x7f21a41a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.644+0000 7f21aa87c640 1 -- 172.21.15.98:0/1880626628 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a419ad30 con 0x7f21a4106f10 2024-09-19T08:11:32.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.645+0000 7f21aa87c640 1 --2- 172.21.15.98:0/1880626628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21a4106f10 0x7f21a419a600 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f219c00b970 tx=0x7f219c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:32.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.645+0000 7f21977fe640 1 -- 172.21.15.98:0/1880626628 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c019070 con 0x7f21a4106f10 2024-09-19T08:11:32.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.645+0000 7f21977fe640 1 -- 172.21.15.98:0/1880626628 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f219c014440 con 0x7f21a4106f10 2024-09-19T08:11:32.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.645+0000 7f21ac2df640 1 -- 172.21.15.98:0/1880626628 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21a419b020 con 0x7f21a4106f10 2024-09-19T08:11:32.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.645+0000 7f21977fe640 1 -- 172.21.15.98:0/1880626628 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c0075d0 con 0x7f21a4106f10 2024-09-19T08:11:32.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.645+0000 7f21ac2df640 1 -- 172.21.15.98:0/1880626628 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21a41a7310 con 0x7f21a4106f10 2024-09-19T08:11:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.647+0000 7f21977fe640 1 -- 172.21.15.98:0/1880626628 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f219c01c070 con 0x7f21a4106f10 2024-09-19T08:11:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.647+0000 7f21ae6b8740 1 Processor -- start 2024-09-19T08:11:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21ae6b8740 1 -- start start 2024-09-19T08:11:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21ae6b8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62338b40 0x55fb62337090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21ae6b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb623375d0 0x55fb62337a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21ae6b8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb62342370 0x55fb62344760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21ae6b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fb6229d2f0 con 0x55fb623375d0 2024-09-19T08:11:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21ae6b8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fb62244cf0 con 0x55fb62342370 2024-09-19T08:11:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21ae6b8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fb622992b0 con 0x55fb62338b40 2024-09-19T08:11:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21aa07b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb623375d0 0x55fb62337a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21aa07b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb623375d0 0x55fb62337a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53774/0 (socket says 172.21.15.98:53774) 2024-09-19T08:11:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21aa07b640 1 -- 172.21.15.98:0/3549092487 learned_addr learned my addr 172.21.15.98:0/3549092487 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21a987a640 1 --2- 172.21.15.98:0/3549092487 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62338b40 0x55fb62337090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21a987a640 1 --2- 172.21.15.98:0/3549092487 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb62342370 0x55fb62344760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21977fe640 1 --2- 172.21.15.98:0/1880626628 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f21900a3430 0x7f21900a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21a987a640 1 -- 172.21.15.98:0/3549092487 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb62342370 msgr2=0x55fb62344760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21a987a640 1 --2- 172.21.15.98:0/3549092487 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb62342370 0x55fb62344760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21a987a640 1 -- 172.21.15.98:0/3549092487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb623375d0 msgr2=0x55fb62337a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:32.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21a987a640 1 --2- 172.21.15.98:0/3549092487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb623375d0 0x55fb62337a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21a987a640 1 -- 172.21.15.98:0/3549092487 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb622293a0 con 0x55fb62338b40 2024-09-19T08:11:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.648+0000 7f21aa07b640 1 --2- 172.21.15.98:0/3549092487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb623375d0 0x55fb62337a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:11:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.649+0000 7f21977fe640 1 -- 172.21.15.98:0/1880626628 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f219c0cf130 con 0x7f21a4106f10 2024-09-19T08:11:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.649+0000 7f21aa87c640 1 --2- 172.21.15.98:0/1880626628 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f21900a3430 0x7f21900a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.649+0000 7f21a987a640 1 --2- 172.21.15.98:0/3549092487 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62338b40 0x55fb62337090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f21a0005940 tx=0x7f21a00314d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c268c3bdce9a90f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.649+0000 7f2194dea640 1 -- 172.21.15.98:0/3549092487 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0035070 con 0x55fb62338b40 2024-09-19T08:11:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.649+0000 7f21aa87c640 1 --2- 172.21.15.98:0/1880626628 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f21900a3430 0x7f21900a58f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f219c09b2d0 tx=0x7f219c0cf430 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.649+0000 7f2194dea640 1 -- 172.21.15.98:0/3549092487 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21a0031a10 con 0x55fb62338b40 2024-09-19T08:11:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.649+0000 7f2194dea640 1 -- 172.21.15.98:0/3549092487 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a003d450 con 0x55fb62338b40 2024-09-19T08:11:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.650+0000 7f21ae6b8740 1 -- 172.21.15.98:0/3549092487 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62338b40 msgr2=0x55fb62337090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.650+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/3549092487 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62338b40 0x55fb62337090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f21a0005940 tx=0x7f21a00314d0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.650+0000 7f21ae6b8740 1 -- 172.21.15.98:0/3549092487 shutdown_connections 2024-09-19T08:11:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.650+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/3549092487 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb62342370 0x55fb62344760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.650+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/3549092487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb623375d0 0x55fb62337a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.650+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/3549092487 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62338b40 0x55fb62337090 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.650+0000 7f21ae6b8740 1 -- 172.21.15.98:0/3549092487 >> 172.21.15.98:0/3549092487 conn(0x55fb62300310 msgr2=0x55fb62333c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.650+0000 7f21ae6b8740 1 -- 172.21.15.98:0/3549092487 shutdown_connections 2024-09-19T08:11:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.650+0000 7f21ae6b8740 1 -- 172.21.15.98:0/3549092487 wait complete. 2024-09-19T08:11:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.650+0000 7f21ae6b8740 1 -- start start 2024-09-19T08:11:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.650+0000 7f21ae6b8740 1 Processor -- start 2024-09-19T08:11:32.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21ae6b8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb623375d0 0x55fb62341db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21ae6b8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62338b40 0x55fb6233f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21ae6b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62342370 0x55fb6233f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21ae6b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fb62345010 con 0x55fb62342370 2024-09-19T08:11:32.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21ae6b8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fb62258060 con 0x55fb623375d0 2024-09-19T08:11:32.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21ae6b8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fb62345190 con 0x55fb62338b40 2024-09-19T08:11:32.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21ae6b8740 10 client.0 _ll_register_callbacks cb 0x55fb61fab8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:32.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21a987a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62342370 0x55fb6233f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21a987a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62342370 0x55fb6233f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53786/0 (socket says 172.21.15.98:53786) 2024-09-19T08:11:32.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21a987a640 1 -- 172.21.15.98:0/2302838871 learned_addr learned my addr 172.21.15.98:0/2302838871 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:32.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21aa07b640 1 --2- 172.21.15.98:0/2302838871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62338b40 0x55fb6233f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21a987a640 1 --2- 172.21.15.98:0/2302838871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb623375d0 0x55fb62341db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21a987a640 1 -- 172.21.15.98:0/2302838871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62338b40 msgr2=0x55fb6233f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21a987a640 1 --2- 172.21.15.98:0/2302838871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62338b40 0x55fb6233f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21a987a640 1 -- 172.21.15.98:0/2302838871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb623375d0 msgr2=0x55fb62341db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21a987a640 1 --2- 172.21.15.98:0/2302838871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb623375d0 0x55fb62341db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21a987a640 1 -- 172.21.15.98:0/2302838871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb6200e3c0 con 0x55fb62342370 2024-09-19T08:11:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21a987a640 1 --2- 172.21.15.98:0/2302838871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62342370 0x55fb6233f7f0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f21a00318a0 tx=0x7f21a0041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b004c2b55109403 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f218dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:11:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f218dffb640 1 -- 172.21.15.98:0/2302838871 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0035030 con 0x55fb62342370 2024-09-19T08:11:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f218dffb640 1 -- 172.21.15.98:0/2302838871 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f21a0003800 con 0x55fb62342370 2024-09-19T08:11:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.651+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2302838871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fb622aa860 con 0x55fb62342370 2024-09-19T08:11:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.652+0000 7f218dffb640 1 -- 172.21.15.98:0/2302838871 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f21a0003a20 con 0x55fb62342370 2024-09-19T08:11:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.652+0000 7f21ae6b8740 10 client.17430 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.652+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2302838871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fb62350b00 con 0x55fb62342370 2024-09-19T08:11:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.652+0000 7f218dffb640 1 -- 172.21.15.98:0/2302838871 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fb622aa860 con 0x55fb62342370 2024-09-19T08:11:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.652+0000 7f218dffb640 1 -- 172.21.15.98:0/2302838871 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f21a00047a0 con 0x55fb62342370 2024-09-19T08:11:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.652+0000 7f21ae6b8740 10 client.17430 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.652+0000 7f21ae6b8740 10 client.17430 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:11:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.652+0000 7f21ae6b8740 20 client.17430 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.652+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/2302838871 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55fb62351e50 0x55fb62354240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.652+0000 7f21ae6b8740 4 client.17430 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:11:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.652+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2302838871 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb622a4d70 con 0x55fb62351e50 2024-09-19T08:11:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.652+0000 7f21aa07b640 1 --2- 172.21.15.98:0/2302838871 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55fb62351e50 0x55fb62354240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.653+0000 7f218dffb640 10 client.17430 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:11:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.653+0000 7f21aa07b640 1 --2- 172.21.15.98:0/2302838871 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55fb62351e50 0x55fb62354240 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=96dd86e8f814a0a5 server_cookie=2fd99cbbb01bf514 in_seq=0 out_seq=0 2024-09-19T08:11:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.653+0000 7f218dffb640 1 -- 172.21.15.98:0/2302838871 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fb622a4d70 con 0x55fb62351e50 2024-09-19T08:11:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.653+0000 7f218dffb640 10 client.17430 handle_command_reply: tid=0 2024-09-19T08:11:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.653+0000 7f21ae6b8740 1 client.17430 shutdown 2024-09-19T08:11:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.653+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2302838871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62342370 msgr2=0x55fb6233f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.654+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/2302838871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62342370 0x55fb6233f7f0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f21a00318a0 tx=0x7f21a0041040 comp rx=0 tx=0).stop 2024-09-19T08:11:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.654+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2302838871 shutdown_connections 2024-09-19T08:11:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.654+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/2302838871 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55fb62351e50 0x55fb62354240 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.654+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/2302838871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62342370 0x55fb6233f7f0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.654+0000 7f218dffb640 0 client.17430 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:11:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.654+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/2302838871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62338b40 0x55fb6233f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.654+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/2302838871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb623375d0 0x55fb62341db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.654+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2302838871 >> 172.21.15.98:0/2302838871 conn(0x55fb62300310 msgr2=0x55fb623399c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.654+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2302838871 shutdown_connections 2024-09-19T08:11:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.654+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2302838871 wait complete. 2024-09-19T08:11:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.654+0000 7f21ae6b8740 20 client.17430 trim_cache size 0 max 16384 2024-09-19T08:11:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.666+0000 7f21ae6b8740 1 Processor -- start 2024-09-19T08:11:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.666+0000 7f21ae6b8740 1 -- start start 2024-09-19T08:11:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21ae6b8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb623375d0 0x55fb6233e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21ae6b8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb62338b40 0x55fb62357260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21ae6b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62342370 0x55fb623577a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21ae6b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41afba0 con 0x55fb62342370 2024-09-19T08:11:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21ae6b8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41b16e0 con 0x55fb62338b40 2024-09-19T08:11:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21ae6b8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41b0270 con 0x55fb623375d0 2024-09-19T08:11:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21a987a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62342370 0x55fb623577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21a987a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62342370 0x55fb623577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53802/0 (socket says 172.21.15.98:53802) 2024-09-19T08:11:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21a987a640 1 -- 172.21.15.98:0/1136337407 learned_addr learned my addr 172.21.15.98:0/1136337407 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21a987a640 1 --2- 172.21.15.98:0/1136337407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb623375d0 0x55fb6233e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21aa07b640 1 --2- 172.21.15.98:0/1136337407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb62338b40 0x55fb62357260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21a987a640 1 -- 172.21.15.98:0/1136337407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb623375d0 msgr2=0x55fb6233e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21a987a640 1 --2- 172.21.15.98:0/1136337407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb623375d0 0x55fb6233e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21a987a640 1 -- 172.21.15.98:0/1136337407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb62338b40 msgr2=0x55fb62357260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21a987a640 1 --2- 172.21.15.98:0/1136337407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb62338b40 0x55fb62357260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21a987a640 1 -- 172.21.15.98:0/1136337407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb62333ed0 con 0x55fb62342370 2024-09-19T08:11:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21aa07b640 1 --2- 172.21.15.98:0/1136337407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb62338b40 0x55fb62357260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:11:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f21a987a640 1 --2- 172.21.15.98:0/1136337407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62342370 0x55fb623577a0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f21a00319e0 tx=0x7f21a0005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4d1d4d844b7c5c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f218ffff640 1 -- 172.21.15.98:0/1136337407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0035070 con 0x55fb62342370 2024-09-19T08:11:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f218ffff640 1 -- 172.21.15.98:0/1136337407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21a0062040 con 0x55fb62342370 2024-09-19T08:11:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.667+0000 7f218ffff640 1 -- 172.21.15.98:0/1136337407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0061040 con 0x55fb62342370 2024-09-19T08:11:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.668+0000 7f21ae6b8740 1 -- 172.21.15.98:0/1136337407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62342370 msgr2=0x55fb623577a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.668+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/1136337407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62342370 0x55fb623577a0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f21a00319e0 tx=0x7f21a0005a20 comp rx=0 tx=0).stop 2024-09-19T08:11:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.668+0000 7f21ae6b8740 1 -- 172.21.15.98:0/1136337407 shutdown_connections 2024-09-19T08:11:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.668+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/1136337407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62342370 0x55fb623577a0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.668+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/1136337407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb62338b40 0x55fb62357260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.668+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/1136337407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb623375d0 0x55fb6233e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.668+0000 7f21ae6b8740 1 -- 172.21.15.98:0/1136337407 >> 172.21.15.98:0/1136337407 conn(0x55fb62300310 msgr2=0x55fb6233d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.668+0000 7f21ae6b8740 1 -- 172.21.15.98:0/1136337407 shutdown_connections 2024-09-19T08:11:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.668+0000 7f21ae6b8740 1 -- 172.21.15.98:0/1136337407 wait complete. 2024-09-19T08:11:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.668+0000 7f21ae6b8740 1 -- start start 2024-09-19T08:11:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.668+0000 7f21ae6b8740 1 Processor -- start 2024-09-19T08:11:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21ae6b8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb623375d0 0x55fb62354ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21ae6b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62338b40 0x55fb62355530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21ae6b8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62342370 0x55fb62355a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21ae6b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41b1010 con 0x55fb62338b40 2024-09-19T08:11:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21ae6b8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41b0940 con 0x55fb623375d0 2024-09-19T08:11:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21ae6b8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41adba0 con 0x55fb62342370 2024-09-19T08:11:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21ae6b8740 10 client.0 _ll_register_callbacks cb 0x7f21980094a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21aa07b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62338b40 0x55fb62355530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21aa07b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62338b40 0x55fb62355530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53806/0 (socket says 172.21.15.98:53806) 2024-09-19T08:11:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21aa07b640 1 -- 172.21.15.98:0/2682779060 learned_addr learned my addr 172.21.15.98:0/2682779060 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21a987a640 1 --2- 172.21.15.98:0/2682779060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62342370 0x55fb62355a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21a987a640 1 --2- 172.21.15.98:0/2682779060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb623375d0 0x55fb62354ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21aa07b640 1 -- 172.21.15.98:0/2682779060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62342370 msgr2=0x55fb62355a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21aa07b640 1 --2- 172.21.15.98:0/2682779060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62342370 0x55fb62355a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21aa07b640 1 -- 172.21.15.98:0/2682779060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb623375d0 msgr2=0x55fb62354ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21aa07b640 1 --2- 172.21.15.98:0/2682779060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb623375d0 0x55fb62354ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21aa07b640 1 -- 172.21.15.98:0/2682779060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb623bb790 con 0x55fb62338b40 2024-09-19T08:11:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21a987a640 1 --2- 172.21.15.98:0/2682779060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62342370 0x55fb62355a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21aa07b640 1 --2- 172.21.15.98:0/2682779060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62338b40 0x55fb62355530 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f2198006a70 tx=0x7f2198017440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8a578016c78db7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f218dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:11:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f218dffb640 1 -- 172.21.15.98:0/2682779060 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198017dd0 con 0x55fb62338b40 2024-09-19T08:11:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f218dffb640 1 -- 172.21.15.98:0/2682779060 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f21980170d0 con 0x55fb62338b40 2024-09-19T08:11:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2682779060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21a0062040 con 0x55fb62338b40 2024-09-19T08:11:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f218dffb640 1 -- 172.21.15.98:0/2682779060 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f219802d050 con 0x55fb62338b40 2024-09-19T08:11:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.669+0000 7f21ae6b8740 10 client.17436 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.670+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2682779060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fb623bca70 con 0x55fb62338b40 2024-09-19T08:11:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.670+0000 7f218dffb640 1 -- 172.21.15.98:0/2682779060 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21a0062040 con 0x55fb62338b40 2024-09-19T08:11:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.670+0000 7f218dffb640 1 -- 172.21.15.98:0/2682779060 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f21980066d0 con 0x55fb62338b40 2024-09-19T08:11:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.670+0000 7f21ae6b8740 10 client.17436 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.670+0000 7f21ae6b8740 10 client.17436 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:11:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.670+0000 7f21ae6b8740 20 client.17436 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.670+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/2682779060 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55fb623bd220 0x55fb623bf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.670+0000 7f21ae6b8740 4 client.17436 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:11:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.670+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2682779060 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fb62221170 con 0x55fb623bd220 2024-09-19T08:11:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.670+0000 7f21a987a640 1 --2- 172.21.15.98:0/2682779060 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55fb623bd220 0x55fb623bf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f218dffb640 10 client.17436 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:11:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f21a987a640 1 --2- 172.21.15.98:0/2682779060 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55fb623bd220 0x55fb623bf640 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fbfa6f547d604087 server_cookie=e3abe10ac3db2834 in_seq=0 out_seq=0 2024-09-19T08:11:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f218dffb640 1 -- 172.21.15.98:0/2682779060 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fb62221170 con 0x55fb623bd220 2024-09-19T08:11:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f218dffb640 10 client.17436 handle_command_reply: tid=0 2024-09-19T08:11:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f21ae6b8740 1 client.17436 shutdown 2024-09-19T08:11:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2682779060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62338b40 msgr2=0x55fb62355530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/2682779060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62338b40 0x55fb62355530 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f2198006a70 tx=0x7f2198017440 comp rx=0 tx=0).stop 2024-09-19T08:11:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2682779060 shutdown_connections 2024-09-19T08:11:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/2682779060 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55fb623bd220 0x55fb623bf640 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/2682779060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb62342370 0x55fb62355a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/2682779060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb62338b40 0x55fb62355530 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f218dffb640 0 client.17436 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:11:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f21ae6b8740 1 --2- 172.21.15.98:0/2682779060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb623375d0 0x55fb62354ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2682779060 >> 172.21.15.98:0/2682779060 conn(0x55fb62300310 msgr2=0x55fb623408c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2682779060 shutdown_connections 2024-09-19T08:11:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f21ae6b8740 1 -- 172.21.15.98:0/2682779060 wait complete. 2024-09-19T08:11:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.671+0000 7f21ae6b8740 20 client.17436 trim_cache size 0 max 16384 2024-09-19T08:11:32.680 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:11:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.672+0000 7f21ac2df640 1 -- 172.21.15.98:0/1880626628 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f21900a3430 msgr2=0x7f21900a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.672+0000 7f21ac2df640 1 --2- 172.21.15.98:0/1880626628 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f21900a3430 0x7f21900a58f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f219c09b2d0 tx=0x7f219c0cf430 comp rx=0 tx=0).stop 2024-09-19T08:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.672+0000 7f21ac2df640 1 -- 172.21.15.98:0/1880626628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21a4106f10 msgr2=0x7f21a419a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.672+0000 7f21ac2df640 1 --2- 172.21.15.98:0/1880626628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21a4106f10 0x7f21a419a600 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f219c00b970 tx=0x7f219c00bd70 comp rx=0 tx=0).stop 2024-09-19T08:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.672+0000 7f21ac2df640 1 -- 172.21.15.98:0/1880626628 shutdown_connections 2024-09-19T08:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.672+0000 7f21ac2df640 1 --2- 172.21.15.98:0/1880626628 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f21900a3430 0x7f21900a58f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.672+0000 7f21ac2df640 1 --2- 172.21.15.98:0/1880626628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21a4106f10 0x7f21a419a600 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.672+0000 7f21ac2df640 1 --2- 172.21.15.98:0/1880626628 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21a4102790 0x7f21a41a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.672+0000 7f21ac2df640 1 --2- 172.21.15.98:0/1880626628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21a4101de0 0x7f21a41a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.672+0000 7f21ac2df640 1 -- 172.21.15.98:0/1880626628 >> 172.21.15.98:0/1880626628 conn(0x7f21a40fd8c0 msgr2=0x7f21a40fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.672+0000 7f21ac2df640 1 -- 172.21.15.98:0/1880626628 shutdown_connections 2024-09-19T08:11:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:32.672+0000 7f21ac2df640 1 -- 172.21.15.98:0/1880626628 wait complete. 2024-09-19T08:11:33.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:32 smithi162 ceph-mon[35260]: pgmap v262: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:11:33.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:32 smithi139 ceph-mon[34874]: pgmap v262: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:11:33.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:32 smithi098 ceph-mon[31333]: pgmap v262: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:11:33.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:11:33.572 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:11:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.119+0000 7fa928c16640 1 -- 172.21.15.98:0/887253951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa924102790 msgr2=0x7fa9241067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.119+0000 7fa928c16640 1 --2- 172.21.15.98:0/887253951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa924102790 0x7fa9241067e0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fa918009900 tx=0x7fa91802a3f0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.119+0000 7fa928c16640 1 -- 172.21.15.98:0/887253951 shutdown_connections 2024-09-19T08:11:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.119+0000 7fa928c16640 1 --2- 172.21.15.98:0/887253951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa924106f10 0x7fa92410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.119+0000 7fa928c16640 1 --2- 172.21.15.98:0/887253951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa924102790 0x7fa9241067e0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.119+0000 7fa928c16640 1 --2- 172.21.15.98:0/887253951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa924101de0 0x7fa9241021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.119+0000 7fa928c16640 1 -- 172.21.15.98:0/887253951 >> 172.21.15.98:0/887253951 conn(0x7fa9240fd8c0 msgr2=0x7fa9240ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.120+0000 7fa928c16640 1 -- 172.21.15.98:0/887253951 shutdown_connections 2024-09-19T08:11:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.120+0000 7fa928c16640 1 -- 172.21.15.98:0/887253951 wait complete. 2024-09-19T08:11:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.120+0000 7fa928c16640 1 Processor -- start 2024-09-19T08:11:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.120+0000 7fa928c16640 1 -- start start 2024-09-19T08:11:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.120+0000 7fa928c16640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa924101de0 0x7fa92419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.120+0000 7fa928c16640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa924102790 0x7fa92419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa928c16640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa924106f10 0x7fa92419eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa928c16640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa924111f70 con 0x7fa924106f10 2024-09-19T08:11:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa928c16640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa924111df0 con 0x7fa924102790 2024-09-19T08:11:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa928c16640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241120f0 con 0x7fa924101de0 2024-09-19T08:11:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa922d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa924106f10 0x7fa92419eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa922d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa924106f10 0x7fa92419eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53826/0 (socket says 172.21.15.98:53826) 2024-09-19T08:11:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa922d9d640 1 -- 172.21.15.98:0/3073753630 learned_addr learned my addr 172.21.15.98:0/3073753630 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa92259c640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa924101de0 0x7fa92419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa921d9b640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa924102790 0x7fa92419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa922d9d640 1 -- 172.21.15.98:0/3073753630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa924101de0 msgr2=0x7fa92419a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa922d9d640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa924101de0 0x7fa92419a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa922d9d640 1 -- 172.21.15.98:0/3073753630 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa924102790 msgr2=0x7fa92419ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa922d9d640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa924102790 0x7fa92419ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa922d9d640 1 -- 172.21.15.98:0/3073753630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa92419f590 con 0x7fa924106f10 2024-09-19T08:11:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa92259c640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa924101de0 0x7fa92419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa921d9b640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa924102790 0x7fa92419ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa922d9d640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa924106f10 0x7fa92419eeb0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00cce0 tx=0x7fa91c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa9137fe640 1 -- 172.21.15.98:0/3073753630 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91c00f450 con 0x7fa924106f10 2024-09-19T08:11:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa928c16640 1 -- 172.21.15.98:0/3073753630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa92419f880 con 0x7fa924106f10 2024-09-19T08:11:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa9137fe640 1 -- 172.21.15.98:0/3073753630 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa91c00fad0 con 0x7fa924106f10 2024-09-19T08:11:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa9137fe640 1 -- 172.21.15.98:0/3073753630 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91c018c10 con 0x7fa924106f10 2024-09-19T08:11:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.121+0000 7fa928c16640 1 -- 172.21.15.98:0/3073753630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9241a7160 con 0x7fa924106f10 2024-09-19T08:11:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.124+0000 7fa9137fe640 1 -- 172.21.15.98:0/3073753630 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fa91c00f5f0 con 0x7fa924106f10 2024-09-19T08:11:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.124+0000 7fa92afef740 1 Processor -- start 2024-09-19T08:11:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.124+0000 7fa92afef740 1 -- start start 2024-09-19T08:11:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.124+0000 7fa92afef740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008cb40 0x5641a008b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.124+0000 7fa92afef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a008b5d0 0x5641a008ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.124+0000 7fa92afef740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a0096370 0x5641a0098760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.124+0000 7fa92afef740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56419fff12f0 con 0x5641a0096370 2024-09-19T08:11:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.124+0000 7fa92afef740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56419ff98cf0 con 0x5641a008cb40 2024-09-19T08:11:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.124+0000 7fa92afef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56419ffed2b0 con 0x5641a008b5d0 2024-09-19T08:11:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.124+0000 7fa921d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a0096370 0x5641a0098760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.124+0000 7fa921d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a0096370 0x5641a0098760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53838/0 (socket says 172.21.15.98:53838) 2024-09-19T08:11:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.124+0000 7fa921d9b640 1 -- 172.21.15.98:0/2962239883 learned_addr learned my addr 172.21.15.98:0/2962239883 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.125+0000 7fa92259c640 1 --2- 172.21.15.98:0/2962239883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a008b5d0 0x5641a008ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.125+0000 7fa921d9b640 1 --2- 172.21.15.98:0/2962239883 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008cb40 0x5641a008b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.125+0000 7fa9137fe640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa90c0a33b0 0x7fa90c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.125+0000 7fa921d9b640 1 -- 172.21.15.98:0/2962239883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a008b5d0 msgr2=0x5641a008ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.125+0000 7fa921d9b640 1 --2- 172.21.15.98:0/2962239883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a008b5d0 0x5641a008ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.125+0000 7fa921d9b640 1 -- 172.21.15.98:0/2962239883 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008cb40 msgr2=0x5641a008b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.125+0000 7fa921d9b640 1 --2- 172.21.15.98:0/2962239883 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008cb40 0x5641a008b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.125+0000 7fa921d9b640 1 -- 172.21.15.98:0/2962239883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56419ff7d3a0 con 0x5641a0096370 2024-09-19T08:11:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.125+0000 7fa922d9d640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa90c0a33b0 0x7fa90c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.125+0000 7fa921d9b640 1 --2- 172.21.15.98:0/2962239883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a0096370 0x5641a0098760 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fa9180096c0 tx=0x7fa918031480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dfc9c990f2841b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.125+0000 7fa9137fe640 1 -- 172.21.15.98:0/3073753630 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fa91c0d0410 con 0x7fa924106f10 2024-09-19T08:11:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.125+0000 7fa910ff9640 1 -- 172.21.15.98:0/2962239883 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa918035040 con 0x5641a0096370 2024-09-19T08:11:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.125+0000 7fa910ff9640 1 -- 172.21.15.98:0/2962239883 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa918031800 con 0x5641a0096370 2024-09-19T08:11:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.125+0000 7fa910ff9640 1 -- 172.21.15.98:0/2962239883 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9180047a0 con 0x5641a0096370 2024-09-19T08:11:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.126+0000 7fa92afef740 1 -- 172.21.15.98:0/2962239883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a0096370 msgr2=0x5641a0098760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.126+0000 7fa92afef740 1 --2- 172.21.15.98:0/2962239883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a0096370 0x5641a0098760 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fa9180096c0 tx=0x7fa918031480 comp rx=0 tx=0).stop 2024-09-19T08:11:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.126+0000 7fa922d9d640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa90c0a33b0 0x7fa90c0a5870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0cdc10 tx=0x7fa91c098a10 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.126+0000 7fa92afef740 1 -- 172.21.15.98:0/2962239883 shutdown_connections 2024-09-19T08:11:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.126+0000 7fa92afef740 1 --2- 172.21.15.98:0/2962239883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a0096370 0x5641a0098760 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.126+0000 7fa92afef740 1 --2- 172.21.15.98:0/2962239883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a008b5d0 0x5641a008ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.126+0000 7fa92afef740 1 --2- 172.21.15.98:0/2962239883 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008cb40 0x5641a008b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.126+0000 7fa92afef740 1 -- 172.21.15.98:0/2962239883 >> 172.21.15.98:0/2962239883 conn(0x5641a0054310 msgr2=0x5641a0087c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.126+0000 7fa92afef740 1 -- 172.21.15.98:0/2962239883 shutdown_connections 2024-09-19T08:11:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.126+0000 7fa92afef740 1 -- 172.21.15.98:0/2962239883 wait complete. 2024-09-19T08:11:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.126+0000 7fa92afef740 1 -- start start 2024-09-19T08:11:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.126+0000 7fa92afef740 1 Processor -- start 2024-09-19T08:11:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa92afef740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a008b5d0 0x5641a0095db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa92afef740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008cb40 0x5641a0093060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa92afef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a0096370 0x5641a00937f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa92afef740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5641a0099010 con 0x5641a008b5d0 2024-09-19T08:11:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa92afef740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56419ffac060 con 0x5641a008cb40 2024-09-19T08:11:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa92afef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5641a0099190 con 0x5641a0096370 2024-09-19T08:11:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa92afef740 10 client.0 _ll_register_callbacks cb 0x56419fcff8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa921d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a008b5d0 0x5641a0095db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa921d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a008b5d0 0x5641a0095db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53854/0 (socket says 172.21.15.98:53854) 2024-09-19T08:11:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa921d9b640 1 -- 172.21.15.98:0/4035363799 learned_addr learned my addr 172.21.15.98:0/4035363799 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa92259c640 1 --2- 172.21.15.98:0/4035363799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008cb40 0x5641a0093060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa921d9b640 1 --2- 172.21.15.98:0/4035363799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a0096370 0x5641a00937f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa921d9b640 1 -- 172.21.15.98:0/4035363799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a0096370 msgr2=0x5641a00937f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa921d9b640 1 --2- 172.21.15.98:0/4035363799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a0096370 0x5641a00937f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa921d9b640 1 -- 172.21.15.98:0/4035363799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008cb40 msgr2=0x5641a0093060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa921d9b640 1 --2- 172.21.15.98:0/4035363799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008cb40 0x5641a0093060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa921d9b640 1 -- 172.21.15.98:0/4035363799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56419fd623c0 con 0x5641a008b5d0 2024-09-19T08:11:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa92259c640 1 --2- 172.21.15.98:0/4035363799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008cb40 0x5641a0093060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa921d9b640 1 --2- 172.21.15.98:0/4035363799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a008b5d0 0x5641a0095db0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fa918032fd0 tx=0x7fa918005490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86334fc50cdf4f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa909ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:11:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa909ffb640 1 -- 172.21.15.98:0/4035363799 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa918050020 con 0x5641a008b5d0 2024-09-19T08:11:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa909ffb640 1 -- 172.21.15.98:0/4035363799 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fa9180047a0 con 0x5641a008b5d0 2024-09-19T08:11:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.127+0000 7fa92afef740 1 -- 172.21.15.98:0/4035363799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa918031800 con 0x5641a008b5d0 2024-09-19T08:11:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.128+0000 7fa909ffb640 1 -- 172.21.15.98:0/4035363799 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa91805b050 con 0x5641a008b5d0 2024-09-19T08:11:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.128+0000 7fa92afef740 10 client.17448 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.128+0000 7fa92afef740 1 -- 172.21.15.98:0/4035363799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5641a00a4b00 con 0x5641a008b5d0 2024-09-19T08:11:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.128+0000 7fa909ffb640 1 -- 172.21.15.98:0/4035363799 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa918031800 con 0x5641a008b5d0 2024-09-19T08:11:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.128+0000 7fa909ffb640 1 -- 172.21.15.98:0/4035363799 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa91803a040 con 0x5641a008b5d0 2024-09-19T08:11:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.128+0000 7fa92afef740 10 client.17448 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.128+0000 7fa92afef740 10 client.17448 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:11:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.128+0000 7fa92afef740 20 client.17448 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.128+0000 7fa92afef740 1 --2- 172.21.15.98:0/4035363799 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5641a00a5e50 0x5641a00a8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.128+0000 7fa92afef740 4 client.17448 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:11:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.128+0000 7fa92afef740 1 -- 172.21.15.98:0/4035363799 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56419fffe860 con 0x5641a00a5e50 2024-09-19T08:11:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.128+0000 7fa92259c640 1 --2- 172.21.15.98:0/4035363799 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5641a00a5e50 0x5641a00a8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.129+0000 7fa909ffb640 10 client.17448 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:11:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.129+0000 7fa92259c640 1 --2- 172.21.15.98:0/4035363799 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5641a00a5e50 0x5641a00a8240 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ff982133bafb0d58 server_cookie=6e74d53636e1fd27 in_seq=0 out_seq=0 2024-09-19T08:11:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.129+0000 7fa909ffb640 1 -- 172.21.15.98:0/4035363799 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56419fffe860 con 0x5641a00a5e50 2024-09-19T08:11:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.129+0000 7fa909ffb640 10 client.17448 handle_command_reply: tid=0 2024-09-19T08:11:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.129+0000 7fa92afef740 1 client.17448 shutdown 2024-09-19T08:11:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.130+0000 7fa92afef740 1 -- 172.21.15.98:0/4035363799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a008b5d0 msgr2=0x5641a0095db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.130+0000 7fa92afef740 1 --2- 172.21.15.98:0/4035363799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a008b5d0 0x5641a0095db0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fa918032fd0 tx=0x7fa918005490 comp rx=0 tx=0).stop 2024-09-19T08:11:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.130+0000 7fa92afef740 1 -- 172.21.15.98:0/4035363799 shutdown_connections 2024-09-19T08:11:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.130+0000 7fa92afef740 1 --2- 172.21.15.98:0/4035363799 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5641a00a5e50 0x5641a00a8240 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.130+0000 7fa92afef740 1 --2- 172.21.15.98:0/4035363799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a0096370 0x5641a00937f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.130+0000 7fa92afef740 1 --2- 172.21.15.98:0/4035363799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008cb40 0x5641a0093060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.130+0000 7fa909ffb640 0 client.17448 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:11:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.130+0000 7fa92afef740 1 --2- 172.21.15.98:0/4035363799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a008b5d0 0x5641a0095db0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.130+0000 7fa92afef740 1 -- 172.21.15.98:0/4035363799 >> 172.21.15.98:0/4035363799 conn(0x5641a0054310 msgr2=0x5641a008d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.130+0000 7fa92afef740 1 -- 172.21.15.98:0/4035363799 shutdown_connections 2024-09-19T08:11:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.130+0000 7fa92afef740 1 -- 172.21.15.98:0/4035363799 wait complete. 2024-09-19T08:11:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.130+0000 7fa92afef740 20 client.17448 trim_cache size 0 max 16384 2024-09-19T08:11:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.142+0000 7fa92afef740 1 Processor -- start 2024-09-19T08:11:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.142+0000 7fa92afef740 1 -- start start 2024-09-19T08:11:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.142+0000 7fa92afef740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008b5d0 0x5641a0087df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.142+0000 7fa92afef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a008cb40 0x5641a0092920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.142+0000 7fa92afef740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a0096370 0x5641a00ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa92afef740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241af8d0 con 0x5641a0096370 2024-09-19T08:11:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa92afef740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241b1410 con 0x5641a008b5d0 2024-09-19T08:11:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa92afef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241affa0 con 0x5641a008cb40 2024-09-19T08:11:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa921d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a0096370 0x5641a00ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa921d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a0096370 0x5641a00ad7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53856/0 (socket says 172.21.15.98:53856) 2024-09-19T08:11:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa921d9b640 1 -- 172.21.15.98:0/2589671500 learned_addr learned my addr 172.21.15.98:0/2589671500 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa921d9b640 1 --2- 172.21.15.98:0/2589671500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008b5d0 0x5641a0087df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa92259c640 1 --2- 172.21.15.98:0/2589671500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a008cb40 0x5641a0092920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa921d9b640 1 -- 172.21.15.98:0/2589671500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a008cb40 msgr2=0x5641a0092920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa921d9b640 1 --2- 172.21.15.98:0/2589671500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a008cb40 0x5641a0092920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa921d9b640 1 -- 172.21.15.98:0/2589671500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008b5d0 msgr2=0x5641a0087df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa921d9b640 1 --2- 172.21.15.98:0/2589671500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008b5d0 0x5641a0087df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa921d9b640 1 -- 172.21.15.98:0/2589671500 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641a00915b0 con 0x5641a0096370 2024-09-19T08:11:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa921d9b640 1 --2- 172.21.15.98:0/2589671500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a0096370 0x5641a00ad7e0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fa918009900 tx=0x7fa918032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200efa456e664b20 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa90bfff640 1 -- 172.21.15.98:0/2589671500 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa918035040 con 0x5641a0096370 2024-09-19T08:11:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa90bfff640 1 -- 172.21.15.98:0/2589671500 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa918004480 con 0x5641a0096370 2024-09-19T08:11:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.143+0000 7fa90bfff640 1 -- 172.21.15.98:0/2589671500 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa918031e10 con 0x5641a0096370 2024-09-19T08:11:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.144+0000 7fa92afef740 1 -- 172.21.15.98:0/2589671500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a0096370 msgr2=0x5641a00ad7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.144+0000 7fa92afef740 1 --2- 172.21.15.98:0/2589671500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a0096370 0x5641a00ad7e0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fa918009900 tx=0x7fa918032f20 comp rx=0 tx=0).stop 2024-09-19T08:11:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.144+0000 7fa92afef740 1 -- 172.21.15.98:0/2589671500 shutdown_connections 2024-09-19T08:11:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.144+0000 7fa92afef740 1 --2- 172.21.15.98:0/2589671500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a0096370 0x5641a00ad7e0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.144+0000 7fa92afef740 1 --2- 172.21.15.98:0/2589671500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a008cb40 0x5641a0092920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.144+0000 7fa92afef740 1 --2- 172.21.15.98:0/2589671500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008b5d0 0x5641a0087df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.144+0000 7fa92afef740 1 -- 172.21.15.98:0/2589671500 >> 172.21.15.98:0/2589671500 conn(0x5641a0054310 msgr2=0x5641a00876e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.144+0000 7fa92afef740 1 -- 172.21.15.98:0/2589671500 shutdown_connections 2024-09-19T08:11:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.144+0000 7fa92afef740 1 -- 172.21.15.98:0/2589671500 wait complete. 2024-09-19T08:11:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.144+0000 7fa92afef740 1 -- start start 2024-09-19T08:11:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.144+0000 7fa92afef740 1 Processor -- start 2024-09-19T08:11:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92afef740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008b5d0 0x5641a009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92afef740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a008cb40 0x5641a0099d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92afef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a0096370 0x5641a009a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92afef740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241b0d40 con 0x5641a008cb40 2024-09-19T08:11:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92afef740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241b0670 con 0x5641a008b5d0 2024-09-19T08:11:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92afef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241ad8d0 con 0x5641a0096370 2024-09-19T08:11:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92afef740 10 client.0 _ll_register_callbacks cb 0x7fa91400c390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a008cb40 0x5641a0099d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a008cb40 0x5641a0099d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53862/0 (socket says 172.21.15.98:53862) 2024-09-19T08:11:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92259c640 1 -- 172.21.15.98:0/1198984113 learned_addr learned my addr 172.21.15.98:0/1198984113 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa921d9b640 1 --2- 172.21.15.98:0/1198984113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008b5d0 0x5641a009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92259c640 1 -- 172.21.15.98:0/1198984113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a0096370 msgr2=0x5641a009a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa921d9b640 1 --2- 172.21.15.98:0/1198984113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a0096370 0x5641a009a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92259c640 1 --2- 172.21.15.98:0/1198984113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a0096370 0x5641a009a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92259c640 1 -- 172.21.15.98:0/1198984113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008b5d0 msgr2=0x5641a009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92259c640 1 --2- 172.21.15.98:0/1198984113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008b5d0 0x5641a009ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92259c640 1 -- 172.21.15.98:0/1198984113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641a009a9a0 con 0x5641a008cb40 2024-09-19T08:11:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa921d9b640 1 --2- 172.21.15.98:0/1198984113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a0096370 0x5641a009a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:11:34.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92259c640 1 --2- 172.21.15.98:0/1198984113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a008cb40 0x5641a0099d70 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fa914004700 tx=0x7fa914010af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690f0cfd1200ec24 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:34.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa909ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:11:34.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa909ffb640 1 -- 172.21.15.98:0/1198984113 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91401c970 con 0x5641a008cb40 2024-09-19T08:11:34.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa909ffb640 1 -- 172.21.15.98:0/1198984113 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fa91401cb10 con 0x5641a008cb40 2024-09-19T08:11:34.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.145+0000 7fa92afef740 1 -- 172.21.15.98:0/1198984113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa918004480 con 0x5641a008cb40 2024-09-19T08:11:34.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.146+0000 7fa909ffb640 1 -- 172.21.15.98:0/1198984113 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa914005020 con 0x5641a008cb40 2024-09-19T08:11:34.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.146+0000 7fa92afef740 10 client.17454 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:34.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.146+0000 7fa92afef740 1 -- 172.21.15.98:0/1198984113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5641a0118f10 con 0x5641a008cb40 2024-09-19T08:11:34.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.146+0000 7fa909ffb640 1 -- 172.21.15.98:0/1198984113 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa918004480 con 0x5641a008cb40 2024-09-19T08:11:34.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.146+0000 7fa909ffb640 1 -- 172.21.15.98:0/1198984113 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa914023070 con 0x5641a008cb40 2024-09-19T08:11:34.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.146+0000 7fa92afef740 10 client.17454 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:34.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.146+0000 7fa92afef740 10 client.17454 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:11:34.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.146+0000 7fa92afef740 20 client.17454 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:34.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.146+0000 7fa92afef740 1 --2- 172.21.15.98:0/1198984113 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5641a0119710 0x5641a011bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:34.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.146+0000 7fa92afef740 4 client.17454 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:11:34.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.146+0000 7fa92afef740 1 -- 172.21.15.98:0/1198984113 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56419fff8d70 con 0x5641a0119710 2024-09-19T08:11:34.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.146+0000 7fa921d9b640 1 --2- 172.21.15.98:0/1198984113 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5641a0119710 0x5641a011bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:34.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa909ffb640 10 client.17454 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:11:34.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa921d9b640 1 --2- 172.21.15.98:0/1198984113 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5641a0119710 0x5641a011bb30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fd1c69b734534b4f server_cookie=f2452fff6f94f2f in_seq=0 out_seq=0 2024-09-19T08:11:34.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa909ffb640 1 -- 172.21.15.98:0/1198984113 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56419fff8d70 con 0x5641a0119710 2024-09-19T08:11:34.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa909ffb640 10 client.17454 handle_command_reply: tid=0 2024-09-19T08:11:34.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa92afef740 1 client.17454 shutdown 2024-09-19T08:11:34.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa92afef740 1 -- 172.21.15.98:0/1198984113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a008cb40 msgr2=0x5641a0099d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:34.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa92afef740 1 --2- 172.21.15.98:0/1198984113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a008cb40 0x5641a0099d70 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fa914004700 tx=0x7fa914010af0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa92afef740 1 -- 172.21.15.98:0/1198984113 shutdown_connections 2024-09-19T08:11:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa92afef740 1 --2- 172.21.15.98:0/1198984113 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5641a0119710 0x5641a011bb30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa92afef740 1 --2- 172.21.15.98:0/1198984113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641a0096370 0x5641a009a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa92afef740 1 --2- 172.21.15.98:0/1198984113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5641a008cb40 0x5641a0099d70 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa92afef740 1 --2- 172.21.15.98:0/1198984113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5641a008b5d0 0x5641a009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa92afef740 1 -- 172.21.15.98:0/1198984113 >> 172.21.15.98:0/1198984113 conn(0x5641a0054310 msgr2=0x5641a00a1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa909ffb640 0 client.17454 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:11:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa92afef740 1 -- 172.21.15.98:0/1198984113 shutdown_connections 2024-09-19T08:11:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa92afef740 1 -- 172.21.15.98:0/1198984113 wait complete. 2024-09-19T08:11:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.147+0000 7fa92afef740 20 client.17454 trim_cache size 0 max 16384 2024-09-19T08:11:34.153 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:11:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.148+0000 7fa928c16640 1 -- 172.21.15.98:0/3073753630 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa90c0a33b0 msgr2=0x7fa90c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.148+0000 7fa928c16640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa90c0a33b0 0x7fa90c0a5870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0cdc10 tx=0x7fa91c098a10 comp rx=0 tx=0).stop 2024-09-19T08:11:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.148+0000 7fa928c16640 1 -- 172.21.15.98:0/3073753630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa924106f10 msgr2=0x7fa92419eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.148+0000 7fa928c16640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa924106f10 0x7fa92419eeb0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00cce0 tx=0x7fa91c00a620 comp rx=0 tx=0).stop 2024-09-19T08:11:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.148+0000 7fa928c16640 1 -- 172.21.15.98:0/3073753630 shutdown_connections 2024-09-19T08:11:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.148+0000 7fa928c16640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa90c0a33b0 0x7fa90c0a5870 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.148+0000 7fa928c16640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa924106f10 0x7fa92419eeb0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.148+0000 7fa928c16640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa924102790 0x7fa92419ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.148+0000 7fa928c16640 1 --2- 172.21.15.98:0/3073753630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa924101de0 0x7fa92419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.148+0000 7fa928c16640 1 -- 172.21.15.98:0/3073753630 >> 172.21.15.98:0/3073753630 conn(0x7fa9240fd8c0 msgr2=0x7fa9240ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.148+0000 7fa928c16640 1 -- 172.21.15.98:0/3073753630 shutdown_connections 2024-09-19T08:11:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:34.148+0000 7fa928c16640 1 -- 172.21.15.98:0/3073753630 wait complete. 2024-09-19T08:11:35.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:11:35.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:34 smithi139 ceph-mon[34874]: pgmap v263: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:11:35.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:34 smithi098 ceph-mon[31333]: pgmap v263: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:11:35.252 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:11:35.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:34 smithi162 ceph-mon[35260]: pgmap v263: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:11:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.719+0000 7f3d3a18e640 1 -- 172.21.15.98:0/3775253307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d34102da0 msgr2=0x7f3d34103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.719+0000 7f3d3a18e640 1 --2- 172.21.15.98:0/3775253307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d34102da0 0x7f3d34103180 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009ca0 tx=0x7f3d2402a950 comp rx=0 tx=0).stop 2024-09-19T08:11:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.720+0000 7f3d3a18e640 1 -- 172.21.15.98:0/3775253307 shutdown_connections 2024-09-19T08:11:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.720+0000 7f3d3a18e640 1 --2- 172.21.15.98:0/3775253307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d3410c180 0x7f3d3410e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.720+0000 7f3d3a18e640 1 --2- 172.21.15.98:0/3775253307 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d341036c0 0x7f3d3410bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.720+0000 7f3d3a18e640 1 --2- 172.21.15.98:0/3775253307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d34102da0 0x7f3d34103180 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.720+0000 7f3d3a18e640 1 -- 172.21.15.98:0/3775253307 >> 172.21.15.98:0/3775253307 conn(0x7f3d340fc7a0 msgr2=0x7f3d340febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.720+0000 7f3d3a18e640 1 -- 172.21.15.98:0/3775253307 shutdown_connections 2024-09-19T08:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.720+0000 7f3d3a18e640 1 -- 172.21.15.98:0/3775253307 wait complete. 2024-09-19T08:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.720+0000 7f3d3a18e640 1 Processor -- start 2024-09-19T08:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.720+0000 7f3d3a18e640 1 -- start start 2024-09-19T08:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.720+0000 7f3d3a18e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d34102da0 0x7f3d34100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d3a18e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d341036c0 0x7f3d34100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d3a18e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d3410c180 0x7f3d34101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d3a18e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3d34111f60 con 0x7f3d3410c180 2024-09-19T08:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d3a18e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3d34111de0 con 0x7f3d34102da0 2024-09-19T08:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d3a18e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3d341120e0 con 0x7f3d341036c0 2024-09-19T08:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d3410c180 0x7f3d34101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d3410c180 0x7f3d34101480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53876/0 (socket says 172.21.15.98:53876) 2024-09-19T08:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d33fff640 1 -- 172.21.15.98:0/2821609992 learned_addr learned my addr 172.21.15.98:0/2821609992 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d337fe640 1 --2- 172.21.15.98:0/2821609992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d34102da0 0x7f3d34100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/2821609992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d341036c0 0x7f3d34100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d33fff640 1 -- 172.21.15.98:0/2821609992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d341036c0 msgr2=0x7f3d34100f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d33fff640 1 --2- 172.21.15.98:0/2821609992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d341036c0 0x7f3d34100f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d33fff640 1 -- 172.21.15.98:0/2821609992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d34102da0 msgr2=0x7f3d34100a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d33fff640 1 --2- 172.21.15.98:0/2821609992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d34102da0 0x7f3d34100a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d33fff640 1 -- 172.21.15.98:0/2821609992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d341a68e0 con 0x7f3d3410c180 2024-09-19T08:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d33fff640 1 --2- 172.21.15.98:0/2821609992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d3410c180 0x7f3d34101480 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00d750 tx=0x7f3d2c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d30ff9640 1 -- 172.21.15.98:0/2821609992 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c017070 con 0x7f3d3410c180 2024-09-19T08:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d30ff9640 1 -- 172.21.15.98:0/2821609992 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d2c005500 con 0x7f3d3410c180 2024-09-19T08:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d3a18e640 1 -- 172.21.15.98:0/2821609992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d341a6bd0 con 0x7f3d3410c180 2024-09-19T08:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d30ff9640 1 -- 172.21.15.98:0/2821609992 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c004d90 con 0x7f3d3410c180 2024-09-19T08:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.721+0000 7f3d3a18e640 1 -- 172.21.15.98:0/2821609992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d341a7110 con 0x7f3d3410c180 2024-09-19T08:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.723+0000 7f3d30ff9640 1 -- 172.21.15.98:0/2821609992 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f3d2c004f30 con 0x7f3d3410c180 2024-09-19T08:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d3c567740 1 Processor -- start 2024-09-19T08:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d3c567740 1 -- start start 2024-09-19T08:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d3c567740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d3c567740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4ad5d0 0x5604ba4ada10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d3c567740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4b8370 0x5604ba4ba760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d3c567740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5604ba4132f0 con 0x5604ba4aeb40 2024-09-19T08:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d3c567740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5604ba3bacf0 con 0x5604ba4b8370 2024-09-19T08:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d3c567740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5604ba40f2b0 con 0x5604ba4ad5d0 2024-09-19T08:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d30ff9640 1 --2- 172.21.15.98:0/2821609992 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3d1c0a33b0 0x7f3d1c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d32ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d32ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4ad090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53888/0 (socket says 172.21.15.98:53888) 2024-09-19T08:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d32ffd640 1 -- 172.21.15.98:0/2132920940 learned_addr learned my addr 172.21.15.98:0/2132920940 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d337fe640 1 --2- 172.21.15.98:0/2132920940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4ad5d0 0x5604ba4ada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d33fff640 1 --2- 172.21.15.98:0/2821609992 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3d1c0a33b0 0x7f3d1c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d32ffd640 1 -- 172.21.15.98:0/2132920940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4ad5d0 msgr2=0x5604ba4ada10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/2132920940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4ad5d0 0x5604ba4ada10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d32ffd640 1 -- 172.21.15.98:0/2132920940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4b8370 msgr2=0x5604ba4ba760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.724+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/2132920940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4b8370 0x5604ba4ba760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.725+0000 7f3d32ffd640 1 -- 172.21.15.98:0/2132920940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604ba39f3a0 con 0x5604ba4aeb40 2024-09-19T08:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.725+0000 7f3d337fe640 1 --2- 172.21.15.98:0/2132920940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4ad5d0 0x5604ba4ada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.725+0000 7f3d30ff9640 1 -- 172.21.15.98:0/2821609992 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f3d2c0cf620 con 0x7f3d3410c180 2024-09-19T08:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.725+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/2132920940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4ad090 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f3d28005d90 tx=0x7f3d28010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d24e9ac104099fa server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.725+0000 7f3d33fff640 1 --2- 172.21.15.98:0/2821609992 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3d1c0a33b0 0x7f3d1c0a5870 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c0cce20 tx=0x7f3d2c097cd0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.725+0000 7f3d19ffb640 1 -- 172.21.15.98:0/2132920940 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d28016070 con 0x5604ba4aeb40 2024-09-19T08:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.725+0000 7f3d19ffb640 1 -- 172.21.15.98:0/2132920940 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d280047a0 con 0x5604ba4aeb40 2024-09-19T08:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.725+0000 7f3d19ffb640 1 -- 172.21.15.98:0/2132920940 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d28012840 con 0x5604ba4aeb40 2024-09-19T08:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.726+0000 7f3d3c567740 1 -- 172.21.15.98:0/2132920940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 msgr2=0x5604ba4ad090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.726+0000 7f3d3c567740 1 --2- 172.21.15.98:0/2132920940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4ad090 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f3d28005d90 tx=0x7f3d28010740 comp rx=0 tx=0).stop 2024-09-19T08:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.726+0000 7f3d3c567740 1 -- 172.21.15.98:0/2132920940 shutdown_connections 2024-09-19T08:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.726+0000 7f3d3c567740 1 --2- 172.21.15.98:0/2132920940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4b8370 0x5604ba4ba760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.726+0000 7f3d3c567740 1 --2- 172.21.15.98:0/2132920940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4ad5d0 0x5604ba4ada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.726+0000 7f3d3c567740 1 --2- 172.21.15.98:0/2132920940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4ad090 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.726+0000 7f3d3c567740 1 -- 172.21.15.98:0/2132920940 >> 172.21.15.98:0/2132920940 conn(0x5604ba476310 msgr2=0x5604ba4a9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.726+0000 7f3d3c567740 1 -- 172.21.15.98:0/2132920940 shutdown_connections 2024-09-19T08:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.726+0000 7f3d3c567740 1 -- 172.21.15.98:0/2132920940 wait complete. 2024-09-19T08:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.726+0000 7f3d3c567740 1 -- start start 2024-09-19T08:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.726+0000 7f3d3c567740 1 Processor -- start 2024-09-19T08:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d3c567740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4ad5d0 0x5604ba4b7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d3c567740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d3c567740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4b8370 0x5604ba4b5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d3c567740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5604ba4bb070 con 0x5604ba4aeb40 2024-09-19T08:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d3c567740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5604ba3ce060 con 0x5604ba4ad5d0 2024-09-19T08:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d3c567740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5604ba4bb1f0 con 0x5604ba4b8370 2024-09-19T08:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d3c567740 10 client.0 _ll_register_callbacks cb 0x5604ba1218c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d337fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d337fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53890/0 (socket says 172.21.15.98:53890) 2024-09-19T08:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d337fe640 1 -- 172.21.15.98:0/1184554561 learned_addr learned my addr 172.21.15.98:0/1184554561 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/1184554561 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4ad5d0 0x5604ba4b7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/1184554561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4b8370 0x5604ba4b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d337fe640 1 -- 172.21.15.98:0/1184554561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4b8370 msgr2=0x5604ba4b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d337fe640 1 --2- 172.21.15.98:0/1184554561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4b8370 0x5604ba4b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d337fe640 1 -- 172.21.15.98:0/1184554561 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4ad5d0 msgr2=0x5604ba4b7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d337fe640 1 --2- 172.21.15.98:0/1184554561 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4ad5d0 0x5604ba4b7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d337fe640 1 -- 172.21.15.98:0/1184554561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604ba1843c0 con 0x5604ba4aeb40 2024-09-19T08:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/1184554561 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4ad5d0 0x5604ba4b7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d337fe640 1 --2- 172.21.15.98:0/1184554561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b4fc0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009ca0 tx=0x7f3d2402f3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb46f0c20fce4ce server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d12ffd640 1 -- 172.21.15.98:0/1184554561 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24049060 con 0x5604ba4aeb40 2024-09-19T08:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d12ffd640 1 -- 172.21.15.98:0/1184554561 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f3d24003b70 con 0x5604ba4aeb40 2024-09-19T08:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.727+0000 7f3d3c567740 1 -- 172.21.15.98:0/1184554561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d280047a0 con 0x5604ba4aeb40 2024-09-19T08:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.728+0000 7f3d12ffd640 1 -- 172.21.15.98:0/1184554561 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3d24034040 con 0x5604ba4aeb40 2024-09-19T08:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.728+0000 7f3d3c567740 10 client.17466 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.728+0000 7f3d3c567740 1 -- 172.21.15.98:0/1184554561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5604ba4c6af0 con 0x5604ba4aeb40 2024-09-19T08:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.728+0000 7f3d12ffd640 1 -- 172.21.15.98:0/1184554561 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d280047a0 con 0x5604ba4aeb40 2024-09-19T08:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.728+0000 7f3d12ffd640 1 -- 172.21.15.98:0/1184554561 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3d2404d4f0 con 0x5604ba4aeb40 2024-09-19T08:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.728+0000 7f3d3c567740 10 client.17466 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.728+0000 7f3d3c567740 10 client.17466 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.728+0000 7f3d3c567740 20 client.17466 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.728+0000 7f3d3c567740 1 --2- 172.21.15.98:0/1184554561 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5604ba4c8000 0x5604ba4ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.728+0000 7f3d3c567740 4 client.17466 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.728+0000 7f3d3c567740 1 -- 172.21.15.98:0/1184554561 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604ba420860 con 0x5604ba4c8000 2024-09-19T08:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.728+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/1184554561 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5604ba4c8000 0x5604ba4ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.729+0000 7f3d12ffd640 10 client.17466 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.729+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/1184554561 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5604ba4c8000 0x5604ba4ca420 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2bf3ec5d8676dbe9 server_cookie=893ae0d1cc12895d in_seq=0 out_seq=0 2024-09-19T08:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.729+0000 7f3d12ffd640 1 -- 172.21.15.98:0/1184554561 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5604ba420860 con 0x5604ba4c8000 2024-09-19T08:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.729+0000 7f3d12ffd640 10 client.17466 handle_command_reply: tid=0 2024-09-19T08:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.729+0000 7f3d3c567740 1 client.17466 shutdown 2024-09-19T08:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.730+0000 7f3d3c567740 1 -- 172.21.15.98:0/1184554561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 msgr2=0x5604ba4b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.730+0000 7f3d3c567740 1 --2- 172.21.15.98:0/1184554561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b4fc0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009ca0 tx=0x7f3d2402f3a0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.730+0000 7f3d3c567740 1 -- 172.21.15.98:0/1184554561 shutdown_connections 2024-09-19T08:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.730+0000 7f3d3c567740 1 --2- 172.21.15.98:0/1184554561 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5604ba4c8000 0x5604ba4ca420 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.730+0000 7f3d3c567740 1 --2- 172.21.15.98:0/1184554561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4b8370 0x5604ba4b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.730+0000 7f3d3c567740 1 --2- 172.21.15.98:0/1184554561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b4fc0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.730+0000 7f3d12ffd640 0 client.17466 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.730+0000 7f3d3c567740 1 --2- 172.21.15.98:0/1184554561 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4ad5d0 0x5604ba4b7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.730+0000 7f3d3c567740 1 -- 172.21.15.98:0/1184554561 >> 172.21.15.98:0/1184554561 conn(0x5604ba476310 msgr2=0x5604ba4af930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.730+0000 7f3d3c567740 1 -- 172.21.15.98:0/1184554561 shutdown_connections 2024-09-19T08:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.730+0000 7f3d3c567740 1 -- 172.21.15.98:0/1184554561 wait complete. 2024-09-19T08:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.730+0000 7f3d3c567740 20 client.17466 trim_cache size 0 max 16384 2024-09-19T08:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.742+0000 7f3d3c567740 1 Processor -- start 2024-09-19T08:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d3c567740 1 -- start start 2024-09-19T08:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d3c567740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4ad5d0 0x5604ba4b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d3c567740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d3c567740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4b8370 0x5604ba4b0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d3c567740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3d341afb80 con 0x5604ba4aeb40 2024-09-19T08:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d3c567740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3d341b16c0 con 0x5604ba4ad5d0 2024-09-19T08:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d3c567740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3d341b0250 con 0x5604ba4b8370 2024-09-19T08:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d337fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d337fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53892/0 (socket says 172.21.15.98:53892) 2024-09-19T08:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d337fe640 1 -- 172.21.15.98:0/452669157 learned_addr learned my addr 172.21.15.98:0/452669157 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/452669157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4b8370 0x5604ba4b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/452669157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4ad5d0 0x5604ba4b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d337fe640 1 -- 172.21.15.98:0/452669157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4b8370 msgr2=0x5604ba4b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d337fe640 1 --2- 172.21.15.98:0/452669157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4b8370 0x5604ba4b0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d337fe640 1 -- 172.21.15.98:0/452669157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4ad5d0 msgr2=0x5604ba4b6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d337fe640 1 --2- 172.21.15.98:0/452669157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4ad5d0 0x5604ba4b6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d337fe640 1 -- 172.21.15.98:0/452669157 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604ba4aa960 con 0x5604ba4aeb40 2024-09-19T08:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d337fe640 1 --2- 172.21.15.98:0/452669157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b0120 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f3d2404d8f0 tx=0x7f3d24000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982c8b83c84747cb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.743+0000 7f3d197fa640 1 -- 172.21.15.98:0/452669157 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24049040 con 0x5604ba4aeb40 2024-09-19T08:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.744+0000 7f3d197fa640 1 -- 172.21.15.98:0/452669157 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d2404dce0 con 0x5604ba4aeb40 2024-09-19T08:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.744+0000 7f3d197fa640 1 -- 172.21.15.98:0/452669157 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24004420 con 0x5604ba4aeb40 2024-09-19T08:11:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.744+0000 7f3d3c567740 1 -- 172.21.15.98:0/452669157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 msgr2=0x5604ba4b0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.744+0000 7f3d3c567740 1 --2- 172.21.15.98:0/452669157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b0120 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f3d2404d8f0 tx=0x7f3d24000f40 comp rx=0 tx=0).stop 2024-09-19T08:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.744+0000 7f3d3c567740 1 -- 172.21.15.98:0/452669157 shutdown_connections 2024-09-19T08:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.744+0000 7f3d3c567740 1 --2- 172.21.15.98:0/452669157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4b8370 0x5604ba4b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.744+0000 7f3d3c567740 1 --2- 172.21.15.98:0/452669157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b0120 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.744+0000 7f3d3c567740 1 --2- 172.21.15.98:0/452669157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4ad5d0 0x5604ba4b6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.744+0000 7f3d3c567740 1 -- 172.21.15.98:0/452669157 >> 172.21.15.98:0/452669157 conn(0x5604ba476310 msgr2=0x5604ba4b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.744+0000 7f3d3c567740 1 -- 172.21.15.98:0/452669157 shutdown_connections 2024-09-19T08:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.744+0000 7f3d3c567740 1 -- 172.21.15.98:0/452669157 wait complete. 2024-09-19T08:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.744+0000 7f3d3c567740 1 -- start start 2024-09-19T08:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d3c567740 1 Processor -- start 2024-09-19T08:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d3c567740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4ad5d0 0x5604ba4b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d3c567740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d3c567740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4b8370 0x5604ba4b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d3c567740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3d341b0ff0 con 0x5604ba4aeb40 2024-09-19T08:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d3c567740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3d341b0920 con 0x5604ba4b8370 2024-09-19T08:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d3c567740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3d341adb80 con 0x5604ba4ad5d0 2024-09-19T08:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d3c567740 10 client.0 _ll_register_callbacks cb 0x7f3d2402ff60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d337fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d337fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53906/0 (socket says 172.21.15.98:53906) 2024-09-19T08:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d337fe640 1 -- 172.21.15.98:0/2814516595 learned_addr learned my addr 172.21.15.98:0/2814516595 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/2814516595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4b8370 0x5604ba4b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/2814516595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4ad5d0 0x5604ba4b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d337fe640 1 -- 172.21.15.98:0/2814516595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4ad5d0 msgr2=0x5604ba4b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d337fe640 1 --2- 172.21.15.98:0/2814516595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4ad5d0 0x5604ba4b21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d337fe640 1 -- 172.21.15.98:0/2814516595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4b8370 msgr2=0x5604ba4b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d337fe640 1 --2- 172.21.15.98:0/2814516595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4b8370 0x5604ba4b2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d337fe640 1 -- 172.21.15.98:0/2814516595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604ba531ec0 con 0x5604ba4aeb40 2024-09-19T08:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/2814516595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4b8370 0x5604ba4b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d337fe640 1 --2- 172.21.15.98:0/2814516595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b2710 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f3d2404da60 tx=0x7f3d24009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1d498d691d049e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.745+0000 7f3d12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.746+0000 7f3d12ffd640 1 -- 172.21.15.98:0/2814516595 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24049040 con 0x5604ba4aeb40 2024-09-19T08:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.746+0000 7f3d12ffd640 1 -- 172.21.15.98:0/2814516595 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f3d24003b70 con 0x5604ba4aeb40 2024-09-19T08:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.746+0000 7f3d3c567740 1 -- 172.21.15.98:0/2814516595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d2404dce0 con 0x5604ba4aeb40 2024-09-19T08:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.746+0000 7f3d12ffd640 1 -- 172.21.15.98:0/2814516595 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3d24065050 con 0x5604ba4aeb40 2024-09-19T08:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.746+0000 7f3d3c567740 10 client.17472 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.746+0000 7f3d3c567740 1 -- 172.21.15.98:0/2814516595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5604ba5331a0 con 0x5604ba4aeb40 2024-09-19T08:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.746+0000 7f3d12ffd640 1 -- 172.21.15.98:0/2814516595 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d2404dce0 con 0x5604ba4aeb40 2024-09-19T08:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.746+0000 7f3d12ffd640 1 -- 172.21.15.98:0/2814516595 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3d24056480 con 0x5604ba4aeb40 2024-09-19T08:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.746+0000 7f3d3c567740 10 client.17472 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.746+0000 7f3d3c567740 10 client.17472 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:11:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.746+0000 7f3d3c567740 20 client.17472 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.746+0000 7f3d3c567740 1 --2- 172.21.15.98:0/2814516595 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5604ba533800 0x5604ba535c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.746+0000 7f3d3c567740 4 client.17472 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:11:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.746+0000 7f3d3c567740 1 -- 172.21.15.98:0/2814516595 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5604ba41ad70 con 0x5604ba533800 2024-09-19T08:11:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/2814516595 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5604ba533800 0x5604ba535c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d32ffd640 1 --2- 172.21.15.98:0/2814516595 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5604ba533800 0x5604ba535c20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ed565d704ccb599b server_cookie=a1ebc4a5f3ca3b9e in_seq=0 out_seq=0 2024-09-19T08:11:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d12ffd640 10 client.17472 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:11:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d12ffd640 1 -- 172.21.15.98:0/2814516595 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5604ba41ad70 con 0x5604ba533800 2024-09-19T08:11:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d12ffd640 10 client.17472 handle_command_reply: tid=0 2024-09-19T08:11:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d3c567740 1 client.17472 shutdown 2024-09-19T08:11:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d3c567740 1 -- 172.21.15.98:0/2814516595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 msgr2=0x5604ba4b2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d3c567740 1 --2- 172.21.15.98:0/2814516595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b2710 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f3d2404da60 tx=0x7f3d24009800 comp rx=0 tx=0).stop 2024-09-19T08:11:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d3c567740 1 -- 172.21.15.98:0/2814516595 shutdown_connections 2024-09-19T08:11:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d3c567740 1 --2- 172.21.15.98:0/2814516595 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5604ba533800 0x5604ba535c20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d3c567740 1 --2- 172.21.15.98:0/2814516595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5604ba4b8370 0x5604ba4b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d3c567740 1 --2- 172.21.15.98:0/2814516595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5604ba4aeb40 0x5604ba4b2710 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d3c567740 1 --2- 172.21.15.98:0/2814516595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604ba4ad5d0 0x5604ba4b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d3c567740 1 -- 172.21.15.98:0/2814516595 >> 172.21.15.98:0/2814516595 conn(0x5604ba476310 msgr2=0x5604ba4b7040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.747+0000 7f3d12ffd640 0 client.17472 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:11:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3c567740 1 -- 172.21.15.98:0/2814516595 shutdown_connections 2024-09-19T08:11:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3c567740 1 -- 172.21.15.98:0/2814516595 wait complete. 2024-09-19T08:11:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3c567740 20 client.17472 trim_cache size 0 max 16384 2024-09-19T08:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:11:35.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3a18e640 1 -- 172.21.15.98:0/2821609992 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3d1c0a33b0 msgr2=0x7f3d1c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:35.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3a18e640 1 --2- 172.21.15.98:0/2821609992 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3d1c0a33b0 0x7f3d1c0a5870 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c0cce20 tx=0x7f3d2c097cd0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3a18e640 1 -- 172.21.15.98:0/2821609992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d3410c180 msgr2=0x7f3d34101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:35.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3a18e640 1 --2- 172.21.15.98:0/2821609992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d3410c180 0x7f3d34101480 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00d750 tx=0x7f3d2c00dc10 comp rx=0 tx=0).stop 2024-09-19T08:11:35.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3a18e640 1 -- 172.21.15.98:0/2821609992 shutdown_connections 2024-09-19T08:11:35.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3a18e640 1 --2- 172.21.15.98:0/2821609992 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f3d1c0a33b0 0x7f3d1c0a5870 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3a18e640 1 --2- 172.21.15.98:0/2821609992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d3410c180 0x7f3d34101480 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3a18e640 1 --2- 172.21.15.98:0/2821609992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d341036c0 0x7f3d34100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3a18e640 1 --2- 172.21.15.98:0/2821609992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d34102da0 0x7f3d34100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:35.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3a18e640 1 -- 172.21.15.98:0/2821609992 >> 172.21.15.98:0/2821609992 conn(0x7f3d340fc7a0 msgr2=0x7f3d340fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:35.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3a18e640 1 -- 172.21.15.98:0/2821609992 shutdown_connections 2024-09-19T08:11:35.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:35.748+0000 7f3d3a18e640 1 -- 172.21.15.98:0/2821609992 wait complete. 2024-09-19T08:11:36.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:11:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:11:37.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:36 smithi098 ceph-mon[31333]: pgmap v264: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-19T08:11:37.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:36 smithi139 ceph-mon[34874]: pgmap v264: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-19T08:11:37.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.171+0000 7fae1dfa1640 1 -- 172.21.15.98:0/3119739077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae18102570 msgr2=0x7fae181029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:37.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.171+0000 7fae1dfa1640 1 --2- 172.21.15.98:0/3119739077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae18102570 0x7fae181029b0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fae00009900 tx=0x7fae0002a480 comp rx=0 tx=0).stop 2024-09-19T08:11:37.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.171+0000 7fae1dfa1640 1 -- 172.21.15.98:0/3119739077 shutdown_connections 2024-09-19T08:11:37.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.171+0000 7fae1dfa1640 1 --2- 172.21.15.98:0/3119739077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae18102ef0 0x7fae1810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.171+0000 7fae1dfa1640 1 --2- 172.21.15.98:0/3119739077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae18102570 0x7fae181029b0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.171+0000 7fae1dfa1640 1 --2- 172.21.15.98:0/3119739077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae18100e20 0x7fae18101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.171+0000 7fae1dfa1640 1 -- 172.21.15.98:0/3119739077 >> 172.21.15.98:0/3119739077 conn(0x7fae180fc7d0 msgr2=0x7fae180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:37.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.171+0000 7fae1dfa1640 1 -- 172.21.15.98:0/3119739077 shutdown_connections 2024-09-19T08:11:37.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.171+0000 7fae1dfa1640 1 -- 172.21.15.98:0/3119739077 wait complete. 2024-09-19T08:11:37.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.171+0000 7fae1dfa1640 1 Processor -- start 2024-09-19T08:11:37.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae1dfa1640 1 -- start start 2024-09-19T08:11:37.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae1dfa1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae18100e20 0x7fae181993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae1dfa1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae18102570 0x7fae181998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae1dfa1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae18102ef0 0x7fae181a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae1dfa1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae18110fc0 con 0x7fae18100e20 2024-09-19T08:11:37.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae1dfa1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fae18110e40 con 0x7fae18102ef0 2024-09-19T08:11:37.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae1dfa1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fae18111140 con 0x7fae18102570 2024-09-19T08:11:37.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae177fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae18100e20 0x7fae181993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae177fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae18100e20 0x7fae181993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43816/0 (socket says 172.21.15.98:43816) 2024-09-19T08:11:37.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae177fe640 1 -- 172.21.15.98:0/3999787220 learned_addr learned my addr 172.21.15.98:0/3999787220 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:37.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae0ffff640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae18102570 0x7fae181998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae17fff640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae18102ef0 0x7fae181a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae177fe640 1 -- 172.21.15.98:0/3999787220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae18102570 msgr2=0x7fae181998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:37.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae177fe640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae18102570 0x7fae181998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae177fe640 1 -- 172.21.15.98:0/3999787220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae18102ef0 msgr2=0x7fae181a4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:37.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae177fe640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae18102ef0 0x7fae181a4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae177fe640 1 -- 172.21.15.98:0/3999787220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae181a4970 con 0x7fae18100e20 2024-09-19T08:11:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae17fff640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae18102ef0 0x7fae181a4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae177fe640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae18100e20 0x7fae181993b0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fae0800c970 tx=0x7fae0800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.172+0000 7fae157fa640 1 -- 172.21.15.98:0/3999787220 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae08007bb0 con 0x7fae18100e20 2024-09-19T08:11:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.173+0000 7fae1dfa1640 1 -- 172.21.15.98:0/3999787220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae181a4b40 con 0x7fae18100e20 2024-09-19T08:11:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.173+0000 7fae157fa640 1 -- 172.21.15.98:0/3999787220 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae08007d50 con 0x7fae18100e20 2024-09-19T08:11:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.173+0000 7fae157fa640 1 -- 172.21.15.98:0/3999787220 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0800f450 con 0x7fae18100e20 2024-09-19T08:11:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.173+0000 7fae1dfa1640 1 -- 172.21.15.98:0/3999787220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae181a5000 con 0x7fae18100e20 2024-09-19T08:11:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.175+0000 7fae157fa640 1 -- 172.21.15.98:0/3999787220 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fae08005020 con 0x7fae18100e20 2024-09-19T08:11:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.175+0000 7fae2037a740 1 Processor -- start 2024-09-19T08:11:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.175+0000 7fae2037a740 1 -- start start 2024-09-19T08:11:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.175+0000 7fae2037a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d8b40 0x5633b67d7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.175+0000 7fae2037a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d75d0 0x5633b67d7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.175+0000 7fae2037a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67e2370 0x5633b67e4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.175+0000 7fae2037a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5633b673d2f0 con 0x5633b67d75d0 2024-09-19T08:11:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.175+0000 7fae2037a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5633b66e4cf0 con 0x5633b67e2370 2024-09-19T08:11:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.175+0000 7fae2037a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5633b67392b0 con 0x5633b67d8b40 2024-09-19T08:11:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.175+0000 7fae157fa640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae040a33b0 0x7fae040a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.175+0000 7fae177fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d75d0 0x5633b67d7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.175+0000 7fae177fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d75d0 0x5633b67d7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43832/0 (socket says 172.21.15.98:43832) 2024-09-19T08:11:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.175+0000 7fae177fe640 1 -- 172.21.15.98:0/3288870204 learned_addr learned my addr 172.21.15.98:0/3288870204 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae17fff640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae040a33b0 0x7fae040a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae0ffff640 1 --2- 172.21.15.98:0/3288870204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67e2370 0x5633b67e4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae17fff640 1 --2- 172.21.15.98:0/3288870204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d8b40 0x5633b67d7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae177fe640 1 -- 172.21.15.98:0/3288870204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d8b40 msgr2=0x5633b67d7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae177fe640 1 --2- 172.21.15.98:0/3288870204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d8b40 0x5633b67d7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae177fe640 1 -- 172.21.15.98:0/3288870204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67e2370 msgr2=0x5633b67e4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:37.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae177fe640 1 --2- 172.21.15.98:0/3288870204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67e2370 0x5633b67e4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae157fa640 1 -- 172.21.15.98:0/3999787220 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fae080cfb10 con 0x7fae18100e20 2024-09-19T08:11:37.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae177fe640 1 -- 172.21.15.98:0/3288870204 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633b66c93a0 con 0x5633b67d75d0 2024-09-19T08:11:37.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae17fff640 1 --2- 172.21.15.98:0/3288870204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d8b40 0x5633b67d7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:37.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae0ffff640 1 --2- 172.21.15.98:0/3288870204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67e2370 0x5633b67e4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:37.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae177fe640 1 --2- 172.21.15.98:0/3288870204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d75d0 0x5633b67d7a10 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fae080a4c20 tx=0x7fae08004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a0eed358be6046 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:37.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae17fff640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae040a33b0 0x7fae040a5870 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fae10009440 tx=0x7fae10009600 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:37.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae0dffb640 1 -- 172.21.15.98:0/3288870204 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0801d070 con 0x5633b67d75d0 2024-09-19T08:11:37.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae0dffb640 1 -- 172.21.15.98:0/3288870204 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae0809c4f0 con 0x5633b67d75d0 2024-09-19T08:11:37.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.176+0000 7fae0dffb640 1 -- 172.21.15.98:0/3288870204 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae080cd4e0 con 0x5633b67d75d0 2024-09-19T08:11:37.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.177+0000 7fae2037a740 1 -- 172.21.15.98:0/3288870204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d75d0 msgr2=0x5633b67d7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:37.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.177+0000 7fae2037a740 1 --2- 172.21.15.98:0/3288870204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d75d0 0x5633b67d7a10 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fae080a4c20 tx=0x7fae08004a20 comp rx=0 tx=0).stop 2024-09-19T08:11:37.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.177+0000 7fae2037a740 1 -- 172.21.15.98:0/3288870204 shutdown_connections 2024-09-19T08:11:37.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.177+0000 7fae2037a740 1 --2- 172.21.15.98:0/3288870204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67e2370 0x5633b67e4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.177+0000 7fae2037a740 1 --2- 172.21.15.98:0/3288870204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d75d0 0x5633b67d7a10 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.177+0000 7fae2037a740 1 --2- 172.21.15.98:0/3288870204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d8b40 0x5633b67d7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.177+0000 7fae2037a740 1 -- 172.21.15.98:0/3288870204 >> 172.21.15.98:0/3288870204 conn(0x5633b67a0310 msgr2=0x5633b67d3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:37.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.177+0000 7fae2037a740 1 -- 172.21.15.98:0/3288870204 shutdown_connections 2024-09-19T08:11:37.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.177+0000 7fae2037a740 1 -- 172.21.15.98:0/3288870204 wait complete. 2024-09-19T08:11:37.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.177+0000 7fae2037a740 1 -- start start 2024-09-19T08:11:37.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae2037a740 1 Processor -- start 2024-09-19T08:11:37.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae2037a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d75d0 0x5633b67e1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae2037a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d8b40 0x5633b67defd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae2037a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67e2370 0x5633b67df570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae2037a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5633b67e4fb0 con 0x5633b67d8b40 2024-09-19T08:11:37.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae2037a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5633b66f8060 con 0x5633b67e2370 2024-09-19T08:11:37.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae2037a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5633b67e5130 con 0x5633b67d75d0 2024-09-19T08:11:37.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae2037a740 10 client.0 _ll_register_callbacks cb 0x5633b644b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:37.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae0ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d8b40 0x5633b67defd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae0ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d8b40 0x5633b67defd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43840/0 (socket says 172.21.15.98:43840) 2024-09-19T08:11:37.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae0ffff640 1 -- 172.21.15.98:0/4046393255 learned_addr learned my addr 172.21.15.98:0/4046393255 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:37.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae17fff640 1 --2- 172.21.15.98:0/4046393255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67e2370 0x5633b67df570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae177fe640 1 --2- 172.21.15.98:0/4046393255 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d75d0 0x5633b67e1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae0ffff640 1 -- 172.21.15.98:0/4046393255 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d75d0 msgr2=0x5633b67e1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:37.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae0ffff640 1 --2- 172.21.15.98:0/4046393255 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d75d0 0x5633b67e1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae0ffff640 1 -- 172.21.15.98:0/4046393255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67e2370 msgr2=0x5633b67df570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:37.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae0ffff640 1 --2- 172.21.15.98:0/4046393255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67e2370 0x5633b67df570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae0ffff640 1 -- 172.21.15.98:0/4046393255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633b64ae3c0 con 0x5633b67d8b40 2024-09-19T08:11:37.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae177fe640 1 --2- 172.21.15.98:0/4046393255 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d75d0 0x5633b67e1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:37.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fae0ffff640 1 --2- 172.21.15.98:0/4046393255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d8b40 0x5633b67defd0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fae00009900 tx=0x7fae000044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4be553ce4bac94 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:37.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.178+0000 7fadf6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:11:37.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fadf6ffd640 1 -- 172.21.15.98:0/4046393255 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae00035070 con 0x5633b67d8b40 2024-09-19T08:11:37.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fadf6ffd640 1 -- 172.21.15.98:0/4046393255 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fae00003b40 con 0x5633b67d8b40 2024-09-19T08:11:37.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fae2037a740 1 -- 172.21.15.98:0/4046393255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae0809c4f0 con 0x5633b67d8b40 2024-09-19T08:11:37.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fadf6ffd640 1 -- 172.21.15.98:0/4046393255 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fae00003d40 con 0x5633b67d8b40 2024-09-19T08:11:37.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fae2037a740 10 client.17484 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:37.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fae2037a740 1 -- 172.21.15.98:0/4046393255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5633b67f0a00 con 0x5633b67d8b40 2024-09-19T08:11:37.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fadf6ffd640 1 -- 172.21.15.98:0/4046393255 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae0809c4f0 con 0x5633b67d8b40 2024-09-19T08:11:37.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fadf6ffd640 1 -- 172.21.15.98:0/4046393255 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fae0004e760 con 0x5633b67d8b40 2024-09-19T08:11:37.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fae2037a740 10 client.17484 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:37.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fae2037a740 10 client.17484 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:11:37.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fae2037a740 20 client.17484 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:37.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fae2037a740 1 --2- 172.21.15.98:0/4046393255 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5633b67f1e60 0x5633b67f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fae2037a740 4 client.17484 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:11:37.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fae2037a740 1 -- 172.21.15.98:0/4046393255 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633b674a860 con 0x5633b67f1e60 2024-09-19T08:11:37.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.179+0000 7fae177fe640 1 --2- 172.21.15.98:0/4046393255 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5633b67f1e60 0x5633b67f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.180+0000 7fae177fe640 1 --2- 172.21.15.98:0/4046393255 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5633b67f1e60 0x5633b67f42e0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=44e42d3aad1b4fac server_cookie=5fc18a8db7da1fde in_seq=0 out_seq=0 2024-09-19T08:11:37.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.180+0000 7fadf6ffd640 10 client.17484 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:11:37.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.180+0000 7fadf6ffd640 1 -- 172.21.15.98:0/4046393255 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5633b674a860 con 0x5633b67f1e60 2024-09-19T08:11:37.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.180+0000 7fadf6ffd640 10 client.17484 handle_command_reply: tid=0 2024-09-19T08:11:37.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.180+0000 7fae2037a740 1 client.17484 shutdown 2024-09-19T08:11:37.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.181+0000 7fae2037a740 1 -- 172.21.15.98:0/4046393255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d8b40 msgr2=0x5633b67defd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:37.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.181+0000 7fae2037a740 1 --2- 172.21.15.98:0/4046393255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d8b40 0x5633b67defd0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fae00009900 tx=0x7fae000044c0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.181+0000 7fae2037a740 1 -- 172.21.15.98:0/4046393255 shutdown_connections 2024-09-19T08:11:37.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.181+0000 7fae2037a740 1 --2- 172.21.15.98:0/4046393255 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5633b67f1e60 0x5633b67f42e0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.181+0000 7fae2037a740 1 --2- 172.21.15.98:0/4046393255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67e2370 0x5633b67df570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.181+0000 7fadf6ffd640 0 client.17484 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:11:37.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.181+0000 7fae2037a740 1 --2- 172.21.15.98:0/4046393255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d8b40 0x5633b67defd0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.181+0000 7fae2037a740 1 --2- 172.21.15.98:0/4046393255 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d75d0 0x5633b67e1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.181+0000 7fae2037a740 1 -- 172.21.15.98:0/4046393255 >> 172.21.15.98:0/4046393255 conn(0x5633b67a0310 msgr2=0x5633b67d9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:37.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.181+0000 7fae2037a740 1 -- 172.21.15.98:0/4046393255 shutdown_connections 2024-09-19T08:11:37.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.181+0000 7fae2037a740 1 -- 172.21.15.98:0/4046393255 wait complete. 2024-09-19T08:11:37.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.181+0000 7fae2037a740 20 client.17484 trim_cache size 0 max 16384 2024-09-19T08:11:37.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.193+0000 7fae2037a740 1 Processor -- start 2024-09-19T08:11:37.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae2037a740 1 -- start start 2024-09-19T08:11:37.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae2037a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67d75d0 0x5633b67d2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae2037a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d8b40 0x5633b67da120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae2037a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67e2370 0x5633b67da660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae2037a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae181ad880 con 0x5633b67d8b40 2024-09-19T08:11:37.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae2037a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fae181af3c0 con 0x5633b67d75d0 2024-09-19T08:11:37.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae2037a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fae181adf50 con 0x5633b67e2370 2024-09-19T08:11:37.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae0ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d8b40 0x5633b67da120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae0ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d8b40 0x5633b67da120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43856/0 (socket says 172.21.15.98:43856) 2024-09-19T08:11:37.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae0ffff640 1 -- 172.21.15.98:0/321398765 learned_addr learned my addr 172.21.15.98:0/321398765 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:37.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae177fe640 1 --2- 172.21.15.98:0/321398765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67d75d0 0x5633b67d2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae17fff640 1 --2- 172.21.15.98:0/321398765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67e2370 0x5633b67da660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae0ffff640 1 -- 172.21.15.98:0/321398765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67e2370 msgr2=0x5633b67da660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:37.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae0ffff640 1 --2- 172.21.15.98:0/321398765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67e2370 0x5633b67da660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae0ffff640 1 -- 172.21.15.98:0/321398765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67d75d0 msgr2=0x5633b67d2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:37.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae0ffff640 1 --2- 172.21.15.98:0/321398765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67d75d0 0x5633b67d2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae0ffff640 1 -- 172.21.15.98:0/321398765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633b67df8e0 con 0x5633b67d8b40 2024-09-19T08:11:37.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae17fff640 1 --2- 172.21.15.98:0/321398765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67e2370 0x5633b67da660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:11:37.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae177fe640 1 --2- 172.21.15.98:0/321398765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67d75d0 0x5633b67d2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:37.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae0ffff640 1 --2- 172.21.15.98:0/321398765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d8b40 0x5633b67da120 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fae0003b450 tx=0x7fae0002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6108005da4819f4d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:37.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae0d7fa640 1 -- 172.21.15.98:0/321398765 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae00000e60 con 0x5633b67d8b40 2024-09-19T08:11:37.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae0d7fa640 1 -- 172.21.15.98:0/321398765 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae00053070 con 0x5633b67d8b40 2024-09-19T08:11:37.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.194+0000 7fae0d7fa640 1 -- 172.21.15.98:0/321398765 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae00035040 con 0x5633b67d8b40 2024-09-19T08:11:37.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.195+0000 7fae2037a740 1 -- 172.21.15.98:0/321398765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d8b40 msgr2=0x5633b67da120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:37.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.195+0000 7fae2037a740 1 --2- 172.21.15.98:0/321398765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d8b40 0x5633b67da120 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fae0003b450 tx=0x7fae0002a480 comp rx=0 tx=0).stop 2024-09-19T08:11:37.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.195+0000 7fae2037a740 1 -- 172.21.15.98:0/321398765 shutdown_connections 2024-09-19T08:11:37.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.195+0000 7fae2037a740 1 --2- 172.21.15.98:0/321398765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67e2370 0x5633b67da660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.195+0000 7fae2037a740 1 --2- 172.21.15.98:0/321398765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67d8b40 0x5633b67da120 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.195+0000 7fae2037a740 1 --2- 172.21.15.98:0/321398765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67d75d0 0x5633b67d2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.195+0000 7fae2037a740 1 -- 172.21.15.98:0/321398765 >> 172.21.15.98:0/321398765 conn(0x5633b67a0310 msgr2=0x5633b67d24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.195+0000 7fae2037a740 1 -- 172.21.15.98:0/321398765 shutdown_connections 2024-09-19T08:11:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.195+0000 7fae2037a740 1 -- 172.21.15.98:0/321398765 wait complete. 2024-09-19T08:11:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.195+0000 7fae2037a740 1 -- start start 2024-09-19T08:11:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae2037a740 1 Processor -- start 2024-09-19T08:11:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae2037a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d75d0 0x5633b67dc180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae2037a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67d8b40 0x5633b67dc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae2037a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67e2370 0x5633b67dcc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae2037a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae181aecf0 con 0x5633b67e2370 2024-09-19T08:11:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae2037a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fae181ae620 con 0x5633b67d8b40 2024-09-19T08:11:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae2037a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fae181ab880 con 0x5633b67d75d0 2024-09-19T08:11:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae2037a740 10 client.0 _ll_register_callbacks cb 0x7fae0002ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae17fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67e2370 0x5633b67dcc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae17fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67e2370 0x5633b67dcc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43866/0 (socket says 172.21.15.98:43866) 2024-09-19T08:11:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae17fff640 1 -- 172.21.15.98:0/1455976008 learned_addr learned my addr 172.21.15.98:0/1455976008 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae0ffff640 1 --2- 172.21.15.98:0/1455976008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67d8b40 0x5633b67dc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae177fe640 1 --2- 172.21.15.98:0/1455976008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d75d0 0x5633b67dc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae17fff640 1 -- 172.21.15.98:0/1455976008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d75d0 msgr2=0x5633b67dc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae17fff640 1 --2- 172.21.15.98:0/1455976008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d75d0 0x5633b67dc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae17fff640 1 -- 172.21.15.98:0/1455976008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67d8b40 msgr2=0x5633b67dc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.196+0000 7fae17fff640 1 --2- 172.21.15.98:0/1455976008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67d8b40 0x5633b67dc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.197+0000 7fae17fff640 1 -- 172.21.15.98:0/1455976008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633b685bd30 con 0x5633b67e2370 2024-09-19T08:11:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.197+0000 7fae0ffff640 1 --2- 172.21.15.98:0/1455976008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67d8b40 0x5633b67dc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:11:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.197+0000 7fae17fff640 1 --2- 172.21.15.98:0/1455976008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67e2370 0x5633b67dcc00 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fae100186c0 tx=0x7fae10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11a5eb11016a49b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.197+0000 7fadf6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:11:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.197+0000 7fadf6ffd640 1 -- 172.21.15.98:0/1455976008 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10013040 con 0x5633b67e2370 2024-09-19T08:11:37.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.197+0000 7fadf6ffd640 1 -- 172.21.15.98:0/1455976008 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fae10018e00 con 0x5633b67e2370 2024-09-19T08:11:37.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.197+0000 7fae2037a740 1 -- 172.21.15.98:0/1455976008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae00053070 con 0x5633b67e2370 2024-09-19T08:11:37.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.197+0000 7fadf6ffd640 1 -- 172.21.15.98:0/1455976008 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fae10033050 con 0x5633b67e2370 2024-09-19T08:11:37.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.197+0000 7fae2037a740 10 client.17490 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:37.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.197+0000 7fae2037a740 1 -- 172.21.15.98:0/1455976008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5633b685cfc0 con 0x5633b67e2370 2024-09-19T08:11:37.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.197+0000 7fadf6ffd640 1 -- 172.21.15.98:0/1455976008 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae00053070 con 0x5633b67e2370 2024-09-19T08:11:37.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.197+0000 7fadf6ffd640 1 -- 172.21.15.98:0/1455976008 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fae1002e4d0 con 0x5633b67e2370 2024-09-19T08:11:37.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.197+0000 7fae2037a740 10 client.17490 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:37.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.197+0000 7fae2037a740 10 client.17490 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:11:37.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.198+0000 7fae2037a740 20 client.17490 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:37.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.198+0000 7fae2037a740 1 --2- 172.21.15.98:0/1455976008 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5633b685d7c0 0x5633b685fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:37.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.198+0000 7fae2037a740 4 client.17490 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:11:37.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.198+0000 7fae2037a740 1 -- 172.21.15.98:0/1455976008 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5633b6744d70 con 0x5633b685d7c0 2024-09-19T08:11:37.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.198+0000 7fae177fe640 1 --2- 172.21.15.98:0/1455976008 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5633b685d7c0 0x5633b685fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:37.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.198+0000 7fae177fe640 1 --2- 172.21.15.98:0/1455976008 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5633b685d7c0 0x5633b685fbe0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=18df1684d902ec9c server_cookie=8971c3fa722e2e70 in_seq=0 out_seq=0 2024-09-19T08:11:37.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.198+0000 7fadf6ffd640 10 client.17490 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:11:37.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.198+0000 7fadf6ffd640 1 -- 172.21.15.98:0/1455976008 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5633b6744d70 con 0x5633b685d7c0 2024-09-19T08:11:37.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.198+0000 7fadf6ffd640 10 client.17490 handle_command_reply: tid=0 2024-09-19T08:11:37.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.198+0000 7fae2037a740 1 client.17490 shutdown 2024-09-19T08:11:37.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae2037a740 1 -- 172.21.15.98:0/1455976008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67e2370 msgr2=0x5633b67dcc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:37.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae2037a740 1 --2- 172.21.15.98:0/1455976008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67e2370 0x5633b67dcc00 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fae100186c0 tx=0x7fae10004a40 comp rx=0 tx=0).stop 2024-09-19T08:11:37.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae2037a740 1 -- 172.21.15.98:0/1455976008 shutdown_connections 2024-09-19T08:11:37.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae2037a740 1 --2- 172.21.15.98:0/1455976008 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5633b685d7c0 0x5633b685fbe0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae2037a740 1 --2- 172.21.15.98:0/1455976008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b67e2370 0x5633b67dcc00 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae2037a740 1 --2- 172.21.15.98:0/1455976008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5633b67d8b40 0x5633b67dc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fadf6ffd640 0 client.17490 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:11:37.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae2037a740 1 --2- 172.21.15.98:0/1455976008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633b67d75d0 0x5633b67dc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae2037a740 1 -- 172.21.15.98:0/1455976008 >> 172.21.15.98:0/1455976008 conn(0x5633b67a0310 msgr2=0x5633b67e1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:37.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae2037a740 1 -- 172.21.15.98:0/1455976008 shutdown_connections 2024-09-19T08:11:37.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae2037a740 1 -- 172.21.15.98:0/1455976008 wait complete. 2024-09-19T08:11:37.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae2037a740 20 client.17490 trim_cache size 0 max 16384 2024-09-19T08:11:37.215 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:11:37.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae1dfa1640 1 -- 172.21.15.98:0/3999787220 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae040a33b0 msgr2=0x7fae040a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:37.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae1dfa1640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae040a33b0 0x7fae040a5870 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fae10009440 tx=0x7fae10009600 comp rx=0 tx=0).stop 2024-09-19T08:11:37.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae1dfa1640 1 -- 172.21.15.98:0/3999787220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae18100e20 msgr2=0x7fae181993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:37.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae1dfa1640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae18100e20 0x7fae181993b0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fae0800c970 tx=0x7fae0800ce30 comp rx=0 tx=0).stop 2024-09-19T08:11:37.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae1dfa1640 1 -- 172.21.15.98:0/3999787220 shutdown_connections 2024-09-19T08:11:37.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae1dfa1640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae040a33b0 0x7fae040a5870 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae1dfa1640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae18102ef0 0x7fae181a4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae1dfa1640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae18102570 0x7fae181998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae1dfa1640 1 --2- 172.21.15.98:0/3999787220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae18100e20 0x7fae181993b0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:37.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae1dfa1640 1 -- 172.21.15.98:0/3999787220 >> 172.21.15.98:0/3999787220 conn(0x7fae180fc7d0 msgr2=0x7fae180febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:37.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae1dfa1640 1 -- 172.21.15.98:0/3999787220 shutdown_connections 2024-09-19T08:11:37.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:37.199+0000 7fae1dfa1640 1 -- 172.21.15.98:0/3999787220 wait complete. 2024-09-19T08:11:37.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:36 smithi162 ceph-mon[35260]: pgmap v264: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-19T08:11:37.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:11:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:11:38.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.655+0000 7f192e3ed640 1 -- 172.21.15.98:0/3254725938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1928102da0 msgr2=0x7f1928103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:38.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.655+0000 7f192e3ed640 1 --2- 172.21.15.98:0/3254725938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1928102da0 0x7f1928103180 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f1918009900 tx=0x7f191802a480 comp rx=0 tx=0).stop 2024-09-19T08:11:38.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.655+0000 7f192e3ed640 1 -- 172.21.15.98:0/3254725938 shutdown_connections 2024-09-19T08:11:38.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.655+0000 7f192e3ed640 1 --2- 172.21.15.98:0/3254725938 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f192810c180 0x7f192810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.655+0000 7f192e3ed640 1 --2- 172.21.15.98:0/3254725938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19281036c0 0x7f192810bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.655+0000 7f192e3ed640 1 --2- 172.21.15.98:0/3254725938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1928102da0 0x7f1928103180 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.655+0000 7f192e3ed640 1 -- 172.21.15.98:0/3254725938 >> 172.21.15.98:0/3254725938 conn(0x7f19280fc7a0 msgr2=0x7f19280febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:38.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.655+0000 7f192e3ed640 1 -- 172.21.15.98:0/3254725938 shutdown_connections 2024-09-19T08:11:38.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.655+0000 7f192e3ed640 1 -- 172.21.15.98:0/3254725938 wait complete. 2024-09-19T08:11:38.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.656+0000 7f192e3ed640 1 Processor -- start 2024-09-19T08:11:38.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.656+0000 7f192e3ed640 1 -- start start 2024-09-19T08:11:38.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.656+0000 7f192e3ed640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1928102da0 0x7f19281983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.656+0000 7f192e3ed640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19281036c0 0x7f1928198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.656+0000 7f192e3ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f192810c180 0x7f192819ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.656+0000 7f192e3ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1928111f60 con 0x7f192810c180 2024-09-19T08:11:38.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.656+0000 7f192e3ed640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1928111de0 con 0x7f1928102da0 2024-09-19T08:11:38.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.656+0000 7f192e3ed640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f19281120e0 con 0x7f19281036c0 2024-09-19T08:11:38.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f192c98a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f192810c180 0x7f192819ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f192c98a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f192810c180 0x7f192819ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43882/0 (socket says 172.21.15.98:43882) 2024-09-19T08:11:38.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f192c98a640 1 -- 172.21.15.98:0/2183132854 learned_addr learned my addr 172.21.15.98:0/2183132854 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:38.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f19277fe640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19281036c0 0x7f1928198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f1927fff640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1928102da0 0x7f19281983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f192c98a640 1 -- 172.21.15.98:0/2183132854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19281036c0 msgr2=0x7f1928198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:38.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f192c98a640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19281036c0 0x7f1928198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f192c98a640 1 -- 172.21.15.98:0/2183132854 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1928102da0 msgr2=0x7f19281983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:38.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f192c98a640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1928102da0 0x7f19281983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f192c98a640 1 -- 172.21.15.98:0/2183132854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f192819d390 con 0x7f192810c180 2024-09-19T08:11:38.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f1927fff640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1928102da0 0x7f19281983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:11:38.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f192c98a640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f192810c180 0x7f192819ccb0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f192000cad0 tx=0x7f192000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:38.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f19257fa640 1 -- 172.21.15.98:0/2183132854 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192000ae20 con 0x7f192810c180 2024-09-19T08:11:38.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f19257fa640 1 -- 172.21.15.98:0/2183132854 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1920005020 con 0x7f192810c180 2024-09-19T08:11:38.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f192e3ed640 1 -- 172.21.15.98:0/2183132854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f192819d680 con 0x7f192810c180 2024-09-19T08:11:38.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f19257fa640 1 -- 172.21.15.98:0/2183132854 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192000f6f0 con 0x7f192810c180 2024-09-19T08:11:38.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.657+0000 7f192e3ed640 1 -- 172.21.15.98:0/2183132854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19281a4f60 con 0x7f192810c180 2024-09-19T08:11:38.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.659+0000 7f19257fa640 1 -- 172.21.15.98:0/2183132854 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f19200051d0 con 0x7f192810c180 2024-09-19T08:11:38.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f19307c6740 1 Processor -- start 2024-09-19T08:11:38.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f19307c6740 1 -- start start 2024-09-19T08:11:38.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f19307c6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb43f50 0x5562fbb424a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f19307c6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb429e0 0x5562fbb42e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f19307c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb4d780 0x5562fbb4fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f19307c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5562fbaa42f0 con 0x5562fbb4d780 2024-09-19T08:11:38.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f19257fa640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f19100a33b0 0x7f19100a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f19307c6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5562fba4bcf0 con 0x5562fbb429e0 2024-09-19T08:11:38.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f19307c6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5562fbaa02b0 con 0x5562fbb43f50 2024-09-19T08:11:38.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f19277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb4d780 0x5562fbb4fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f192c98a640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f19100a33b0 0x7f19100a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f19277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb4d780 0x5562fbb4fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43898/0 (socket says 172.21.15.98:43898) 2024-09-19T08:11:38.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f19277fe640 1 -- 172.21.15.98:0/1037809773 learned_addr learned my addr 172.21.15.98:0/1037809773 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:38.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f19257fa640 1 -- 172.21.15.98:0/2183132854 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f19200cfdc0 con 0x7f192810c180 2024-09-19T08:11:38.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f1927fff640 1 --2- 172.21.15.98:0/1037809773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb429e0 0x5562fbb42e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.660+0000 7f19277fe640 1 --2- 172.21.15.98:0/1037809773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb43f50 0x5562fbb424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.661+0000 7f19277fe640 1 -- 172.21.15.98:0/1037809773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb43f50 msgr2=0x5562fbb424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:38.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.661+0000 7f19277fe640 1 --2- 172.21.15.98:0/1037809773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb43f50 0x5562fbb424a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.661+0000 7f19277fe640 1 -- 172.21.15.98:0/1037809773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb429e0 msgr2=0x5562fbb42e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:38.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.661+0000 7f19277fe640 1 --2- 172.21.15.98:0/1037809773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb429e0 0x5562fbb42e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.661+0000 7f19277fe640 1 -- 172.21.15.98:0/1037809773 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562fba303a0 con 0x5562fbb4d780 2024-09-19T08:11:38.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.661+0000 7f19277fe640 1 --2- 172.21.15.98:0/1037809773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb4d780 0x5562fbb4fb70 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f191c0128b0 tx=0x7f191c00b640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=775c760451202d7b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:38.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.661+0000 7f192c98a640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f19100a33b0 0x7f19100a5870 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f19200cd5c0 tx=0x7f192001baf0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:38.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.661+0000 7f190e7fc640 1 -- 172.21.15.98:0/1037809773 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c010c60 con 0x5562fbb4d780 2024-09-19T08:11:38.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.661+0000 7f190e7fc640 1 -- 172.21.15.98:0/1037809773 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f191c010e00 con 0x5562fbb4d780 2024-09-19T08:11:38.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.661+0000 7f190e7fc640 1 -- 172.21.15.98:0/1037809773 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c004bd0 con 0x5562fbb4d780 2024-09-19T08:11:38.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.662+0000 7f19307c6740 1 -- 172.21.15.98:0/1037809773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb4d780 msgr2=0x5562fbb4fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:38.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.662+0000 7f19307c6740 1 --2- 172.21.15.98:0/1037809773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb4d780 0x5562fbb4fb70 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f191c0128b0 tx=0x7f191c00b640 comp rx=0 tx=0).stop 2024-09-19T08:11:38.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.662+0000 7f19307c6740 1 -- 172.21.15.98:0/1037809773 shutdown_connections 2024-09-19T08:11:38.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.662+0000 7f19307c6740 1 --2- 172.21.15.98:0/1037809773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb4d780 0x5562fbb4fb70 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.662+0000 7f19307c6740 1 --2- 172.21.15.98:0/1037809773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb429e0 0x5562fbb42e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.662+0000 7f19307c6740 1 --2- 172.21.15.98:0/1037809773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb43f50 0x5562fbb424a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.662+0000 7f19307c6740 1 -- 172.21.15.98:0/1037809773 >> 172.21.15.98:0/1037809773 conn(0x5562fbb07310 msgr2=0x5562fbb3f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:38.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.662+0000 7f19307c6740 1 -- 172.21.15.98:0/1037809773 shutdown_connections 2024-09-19T08:11:38.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.662+0000 7f19307c6740 1 -- 172.21.15.98:0/1037809773 wait complete. 2024-09-19T08:11:38.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.662+0000 7f19307c6740 1 -- start start 2024-09-19T08:11:38.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.662+0000 7f19307c6740 1 Processor -- start 2024-09-19T08:11:38.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19307c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb429e0 0x5562fbb4d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19307c6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb43f50 0x5562fbb4a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19307c6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb4d780 0x5562fbb4aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19307c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5562fbb50480 con 0x5562fbb429e0 2024-09-19T08:11:38.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19307c6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5562fba5f060 con 0x5562fbb43f50 2024-09-19T08:11:38.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19307c6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5562fbb50600 con 0x5562fbb4d780 2024-09-19T08:11:38.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19307c6740 10 client.0 _ll_register_callbacks cb 0x5562fb7b28c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:38.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb429e0 0x5562fbb4d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb429e0 0x5562fbb4d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43912/0 (socket says 172.21.15.98:43912) 2024-09-19T08:11:38.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19277fe640 1 -- 172.21.15.98:0/224135866 learned_addr learned my addr 172.21.15.98:0/224135866 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:38.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19277fe640 1 --2- 172.21.15.98:0/224135866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb4d780 0x5562fbb4aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f1927fff640 1 --2- 172.21.15.98:0/224135866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb43f50 0x5562fbb4a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19277fe640 1 -- 172.21.15.98:0/224135866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb4d780 msgr2=0x5562fbb4aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:38.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19277fe640 1 --2- 172.21.15.98:0/224135866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb4d780 0x5562fbb4aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19277fe640 1 -- 172.21.15.98:0/224135866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb43f50 msgr2=0x5562fbb4a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:38.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19277fe640 1 --2- 172.21.15.98:0/224135866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb43f50 0x5562fbb4a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19277fe640 1 -- 172.21.15.98:0/224135866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562fb8153c0 con 0x5562fbb429e0 2024-09-19T08:11:38.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f1927fff640 1 --2- 172.21.15.98:0/224135866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb43f50 0x5562fbb4a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:38.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19277fe640 1 --2- 172.21.15.98:0/224135866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb429e0 0x5562fbb4d1a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f191c010820 tx=0x7f191c012ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b4844945e355db server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:38.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19077fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:11:38.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19077fe640 1 -- 172.21.15.98:0/224135866 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c02d640 con 0x5562fbb429e0 2024-09-19T08:11:38.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19077fe640 1 -- 172.21.15.98:0/224135866 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f191c02d7e0 con 0x5562fbb429e0 2024-09-19T08:11:38.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.663+0000 7f19307c6740 1 -- 172.21.15.98:0/224135866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f191c010e00 con 0x5562fbb429e0 2024-09-19T08:11:38.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.664+0000 7f19077fe640 1 -- 172.21.15.98:0/224135866 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f191c012020 con 0x5562fbb429e0 2024-09-19T08:11:38.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.664+0000 7f19307c6740 10 client.17502 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:38.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.664+0000 7f19307c6740 1 -- 172.21.15.98:0/224135866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5562fbb5bf50 con 0x5562fbb429e0 2024-09-19T08:11:38.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.664+0000 7f19077fe640 1 -- 172.21.15.98:0/224135866 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f191c010e00 con 0x5562fbb429e0 2024-09-19T08:11:38.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.664+0000 7f19077fe640 1 -- 172.21.15.98:0/224135866 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f191c02b020 con 0x5562fbb429e0 2024-09-19T08:11:38.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.664+0000 7f19307c6740 10 client.17502 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:38.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.664+0000 7f19307c6740 10 client.17502 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:11:38.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.664+0000 7f19307c6740 20 client.17502 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:38.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.664+0000 7f19307c6740 1 --2- 172.21.15.98:0/224135866 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5562fbb5d2a0 0x5562fbb5f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.664+0000 7f19307c6740 4 client.17502 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:11:38.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.664+0000 7f19307c6740 1 -- 172.21.15.98:0/224135866 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562fbab1860 con 0x5562fbb5d2a0 2024-09-19T08:11:38.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.664+0000 7f1927fff640 1 --2- 172.21.15.98:0/224135866 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5562fbb5d2a0 0x5562fbb5f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.665+0000 7f1927fff640 1 --2- 172.21.15.98:0/224135866 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5562fbb5d2a0 0x5562fbb5f690 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b6a8bdbf25f65f6 server_cookie=517b8030c04d7b58 in_seq=0 out_seq=0 2024-09-19T08:11:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.665+0000 7f19077fe640 10 client.17502 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.665+0000 7f19077fe640 1 -- 172.21.15.98:0/224135866 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5562fbab1860 con 0x5562fbb5d2a0 2024-09-19T08:11:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.665+0000 7f19077fe640 10 client.17502 handle_command_reply: tid=0 2024-09-19T08:11:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.665+0000 7f19307c6740 1 client.17502 shutdown 2024-09-19T08:11:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.666+0000 7f19307c6740 1 -- 172.21.15.98:0/224135866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb429e0 msgr2=0x5562fbb4d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.666+0000 7f19307c6740 1 --2- 172.21.15.98:0/224135866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb429e0 0x5562fbb4d1a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f191c010820 tx=0x7f191c012ef0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.666+0000 7f19307c6740 1 -- 172.21.15.98:0/224135866 shutdown_connections 2024-09-19T08:11:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.666+0000 7f19307c6740 1 --2- 172.21.15.98:0/224135866 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5562fbb5d2a0 0x5562fbb5f690 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.666+0000 7f19307c6740 1 --2- 172.21.15.98:0/224135866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb4d780 0x5562fbb4aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.666+0000 7f19307c6740 1 --2- 172.21.15.98:0/224135866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb43f50 0x5562fbb4a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.666+0000 7f19307c6740 1 --2- 172.21.15.98:0/224135866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb429e0 0x5562fbb4d1a0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.666+0000 7f19077fe640 0 client.17502 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.666+0000 7f19307c6740 1 -- 172.21.15.98:0/224135866 >> 172.21.15.98:0/224135866 conn(0x5562fbb07310 msgr2=0x5562fbb44d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:38.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.666+0000 7f19307c6740 1 -- 172.21.15.98:0/224135866 shutdown_connections 2024-09-19T08:11:38.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.666+0000 7f19307c6740 1 -- 172.21.15.98:0/224135866 wait complete. 2024-09-19T08:11:38.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.666+0000 7f19307c6740 20 client.17502 trim_cache size 0 max 16384 2024-09-19T08:11:38.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.678+0000 7f19307c6740 1 Processor -- start 2024-09-19T08:11:38.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.678+0000 7f19307c6740 1 -- start start 2024-09-19T08:11:38.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f19307c6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb429e0 0x5562fbb59720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f19307c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb43f50 0x5562fbb3f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f19307c6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb4d780 0x5562fbb64cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f19307c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19281ab700 con 0x5562fbb43f50 2024-09-19T08:11:38.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f19307c6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f19281ad240 con 0x5562fbb429e0 2024-09-19T08:11:38.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f19307c6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f19281abdd0 con 0x5562fbb4d780 2024-09-19T08:11:38.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f1927fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb43f50 0x5562fbb3f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f1927fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb43f50 0x5562fbb3f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43920/0 (socket says 172.21.15.98:43920) 2024-09-19T08:11:38.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f1927fff640 1 -- 172.21.15.98:0/1290716944 learned_addr learned my addr 172.21.15.98:0/1290716944 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:38.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f19277fe640 1 --2- 172.21.15.98:0/1290716944 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb429e0 0x5562fbb59720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f1927fff640 1 -- 172.21.15.98:0/1290716944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb4d780 msgr2=0x5562fbb64cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:38.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f1927fff640 1 --2- 172.21.15.98:0/1290716944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb4d780 0x5562fbb64cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f1927fff640 1 -- 172.21.15.98:0/1290716944 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb429e0 msgr2=0x5562fbb59720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:38.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f1927fff640 1 --2- 172.21.15.98:0/1290716944 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb429e0 0x5562fbb59720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f1927fff640 1 -- 172.21.15.98:0/1290716944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562fbb4add0 con 0x5562fbb43f50 2024-09-19T08:11:38.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f1927fff640 1 --2- 172.21.15.98:0/1290716944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb43f50 0x5562fbb3f0b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f1918009230 tx=0x7f1918005a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990854737fa26fc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:38.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f190dffb640 1 -- 172.21.15.98:0/1290716944 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1918035040 con 0x5562fbb43f50 2024-09-19T08:11:38.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f190dffb640 1 -- 172.21.15.98:0/1290716944 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1918032960 con 0x5562fbb43f50 2024-09-19T08:11:38.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.679+0000 7f190dffb640 1 -- 172.21.15.98:0/1290716944 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1918004920 con 0x5562fbb43f50 2024-09-19T08:11:38.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.680+0000 7f19307c6740 1 -- 172.21.15.98:0/1290716944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb43f50 msgr2=0x5562fbb3f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:38.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.680+0000 7f19307c6740 1 --2- 172.21.15.98:0/1290716944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb43f50 0x5562fbb3f0b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f1918009230 tx=0x7f1918005a40 comp rx=0 tx=0).stop 2024-09-19T08:11:38.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.680+0000 7f19307c6740 1 -- 172.21.15.98:0/1290716944 shutdown_connections 2024-09-19T08:11:38.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.680+0000 7f19307c6740 1 --2- 172.21.15.98:0/1290716944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb4d780 0x5562fbb64cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.680+0000 7f19307c6740 1 --2- 172.21.15.98:0/1290716944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb43f50 0x5562fbb3f0b0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.680+0000 7f19307c6740 1 --2- 172.21.15.98:0/1290716944 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb429e0 0x5562fbb59720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.680+0000 7f19307c6740 1 -- 172.21.15.98:0/1290716944 >> 172.21.15.98:0/1290716944 conn(0x5562fbb07310 msgr2=0x5562fbb49540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:38.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.680+0000 7f19307c6740 1 -- 172.21.15.98:0/1290716944 shutdown_connections 2024-09-19T08:11:38.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.680+0000 7f19307c6740 1 -- 172.21.15.98:0/1290716944 wait complete. 2024-09-19T08:11:38.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.680+0000 7f19307c6740 1 -- start start 2024-09-19T08:11:38.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19307c6740 1 Processor -- start 2024-09-19T08:11:38.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19307c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb429e0 0x5562fbb55bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19307c6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb43f50 0x5562fbb560f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19307c6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb4d780 0x5562fbb46e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19307c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19281acb70 con 0x5562fbb429e0 2024-09-19T08:11:38.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19307c6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f19281ac4a0 con 0x5562fbb43f50 2024-09-19T08:11:38.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19307c6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f19281a9700 con 0x5562fbb4d780 2024-09-19T08:11:38.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19307c6740 10 client.0 _ll_register_callbacks cb 0x7f1918009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:38.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb429e0 0x5562fbb55bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb429e0 0x5562fbb55bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43928/0 (socket says 172.21.15.98:43928) 2024-09-19T08:11:38.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19277fe640 1 -- 172.21.15.98:0/3292652419 learned_addr learned my addr 172.21.15.98:0/3292652419 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:38.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f1927fff640 1 --2- 172.21.15.98:0/3292652419 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb43f50 0x5562fbb560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19277fe640 1 --2- 172.21.15.98:0/3292652419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb4d780 0x5562fbb46e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19277fe640 1 -- 172.21.15.98:0/3292652419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb4d780 msgr2=0x5562fbb46e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:38.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19277fe640 1 --2- 172.21.15.98:0/3292652419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb4d780 0x5562fbb46e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19277fe640 1 -- 172.21.15.98:0/3292652419 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb43f50 msgr2=0x5562fbb560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19277fe640 1 --2- 172.21.15.98:0/3292652419 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb43f50 0x5562fbb560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.681+0000 7f19277fe640 1 -- 172.21.15.98:0/3292652419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562fbb47640 con 0x5562fbb429e0 2024-09-19T08:11:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19277fe640 1 --2- 172.21.15.98:0/3292652419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb429e0 0x5562fbb55bb0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f191c011740 tx=0x7f191c011540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd5005792c03847 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19077fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:11:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19077fe640 1 -- 172.21.15.98:0/3292652419 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c036a10 con 0x5562fbb429e0 2024-09-19T08:11:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19077fe640 1 -- 172.21.15.98:0/3292652419 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f191c036bb0 con 0x5562fbb429e0 2024-09-19T08:11:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19307c6740 1 -- 172.21.15.98:0/3292652419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1918032960 con 0x5562fbb429e0 2024-09-19T08:11:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19077fe640 1 -- 172.21.15.98:0/3292652419 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f191c047050 con 0x5562fbb429e0 2024-09-19T08:11:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19307c6740 10 client.17508 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19307c6740 1 -- 172.21.15.98:0/3292652419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5562fbbc7850 con 0x5562fbb429e0 2024-09-19T08:11:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19077fe640 1 -- 172.21.15.98:0/3292652419 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1918032960 con 0x5562fbb429e0 2024-09-19T08:11:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19077fe640 1 -- 172.21.15.98:0/3292652419 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f191c02b050 con 0x5562fbb429e0 2024-09-19T08:11:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19307c6740 10 client.17508 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19307c6740 10 client.17508 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:11:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19307c6740 20 client.17508 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19307c6740 1 --2- 172.21.15.98:0/3292652419 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5562fbbc7e80 0x5562fbbca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19307c6740 4 client.17508 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:11:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.682+0000 7f19307c6740 1 -- 172.21.15.98:0/3292652419 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5562fbaabd70 con 0x5562fbbc7e80 2024-09-19T08:11:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.683+0000 7f1927fff640 1 --2- 172.21.15.98:0/3292652419 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5562fbbc7e80 0x5562fbbca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.683+0000 7f1927fff640 1 --2- 172.21.15.98:0/3292652419 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5562fbbc7e80 0x5562fbbca2a0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe2c231b0ebd0f4e server_cookie=4dd87bf8c80555c7 in_seq=0 out_seq=0 2024-09-19T08:11:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.683+0000 7f19077fe640 10 client.17508 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.683+0000 7f19077fe640 1 -- 172.21.15.98:0/3292652419 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5562fbaabd70 con 0x5562fbbc7e80 2024-09-19T08:11:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.683+0000 7f19077fe640 10 client.17508 handle_command_reply: tid=0 2024-09-19T08:11:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.683+0000 7f19307c6740 1 client.17508 shutdown 2024-09-19T08:11:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f19307c6740 1 -- 172.21.15.98:0/3292652419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb429e0 msgr2=0x5562fbb55bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f19307c6740 1 --2- 172.21.15.98:0/3292652419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb429e0 0x5562fbb55bb0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f191c011740 tx=0x7f191c011540 comp rx=0 tx=0).stop 2024-09-19T08:11:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f19307c6740 1 -- 172.21.15.98:0/3292652419 shutdown_connections 2024-09-19T08:11:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f19307c6740 1 --2- 172.21.15.98:0/3292652419 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5562fbbc7e80 0x5562fbbca2a0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f19307c6740 1 --2- 172.21.15.98:0/3292652419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562fbb4d780 0x5562fbb46e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f19307c6740 1 --2- 172.21.15.98:0/3292652419 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5562fbb43f50 0x5562fbb560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f19077fe640 0 client.17508 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f19307c6740 1 --2- 172.21.15.98:0/3292652419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5562fbb429e0 0x5562fbb55bb0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f19307c6740 1 -- 172.21.15.98:0/3292652419 >> 172.21.15.98:0/3292652419 conn(0x5562fbb07310 msgr2=0x5562fbb44520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f19307c6740 1 -- 172.21.15.98:0/3292652419 shutdown_connections 2024-09-19T08:11:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f19307c6740 1 -- 172.21.15.98:0/3292652419 wait complete. 2024-09-19T08:11:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f19307c6740 20 client.17508 trim_cache size 0 max 16384 2024-09-19T08:11:38.691 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:11:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f192e3ed640 1 -- 172.21.15.98:0/2183132854 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f19100a33b0 msgr2=0x7f19100a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f192e3ed640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f19100a33b0 0x7f19100a5870 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f19200cd5c0 tx=0x7f192001baf0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f192e3ed640 1 -- 172.21.15.98:0/2183132854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f192810c180 msgr2=0x7f192819ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f192e3ed640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f192810c180 0x7f192819ccb0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f192000cad0 tx=0x7f192000cf90 comp rx=0 tx=0).stop 2024-09-19T08:11:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f192e3ed640 1 -- 172.21.15.98:0/2183132854 shutdown_connections 2024-09-19T08:11:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f192e3ed640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f19100a33b0 0x7f19100a5870 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f192e3ed640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f192810c180 0x7f192819ccb0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f192e3ed640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19281036c0 0x7f1928198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f192e3ed640 1 --2- 172.21.15.98:0/2183132854 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1928102da0 0x7f19281983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:38.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.684+0000 7f192e3ed640 1 -- 172.21.15.98:0/2183132854 >> 172.21.15.98:0/2183132854 conn(0x7f19280fc7a0 msgr2=0x7f19280fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:38.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.685+0000 7f192e3ed640 1 -- 172.21.15.98:0/2183132854 shutdown_connections 2024-09-19T08:11:38.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:38.685+0000 7f192e3ed640 1 -- 172.21.15.98:0/2183132854 wait complete. 2024-09-19T08:11:39.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:38 smithi139 ceph-mon[34874]: pgmap v265: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 11 op/s 2024-09-19T08:11:39.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:38 smithi098 ceph-mon[31333]: pgmap v265: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 11 op/s 2024-09-19T08:11:39.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:38 smithi162 ceph-mon[35260]: pgmap v265: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 11 op/s 2024-09-19T08:11:40.184 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:39 smithi098 ceph-mon[31333]: pgmap v266: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-09-19T08:11:40.327 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:11:40.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:11:40.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:39 smithi162 ceph-mon[35260]: pgmap v266: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-09-19T08:11:40.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:39 smithi139 ceph-mon[34874]: pgmap v266: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-09-19T08:11:40.578 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:11:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.100+0000 7f08348b5640 1 -- 172.21.15.98:0/3072940276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0830100a80 msgr2=0x7f0830100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.100+0000 7f08348b5640 1 --2- 172.21.15.98:0/3072940276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0830100a80 0x7f0830100ee0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f0824009930 tx=0x7f082402a420 comp rx=0 tx=0).stop 2024-09-19T08:11:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.100+0000 7f08348b5640 1 -- 172.21.15.98:0/3072940276 shutdown_connections 2024-09-19T08:11:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.100+0000 7f08348b5640 1 --2- 172.21.15.98:0/3072940276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0830101420 0x7f083010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.100+0000 7f08348b5640 1 --2- 172.21.15.98:0/3072940276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0830100a80 0x7f0830100ee0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.100+0000 7f08348b5640 1 --2- 172.21.15.98:0/3072940276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0830106a80 0x7f0830106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.100+0000 7f08348b5640 1 -- 172.21.15.98:0/3072940276 >> 172.21.15.98:0/3072940276 conn(0x7f08300fc7a0 msgr2=0x7f08300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.100+0000 7f08348b5640 1 -- 172.21.15.98:0/3072940276 shutdown_connections 2024-09-19T08:11:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.100+0000 7f08348b5640 1 -- 172.21.15.98:0/3072940276 wait complete. 2024-09-19T08:11:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.101+0000 7f08348b5640 1 Processor -- start 2024-09-19T08:11:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.101+0000 7f08348b5640 1 -- start start 2024-09-19T08:11:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.101+0000 7f08348b5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0830100a80 0x7f0830197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.101+0000 7f08348b5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0830101420 0x7f0830197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.101+0000 7f08348b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0830106a80 0x7f0830197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.101+0000 7f08348b5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0830110f20 con 0x7f0830106a80 2024-09-19T08:11:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.101+0000 7f08348b5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0830110da0 con 0x7f0830100a80 2024-09-19T08:11:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.101+0000 7f08348b5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08301110a0 con 0x7f0830101420 2024-09-19T08:11:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.101+0000 7f082ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0830106a80 0x7f0830197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.101+0000 7f082ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0830106a80 0x7f0830197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43954/0 (socket says 172.21.15.98:43954) 2024-09-19T08:11:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.101+0000 7f082ed9d640 1 -- 172.21.15.98:0/2241623061 learned_addr learned my addr 172.21.15.98:0/2241623061 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.101+0000 7f082dd9b640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0830101420 0x7f0830197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.101+0000 7f082e59c640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0830100a80 0x7f0830197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.102+0000 7f082ed9d640 1 -- 172.21.15.98:0/2241623061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0830101420 msgr2=0x7f0830197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.102+0000 7f082ed9d640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0830101420 0x7f0830197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.102+0000 7f082ed9d640 1 -- 172.21.15.98:0/2241623061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0830100a80 msgr2=0x7f0830197220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.102+0000 7f082ed9d640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0830100a80 0x7f0830197220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.102+0000 7f082ed9d640 1 -- 172.21.15.98:0/2241623061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08301a6a90 con 0x7f0830106a80 2024-09-19T08:11:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.102+0000 7f082e59c640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0830100a80 0x7f0830197220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:11:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.102+0000 7f082ed9d640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0830106a80 0x7f0830197ca0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f082800b970 tx=0x7f082800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.102+0000 7f081f7fe640 1 -- 172.21.15.98:0/2241623061 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0828019070 con 0x7f0830106a80 2024-09-19T08:11:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.102+0000 7f08348b5640 1 -- 172.21.15.98:0/2241623061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08301a6d80 con 0x7f0830106a80 2024-09-19T08:11:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.102+0000 7f081f7fe640 1 -- 172.21.15.98:0/2241623061 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f082800cb40 con 0x7f0830106a80 2024-09-19T08:11:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.102+0000 7f081f7fe640 1 -- 172.21.15.98:0/2241623061 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0828007610 con 0x7f0830106a80 2024-09-19T08:11:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.102+0000 7f08348b5640 1 -- 172.21.15.98:0/2241623061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08301a7290 con 0x7f0830106a80 2024-09-19T08:11:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.104+0000 7f081f7fe640 1 -- 172.21.15.98:0/2241623061 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f082801c070 con 0x7f0830106a80 2024-09-19T08:11:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.104+0000 7f0836c8e740 1 Processor -- start 2024-09-19T08:11:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f0836c8e740 1 -- start start 2024-09-19T08:11:41.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f0836c8e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8d9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f0836c8e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8da480 0x55ee5c8e4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f0836c8e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8e7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f0836c8e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ee5c83f280 con 0x55ee5c8d9b60 2024-09-19T08:11:41.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f0836c8e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ee5c7e6b10 con 0x55ee5c8da480 2024-09-19T08:11:41.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f0836c8e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ee5c83b240 con 0x55ee5c8e4d40 2024-09-19T08:11:41.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f082dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8d9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f082dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8d9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43960/0 (socket says 172.21.15.98:43960) 2024-09-19T08:11:41.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f082dd9b640 1 -- 172.21.15.98:0/2875171439 learned_addr learned my addr 172.21.15.98:0/2875171439 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:41.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f081f7fe640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08180a33b0 0x7f08180a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f082e59c640 1 --2- 172.21.15.98:0/2875171439 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8da480 0x55ee5c8e4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f082ed9d640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08180a33b0 0x7f08180a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f082dd9b640 1 --2- 172.21.15.98:0/2875171439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8e7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f082dd9b640 1 -- 172.21.15.98:0/2875171439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8e4d40 msgr2=0x55ee5c8e7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:41.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f082dd9b640 1 --2- 172.21.15.98:0/2875171439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8e7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f082dd9b640 1 -- 172.21.15.98:0/2875171439 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8da480 msgr2=0x55ee5c8e4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:41.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f081f7fe640 1 -- 172.21.15.98:0/2241623061 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f08280cf0e0 con 0x7f0830106a80 2024-09-19T08:11:41.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f082dd9b640 1 --2- 172.21.15.98:0/2875171439 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8da480 0x55ee5c8e4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.105+0000 7f082dd9b640 1 -- 172.21.15.98:0/2875171439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee5c7f31f0 con 0x55ee5c8d9b60 2024-09-19T08:11:41.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.106+0000 7f082dd9b640 1 --2- 172.21.15.98:0/2875171439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8d9f40 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f0824005ec0 tx=0x7f0824005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96064bc8969df42 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:41.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.106+0000 7f082ed9d640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08180a33b0 0x7f08180a5870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f08280cca90 tx=0x7f0828075040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:41.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.106+0000 7f081cff9640 1 -- 172.21.15.98:0/2875171439 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0824035040 con 0x55ee5c8d9b60 2024-09-19T08:11:41.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.106+0000 7f081cff9640 1 -- 172.21.15.98:0/2875171439 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0824005020 con 0x55ee5c8d9b60 2024-09-19T08:11:41.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.106+0000 7f081cff9640 1 -- 172.21.15.98:0/2875171439 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0824004bc0 con 0x55ee5c8d9b60 2024-09-19T08:11:41.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.106+0000 7f0836c8e740 1 -- 172.21.15.98:0/2875171439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 msgr2=0x55ee5c8d9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:41.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.106+0000 7f0836c8e740 1 --2- 172.21.15.98:0/2875171439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8d9f40 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f0824005ec0 tx=0x7f0824005fd0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 -- 172.21.15.98:0/2875171439 shutdown_connections 2024-09-19T08:11:41.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 --2- 172.21.15.98:0/2875171439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8e7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 --2- 172.21.15.98:0/2875171439 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8da480 0x55ee5c8e4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 --2- 172.21.15.98:0/2875171439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8d9f40 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 -- 172.21.15.98:0/2875171439 >> 172.21.15.98:0/2875171439 conn(0x55ee5c8d60f0 msgr2=0x55ee5c8d64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:41.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 -- 172.21.15.98:0/2875171439 shutdown_connections 2024-09-19T08:11:41.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 -- 172.21.15.98:0/2875171439 wait complete. 2024-09-19T08:11:41.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 -- start start 2024-09-19T08:11:41.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 Processor -- start 2024-09-19T08:11:41.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8e1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8da480 0x55ee5c8e2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8e2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ee5c8e7910 con 0x55ee5c8d9b60 2024-09-19T08:11:41.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ee5c7f9f10 con 0x55ee5c8da480 2024-09-19T08:11:41.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ee5c8e7a90 con 0x55ee5c8e4d40 2024-09-19T08:11:41.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f0836c8e740 10 client.0 _ll_register_callbacks cb 0x55ee5c7c2a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:41.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f082dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8e1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.107+0000 7f082dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8e1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43964/0 (socket says 172.21.15.98:43964) 2024-09-19T08:11:41.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.108+0000 7f082dd9b640 1 -- 172.21.15.98:0/4108201831 learned_addr learned my addr 172.21.15.98:0/4108201831 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:41.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.108+0000 7f082dd9b640 1 -- 172.21.15.98:0/4108201831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8e4d40 msgr2=0x55ee5c8e2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:41.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.108+0000 7f082dd9b640 1 --2- 172.21.15.98:0/4108201831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8e2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.108+0000 7f082dd9b640 1 -- 172.21.15.98:0/4108201831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8da480 msgr2=0x55ee5c8e2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:41.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.108+0000 7f082dd9b640 1 --2- 172.21.15.98:0/4108201831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8da480 0x55ee5c8e2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.108+0000 7f082dd9b640 1 -- 172.21.15.98:0/4108201831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee5c5b06a0 con 0x55ee5c8d9b60 2024-09-19T08:11:41.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.108+0000 7f082dd9b640 1 --2- 172.21.15.98:0/4108201831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8e1bc0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f0824000c00 tx=0x7f082402ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3933c35406293f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:41.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.108+0000 7f0815ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:11:41.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.108+0000 7f0815ffb640 1 -- 172.21.15.98:0/4108201831 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082404d070 con 0x55ee5c8d9b60 2024-09-19T08:11:41.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.108+0000 7f0815ffb640 1 -- 172.21.15.98:0/4108201831 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f08240051d0 con 0x55ee5c8d9b60 2024-09-19T08:11:41.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.108+0000 7f0836c8e740 1 -- 172.21.15.98:0/4108201831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0824005020 con 0x55ee5c8d9b60 2024-09-19T08:11:41.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.108+0000 7f0815ffb640 1 -- 172.21.15.98:0/4108201831 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0824035040 con 0x55ee5c8d9b60 2024-09-19T08:11:41.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.108+0000 7f0836c8e740 10 client.17520 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:41.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.108+0000 7f0836c8e740 1 -- 172.21.15.98:0/4108201831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ee5c8f34f0 con 0x55ee5c8d9b60 2024-09-19T08:11:41.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.109+0000 7f0815ffb640 1 -- 172.21.15.98:0/4108201831 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0824005020 con 0x55ee5c8d9b60 2024-09-19T08:11:41.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.109+0000 7f0815ffb640 1 -- 172.21.15.98:0/4108201831 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f08240036f0 con 0x55ee5c8d9b60 2024-09-19T08:11:41.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.109+0000 7f0836c8e740 10 client.17520 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:41.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.109+0000 7f0836c8e740 10 client.17520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:11:41.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.109+0000 7f0836c8e740 20 client.17520 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:41.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.109+0000 7f0836c8e740 1 --2- 172.21.15.98:0/4108201831 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ee5c8f4840 0x55ee5c8f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.109+0000 7f0836c8e740 4 client.17520 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:11:41.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.109+0000 7f0836c8e740 1 -- 172.21.15.98:0/4108201831 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee5c84c7f0 con 0x55ee5c8f4840 2024-09-19T08:11:41.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.109+0000 7f082e59c640 1 --2- 172.21.15.98:0/4108201831 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ee5c8f4840 0x55ee5c8f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.110+0000 7f0815ffb640 10 client.17520 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:41.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.110+0000 7f082e59c640 1 --2- 172.21.15.98:0/4108201831 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ee5c8f4840 0x55ee5c8f6c30 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f68692c5a0054c99 server_cookie=fab0f6eb8585b5db in_seq=0 out_seq=0 2024-09-19T08:11:41.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.110+0000 7f0815ffb640 1 -- 172.21.15.98:0/4108201831 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ee5c84c7f0 con 0x55ee5c8f4840 2024-09-19T08:11:41.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.110+0000 7f0815ffb640 10 client.17520 handle_command_reply: tid=0 2024-09-19T08:11:41.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.110+0000 7f0836c8e740 1 client.17520 shutdown 2024-09-19T08:11:41.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.110+0000 7f0836c8e740 1 -- 172.21.15.98:0/4108201831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 msgr2=0x55ee5c8e1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:41.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.110+0000 7f0836c8e740 1 --2- 172.21.15.98:0/4108201831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8e1bc0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f0824000c00 tx=0x7f082402ab80 comp rx=0 tx=0).stop 2024-09-19T08:11:41.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.110+0000 7f0836c8e740 1 -- 172.21.15.98:0/4108201831 shutdown_connections 2024-09-19T08:11:41.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.110+0000 7f0836c8e740 1 --2- 172.21.15.98:0/4108201831 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ee5c8f4840 0x55ee5c8f6c30 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.110+0000 7f0836c8e740 1 --2- 172.21.15.98:0/4108201831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8e2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.110+0000 7f0836c8e740 1 --2- 172.21.15.98:0/4108201831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8da480 0x55ee5c8e2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.110+0000 7f0836c8e740 1 --2- 172.21.15.98:0/4108201831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8e1bc0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.110+0000 7f0836c8e740 1 -- 172.21.15.98:0/4108201831 >> 172.21.15.98:0/4108201831 conn(0x55ee5c8d60f0 msgr2=0x55ee5c8dbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:41.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.110+0000 7f0815ffb640 0 client.17520 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:41.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.111+0000 7f0836c8e740 1 -- 172.21.15.98:0/4108201831 shutdown_connections 2024-09-19T08:11:41.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.111+0000 7f0836c8e740 1 -- 172.21.15.98:0/4108201831 wait complete. 2024-09-19T08:11:41.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.111+0000 7f0836c8e740 20 client.17520 trim_cache size 0 max 16384 2024-09-19T08:11:41.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.127+0000 7f0836c8e740 1 Processor -- start 2024-09-19T08:11:41.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f0836c8e740 1 -- start start 2024-09-19T08:11:41.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f0836c8e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8dc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f0836c8e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8da480 0x55ee5c8e3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f0836c8e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8fc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f0836c8e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08301afac0 con 0x55ee5c8d9b60 2024-09-19T08:11:41.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f0836c8e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f08301b1600 con 0x55ee5c8da480 2024-09-19T08:11:41.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f0836c8e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08301b0190 con 0x55ee5c8e4d40 2024-09-19T08:11:41.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f082dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8dc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f082dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8dc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43972/0 (socket says 172.21.15.98:43972) 2024-09-19T08:11:41.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f082dd9b640 1 -- 172.21.15.98:0/284668621 learned_addr learned my addr 172.21.15.98:0/284668621 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:41.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f082e59c640 1 --2- 172.21.15.98:0/284668621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8da480 0x55ee5c8e3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f082dd9b640 1 -- 172.21.15.98:0/284668621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8e4d40 msgr2=0x55ee5c8fc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:41.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f082dd9b640 1 --2- 172.21.15.98:0/284668621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f082dd9b640 1 -- 172.21.15.98:0/284668621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8da480 msgr2=0x55ee5c8e3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:41.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f082dd9b640 1 --2- 172.21.15.98:0/284668621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8da480 0x55ee5c8e3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f082dd9b640 1 -- 172.21.15.98:0/284668621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee5c8d1830 con 0x55ee5c8d9b60 2024-09-19T08:11:41.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f082dd9b640 1 --2- 172.21.15.98:0/284668621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8dc5c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f0824031b70 tx=0x7f0824031ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf088c79985dda69 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:41.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f0817fff640 1 -- 172.21.15.98:0/284668621 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082405b030 con 0x55ee5c8d9b60 2024-09-19T08:11:41.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f0817fff640 1 -- 172.21.15.98:0/284668621 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0824031750 con 0x55ee5c8d9b60 2024-09-19T08:11:41.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.128+0000 7f0817fff640 1 -- 172.21.15.98:0/284668621 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082404d040 con 0x55ee5c8d9b60 2024-09-19T08:11:41.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.129+0000 7f0836c8e740 1 -- 172.21.15.98:0/284668621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 msgr2=0x55ee5c8dc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:41.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.129+0000 7f0836c8e740 1 --2- 172.21.15.98:0/284668621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8dc5c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f0824031b70 tx=0x7f0824031ba0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.129+0000 7f0836c8e740 1 -- 172.21.15.98:0/284668621 shutdown_connections 2024-09-19T08:11:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.129+0000 7f0836c8e740 1 --2- 172.21.15.98:0/284668621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8fc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.129+0000 7f0836c8e740 1 --2- 172.21.15.98:0/284668621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8da480 0x55ee5c8e3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.129+0000 7f0836c8e740 1 --2- 172.21.15.98:0/284668621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8dc5c0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.129+0000 7f0836c8e740 1 -- 172.21.15.98:0/284668621 >> 172.21.15.98:0/284668621 conn(0x55ee5c8d60f0 msgr2=0x55ee5c8e3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.129+0000 7f0836c8e740 1 -- 172.21.15.98:0/284668621 shutdown_connections 2024-09-19T08:11:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.129+0000 7f0836c8e740 1 -- 172.21.15.98:0/284668621 wait complete. 2024-09-19T08:11:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.129+0000 7f0836c8e740 1 -- start start 2024-09-19T08:11:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.129+0000 7f0836c8e740 1 Processor -- start 2024-09-19T08:11:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f0836c8e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8ed320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f0836c8e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8da480 0x55ee5c8de830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f0836c8e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8ded70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f0836c8e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08301b0f30 con 0x55ee5c8e4d40 2024-09-19T08:11:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f0836c8e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f08301b0860 con 0x55ee5c8d9b60 2024-09-19T08:11:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f0836c8e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08301adac0 con 0x55ee5c8da480 2024-09-19T08:11:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f0836c8e740 10 client.0 _ll_register_callbacks cb 0x7f08200071a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f082dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8ded70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f082dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8ded70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43978/0 (socket says 172.21.15.98:43978) 2024-09-19T08:11:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f082dd9b640 1 -- 172.21.15.98:0/3839762887 learned_addr learned my addr 172.21.15.98:0/3839762887 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f082e59c640 1 --2- 172.21.15.98:0/3839762887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8da480 0x55ee5c8de830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f082dd9b640 1 --2- 172.21.15.98:0/3839762887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8ed320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f082e59c640 1 -- 172.21.15.98:0/3839762887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8d9b60 msgr2=0x55ee5c8ed320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f082e59c640 1 --2- 172.21.15.98:0/3839762887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8ed320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f082e59c640 1 -- 172.21.15.98:0/3839762887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8e4d40 msgr2=0x55ee5c8ded70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f082e59c640 1 --2- 172.21.15.98:0/3839762887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8ded70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f082e59c640 1 -- 172.21.15.98:0/3839762887 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee5c8df480 con 0x55ee5c8da480 2024-09-19T08:11:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f082dd9b640 1 --2- 172.21.15.98:0/3839762887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8ded70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:11:41.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f082e59c640 1 --2- 172.21.15.98:0/3839762887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8da480 0x55ee5c8de830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x55ee5c8f7200 tx=0x7f0820007e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cce0305b71902f97 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:41.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.130+0000 7f0815ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-09-19T08:11:41.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f0815ffb640 1 -- 172.21.15.98:0/3839762887 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0820014b30 con 0x55ee5c8da480 2024-09-19T08:11:41.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f0815ffb640 1 -- 172.21.15.98:0/3839762887 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f0820014cd0 con 0x55ee5c8da480 2024-09-19T08:11:41.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f0836c8e740 1 -- 172.21.15.98:0/3839762887 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0824031750 con 0x55ee5c8da480 2024-09-19T08:11:41.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f0815ffb640 1 -- 172.21.15.98:0/3839762887 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0820023480 con 0x55ee5c8da480 2024-09-19T08:11:41.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f0836c8e740 10 client.26428 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:41.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f0836c8e740 1 -- 172.21.15.98:0/3839762887 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ee5c95f1f0 con 0x55ee5c8da480 2024-09-19T08:11:41.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f0815ffb640 1 -- 172.21.15.98:0/3839762887 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0824031750 con 0x55ee5c8da480 2024-09-19T08:11:41.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f0815ffb640 1 -- 172.21.15.98:0/3839762887 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f082002b590 con 0x55ee5c8da480 2024-09-19T08:11:41.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f0836c8e740 10 client.26428 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:41.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f0836c8e740 10 client.26428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:11:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f0836c8e740 20 client.26428 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f0836c8e740 1 --2- 172.21.15.98:0/3839762887 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ee5c95f920 0x55ee5c961d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f0836c8e740 4 client.26428 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:11:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f0836c8e740 1 -- 172.21.15.98:0/3839762887 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ee5c846d00 con 0x55ee5c95f920 2024-09-19T08:11:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.131+0000 7f082dd9b640 1 --2- 172.21.15.98:0/3839762887 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ee5c95f920 0x55ee5c961d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.132+0000 7f082dd9b640 1 --2- 172.21.15.98:0/3839762887 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ee5c95f920 0x55ee5c961d40 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8266961f307b9343 server_cookie=5c140fb661d8534d in_seq=0 out_seq=0 2024-09-19T08:11:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.132+0000 7f0815ffb640 10 client.26428 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0815ffb640 1 -- 172.21.15.98:0/3839762887 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ee5c846d00 con 0x55ee5c95f920 2024-09-19T08:11:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0815ffb640 10 client.26428 handle_command_reply: tid=0 2024-09-19T08:11:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0836c8e740 1 client.26428 shutdown 2024-09-19T08:11:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0836c8e740 1 -- 172.21.15.98:0/3839762887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8da480 msgr2=0x55ee5c8de830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0836c8e740 1 --2- 172.21.15.98:0/3839762887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8da480 0x55ee5c8de830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x55ee5c8f7200 tx=0x7f0820007e60 comp rx=0 tx=0).stop 2024-09-19T08:11:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0836c8e740 1 -- 172.21.15.98:0/3839762887 shutdown_connections 2024-09-19T08:11:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0836c8e740 1 --2- 172.21.15.98:0/3839762887 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55ee5c95f920 0x55ee5c961d40 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0836c8e740 1 --2- 172.21.15.98:0/3839762887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee5c8e4d40 0x55ee5c8ded70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0836c8e740 1 --2- 172.21.15.98:0/3839762887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee5c8da480 0x55ee5c8de830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0836c8e740 1 --2- 172.21.15.98:0/3839762887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee5c8d9b60 0x55ee5c8ed320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0836c8e740 1 -- 172.21.15.98:0/3839762887 >> 172.21.15.98:0/3839762887 conn(0x55ee5c8d60f0 msgr2=0x55ee5c8e23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:41.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0815ffb640 0 client.26428 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:41.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0836c8e740 1 -- 172.21.15.98:0/3839762887 shutdown_connections 2024-09-19T08:11:41.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0836c8e740 1 -- 172.21.15.98:0/3839762887 wait complete. 2024-09-19T08:11:41.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.133+0000 7f0836c8e740 20 client.26428 trim_cache size 0 max 16384 2024-09-19T08:11:41.138 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:11:41.138 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:11:41.139 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "56c41dbf-aa2d-457f-9739-1f07f0d4c518", 2024-09-19T08:11:41.139 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:11:41.139 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:11:41.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.134+0000 7f08348b5640 1 -- 172.21.15.98:0/2241623061 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08180a33b0 msgr2=0x7f08180a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:41.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.134+0000 7f08348b5640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08180a33b0 0x7f08180a5870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f08280cca90 tx=0x7f0828075040 comp rx=0 tx=0).stop 2024-09-19T08:11:41.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.134+0000 7f08348b5640 1 -- 172.21.15.98:0/2241623061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0830106a80 msgr2=0x7f0830197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:41.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.134+0000 7f08348b5640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0830106a80 0x7f0830197ca0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f082800b970 tx=0x7f082800be30 comp rx=0 tx=0).stop 2024-09-19T08:11:41.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.134+0000 7f08348b5640 1 -- 172.21.15.98:0/2241623061 shutdown_connections 2024-09-19T08:11:41.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.134+0000 7f08348b5640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08180a33b0 0x7f08180a5870 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.134+0000 7f08348b5640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0830106a80 0x7f0830197ca0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.134+0000 7f08348b5640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0830101420 0x7f0830197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.134+0000 7f08348b5640 1 --2- 172.21.15.98:0/2241623061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0830100a80 0x7f0830197220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:41.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.134+0000 7f08348b5640 1 -- 172.21.15.98:0/2241623061 >> 172.21.15.98:0/2241623061 conn(0x7f08300fc7a0 msgr2=0x7f08300fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:41.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.134+0000 7f08348b5640 1 -- 172.21.15.98:0/2241623061 shutdown_connections 2024-09-19T08:11:41.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:41.134+0000 7f08348b5640 1 -- 172.21.15.98:0/2241623061 wait complete. 2024-09-19T08:11:41.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:11:42.065 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:11:42.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:41 smithi098 ceph-mon[31333]: pgmap v267: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-09-19T08:11:42.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:41 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:11:42.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:41 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:11:42.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:41 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:11:42.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:41 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:11:42.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:41 smithi162 ceph-mon[35260]: pgmap v267: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-09-19T08:11:42.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:41 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:11:42.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:41 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:11:42.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:41 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:11:42.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:41 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:11:42.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:41 smithi139 ceph-mon[34874]: pgmap v267: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-09-19T08:11:42.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:41 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:11:42.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:41 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:11:42.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:41 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:11:42.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:41 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:11:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.574+0000 7fcb76834640 1 -- 172.21.15.98:0/414808217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb7010e440 msgr2=0x7fcb70110830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.574+0000 7fcb76834640 1 --2- 172.21.15.98:0/414808217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb7010e440 0x7fcb70110830 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009ca0 tx=0x7fcb6802a950 comp rx=0 tx=0).stop 2024-09-19T08:11:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.574+0000 7fcb76834640 1 -- 172.21.15.98:0/414808217 shutdown_connections 2024-09-19T08:11:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.574+0000 7fcb76834640 1 --2- 172.21.15.98:0/414808217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb7010e440 0x7fcb70110830 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.574+0000 7fcb76834640 1 --2- 172.21.15.98:0/414808217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb70074e30 0x7fcb7010df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.574+0000 7fcb76834640 1 --2- 172.21.15.98:0/414808217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb70074510 0x7fcb700748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.574+0000 7fcb76834640 1 -- 172.21.15.98:0/414808217 >> 172.21.15.98:0/414808217 conn(0x7fcb700fd8c0 msgr2=0x7fcb700ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:42.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.574+0000 7fcb76834640 1 -- 172.21.15.98:0/414808217 shutdown_connections 2024-09-19T08:11:42.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.574+0000 7fcb76834640 1 -- 172.21.15.98:0/414808217 wait complete. 2024-09-19T08:11:42.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.575+0000 7fcb76834640 1 Processor -- start 2024-09-19T08:11:42.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.575+0000 7fcb76834640 1 -- start start 2024-09-19T08:11:42.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.575+0000 7fcb76834640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb70074510 0x7fcb7019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.575+0000 7fcb76834640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb70074e30 0x7fcb7019ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.575+0000 7fcb76834640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb7010e440 0x7fcb7019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.575+0000 7fcb76834640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb70114160 con 0x7fcb70074e30 2024-09-19T08:11:42.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.575+0000 7fcb76834640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcb70113fe0 con 0x7fcb70074510 2024-09-19T08:11:42.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.575+0000 7fcb76834640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcb701142e0 con 0x7fcb7010e440 2024-09-19T08:11:42.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.575+0000 7fcb6f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb70074e30 0x7fcb7019ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.575+0000 7fcb6f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb70074e30 0x7fcb7019ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43988/0 (socket says 172.21.15.98:43988) 2024-09-19T08:11:42.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.575+0000 7fcb6f7fe640 1 -- 172.21.15.98:0/1454717553 learned_addr learned my addr 172.21.15.98:0/1454717553 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:42.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.575+0000 7fcb6ffff640 1 --2- 172.21.15.98:0/1454717553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb70074510 0x7fcb7019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.575+0000 7fcb74dd1640 1 --2- 172.21.15.98:0/1454717553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb7010e440 0x7fcb7019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.576+0000 7fcb6f7fe640 1 -- 172.21.15.98:0/1454717553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb7010e440 msgr2=0x7fcb7019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:42.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.576+0000 7fcb6f7fe640 1 --2- 172.21.15.98:0/1454717553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb7010e440 0x7fcb7019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.576+0000 7fcb6f7fe640 1 -- 172.21.15.98:0/1454717553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb70074510 msgr2=0x7fcb7019a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:42.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.576+0000 7fcb6f7fe640 1 --2- 172.21.15.98:0/1454717553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb70074510 0x7fcb7019a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.576+0000 7fcb6f7fe640 1 -- 172.21.15.98:0/1454717553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7019f5a0 con 0x7fcb70074e30 2024-09-19T08:11:42.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.576+0000 7fcb6f7fe640 1 --2- 172.21.15.98:0/1454717553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb70074e30 0x7fcb7019ab30 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800c920 tx=0x7fcb5800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:42.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.576+0000 7fcb6d7fa640 1 -- 172.21.15.98:0/1454717553 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58007c70 con 0x7fcb70074e30 2024-09-19T08:11:42.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.576+0000 7fcb76834640 1 -- 172.21.15.98:0/1454717553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb7019f890 con 0x7fcb70074e30 2024-09-19T08:11:42.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.576+0000 7fcb6d7fa640 1 -- 172.21.15.98:0/1454717553 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb58007e10 con 0x7fcb70074e30 2024-09-19T08:11:42.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.576+0000 7fcb6d7fa640 1 -- 172.21.15.98:0/1454717553 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58018890 con 0x7fcb70074e30 2024-09-19T08:11:42.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.576+0000 7fcb76834640 1 -- 172.21.15.98:0/1454717553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb701a7170 con 0x7fcb70074e30 2024-09-19T08:11:42.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.579+0000 7fcb6d7fa640 1 -- 172.21.15.98:0/1454717553 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fcb58018a50 con 0x7fcb70074e30 2024-09-19T08:11:42.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.579+0000 7fcb78c0d740 1 Processor -- start 2024-09-19T08:11:42.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.579+0000 7fcb78c0d740 1 -- start start 2024-09-19T08:11:42.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.579+0000 7fcb78c0d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b5564a0 0x55af8b556880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.579+0000 7fcb78c0d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b561140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.579+0000 7fcb78c0d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b561680 0x55af8b563a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.579+0000 7fcb78c0d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55af8b4bc2e0 con 0x55af8b556dc0 2024-09-19T08:11:42.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.579+0000 7fcb78c0d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55af8b463ce0 con 0x55af8b561680 2024-09-19T08:11:42.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.579+0000 7fcb78c0d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55af8b4b82a0 con 0x55af8b5564a0 2024-09-19T08:11:42.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.579+0000 7fcb6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b561140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.579+0000 7fcb6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b561140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43996/0 (socket says 172.21.15.98:43996) 2024-09-19T08:11:42.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.579+0000 7fcb6ffff640 1 -- 172.21.15.98:0/4064566729 learned_addr learned my addr 172.21.15.98:0/4064566729 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:42.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.579+0000 7fcb74dd1640 1 --2- 172.21.15.98:0/4064566729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b5564a0 0x55af8b556880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.579+0000 7fcb6f7fe640 1 --2- 172.21.15.98:0/4064566729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b561680 0x55af8b563a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.580+0000 7fcb6d7fa640 1 --2- 172.21.15.98:0/1454717553 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcb5c0a33b0 0x7fcb5c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.580+0000 7fcb6ffff640 1 -- 172.21.15.98:0/4064566729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b5564a0 msgr2=0x55af8b556880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:42.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.580+0000 7fcb6ffff640 1 --2- 172.21.15.98:0/4064566729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b5564a0 0x55af8b556880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.580+0000 7fcb6ffff640 1 -- 172.21.15.98:0/4064566729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b561680 msgr2=0x55af8b563a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:42.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.580+0000 7fcb6ffff640 1 --2- 172.21.15.98:0/4064566729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b561680 0x55af8b563a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.580+0000 7fcb6ffff640 1 -- 172.21.15.98:0/4064566729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af8b448440 con 0x55af8b556dc0 2024-09-19T08:11:42.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.580+0000 7fcb6f7fe640 1 --2- 172.21.15.98:0/4064566729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b561680 0x55af8b563a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:11:42.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.580+0000 7fcb74dd1640 1 --2- 172.21.15.98:0/1454717553 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcb5c0a33b0 0x7fcb5c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.580+0000 7fcb6d7fa640 1 -- 172.21.15.98:0/1454717553 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fcb580cfea0 con 0x7fcb70074e30 2024-09-19T08:11:42.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.580+0000 7fcb6ffff640 1 --2- 172.21.15.98:0/4064566729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b561140 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000cbe0 tx=0x7fcb6000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=691b05f335dcc54c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:42.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.580+0000 7fcb667fc640 1 -- 172.21.15.98:0/4064566729 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb60010630 con 0x55af8b556dc0 2024-09-19T08:11:42.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.580+0000 7fcb667fc640 1 -- 172.21.15.98:0/4064566729 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb60010cb0 con 0x55af8b556dc0 2024-09-19T08:11:42.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.580+0000 7fcb667fc640 1 -- 172.21.15.98:0/4064566729 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6001ac20 con 0x55af8b556dc0 2024-09-19T08:11:42.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.581+0000 7fcb78c0d740 1 -- 172.21.15.98:0/4064566729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 msgr2=0x55af8b561140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:42.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.581+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/4064566729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b561140 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000cbe0 tx=0x7fcb6000e720 comp rx=0 tx=0).stop 2024-09-19T08:11:42.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.581+0000 7fcb74dd1640 1 --2- 172.21.15.98:0/1454717553 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcb5c0a33b0 0x7fcb5c0a5870 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009430 tx=0x7fcb680095c0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:42.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.581+0000 7fcb78c0d740 1 -- 172.21.15.98:0/4064566729 shutdown_connections 2024-09-19T08:11:42.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.581+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/4064566729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b561680 0x55af8b563a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.581+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/4064566729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b561140 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.581+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/4064566729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b5564a0 0x55af8b556880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.581+0000 7fcb78c0d740 1 -- 172.21.15.98:0/4064566729 >> 172.21.15.98:0/4064566729 conn(0x55af8b552a30 msgr2=0x55af8b552e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:42.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.581+0000 7fcb78c0d740 1 -- 172.21.15.98:0/4064566729 shutdown_connections 2024-09-19T08:11:42.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.581+0000 7fcb78c0d740 1 -- 172.21.15.98:0/4064566729 wait complete. 2024-09-19T08:11:42.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.581+0000 7fcb78c0d740 1 -- start start 2024-09-19T08:11:42.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.581+0000 7fcb78c0d740 1 Processor -- start 2024-09-19T08:11:42.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb78c0d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b5564a0 0x55af8b560f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb78c0d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b556dc0 0x55af8b55e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb78c0d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b561680 0x55af8b55e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb78c0d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55af8b564250 con 0x55af8b5564a0 2024-09-19T08:11:42.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb78c0d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55af8b477050 con 0x55af8b561680 2024-09-19T08:11:42.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb78c0d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55af8b5643d0 con 0x55af8b556dc0 2024-09-19T08:11:42.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb78c0d740 10 client.0 _ll_register_callbacks cb 0x55af8b4b7160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:42.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b5564a0 0x55af8b560f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b5564a0 0x55af8b560f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44010/0 (socket says 172.21.15.98:44010) 2024-09-19T08:11:42.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb6ffff640 1 -- 172.21.15.98:0/3778652431 learned_addr learned my addr 172.21.15.98:0/3778652431 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:42.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb6f7fe640 1 --2- 172.21.15.98:0/3778652431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b556dc0 0x55af8b55e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb74dd1640 1 --2- 172.21.15.98:0/3778652431 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b561680 0x55af8b55e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb6ffff640 1 -- 172.21.15.98:0/3778652431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b556dc0 msgr2=0x55af8b55e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:42.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb6ffff640 1 --2- 172.21.15.98:0/3778652431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b556dc0 0x55af8b55e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb6ffff640 1 -- 172.21.15.98:0/3778652431 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b561680 msgr2=0x55af8b55e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:42.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb6ffff640 1 --2- 172.21.15.98:0/3778652431 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b561680 0x55af8b55e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb6ffff640 1 -- 172.21.15.98:0/3778652431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af8b22d3c0 con 0x55af8b5564a0 2024-09-19T08:11:42.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb6ffff640 1 --2- 172.21.15.98:0/3778652431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b5564a0 0x55af8b560f50 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000f010 tx=0x7fcb6000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f44102cbe00d7c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:42.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:11:42.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb4f7fe640 1 -- 172.21.15.98:0/3778652431 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb60010e60 con 0x55af8b5564a0 2024-09-19T08:11:42.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb4f7fe640 1 -- 172.21.15.98:0/3778652431 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fcb6001acd0 con 0x55af8b5564a0 2024-09-19T08:11:42.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb78c0d740 1 -- 172.21.15.98:0/3778652431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb60010cb0 con 0x55af8b5564a0 2024-09-19T08:11:42.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb4f7fe640 1 -- 172.21.15.98:0/3778652431 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcb60004020 con 0x55af8b5564a0 2024-09-19T08:11:42.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb78c0d740 10 client.17538 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:42.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.582+0000 7fcb78c0d740 1 -- 172.21.15.98:0/3778652431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55af8b56fb10 con 0x55af8b5564a0 2024-09-19T08:11:42.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.583+0000 7fcb4f7fe640 1 -- 172.21.15.98:0/3778652431 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb60010cb0 con 0x55af8b5564a0 2024-09-19T08:11:42.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.583+0000 7fcb4f7fe640 1 -- 172.21.15.98:0/3778652431 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcb60003680 con 0x55af8b5564a0 2024-09-19T08:11:42.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.583+0000 7fcb78c0d740 10 client.17538 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:42.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.583+0000 7fcb78c0d740 10 client.17538 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:11:42.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.583+0000 7fcb78c0d740 20 client.17538 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:42.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.583+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/3778652431 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55af8b570f70 0x55af8b573360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.583+0000 7fcb78c0d740 4 client.17538 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:11:42.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.583+0000 7fcb78c0d740 1 -- 172.21.15.98:0/3778652431 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af8b4c9850 con 0x55af8b570f70 2024-09-19T08:11:42.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.583+0000 7fcb6f7fe640 1 --2- 172.21.15.98:0/3778652431 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55af8b570f70 0x55af8b573360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.584+0000 7fcb4f7fe640 10 client.17538 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:42.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.584+0000 7fcb6f7fe640 1 --2- 172.21.15.98:0/3778652431 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55af8b570f70 0x55af8b573360 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29a582d93bcd8bbe server_cookie=fa07f62d594a4442 in_seq=0 out_seq=0 2024-09-19T08:11:42.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.584+0000 7fcb4f7fe640 1 -- 172.21.15.98:0/3778652431 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55af8b4c9850 con 0x55af8b570f70 2024-09-19T08:11:42.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.584+0000 7fcb4f7fe640 10 client.17538 handle_command_reply: tid=0 2024-09-19T08:11:42.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.584+0000 7fcb78c0d740 1 client.17538 shutdown 2024-09-19T08:11:42.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.585+0000 7fcb78c0d740 1 -- 172.21.15.98:0/3778652431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b5564a0 msgr2=0x55af8b560f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:42.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.585+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/3778652431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b5564a0 0x55af8b560f50 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000f010 tx=0x7fcb6000c920 comp rx=0 tx=0).stop 2024-09-19T08:11:42.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.585+0000 7fcb78c0d740 1 -- 172.21.15.98:0/3778652431 shutdown_connections 2024-09-19T08:11:42.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.585+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/3778652431 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55af8b570f70 0x55af8b573360 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.585+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/3778652431 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b561680 0x55af8b55e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.585+0000 7fcb4f7fe640 0 client.17538 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:42.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.585+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/3778652431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b556dc0 0x55af8b55e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.585+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/3778652431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b5564a0 0x55af8b560f50 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.585+0000 7fcb78c0d740 1 -- 172.21.15.98:0/3778652431 >> 172.21.15.98:0/3778652431 conn(0x55af8b552a30 msgr2=0x55af8b558710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:42.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.585+0000 7fcb78c0d740 1 -- 172.21.15.98:0/3778652431 shutdown_connections 2024-09-19T08:11:42.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.585+0000 7fcb78c0d740 1 -- 172.21.15.98:0/3778652431 wait complete. 2024-09-19T08:11:42.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.585+0000 7fcb78c0d740 20 client.17538 trim_cache size 0 max 16384 2024-09-19T08:11:42.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.601+0000 7fcb78c0d740 1 Processor -- start 2024-09-19T08:11:42.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.601+0000 7fcb78c0d740 1 -- start start 2024-09-19T08:11:42.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb78c0d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b5564a0 0x55af8b56d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb78c0d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b578960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb78c0d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b561680 0x55af8b578ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb78c0d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb701afad0 con 0x55af8b556dc0 2024-09-19T08:11:42.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb78c0d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcb701b1610 con 0x55af8b5564a0 2024-09-19T08:11:42.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb78c0d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcb701b01a0 con 0x55af8b561680 2024-09-19T08:11:42.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb6f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b578960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb6f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b578960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44022/0 (socket says 172.21.15.98:44022) 2024-09-19T08:11:42.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb6f7fe640 1 -- 172.21.15.98:0/2694492250 learned_addr learned my addr 172.21.15.98:0/2694492250 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:42.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb74dd1640 1 --2- 172.21.15.98:0/2694492250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b561680 0x55af8b578ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb6ffff640 1 --2- 172.21.15.98:0/2694492250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b5564a0 0x55af8b56d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb6f7fe640 1 -- 172.21.15.98:0/2694492250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b561680 msgr2=0x55af8b578ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:42.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb6f7fe640 1 --2- 172.21.15.98:0/2694492250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b561680 0x55af8b578ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb6f7fe640 1 -- 172.21.15.98:0/2694492250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b5564a0 msgr2=0x55af8b56d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:42.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb6f7fe640 1 --2- 172.21.15.98:0/2694492250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b5564a0 0x55af8b56d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb6f7fe640 1 -- 172.21.15.98:0/2694492250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af8b552860 con 0x55af8b556dc0 2024-09-19T08:11:42.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb6ffff640 1 --2- 172.21.15.98:0/2694492250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b5564a0 0x55af8b56d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:11:42.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb6f7fe640 1 --2- 172.21.15.98:0/2694492250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b578960 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fcb5809c110 tx=0x7fcb580d23b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8919378e88ba52dd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:42.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb65ffb640 1 -- 172.21.15.98:0/2694492250 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb580b4070 con 0x55af8b556dc0 2024-09-19T08:11:42.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb65ffb640 1 -- 172.21.15.98:0/2694492250 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb580a0530 con 0x55af8b556dc0 2024-09-19T08:11:42.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.602+0000 7fcb65ffb640 1 -- 172.21.15.98:0/2694492250 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5800fc50 con 0x55af8b556dc0 2024-09-19T08:11:42.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.603+0000 7fcb78c0d740 1 -- 172.21.15.98:0/2694492250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 msgr2=0x55af8b578960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:42.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.603+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/2694492250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b578960 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fcb5809c110 tx=0x7fcb580d23b0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.603+0000 7fcb78c0d740 1 -- 172.21.15.98:0/2694492250 shutdown_connections 2024-09-19T08:11:42.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.603+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/2694492250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b561680 0x55af8b578ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.603+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/2694492250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b578960 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.603+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/2694492250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b5564a0 0x55af8b56d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.603+0000 7fcb78c0d740 1 -- 172.21.15.98:0/2694492250 >> 172.21.15.98:0/2694492250 conn(0x55af8b552a30 msgr2=0x55af8b55d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:42.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.603+0000 7fcb78c0d740 1 -- 172.21.15.98:0/2694492250 shutdown_connections 2024-09-19T08:11:42.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.603+0000 7fcb78c0d740 1 -- 172.21.15.98:0/2694492250 wait complete. 2024-09-19T08:11:42.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.603+0000 7fcb78c0d740 1 -- start start 2024-09-19T08:11:42.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb78c0d740 1 Processor -- start 2024-09-19T08:11:42.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb78c0d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b5564a0 0x55af8b569b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb78c0d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b566e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb78c0d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b561680 0x55af8b567390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb78c0d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb701b0f40 con 0x55af8b556dc0 2024-09-19T08:11:42.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb78c0d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcb701b0870 con 0x55af8b5564a0 2024-09-19T08:11:42.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb78c0d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcb701adad0 con 0x55af8b561680 2024-09-19T08:11:42.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb78c0d740 10 client.0 _ll_register_callbacks cb 0x7fcb60014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:11:42.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb6f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b566e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb6f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b566e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44024/0 (socket says 172.21.15.98:44024) 2024-09-19T08:11:42.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb6f7fe640 1 -- 172.21.15.98:0/910603427 learned_addr learned my addr 172.21.15.98:0/910603427 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:11:42.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb74dd1640 1 --2- 172.21.15.98:0/910603427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b561680 0x55af8b567390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb6ffff640 1 --2- 172.21.15.98:0/910603427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b5564a0 0x55af8b569b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb6f7fe640 1 -- 172.21.15.98:0/910603427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b561680 msgr2=0x55af8b567390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:42.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb6f7fe640 1 --2- 172.21.15.98:0/910603427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b561680 0x55af8b567390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb6f7fe640 1 -- 172.21.15.98:0/910603427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b5564a0 msgr2=0x55af8b569b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:11:42.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb6f7fe640 1 --2- 172.21.15.98:0/910603427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b5564a0 0x55af8b569b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.604+0000 7fcb6f7fe640 1 -- 172.21.15.98:0/910603427 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af8b567950 con 0x55af8b556dc0 2024-09-19T08:11:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb6f7fe640 1 --2- 172.21.15.98:0/910603427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b566e50 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fcb580a48c0 tx=0x7fcb580a4e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=787699b882f39ca4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:11:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:11:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb4f7fe640 1 -- 172.21.15.98:0/910603427 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58066060 con 0x55af8b556dc0 2024-09-19T08:11:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb4f7fe640 1 -- 172.21.15.98:0/910603427 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fcb580cd870 con 0x55af8b556dc0 2024-09-19T08:11:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb78c0d740 1 -- 172.21.15.98:0/910603427 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb580a0530 con 0x55af8b556dc0 2024-09-19T08:11:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb4f7fe640 1 -- 172.21.15.98:0/910603427 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcb580b4070 con 0x55af8b556dc0 2024-09-19T08:11:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb78c0d740 10 client.17544 fetch_fsmap learned FSMap version 32 2024-09-19T08:11:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb78c0d740 1 -- 172.21.15.98:0/910603427 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55af8b5db650 con 0x55af8b556dc0 2024-09-19T08:11:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb4f7fe640 1 -- 172.21.15.98:0/910603427 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb580a0530 con 0x55af8b556dc0 2024-09-19T08:11:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb4f7fe640 1 -- 172.21.15.98:0/910603427 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcb580a4960 con 0x55af8b556dc0 2024-09-19T08:11:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb78c0d740 10 client.17544 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:11:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb78c0d740 10 client.17544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:11:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb78c0d740 20 client.17544 populate_metadata read hostname 'smithi098' 2024-09-19T08:11:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/910603427 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55af8b5dbdb0 0x55af8b5de1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:11:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb78c0d740 4 client.17544 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:11:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.605+0000 7fcb78c0d740 1 -- 172.21.15.98:0/910603427 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55af8b4c3d60 con 0x55af8b5dbdb0 2024-09-19T08:11:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb6ffff640 1 --2- 172.21.15.98:0/910603427 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55af8b5dbdb0 0x55af8b5de1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:11:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb6ffff640 1 --2- 172.21.15.98:0/910603427 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55af8b5dbdb0 0x55af8b5de1d0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6098f9db3414f5a2 server_cookie=658bb6e458fb3a4f in_seq=0 out_seq=0 2024-09-19T08:11:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb4f7fe640 10 client.17544 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb4f7fe640 1 -- 172.21.15.98:0/910603427 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55af8b4c3d60 con 0x55af8b5dbdb0 2024-09-19T08:11:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb4f7fe640 10 client.17544 handle_command_reply: tid=0 2024-09-19T08:11:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb78c0d740 1 client.17544 shutdown 2024-09-19T08:11:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb78c0d740 1 -- 172.21.15.98:0/910603427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 msgr2=0x55af8b566e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/910603427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b566e50 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fcb580a48c0 tx=0x7fcb580a4e60 comp rx=0 tx=0).stop 2024-09-19T08:11:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb78c0d740 1 -- 172.21.15.98:0/910603427 shutdown_connections 2024-09-19T08:11:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/910603427 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55af8b5dbdb0 0x55af8b5de1d0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/910603427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55af8b561680 0x55af8b567390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/910603427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af8b556dc0 0x55af8b566e50 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb4f7fe640 0 client.17544 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:11:42.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb78c0d740 1 --2- 172.21.15.98:0/910603427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55af8b5564a0 0x55af8b569b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb78c0d740 1 -- 172.21.15.98:0/910603427 >> 172.21.15.98:0/910603427 conn(0x55af8b552a30 msgr2=0x55af8b551350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:42.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb78c0d740 1 -- 172.21.15.98:0/910603427 shutdown_connections 2024-09-19T08:11:42.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.606+0000 7fcb78c0d740 1 -- 172.21.15.98:0/910603427 wait complete. 2024-09-19T08:11:42.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.607+0000 7fcb78c0d740 20 client.17544 trim_cache size 0 max 16384 2024-09-19T08:11:42.615 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:11:42.616 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-19T08:11:42.616 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-09-19T08:11:42.616 INFO:teuthology.orchestra.run.smithi098.stdout: "56c41dbf-aa2d-457f-9739-1f07f0d4c518": { 2024-09-19T08:11:42.616 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-09-19T08:11:42.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "56c41dbf-aa2d-457f-9739-1f07f0d4c518", 2024-09-19T08:11:42.616 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-09-19T08:11:42.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:11:42.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:11:42.616 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:11:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.607+0000 7fcb76834640 1 -- 172.21.15.98:0/1454717553 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcb5c0a33b0 msgr2=0x7fcb5c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.607+0000 7fcb76834640 1 --2- 172.21.15.98:0/1454717553 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcb5c0a33b0 0x7fcb5c0a5870 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009430 tx=0x7fcb680095c0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.607+0000 7fcb76834640 1 -- 172.21.15.98:0/1454717553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb70074e30 msgr2=0x7fcb7019ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:11:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.607+0000 7fcb76834640 1 --2- 172.21.15.98:0/1454717553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb70074e30 0x7fcb7019ab30 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800c920 tx=0x7fcb5800cde0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.607+0000 7fcb76834640 1 -- 172.21.15.98:0/1454717553 shutdown_connections 2024-09-19T08:11:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.607+0000 7fcb76834640 1 --2- 172.21.15.98:0/1454717553 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fcb5c0a33b0 0x7fcb5c0a5870 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.607+0000 7fcb76834640 1 --2- 172.21.15.98:0/1454717553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb7010e440 0x7fcb7019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.607+0000 7fcb76834640 1 --2- 172.21.15.98:0/1454717553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb70074e30 0x7fcb7019ab30 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.607+0000 7fcb76834640 1 --2- 172.21.15.98:0/1454717553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb70074510 0x7fcb7019a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:11:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.607+0000 7fcb76834640 1 -- 172.21.15.98:0/1454717553 >> 172.21.15.98:0/1454717553 conn(0x7fcb700fd8c0 msgr2=0x7fcb700fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:11:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.607+0000 7fcb76834640 1 -- 172.21.15.98:0/1454717553 shutdown_connections 2024-09-19T08:11:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:11:42.607+0000 7fcb76834640 1 -- 172.21.15.98:0/1454717553 wait complete. 2024-09-19T08:11:43.319 INFO:tasks.cephfs.filesystem:scrub status for tag:56c41dbf-aa2d-457f-9739-1f07f0d4c518 - {'path': '/', 'tag': '56c41dbf-aa2d-457f-9739-1f07f0d4c518', 'options': 'recursive,force'} 2024-09-19T08:11:44.229 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:43 smithi139 ceph-mon[34874]: pgmap v268: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:11:44.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:43 smithi162 ceph-mon[35260]: pgmap v268: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:11:44.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:43 smithi098 ceph-mon[31333]: pgmap v268: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:11:46.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:45 smithi162 ceph-mon[35260]: pgmap v269: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:11:46.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:45 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:11:46.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:45 smithi139 ceph-mon[34874]: pgmap v269: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:11:46.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:45 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:11:46.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:45 smithi098 ceph-mon[31333]: pgmap v269: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:11:46.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:45 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:11:48.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:47 smithi162 ceph-mon[35260]: pgmap v270: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-09-19T08:11:48.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:47 smithi139 ceph-mon[34874]: pgmap v270: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-09-19T08:11:48.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:47 smithi098 ceph-mon[31333]: pgmap v270: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-09-19T08:11:50.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:49 smithi162 ceph-mon[35260]: pgmap v271: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-09-19T08:11:50.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:49 smithi139 ceph-mon[34874]: pgmap v271: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-09-19T08:11:50.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:49 smithi098 ceph-mon[31333]: pgmap v271: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-09-19T08:11:52.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:51 smithi162 ceph-mon[35260]: pgmap v272: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:11:52.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:51 smithi139 ceph-mon[34874]: pgmap v272: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:11:52.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:51 smithi098 ceph-mon[31333]: pgmap v272: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:11:54.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:53 smithi162 ceph-mon[35260]: pgmap v273: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:11:54.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:53 smithi139 ceph-mon[34874]: pgmap v273: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:11:54.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:53 smithi098 ceph-mon[31333]: pgmap v273: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:11:56.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:55 smithi162 ceph-mon[35260]: pgmap v274: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:11:56.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:55 smithi139 ceph-mon[34874]: pgmap v274: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:11:56.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:55 smithi098 ceph-mon[31333]: pgmap v274: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:11:58.234 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:57 smithi098 ceph-mon[31333]: pgmap v275: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:11:58.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:57 smithi162 ceph-mon[35260]: pgmap v275: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:11:58.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:57 smithi139 ceph-mon[34874]: pgmap v275: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:12:00.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:11:59 smithi162 ceph-mon[35260]: pgmap v276: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:12:00.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:11:59 smithi139 ceph-mon[34874]: pgmap v276: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:12:00.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:11:59 smithi098 ceph-mon[31333]: pgmap v276: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:12:02.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:01 smithi162 ceph-mon[35260]: pgmap v277: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:02.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:01 smithi139 ceph-mon[34874]: pgmap v277: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:02.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:01 smithi098 ceph-mon[31333]: pgmap v277: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:04.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:03 smithi162 ceph-mon[35260]: pgmap v278: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:12:04.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:03 smithi139 ceph-mon[34874]: pgmap v278: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:12:04.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:03 smithi098 ceph-mon[31333]: pgmap v278: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:12:06.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:05 smithi162 ceph-mon[35260]: pgmap v279: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:12:06.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:05 smithi139 ceph-mon[34874]: pgmap v279: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:12:06.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:05 smithi098 ceph-mon[31333]: pgmap v279: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:12:08.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:07 smithi162 ceph-mon[35260]: pgmap v280: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-09-19T08:12:08.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:07 smithi139 ceph-mon[34874]: pgmap v280: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-09-19T08:12:08.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:07 smithi098 ceph-mon[31333]: pgmap v280: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-09-19T08:12:10.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:09 smithi162 ceph-mon[35260]: pgmap v281: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:12:10.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:09 smithi139 ceph-mon[34874]: pgmap v281: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:12:10.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:09 smithi098 ceph-mon[31333]: pgmap v281: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:12:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:11 smithi162 ceph-mon[35260]: pgmap v282: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:12:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:11 smithi162 ceph-mon[35260]: osdmap e86: 12 total, 12 up, 12 in 2024-09-19T08:12:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:11 smithi139 ceph-mon[34874]: pgmap v282: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:12:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:11 smithi139 ceph-mon[34874]: osdmap e86: 12 total, 12 up, 12 in 2024-09-19T08:12:12.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:11 smithi098 ceph-mon[31333]: pgmap v282: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:12:12.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:11 smithi098 ceph-mon[31333]: osdmap e86: 12 total, 12 up, 12 in 2024-09-19T08:12:13.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:12:13.582 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:12:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.133+0000 7f0f3404b640 1 -- 172.21.15.98:0/2128762681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f2c102d30 msgr2=0x7f0f2c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.133+0000 7f0f3404b640 1 --2- 172.21.15.98:0/2128762681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f2c102d30 0x7f0f2c103110 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f0f20009930 tx=0x7f0f2002a420 comp rx=0 tx=0).stop 2024-09-19T08:12:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.133+0000 7f0f3404b640 1 -- 172.21.15.98:0/2128762681 shutdown_connections 2024-09-19T08:12:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.133+0000 7f0f3404b640 1 --2- 172.21.15.98:0/2128762681 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0f2c1082a0 0x7f0f2c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.133+0000 7f0f3404b640 1 --2- 172.21.15.98:0/2128762681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f2c1036e0 0x7f0f2c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.133+0000 7f0f3404b640 1 --2- 172.21.15.98:0/2128762681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f2c102d30 0x7f0f2c103110 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.133+0000 7f0f3404b640 1 -- 172.21.15.98:0/2128762681 >> 172.21.15.98:0/2128762681 conn(0x7f0f2c0fc7d0 msgr2=0x7f0f2c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.133+0000 7f0f3404b640 1 -- 172.21.15.98:0/2128762681 shutdown_connections 2024-09-19T08:12:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.133+0000 7f0f3404b640 1 -- 172.21.15.98:0/2128762681 wait complete. 2024-09-19T08:12:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.133+0000 7f0f3404b640 1 Processor -- start 2024-09-19T08:12:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.133+0000 7f0f3404b640 1 -- start start 2024-09-19T08:12:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f3404b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f2c102d30 0x7f0f2c100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f3404b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f2c1036e0 0x7f0f2c100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f3404b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0f2c1082a0 0x7f0f2c101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f3404b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1120a0 con 0x7f0f2c1036e0 2024-09-19T08:12:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f3404b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c111f20 con 0x7f0f2c1082a0 2024-09-19T08:12:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f3404b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c112220 con 0x7f0f2c102d30 2024-09-19T08:12:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f315e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f2c1036e0 0x7f0f2c100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f315e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f2c1036e0 0x7f0f2c100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40164/0 (socket says 172.21.15.98:40164) 2024-09-19T08:12:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f315e6640 1 -- 172.21.15.98:0/1879677897 learned_addr learned my addr 172.21.15.98:0/1879677897 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f325e8640 1 --2- 172.21.15.98:0/1879677897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0f2c1082a0 0x7f0f2c101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f31de7640 1 --2- 172.21.15.98:0/1879677897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f2c102d30 0x7f0f2c100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f315e6640 1 -- 172.21.15.98:0/1879677897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f2c102d30 msgr2=0x7f0f2c100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f315e6640 1 --2- 172.21.15.98:0/1879677897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f2c102d30 0x7f0f2c100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f315e6640 1 -- 172.21.15.98:0/1879677897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0f2c1082a0 msgr2=0x7f0f2c101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f315e6640 1 --2- 172.21.15.98:0/1879677897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0f2c1082a0 0x7f0f2c101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f315e6640 1 -- 172.21.15.98:0/1879677897 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2c1a6a60 con 0x7f0f2c1036e0 2024-09-19T08:12:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f315e6640 1 --2- 172.21.15.98:0/1879677897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f2c1036e0 0x7f0f2c100ed0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800d740 tx=0x7f0f2800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.134+0000 7f0f1effd640 1 -- 172.21.15.98:0/1879677897 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28017070 con 0x7f0f2c1036e0 2024-09-19T08:12:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.135+0000 7f0f3404b640 1 -- 172.21.15.98:0/1879677897 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f2c1a6d50 con 0x7f0f2c1036e0 2024-09-19T08:12:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.135+0000 7f0f1effd640 1 -- 172.21.15.98:0/1879677897 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f2800de50 con 0x7f0f2c1036e0 2024-09-19T08:12:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.135+0000 7f0f1effd640 1 -- 172.21.15.98:0/1879677897 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2800f6b0 con 0x7f0f2c1036e0 2024-09-19T08:12:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.135+0000 7f0f3404b640 1 -- 172.21.15.98:0/1879677897 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f2c1a7250 con 0x7f0f2c1036e0 2024-09-19T08:12:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.137+0000 7f0f36424740 1 Processor -- start 2024-09-19T08:12:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f36424740 1 -- start start 2024-09-19T08:12:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f36424740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdc94a0 0x55556cdc9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f36424740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc9dc0 0x55556cdd4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f36424740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdd4680 0x55556cdd6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f36424740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55556cd2f2e0 con 0x55556cdd4680 2024-09-19T08:12:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f36424740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55556ccd6ce0 con 0x55556cdc9dc0 2024-09-19T08:12:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f36424740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55556cd2b2a0 con 0x55556cdc94a0 2024-09-19T08:12:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f1effd640 1 -- 172.21.15.98:0/1879677897 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f0f2801a070 con 0x7f0f2c1036e0 2024-09-19T08:12:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f325e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdc94a0 0x55556cdc9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f325e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdc94a0 0x55556cdc9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.98:39544/0 (socket says 172.21.15.98:39544) 2024-09-19T08:12:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f325e8640 1 -- 172.21.15.98:0/2110431091 learned_addr learned my addr 172.21.15.98:0/2110431091 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f31de7640 1 --2- 172.21.15.98:0/2110431091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc9dc0 0x55556cdd4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f315e6640 1 --2- 172.21.15.98:0/2110431091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdd4680 0x55556cdd6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f325e8640 1 -- 172.21.15.98:0/2110431091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc9dc0 msgr2=0x55556cdd4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:14.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f325e8640 1 --2- 172.21.15.98:0/2110431091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc9dc0 0x55556cdd4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f325e8640 1 -- 172.21.15.98:0/2110431091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdd4680 msgr2=0x55556cdd6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:14.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f325e8640 1 --2- 172.21.15.98:0/2110431091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdd4680 0x55556cdd6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f325e8640 1 -- 172.21.15.98:0/2110431091 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55556ccbb440 con 0x55556cdc94a0 2024-09-19T08:12:14.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.138+0000 7f0f315e6640 1 --2- 172.21.15.98:0/2110431091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdd4680 0x55556cdd6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:12:14.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.139+0000 7f0f325e8640 1 --2- 172.21.15.98:0/2110431091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdc94a0 0x55556cdc9880 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009f70 tx=0x7f0f24010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=314e1e0e39f68feb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:14.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.139+0000 7f0f17fff640 1 -- 172.21.15.98:0/2110431091 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24016070 con 0x55556cdc94a0 2024-09-19T08:12:14.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.139+0000 7f0f1effd640 1 --2- 172.21.15.98:0/1879677897 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f0f180a3430 0x7f0f180a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.139+0000 7f0f17fff640 1 -- 172.21.15.98:0/2110431091 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f24004d20 con 0x55556cdc94a0 2024-09-19T08:12:14.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.139+0000 7f0f17fff640 1 -- 172.21.15.98:0/2110431091 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24012870 con 0x55556cdc94a0 2024-09-19T08:12:14.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.139+0000 7f0f31de7640 1 --2- 172.21.15.98:0/1879677897 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f0f180a3430 0x7f0f180a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.139+0000 7f0f1effd640 1 -- 172.21.15.98:0/1879677897 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7f0f280cf630 con 0x7f0f2c1036e0 2024-09-19T08:12:14.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.140+0000 7f0f36424740 1 -- 172.21.15.98:0/2110431091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdc94a0 msgr2=0x55556cdc9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:14.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.140+0000 7f0f31de7640 1 --2- 172.21.15.98:0/1879677897 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f0f180a3430 0x7f0f180a58f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0f20009900 tx=0x7f0f200058d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:14.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.140+0000 7f0f36424740 1 --2- 172.21.15.98:0/2110431091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdc94a0 0x55556cdc9880 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009f70 tx=0x7f0f24010740 comp rx=0 tx=0).stop 2024-09-19T08:12:14.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.140+0000 7f0f36424740 1 -- 172.21.15.98:0/2110431091 shutdown_connections 2024-09-19T08:12:14.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.140+0000 7f0f36424740 1 --2- 172.21.15.98:0/2110431091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdd4680 0x55556cdd6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.140+0000 7f0f36424740 1 --2- 172.21.15.98:0/2110431091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc9dc0 0x55556cdd4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.140+0000 7f0f36424740 1 --2- 172.21.15.98:0/2110431091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdc94a0 0x55556cdc9880 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.140+0000 7f0f36424740 1 -- 172.21.15.98:0/2110431091 >> 172.21.15.98:0/2110431091 conn(0x55556cdc5a30 msgr2=0x55556cdc5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:14.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.140+0000 7f0f36424740 1 -- 172.21.15.98:0/2110431091 shutdown_connections 2024-09-19T08:12:14.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.140+0000 7f0f36424740 1 -- 172.21.15.98:0/2110431091 wait complete. 2024-09-19T08:12:14.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.140+0000 7f0f36424740 1 -- start start 2024-09-19T08:12:14.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.140+0000 7f0f36424740 1 Processor -- start 2024-09-19T08:12:14.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f36424740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc94a0 0x55556cdd3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f36424740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdd1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f36424740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdd4680 0x55556cdd1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f36424740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55556cdd7250 con 0x55556cdc9dc0 2024-09-19T08:12:14.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f36424740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55556ccea050 con 0x55556cdc94a0 2024-09-19T08:12:14.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f36424740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55556cdd73d0 con 0x55556cdd4680 2024-09-19T08:12:14.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f36424740 10 client.0 _ll_register_callbacks cb 0x55556cd2a160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:14.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f315e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdd1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f315e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdd1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40180/0 (socket says 172.21.15.98:40180) 2024-09-19T08:12:14.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f315e6640 1 -- 172.21.15.98:0/325108967 learned_addr learned my addr 172.21.15.98:0/325108967 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:14.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f31de7640 1 --2- 172.21.15.98:0/325108967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc94a0 0x55556cdd3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f325e8640 1 --2- 172.21.15.98:0/325108967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdd4680 0x55556cdd1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f315e6640 1 -- 172.21.15.98:0/325108967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdd4680 msgr2=0x55556cdd1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:14.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f315e6640 1 --2- 172.21.15.98:0/325108967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdd4680 0x55556cdd1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f315e6640 1 -- 172.21.15.98:0/325108967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc94a0 msgr2=0x55556cdd3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:14.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f315e6640 1 --2- 172.21.15.98:0/325108967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc94a0 0x55556cdd3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.141+0000 7f0f315e6640 1 -- 172.21.15.98:0/325108967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55556caa03c0 con 0x55556cdc9dc0 2024-09-19T08:12:14.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.142+0000 7f0f315e6640 1 --2- 172.21.15.98:0/325108967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdd1220 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009610 tx=0x7f0f2809b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad335a91d833a30e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:14.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.142+0000 7f0f157fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:14.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.142+0000 7f0f157fa640 1 -- 172.21.15.98:0/325108967 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28021070 con 0x55556cdc9dc0 2024-09-19T08:12:14.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.142+0000 7f0f36424740 1 -- 172.21.15.98:0/325108967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f24004d20 con 0x55556cdc9dc0 2024-09-19T08:12:14.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.142+0000 7f0f157fa640 1 -- 172.21.15.98:0/325108967 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7f0f28098000 con 0x55556cdc9dc0 2024-09-19T08:12:14.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.142+0000 7f0f157fa640 1 -- 172.21.15.98:0/325108967 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0f2809fc90 con 0x55556cdc9dc0 2024-09-19T08:12:14.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.142+0000 7f0f36424740 10 client.17556 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:14.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.142+0000 7f0f36424740 1 -- 172.21.15.98:0/325108967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55556cde2bc0 con 0x55556cdc9dc0 2024-09-19T08:12:14.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.142+0000 7f0f157fa640 1 -- 172.21.15.98:0/325108967 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f24004d20 con 0x55556cdc9dc0 2024-09-19T08:12:14.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.142+0000 7f0f157fa640 1 -- 172.21.15.98:0/325108967 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0f2809ae70 con 0x55556cdc9dc0 2024-09-19T08:12:14.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.143+0000 7f0f36424740 10 client.17556 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:14.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.143+0000 7f0f36424740 10 client.17556 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:12:14.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.143+0000 7f0f36424740 20 client.17556 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:14.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.143+0000 7f0f36424740 1 --2- 172.21.15.98:0/325108967 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55556cde3fd0 0x55556cde63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.143+0000 7f0f36424740 4 client.17556 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:12:14.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.143+0000 7f0f36424740 1 -- 172.21.15.98:0/325108967 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55556cd3c850 con 0x55556cde3fd0 2024-09-19T08:12:14.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.143+0000 7f0f31de7640 1 --2- 172.21.15.98:0/325108967 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55556cde3fd0 0x55556cde63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.143+0000 7f0f31de7640 1 --2- 172.21.15.98:0/325108967 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55556cde3fd0 0x55556cde63e0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cb7d83460c7bbc9 server_cookie=add0e76a24d0e21f in_seq=0 out_seq=0 2024-09-19T08:12:14.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.143+0000 7f0f157fa640 10 client.17556 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:14.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f157fa640 1 -- 172.21.15.98:0/325108967 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55556cd3c850 con 0x55556cde3fd0 2024-09-19T08:12:14.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f157fa640 10 client.17556 handle_command_reply: tid=0 2024-09-19T08:12:14.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f36424740 1 client.17556 shutdown 2024-09-19T08:12:14.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f36424740 1 -- 172.21.15.98:0/325108967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 msgr2=0x55556cdd1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:14.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f36424740 1 --2- 172.21.15.98:0/325108967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdd1220 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009610 tx=0x7f0f2809b4d0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f36424740 1 -- 172.21.15.98:0/325108967 shutdown_connections 2024-09-19T08:12:14.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f36424740 1 --2- 172.21.15.98:0/325108967 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55556cde3fd0 0x55556cde63e0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f36424740 1 --2- 172.21.15.98:0/325108967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdd4680 0x55556cdd1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f36424740 1 --2- 172.21.15.98:0/325108967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdd1220 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f157fa640 0 client.17556 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f36424740 1 --2- 172.21.15.98:0/325108967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc94a0 0x55556cdd3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f36424740 1 -- 172.21.15.98:0/325108967 >> 172.21.15.98:0/325108967 conn(0x55556cdc5a30 msgr2=0x55556cdcb710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f36424740 1 -- 172.21.15.98:0/325108967 shutdown_connections 2024-09-19T08:12:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f36424740 1 -- 172.21.15.98:0/325108967 wait complete. 2024-09-19T08:12:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.144+0000 7f0f36424740 20 client.17556 trim_cache size 0 max 16384 2024-09-19T08:12:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.161+0000 7f0f36424740 1 Processor -- start 2024-09-19T08:12:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.161+0000 7f0f36424740 1 -- start start 2024-09-19T08:12:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.161+0000 7f0f36424740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc94a0 0x55556cdcfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.161+0000 7f0f36424740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdebcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.161+0000 7f0f36424740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdd4680 0x55556cdec1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.161+0000 7f0f36424740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1afb80 con 0x55556cdc9dc0 2024-09-19T08:12:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.161+0000 7f0f36424740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1b16c0 con 0x55556cdc94a0 2024-09-19T08:12:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.161+0000 7f0f36424740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1b0250 con 0x55556cdd4680 2024-09-19T08:12:14.184 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:12:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:12:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:12:14.184 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:12:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.161+0000 7f0f315e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdebcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.161+0000 7f0f315e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdebcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40190/0 (socket says 172.21.15.98:40190) 2024-09-19T08:12:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.161+0000 7f0f315e6640 1 -- 172.21.15.98:0/4034781869 learned_addr learned my addr 172.21.15.98:0/4034781869 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.161+0000 7f0f31de7640 1 --2- 172.21.15.98:0/4034781869 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc94a0 0x55556cdcfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.162+0000 7f0f325e8640 1 --2- 172.21.15.98:0/4034781869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdd4680 0x55556cdec1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.162+0000 7f0f315e6640 1 -- 172.21.15.98:0/4034781869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdd4680 msgr2=0x55556cdec1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.162+0000 7f0f315e6640 1 --2- 172.21.15.98:0/4034781869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdd4680 0x55556cdec1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.162+0000 7f0f315e6640 1 -- 172.21.15.98:0/4034781869 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc94a0 msgr2=0x55556cdcfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.162+0000 7f0f315e6640 1 --2- 172.21.15.98:0/4034781869 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc94a0 0x55556cdcfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.162+0000 7f0f315e6640 1 -- 172.21.15.98:0/4034781869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55556cdc54d0 con 0x55556cdc9dc0 2024-09-19T08:12:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.162+0000 7f0f31de7640 1 --2- 172.21.15.98:0/4034781869 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc94a0 0x55556cdcfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:12:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.162+0000 7f0f325e8640 1 --2- 172.21.15.98:0/4034781869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdd4680 0x55556cdec1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.162+0000 7f0f315e6640 1 --2- 172.21.15.98:0/4034781869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdebcb0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f0f2809b690 tx=0x7f0f2809b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e593108ca152d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.162+0000 7f0f177fe640 1 -- 172.21.15.98:0/4034781869 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28021070 con 0x55556cdc9dc0 2024-09-19T08:12:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.162+0000 7f0f177fe640 1 -- 172.21.15.98:0/4034781869 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f280a4550 con 0x55556cdc9dc0 2024-09-19T08:12:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.162+0000 7f0f177fe640 1 -- 172.21.15.98:0/4034781869 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2809ade0 con 0x55556cdc9dc0 2024-09-19T08:12:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.163+0000 7f0f36424740 1 -- 172.21.15.98:0/4034781869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 msgr2=0x55556cdebcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.163+0000 7f0f36424740 1 --2- 172.21.15.98:0/4034781869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdebcb0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f0f2809b690 tx=0x7f0f2809b600 comp rx=0 tx=0).stop 2024-09-19T08:12:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.163+0000 7f0f36424740 1 -- 172.21.15.98:0/4034781869 shutdown_connections 2024-09-19T08:12:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.163+0000 7f0f36424740 1 --2- 172.21.15.98:0/4034781869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdd4680 0x55556cdec1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.163+0000 7f0f36424740 1 --2- 172.21.15.98:0/4034781869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdebcb0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.163+0000 7f0f36424740 1 --2- 172.21.15.98:0/4034781869 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdc94a0 0x55556cdcfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.163+0000 7f0f36424740 1 -- 172.21.15.98:0/4034781869 >> 172.21.15.98:0/4034781869 conn(0x55556cdc5a30 msgr2=0x55556cdd32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.163+0000 7f0f36424740 1 -- 172.21.15.98:0/4034781869 shutdown_connections 2024-09-19T08:12:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.163+0000 7f0f36424740 1 -- 172.21.15.98:0/4034781869 wait complete. 2024-09-19T08:12:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.163+0000 7f0f36424740 1 -- start start 2024-09-19T08:12:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.163+0000 7f0f36424740 1 Processor -- start 2024-09-19T08:12:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f36424740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdc94a0 0x55556cdda300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f36424740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdda840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f36424740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdd4680 0x55556cddad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f36424740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1b0ff0 con 0x55556cdc9dc0 2024-09-19T08:12:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f36424740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1b0920 con 0x55556cdd4680 2024-09-19T08:12:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f36424740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1adb80 con 0x55556cdc94a0 2024-09-19T08:12:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f36424740 10 client.0 _ll_register_callbacks cb 0x7f0f28010c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f315e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdda840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f315e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdda840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40202/0 (socket says 172.21.15.98:40202) 2024-09-19T08:12:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f315e6640 1 -- 172.21.15.98:0/2244462908 learned_addr learned my addr 172.21.15.98:0/2244462908 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f31de7640 1 --2- 172.21.15.98:0/2244462908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdc94a0 0x55556cdda300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f325e8640 1 --2- 172.21.15.98:0/2244462908 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdd4680 0x55556cddad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f315e6640 1 -- 172.21.15.98:0/2244462908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdc94a0 msgr2=0x55556cdda300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f315e6640 1 --2- 172.21.15.98:0/2244462908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdc94a0 0x55556cdda300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f315e6640 1 -- 172.21.15.98:0/2244462908 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdd4680 msgr2=0x55556cddad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f315e6640 1 --2- 172.21.15.98:0/2244462908 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdd4680 0x55556cddad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f315e6640 1 -- 172.21.15.98:0/2244462908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55556ce4dc90 con 0x55556cdc9dc0 2024-09-19T08:12:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f315e6640 1 --2- 172.21.15.98:0/2244462908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdda840 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f0f28098360 tx=0x7f0f2809b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb19321559ed3e01 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f157fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f157fa640 1 -- 172.21.15.98:0/2244462908 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28021030 con 0x55556cdc9dc0 2024-09-19T08:12:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.164+0000 7f0f157fa640 1 -- 172.21.15.98:0/2244462908 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7f0f2809ade0 con 0x55556cdc9dc0 2024-09-19T08:12:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.165+0000 7f0f36424740 1 -- 172.21.15.98:0/2244462908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f280a4550 con 0x55556cdc9dc0 2024-09-19T08:12:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.165+0000 7f0f157fa640 1 -- 172.21.15.98:0/2244462908 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0f280ba300 con 0x55556cdc9dc0 2024-09-19T08:12:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.165+0000 7f0f36424740 10 client.17562 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.165+0000 7f0f36424740 1 -- 172.21.15.98:0/2244462908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55556ce4ee10 con 0x55556cdc9dc0 2024-09-19T08:12:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.165+0000 7f0f157fa640 1 -- 172.21.15.98:0/2244462908 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f280a4550 con 0x55556cdc9dc0 2024-09-19T08:12:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.165+0000 7f0f157fa640 1 -- 172.21.15.98:0/2244462908 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0f280cfa90 con 0x55556cdc9dc0 2024-09-19T08:12:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.165+0000 7f0f36424740 10 client.17562 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.165+0000 7f0f36424740 10 client.17562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:12:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.165+0000 7f0f36424740 20 client.17562 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.165+0000 7f0f36424740 1 --2- 172.21.15.98:0/2244462908 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55556ce4f220 0x55556ce51640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.165+0000 7f0f36424740 4 client.17562 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:12:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.165+0000 7f0f36424740 1 -- 172.21.15.98:0/2244462908 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55556cdd1af0 con 0x55556ce4f220 2024-09-19T08:12:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.165+0000 7f0f31de7640 1 --2- 172.21.15.98:0/2244462908 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55556ce4f220 0x55556ce51640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f157fa640 10 client.17562 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f31de7640 1 --2- 172.21.15.98:0/2244462908 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55556ce4f220 0x55556ce51640 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2145cc66d5c62e8e server_cookie=a48a2d2ec51290e8 in_seq=0 out_seq=0 2024-09-19T08:12:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f157fa640 1 -- 172.21.15.98:0/2244462908 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55556cdd1af0 con 0x55556ce4f220 2024-09-19T08:12:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f157fa640 10 client.17562 handle_command_reply: tid=0 2024-09-19T08:12:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f36424740 1 client.17562 shutdown 2024-09-19T08:12:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f36424740 1 -- 172.21.15.98:0/2244462908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 msgr2=0x55556cdda840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f36424740 1 --2- 172.21.15.98:0/2244462908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdda840 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f0f28098360 tx=0x7f0f2809b4d0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f36424740 1 -- 172.21.15.98:0/2244462908 shutdown_connections 2024-09-19T08:12:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f36424740 1 --2- 172.21.15.98:0/2244462908 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55556ce4f220 0x55556ce51640 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f36424740 1 --2- 172.21.15.98:0/2244462908 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55556cdd4680 0x55556cddad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f36424740 1 --2- 172.21.15.98:0/2244462908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55556cdc9dc0 0x55556cdda840 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f157fa640 0 client.17562 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f36424740 1 --2- 172.21.15.98:0/2244462908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55556cdc94a0 0x55556cdda300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f36424740 1 -- 172.21.15.98:0/2244462908 >> 172.21.15.98:0/2244462908 conn(0x55556cdc5a30 msgr2=0x55556cdd2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f36424740 1 -- 172.21.15.98:0/2244462908 shutdown_connections 2024-09-19T08:12:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f36424740 1 -- 172.21.15.98:0/2244462908 wait complete. 2024-09-19T08:12:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.166+0000 7f0f36424740 20 client.17562 trim_cache size 0 max 16384 2024-09-19T08:12:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.167+0000 7f0f3404b640 1 -- 172.21.15.98:0/1879677897 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f0f180a3430 msgr2=0x7f0f180a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.167+0000 7f0f3404b640 1 --2- 172.21.15.98:0/1879677897 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f0f180a3430 0x7f0f180a58f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0f20009900 tx=0x7f0f200058d0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.167+0000 7f0f3404b640 1 -- 172.21.15.98:0/1879677897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f2c1036e0 msgr2=0x7f0f2c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.167+0000 7f0f3404b640 1 --2- 172.21.15.98:0/1879677897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f2c1036e0 0x7f0f2c100ed0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800d740 tx=0x7f0f2800dc00 comp rx=0 tx=0).stop 2024-09-19T08:12:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.167+0000 7f0f3404b640 1 -- 172.21.15.98:0/1879677897 shutdown_connections 2024-09-19T08:12:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.167+0000 7f0f3404b640 1 --2- 172.21.15.98:0/1879677897 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f0f180a3430 0x7f0f180a58f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.167+0000 7f0f3404b640 1 --2- 172.21.15.98:0/1879677897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0f2c1082a0 0x7f0f2c101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.167+0000 7f0f3404b640 1 --2- 172.21.15.98:0/1879677897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f2c1036e0 0x7f0f2c100ed0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.167+0000 7f0f3404b640 1 --2- 172.21.15.98:0/1879677897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f2c102d30 0x7f0f2c100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.167+0000 7f0f3404b640 1 -- 172.21.15.98:0/1879677897 >> 172.21.15.98:0/1879677897 conn(0x7f0f2c0fc7d0 msgr2=0x7f0f2c105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.167+0000 7f0f3404b640 1 -- 172.21.15.98:0/1879677897 shutdown_connections 2024-09-19T08:12:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:14.167+0000 7f0f3404b640 1 -- 172.21.15.98:0/1879677897 wait complete. 2024-09-19T08:12:14.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:13 smithi098 ceph-mon[31333]: pgmap v284: 97 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 66 active+clean; 735 MiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:12:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:13 smithi162 ceph-mon[35260]: pgmap v284: 97 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 66 active+clean; 735 MiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:12:14.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:13 smithi139 ceph-mon[34874]: pgmap v284: 97 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 66 active+clean; 735 MiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:12:15.105 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:12:15.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:12:15.373 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:12:15.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.886+0000 7f54810b8640 1 -- 172.21.15.98:0/3235248481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c101420 msgr2=0x7f547c10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:15.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.886+0000 7f54810b8640 1 --2- 172.21.15.98:0/3235248481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c101420 0x7f547c10d950 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f5474009cd0 tx=0x7f547402a8e0 comp rx=0 tx=0).stop 2024-09-19T08:12:15.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.886+0000 7f54810b8640 1 -- 172.21.15.98:0/3235248481 shutdown_connections 2024-09-19T08:12:15.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.886+0000 7f54810b8640 1 --2- 172.21.15.98:0/3235248481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c101420 0x7f547c10d950 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:15.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.886+0000 7f54810b8640 1 --2- 172.21.15.98:0/3235248481 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f547c100a80 0x7f547c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:15.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.886+0000 7f54810b8640 1 --2- 172.21.15.98:0/3235248481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f547c106a80 0x7f547c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:15.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.886+0000 7f54810b8640 1 -- 172.21.15.98:0/3235248481 >> 172.21.15.98:0/3235248481 conn(0x7f547c0fc7a0 msgr2=0x7f547c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:15.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.886+0000 7f54810b8640 1 -- 172.21.15.98:0/3235248481 shutdown_connections 2024-09-19T08:12:15.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.886+0000 7f54810b8640 1 -- 172.21.15.98:0/3235248481 wait complete. 2024-09-19T08:12:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.887+0000 7f54810b8640 1 Processor -- start 2024-09-19T08:12:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.887+0000 7f54810b8640 1 -- start start 2024-09-19T08:12:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.887+0000 7f54810b8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c100a80 0x7f547c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.887+0000 7f54810b8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f547c101420 0x7f547c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.887+0000 7f54810b8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f547c106a80 0x7f547c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.887+0000 7f54810b8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f547c110e80 con 0x7f547c100a80 2024-09-19T08:12:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.887+0000 7f54810b8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f547c110d00 con 0x7f547c101420 2024-09-19T08:12:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.887+0000 7f54810b8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f547c111000 con 0x7f547c106a80 2024-09-19T08:12:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.887+0000 7f547ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c100a80 0x7f547c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.887+0000 7f547ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c100a80 0x7f547c198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40218/0 (socket says 172.21.15.98:40218) 2024-09-19T08:12:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.887+0000 7f547ad9d640 1 -- 172.21.15.98:0/1288878612 learned_addr learned my addr 172.21.15.98:0/1288878612 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.887+0000 7f547a59c640 1 --2- 172.21.15.98:0/1288878612 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f547c101420 0x7f547c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.887+0000 7f547b59e640 1 --2- 172.21.15.98:0/1288878612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f547c106a80 0x7f547c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.888+0000 7f547ad9d640 1 -- 172.21.15.98:0/1288878612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f547c106a80 msgr2=0x7f547c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.888+0000 7f547ad9d640 1 --2- 172.21.15.98:0/1288878612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f547c106a80 0x7f547c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.888+0000 7f547ad9d640 1 -- 172.21.15.98:0/1288878612 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f547c101420 msgr2=0x7f547c1989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.888+0000 7f547ad9d640 1 --2- 172.21.15.98:0/1288878612 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f547c101420 0x7f547c1989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.888+0000 7f547ad9d640 1 -- 172.21.15.98:0/1288878612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547c19d440 con 0x7f547c100a80 2024-09-19T08:12:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.888+0000 7f547ad9d640 1 --2- 172.21.15.98:0/1288878612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c100a80 0x7f547c198490 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f546c00d770 tx=0x7f546c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.888+0000 7f5463fff640 1 -- 172.21.15.98:0/1288878612 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c017070 con 0x7f547c100a80 2024-09-19T08:12:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.888+0000 7f54810b8640 1 -- 172.21.15.98:0/1288878612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f547c19d730 con 0x7f547c100a80 2024-09-19T08:12:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.888+0000 7f5463fff640 1 -- 172.21.15.98:0/1288878612 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f546c005500 con 0x7f547c100a80 2024-09-19T08:12:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.888+0000 7f5463fff640 1 -- 172.21.15.98:0/1288878612 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c004e00 con 0x7f547c100a80 2024-09-19T08:12:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.888+0000 7f54810b8640 1 -- 172.21.15.98:0/1288878612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f547c1a5010 con 0x7f547c100a80 2024-09-19T08:12:15.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.889+0000 7f54810b8640 1 -- 172.21.15.98:0/1288878612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f547c100ee0 con 0x7f547c100a80 2024-09-19T08:12:15.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.890+0000 7f5463fff640 1 -- 172.21.15.98:0/1288878612 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f546c004fa0 con 0x7f547c100a80 2024-09-19T08:12:15.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.891+0000 7f5463fff640 1 --2- 172.21.15.98:0/1288878612 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f54640a3430 0x7f54640a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:15.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.891+0000 7f547a59c640 1 --2- 172.21.15.98:0/1288878612 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f54640a3430 0x7f54640a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:15.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.891+0000 7f5463fff640 1 -- 172.21.15.98:0/1288878612 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7f546c0cf640 con 0x7f547c100a80 2024-09-19T08:12:15.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.892+0000 7f547a59c640 1 --2- 172.21.15.98:0/1288878612 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f54640a3430 0x7f54640a58f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f547c105200 tx=0x7f547000a2d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:15.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:15.896+0000 7f5463fff640 1 -- 172.21.15.98:0/1288878612 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f546c09b9a0 con 0x7f547c100a80 2024-09-19T08:12:16.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:15 smithi098 ceph-mon[31333]: pgmap v285: 97 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 66 active+clean; 735 MiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:12:16.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:15 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:12:16.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:15 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:12:16.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:15 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:12:16.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:15 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:12:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:16.096+0000 7f54810b8640 1 -- 172.21.15.98:0/1288878612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f547c10a050 con 0x7f547c100a80 2024-09-19T08:12:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:16.097+0000 7f5463fff640 1 -- 172.21.15.98:0/1288878612 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11983 (secure 0 0 0) 0x7f547c10a050 con 0x7f547c100a80 2024-09-19T08:12:16.098 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:12:16.098 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"btime":"2024-09-19T08:08:26:985720+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:08:26.555658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:12:16.099 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2024-09-19T08:12:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:16.099+0000 7f54810b8640 1 -- 172.21.15.98:0/1288878612 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f54640a3430 msgr2=0x7f54640a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:16.099+0000 7f54810b8640 1 --2- 172.21.15.98:0/1288878612 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f54640a3430 0x7f54640a58f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f547c105200 tx=0x7f547000a2d0 comp rx=0 tx=0).stop 2024-09-19T08:12:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:16.099+0000 7f54810b8640 1 -- 172.21.15.98:0/1288878612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c100a80 msgr2=0x7f547c198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:16.099+0000 7f54810b8640 1 --2- 172.21.15.98:0/1288878612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c100a80 0x7f547c198490 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f546c00d770 tx=0x7f546c00dc30 comp rx=0 tx=0).stop 2024-09-19T08:12:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:16.099+0000 7f54810b8640 1 -- 172.21.15.98:0/1288878612 shutdown_connections 2024-09-19T08:12:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:16.099+0000 7f54810b8640 1 --2- 172.21.15.98:0/1288878612 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f54640a3430 0x7f54640a58f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:16.099+0000 7f54810b8640 1 --2- 172.21.15.98:0/1288878612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f547c106a80 0x7f547c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:16.099+0000 7f54810b8640 1 --2- 172.21.15.98:0/1288878612 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f547c101420 0x7f547c1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:16.099+0000 7f54810b8640 1 --2- 172.21.15.98:0/1288878612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c100a80 0x7f547c198490 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:16.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:16.099+0000 7f54810b8640 1 -- 172.21.15.98:0/1288878612 >> 172.21.15.98:0/1288878612 conn(0x7f547c0fc7a0 msgr2=0x7f547c06aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:16.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:16.099+0000 7f54810b8640 1 -- 172.21.15.98:0/1288878612 shutdown_connections 2024-09-19T08:12:16.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:16.099+0000 7f54810b8640 1 -- 172.21.15.98:0/1288878612 wait complete. 2024-09-19T08:12:16.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:15 smithi162 ceph-mon[35260]: pgmap v285: 97 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 66 active+clean; 735 MiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:12:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:15 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:12:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:15 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:12:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:15 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:12:16.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:15 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:12:16.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:15 smithi139 ceph-mon[34874]: pgmap v285: 97 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 66 active+clean; 735 MiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:12:16.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:15 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:12:16.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:15 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:12:16.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:15 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:12:16.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:15 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:12:16.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:12:17.181 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:12:17.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:17 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1288878612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:12:17.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:17 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1288878612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:12:17.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:17 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1288878612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:12:17.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.639+0000 7feb7ad26640 1 -- 172.21.15.98:0/4053982709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb74106f20 msgr2=0x7feb7410eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:17.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.639+0000 7feb7ad26640 1 --2- 172.21.15.98:0/4053982709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb74106f20 0x7feb7410eaa0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7feb6c009ca0 tx=0x7feb6c02a8c0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.639+0000 7feb7ad26640 1 -- 172.21.15.98:0/4053982709 shutdown_connections 2024-09-19T08:12:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.639+0000 7feb7ad26640 1 --2- 172.21.15.98:0/4053982709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb74106f20 0x7feb7410eaa0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.639+0000 7feb7ad26640 1 --2- 172.21.15.98:0/4053982709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb741027a0 0x7feb741067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.639+0000 7feb7ad26640 1 --2- 172.21.15.98:0/4053982709 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb74101df0 0x7feb741021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.639+0000 7feb7ad26640 1 -- 172.21.15.98:0/4053982709 >> 172.21.15.98:0/4053982709 conn(0x7feb740fd8b0 msgr2=0x7feb740ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.639+0000 7feb7ad26640 1 -- 172.21.15.98:0/4053982709 shutdown_connections 2024-09-19T08:12:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.639+0000 7feb7ad26640 1 -- 172.21.15.98:0/4053982709 wait complete. 2024-09-19T08:12:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.639+0000 7feb7ad26640 1 Processor -- start 2024-09-19T08:12:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb7ad26640 1 -- start start 2024-09-19T08:12:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb7ad26640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb74101df0 0x7feb74198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb7ad26640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb741027a0 0x7feb74198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb7ad26640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb74106f20 0x7feb7419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb7ad26640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb74111f80 con 0x7feb74106f20 2024-09-19T08:12:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb7ad26640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feb74111e00 con 0x7feb74101df0 2024-09-19T08:12:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb7ad26640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb74112100 con 0x7feb741027a0 2024-09-19T08:12:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb792c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb74106f20 0x7feb7419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb792c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb74106f20 0x7feb7419ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54238/0 (socket says 172.21.15.98:54238) 2024-09-19T08:12:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb792c3640 1 -- 172.21.15.98:0/3592651663 learned_addr learned my addr 172.21.15.98:0/3592651663 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb78ac2640 1 --2- 172.21.15.98:0/3592651663 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb74101df0 0x7feb74198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb73fff640 1 --2- 172.21.15.98:0/3592651663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb741027a0 0x7feb74198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb792c3640 1 -- 172.21.15.98:0/3592651663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb741027a0 msgr2=0x7feb74198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb792c3640 1 --2- 172.21.15.98:0/3592651663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb741027a0 0x7feb74198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb792c3640 1 -- 172.21.15.98:0/3592651663 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb74101df0 msgr2=0x7feb74198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb792c3640 1 --2- 172.21.15.98:0/3592651663 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb74101df0 0x7feb74198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb792c3640 1 -- 172.21.15.98:0/3592651663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7419d3d0 con 0x7feb74106f20 2024-09-19T08:12:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.640+0000 7feb792c3640 1 --2- 172.21.15.98:0/3592651663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb74106f20 0x7feb7419ccf0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7feb6c031a20 tx=0x7feb6c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.641+0000 7feb71ffb640 1 -- 172.21.15.98:0/3592651663 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c003c30 con 0x7feb74106f20 2024-09-19T08:12:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.641+0000 7feb71ffb640 1 -- 172.21.15.98:0/3592651663 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6c003dd0 con 0x7feb74106f20 2024-09-19T08:12:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.641+0000 7feb7ad26640 1 -- 172.21.15.98:0/3592651663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7419d660 con 0x7feb74106f20 2024-09-19T08:12:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.641+0000 7feb71ffb640 1 -- 172.21.15.98:0/3592651663 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c02fcf0 con 0x7feb74106f20 2024-09-19T08:12:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.641+0000 7feb7ad26640 1 -- 172.21.15.98:0/3592651663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb741a4fa0 con 0x7feb74106f20 2024-09-19T08:12:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.643+0000 7feb71ffb640 1 -- 172.21.15.98:0/3592651663 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7feb6c004050 con 0x7feb74106f20 2024-09-19T08:12:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.643+0000 7feb7d0ff740 1 Processor -- start 2024-09-19T08:12:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.643+0000 7feb7d0ff740 1 -- start start 2024-09-19T08:12:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.643+0000 7feb7d0ff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 0x5644ac3ef090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.643+0000 7feb7d0ff740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3ef5d0 0x5644ac3efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.643+0000 7feb7d0ff740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3fe780 0x5644ac400b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.643+0000 7feb7d0ff740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5644ac3552f0 con 0x5644ac3ef5d0 2024-09-19T08:12:17.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.643+0000 7feb7d0ff740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5644ac2fccf0 con 0x5644ac3fe780 2024-09-19T08:12:17.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.643+0000 7feb7d0ff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5644ac3512b0 con 0x5644ac3f0b40 2024-09-19T08:12:17.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb78ac2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3ef5d0 0x5644ac3efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb78ac2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3ef5d0 0x5644ac3efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54246/0 (socket says 172.21.15.98:54246) 2024-09-19T08:12:17.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb78ac2640 1 -- 172.21.15.98:0/1816476042 learned_addr learned my addr 172.21.15.98:0/1816476042 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:17.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb73fff640 1 --2- 172.21.15.98:0/1816476042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 0x5644ac3ef090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb73fff640 1 --2- 172.21.15.98:0/1816476042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3fe780 0x5644ac400b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb71ffb640 1 --2- 172.21.15.98:0/3592651663 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feb600a3430 0x7feb600a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb78ac2640 1 -- 172.21.15.98:0/1816476042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 msgr2=0x5644ac3ef090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:17.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb78ac2640 1 --2- 172.21.15.98:0/1816476042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 0x5644ac3ef090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb78ac2640 1 -- 172.21.15.98:0/1816476042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3fe780 msgr2=0x5644ac400b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:17.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb78ac2640 1 --2- 172.21.15.98:0/1816476042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3fe780 0x5644ac400b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb78ac2640 1 -- 172.21.15.98:0/1816476042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644ac2e13a0 con 0x5644ac3ef5d0 2024-09-19T08:12:17.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb792c3640 1 --2- 172.21.15.98:0/3592651663 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feb600a3430 0x7feb600a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb71ffb640 1 -- 172.21.15.98:0/3592651663 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7feb6c0f0ea0 con 0x7feb74106f20 2024-09-19T08:12:17.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb78ac2640 1 --2- 172.21.15.98:0/1816476042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3ef5d0 0x5644ac3efa10 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7feb6400bd10 tx=0x7feb640107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a78aab1464e373 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:17.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb6affd640 1 -- 172.21.15.98:0/1816476042 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64004030 con 0x5644ac3ef5d0 2024-09-19T08:12:17.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb6affd640 1 -- 172.21.15.98:0/1816476042 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb640041d0 con 0x5644ac3ef5d0 2024-09-19T08:12:17.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.644+0000 7feb6affd640 1 -- 172.21.15.98:0/1816476042 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb640117e0 con 0x5644ac3ef5d0 2024-09-19T08:12:17.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.645+0000 7feb7d0ff740 1 -- 172.21.15.98:0/1816476042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3ef5d0 msgr2=0x5644ac3efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:17.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.645+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/1816476042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3ef5d0 0x5644ac3efa10 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7feb6400bd10 tx=0x7feb640107a0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.645+0000 7feb792c3640 1 --2- 172.21.15.98:0/3592651663 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feb600a3430 0x7feb600a58f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0eebc0 tx=0x7feb6c03d040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:17.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.645+0000 7feb7d0ff740 1 -- 172.21.15.98:0/1816476042 shutdown_connections 2024-09-19T08:12:17.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.645+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/1816476042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3fe780 0x5644ac400b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.645+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/1816476042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3ef5d0 0x5644ac3efa10 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.645+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/1816476042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 0x5644ac3ef090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.645+0000 7feb7d0ff740 1 -- 172.21.15.98:0/1816476042 >> 172.21.15.98:0/1816476042 conn(0x5644ac3b8310 msgr2=0x5644ac3ebc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:17.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.645+0000 7feb7d0ff740 1 -- 172.21.15.98:0/1816476042 shutdown_connections 2024-09-19T08:12:17.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.645+0000 7feb7d0ff740 1 -- 172.21.15.98:0/1816476042 wait complete. 2024-09-19T08:12:17.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.645+0000 7feb7d0ff740 1 -- start start 2024-09-19T08:12:17.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.645+0000 7feb7d0ff740 1 Processor -- start 2024-09-19T08:12:17.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb7d0ff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3ef5d0 0x5644ac3fe190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb7d0ff740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3f0b40 0x5644ac3fb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb7d0ff740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3fe780 0x5644ac3fba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb7d0ff740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5644ac401480 con 0x5644ac3f0b40 2024-09-19T08:12:17.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb7d0ff740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5644ac310060 con 0x5644ac3fe780 2024-09-19T08:12:17.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb7d0ff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5644ac401600 con 0x5644ac3ef5d0 2024-09-19T08:12:17.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb7d0ff740 10 client.0 _ll_register_callbacks cb 0x5644ac0638c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:17.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb78ac2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3f0b40 0x5644ac3fb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb78ac2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3f0b40 0x5644ac3fb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54260/0 (socket says 172.21.15.98:54260) 2024-09-19T08:12:17.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb78ac2640 1 -- 172.21.15.98:0/3523782039 learned_addr learned my addr 172.21.15.98:0/3523782039 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:17.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb73fff640 1 --2- 172.21.15.98:0/3523782039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3fe780 0x5644ac3fba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb73fff640 1 --2- 172.21.15.98:0/3523782039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3ef5d0 0x5644ac3fe190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb78ac2640 1 -- 172.21.15.98:0/3523782039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3ef5d0 msgr2=0x5644ac3fe190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:17.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb78ac2640 1 --2- 172.21.15.98:0/3523782039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3ef5d0 0x5644ac3fe190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb78ac2640 1 -- 172.21.15.98:0/3523782039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3fe780 msgr2=0x5644ac3fba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:17.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb78ac2640 1 --2- 172.21.15.98:0/3523782039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3fe780 0x5644ac3fba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb78ac2640 1 -- 172.21.15.98:0/3523782039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644ac0c63c0 con 0x5644ac3f0b40 2024-09-19T08:12:17.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb73fff640 1 --2- 172.21.15.98:0/3523782039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3ef5d0 0x5644ac3fe190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:12:17.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb78ac2640 1 --2- 172.21.15.98:0/3523782039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3f0b40 0x5644ac3fb440 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7feb64005920 tx=0x7feb6400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11fcb09d81a880 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.646+0000 7feb53fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.647+0000 7feb53fff640 1 -- 172.21.15.98:0/3523782039 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64029060 con 0x5644ac3f0b40 2024-09-19T08:12:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.647+0000 7feb53fff640 1 -- 172.21.15.98:0/3523782039 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7feb640035f0 con 0x5644ac3f0b40 2024-09-19T08:12:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.647+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3523782039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb640041d0 con 0x5644ac3f0b40 2024-09-19T08:12:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.647+0000 7feb53fff640 1 -- 172.21.15.98:0/3523782039 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feb640037f0 con 0x5644ac3f0b40 2024-09-19T08:12:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.647+0000 7feb7d0ff740 10 client.17580 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.647+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3523782039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5644ac40cf50 con 0x5644ac3f0b40 2024-09-19T08:12:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.647+0000 7feb53fff640 1 -- 172.21.15.98:0/3523782039 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb640041d0 con 0x5644ac3f0b40 2024-09-19T08:12:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.647+0000 7feb53fff640 1 -- 172.21.15.98:0/3523782039 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7feb64004880 con 0x5644ac3f0b40 2024-09-19T08:12:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.647+0000 7feb7d0ff740 10 client.17580 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.647+0000 7feb7d0ff740 10 client.17580 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:12:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.647+0000 7feb7d0ff740 20 client.17580 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.647+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/3523782039 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5644ac40e350 0x5644ac410740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.647+0000 7feb7d0ff740 4 client.17580 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:12:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.647+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3523782039 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644ac362860 con 0x5644ac40e350 2024-09-19T08:12:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.648+0000 7feb73fff640 1 --2- 172.21.15.98:0/3523782039 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5644ac40e350 0x5644ac410740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.648+0000 7feb73fff640 1 --2- 172.21.15.98:0/3523782039 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5644ac40e350 0x5644ac410740 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8970fc088ba38c7b server_cookie=66022898976a070 in_seq=0 out_seq=0 2024-09-19T08:12:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.648+0000 7feb53fff640 10 client.17580 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:12:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb53fff640 1 -- 172.21.15.98:0/3523782039 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5644ac362860 con 0x5644ac40e350 2024-09-19T08:12:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb53fff640 10 client.17580 handle_command_reply: tid=0 2024-09-19T08:12:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb7d0ff740 1 client.17580 shutdown 2024-09-19T08:12:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3523782039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3f0b40 msgr2=0x5644ac3fb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/3523782039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3f0b40 0x5644ac3fb440 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7feb64005920 tx=0x7feb6400b8e0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3523782039 shutdown_connections 2024-09-19T08:12:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/3523782039 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5644ac40e350 0x5644ac410740 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb53fff640 0 client.17580 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:12:17.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/3523782039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3fe780 0x5644ac3fba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/3523782039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3f0b40 0x5644ac3fb440 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/3523782039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3ef5d0 0x5644ac3fe190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3523782039 >> 172.21.15.98:0/3523782039 conn(0x5644ac3b8310 msgr2=0x5644ac3f1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:17.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3523782039 shutdown_connections 2024-09-19T08:12:17.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3523782039 wait complete. 2024-09-19T08:12:17.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.649+0000 7feb7d0ff740 20 client.17580 trim_cache size 0 max 16384 2024-09-19T08:12:17.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.662+0000 7feb7d0ff740 1 Processor -- start 2024-09-19T08:12:17.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.662+0000 7feb7d0ff740 1 -- start start 2024-09-19T08:12:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.662+0000 7feb7d0ff740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3ef5d0 0x5644ac3ebe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.662+0000 7feb7d0ff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 0x5644ac40a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.662+0000 7feb7d0ff740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3fe780 0x5644ac415ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.662+0000 7feb7d0ff740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb741ab6f0 con 0x5644ac3fe780 2024-09-19T08:12:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.662+0000 7feb7d0ff740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feb741ad230 con 0x5644ac3ef5d0 2024-09-19T08:12:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.662+0000 7feb7d0ff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb741abdc0 con 0x5644ac3f0b40 2024-09-19T08:12:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.662+0000 7feb73fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3fe780 0x5644ac415ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.662+0000 7feb73fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3fe780 0x5644ac415ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54276/0 (socket says 172.21.15.98:54276) 2024-09-19T08:12:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.662+0000 7feb73fff640 1 -- 172.21.15.98:0/179785167 learned_addr learned my addr 172.21.15.98:0/179785167 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.663+0000 7feb73fff640 1 -- 172.21.15.98:0/179785167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 msgr2=0x5644ac40a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.663+0000 7feb73fff640 1 --2- 172.21.15.98:0/179785167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 0x5644ac40a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.663+0000 7feb73fff640 1 -- 172.21.15.98:0/179785167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3ef5d0 msgr2=0x5644ac3ebe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:17.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.663+0000 7feb73fff640 1 --2- 172.21.15.98:0/179785167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3ef5d0 0x5644ac3ebe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.663+0000 7feb73fff640 1 -- 172.21.15.98:0/179785167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644ac3ec960 con 0x5644ac3fe780 2024-09-19T08:12:17.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.663+0000 7feb73fff640 1 --2- 172.21.15.98:0/179785167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3fe780 0x5644ac415ce0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7feb5c005fd0 tx=0x7feb5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4734b886905893 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:17.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.663+0000 7feb6a7fc640 1 -- 172.21.15.98:0/179785167 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5c025070 con 0x5644ac3fe780 2024-09-19T08:12:17.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.663+0000 7feb6a7fc640 1 -- 172.21.15.98:0/179785167 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb5c005500 con 0x5644ac3fe780 2024-09-19T08:12:17.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.663+0000 7feb6a7fc640 1 -- 172.21.15.98:0/179785167 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5c023640 con 0x5644ac3fe780 2024-09-19T08:12:17.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.664+0000 7feb7d0ff740 1 -- 172.21.15.98:0/179785167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3fe780 msgr2=0x5644ac415ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:17.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.664+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/179785167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3fe780 0x5644ac415ce0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7feb5c005fd0 tx=0x7feb5c004730 comp rx=0 tx=0).stop 2024-09-19T08:12:17.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.664+0000 7feb7d0ff740 1 -- 172.21.15.98:0/179785167 shutdown_connections 2024-09-19T08:12:17.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.664+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/179785167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3fe780 0x5644ac415ce0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.664+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/179785167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 0x5644ac40a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.664+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/179785167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3ef5d0 0x5644ac3ebe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.664+0000 7feb7d0ff740 1 -- 172.21.15.98:0/179785167 >> 172.21.15.98:0/179785167 conn(0x5644ac3b8310 msgr2=0x5644ac3eb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.664+0000 7feb7d0ff740 1 -- 172.21.15.98:0/179785167 shutdown_connections 2024-09-19T08:12:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.664+0000 7feb7d0ff740 1 -- 172.21.15.98:0/179785167 wait complete. 2024-09-19T08:12:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.664+0000 7feb7d0ff740 1 -- start start 2024-09-19T08:12:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.664+0000 7feb7d0ff740 1 Processor -- start 2024-09-19T08:12:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.664+0000 7feb7d0ff740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3ef5d0 0x5644ac3f8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.664+0000 7feb7d0ff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 0x5644ac3f8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb7d0ff740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3fe780 0x5644ac3f8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb7d0ff740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb741acb60 con 0x5644ac3ef5d0 2024-09-19T08:12:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb7d0ff740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feb741ac490 con 0x5644ac3fe780 2024-09-19T08:12:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb7d0ff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb741a9710 con 0x5644ac3f0b40 2024-09-19T08:12:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb7d0ff740 10 client.0 _ll_register_callbacks cb 0x7feb5c00b9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb73fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3ef5d0 0x5644ac3f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb73fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3ef5d0 0x5644ac3f8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54288/0 (socket says 172.21.15.98:54288) 2024-09-19T08:12:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb73fff640 1 -- 172.21.15.98:0/3180143335 learned_addr learned my addr 172.21.15.98:0/3180143335 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb78ac2640 1 --2- 172.21.15.98:0/3180143335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 0x5644ac3f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb73fff640 1 -- 172.21.15.98:0/3180143335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 msgr2=0x5644ac3f8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb73fff640 1 --2- 172.21.15.98:0/3180143335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 0x5644ac3f8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb73fff640 1 -- 172.21.15.98:0/3180143335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3fe780 msgr2=0x5644ac3f8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb73fff640 1 --2- 172.21.15.98:0/3180143335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3fe780 0x5644ac3f8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb73fff640 1 -- 172.21.15.98:0/3180143335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644ac3eb4f0 con 0x5644ac3ef5d0 2024-09-19T08:12:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb78ac2640 1 --2- 172.21.15.98:0/3180143335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 0x5644ac3f8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb73fff640 1 --2- 172.21.15.98:0/3180143335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3ef5d0 0x5644ac3f8430 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7feb5c004160 tx=0x7feb5c004190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ca282a60ffbde6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb53fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb53fff640 1 -- 172.21.15.98:0/3180143335 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5c036060 con 0x5644ac3ef5d0 2024-09-19T08:12:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb53fff640 1 -- 172.21.15.98:0/3180143335 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7feb5c004860 con 0x5644ac3ef5d0 2024-09-19T08:12:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3180143335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb5c005500 con 0x5644ac3ef5d0 2024-09-19T08:12:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.665+0000 7feb53fff640 1 -- 172.21.15.98:0/3180143335 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feb5c025030 con 0x5644ac3ef5d0 2024-09-19T08:12:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.666+0000 7feb7d0ff740 10 client.17586 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.666+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3180143335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5644ac478d80 con 0x5644ac3ef5d0 2024-09-19T08:12:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.666+0000 7feb53fff640 1 -- 172.21.15.98:0/3180143335 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb5c005500 con 0x5644ac3ef5d0 2024-09-19T08:12:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.666+0000 7feb53fff640 1 -- 172.21.15.98:0/3180143335 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7feb5c021040 con 0x5644ac3ef5d0 2024-09-19T08:12:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.666+0000 7feb7d0ff740 10 client.17586 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.666+0000 7feb7d0ff740 10 client.17586 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:12:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.666+0000 7feb7d0ff740 20 client.17586 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.666+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/3180143335 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5644ac40e350 0x5644ac47b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.666+0000 7feb7d0ff740 4 client.17586 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:12:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.666+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3180143335 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5644ac35cd70 con 0x5644ac40e350 2024-09-19T08:12:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.666+0000 7feb78ac2640 1 --2- 172.21.15.98:0/3180143335 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5644ac40e350 0x5644ac47b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb78ac2640 1 --2- 172.21.15.98:0/3180143335 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5644ac40e350 0x5644ac47b3e0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a9290ed8e87a106f server_cookie=34293eedaf2071f4 in_seq=0 out_seq=0 2024-09-19T08:12:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb53fff640 10 client.17586 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:12:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb53fff640 1 -- 172.21.15.98:0/3180143335 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5644ac35cd70 con 0x5644ac40e350 2024-09-19T08:12:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb53fff640 10 client.17586 handle_command_reply: tid=0 2024-09-19T08:12:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb7d0ff740 1 client.17586 shutdown 2024-09-19T08:12:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3180143335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3ef5d0 msgr2=0x5644ac3f8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/3180143335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3ef5d0 0x5644ac3f8430 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7feb5c004160 tx=0x7feb5c004190 comp rx=0 tx=0).stop 2024-09-19T08:12:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3180143335 shutdown_connections 2024-09-19T08:12:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/3180143335 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x5644ac40e350 0x5644ac47b3e0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/3180143335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644ac3fe780 0x5644ac3f8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/3180143335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644ac3f0b40 0x5644ac3f8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb7d0ff740 1 --2- 172.21.15.98:0/3180143335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ac3ef5d0 0x5644ac3f8430 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3180143335 >> 172.21.15.98:0/3180143335 conn(0x5644ac3b8310 msgr2=0x5644ac3fcca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:17.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb53fff640 0 client.17586 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:12:17.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3180143335 shutdown_connections 2024-09-19T08:12:17.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb7d0ff740 1 -- 172.21.15.98:0/3180143335 wait complete. 2024-09-19T08:12:17.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.667+0000 7feb7d0ff740 20 client.17586 trim_cache size 0 max 16384 2024-09-19T08:12:17.679 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:12:17.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.668+0000 7feb7ad26640 1 -- 172.21.15.98:0/3592651663 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feb600a3430 msgr2=0x7feb600a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:17.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.668+0000 7feb7ad26640 1 --2- 172.21.15.98:0/3592651663 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feb600a3430 0x7feb600a58f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0eebc0 tx=0x7feb6c03d040 comp rx=0 tx=0).stop 2024-09-19T08:12:17.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.668+0000 7feb7ad26640 1 -- 172.21.15.98:0/3592651663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb74106f20 msgr2=0x7feb7419ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:17.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.668+0000 7feb7ad26640 1 --2- 172.21.15.98:0/3592651663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb74106f20 0x7feb7419ccf0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7feb6c031a20 tx=0x7feb6c031a50 comp rx=0 tx=0).stop 2024-09-19T08:12:17.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.668+0000 7feb7ad26640 1 -- 172.21.15.98:0/3592651663 shutdown_connections 2024-09-19T08:12:17.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.668+0000 7feb7ad26640 1 --2- 172.21.15.98:0/3592651663 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7feb600a3430 0x7feb600a58f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.668+0000 7feb7ad26640 1 --2- 172.21.15.98:0/3592651663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb74106f20 0x7feb7419ccf0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.668+0000 7feb7ad26640 1 --2- 172.21.15.98:0/3592651663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb741027a0 0x7feb74198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.668+0000 7feb7ad26640 1 --2- 172.21.15.98:0/3592651663 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb74101df0 0x7feb74198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:17.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.668+0000 7feb7ad26640 1 -- 172.21.15.98:0/3592651663 >> 172.21.15.98:0/3592651663 conn(0x7feb740fd8b0 msgr2=0x7feb740ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:17.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.668+0000 7feb7ad26640 1 -- 172.21.15.98:0/3592651663 shutdown_connections 2024-09-19T08:12:17.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:17.668+0000 7feb7ad26640 1 -- 172.21.15.98:0/3592651663 wait complete. 2024-09-19T08:12:18.295 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:18 smithi098 ceph-mon[31333]: pgmap v286: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 147 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:12:18.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:12:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:18 smithi162 ceph-mon[35260]: pgmap v286: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 147 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:12:18.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:18 smithi139 ceph-mon[34874]: pgmap v286: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 147 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:12:18.567 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:12:19.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.073+0000 7fe100cc4640 1 -- 172.21.15.98:0/2026846205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc102790 msgr2=0x7fe0fc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.073+0000 7fe100cc4640 1 --2- 172.21.15.98:0/2026846205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc102790 0x7fe0fc1067e0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0009900 tx=0x7fe0f002a410 comp rx=0 tx=0).stop 2024-09-19T08:12:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.073+0000 7fe100cc4640 1 -- 172.21.15.98:0/2026846205 shutdown_connections 2024-09-19T08:12:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.073+0000 7fe100cc4640 1 --2- 172.21.15.98:0/2026846205 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0fc106f10 0x7fe0fc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.073+0000 7fe100cc4640 1 --2- 172.21.15.98:0/2026846205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc102790 0x7fe0fc1067e0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.073+0000 7fe100cc4640 1 --2- 172.21.15.98:0/2026846205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0fc101de0 0x7fe0fc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.073+0000 7fe100cc4640 1 -- 172.21.15.98:0/2026846205 >> 172.21.15.98:0/2026846205 conn(0x7fe0fc0fd8c0 msgr2=0x7fe0fc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.073+0000 7fe100cc4640 1 -- 172.21.15.98:0/2026846205 shutdown_connections 2024-09-19T08:12:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.073+0000 7fe100cc4640 1 -- 172.21.15.98:0/2026846205 wait complete. 2024-09-19T08:12:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.074+0000 7fe100cc4640 1 Processor -- start 2024-09-19T08:12:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.074+0000 7fe100cc4640 1 -- start start 2024-09-19T08:12:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.074+0000 7fe100cc4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0fc101de0 0x7fe0fc198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.074+0000 7fe100cc4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc102790 0x7fe0fc198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.074+0000 7fe100cc4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0fc106f10 0x7fe0fc19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.074+0000 7fe100cc4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe0fc111f90 con 0x7fe0fc102790 2024-09-19T08:12:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.074+0000 7fe100cc4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe0fc111e10 con 0x7fe0fc101de0 2024-09-19T08:12:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.074+0000 7fe100cc4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe0fc112110 con 0x7fe0fc106f10 2024-09-19T08:12:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.074+0000 7fe0f9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc102790 0x7fe0fc198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.074+0000 7fe0f9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc102790 0x7fe0fc198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54320/0 (socket says 172.21.15.98:54320) 2024-09-19T08:12:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.074+0000 7fe0f9d9b640 1 -- 172.21.15.98:0/981715335 learned_addr learned my addr 172.21.15.98:0/981715335 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.074+0000 7fe0fad9d640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0fc106f10 0x7fe0fc19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.075+0000 7fe0f9d9b640 1 -- 172.21.15.98:0/981715335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0fc106f10 msgr2=0x7fe0fc19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.075+0000 7fe0fa59c640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0fc101de0 0x7fe0fc198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.075+0000 7fe0f9d9b640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0fc106f10 0x7fe0fc19cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.075+0000 7fe0f9d9b640 1 -- 172.21.15.98:0/981715335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0fc101de0 msgr2=0x7fe0fc198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.075+0000 7fe0f9d9b640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0fc101de0 0x7fe0fc198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.075+0000 7fe0f9d9b640 1 -- 172.21.15.98:0/981715335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0fc19d3c0 con 0x7fe0fc102790 2024-09-19T08:12:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.075+0000 7fe0fad9d640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0fc106f10 0x7fe0fc19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:12:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.075+0000 7fe0fa59c640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0fc101de0 0x7fe0fc198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.075+0000 7fe0f9d9b640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc102790 0x7fe0fc198950 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0005ab0 tx=0x7fe0f00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.075+0000 7fe0eb7fe640 1 -- 172.21.15.98:0/981715335 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f003f030 con 0x7fe0fc102790 2024-09-19T08:12:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.075+0000 7fe0eb7fe640 1 -- 172.21.15.98:0/981715335 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0f0003ce0 con 0x7fe0fc102790 2024-09-19T08:12:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.075+0000 7fe0eb7fe640 1 -- 172.21.15.98:0/981715335 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0035070 con 0x7fe0fc102790 2024-09-19T08:12:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.075+0000 7fe100cc4640 1 -- 172.21.15.98:0/981715335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0fc19d650 con 0x7fe0fc102790 2024-09-19T08:12:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.075+0000 7fe100cc4640 1 -- 172.21.15.98:0/981715335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0fc19dae0 con 0x7fe0fc102790 2024-09-19T08:12:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.077+0000 7fe0eb7fe640 1 -- 172.21.15.98:0/981715335 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fe0f0042030 con 0x7fe0fc102790 2024-09-19T08:12:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.078+0000 7fe10309d740 1 Processor -- start 2024-09-19T08:12:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.078+0000 7fe10309d740 1 -- start start 2024-09-19T08:12:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.078+0000 7fe10309d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad58ab40 0x5622ad589090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.078+0000 7fe10309d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad5895d0 0x5622ad589a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.078+0000 7fe10309d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 0x5622ad596760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.078+0000 7fe10309d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5622ad4ef2f0 con 0x5622ad5895d0 2024-09-19T08:12:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.078+0000 7fe10309d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5622ad496cf0 con 0x5622ad594370 2024-09-19T08:12:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.078+0000 7fe10309d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5622ad4eb2b0 con 0x5622ad58ab40 2024-09-19T08:12:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.078+0000 7fe0fa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad5895d0 0x5622ad589a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.078+0000 7fe0fa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad5895d0 0x5622ad589a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54322/0 (socket says 172.21.15.98:54322) 2024-09-19T08:12:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.078+0000 7fe0fa59c640 1 -- 172.21.15.98:0/318916744 learned_addr learned my addr 172.21.15.98:0/318916744 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.078+0000 7fe0eb7fe640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe0e40a3430 0x7fe0e40a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0fad9d640 1 --2- 172.21.15.98:0/318916744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad58ab40 0x5622ad589090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0f9d9b640 1 --2- 172.21.15.98:0/318916744 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 0x5622ad596760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0fad9d640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe0e40a3430 0x7fe0e40a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0fa59c640 1 -- 172.21.15.98:0/318916744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad58ab40 msgr2=0x5622ad589090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:19.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0fa59c640 1 --2- 172.21.15.98:0/318916744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad58ab40 0x5622ad589090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0fa59c640 1 -- 172.21.15.98:0/318916744 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 msgr2=0x5622ad596760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:19.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0fa59c640 1 --2- 172.21.15.98:0/318916744 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 0x5622ad596760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0fa59c640 1 -- 172.21.15.98:0/318916744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622ad47b3a0 con 0x5622ad5895d0 2024-09-19T08:12:19.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0f9d9b640 1 --2- 172.21.15.98:0/318916744 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 0x5622ad596760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:19.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0eb7fe640 1 -- 172.21.15.98:0/981715335 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7fe0f00f0690 con 0x7fe0fc102790 2024-09-19T08:12:19.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0fa59c640 1 --2- 172.21.15.98:0/318916744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad5895d0 0x5622ad589a10 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00aa10 tx=0x7fe0ec00aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb079ac6f47217a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:19.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0fad9d640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe0e40a3430 0x7fe0e40a58f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4007fd0 tx=0x7fe0f40072a0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:19.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0e8ff9640 1 -- 172.21.15.98:0/318916744 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec015070 con 0x5622ad5895d0 2024-09-19T08:12:19.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0e8ff9640 1 -- 172.21.15.98:0/318916744 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0ec00cc60 con 0x5622ad5895d0 2024-09-19T08:12:19.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.079+0000 7fe0e8ff9640 1 -- 172.21.15.98:0/318916744 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec018760 con 0x5622ad5895d0 2024-09-19T08:12:19.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.080+0000 7fe10309d740 1 -- 172.21.15.98:0/318916744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad5895d0 msgr2=0x5622ad589a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:19.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.080+0000 7fe10309d740 1 --2- 172.21.15.98:0/318916744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad5895d0 0x5622ad589a10 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00aa10 tx=0x7fe0ec00aed0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.080+0000 7fe10309d740 1 -- 172.21.15.98:0/318916744 shutdown_connections 2024-09-19T08:12:19.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.080+0000 7fe10309d740 1 --2- 172.21.15.98:0/318916744 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 0x5622ad596760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.080+0000 7fe10309d740 1 --2- 172.21.15.98:0/318916744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad5895d0 0x5622ad589a10 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.080+0000 7fe10309d740 1 --2- 172.21.15.98:0/318916744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad58ab40 0x5622ad589090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.080+0000 7fe10309d740 1 -- 172.21.15.98:0/318916744 >> 172.21.15.98:0/318916744 conn(0x5622ad552310 msgr2=0x5622ad585c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:19.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.080+0000 7fe10309d740 1 -- 172.21.15.98:0/318916744 shutdown_connections 2024-09-19T08:12:19.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.080+0000 7fe10309d740 1 -- 172.21.15.98:0/318916744 wait complete. 2024-09-19T08:12:19.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.080+0000 7fe10309d740 1 -- start start 2024-09-19T08:12:19.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe10309d740 1 Processor -- start 2024-09-19T08:12:19.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe10309d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad5895d0 0x5622ad593d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe10309d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad58ab40 0x5622ad591010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe10309d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad594370 0x5622ad591600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe10309d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5622ad597070 con 0x5622ad594370 2024-09-19T08:12:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe10309d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5622ad4aa060 con 0x5622ad58ab40 2024-09-19T08:12:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe10309d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5622ad5971f0 con 0x5622ad5895d0 2024-09-19T08:12:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe10309d740 10 client.0 _ll_register_callbacks cb 0x5622ad1fd8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe0fad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad594370 0x5622ad591600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe0fad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad594370 0x5622ad591600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54338/0 (socket says 172.21.15.98:54338) 2024-09-19T08:12:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe0fad9d640 1 -- 172.21.15.98:0/1271955191 learned_addr learned my addr 172.21.15.98:0/1271955191 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe0fa59c640 1 --2- 172.21.15.98:0/1271955191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad5895d0 0x5622ad593d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe0f9d9b640 1 --2- 172.21.15.98:0/1271955191 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad58ab40 0x5622ad591010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe0fad9d640 1 -- 172.21.15.98:0/1271955191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad5895d0 msgr2=0x5622ad593d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe0fad9d640 1 --2- 172.21.15.98:0/1271955191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad5895d0 0x5622ad593d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe0fad9d640 1 -- 172.21.15.98:0/1271955191 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad58ab40 msgr2=0x5622ad591010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe0fad9d640 1 --2- 172.21.15.98:0/1271955191 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad58ab40 0x5622ad591010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe0fad9d640 1 -- 172.21.15.98:0/1271955191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622ad2603c0 con 0x5622ad594370 2024-09-19T08:12:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe0fad9d640 1 --2- 172.21.15.98:0/1271955191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad594370 0x5622ad591600 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4010f20 tx=0x7fe0f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db780da65fb3973d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.081+0000 7fe0e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.082+0000 7fe0e1ffb640 1 -- 172.21.15.98:0/1271955191 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4024020 con 0x5622ad594370 2024-09-19T08:12:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.082+0000 7fe0e1ffb640 1 -- 172.21.15.98:0/1271955191 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7fe0f400ed10 con 0x5622ad594370 2024-09-19T08:12:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.082+0000 7fe10309d740 1 -- 172.21.15.98:0/1271955191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0ec00cc60 con 0x5622ad594370 2024-09-19T08:12:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.082+0000 7fe0e1ffb640 1 -- 172.21.15.98:0/1271955191 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe0f402f050 con 0x5622ad594370 2024-09-19T08:12:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.082+0000 7fe10309d740 10 client.17598 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.082+0000 7fe10309d740 1 -- 172.21.15.98:0/1271955191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5622ad5a2ae0 con 0x5622ad594370 2024-09-19T08:12:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.082+0000 7fe0e1ffb640 1 -- 172.21.15.98:0/1271955191 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0ec00cc60 con 0x5622ad594370 2024-09-19T08:12:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.082+0000 7fe0e1ffb640 1 -- 172.21.15.98:0/1271955191 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe0f401f030 con 0x5622ad594370 2024-09-19T08:12:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.082+0000 7fe10309d740 10 client.17598 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.082+0000 7fe10309d740 10 client.17598 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:12:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.082+0000 7fe10309d740 20 client.17598 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.082+0000 7fe10309d740 1 --2- 172.21.15.98:0/1271955191 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5622ad5a3fa0 0x5622ad5a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.082+0000 7fe10309d740 4 client.17598 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:12:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.082+0000 7fe10309d740 1 -- 172.21.15.98:0/1271955191 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622ad4fc860 con 0x5622ad5a3fa0 2024-09-19T08:12:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.083+0000 7fe0fa59c640 1 --2- 172.21.15.98:0/1271955191 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5622ad5a3fa0 0x5622ad5a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.083+0000 7fe0fa59c640 1 --2- 172.21.15.98:0/1271955191 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5622ad5a3fa0 0x5622ad5a63f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=471ec01a4b322510 server_cookie=ce5b2a074cdf60d4 in_seq=0 out_seq=0 2024-09-19T08:12:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.083+0000 7fe0e1ffb640 10 client.17598 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:12:19.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe0e1ffb640 1 -- 172.21.15.98:0/1271955191 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5622ad4fc860 con 0x5622ad5a3fa0 2024-09-19T08:12:19.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe0e1ffb640 10 client.17598 handle_command_reply: tid=0 2024-09-19T08:12:19.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe10309d740 1 client.17598 shutdown 2024-09-19T08:12:19.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe10309d740 1 -- 172.21.15.98:0/1271955191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad594370 msgr2=0x5622ad591600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:19.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe10309d740 1 --2- 172.21.15.98:0/1271955191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad594370 0x5622ad591600 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4010f20 tx=0x7fe0f4004a40 comp rx=0 tx=0).stop 2024-09-19T08:12:19.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe10309d740 1 -- 172.21.15.98:0/1271955191 shutdown_connections 2024-09-19T08:12:19.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe10309d740 1 --2- 172.21.15.98:0/1271955191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad594370 0x5622ad591600 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe10309d740 1 --2- 172.21.15.98:0/1271955191 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad58ab40 0x5622ad591010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe10309d740 1 --2- 172.21.15.98:0/1271955191 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5622ad5a3fa0 0x5622ad5a63f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe10309d740 1 --2- 172.21.15.98:0/1271955191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad5895d0 0x5622ad593d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe10309d740 1 -- 172.21.15.98:0/1271955191 >> 172.21.15.98:0/1271955191 conn(0x5622ad552310 msgr2=0x5622ad58b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:19.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe0e1ffb640 0 client.17598 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:12:19.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe10309d740 1 -- 172.21.15.98:0/1271955191 shutdown_connections 2024-09-19T08:12:19.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe10309d740 1 -- 172.21.15.98:0/1271955191 wait complete. 2024-09-19T08:12:19.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.084+0000 7fe10309d740 20 client.17598 trim_cache size 0 max 16384 2024-09-19T08:12:19.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.097+0000 7fe10309d740 1 Processor -- start 2024-09-19T08:12:19.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.097+0000 7fe10309d740 1 -- start start 2024-09-19T08:12:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.097+0000 7fe10309d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad5895d0 0x5622ad592f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.097+0000 7fe10309d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad58ab40 0x5622ad58c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.097+0000 7fe10309d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 0x5622ad58c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.097+0000 7fe10309d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe0fc1ad8c0 con 0x5622ad5895d0 2024-09-19T08:12:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.097+0000 7fe10309d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe0fc1af400 con 0x5622ad594370 2024-09-19T08:12:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.097+0000 7fe10309d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe0fc1adf90 con 0x5622ad58ab40 2024-09-19T08:12:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.097+0000 7fe0fa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad5895d0 0x5622ad592f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.097+0000 7fe0fa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad5895d0 0x5622ad592f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54344/0 (socket says 172.21.15.98:54344) 2024-09-19T08:12:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.097+0000 7fe0fa59c640 1 -- 172.21.15.98:0/718817584 learned_addr learned my addr 172.21.15.98:0/718817584 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.097+0000 7fe0f9d9b640 1 --2- 172.21.15.98:0/718817584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad58ab40 0x5622ad58c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.097+0000 7fe0fa59c640 1 -- 172.21.15.98:0/718817584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad58ab40 msgr2=0x5622ad58c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.098+0000 7fe0fa59c640 1 --2- 172.21.15.98:0/718817584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad58ab40 0x5622ad58c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.098+0000 7fe0fa59c640 1 -- 172.21.15.98:0/718817584 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 msgr2=0x5622ad58c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.098+0000 7fe0fad9d640 1 --2- 172.21.15.98:0/718817584 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 0x5622ad58c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.098+0000 7fe0fa59c640 1 --2- 172.21.15.98:0/718817584 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 0x5622ad58c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.098+0000 7fe0fa59c640 1 -- 172.21.15.98:0/718817584 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622ad5a6a70 con 0x5622ad5895d0 2024-09-19T08:12:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.098+0000 7fe0fa59c640 1 --2- 172.21.15.98:0/718817584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad5895d0 0x5622ad592f80 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00aa10 tx=0x7fe0ec018360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c2fed092c5c586 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.098+0000 7fe0e3fff640 1 -- 172.21.15.98:0/718817584 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec015040 con 0x5622ad5895d0 2024-09-19T08:12:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.098+0000 7fe0e3fff640 1 -- 172.21.15.98:0/718817584 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0ec02e070 con 0x5622ad5895d0 2024-09-19T08:12:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.098+0000 7fe0e3fff640 1 -- 172.21.15.98:0/718817584 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec027620 con 0x5622ad5895d0 2024-09-19T08:12:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.098+0000 7fe10309d740 1 -- 172.21.15.98:0/718817584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad5895d0 msgr2=0x5622ad592f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 --2- 172.21.15.98:0/718817584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad5895d0 0x5622ad592f80 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00aa10 tx=0x7fe0ec018360 comp rx=0 tx=0).stop 2024-09-19T08:12:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 -- 172.21.15.98:0/718817584 shutdown_connections 2024-09-19T08:12:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 --2- 172.21.15.98:0/718817584 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 0x5622ad58c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 --2- 172.21.15.98:0/718817584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad58ab40 0x5622ad58c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 --2- 172.21.15.98:0/718817584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad5895d0 0x5622ad592f80 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 -- 172.21.15.98:0/718817584 >> 172.21.15.98:0/718817584 conn(0x5622ad552310 msgr2=0x5622ad592870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 -- 172.21.15.98:0/718817584 shutdown_connections 2024-09-19T08:12:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 -- 172.21.15.98:0/718817584 wait complete. 2024-09-19T08:12:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 -- start start 2024-09-19T08:12:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 Processor -- start 2024-09-19T08:12:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad5895d0 0x5622ad59cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad58ab40 0x5622ad58dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 0x5622ad58e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe0fc1aed30 con 0x5622ad58ab40 2024-09-19T08:12:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe0fc1ae660 con 0x5622ad594370 2024-09-19T08:12:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe0fc1ab8c0 con 0x5622ad5895d0 2024-09-19T08:12:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe10309d740 10 client.0 _ll_register_callbacks cb 0x7fe0f4007c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe0f9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad58ab40 0x5622ad58dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe0f9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad58ab40 0x5622ad58dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54360/0 (socket says 172.21.15.98:54360) 2024-09-19T08:12:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.099+0000 7fe0f9d9b640 1 -- 172.21.15.98:0/1292188460 learned_addr learned my addr 172.21.15.98:0/1292188460 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe0fad9d640 1 --2- 172.21.15.98:0/1292188460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 0x5622ad58e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe0fa59c640 1 --2- 172.21.15.98:0/1292188460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad5895d0 0x5622ad59cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe0f9d9b640 1 -- 172.21.15.98:0/1292188460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad5895d0 msgr2=0x5622ad59cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe0f9d9b640 1 --2- 172.21.15.98:0/1292188460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad5895d0 0x5622ad59cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe0f9d9b640 1 -- 172.21.15.98:0/1292188460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 msgr2=0x5622ad58e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe0f9d9b640 1 --2- 172.21.15.98:0/1292188460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 0x5622ad58e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe0f9d9b640 1 -- 172.21.15.98:0/1292188460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622ad58e9b0 con 0x5622ad58ab40 2024-09-19T08:12:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe0fad9d640 1 --2- 172.21.15.98:0/1292188460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 0x5622ad58e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:12:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe0f9d9b640 1 --2- 172.21.15.98:0/1292188460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad58ab40 0x5622ad58dc90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00b9310 tx=0x7fe0f002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba7b5de908d34e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe0e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:19.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe0e1ffb640 1 -- 172.21.15.98:0/1292188460 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0049040 con 0x5622ad58ab40 2024-09-19T08:12:19.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe0e1ffb640 1 -- 172.21.15.98:0/1292188460 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7fe0f00b8fb0 con 0x5622ad58ab40 2024-09-19T08:12:19.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe10309d740 1 -- 172.21.15.98:0/1292188460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0ec02e070 con 0x5622ad58ab40 2024-09-19T08:12:19.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe0e1ffb640 1 -- 172.21.15.98:0/1292188460 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe0f00bd040 con 0x5622ad58ab40 2024-09-19T08:12:19.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe10309d740 10 client.17604 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:19.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe10309d740 1 -- 172.21.15.98:0/1292188460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5622ad60ecf0 con 0x5622ad58ab40 2024-09-19T08:12:19.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.100+0000 7fe0e1ffb640 1 -- 172.21.15.98:0/1292188460 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0ec02e070 con 0x5622ad58ab40 2024-09-19T08:12:19.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.101+0000 7fe0e1ffb640 1 -- 172.21.15.98:0/1292188460 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe0f003b460 con 0x5622ad58ab40 2024-09-19T08:12:19.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.101+0000 7fe10309d740 10 client.17604 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:19.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.101+0000 7fe10309d740 10 client.17604 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:12:19.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.101+0000 7fe10309d740 20 client.17604 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:19.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.101+0000 7fe10309d740 1 --2- 172.21.15.98:0/1292188460 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5622ad60f440 0x5622ad611860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:19.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.101+0000 7fe10309d740 4 client.17604 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:12:19.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.101+0000 7fe10309d740 1 -- 172.21.15.98:0/1292188460 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5622ad4f6d70 con 0x5622ad60f440 2024-09-19T08:12:19.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.101+0000 7fe0fa59c640 1 --2- 172.21.15.98:0/1292188460 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5622ad60f440 0x5622ad611860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:19.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.101+0000 7fe0fa59c640 1 --2- 172.21.15.98:0/1292188460 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5622ad60f440 0x5622ad611860 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=91a4f187014846f6 server_cookie=41af5eec3a457e31 in_seq=0 out_seq=0 2024-09-19T08:12:19.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.101+0000 7fe0e1ffb640 10 client.17604 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:12:19.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.101+0000 7fe0e1ffb640 1 -- 172.21.15.98:0/1292188460 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5622ad4f6d70 con 0x5622ad60f440 2024-09-19T08:12:19.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.101+0000 7fe0e1ffb640 10 client.17604 handle_command_reply: tid=0 2024-09-19T08:12:19.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.101+0000 7fe10309d740 1 client.17604 shutdown 2024-09-19T08:12:19.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe10309d740 1 -- 172.21.15.98:0/1292188460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad58ab40 msgr2=0x5622ad58dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:19.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe10309d740 1 --2- 172.21.15.98:0/1292188460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad58ab40 0x5622ad58dc90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00b9310 tx=0x7fe0f002a410 comp rx=0 tx=0).stop 2024-09-19T08:12:19.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe10309d740 1 -- 172.21.15.98:0/1292188460 shutdown_connections 2024-09-19T08:12:19.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe10309d740 1 --2- 172.21.15.98:0/1292188460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5622ad594370 0x5622ad58e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe10309d740 1 --2- 172.21.15.98:0/1292188460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5622ad58ab40 0x5622ad58dc90 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe10309d740 1 --2- 172.21.15.98:0/1292188460 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x5622ad60f440 0x5622ad611860 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe10309d740 1 --2- 172.21.15.98:0/1292188460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5622ad5895d0 0x5622ad59cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe10309d740 1 -- 172.21.15.98:0/1292188460 >> 172.21.15.98:0/1292188460 conn(0x5622ad552310 msgr2=0x5622ad584d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:19.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe0e1ffb640 0 client.17604 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:12:19.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe10309d740 1 -- 172.21.15.98:0/1292188460 shutdown_connections 2024-09-19T08:12:19.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe10309d740 1 -- 172.21.15.98:0/1292188460 wait complete. 2024-09-19T08:12:19.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe10309d740 20 client.17604 trim_cache size 0 max 16384 2024-09-19T08:12:19.109 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:12:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe100cc4640 1 -- 172.21.15.98:0/981715335 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe0e40a3430 msgr2=0x7fe0e40a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe100cc4640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe0e40a3430 0x7fe0e40a58f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4007fd0 tx=0x7fe0f40072a0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe100cc4640 1 -- 172.21.15.98:0/981715335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc102790 msgr2=0x7fe0fc198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe100cc4640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc102790 0x7fe0fc198950 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0005ab0 tx=0x7fe0f00042f0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe100cc4640 1 -- 172.21.15.98:0/981715335 shutdown_connections 2024-09-19T08:12:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe100cc4640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe0e40a3430 0x7fe0e40a58f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe100cc4640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0fc106f10 0x7fe0fc19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe100cc4640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc102790 0x7fe0fc198950 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe100cc4640 1 --2- 172.21.15.98:0/981715335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0fc101de0 0x7fe0fc198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe100cc4640 1 -- 172.21.15.98:0/981715335 >> 172.21.15.98:0/981715335 conn(0x7fe0fc0fd8c0 msgr2=0x7fe0fc0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.102+0000 7fe100cc4640 1 -- 172.21.15.98:0/981715335 shutdown_connections 2024-09-19T08:12:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:19.103+0000 7fe100cc4640 1 -- 172.21.15.98:0/981715335 wait complete. 2024-09-19T08:12:19.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:12:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:12:20.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:20 smithi098 ceph-mon[31333]: pgmap v287: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:12:20.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:20 smithi162 ceph-mon[35260]: pgmap v287: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:12:20.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:20 smithi139 ceph-mon[34874]: pgmap v287: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:12:20.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.671+0000 7f9ab5495640 1 -- 172.21.15.98:0/1830684918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ab01037a0 msgr2=0x7f9ab010bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:20.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.671+0000 7f9ab5495640 1 --2- 172.21.15.98:0/1830684918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ab01037a0 0x7f9ab010bc00 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009900 tx=0x7f9a9802a410 comp rx=0 tx=0).stop 2024-09-19T08:12:20.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.671+0000 7f9ab5495640 1 -- 172.21.15.98:0/1830684918 shutdown_connections 2024-09-19T08:12:20.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.671+0000 7f9ab5495640 1 --2- 172.21.15.98:0/1830684918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9ab010c2f0 0x7f9ab010e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.671+0000 7f9ab5495640 1 --2- 172.21.15.98:0/1830684918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ab01037a0 0x7f9ab010bc00 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.671+0000 7f9ab5495640 1 --2- 172.21.15.98:0/1830684918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ab0102df0 0x7f9ab01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.671+0000 7f9ab5495640 1 -- 172.21.15.98:0/1830684918 >> 172.21.15.98:0/1830684918 conn(0x7f9ab00fc7d0 msgr2=0x7f9ab00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.671+0000 7f9ab5495640 1 -- 172.21.15.98:0/1830684918 shutdown_connections 2024-09-19T08:12:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.671+0000 7f9ab5495640 1 -- 172.21.15.98:0/1830684918 wait complete. 2024-09-19T08:12:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.671+0000 7f9ab5495640 1 Processor -- start 2024-09-19T08:12:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.671+0000 7f9ab5495640 1 -- start start 2024-09-19T08:12:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9ab5495640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ab0102df0 0x7f9ab0198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9ab5495640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9ab01037a0 0x7f9ab0198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9ab5495640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ab010c2f0 0x7f9ab019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9ab5495640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab0112150 con 0x7f9ab010c2f0 2024-09-19T08:12:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9ab5495640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab0111fd0 con 0x7f9ab01037a0 2024-09-19T08:12:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9ab5495640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab01122d0 con 0x7f9ab0102df0 2024-09-19T08:12:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aaf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ab010c2f0 0x7f9ab019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aaf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ab010c2f0 0x7f9ab019cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54384/0 (socket says 172.21.15.98:54384) 2024-09-19T08:12:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aaf7fe640 1 -- 172.21.15.98:0/4228176589 learned_addr learned my addr 172.21.15.98:0/4228176589 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aaeffd640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ab0102df0 0x7f9ab0198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9ab01037a0 0x7f9ab0198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aaf7fe640 1 -- 172.21.15.98:0/4228176589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ab0102df0 msgr2=0x7f9ab0198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aaf7fe640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ab0102df0 0x7f9ab0198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aaf7fe640 1 -- 172.21.15.98:0/4228176589 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9ab01037a0 msgr2=0x7f9ab0198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aaf7fe640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9ab01037a0 0x7f9ab0198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aaf7fe640 1 -- 172.21.15.98:0/4228176589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab019d620 con 0x7f9ab010c2f0 2024-09-19T08:12:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9ab01037a0 0x7f9ab0198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:12:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aaf7fe640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ab010c2f0 0x7f9ab019cf40 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f9aa800b970 tx=0x7f9aa800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aacff9640 1 -- 172.21.15.98:0/4228176589 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8019070 con 0x7f9ab010c2f0 2024-09-19T08:12:20.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aacff9640 1 -- 172.21.15.98:0/4228176589 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aa800cb40 con 0x7f9ab010c2f0 2024-09-19T08:12:20.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9ab5495640 1 -- 172.21.15.98:0/4228176589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab019d910 con 0x7f9ab010c2f0 2024-09-19T08:12:20.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9aacff9640 1 -- 172.21.15.98:0/4228176589 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8007650 con 0x7f9ab010c2f0 2024-09-19T08:12:20.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.672+0000 7f9ab5495640 1 -- 172.21.15.98:0/4228176589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab01a51f0 con 0x7f9ab010c2f0 2024-09-19T08:12:20.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.675+0000 7f9aacff9640 1 -- 172.21.15.98:0/4228176589 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f9aa801c070 con 0x7f9ab010c2f0 2024-09-19T08:12:20.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.675+0000 7f9ab786e740 1 Processor -- start 2024-09-19T08:12:20.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.675+0000 7f9ab786e740 1 -- start start 2024-09-19T08:12:20.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.675+0000 7f9ab786e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f00fb40 0x55733f00e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.675+0000 7f9ab786e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f00ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.675+0000 7f9ab786e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f019370 0x55733f01b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.675+0000 7f9ab786e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55733ef742f0 con 0x55733f00e5d0 2024-09-19T08:12:20.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.675+0000 7f9ab786e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55733ef1bcf0 con 0x55733f019370 2024-09-19T08:12:20.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.675+0000 7f9ab786e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55733ef702b0 con 0x55733f00fb40 2024-09-19T08:12:20.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.675+0000 7f9aacff9640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9a9c0a3430 0x7f9a9c0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aaeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f00ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aaeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f00ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54392/0 (socket says 172.21.15.98:54392) 2024-09-19T08:12:20.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aaeffd640 1 -- 172.21.15.98:0/1718272788 learned_addr learned my addr 172.21.15.98:0/1718272788 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:20.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aaf7fe640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9a9c0a3430 0x7f9a9c0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/1718272788 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f019370 0x55733f01b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aacff9640 1 -- 172.21.15.98:0/4228176589 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7f9aa80cf120 con 0x7f9ab010c2f0 2024-09-19T08:12:20.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/1718272788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f00fb40 0x55733f00e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aaeffd640 1 -- 172.21.15.98:0/1718272788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f00fb40 msgr2=0x55733f00e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:20.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aaeffd640 1 --2- 172.21.15.98:0/1718272788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f00fb40 0x55733f00e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aaeffd640 1 -- 172.21.15.98:0/1718272788 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f019370 msgr2=0x55733f01b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:20.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aaeffd640 1 --2- 172.21.15.98:0/1718272788 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f019370 0x55733f01b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aaeffd640 1 -- 172.21.15.98:0/1718272788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55733ef003a0 con 0x55733f00e5d0 2024-09-19T08:12:20.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/1718272788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f00fb40 0x55733f00e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:12:20.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aaf7fe640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9a9c0a3430 0x7f9a9c0a58f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80ccaa0 tx=0x7f9aa8004a40 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:20.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aaeffd640 1 --2- 172.21.15.98:0/1718272788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f00ea10 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000b760 tx=0x7f9aa000bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a226ed80cade514 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:20.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aa57fa640 1 -- 172.21.15.98:0/1718272788 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0003dd0 con 0x55733f00e5d0 2024-09-19T08:12:20.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aa57fa640 1 -- 172.21.15.98:0/1718272788 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aa0015070 con 0x55733f00e5d0 2024-09-19T08:12:20.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.676+0000 7f9aa57fa640 1 -- 172.21.15.98:0/1718272788 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0004e60 con 0x55733f00e5d0 2024-09-19T08:12:20.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.677+0000 7f9ab786e740 1 -- 172.21.15.98:0/1718272788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 msgr2=0x55733f00ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:20.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.677+0000 7f9ab786e740 1 --2- 172.21.15.98:0/1718272788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f00ea10 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000b760 tx=0x7f9aa000bc20 comp rx=0 tx=0).stop 2024-09-19T08:12:20.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.677+0000 7f9ab786e740 1 -- 172.21.15.98:0/1718272788 shutdown_connections 2024-09-19T08:12:20.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.677+0000 7f9ab786e740 1 --2- 172.21.15.98:0/1718272788 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f019370 0x55733f01b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.677+0000 7f9ab786e740 1 --2- 172.21.15.98:0/1718272788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f00ea10 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.677+0000 7f9ab786e740 1 --2- 172.21.15.98:0/1718272788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f00fb40 0x55733f00e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.677+0000 7f9ab786e740 1 -- 172.21.15.98:0/1718272788 >> 172.21.15.98:0/1718272788 conn(0x55733efd7310 msgr2=0x55733f00ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:20.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.677+0000 7f9ab786e740 1 -- 172.21.15.98:0/1718272788 shutdown_connections 2024-09-19T08:12:20.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.677+0000 7f9ab786e740 1 -- 172.21.15.98:0/1718272788 wait complete. 2024-09-19T08:12:20.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.677+0000 7f9ab786e740 1 -- start start 2024-09-19T08:12:20.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9ab786e740 1 Processor -- start 2024-09-19T08:12:20.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9ab786e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f00e5d0 0x55733f018d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9ab786e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00fb40 0x55733f016000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9ab786e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f019370 0x55733f0165d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9ab786e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55733f01c070 con 0x55733f00fb40 2024-09-19T08:12:20.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9ab786e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55733ef2f060 con 0x55733f019370 2024-09-19T08:12:20.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9ab786e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55733f01c1f0 con 0x55733f00e5d0 2024-09-19T08:12:20.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9ab786e740 10 client.0 _ll_register_callbacks cb 0x55733ec828c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:20.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9aaeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00fb40 0x55733f016000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9aaeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00fb40 0x55733f016000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54402/0 (socket says 172.21.15.98:54402) 2024-09-19T08:12:20.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9aaeffd640 1 -- 172.21.15.98:0/1985401445 learned_addr learned my addr 172.21.15.98:0/1985401445 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:20.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/1985401445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f00e5d0 0x55733f018d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/1985401445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f019370 0x55733f0165d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9aaeffd640 1 -- 172.21.15.98:0/1985401445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f00e5d0 msgr2=0x55733f018d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:20.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9aaeffd640 1 --2- 172.21.15.98:0/1985401445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f00e5d0 0x55733f018d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9aaeffd640 1 -- 172.21.15.98:0/1985401445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f019370 msgr2=0x55733f0165d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:20.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9aaeffd640 1 --2- 172.21.15.98:0/1985401445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f019370 0x55733f0165d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9aaeffd640 1 -- 172.21.15.98:0/1985401445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55733ece53c0 con 0x55733f00fb40 2024-09-19T08:12:20.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/1985401445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f00e5d0 0x55733f018d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:12:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9aaeffd640 1 --2- 172.21.15.98:0/1985401445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00fb40 0x55733f016000 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00039c0 tx=0x7f9aa000bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ea3bee1e60396e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.678+0000 7f9a967fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9a967fc640 1 -- 172.21.15.98:0/1985401445 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0028060 con 0x55733f00fb40 2024-09-19T08:12:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9a967fc640 1 -- 172.21.15.98:0/1985401445 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7f9aa0011640 con 0x55733f00fb40 2024-09-19T08:12:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9ab786e740 1 -- 172.21.15.98:0/1985401445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9aa0015070 con 0x55733f00fb40 2024-09-19T08:12:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9a967fc640 1 -- 172.21.15.98:0/1985401445 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9aa0003e60 con 0x55733f00fb40 2024-09-19T08:12:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9ab786e740 10 client.17616 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9ab786e740 1 -- 172.21.15.98:0/1985401445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55733f027af0 con 0x55733f00fb40 2024-09-19T08:12:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9a967fc640 1 -- 172.21.15.98:0/1985401445 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aa0015070 con 0x55733f00fb40 2024-09-19T08:12:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9a967fc640 1 -- 172.21.15.98:0/1985401445 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9aa0022070 con 0x55733f00fb40 2024-09-19T08:12:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9ab786e740 10 client.17616 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9ab786e740 10 client.17616 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:12:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9ab786e740 20 client.17616 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9ab786e740 1 --2- 172.21.15.98:0/1985401445 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55733f028ef0 0x55733f02b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9ab786e740 4 client.17616 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:12:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9ab786e740 1 -- 172.21.15.98:0/1985401445 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55733ef81860 con 0x55733f028ef0 2024-09-19T08:12:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.679+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/1985401445 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55733f028ef0 0x55733f02b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.680+0000 7f9a967fc640 10 client.17616 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:12:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.680+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/1985401445 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55733f028ef0 0x55733f02b2e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=35a6ff63f6f6b93d server_cookie=c87f42e1940617d5 in_seq=0 out_seq=0 2024-09-19T08:12:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9a967fc640 1 -- 172.21.15.98:0/1985401445 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55733ef81860 con 0x55733f028ef0 2024-09-19T08:12:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9a967fc640 10 client.17616 handle_command_reply: tid=0 2024-09-19T08:12:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9ab786e740 1 client.17616 shutdown 2024-09-19T08:12:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9ab786e740 1 -- 172.21.15.98:0/1985401445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00fb40 msgr2=0x55733f016000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9ab786e740 1 --2- 172.21.15.98:0/1985401445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00fb40 0x55733f016000 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00039c0 tx=0x7f9aa000bf00 comp rx=0 tx=0).stop 2024-09-19T08:12:20.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9ab786e740 1 -- 172.21.15.98:0/1985401445 shutdown_connections 2024-09-19T08:12:20.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9ab786e740 1 --2- 172.21.15.98:0/1985401445 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55733f028ef0 0x55733f02b2e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9a967fc640 0 client.17616 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:12:20.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9ab786e740 1 --2- 172.21.15.98:0/1985401445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f019370 0x55733f0165d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9ab786e740 1 --2- 172.21.15.98:0/1985401445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00fb40 0x55733f016000 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9ab786e740 1 --2- 172.21.15.98:0/1985401445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f00e5d0 0x55733f018d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9ab786e740 1 -- 172.21.15.98:0/1985401445 >> 172.21.15.98:0/1985401445 conn(0x55733efd7310 msgr2=0x55733f010960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:20.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9ab786e740 1 -- 172.21.15.98:0/1985401445 shutdown_connections 2024-09-19T08:12:20.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9ab786e740 1 -- 172.21.15.98:0/1985401445 wait complete. 2024-09-19T08:12:20.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.681+0000 7f9ab786e740 20 client.17616 trim_cache size 0 max 16384 2024-09-19T08:12:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.694+0000 7f9ab786e740 1 Processor -- start 2024-09-19T08:12:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.694+0000 7f9ab786e740 1 -- start start 2024-09-19T08:12:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9ab786e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f025310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9ab786e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f00fb40 0x55733f011120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9ab786e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f019370 0x55733f011660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9ab786e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab01ad920 con 0x55733f00e5d0 2024-09-19T08:12:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9ab786e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab01af460 con 0x55733f00fb40 2024-09-19T08:12:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9ab786e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab01adff0 con 0x55733f019370 2024-09-19T08:12:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9aa67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f025310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9aa67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f025310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54418/0 (socket says 172.21.15.98:54418) 2024-09-19T08:12:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9aa67fc640 1 -- 172.21.15.98:0/314261091 learned_addr learned my addr 172.21.15.98:0/314261091 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9aa67fc640 1 -- 172.21.15.98:0/314261091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f019370 msgr2=0x55733f011660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/314261091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f019370 0x55733f011660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9aaeffd640 1 --2- 172.21.15.98:0/314261091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f00fb40 0x55733f011120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9aa67fc640 1 -- 172.21.15.98:0/314261091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f00fb40 msgr2=0x55733f011120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/314261091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f00fb40 0x55733f011120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9aa67fc640 1 -- 172.21.15.98:0/314261091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55733f00b960 con 0x55733f00e5d0 2024-09-19T08:12:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/314261091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f025310 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f9a98005ec0 tx=0x7f9a98009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f410f046b7dd9603 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9aa4ff9640 1 -- 172.21.15.98:0/314261091 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9804f070 con 0x55733f00e5d0 2024-09-19T08:12:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9aa4ff9640 1 -- 172.21.15.98:0/314261091 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a980044d0 con 0x55733f00e5d0 2024-09-19T08:12:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.695+0000 7f9aa4ff9640 1 -- 172.21.15.98:0/314261091 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a98035040 con 0x55733f00e5d0 2024-09-19T08:12:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.696+0000 7f9ab786e740 1 -- 172.21.15.98:0/314261091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 msgr2=0x55733f025310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.696+0000 7f9ab786e740 1 --2- 172.21.15.98:0/314261091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f025310 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f9a98005ec0 tx=0x7f9a98009260 comp rx=0 tx=0).stop 2024-09-19T08:12:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.696+0000 7f9ab786e740 1 -- 172.21.15.98:0/314261091 shutdown_connections 2024-09-19T08:12:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.696+0000 7f9ab786e740 1 --2- 172.21.15.98:0/314261091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f019370 0x55733f011660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.696+0000 7f9ab786e740 1 --2- 172.21.15.98:0/314261091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f00fb40 0x55733f011120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.696+0000 7f9ab786e740 1 --2- 172.21.15.98:0/314261091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f025310 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.696+0000 7f9ab786e740 1 -- 172.21.15.98:0/314261091 >> 172.21.15.98:0/314261091 conn(0x55733efd7310 msgr2=0x55733f015110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.696+0000 7f9ab786e740 1 -- 172.21.15.98:0/314261091 shutdown_connections 2024-09-19T08:12:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.696+0000 7f9ab786e740 1 -- 172.21.15.98:0/314261091 wait complete. 2024-09-19T08:12:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9ab786e740 1 -- start start 2024-09-19T08:12:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9ab786e740 1 Processor -- start 2024-09-19T08:12:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9ab786e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f021a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9ab786e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f00fb40 0x55733f012b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9ab786e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f019370 0x55733f013070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9ab786e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab01aed90 con 0x55733f00e5d0 2024-09-19T08:12:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9ab786e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab01ae6c0 con 0x55733f00fb40 2024-09-19T08:12:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9ab786e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab01ab920 con 0x55733f019370 2024-09-19T08:12:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9ab786e740 10 client.0 _ll_register_callbacks cb 0x7f9a98009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:20.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9aa67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f021a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9aa67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f021a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54432/0 (socket says 172.21.15.98:54432) 2024-09-19T08:12:20.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9aa67fc640 1 -- 172.21.15.98:0/2925655356 learned_addr learned my addr 172.21.15.98:0/2925655356 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:20.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9aaeffd640 1 --2- 172.21.15.98:0/2925655356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f00fb40 0x55733f012b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/2925655356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f019370 0x55733f013070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9aa67fc640 1 -- 172.21.15.98:0/2925655356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f019370 msgr2=0x55733f013070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:20.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/2925655356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f019370 0x55733f013070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9aa67fc640 1 -- 172.21.15.98:0/2925655356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f00fb40 msgr2=0x55733f012b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:20.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/2925655356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f00fb40 0x55733f012b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.697+0000 7f9aa67fc640 1 -- 172.21.15.98:0/2925655356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55733f013660 con 0x55733f00e5d0 2024-09-19T08:12:20.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9aa67fc640 1 --2- 172.21.15.98:0/2925655356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f021a00 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f9a98005e10 tx=0x7f9a980314e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f18a745fabb8a8f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9a967fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9a967fc640 1 -- 172.21.15.98:0/2925655356 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a98061060 con 0x55733f00e5d0 2024-09-19T08:12:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9a967fc640 1 -- 172.21.15.98:0/2925655356 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7f9a9804b580 con 0x55733f00e5d0 2024-09-19T08:12:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9ab786e740 1 -- 172.21.15.98:0/2925655356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a980044d0 con 0x55733f00e5d0 2024-09-19T08:12:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9a967fc640 1 -- 172.21.15.98:0/2925655356 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9a98035050 con 0x55733f00e5d0 2024-09-19T08:12:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9ab786e740 10 client.17622 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9ab786e740 1 -- 172.21.15.98:0/2925655356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55733f093630 con 0x55733f00e5d0 2024-09-19T08:12:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9a967fc640 1 -- 172.21.15.98:0/2925655356 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a980044d0 con 0x55733f00e5d0 2024-09-19T08:12:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9a967fc640 1 -- 172.21.15.98:0/2925655356 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9a9803a040 con 0x55733f00e5d0 2024-09-19T08:12:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9ab786e740 10 client.17622 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9ab786e740 10 client.17622 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:12:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9ab786e740 20 client.17622 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9ab786e740 1 --2- 172.21.15.98:0/2925655356 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55733f028ef0 0x55733f095d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9ab786e740 4 client.17622 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:12:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.698+0000 7f9ab786e740 1 -- 172.21.15.98:0/2925655356 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55733ef7bd70 con 0x55733f028ef0 2024-09-19T08:12:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9aaeffd640 1 --2- 172.21.15.98:0/2925655356 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55733f028ef0 0x55733f095d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9a967fc640 10 client.17622 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:12:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9aaeffd640 1 --2- 172.21.15.98:0/2925655356 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55733f028ef0 0x55733f095d70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7b839bd40cca4e38 server_cookie=f173a40bf862946 in_seq=0 out_seq=0 2024-09-19T08:12:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9a967fc640 1 -- 172.21.15.98:0/2925655356 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55733ef7bd70 con 0x55733f028ef0 2024-09-19T08:12:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9a967fc640 10 client.17622 handle_command_reply: tid=0 2024-09-19T08:12:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9ab786e740 1 client.17622 shutdown 2024-09-19T08:12:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9ab786e740 1 -- 172.21.15.98:0/2925655356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 msgr2=0x55733f021a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9ab786e740 1 --2- 172.21.15.98:0/2925655356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f021a00 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f9a98005e10 tx=0x7f9a980314e0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9ab786e740 1 -- 172.21.15.98:0/2925655356 shutdown_connections 2024-09-19T08:12:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9ab786e740 1 --2- 172.21.15.98:0/2925655356 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55733f028ef0 0x55733f095d70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9ab786e740 1 --2- 172.21.15.98:0/2925655356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55733f019370 0x55733f013070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9ab786e740 1 --2- 172.21.15.98:0/2925655356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55733f00fb40 0x55733f012b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9ab786e740 1 --2- 172.21.15.98:0/2925655356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55733f00e5d0 0x55733f021a00 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9a967fc640 0 client.17622 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:12:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9ab786e740 1 -- 172.21.15.98:0/2925655356 >> 172.21.15.98:0/2925655356 conn(0x55733efd7310 msgr2=0x55733f017860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9ab786e740 1 -- 172.21.15.98:0/2925655356 shutdown_connections 2024-09-19T08:12:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.699+0000 7f9ab786e740 1 -- 172.21.15.98:0/2925655356 wait complete. 2024-09-19T08:12:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.700+0000 7f9ab786e740 20 client.17622 trim_cache size 0 max 16384 2024-09-19T08:12:20.719 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:12:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.700+0000 7f9ab5495640 1 -- 172.21.15.98:0/4228176589 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9a9c0a3430 msgr2=0x7f9a9c0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.700+0000 7f9ab5495640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9a9c0a3430 0x7f9a9c0a58f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80ccaa0 tx=0x7f9aa8004a40 comp rx=0 tx=0).stop 2024-09-19T08:12:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.700+0000 7f9ab5495640 1 -- 172.21.15.98:0/4228176589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ab010c2f0 msgr2=0x7f9ab019cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.700+0000 7f9ab5495640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ab010c2f0 0x7f9ab019cf40 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f9aa800b970 tx=0x7f9aa800be30 comp rx=0 tx=0).stop 2024-09-19T08:12:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.700+0000 7f9ab5495640 1 -- 172.21.15.98:0/4228176589 shutdown_connections 2024-09-19T08:12:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.700+0000 7f9ab5495640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f9a9c0a3430 0x7f9a9c0a58f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.700+0000 7f9ab5495640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ab010c2f0 0x7f9ab019cf40 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.700+0000 7f9ab5495640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9ab01037a0 0x7f9ab0198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.700+0000 7f9ab5495640 1 --2- 172.21.15.98:0/4228176589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ab0102df0 0x7f9ab0198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.700+0000 7f9ab5495640 1 -- 172.21.15.98:0/4228176589 >> 172.21.15.98:0/4228176589 conn(0x7f9ab00fc7d0 msgr2=0x7f9ab00fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.700+0000 7f9ab5495640 1 -- 172.21.15.98:0/4228176589 shutdown_connections 2024-09-19T08:12:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:20.701+0000 7f9ab5495640 1 -- 172.21.15.98:0/4228176589 wait complete. 2024-09-19T08:12:21.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:12:21.712 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:12:22.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:21 smithi162 ceph-mon[35260]: osdmap e87: 12 total, 12 up, 12 in 2024-09-19T08:12:22.152 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:21 smithi098 ceph-mon[31333]: osdmap e87: 12 total, 12 up, 12 in 2024-09-19T08:12:22.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:21 smithi139 ceph-mon[34874]: osdmap e87: 12 total, 12 up, 12 in 2024-09-19T08:12:22.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.232+0000 7fdabbd96640 1 -- 172.21.15.98:0/1975029896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab4106f10 msgr2=0x7fdab410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.232+0000 7fdabbd96640 1 --2- 172.21.15.98:0/1975029896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab4106f10 0x7fdab410ea90 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fdaac009ca0 tx=0x7fdaac02a8c0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.232+0000 7fdabbd96640 1 -- 172.21.15.98:0/1975029896 shutdown_connections 2024-09-19T08:12:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.232+0000 7fdabbd96640 1 --2- 172.21.15.98:0/1975029896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab4106f10 0x7fdab410ea90 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.232+0000 7fdabbd96640 1 --2- 172.21.15.98:0/1975029896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdab4102790 0x7fdab41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.232+0000 7fdabbd96640 1 --2- 172.21.15.98:0/1975029896 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdab4101de0 0x7fdab41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.232+0000 7fdabbd96640 1 -- 172.21.15.98:0/1975029896 >> 172.21.15.98:0/1975029896 conn(0x7fdab40fd8c0 msgr2=0x7fdab40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.232+0000 7fdabbd96640 1 -- 172.21.15.98:0/1975029896 shutdown_connections 2024-09-19T08:12:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.232+0000 7fdabbd96640 1 -- 172.21.15.98:0/1975029896 wait complete. 2024-09-19T08:12:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdabbd96640 1 Processor -- start 2024-09-19T08:12:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdabbd96640 1 -- start start 2024-09-19T08:12:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdabbd96640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdab4101de0 0x7fdab40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdabbd96640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdab4102790 0x7fdab4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdabbd96640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab4106f10 0x7fdab4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdabbd96640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdab4111f70 con 0x7fdab4106f10 2024-09-19T08:12:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdabbd96640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdab4111df0 con 0x7fdab4101de0 2024-09-19T08:12:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdabbd96640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdab41120f0 con 0x7fdab4102790 2024-09-19T08:12:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdaba333640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab4106f10 0x7fdab4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdaba333640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab4106f10 0x7fdab4074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54444/0 (socket says 172.21.15.98:54444) 2024-09-19T08:12:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdaba333640 1 -- 172.21.15.98:0/1240865741 learned_addr learned my addr 172.21.15.98:0/1240865741 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdab9b32640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdab4101de0 0x7fdab40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdab9331640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdab4102790 0x7fdab4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdaba333640 1 -- 172.21.15.98:0/1240865741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdab4102790 msgr2=0x7fdab4074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdaba333640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdab4102790 0x7fdab4074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.233+0000 7fdaba333640 1 -- 172.21.15.98:0/1240865741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdab4101de0 msgr2=0x7fdab40744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.234+0000 7fdaba333640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdab4101de0 0x7fdab40744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.234+0000 7fdaba333640 1 -- 172.21.15.98:0/1240865741 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab41a68d0 con 0x7fdab4106f10 2024-09-19T08:12:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.234+0000 7fdab9b32640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdab4101de0 0x7fdab40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:12:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.234+0000 7fdaba333640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab4106f10 0x7fdab4074f70 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fdaac031a20 tx=0x7fdaac031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.234+0000 7fdaaaffd640 1 -- 172.21.15.98:0/1240865741 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac003c30 con 0x7fdab4106f10 2024-09-19T08:12:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.234+0000 7fdaaaffd640 1 -- 172.21.15.98:0/1240865741 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaac003dd0 con 0x7fdab4106f10 2024-09-19T08:12:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.234+0000 7fdabbd96640 1 -- 172.21.15.98:0/1240865741 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab41a6b60 con 0x7fdab4106f10 2024-09-19T08:12:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.234+0000 7fdaaaffd640 1 -- 172.21.15.98:0/1240865741 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac02fcf0 con 0x7fdab4106f10 2024-09-19T08:12:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.234+0000 7fdabbd96640 1 -- 172.21.15.98:0/1240865741 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab41a70a0 con 0x7fdab4106f10 2024-09-19T08:12:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.236+0000 7fdaaaffd640 1 -- 172.21.15.98:0/1240865741 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fdaac004050 con 0x7fdab4106f10 2024-09-19T08:12:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.236+0000 7fdabe16f740 1 Processor -- start 2024-09-19T08:12:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdabe16f740 1 -- start start 2024-09-19T08:12:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdabe16f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e26b40 0x55bb90e25090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdabe16f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e255d0 0x55bb90e25a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdabe16f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e30370 0x55bb90e32760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdabe16f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bb90d8b2f0 con 0x55bb90e26b40 2024-09-19T08:12:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdabe16f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bb90d32cf0 con 0x55bb90e255d0 2024-09-19T08:12:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdabe16f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bb90d872b0 con 0x55bb90e30370 2024-09-19T08:12:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdab9331640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e26b40 0x55bb90e25090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdab9331640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e26b40 0x55bb90e25090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54460/0 (socket says 172.21.15.98:54460) 2024-09-19T08:12:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdaaaffd640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdaa00a33b0 0x7fdaa00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdab9331640 1 -- 172.21.15.98:0/2830923391 learned_addr learned my addr 172.21.15.98:0/2830923391 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdab9b32640 1 --2- 172.21.15.98:0/2830923391 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e255d0 0x55bb90e25a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdab9331640 1 --2- 172.21.15.98:0/2830923391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e30370 0x55bb90e32760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdaba333640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdaa00a33b0 0x7fdaa00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdaaaffd640 1 -- 172.21.15.98:0/1240865741 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7fdaac0f1020 con 0x7fdab4106f10 2024-09-19T08:12:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdab9331640 1 -- 172.21.15.98:0/2830923391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e30370 msgr2=0x55bb90e32760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdab9331640 1 --2- 172.21.15.98:0/2830923391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e30370 0x55bb90e32760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdab9331640 1 -- 172.21.15.98:0/2830923391 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e255d0 msgr2=0x55bb90e25a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdab9331640 1 --2- 172.21.15.98:0/2830923391 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e255d0 0x55bb90e25a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.237+0000 7fdab9331640 1 -- 172.21.15.98:0/2830923391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb90d173a0 con 0x55bb90e26b40 2024-09-19T08:12:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.238+0000 7fdab9331640 1 --2- 172.21.15.98:0/2830923391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e26b40 0x55bb90e25090 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x55bb90e26070 tx=0x7fdaa400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73825771966e1a09 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.238+0000 7fdaba333640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdaa00a33b0 0x7fdaa00a5870 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0bd430 tx=0x7fdaac08d0c0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.238+0000 7fda9ffff640 1 -- 172.21.15.98:0/2830923391 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4005220 con 0x55bb90e26b40 2024-09-19T08:12:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.238+0000 7fda9ffff640 1 -- 172.21.15.98:0/2830923391 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaa4004c80 con 0x55bb90e26b40 2024-09-19T08:12:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.238+0000 7fda9ffff640 1 -- 172.21.15.98:0/2830923391 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4010a00 con 0x55bb90e26b40 2024-09-19T08:12:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.238+0000 7fdabe16f740 1 -- 172.21.15.98:0/2830923391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e26b40 msgr2=0x55bb90e25090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:22.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.238+0000 7fdabe16f740 1 --2- 172.21.15.98:0/2830923391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e26b40 0x55bb90e25090 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x55bb90e26070 tx=0x7fdaa400c520 comp rx=0 tx=0).stop 2024-09-19T08:12:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.238+0000 7fdabe16f740 1 -- 172.21.15.98:0/2830923391 shutdown_connections 2024-09-19T08:12:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.238+0000 7fdabe16f740 1 --2- 172.21.15.98:0/2830923391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e30370 0x55bb90e32760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.238+0000 7fdabe16f740 1 --2- 172.21.15.98:0/2830923391 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e255d0 0x55bb90e25a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.238+0000 7fdabe16f740 1 --2- 172.21.15.98:0/2830923391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e26b40 0x55bb90e25090 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.238+0000 7fdabe16f740 1 -- 172.21.15.98:0/2830923391 >> 172.21.15.98:0/2830923391 conn(0x55bb90dee310 msgr2=0x55bb90e21c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.239+0000 7fdabe16f740 1 -- 172.21.15.98:0/2830923391 shutdown_connections 2024-09-19T08:12:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.239+0000 7fdabe16f740 1 -- 172.21.15.98:0/2830923391 wait complete. 2024-09-19T08:12:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.239+0000 7fdabe16f740 1 -- start start 2024-09-19T08:12:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.239+0000 7fdabe16f740 1 Processor -- start 2024-09-19T08:12:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.239+0000 7fdabe16f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e255d0 0x55bb90e2fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.239+0000 7fdabe16f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e26b40 0x55bb90e2cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.239+0000 7fdabe16f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e30370 0x55bb90e2d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.239+0000 7fdabe16f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bb90e33070 con 0x55bb90e26b40 2024-09-19T08:12:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.239+0000 7fdabe16f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bb90d46060 con 0x55bb90e255d0 2024-09-19T08:12:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.239+0000 7fdabe16f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bb90e331f0 con 0x55bb90e30370 2024-09-19T08:12:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.239+0000 7fdabe16f740 10 client.0 _ll_register_callbacks cb 0x55bb90a998c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fdab9b32640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e26b40 0x55bb90e2cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fdab9b32640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e26b40 0x55bb90e2cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54466/0 (socket says 172.21.15.98:54466) 2024-09-19T08:12:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fdab9b32640 1 -- 172.21.15.98:0/3552694918 learned_addr learned my addr 172.21.15.98:0/3552694918 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fdab9331640 1 --2- 172.21.15.98:0/3552694918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e30370 0x55bb90e2d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fdab9b32640 1 -- 172.21.15.98:0/3552694918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e30370 msgr2=0x55bb90e2d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fdab9b32640 1 --2- 172.21.15.98:0/3552694918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e30370 0x55bb90e2d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fdab9b32640 1 -- 172.21.15.98:0/3552694918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e255d0 msgr2=0x55bb90e2fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:22.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fdab9b32640 1 --2- 172.21.15.98:0/3552694918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e255d0 0x55bb90e2fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fdab9b32640 1 -- 172.21.15.98:0/3552694918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb90afc3c0 con 0x55bb90e26b40 2024-09-19T08:12:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fdab9331640 1 --2- 172.21.15.98:0/3552694918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e30370 0x55bb90e2d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fdab9b32640 1 --2- 172.21.15.98:0/3552694918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e26b40 0x55bb90e2cfb0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fdab000bfb0 tx=0x7fdab00128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6dc6460233f3c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fda9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fda9d7fa640 1 -- 172.21.15.98:0/3552694918 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab00035f0 con 0x55bb90e26b40 2024-09-19T08:12:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fda9d7fa640 1 -- 172.21.15.98:0/3552694918 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7fdab0003790 con 0x55bb90e26b40 2024-09-19T08:12:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fdabe16f740 1 -- 172.21.15.98:0/3552694918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdaa4004c80 con 0x55bb90e26b40 2024-09-19T08:12:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fda9d7fa640 1 -- 172.21.15.98:0/3552694918 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdab0003990 con 0x55bb90e26b40 2024-09-19T08:12:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fdabe16f740 10 client.17634 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fdabe16f740 1 -- 172.21.15.98:0/3552694918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bb90e3eb60 con 0x55bb90e26b40 2024-09-19T08:12:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.240+0000 7fda9d7fa640 1 -- 172.21.15.98:0/3552694918 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaa4004c80 con 0x55bb90e26b40 2024-09-19T08:12:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.241+0000 7fda9d7fa640 1 -- 172.21.15.98:0/3552694918 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fdab0019070 con 0x55bb90e26b40 2024-09-19T08:12:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.241+0000 7fdabe16f740 10 client.17634 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.241+0000 7fdabe16f740 10 client.17634 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:12:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.241+0000 7fdabe16f740 20 client.17634 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.241+0000 7fdabe16f740 1 --2- 172.21.15.98:0/3552694918 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55bb90e400c0 0x55bb90e424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.241+0000 7fdabe16f740 4 client.17634 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:12:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.241+0000 7fdabe16f740 1 -- 172.21.15.98:0/3552694918 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb90d98860 con 0x55bb90e400c0 2024-09-19T08:12:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.241+0000 7fdab9331640 1 --2- 172.21.15.98:0/3552694918 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55bb90e400c0 0x55bb90e424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.241+0000 7fda9d7fa640 10 client.17634 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:12:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.241+0000 7fdab9331640 1 --2- 172.21.15.98:0/3552694918 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55bb90e400c0 0x55bb90e424e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d7719338e52e26f5 server_cookie=f38a578630caaaae in_seq=0 out_seq=0 2024-09-19T08:12:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fda9d7fa640 1 -- 172.21.15.98:0/3552694918 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bb90d98860 con 0x55bb90e400c0 2024-09-19T08:12:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fda9d7fa640 10 client.17634 handle_command_reply: tid=0 2024-09-19T08:12:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fdabe16f740 1 client.17634 shutdown 2024-09-19T08:12:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fdabe16f740 1 -- 172.21.15.98:0/3552694918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e26b40 msgr2=0x55bb90e2cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fdabe16f740 1 --2- 172.21.15.98:0/3552694918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e26b40 0x55bb90e2cfb0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fdab000bfb0 tx=0x7fdab00128e0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fdabe16f740 1 -- 172.21.15.98:0/3552694918 shutdown_connections 2024-09-19T08:12:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fdabe16f740 1 --2- 172.21.15.98:0/3552694918 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55bb90e400c0 0x55bb90e424e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fdabe16f740 1 --2- 172.21.15.98:0/3552694918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e30370 0x55bb90e2d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fda9d7fa640 0 client.17634 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:12:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fdabe16f740 1 --2- 172.21.15.98:0/3552694918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e26b40 0x55bb90e2cfb0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fdabe16f740 1 --2- 172.21.15.98:0/3552694918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e255d0 0x55bb90e2fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fdabe16f740 1 -- 172.21.15.98:0/3552694918 >> 172.21.15.98:0/3552694918 conn(0x55bb90dee310 msgr2=0x55bb90e27930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fdabe16f740 1 -- 172.21.15.98:0/3552694918 shutdown_connections 2024-09-19T08:12:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fdabe16f740 1 -- 172.21.15.98:0/3552694918 wait complete. 2024-09-19T08:12:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.242+0000 7fdabe16f740 20 client.17634 trim_cache size 0 max 16384 2024-09-19T08:12:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.255+0000 7fdabe16f740 1 Processor -- start 2024-09-19T08:12:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.255+0000 7fdabe16f740 1 -- start start 2024-09-19T08:12:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.255+0000 7fdabe16f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e255d0 0x55bb90e3c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.255+0000 7fdabe16f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e26b40 0x55bb90e28120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.255+0000 7fdabe16f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e30370 0x55bb90e28660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.255+0000 7fdabe16f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdab41afab0 con 0x55bb90e30370 2024-09-19T08:12:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.255+0000 7fdabe16f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdab41b15f0 con 0x55bb90e26b40 2024-09-19T08:12:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.255+0000 7fdabe16f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdab41b0180 con 0x55bb90e255d0 2024-09-19T08:12:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.255+0000 7fdab9331640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e30370 0x55bb90e28660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.255+0000 7fdab9331640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e30370 0x55bb90e28660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54470/0 (socket says 172.21.15.98:54470) 2024-09-19T08:12:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.256+0000 7fdab9331640 1 -- 172.21.15.98:0/3955594701 learned_addr learned my addr 172.21.15.98:0/3955594701 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.256+0000 7fdab9b32640 1 --2- 172.21.15.98:0/3955594701 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e26b40 0x55bb90e28120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.256+0000 7fdab9331640 1 --2- 172.21.15.98:0/3955594701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e255d0 0x55bb90e3c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.256+0000 7fdab9331640 1 -- 172.21.15.98:0/3955594701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e255d0 msgr2=0x55bb90e3c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.256+0000 7fdab9331640 1 --2- 172.21.15.98:0/3955594701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e255d0 0x55bb90e3c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.256+0000 7fdab9331640 1 -- 172.21.15.98:0/3955594701 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e26b40 msgr2=0x55bb90e28120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.256+0000 7fdab9331640 1 --2- 172.21.15.98:0/3955594701 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e26b40 0x55bb90e28120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.256+0000 7fdab9331640 1 -- 172.21.15.98:0/3955594701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb90e20310 con 0x55bb90e30370 2024-09-19T08:12:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.256+0000 7fdab9331640 1 --2- 172.21.15.98:0/3955594701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e30370 0x55bb90e28660 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4005790 tx=0x7fdaa40056c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2be362f80b9c35c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.256+0000 7fda9f7fe640 1 -- 172.21.15.98:0/3955594701 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4004020 con 0x55bb90e30370 2024-09-19T08:12:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.256+0000 7fda9f7fe640 1 -- 172.21.15.98:0/3955594701 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaa40054e0 con 0x55bb90e30370 2024-09-19T08:12:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.256+0000 7fda9f7fe640 1 -- 172.21.15.98:0/3955594701 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4004c90 con 0x55bb90e30370 2024-09-19T08:12:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 -- 172.21.15.98:0/3955594701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e30370 msgr2=0x55bb90e28660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 --2- 172.21.15.98:0/3955594701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e30370 0x55bb90e28660 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4005790 tx=0x7fdaa40056c0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 -- 172.21.15.98:0/3955594701 shutdown_connections 2024-09-19T08:12:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 --2- 172.21.15.98:0/3955594701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e30370 0x55bb90e28660 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 --2- 172.21.15.98:0/3955594701 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e26b40 0x55bb90e28120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 --2- 172.21.15.98:0/3955594701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e255d0 0x55bb90e3c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 -- 172.21.15.98:0/3955594701 >> 172.21.15.98:0/3955594701 conn(0x55bb90dee310 msgr2=0x55bb90e2f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 -- 172.21.15.98:0/3955594701 shutdown_connections 2024-09-19T08:12:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 -- 172.21.15.98:0/3955594701 wait complete. 2024-09-19T08:12:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 -- start start 2024-09-19T08:12:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 Processor -- start 2024-09-19T08:12:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e255d0 0x55bb90e34230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e26b40 0x55bb90e34770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e30370 0x55bb90e34cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdab41b0f20 con 0x55bb90e30370 2024-09-19T08:12:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdab41b0850 con 0x55bb90e26b40 2024-09-19T08:12:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdab41adab0 con 0x55bb90e255d0 2024-09-19T08:12:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdabe16f740 10 client.0 _ll_register_callbacks cb 0x7fdab000b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.257+0000 7fdab9331640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e30370 0x55bb90e34cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fdab9331640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e30370 0x55bb90e34cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54486/0 (socket says 172.21.15.98:54486) 2024-09-19T08:12:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fdab9331640 1 -- 172.21.15.98:0/2647490579 learned_addr learned my addr 172.21.15.98:0/2647490579 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fdab9b32640 1 --2- 172.21.15.98:0/2647490579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e26b40 0x55bb90e34770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fdab9331640 1 --2- 172.21.15.98:0/2647490579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e255d0 0x55bb90e34230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fdab9331640 1 -- 172.21.15.98:0/2647490579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e255d0 msgr2=0x55bb90e34230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fdab9331640 1 --2- 172.21.15.98:0/2647490579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e255d0 0x55bb90e34230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fdab9331640 1 -- 172.21.15.98:0/2647490579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e26b40 msgr2=0x55bb90e34770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fdab9331640 1 --2- 172.21.15.98:0/2647490579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e26b40 0x55bb90e34770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fdab9331640 1 -- 172.21.15.98:0/2647490579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb90e21510 con 0x55bb90e30370 2024-09-19T08:12:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fdab9331640 1 --2- 172.21.15.98:0/2647490579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e30370 0x55bb90e34cb0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4012fd0 tx=0x7fdaa40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e528170ec166d2e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fda9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fda9d7fa640 1 -- 172.21.15.98:0/2647490579 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4004f50 con 0x55bb90e30370 2024-09-19T08:12:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fda9d7fa640 1 -- 172.21.15.98:0/2647490579 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7fdaa40050f0 con 0x55bb90e30370 2024-09-19T08:12:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fdabe16f740 1 -- 172.21.15.98:0/2647490579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdaa40054e0 con 0x55bb90e30370 2024-09-19T08:12:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fda9d7fa640 1 -- 172.21.15.98:0/2647490579 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdaa40052f0 con 0x55bb90e30370 2024-09-19T08:12:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fdabe16f740 10 client.17640 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fdabe16f740 1 -- 172.21.15.98:0/2647490579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bb90e22960 con 0x55bb90e30370 2024-09-19T08:12:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.258+0000 7fda9d7fa640 1 -- 172.21.15.98:0/2647490579 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaa40054e0 con 0x55bb90e30370 2024-09-19T08:12:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fda9d7fa640 1 -- 172.21.15.98:0/2647490579 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fdaa402d6f0 con 0x55bb90e30370 2024-09-19T08:12:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fdabe16f740 10 client.17640 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fdabe16f740 10 client.17640 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:12:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fdabe16f740 20 client.17640 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fdabe16f740 1 --2- 172.21.15.98:0/2647490579 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55bb90eab130 0x55bb90ead550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fdabe16f740 4 client.17640 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:12:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fdabe16f740 1 -- 172.21.15.98:0/2647490579 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb90d92d70 con 0x55bb90eab130 2024-09-19T08:12:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fdab9b32640 1 --2- 172.21.15.98:0/2647490579 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55bb90eab130 0x55bb90ead550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fda9d7fa640 10 client.17640 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:12:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fdab9b32640 1 --2- 172.21.15.98:0/2647490579 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55bb90eab130 0x55bb90ead550 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f0443180590ed93c server_cookie=bfd3a98bc2f4786d in_seq=0 out_seq=0 2024-09-19T08:12:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fda9d7fa640 1 -- 172.21.15.98:0/2647490579 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb90d92d70 con 0x55bb90eab130 2024-09-19T08:12:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fda9d7fa640 10 client.17640 handle_command_reply: tid=0 2024-09-19T08:12:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fdabe16f740 1 client.17640 shutdown 2024-09-19T08:12:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fdabe16f740 1 -- 172.21.15.98:0/2647490579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e30370 msgr2=0x55bb90e34cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.259+0000 7fdabe16f740 1 --2- 172.21.15.98:0/2647490579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e30370 0x55bb90e34cb0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4012fd0 tx=0x7fdaa40057c0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabe16f740 1 -- 172.21.15.98:0/2647490579 shutdown_connections 2024-09-19T08:12:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabe16f740 1 --2- 172.21.15.98:0/2647490579 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x55bb90eab130 0x55bb90ead550 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabe16f740 1 --2- 172.21.15.98:0/2647490579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb90e30370 0x55bb90e34cb0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabe16f740 1 --2- 172.21.15.98:0/2647490579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb90e26b40 0x55bb90e34770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fda9d7fa640 0 client.17640 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:12:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabe16f740 1 --2- 172.21.15.98:0/2647490579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb90e255d0 0x55bb90e34230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabe16f740 1 -- 172.21.15.98:0/2647490579 >> 172.21.15.98:0/2647490579 conn(0x55bb90dee310 msgr2=0x55bb90e204e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:22.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabe16f740 1 -- 172.21.15.98:0/2647490579 shutdown_connections 2024-09-19T08:12:22.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabe16f740 1 -- 172.21.15.98:0/2647490579 wait complete. 2024-09-19T08:12:22.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabe16f740 20 client.17640 trim_cache size 0 max 16384 2024-09-19T08:12:22.278 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:12:22.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabbd96640 1 -- 172.21.15.98:0/1240865741 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdaa00a33b0 msgr2=0x7fdaa00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:22.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabbd96640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdaa00a33b0 0x7fdaa00a5870 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0bd430 tx=0x7fdaac08d0c0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabbd96640 1 -- 172.21.15.98:0/1240865741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab4106f10 msgr2=0x7fdab4074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:22.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabbd96640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab4106f10 0x7fdab4074f70 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fdaac031a20 tx=0x7fdaac031a50 comp rx=0 tx=0).stop 2024-09-19T08:12:22.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabbd96640 1 -- 172.21.15.98:0/1240865741 shutdown_connections 2024-09-19T08:12:22.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabbd96640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fdaa00a33b0 0x7fdaa00a5870 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabbd96640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab4106f10 0x7fdab4074f70 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabbd96640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdab4102790 0x7fdab4074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabbd96640 1 --2- 172.21.15.98:0/1240865741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdab4101de0 0x7fdab40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:22.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabbd96640 1 -- 172.21.15.98:0/1240865741 >> 172.21.15.98:0/1240865741 conn(0x7fdab40fd8c0 msgr2=0x7fdab40feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:22.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.260+0000 7fdabbd96640 1 -- 172.21.15.98:0/1240865741 shutdown_connections 2024-09-19T08:12:22.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:22.261+0000 7fdabbd96640 1 -- 172.21.15.98:0/1240865741 wait complete. 2024-09-19T08:12:22.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:12:23.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:22 smithi162 ceph-mon[35260]: pgmap v289: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2024-09-19T08:12:23.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:22 smithi139 ceph-mon[34874]: pgmap v289: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2024-09-19T08:12:23.180 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:12:23.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:22 smithi098 ceph-mon[31333]: pgmap v289: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2024-09-19T08:12:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.674+0000 7ff9e487f640 1 -- 172.21.15.98:0/2170473874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e0100a80 msgr2=0x7ff9e0100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.674+0000 7ff9e487f640 1 --2- 172.21.15.98:0/2170473874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e0100a80 0x7ff9e0100ee0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8009930 tx=0x7ff9c802a420 comp rx=0 tx=0).stop 2024-09-19T08:12:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.674+0000 7ff9e487f640 1 -- 172.21.15.98:0/2170473874 shutdown_connections 2024-09-19T08:12:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.674+0000 7ff9e487f640 1 --2- 172.21.15.98:0/2170473874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9e0101420 0x7ff9e010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.674+0000 7ff9e487f640 1 --2- 172.21.15.98:0/2170473874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e0100a80 0x7ff9e0100ee0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.674+0000 7ff9e487f640 1 --2- 172.21.15.98:0/2170473874 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9e0106a80 0x7ff9e0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.674+0000 7ff9e487f640 1 -- 172.21.15.98:0/2170473874 >> 172.21.15.98:0/2170473874 conn(0x7ff9e00fc7a0 msgr2=0x7ff9e00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.674+0000 7ff9e487f640 1 -- 172.21.15.98:0/2170473874 shutdown_connections 2024-09-19T08:12:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.674+0000 7ff9e487f640 1 -- 172.21.15.98:0/2170473874 wait complete. 2024-09-19T08:12:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.675+0000 7ff9e487f640 1 Processor -- start 2024-09-19T08:12:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.675+0000 7ff9e487f640 1 -- start start 2024-09-19T08:12:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.675+0000 7ff9e487f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9e0100a80 0x7ff9e0198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.675+0000 7ff9e487f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9e0101420 0x7ff9e0198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.675+0000 7ff9e487f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e0106a80 0x7ff9e019cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.675+0000 7ff9e487f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e0110e80 con 0x7ff9e0106a80 2024-09-19T08:12:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.675+0000 7ff9e487f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e0110d00 con 0x7ff9e0101420 2024-09-19T08:12:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.675+0000 7ff9e487f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e0111000 con 0x7ff9e0100a80 2024-09-19T08:12:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.675+0000 7ff9ded9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e0106a80 0x7ff9e019cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.675+0000 7ff9ded9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e0106a80 0x7ff9e019cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54496/0 (socket says 172.21.15.98:54496) 2024-09-19T08:12:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.675+0000 7ff9ded9d640 1 -- 172.21.15.98:0/1859353742 learned_addr learned my addr 172.21.15.98:0/1859353742 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.675+0000 7ff9de59c640 1 --2- 172.21.15.98:0/1859353742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9e0100a80 0x7ff9e0198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.675+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/1859353742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9e0101420 0x7ff9e0198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.676+0000 7ff9ded9d640 1 -- 172.21.15.98:0/1859353742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9e0100a80 msgr2=0x7ff9e0198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.676+0000 7ff9ded9d640 1 --2- 172.21.15.98:0/1859353742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9e0100a80 0x7ff9e0198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.676+0000 7ff9ded9d640 1 -- 172.21.15.98:0/1859353742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9e0101420 msgr2=0x7ff9e0198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.676+0000 7ff9ded9d640 1 --2- 172.21.15.98:0/1859353742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9e0101420 0x7ff9e0198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.676+0000 7ff9ded9d640 1 -- 172.21.15.98:0/1859353742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e019d3f0 con 0x7ff9e0106a80 2024-09-19T08:12:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.676+0000 7ff9ded9d640 1 --2- 172.21.15.98:0/1859353742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e0106a80 0x7ff9e019cd10 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800cce0 tx=0x7ff9d800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.676+0000 7ff9d77fe640 1 -- 172.21.15.98:0/1859353742 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d800f450 con 0x7ff9e0106a80 2024-09-19T08:12:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.676+0000 7ff9d77fe640 1 -- 172.21.15.98:0/1859353742 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9d800fad0 con 0x7ff9e0106a80 2024-09-19T08:12:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.676+0000 7ff9e487f640 1 -- 172.21.15.98:0/1859353742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9e019d6e0 con 0x7ff9e0106a80 2024-09-19T08:12:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.676+0000 7ff9d77fe640 1 -- 172.21.15.98:0/1859353742 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8018c10 con 0x7ff9e0106a80 2024-09-19T08:12:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.676+0000 7ff9e487f640 1 -- 172.21.15.98:0/1859353742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9e01a4fc0 con 0x7ff9e0106a80 2024-09-19T08:12:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.678+0000 7ff9d77fe640 1 -- 172.21.15.98:0/1859353742 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7ff9d800f5f0 con 0x7ff9e0106a80 2024-09-19T08:12:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.679+0000 7ff9e6c58740 1 Processor -- start 2024-09-19T08:12:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.679+0000 7ff9e6c58740 1 -- start start 2024-09-19T08:12:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.679+0000 7ff9e6c58740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 0x55e4f99a6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.679+0000 7ff9e6c58740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99a65d0 0x55e4f99a6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.679+0000 7ff9e6c58740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99b1370 0x55e4f99b3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.679+0000 7ff9e6c58740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e4f990c2f0 con 0x55e4f99b1370 2024-09-19T08:12:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.679+0000 7ff9e6c58740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e4f98b3cf0 con 0x55e4f99a7b40 2024-09-19T08:12:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.679+0000 7ff9e6c58740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e4f99082b0 con 0x55e4f99a65d0 2024-09-19T08:12:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.679+0000 7ff9ddd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99b1370 0x55e4f99b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.679+0000 7ff9d77fe640 1 --2- 172.21.15.98:0/1859353742 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff9cc0a3430 0x7ff9cc0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.679+0000 7ff9ddd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99b1370 0x55e4f99b3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54508/0 (socket says 172.21.15.98:54508) 2024-09-19T08:12:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.679+0000 7ff9ddd9b640 1 -- 172.21.15.98:0/1739509220 learned_addr learned my addr 172.21.15.98:0/1739509220 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.679+0000 7ff9ded9d640 1 --2- 172.21.15.98:0/1859353742 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff9cc0a3430 0x7ff9cc0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.679+0000 7ff9de59c640 1 --2- 172.21.15.98:0/1739509220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99a65d0 0x55e4f99a6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.680+0000 7ff9d77fe640 1 -- 172.21.15.98:0/1859353742 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7ff9d80d03d0 con 0x7ff9e0106a80 2024-09-19T08:12:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.680+0000 7ff9ddd9b640 1 -- 172.21.15.98:0/1739509220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99a65d0 msgr2=0x55e4f99a6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.680+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/1739509220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99a65d0 0x55e4f99a6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.680+0000 7ff9ddd9b640 1 -- 172.21.15.98:0/1739509220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 msgr2=0x55e4f99a6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.680+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/1739509220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 0x55e4f99a6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.680+0000 7ff9ddd9b640 1 -- 172.21.15.98:0/1739509220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4f98983a0 con 0x55e4f99b1370 2024-09-19T08:12:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.680+0000 7ff9ded9d640 1 --2- 172.21.15.98:0/1859353742 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff9cc0a3430 0x7ff9cc0a58f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff9d809c350 tx=0x7ff9d80be040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.680+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/1739509220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99b1370 0x55e4f99b3760 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8009930 tx=0x7ff9c8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5eab1138e77e81c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.680+0000 7ff9d4ff9640 1 -- 172.21.15.98:0/1739509220 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8041070 con 0x55e4f99b1370 2024-09-19T08:12:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.680+0000 7ff9d4ff9640 1 -- 172.21.15.98:0/1739509220 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9c8004030 con 0x55e4f99b1370 2024-09-19T08:12:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.680+0000 7ff9d4ff9640 1 -- 172.21.15.98:0/1739509220 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8035040 con 0x55e4f99b1370 2024-09-19T08:12:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.681+0000 7ff9e6c58740 1 -- 172.21.15.98:0/1739509220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99b1370 msgr2=0x55e4f99b3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.681+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/1739509220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99b1370 0x55e4f99b3760 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8009930 tx=0x7ff9c8005e80 comp rx=0 tx=0).stop 2024-09-19T08:12:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.681+0000 7ff9e6c58740 1 -- 172.21.15.98:0/1739509220 shutdown_connections 2024-09-19T08:12:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.681+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/1739509220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99b1370 0x55e4f99b3760 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.681+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/1739509220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99a65d0 0x55e4f99a6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.681+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/1739509220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 0x55e4f99a6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.681+0000 7ff9e6c58740 1 -- 172.21.15.98:0/1739509220 >> 172.21.15.98:0/1739509220 conn(0x55e4f996f310 msgr2=0x55e4f99a2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.681+0000 7ff9e6c58740 1 -- 172.21.15.98:0/1739509220 shutdown_connections 2024-09-19T08:12:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.681+0000 7ff9e6c58740 1 -- 172.21.15.98:0/1739509220 wait complete. 2024-09-19T08:12:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9e6c58740 1 -- start start 2024-09-19T08:12:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9e6c58740 1 Processor -- start 2024-09-19T08:12:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9e6c58740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a65d0 0x55e4f99b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9e6c58740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 0x55e4f99ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9e6c58740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99b1370 0x55e4f99ae610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9e6c58740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e4f99b4010 con 0x55e4f99a65d0 2024-09-19T08:12:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9e6c58740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e4f98c7060 con 0x55e4f99a7b40 2024-09-19T08:12:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9e6c58740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e4f99b4190 con 0x55e4f99b1370 2024-09-19T08:12:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9e6c58740 10 client.0 _ll_register_callbacks cb 0x55e4f961a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9ddd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a65d0 0x55e4f99b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9ddd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a65d0 0x55e4f99b0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54524/0 (socket says 172.21.15.98:54524) 2024-09-19T08:12:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9ddd9b640 1 -- 172.21.15.98:0/655521756 learned_addr learned my addr 172.21.15.98:0/655521756 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/655521756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99b1370 0x55e4f99ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9de59c640 1 --2- 172.21.15.98:0/655521756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 0x55e4f99ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9ddd9b640 1 -- 172.21.15.98:0/655521756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99b1370 msgr2=0x55e4f99ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/655521756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99b1370 0x55e4f99ae610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9ddd9b640 1 -- 172.21.15.98:0/655521756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 msgr2=0x55e4f99ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/655521756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 0x55e4f99ae020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9ddd9b640 1 -- 172.21.15.98:0/655521756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4f967d3c0 con 0x55e4f99a65d0 2024-09-19T08:12:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.682+0000 7ff9de59c640 1 --2- 172.21.15.98:0/655521756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 0x55e4f99ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/655521756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a65d0 0x55e4f99b0d70 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8000c00 tx=0x7ff9c80037d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f174c4048c15d47 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9c5ffb640 1 -- 172.21.15.98:0/655521756 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8053060 con 0x55e4f99a65d0 2024-09-19T08:12:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9c5ffb640 1 -- 172.21.15.98:0/655521756 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7ff9c8003ab0 con 0x55e4f99a65d0 2024-09-19T08:12:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9e6c58740 1 -- 172.21.15.98:0/655521756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9c8004030 con 0x55e4f99a65d0 2024-09-19T08:12:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9c5ffb640 1 -- 172.21.15.98:0/655521756 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff9c8035040 con 0x55e4f99a65d0 2024-09-19T08:12:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9e6c58740 10 client.17652 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9e6c58740 1 -- 172.21.15.98:0/655521756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e4f99bfab0 con 0x55e4f99a65d0 2024-09-19T08:12:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9c5ffb640 1 -- 172.21.15.98:0/655521756 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9c8004030 con 0x55e4f99a65d0 2024-09-19T08:12:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9c5ffb640 1 -- 172.21.15.98:0/655521756 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff9c803a040 con 0x55e4f99a65d0 2024-09-19T08:12:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9e6c58740 10 client.17652 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9e6c58740 10 client.17652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:12:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9e6c58740 20 client.17652 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/655521756 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e4f99c0e00 0x55e4f99c31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9e6c58740 4 client.17652 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:12:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.683+0000 7ff9e6c58740 1 -- 172.21.15.98:0/655521756 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4f9919860 con 0x55e4f99c0e00 2024-09-19T08:12:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.684+0000 7ff9de59c640 1 --2- 172.21.15.98:0/655521756 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e4f99c0e00 0x55e4f99c31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.684+0000 7ff9de59c640 1 --2- 172.21.15.98:0/655521756 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e4f99c0e00 0x55e4f99c31f0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daa978d86dd728dd server_cookie=fccdddeaec153c61 in_seq=0 out_seq=0 2024-09-19T08:12:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.684+0000 7ff9c5ffb640 10 client.17652 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9c5ffb640 1 -- 172.21.15.98:0/655521756 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e4f9919860 con 0x55e4f99c0e00 2024-09-19T08:12:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9c5ffb640 10 client.17652 handle_command_reply: tid=0 2024-09-19T08:12:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9e6c58740 1 client.17652 shutdown 2024-09-19T08:12:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9e6c58740 1 -- 172.21.15.98:0/655521756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a65d0 msgr2=0x55e4f99b0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/655521756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a65d0 0x55e4f99b0d70 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8000c00 tx=0x7ff9c80037d0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9e6c58740 1 -- 172.21.15.98:0/655521756 shutdown_connections 2024-09-19T08:12:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/655521756 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e4f99c0e00 0x55e4f99c31f0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/655521756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99b1370 0x55e4f99ae610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/655521756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 0x55e4f99ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/655521756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a65d0 0x55e4f99b0d70 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9e6c58740 1 -- 172.21.15.98:0/655521756 >> 172.21.15.98:0/655521756 conn(0x55e4f996f310 msgr2=0x55e4f99a8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9c5ffb640 0 client.17652 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9e6c58740 1 -- 172.21.15.98:0/655521756 shutdown_connections 2024-09-19T08:12:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9e6c58740 1 -- 172.21.15.98:0/655521756 wait complete. 2024-09-19T08:12:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.685+0000 7ff9e6c58740 20 client.17652 trim_cache size 0 max 16384 2024-09-19T08:12:23.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9e6c58740 1 Processor -- start 2024-09-19T08:12:23.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9e6c58740 1 -- start start 2024-09-19T08:12:23.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9e6c58740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a65d0 0x55e4f99a9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9e6c58740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 0x55e4f99a9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9e6c58740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99b1370 0x55e4f99c8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9e6c58740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e01ad750 con 0x55e4f99a65d0 2024-09-19T08:12:23.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9e6c58740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e01af290 con 0x55e4f99a7b40 2024-09-19T08:12:23.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9e6c58740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e01ade20 con 0x55e4f99b1370 2024-09-19T08:12:23.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9ddd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a65d0 0x55e4f99a9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9ddd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a65d0 0x55e4f99a9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54532/0 (socket says 172.21.15.98:54532) 2024-09-19T08:12:23.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9ddd9b640 1 -- 172.21.15.98:0/2441738983 learned_addr learned my addr 172.21.15.98:0/2441738983 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:23.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/2441738983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99b1370 0x55e4f99c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9ddd9b640 1 -- 172.21.15.98:0/2441738983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99b1370 msgr2=0x55e4f99c8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:23.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9de59c640 1 --2- 172.21.15.98:0/2441738983 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 0x55e4f99a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/2441738983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99b1370 0x55e4f99c8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9ddd9b640 1 -- 172.21.15.98:0/2441738983 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 msgr2=0x55e4f99a9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:23.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/2441738983 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 0x55e4f99a9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9ddd9b640 1 -- 172.21.15.98:0/2441738983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4f99b0990 con 0x55e4f99a65d0 2024-09-19T08:12:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.698+0000 7ff9de59c640 1 --2- 172.21.15.98:0/2441738983 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 0x55e4f99a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.699+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/2441738983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a65d0 0x55e4f99a9010 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8046fb0 tx=0x7ff9c802a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12f998ef35a07cf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.699+0000 7ff9c7fff640 1 -- 172.21.15.98:0/2441738983 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8053020 con 0x55e4f99a65d0 2024-09-19T08:12:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.699+0000 7ff9c7fff640 1 -- 172.21.15.98:0/2441738983 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9c804b040 con 0x55e4f99a65d0 2024-09-19T08:12:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.699+0000 7ff9c7fff640 1 -- 172.21.15.98:0/2441738983 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c804a070 con 0x55e4f99a65d0 2024-09-19T08:12:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.699+0000 7ff9e6c58740 1 -- 172.21.15.98:0/2441738983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a65d0 msgr2=0x55e4f99a9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.699+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/2441738983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a65d0 0x55e4f99a9010 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8046fb0 tx=0x7ff9c802a420 comp rx=0 tx=0).stop 2024-09-19T08:12:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.699+0000 7ff9e6c58740 1 -- 172.21.15.98:0/2441738983 shutdown_connections 2024-09-19T08:12:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.699+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/2441738983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99b1370 0x55e4f99c8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.699+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/2441738983 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a7b40 0x55e4f99a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.699+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/2441738983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a65d0 0x55e4f99a9010 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.699+0000 7ff9e6c58740 1 -- 172.21.15.98:0/2441738983 >> 172.21.15.98:0/2441738983 conn(0x55e4f996f310 msgr2=0x55e4f99b00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.699+0000 7ff9e6c58740 1 -- 172.21.15.98:0/2441738983 shutdown_connections 2024-09-19T08:12:23.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.699+0000 7ff9e6c58740 1 -- 172.21.15.98:0/2441738983 wait complete. 2024-09-19T08:12:23.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9e6c58740 1 -- start start 2024-09-19T08:12:23.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9e6c58740 1 Processor -- start 2024-09-19T08:12:23.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9e6c58740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a65d0 0x55e4f99b9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9e6c58740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a7b40 0x55e4f99b9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9e6c58740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99b1370 0x55e4f99b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9e6c58740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e01aebc0 con 0x55e4f99a7b40 2024-09-19T08:12:23.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9e6c58740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e01ae4f0 con 0x55e4f99a65d0 2024-09-19T08:12:23.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9e6c58740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e01ab750 con 0x55e4f99b1370 2024-09-19T08:12:23.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9de59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a7b40 0x55e4f99b9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9e6c58740 10 client.0 _ll_register_callbacks cb 0x7ff9d000b690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:23.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9de59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a7b40 0x55e4f99b9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54546/0 (socket says 172.21.15.98:54546) 2024-09-19T08:12:23.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9de59c640 1 -- 172.21.15.98:0/1881630767 learned_addr learned my addr 172.21.15.98:0/1881630767 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:23.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9de59c640 1 -- 172.21.15.98:0/1881630767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99b1370 msgr2=0x55e4f99b63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:23.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9de59c640 1 --2- 172.21.15.98:0/1881630767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99b1370 0x55e4f99b63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/1881630767 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a65d0 0x55e4f99b9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.700+0000 7ff9de59c640 1 -- 172.21.15.98:0/1881630767 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a65d0 msgr2=0x55e4f99b9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9de59c640 1 --2- 172.21.15.98:0/1881630767 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a65d0 0x55e4f99b9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9de59c640 1 -- 172.21.15.98:0/1881630767 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4f99b6a80 con 0x55e4f99a7b40 2024-09-19T08:12:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/1881630767 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a65d0 0x55e4f99b9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9de59c640 1 --2- 172.21.15.98:0/1881630767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a7b40 0x55e4f99b9b40 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0000c00 tx=0x7ff9d000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44fa56bc8ac0d557 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9c5ffb640 1 -- 172.21.15.98:0/1881630767 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0005020 con 0x55e4f99a7b40 2024-09-19T08:12:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9c5ffb640 1 -- 172.21.15.98:0/1881630767 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7ff9d00051c0 con 0x55e4f99a7b40 2024-09-19T08:12:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9e6c58740 1 -- 172.21.15.98:0/1881630767 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9c804b040 con 0x55e4f99a7b40 2024-09-19T08:12:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9c5ffb640 1 -- 172.21.15.98:0/1881630767 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff9d002c050 con 0x55e4f99a7b40 2024-09-19T08:12:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9e6c58740 10 client.17658 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9e6c58740 1 -- 172.21.15.98:0/1881630767 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e4f9a2c180 con 0x55e4f99a7b40 2024-09-19T08:12:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9c5ffb640 1 -- 172.21.15.98:0/1881630767 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9c804b040 con 0x55e4f99a7b40 2024-09-19T08:12:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9c5ffb640 1 -- 172.21.15.98:0/1881630767 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff9d001d4b0 con 0x55e4f99a7b40 2024-09-19T08:12:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9e6c58740 10 client.17658 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9e6c58740 10 client.17658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:12:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9e6c58740 20 client.17658 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/1881630767 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e4f9a2c980 0x55e4f9a2eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9e6c58740 4 client.17658 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:12:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.701+0000 7ff9e6c58740 1 -- 172.21.15.98:0/1881630767 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e4f9913d70 con 0x55e4f9a2c980 2024-09-19T08:12:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.702+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/1881630767 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e4f9a2c980 0x55e4f9a2eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.702+0000 7ff9ddd9b640 1 --2- 172.21.15.98:0/1881630767 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e4f9a2c980 0x55e4f9a2eda0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7b73c725ee9e0e server_cookie=718f6103b218923d in_seq=0 out_seq=0 2024-09-19T08:12:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.702+0000 7ff9c5ffb640 10 client.17658 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9c5ffb640 1 -- 172.21.15.98:0/1881630767 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e4f9913d70 con 0x55e4f9a2c980 2024-09-19T08:12:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9c5ffb640 10 client.17658 handle_command_reply: tid=0 2024-09-19T08:12:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e6c58740 1 client.17658 shutdown 2024-09-19T08:12:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e6c58740 1 -- 172.21.15.98:0/1881630767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a7b40 msgr2=0x55e4f99b9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:23.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/1881630767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a7b40 0x55e4f99b9b40 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0000c00 tx=0x7ff9d000bd40 comp rx=0 tx=0).stop 2024-09-19T08:12:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e6c58740 1 -- 172.21.15.98:0/1881630767 shutdown_connections 2024-09-19T08:12:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/1881630767 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e4f9a2c980 0x55e4f9a2eda0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/1881630767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e4f99b1370 0x55e4f99b63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/1881630767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e4f99a7b40 0x55e4f99b9b40 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9c5ffb640 0 client.17658 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e6c58740 1 --2- 172.21.15.98:0/1881630767 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e4f99a65d0 0x55e4f99b9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e6c58740 1 -- 172.21.15.98:0/1881630767 >> 172.21.15.98:0/1881630767 conn(0x55e4f996f310 msgr2=0x55e4f99a1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e6c58740 1 -- 172.21.15.98:0/1881630767 shutdown_connections 2024-09-19T08:12:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e6c58740 1 -- 172.21.15.98:0/1881630767 wait complete. 2024-09-19T08:12:23.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e6c58740 20 client.17658 trim_cache size 0 max 16384 2024-09-19T08:12:23.709 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:12:23.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e487f640 1 -- 172.21.15.98:0/1859353742 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff9cc0a3430 msgr2=0x7ff9cc0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:23.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e487f640 1 --2- 172.21.15.98:0/1859353742 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff9cc0a3430 0x7ff9cc0a58f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff9d809c350 tx=0x7ff9d80be040 comp rx=0 tx=0).stop 2024-09-19T08:12:23.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e487f640 1 -- 172.21.15.98:0/1859353742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e0106a80 msgr2=0x7ff9e019cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:23.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e487f640 1 --2- 172.21.15.98:0/1859353742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e0106a80 0x7ff9e019cd10 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800cce0 tx=0x7ff9d800a620 comp rx=0 tx=0).stop 2024-09-19T08:12:23.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e487f640 1 -- 172.21.15.98:0/1859353742 shutdown_connections 2024-09-19T08:12:23.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e487f640 1 --2- 172.21.15.98:0/1859353742 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ff9cc0a3430 0x7ff9cc0a58f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e487f640 1 --2- 172.21.15.98:0/1859353742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e0106a80 0x7ff9e019cd10 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e487f640 1 --2- 172.21.15.98:0/1859353742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9e0101420 0x7ff9e0198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e487f640 1 --2- 172.21.15.98:0/1859353742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9e0100a80 0x7ff9e0198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:23.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.703+0000 7ff9e487f640 1 -- 172.21.15.98:0/1859353742 >> 172.21.15.98:0/1859353742 conn(0x7ff9e00fc7a0 msgr2=0x7ff9e006aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:23.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.704+0000 7ff9e487f640 1 -- 172.21.15.98:0/1859353742 shutdown_connections 2024-09-19T08:12:23.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:23.704+0000 7ff9e487f640 1 -- 172.21.15.98:0/1859353742 wait complete. 2024-09-19T08:12:25.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:24 smithi162 ceph-mon[35260]: pgmap v290: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:12:25.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:24 smithi139 ceph-mon[34874]: pgmap v290: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:12:25.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:24 smithi098 ceph-mon[31333]: pgmap v290: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:12:25.572 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:12:25.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:12:25.828 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:12:26.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.312+0000 7f587d1a2640 1 -- 172.21.15.98:0/1161088179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5878105d90 msgr2=0x7f5878106220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:26.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.312+0000 7f587d1a2640 1 --2- 172.21.15.98:0/1161088179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5878105d90 0x7f5878106220 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f5870009ca0 tx=0x7f587002a940 comp rx=0 tx=0).stop 2024-09-19T08:12:26.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.312+0000 7f587d1a2640 1 -- 172.21.15.98:0/1161088179 shutdown_connections 2024-09-19T08:12:26.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.312+0000 7f587d1a2640 1 --2- 172.21.15.98:0/1161088179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5878105d90 0x7f5878106220 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.312+0000 7f587d1a2640 1 --2- 172.21.15.98:0/1161088179 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58781012a0 0x7f5878105770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.312+0000 7f587d1a2640 1 --2- 172.21.15.98:0/1161088179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5878100980 0x7f5878100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.312+0000 7f587d1a2640 1 -- 172.21.15.98:0/1161088179 >> 172.21.15.98:0/1161088179 conn(0x7f58780fc610 msgr2=0x7f58780fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:26.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.312+0000 7f587d1a2640 1 -- 172.21.15.98:0/1161088179 shutdown_connections 2024-09-19T08:12:26.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.312+0000 7f587d1a2640 1 -- 172.21.15.98:0/1161088179 wait complete. 2024-09-19T08:12:26.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.313+0000 7f587d1a2640 1 Processor -- start 2024-09-19T08:12:26.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.313+0000 7f587d1a2640 1 -- start start 2024-09-19T08:12:26.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.313+0000 7f587d1a2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5878100980 0x7f58781a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.313+0000 7f587d1a2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58781012a0 0x7f58781a4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.313+0000 7f587d1a2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5878105d90 0x7f587819e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.313+0000 7f587d1a2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5878118410 con 0x7f58781012a0 2024-09-19T08:12:26.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.313+0000 7f587d1a2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5878118290 con 0x7f5878100980 2024-09-19T08:12:26.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.313+0000 7f587d1a2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5878118590 con 0x7f5878105d90 2024-09-19T08:12:26.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.313+0000 7f586ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58781012a0 0x7f58781a4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.313+0000 7f586ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58781012a0 0x7f58781a4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38026/0 (socket says 172.21.15.98:38026) 2024-09-19T08:12:26.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.313+0000 7f586ffff640 1 -- 172.21.15.98:0/1743998970 learned_addr learned my addr 172.21.15.98:0/1743998970 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:26.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.313+0000 7f5877fff640 1 --2- 172.21.15.98:0/1743998970 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5878100980 0x7f58781a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.313+0000 7f587c9a1640 1 --2- 172.21.15.98:0/1743998970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5878105d90 0x7f587819e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.313+0000 7f586ffff640 1 -- 172.21.15.98:0/1743998970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5878105d90 msgr2=0x7f587819e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:26.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.314+0000 7f586ffff640 1 --2- 172.21.15.98:0/1743998970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5878105d90 0x7f587819e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.314+0000 7f586ffff640 1 -- 172.21.15.98:0/1743998970 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5878100980 msgr2=0x7f58781a4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:26.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.314+0000 7f586ffff640 1 --2- 172.21.15.98:0/1743998970 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5878100980 0x7f58781a4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.314+0000 7f586ffff640 1 -- 172.21.15.98:0/1743998970 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f587819efb0 con 0x7f58781012a0 2024-09-19T08:12:26.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.314+0000 7f586ffff640 1 --2- 172.21.15.98:0/1743998970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58781012a0 0x7f58781a4bb0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f586000b3f0 tx=0x7f586000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:26.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.314+0000 7f5875ffb640 1 -- 172.21.15.98:0/1743998970 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5860015020 con 0x7f58781012a0 2024-09-19T08:12:26.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.314+0000 7f587d1a2640 1 -- 172.21.15.98:0/1743998970 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f587819f2a0 con 0x7f58781012a0 2024-09-19T08:12:26.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.314+0000 7f5875ffb640 1 -- 172.21.15.98:0/1743998970 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5860011d10 con 0x7f58781012a0 2024-09-19T08:12:26.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.314+0000 7f5875ffb640 1 -- 172.21.15.98:0/1743998970 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586000f2e0 con 0x7f58781012a0 2024-09-19T08:12:26.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.314+0000 7f587d1a2640 1 -- 172.21.15.98:0/1743998970 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58781ab5f0 con 0x7f58781012a0 2024-09-19T08:12:26.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.316+0000 7f58807dd740 1 Processor -- start 2024-09-19T08:12:26.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.316+0000 7f5875ffb640 1 -- 172.21.15.98:0/1743998970 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f5860018020 con 0x7f58781012a0 2024-09-19T08:12:26.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.316+0000 7f58807dd740 1 -- start start 2024-09-19T08:12:26.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f58807dd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f68ef150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f58807dd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68ef690 0x5557f68f9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f58807dd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68f9f50 0x5557f68fc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f58807dd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5557f6854280 con 0x5557f68eed70 2024-09-19T08:12:26.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f58807dd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5557f67fbb10 con 0x5557f68ef690 2024-09-19T08:12:26.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f58807dd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5557f6850240 con 0x5557f68f9f50 2024-09-19T08:12:26.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f587c9a1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f68ef150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f587c9a1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f68ef150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38034/0 (socket says 172.21.15.98:38034) 2024-09-19T08:12:26.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f587c9a1640 1 -- 172.21.15.98:0/3908650662 learned_addr learned my addr 172.21.15.98:0/3908650662 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:26.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f5877fff640 1 --2- 172.21.15.98:0/3908650662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68ef690 0x5557f68f9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f586ffff640 1 --2- 172.21.15.98:0/3908650662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68f9f50 0x5557f68fc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f587c9a1640 1 -- 172.21.15.98:0/3908650662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68f9f50 msgr2=0x5557f68fc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:26.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f587c9a1640 1 --2- 172.21.15.98:0/3908650662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68f9f50 0x5557f68fc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f587c9a1640 1 -- 172.21.15.98:0/3908650662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68ef690 msgr2=0x5557f68f9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:26.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f587c9a1640 1 --2- 172.21.15.98:0/3908650662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68ef690 0x5557f68f9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f587c9a1640 1 -- 172.21.15.98:0/3908650662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557f68081f0 con 0x5557f68eed70 2024-09-19T08:12:26.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f586ffff640 1 --2- 172.21.15.98:0/3908650662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68f9f50 0x5557f68fc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:26.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f5875ffb640 1 --2- 172.21.15.98:0/1743998970 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58640a33b0 0x7f58640a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f587c9a1640 1 --2- 172.21.15.98:0/3908650662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f68ef150 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f58700098b0 tx=0x7f58700098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9673fad8d6e7bf3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:26.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f5877fff640 1 --2- 172.21.15.98:0/1743998970 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58640a33b0 0x7f58640a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.317+0000 7f5875ffb640 1 -- 172.21.15.98:0/1743998970 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f58600cf3c0 con 0x7f58781012a0 2024-09-19T08:12:26.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.318+0000 7f586e7fc640 1 -- 172.21.15.98:0/3908650662 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870034040 con 0x5557f68eed70 2024-09-19T08:12:26.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.318+0000 7f586e7fc640 1 -- 172.21.15.98:0/3908650662 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f587002f4d0 con 0x5557f68eed70 2024-09-19T08:12:26.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.318+0000 7f586e7fc640 1 -- 172.21.15.98:0/3908650662 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58700308a0 con 0x5557f68eed70 2024-09-19T08:12:26.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.318+0000 7f58807dd740 1 -- 172.21.15.98:0/3908650662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 msgr2=0x5557f68ef150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:26.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.318+0000 7f5877fff640 1 --2- 172.21.15.98:0/1743998970 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58640a33b0 0x7f58640a5870 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x5557f68ec160 tx=0x7f586800b390 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:26.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.318+0000 7f58807dd740 1 --2- 172.21.15.98:0/3908650662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f68ef150 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f58700098b0 tx=0x7f58700098e0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.318+0000 7f58807dd740 1 -- 172.21.15.98:0/3908650662 shutdown_connections 2024-09-19T08:12:26.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.318+0000 7f58807dd740 1 --2- 172.21.15.98:0/3908650662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68f9f50 0x5557f68fc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.318+0000 7f58807dd740 1 --2- 172.21.15.98:0/3908650662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68ef690 0x5557f68f9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.318+0000 7f58807dd740 1 --2- 172.21.15.98:0/3908650662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f68ef150 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.318+0000 7f58807dd740 1 -- 172.21.15.98:0/3908650662 >> 172.21.15.98:0/3908650662 conn(0x5557f68eb300 msgr2=0x5557f68eb6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:26.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.318+0000 7f58807dd740 1 -- 172.21.15.98:0/3908650662 shutdown_connections 2024-09-19T08:12:26.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.318+0000 7f58807dd740 1 -- 172.21.15.98:0/3908650662 wait complete. 2024-09-19T08:12:26.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.319+0000 7f58807dd740 1 -- start start 2024-09-19T08:12:26.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.319+0000 7f58807dd740 1 Processor -- start 2024-09-19T08:12:26.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.319+0000 7f58807dd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f68f99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.319+0000 7f58807dd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68ef690 0x5557f68f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.319+0000 7f58807dd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f68f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.319+0000 7f58807dd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5557f68fcb20 con 0x5557f68eed70 2024-09-19T08:12:26.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.319+0000 7f58807dd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5557f680ef10 con 0x5557f68f9f50 2024-09-19T08:12:26.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.319+0000 7f58807dd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5557f68fcca0 con 0x5557f68ef690 2024-09-19T08:12:26.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.319+0000 7f58807dd740 10 client.0 _ll_register_callbacks cb 0x5557f67d7a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:26.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.319+0000 7f5877fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f68f99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.319+0000 7f5877fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f68f99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38042/0 (socket says 172.21.15.98:38042) 2024-09-19T08:12:26.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.319+0000 7f5877fff640 1 -- 172.21.15.98:0/3262248137 learned_addr learned my addr 172.21.15.98:0/3262248137 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:26.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f587c9a1640 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f68f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f586ffff640 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68ef690 0x5557f68f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f5877fff640 1 -- 172.21.15.98:0/3262248137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68ef690 msgr2=0x5557f68f6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:26.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f5877fff640 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68ef690 0x5557f68f6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f5877fff640 1 -- 172.21.15.98:0/3262248137 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 msgr2=0x5557f68f74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:26.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f5877fff640 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f68f74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f5877fff640 1 -- 172.21.15.98:0/3262248137 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557f65c56a0 con 0x5557f68eed70 2024-09-19T08:12:26.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f586ffff640 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68ef690 0x5557f68f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:26.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f587c9a1640 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f68f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:12:26.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f5877fff640 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f68f99f0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f5868003e90 tx=0x7f58680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2629f497004aa318 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:26.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f58577fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:26.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f58577fe640 1 -- 172.21.15.98:0/3262248137 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586801b660 con 0x5557f68eed70 2024-09-19T08:12:26.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f58577fe640 1 -- 172.21.15.98:0/3262248137 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f586801b800 con 0x5557f68eed70 2024-09-19T08:12:26.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f58807dd740 1 -- 172.21.15.98:0/3262248137 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f587002f4d0 con 0x5557f68eed70 2024-09-19T08:12:26.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f58577fe640 1 -- 172.21.15.98:0/3262248137 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f586801ba00 con 0x5557f68eed70 2024-09-19T08:12:26.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f58807dd740 10 client.17670 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:26.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f58807dd740 1 -- 172.21.15.98:0/3262248137 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5557f6908600 con 0x5557f68eed70 2024-09-19T08:12:26.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.320+0000 7f58577fe640 1 -- 172.21.15.98:0/3262248137 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f587002f4d0 con 0x5557f68eed70 2024-09-19T08:12:26.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.321+0000 7f58577fe640 1 -- 172.21.15.98:0/3262248137 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5868024070 con 0x5557f68eed70 2024-09-19T08:12:26.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.321+0000 7f58807dd740 10 client.17670 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:26.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.321+0000 7f58807dd740 10 client.17670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:12:26.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.321+0000 7f58807dd740 20 client.17670 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:26.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.321+0000 7f58807dd740 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5557f6909ac0 0x5557f690bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.321+0000 7f58807dd740 4 client.17670 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:12:26.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.321+0000 7f58807dd740 1 -- 172.21.15.98:0/3262248137 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557f68617f0 con 0x5557f6909ac0 2024-09-19T08:12:26.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.321+0000 7f586ffff640 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5557f6909ac0 0x5557f690bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.321+0000 7f586ffff640 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5557f6909ac0 0x5557f690bef0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8a5e5994475bc0f server_cookie=c8d1c32a63848844 in_seq=0 out_seq=0 2024-09-19T08:12:26.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.321+0000 7f58577fe640 10 client.17670 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:26.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58577fe640 1 -- 172.21.15.98:0/3262248137 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5557f68617f0 con 0x5557f6909ac0 2024-09-19T08:12:26.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58577fe640 10 client.17670 handle_command_reply: tid=0 2024-09-19T08:12:26.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58807dd740 1 client.17670 shutdown 2024-09-19T08:12:26.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58807dd740 1 -- 172.21.15.98:0/3262248137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 msgr2=0x5557f68f99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:26.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58807dd740 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f68f99f0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f5868003e90 tx=0x7f58680046c0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58807dd740 1 -- 172.21.15.98:0/3262248137 shutdown_connections 2024-09-19T08:12:26.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58807dd740 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5557f6909ac0 0x5557f690bef0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58807dd740 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f68f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58807dd740 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68ef690 0x5557f68f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58807dd740 1 --2- 172.21.15.98:0/3262248137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f68f99f0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58807dd740 1 -- 172.21.15.98:0/3262248137 >> 172.21.15.98:0/3262248137 conn(0x5557f68eb300 msgr2=0x5557f68f1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58577fe640 0 client.17670 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58807dd740 1 -- 172.21.15.98:0/3262248137 shutdown_connections 2024-09-19T08:12:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58807dd740 1 -- 172.21.15.98:0/3262248137 wait complete. 2024-09-19T08:12:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.322+0000 7f58807dd740 20 client.17670 trim_cache size 0 max 16384 2024-09-19T08:12:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.339+0000 7f58807dd740 1 Processor -- start 2024-09-19T08:12:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.339+0000 7f58807dd740 1 -- start start 2024-09-19T08:12:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.339+0000 7f58807dd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68eed70 0x5557f68f17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.339+0000 7f58807dd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68ef690 0x5557f68ea300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.339+0000 7f58807dd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f69117c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.339+0000 7f58807dd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f58781b3ce0 con 0x5557f68ef690 2024-09-19T08:12:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.339+0000 7f58807dd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f58781b5820 con 0x5557f68f9f50 2024-09-19T08:12:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.339+0000 7f58807dd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f58781b43b0 con 0x5557f68eed70 2024-09-19T08:12:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.339+0000 7f586ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68ef690 0x5557f68ea300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.339+0000 7f586ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68ef690 0x5557f68ea300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38052/0 (socket says 172.21.15.98:38052) 2024-09-19T08:12:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.339+0000 7f586ffff640 1 -- 172.21.15.98:0/2275382164 learned_addr learned my addr 172.21.15.98:0/2275382164 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.340+0000 7f5877fff640 1 --2- 172.21.15.98:0/2275382164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68eed70 0x5557f68f17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.340+0000 7f587c9a1640 1 --2- 172.21.15.98:0/2275382164 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f69117c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.340+0000 7f586ffff640 1 -- 172.21.15.98:0/2275382164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68eed70 msgr2=0x5557f68f17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.340+0000 7f586ffff640 1 --2- 172.21.15.98:0/2275382164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68eed70 0x5557f68f17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.340+0000 7f586ffff640 1 -- 172.21.15.98:0/2275382164 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 msgr2=0x5557f69117c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.340+0000 7f586ffff640 1 --2- 172.21.15.98:0/2275382164 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f69117c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.340+0000 7f586ffff640 1 -- 172.21.15.98:0/2275382164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557f68eb130 con 0x5557f68ef690 2024-09-19T08:12:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.340+0000 7f587c9a1640 1 --2- 172.21.15.98:0/2275382164 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f69117c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:12:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.340+0000 7f586ffff640 1 --2- 172.21.15.98:0/2275382164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68ef690 0x5557f68ea300 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f586009b4b0 tx=0x7f58600047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43506536990970e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.340+0000 7f586dffb640 1 -- 172.21.15.98:0/2275382164 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58600a4420 con 0x5557f68ef690 2024-09-19T08:12:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.340+0000 7f586dffb640 1 -- 172.21.15.98:0/2275382164 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5860099850 con 0x5557f68ef690 2024-09-19T08:12:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.340+0000 7f586dffb640 1 -- 172.21.15.98:0/2275382164 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5860097e20 con 0x5557f68ef690 2024-09-19T08:12:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.341+0000 7f58807dd740 1 -- 172.21.15.98:0/2275382164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68ef690 msgr2=0x5557f68ea300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.341+0000 7f58807dd740 1 --2- 172.21.15.98:0/2275382164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68ef690 0x5557f68ea300 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f586009b4b0 tx=0x7f58600047d0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.341+0000 7f58807dd740 1 -- 172.21.15.98:0/2275382164 shutdown_connections 2024-09-19T08:12:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.341+0000 7f58807dd740 1 --2- 172.21.15.98:0/2275382164 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f69117c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.341+0000 7f58807dd740 1 --2- 172.21.15.98:0/2275382164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68ef690 0x5557f68ea300 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.341+0000 7f58807dd740 1 --2- 172.21.15.98:0/2275382164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68eed70 0x5557f68f17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.341+0000 7f58807dd740 1 -- 172.21.15.98:0/2275382164 >> 172.21.15.98:0/2275382164 conn(0x5557f68eb300 msgr2=0x5557f68e9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.341+0000 7f58807dd740 1 -- 172.21.15.98:0/2275382164 shutdown_connections 2024-09-19T08:12:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.341+0000 7f58807dd740 1 -- 172.21.15.98:0/2275382164 wait complete. 2024-09-19T08:12:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.341+0000 7f58807dd740 1 -- start start 2024-09-19T08:12:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.341+0000 7f58807dd740 1 Processor -- start 2024-09-19T08:12:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f58807dd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f69020d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f58807dd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68ef690 0x5557f6902610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f58807dd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f68feec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f58807dd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f58781b5150 con 0x5557f68eed70 2024-09-19T08:12:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f58807dd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f58781b4a80 con 0x5557f68f9f50 2024-09-19T08:12:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f58807dd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f58781b1ce0 con 0x5557f68ef690 2024-09-19T08:12:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f58807dd740 10 client.0 _ll_register_callbacks cb 0x7f5868003b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f5877fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f69020d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f5877fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f69020d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38062/0 (socket says 172.21.15.98:38062) 2024-09-19T08:12:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f5877fff640 1 -- 172.21.15.98:0/1315161953 learned_addr learned my addr 172.21.15.98:0/1315161953 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f586ffff640 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68ef690 0x5557f6902610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f587c9a1640 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f68feec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f5877fff640 1 -- 172.21.15.98:0/1315161953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68ef690 msgr2=0x5557f6902610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f5877fff640 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68ef690 0x5557f6902610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f5877fff640 1 -- 172.21.15.98:0/1315161953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 msgr2=0x5557f68feec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f5877fff640 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f68feec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f5877fff640 1 -- 172.21.15.98:0/1315161953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557f68ff500 con 0x5557f68eed70 2024-09-19T08:12:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f586ffff640 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68ef690 0x5557f6902610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:12:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f587c9a1640 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f68feec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f5877fff640 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f69020d0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f58680105b0 tx=0x7f58680036c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47179dd9a29b3ce4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f58577fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f58577fe640 1 -- 172.21.15.98:0/1315161953 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5868005450 con 0x5557f68eed70 2024-09-19T08:12:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f58577fe640 1 -- 172.21.15.98:0/1315161953 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f58680055f0 con 0x5557f68eed70 2024-09-19T08:12:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f58807dd740 1 -- 172.21.15.98:0/1315161953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5860099850 con 0x5557f68eed70 2024-09-19T08:12:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.342+0000 7f58577fe640 1 -- 172.21.15.98:0/1315161953 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f58680107d0 con 0x5557f68eed70 2024-09-19T08:12:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.343+0000 7f58807dd740 10 client.17676 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.343+0000 7f58807dd740 1 -- 172.21.15.98:0/1315161953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5557f6975120 con 0x5557f68eed70 2024-09-19T08:12:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.343+0000 7f58577fe640 1 -- 172.21.15.98:0/1315161953 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5860099850 con 0x5557f68eed70 2024-09-19T08:12:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.343+0000 7f58577fe640 1 -- 172.21.15.98:0/1315161953 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5868035070 con 0x5557f68eed70 2024-09-19T08:12:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.343+0000 7f58807dd740 10 client.17676 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.343+0000 7f58807dd740 10 client.17676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:12:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.343+0000 7f58807dd740 20 client.17676 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.343+0000 7f58807dd740 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5557f6975830 0x5557f6977c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.343+0000 7f58807dd740 4 client.17676 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:12:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.343+0000 7f58807dd740 1 -- 172.21.15.98:0/1315161953 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5557f685bd00 con 0x5557f6975830 2024-09-19T08:12:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.343+0000 7f586ffff640 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5557f6975830 0x5557f6977c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.344+0000 7f586ffff640 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5557f6975830 0x5557f6977c70 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=116826a553e22fd9 server_cookie=57a45ab8eff044f9 in_seq=0 out_seq=0 2024-09-19T08:12:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.344+0000 7f58577fe640 10 client.17676 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58577fe640 1 -- 172.21.15.98:0/1315161953 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5557f685bd00 con 0x5557f6975830 2024-09-19T08:12:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58577fe640 10 client.17676 handle_command_reply: tid=0 2024-09-19T08:12:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58807dd740 1 client.17676 shutdown 2024-09-19T08:12:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58807dd740 1 -- 172.21.15.98:0/1315161953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 msgr2=0x5557f69020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58807dd740 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f69020d0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f58680105b0 tx=0x7f58680036c0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58807dd740 1 -- 172.21.15.98:0/1315161953 shutdown_connections 2024-09-19T08:12:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58807dd740 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x5557f6975830 0x5557f6977c70 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58807dd740 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5557f68f9f50 0x5557f68feec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58807dd740 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557f68ef690 0x5557f6902610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58807dd740 1 --2- 172.21.15.98:0/1315161953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f68eed70 0x5557f69020d0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58807dd740 1 -- 172.21.15.98:0/1315161953 >> 172.21.15.98:0/1315161953 conn(0x5557f68eb300 msgr2=0x5557f68ea410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58577fe640 0 client.17676 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58807dd740 1 -- 172.21.15.98:0/1315161953 shutdown_connections 2024-09-19T08:12:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58807dd740 1 -- 172.21.15.98:0/1315161953 wait complete. 2024-09-19T08:12:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.345+0000 7f58807dd740 20 client.17676 trim_cache size 0 max 16384 2024-09-19T08:12:26.352 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:12:26.352 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:12:26.352 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "08bc7f3d-2411-494d-ba2b-4d2aba099064", 2024-09-19T08:12:26.353 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:12:26.353 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:12:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.346+0000 7f587d1a2640 1 -- 172.21.15.98:0/1743998970 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58640a33b0 msgr2=0x7f58640a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.346+0000 7f587d1a2640 1 --2- 172.21.15.98:0/1743998970 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58640a33b0 0x7f58640a5870 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x5557f68ec160 tx=0x7f586800b390 comp rx=0 tx=0).stop 2024-09-19T08:12:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.346+0000 7f587d1a2640 1 -- 172.21.15.98:0/1743998970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58781012a0 msgr2=0x7f58781a4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.346+0000 7f587d1a2640 1 --2- 172.21.15.98:0/1743998970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58781012a0 0x7f58781a4bb0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f586000b3f0 tx=0x7f586000b8b0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.346+0000 7f587d1a2640 1 -- 172.21.15.98:0/1743998970 shutdown_connections 2024-09-19T08:12:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.346+0000 7f587d1a2640 1 --2- 172.21.15.98:0/1743998970 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58640a33b0 0x7f58640a5870 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.346+0000 7f587d1a2640 1 --2- 172.21.15.98:0/1743998970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5878105d90 0x7f587819e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.346+0000 7f587d1a2640 1 --2- 172.21.15.98:0/1743998970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58781012a0 0x7f58781a4bb0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.346+0000 7f587d1a2640 1 --2- 172.21.15.98:0/1743998970 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5878100980 0x7f58781a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.346+0000 7f587d1a2640 1 -- 172.21.15.98:0/1743998970 >> 172.21.15.98:0/1743998970 conn(0x7f58780fc610 msgr2=0x7f5878103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.346+0000 7f587d1a2640 1 -- 172.21.15.98:0/1743998970 shutdown_connections 2024-09-19T08:12:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:26.346+0000 7f587d1a2640 1 -- 172.21.15.98:0/1743998970 wait complete. 2024-09-19T08:12:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:26 smithi162 ceph-mon[35260]: pgmap v291: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:12:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:26 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:12:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:26 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:12:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:26 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:12:27.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:26 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:12:27.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:26 smithi098 ceph-mon[31333]: pgmap v291: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:12:27.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:26 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:12:27.132 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:26 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:12:27.132 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:26 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:12:27.132 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:26 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:12:27.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:12:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:26 smithi139 ceph-mon[34874]: pgmap v291: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:12:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:26 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:12:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:26 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:12:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:26 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:12:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:26 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:12:27.380 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:12:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.894+0000 7f66aff9d640 1 -- 172.21.15.98:0/4079212471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a8102ef0 msgr2=0x7f66a810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.894+0000 7f66aff9d640 1 --2- 172.21.15.98:0/4079212471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a8102ef0 0x7f66a810da40 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009ca0 tx=0x7f66a002a950 comp rx=0 tx=0).stop 2024-09-19T08:12:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.894+0000 7f66aff9d640 1 -- 172.21.15.98:0/4079212471 shutdown_connections 2024-09-19T08:12:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.894+0000 7f66aff9d640 1 --2- 172.21.15.98:0/4079212471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a8102ef0 0x7f66a810da40 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.894+0000 7f66aff9d640 1 --2- 172.21.15.98:0/4079212471 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a8102570 0x7f66a81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.894+0000 7f66aff9d640 1 --2- 172.21.15.98:0/4079212471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66a8100e20 0x7f66a8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.894+0000 7f66aff9d640 1 -- 172.21.15.98:0/4079212471 >> 172.21.15.98:0/4079212471 conn(0x7f66a80fc7d0 msgr2=0x7f66a80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.894+0000 7f66aff9d640 1 -- 172.21.15.98:0/4079212471 shutdown_connections 2024-09-19T08:12:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.894+0000 7f66aff9d640 1 -- 172.21.15.98:0/4079212471 wait complete. 2024-09-19T08:12:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.894+0000 7f66aff9d640 1 Processor -- start 2024-09-19T08:12:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.894+0000 7f66aff9d640 1 -- start start 2024-09-19T08:12:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.894+0000 7f66aff9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a8100e20 0x7f66a81985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.894+0000 7f66aff9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a8102570 0x7f66a8198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66aff9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66a8102ef0 0x7f66a819ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66aff9d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66a8112120 con 0x7f66a8102570 2024-09-19T08:12:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66aff9d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f66a8111fa0 con 0x7f66a8100e20 2024-09-19T08:12:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66aff9d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81122a0 con 0x7f66a8102ef0 2024-09-19T08:12:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66ad538640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a8102570 0x7f66a8198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66ad538640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a8102570 0x7f66a8198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38094/0 (socket says 172.21.15.98:38094) 2024-09-19T08:12:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66ad538640 1 -- 172.21.15.98:0/777963493 learned_addr learned my addr 172.21.15.98:0/777963493 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66ae53a640 1 --2- 172.21.15.98:0/777963493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66a8102ef0 0x7f66a819ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66add39640 1 --2- 172.21.15.98:0/777963493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a8100e20 0x7f66a81985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66ad538640 1 -- 172.21.15.98:0/777963493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66a8102ef0 msgr2=0x7f66a819ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66ad538640 1 --2- 172.21.15.98:0/777963493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66a8102ef0 0x7f66a819ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66ad538640 1 -- 172.21.15.98:0/777963493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a8100e20 msgr2=0x7f66a81985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66ad538640 1 --2- 172.21.15.98:0/777963493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a8100e20 0x7f66a81985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66ad538640 1 -- 172.21.15.98:0/777963493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a819d590 con 0x7f66a8102570 2024-09-19T08:12:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66ad538640 1 --2- 172.21.15.98:0/777963493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a8102570 0x7f66a8198b20 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f669800b4a0 tx=0x7f669800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f669effd640 1 -- 172.21.15.98:0/777963493 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698014070 con 0x7f66a8102570 2024-09-19T08:12:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f669effd640 1 -- 172.21.15.98:0/777963493 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6698010af0 con 0x7f66a8102570 2024-09-19T08:12:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f669effd640 1 -- 172.21.15.98:0/777963493 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698017d10 con 0x7f66a8102570 2024-09-19T08:12:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66aff9d640 1 -- 172.21.15.98:0/777963493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a819d880 con 0x7f66a8102570 2024-09-19T08:12:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.895+0000 7f66aff9d640 1 -- 172.21.15.98:0/777963493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a81a5160 con 0x7f66a8102570 2024-09-19T08:12:27.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.898+0000 7f669effd640 1 -- 172.21.15.98:0/777963493 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f6698017070 con 0x7f66a8102570 2024-09-19T08:12:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.898+0000 7f66b2376740 1 Processor -- start 2024-09-19T08:12:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.898+0000 7f66b2376740 1 -- start start 2024-09-19T08:12:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.898+0000 7f66b2376740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6d4a0 0x55dbeba6d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.898+0000 7f66b2376740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba6ddc0 0x55dbeba78140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.898+0000 7f66b2376740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba78680 0x55dbeba7aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.898+0000 7f66b2376740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dbeb9d32e0 con 0x55dbeba6d4a0 2024-09-19T08:12:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.898+0000 7f66b2376740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dbeb97ace0 con 0x55dbeba6ddc0 2024-09-19T08:12:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.898+0000 7f66b2376740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dbeb9cf2a0 con 0x55dbeba78680 2024-09-19T08:12:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.898+0000 7f66ae53a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6d4a0 0x55dbeba6d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.898+0000 7f66ae53a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6d4a0 0x55dbeba6d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38106/0 (socket says 172.21.15.98:38106) 2024-09-19T08:12:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.898+0000 7f66ae53a640 1 -- 172.21.15.98:0/4010172984 learned_addr learned my addr 172.21.15.98:0/4010172984 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.898+0000 7f669effd640 1 --2- 172.21.15.98:0/777963493 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f66940a33b0 0x7f66940a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.899+0000 7f66ad538640 1 --2- 172.21.15.98:0/4010172984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba78680 0x55dbeba7aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.899+0000 7f66add39640 1 --2- 172.21.15.98:0/4010172984 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba6ddc0 0x55dbeba78140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.899+0000 7f66ae53a640 1 -- 172.21.15.98:0/4010172984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba78680 msgr2=0x55dbeba7aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.899+0000 7f66ae53a640 1 --2- 172.21.15.98:0/4010172984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba78680 0x55dbeba7aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.899+0000 7f66ae53a640 1 -- 172.21.15.98:0/4010172984 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba6ddc0 msgr2=0x55dbeba78140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.899+0000 7f66ae53a640 1 --2- 172.21.15.98:0/4010172984 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba6ddc0 0x55dbeba78140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.899+0000 7f66ae53a640 1 -- 172.21.15.98:0/4010172984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbeb95f440 con 0x55dbeba6d4a0 2024-09-19T08:12:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.899+0000 7f669effd640 1 -- 172.21.15.98:0/777963493 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f66980cf090 con 0x7f66a8102570 2024-09-19T08:12:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.899+0000 7f66ae53a640 1 --2- 172.21.15.98:0/777963493 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f66940a33b0 0x7f66940a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.899+0000 7f66ae53a640 1 --2- 172.21.15.98:0/4010172984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6d4a0 0x55dbeba6d880 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f66a002f9b0 tx=0x7f66a002f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b4ecc54775d775 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.899+0000 7f6693fff640 1 -- 172.21.15.98:0/4010172984 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a0041030 con 0x55dbeba6d4a0 2024-09-19T08:12:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.900+0000 7f6693fff640 1 -- 172.21.15.98:0/4010172984 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a0003e50 con 0x55dbeba6d4a0 2024-09-19T08:12:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.900+0000 7f66ae53a640 1 --2- 172.21.15.98:0/777963493 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f66940a33b0 0x7f66940a5870 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f66a004a000 tx=0x7f66a00394d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.900+0000 7f6693fff640 1 -- 172.21.15.98:0/4010172984 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a0034040 con 0x55dbeba6d4a0 2024-09-19T08:12:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.900+0000 7f66b2376740 1 -- 172.21.15.98:0/4010172984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6d4a0 msgr2=0x55dbeba6d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.900+0000 7f66b2376740 1 --2- 172.21.15.98:0/4010172984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6d4a0 0x55dbeba6d880 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f66a002f9b0 tx=0x7f66a002f9e0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.900+0000 7f66b2376740 1 -- 172.21.15.98:0/4010172984 shutdown_connections 2024-09-19T08:12:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.900+0000 7f66b2376740 1 --2- 172.21.15.98:0/4010172984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba78680 0x55dbeba7aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.900+0000 7f66b2376740 1 --2- 172.21.15.98:0/4010172984 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba6ddc0 0x55dbeba78140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.900+0000 7f66b2376740 1 --2- 172.21.15.98:0/4010172984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6d4a0 0x55dbeba6d880 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.901+0000 7f66b2376740 1 -- 172.21.15.98:0/4010172984 >> 172.21.15.98:0/4010172984 conn(0x55dbeba69a30 msgr2=0x55dbeba69e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.901+0000 7f66b2376740 1 -- 172.21.15.98:0/4010172984 shutdown_connections 2024-09-19T08:12:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.901+0000 7f66b2376740 1 -- 172.21.15.98:0/4010172984 wait complete. 2024-09-19T08:12:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.901+0000 7f66b2376740 1 -- start start 2024-09-19T08:12:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.901+0000 7f66b2376740 1 Processor -- start 2024-09-19T08:12:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.901+0000 7f66b2376740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6d4a0 0x55dbeba78010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.901+0000 7f66b2376740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba6ddc0 0x55dbeba752c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.901+0000 7f66b2376740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba78680 0x55dbeba75840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66b2376740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dbeba7b250 con 0x55dbeba78680 2024-09-19T08:12:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66b2376740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dbeb98e050 con 0x55dbeba6ddc0 2024-09-19T08:12:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66b2376740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dbeba7b3d0 con 0x55dbeba6d4a0 2024-09-19T08:12:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66b2376740 10 client.0 _ll_register_callbacks cb 0x55dbeb9ce160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66ae53a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba78680 0x55dbeba75840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66ae53a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba78680 0x55dbeba75840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38108/0 (socket says 172.21.15.98:38108) 2024-09-19T08:12:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66ae53a640 1 -- 172.21.15.98:0/2886004508 learned_addr learned my addr 172.21.15.98:0/2886004508 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66add39640 1 --2- 172.21.15.98:0/2886004508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6d4a0 0x55dbeba78010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66ad538640 1 --2- 172.21.15.98:0/2886004508 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba6ddc0 0x55dbeba752c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66ae53a640 1 -- 172.21.15.98:0/2886004508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6d4a0 msgr2=0x55dbeba78010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66ae53a640 1 --2- 172.21.15.98:0/2886004508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6d4a0 0x55dbeba78010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66ae53a640 1 -- 172.21.15.98:0/2886004508 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba6ddc0 msgr2=0x55dbeba752c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66ae53a640 1 --2- 172.21.15.98:0/2886004508 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba6ddc0 0x55dbeba752c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66ae53a640 1 -- 172.21.15.98:0/2886004508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbeb7443c0 con 0x55dbeba78680 2024-09-19T08:12:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66ad538640 1 --2- 172.21.15.98:0/2886004508 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba6ddc0 0x55dbeba752c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66ae53a640 1 --2- 172.21.15.98:0/2886004508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba78680 0x55dbeba75840 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f66a002a990 tx=0x7f66a0043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fee38a3c1d8ffdf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.902+0000 7f66917fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.903+0000 7f66917fa640 1 -- 172.21.15.98:0/2886004508 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a005d060 con 0x55dbeba78680 2024-09-19T08:12:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.903+0000 7f66917fa640 1 -- 172.21.15.98:0/2886004508 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f66a002f520 con 0x55dbeba78680 2024-09-19T08:12:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.903+0000 7f66b2376740 1 -- 172.21.15.98:0/2886004508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a0003e50 con 0x55dbeba78680 2024-09-19T08:12:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.903+0000 7f66917fa640 1 -- 172.21.15.98:0/2886004508 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f66a0041040 con 0x55dbeba78680 2024-09-19T08:12:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.903+0000 7f66b2376740 10 client.17688 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.903+0000 7f66b2376740 1 -- 172.21.15.98:0/2886004508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dbeba86c40 con 0x55dbeba78680 2024-09-19T08:12:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.903+0000 7f66917fa640 1 -- 172.21.15.98:0/2886004508 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a0003e50 con 0x55dbeba78680 2024-09-19T08:12:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.903+0000 7f66917fa640 1 -- 172.21.15.98:0/2886004508 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f66a0061580 con 0x55dbeba78680 2024-09-19T08:12:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.903+0000 7f66b2376740 10 client.17688 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.903+0000 7f66b2376740 10 client.17688 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:12:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.903+0000 7f66b2376740 20 client.17688 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.903+0000 7f66b2376740 1 --2- 172.21.15.98:0/2886004508 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55dbeba87f90 0x55dbeba8a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.903+0000 7f66b2376740 4 client.17688 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:12:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.903+0000 7f66b2376740 1 -- 172.21.15.98:0/2886004508 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbeb9e0850 con 0x55dbeba87f90 2024-09-19T08:12:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.904+0000 7f66add39640 1 --2- 172.21.15.98:0/2886004508 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55dbeba87f90 0x55dbeba8a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.904+0000 7f66917fa640 10 client.17688 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.904+0000 7f66add39640 1 --2- 172.21.15.98:0/2886004508 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55dbeba87f90 0x55dbeba8a380 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af3c8ad2228c57a4 server_cookie=752c93df2e8fdd8b in_seq=0 out_seq=0 2024-09-19T08:12:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.905+0000 7f66917fa640 1 -- 172.21.15.98:0/2886004508 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dbeb9e0850 con 0x55dbeba87f90 2024-09-19T08:12:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.905+0000 7f66917fa640 10 client.17688 handle_command_reply: tid=0 2024-09-19T08:12:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.905+0000 7f66b2376740 1 client.17688 shutdown 2024-09-19T08:12:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.905+0000 7f66b2376740 1 -- 172.21.15.98:0/2886004508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba78680 msgr2=0x55dbeba75840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.905+0000 7f66b2376740 1 --2- 172.21.15.98:0/2886004508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba78680 0x55dbeba75840 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f66a002a990 tx=0x7f66a0043000 comp rx=0 tx=0).stop 2024-09-19T08:12:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.905+0000 7f66b2376740 1 -- 172.21.15.98:0/2886004508 shutdown_connections 2024-09-19T08:12:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.905+0000 7f66b2376740 1 --2- 172.21.15.98:0/2886004508 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55dbeba87f90 0x55dbeba8a380 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.905+0000 7f66b2376740 1 --2- 172.21.15.98:0/2886004508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba78680 0x55dbeba75840 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.905+0000 7f66b2376740 1 --2- 172.21.15.98:0/2886004508 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba6ddc0 0x55dbeba752c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.905+0000 7f66b2376740 1 --2- 172.21.15.98:0/2886004508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6d4a0 0x55dbeba78010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.905+0000 7f66b2376740 1 -- 172.21.15.98:0/2886004508 >> 172.21.15.98:0/2886004508 conn(0x55dbeba69a30 msgr2=0x55dbeba6f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.905+0000 7f66917fa640 0 client.17688 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.906+0000 7f66b2376740 1 -- 172.21.15.98:0/2886004508 shutdown_connections 2024-09-19T08:12:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.906+0000 7f66b2376740 1 -- 172.21.15.98:0/2886004508 wait complete. 2024-09-19T08:12:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.906+0000 7f66b2376740 20 client.17688 trim_cache size 0 max 16384 2024-09-19T08:12:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66b2376740 1 Processor -- start 2024-09-19T08:12:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66b2376740 1 -- start start 2024-09-19T08:12:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66b2376740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6d4a0 0x55dbeba6f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66b2376740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6ddc0 0x55dbeba8cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66b2376740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba78680 0x55dbeba8d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66b2376740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81ab870 con 0x55dbeba6d4a0 2024-09-19T08:12:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66b2376740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81ad3b0 con 0x55dbeba78680 2024-09-19T08:12:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66b2376740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81abf40 con 0x55dbeba6ddc0 2024-09-19T08:12:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66add39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6d4a0 0x55dbeba6f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66add39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6d4a0 0x55dbeba6f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38118/0 (socket says 172.21.15.98:38118) 2024-09-19T08:12:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66add39640 1 -- 172.21.15.98:0/2316739235 learned_addr learned my addr 172.21.15.98:0/2316739235 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66ad538640 1 --2- 172.21.15.98:0/2316739235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6ddc0 0x55dbeba8cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66add39640 1 -- 172.21.15.98:0/2316739235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6ddc0 msgr2=0x55dbeba8cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66ae53a640 1 --2- 172.21.15.98:0/2316739235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba78680 0x55dbeba8d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66add39640 1 --2- 172.21.15.98:0/2316739235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6ddc0 0x55dbeba8cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66add39640 1 -- 172.21.15.98:0/2316739235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba78680 msgr2=0x55dbeba8d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66add39640 1 --2- 172.21.15.98:0/2316739235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba78680 0x55dbeba8d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.923+0000 7f66add39640 1 -- 172.21.15.98:0/2316739235 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbeba6dbb0 con 0x55dbeba6d4a0 2024-09-19T08:12:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.924+0000 7f66add39640 1 --2- 172.21.15.98:0/2316739235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6d4a0 0x55dbeba6f650 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f66a4006cd0 tx=0x7f66a4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5783948fa4677e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.924+0000 7f66937fe640 1 -- 172.21.15.98:0/2316739235 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4016d70 con 0x55dbeba6d4a0 2024-09-19T08:12:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.924+0000 7f66937fe640 1 -- 172.21.15.98:0/2316739235 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a4020030 con 0x55dbeba6d4a0 2024-09-19T08:12:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.924+0000 7f66937fe640 1 -- 172.21.15.98:0/2316739235 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4024db0 con 0x55dbeba6d4a0 2024-09-19T08:12:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.924+0000 7f66b2376740 1 -- 172.21.15.98:0/2316739235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6d4a0 msgr2=0x55dbeba6f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.924+0000 7f66b2376740 1 --2- 172.21.15.98:0/2316739235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6d4a0 0x55dbeba6f650 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f66a4006cd0 tx=0x7f66a4008040 comp rx=0 tx=0).stop 2024-09-19T08:12:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.925+0000 7f66b2376740 1 -- 172.21.15.98:0/2316739235 shutdown_connections 2024-09-19T08:12:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.925+0000 7f66b2376740 1 --2- 172.21.15.98:0/2316739235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba78680 0x55dbeba8d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.925+0000 7f66b2376740 1 --2- 172.21.15.98:0/2316739235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6ddc0 0x55dbeba8cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.925+0000 7f66b2376740 1 --2- 172.21.15.98:0/2316739235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6d4a0 0x55dbeba6f650 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.925+0000 7f66b2376740 1 -- 172.21.15.98:0/2316739235 >> 172.21.15.98:0/2316739235 conn(0x55dbeba69a30 msgr2=0x55dbeba6ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.925+0000 7f66b2376740 1 -- 172.21.15.98:0/2316739235 shutdown_connections 2024-09-19T08:12:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.925+0000 7f66b2376740 1 -- 172.21.15.98:0/2316739235 wait complete. 2024-09-19T08:12:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.925+0000 7f66b2376740 1 -- start start 2024-09-19T08:12:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.925+0000 7f66b2376740 1 Processor -- start 2024-09-19T08:12:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.925+0000 7f66b2376740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6d4a0 0x55dbeba8ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.925+0000 7f66b2376740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6ddc0 0x55dbeba8b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.925+0000 7f66b2376740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba78680 0x55dbeba8b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66b2376740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81acce0 con 0x55dbeba6ddc0 2024-09-19T08:12:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66b2376740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81ac610 con 0x55dbeba78680 2024-09-19T08:12:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66b2376740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81a9870 con 0x55dbeba6d4a0 2024-09-19T08:12:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66b2376740 10 client.0 _ll_register_callbacks cb 0x7f66a0009960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66ae53a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba78680 0x55dbeba8b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66ad538640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6ddc0 0x55dbeba8b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66ae53a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba78680 0x55dbeba8b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:53866/0 (socket says 172.21.15.98:53866) 2024-09-19T08:12:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66ae53a640 1 -- 172.21.15.98:0/2274630202 learned_addr learned my addr 172.21.15.98:0/2274630202 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66add39640 1 --2- 172.21.15.98:0/2274630202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6d4a0 0x55dbeba8ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66ad538640 1 -- 172.21.15.98:0/2274630202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6d4a0 msgr2=0x55dbeba8ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66ad538640 1 --2- 172.21.15.98:0/2274630202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6d4a0 0x55dbeba8ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66ad538640 1 -- 172.21.15.98:0/2274630202 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba78680 msgr2=0x55dbeba8b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66ad538640 1 --2- 172.21.15.98:0/2274630202 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba78680 0x55dbeba8b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66ad538640 1 -- 172.21.15.98:0/2274630202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbebaf1920 con 0x55dbeba6ddc0 2024-09-19T08:12:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66ae53a640 1 --2- 172.21.15.98:0/2274630202 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba78680 0x55dbeba8b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:12:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66ad538640 1 --2- 172.21.15.98:0/2274630202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6ddc0 0x55dbeba8b2d0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f66980cc8f0 tx=0x7f669806b0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=891779932401b5a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66917fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.926+0000 7f66917fa640 1 -- 172.21.15.98:0/2274630202 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669801e040 con 0x55dbeba6ddc0 2024-09-19T08:12:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.927+0000 7f66917fa640 1 -- 172.21.15.98:0/2274630202 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f66980cca60 con 0x55dbeba6ddc0 2024-09-19T08:12:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.927+0000 7f66b2376740 1 -- 172.21.15.98:0/2274630202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a4020030 con 0x55dbeba6ddc0 2024-09-19T08:12:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.927+0000 7f66917fa640 1 -- 172.21.15.98:0/2274630202 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f669809bb80 con 0x55dbeba6ddc0 2024-09-19T08:12:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.927+0000 7f66b2376740 10 client.17694 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:27.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.927+0000 7f66b2376740 1 -- 172.21.15.98:0/2274630202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dbebaf2c40 con 0x55dbeba6ddc0 2024-09-19T08:12:27.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.927+0000 7f66917fa640 1 -- 172.21.15.98:0/2274630202 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a4020030 con 0x55dbeba6ddc0 2024-09-19T08:12:27.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.927+0000 7f66917fa640 1 -- 172.21.15.98:0/2274630202 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6698097af0 con 0x55dbeba6ddc0 2024-09-19T08:12:27.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.927+0000 7f66b2376740 10 client.17694 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:27.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.927+0000 7f66b2376740 10 client.17694 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:12:27.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.927+0000 7f66b2376740 20 client.17694 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:27.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.927+0000 7f66b2376740 1 --2- 172.21.15.98:0/2274630202 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55dbebaf3350 0x55dbebaf5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:27.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.927+0000 7f66b2376740 4 client.17694 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:12:27.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.927+0000 7f66b2376740 1 -- 172.21.15.98:0/2274630202 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dbeb9dad60 con 0x55dbebaf3350 2024-09-19T08:12:27.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.927+0000 7f66add39640 1 --2- 172.21.15.98:0/2274630202 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55dbebaf3350 0x55dbebaf5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.928+0000 7f66917fa640 10 client.17694 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.928+0000 7f66add39640 1 --2- 172.21.15.98:0/2274630202 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55dbebaf3350 0x55dbebaf5770 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67cf1ea6fe900400 server_cookie=3308d8604748a50b in_seq=0 out_seq=0 2024-09-19T08:12:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.928+0000 7f66917fa640 1 -- 172.21.15.98:0/2274630202 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55dbeb9dad60 con 0x55dbebaf3350 2024-09-19T08:12:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.928+0000 7f66917fa640 10 client.17694 handle_command_reply: tid=0 2024-09-19T08:12:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.928+0000 7f66b2376740 1 client.17694 shutdown 2024-09-19T08:12:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.928+0000 7f66b2376740 1 -- 172.21.15.98:0/2274630202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6ddc0 msgr2=0x55dbeba8b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.928+0000 7f66b2376740 1 --2- 172.21.15.98:0/2274630202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6ddc0 0x55dbeba8b2d0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f66980cc8f0 tx=0x7f669806b0a0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.928+0000 7f66b2376740 1 -- 172.21.15.98:0/2274630202 shutdown_connections 2024-09-19T08:12:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.928+0000 7f66b2376740 1 --2- 172.21.15.98:0/2274630202 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55dbebaf3350 0x55dbebaf5770 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.928+0000 7f66b2376740 1 --2- 172.21.15.98:0/2274630202 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dbeba78680 0x55dbeba8b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.928+0000 7f66b2376740 1 --2- 172.21.15.98:0/2274630202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbeba6ddc0 0x55dbeba8b2d0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.928+0000 7f66917fa640 0 client.17694 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.928+0000 7f66b2376740 1 --2- 172.21.15.98:0/2274630202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbeba6d4a0 0x55dbeba8ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.928+0000 7f66b2376740 1 -- 172.21.15.98:0/2274630202 >> 172.21.15.98:0/2274630202 conn(0x55dbeba69a30 msgr2=0x55dbeba68b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66b2376740 1 -- 172.21.15.98:0/2274630202 shutdown_connections 2024-09-19T08:12:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66b2376740 1 -- 172.21.15.98:0/2274630202 wait complete. 2024-09-19T08:12:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66b2376740 20 client.17694 trim_cache size 0 max 16384 2024-09-19T08:12:27.935 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:12:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-19T08:12:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-09-19T08:12:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "08bc7f3d-2411-494d-ba2b-4d2aba099064": { 2024-09-19T08:12:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-09-19T08:12:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "08bc7f3d-2411-494d-ba2b-4d2aba099064", 2024-09-19T08:12:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-09-19T08:12:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:12:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:12:27.936 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:12:27.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66aff9d640 1 -- 172.21.15.98:0/777963493 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f66940a33b0 msgr2=0x7f66940a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:27.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66aff9d640 1 --2- 172.21.15.98:0/777963493 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f66940a33b0 0x7f66940a5870 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f66a004a000 tx=0x7f66a00394d0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66aff9d640 1 -- 172.21.15.98:0/777963493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a8102570 msgr2=0x7f66a8198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:27.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66aff9d640 1 --2- 172.21.15.98:0/777963493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a8102570 0x7f66a8198b20 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f669800b4a0 tx=0x7f669800b960 comp rx=0 tx=0).stop 2024-09-19T08:12:27.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66aff9d640 1 -- 172.21.15.98:0/777963493 shutdown_connections 2024-09-19T08:12:27.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66aff9d640 1 --2- 172.21.15.98:0/777963493 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f66940a33b0 0x7f66940a5870 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66aff9d640 1 --2- 172.21.15.98:0/777963493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66a8102ef0 0x7f66a819ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66aff9d640 1 --2- 172.21.15.98:0/777963493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66a8102570 0x7f66a8198b20 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66aff9d640 1 --2- 172.21.15.98:0/777963493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a8100e20 0x7f66a81985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:27.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66aff9d640 1 -- 172.21.15.98:0/777963493 >> 172.21.15.98:0/777963493 conn(0x7f66a80fc7d0 msgr2=0x7f66a80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:27.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66aff9d640 1 -- 172.21.15.98:0/777963493 shutdown_connections 2024-09-19T08:12:27.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:27.929+0000 7f66aff9d640 1 -- 172.21.15.98:0/777963493 wait complete. 2024-09-19T08:12:28.620 INFO:tasks.cephfs.filesystem:scrub status for tag:08bc7f3d-2411-494d-ba2b-4d2aba099064 - {'path': '/', 'tag': '08bc7f3d-2411-494d-ba2b-4d2aba099064', 'options': 'recursive,force'} 2024-09-19T08:12:29.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:28 smithi139 ceph-mon[34874]: pgmap v292: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-09-19T08:12:29.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:28 smithi098 ceph-mon[31333]: pgmap v292: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-09-19T08:12:29.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:28 smithi162 ceph-mon[35260]: pgmap v292: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-09-19T08:12:31.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:30 smithi139 ceph-mon[34874]: pgmap v293: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-19T08:12:31.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:30 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:12:31.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:30 smithi098 ceph-mon[31333]: pgmap v293: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-19T08:12:31.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:30 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:12:31.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:30 smithi162 ceph-mon[35260]: pgmap v293: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-19T08:12:31.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:30 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:12:33.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:32 smithi139 ceph-mon[34874]: pgmap v294: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-19T08:12:33.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:32 smithi098 ceph-mon[31333]: pgmap v294: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-19T08:12:33.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:32 smithi162 ceph-mon[35260]: pgmap v294: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-19T08:12:35.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:34 smithi139 ceph-mon[34874]: pgmap v295: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:12:35.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:34 smithi098 ceph-mon[31333]: pgmap v295: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:12:35.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:34 smithi162 ceph-mon[35260]: pgmap v295: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-09-19T08:12:37.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:36 smithi139 ceph-mon[34874]: pgmap v296: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:12:37.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:36 smithi098 ceph-mon[31333]: pgmap v296: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:12:37.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:36 smithi162 ceph-mon[35260]: pgmap v296: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-19T08:12:39.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:38 smithi139 ceph-mon[34874]: pgmap v297: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-09-19T08:12:39.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:38 smithi098 ceph-mon[31333]: pgmap v297: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-09-19T08:12:39.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:38 smithi162 ceph-mon[35260]: pgmap v297: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-09-19T08:12:41.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:40 smithi139 ceph-mon[34874]: pgmap v298: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-09-19T08:12:41.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:40 smithi098 ceph-mon[31333]: pgmap v298: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-09-19T08:12:41.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:40 smithi162 ceph-mon[35260]: pgmap v298: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-09-19T08:12:43.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:42 smithi139 ceph-mon[34874]: pgmap v299: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:43.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:42 smithi098 ceph-mon[31333]: pgmap v299: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:43.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:42 smithi162 ceph-mon[35260]: pgmap v299: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:45.172 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:44 smithi162 ceph-mon[35260]: pgmap v300: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:12:45.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:44 smithi139 ceph-mon[34874]: pgmap v300: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:12:45.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:44 smithi098 ceph-mon[31333]: pgmap v300: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:12:47.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:46 smithi139 ceph-mon[34874]: pgmap v301: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:47.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:46 smithi098 ceph-mon[31333]: pgmap v301: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:47.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:46 smithi162 ceph-mon[35260]: pgmap v301: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:49.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:48 smithi139 ceph-mon[34874]: pgmap v302: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:12:49.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:48 smithi098 ceph-mon[31333]: pgmap v302: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:12:49.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:48 smithi162 ceph-mon[35260]: pgmap v302: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:12:51.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:50 smithi139 ceph-mon[34874]: pgmap v303: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:12:51.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:50 smithi162 ceph-mon[35260]: pgmap v303: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:12:51.539 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:50 smithi098 ceph-mon[31333]: pgmap v303: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:12:53.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:52 smithi139 ceph-mon[34874]: pgmap v304: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:53.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:52 smithi098 ceph-mon[31333]: pgmap v304: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:53.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:52 smithi162 ceph-mon[35260]: pgmap v304: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:54.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:53 smithi162 ceph-mon[35260]: pgmap v305: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:12:54.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:53 smithi139 ceph-mon[34874]: pgmap v305: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:12:54.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:53 smithi098 ceph-mon[31333]: pgmap v305: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:12:56.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:55 smithi162 ceph-mon[35260]: pgmap v306: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:56.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:55 smithi098 ceph-mon[31333]: pgmap v306: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:56.580 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:55 smithi139 ceph-mon[34874]: pgmap v306: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:12:58.234 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:57 smithi098 ceph-mon[31333]: pgmap v307: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:12:58.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:57 smithi162 ceph-mon[35260]: pgmap v307: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:12:58.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:57 smithi139 ceph-mon[34874]: pgmap v307: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:12:58.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:12:58.900 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:12:59.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.389+0000 7fc478f57640 1 -- 172.21.15.98:0/675515834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc474102780 msgr2=0x7fc4741067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:59.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.389+0000 7fc478f57640 1 --2- 172.21.15.98:0/675515834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc474102780 0x7fc4741067d0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fc468009900 tx=0x7fc46802a410 comp rx=0 tx=0).stop 2024-09-19T08:12:59.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.389+0000 7fc478f57640 1 -- 172.21.15.98:0/675515834 shutdown_connections 2024-09-19T08:12:59.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.389+0000 7fc478f57640 1 --2- 172.21.15.98:0/675515834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc474106f00 0x7fc47410ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.389+0000 7fc478f57640 1 --2- 172.21.15.98:0/675515834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc474102780 0x7fc4741067d0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.389+0000 7fc478f57640 1 --2- 172.21.15.98:0/675515834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc474101dd0 0x7fc4741021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.389+0000 7fc478f57640 1 -- 172.21.15.98:0/675515834 >> 172.21.15.98:0/675515834 conn(0x7fc4740fd8b0 msgr2=0x7fc4740ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:59.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.389+0000 7fc478f57640 1 -- 172.21.15.98:0/675515834 shutdown_connections 2024-09-19T08:12:59.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.390+0000 7fc478f57640 1 -- 172.21.15.98:0/675515834 wait complete. 2024-09-19T08:12:59.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.390+0000 7fc478f57640 1 Processor -- start 2024-09-19T08:12:59.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.390+0000 7fc478f57640 1 -- start start 2024-09-19T08:12:59.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.390+0000 7fc478f57640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc474101dd0 0x7fc4740744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.390+0000 7fc478f57640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc474102780 0x7fc474074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.390+0000 7fc478f57640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc474106f00 0x7fc474074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.390+0000 7fc478f57640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc474111f60 con 0x7fc474101dd0 2024-09-19T08:12:59.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.390+0000 7fc478f57640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc474111de0 con 0x7fc474106f00 2024-09-19T08:12:59.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.390+0000 7fc478f57640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc4741120e0 con 0x7fc474102780 2024-09-19T08:12:59.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc47259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc474101dd0 0x7fc4740744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc47259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc474101dd0 0x7fc4740744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42860/0 (socket says 172.21.15.98:42860) 2024-09-19T08:12:59.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc47259c640 1 -- 172.21.15.98:0/1380451929 learned_addr learned my addr 172.21.15.98:0/1380451929 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:59.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc471d9b640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc474102780 0x7fc474074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc47259c640 1 -- 172.21.15.98:0/1380451929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc474102780 msgr2=0x7fc474074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:59.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc47259c640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc474102780 0x7fc474074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc472d9d640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc474106f00 0x7fc474074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc47259c640 1 -- 172.21.15.98:0/1380451929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc474106f00 msgr2=0x7fc474074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:59.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc47259c640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc474106f00 0x7fc474074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc47259c640 1 -- 172.21.15.98:0/1380451929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4741a6860 con 0x7fc474101dd0 2024-09-19T08:12:59.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc471d9b640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc474102780 0x7fc474074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:59.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc47259c640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc474101dd0 0x7fc4740744f0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fc46400d770 tx=0x7fc46400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:59.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc4637fe640 1 -- 172.21.15.98:0/1380451929 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc464017070 con 0x7fc474101dd0 2024-09-19T08:12:59.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc478f57640 1 -- 172.21.15.98:0/1380451929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4741a6b50 con 0x7fc474101dd0 2024-09-19T08:12:59.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc4637fe640 1 -- 172.21.15.98:0/1380451929 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc464005500 con 0x7fc474101dd0 2024-09-19T08:12:59.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc4637fe640 1 -- 172.21.15.98:0/1380451929 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc464004e00 con 0x7fc474101dd0 2024-09-19T08:12:59.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.391+0000 7fc478f57640 1 -- 172.21.15.98:0/1380451929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4741a7040 con 0x7fc474101dd0 2024-09-19T08:12:59.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.394+0000 7fc4637fe640 1 -- 172.21.15.98:0/1380451929 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fc464004fa0 con 0x7fc474101dd0 2024-09-19T08:12:59.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.394+0000 7fc47b330740 1 Processor -- start 2024-09-19T08:12:59.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.394+0000 7fc47b330740 1 -- start start 2024-09-19T08:12:59.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.394+0000 7fc47b330740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5c84a0 0x561c0d5c8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.394+0000 7fc47b330740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.394+0000 7fc47b330740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5d3680 0x561c0d5d5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.394+0000 7fc47b330740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561c0d52e2e0 con 0x561c0d5c84a0 2024-09-19T08:12:59.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.394+0000 7fc47b330740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561c0d4d5ce0 con 0x561c0d5c8dc0 2024-09-19T08:12:59.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.394+0000 7fc47b330740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561c0d52a2a0 con 0x561c0d5d3680 2024-09-19T08:12:59.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.394+0000 7fc472d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5c84a0 0x561c0d5c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.394+0000 7fc472d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5c84a0 0x561c0d5c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42870/0 (socket says 172.21.15.98:42870) 2024-09-19T08:12:59.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.394+0000 7fc472d9d640 1 -- 172.21.15.98:0/2005871975 learned_addr learned my addr 172.21.15.98:0/2005871975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:59.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc471d9b640 1 --2- 172.21.15.98:0/2005871975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5d3680 0x561c0d5d5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc47259c640 1 --2- 172.21.15.98:0/2005871975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc472d9d640 1 -- 172.21.15.98:0/2005871975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5d3680 msgr2=0x561c0d5d5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:59.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc472d9d640 1 --2- 172.21.15.98:0/2005871975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5d3680 0x561c0d5d5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc472d9d640 1 -- 172.21.15.98:0/2005871975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c8dc0 msgr2=0x561c0d5d3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:59.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc472d9d640 1 --2- 172.21.15.98:0/2005871975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc472d9d640 1 -- 172.21.15.98:0/2005871975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c0d4ba440 con 0x561c0d5c84a0 2024-09-19T08:12:59.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc47259c640 1 --2- 172.21.15.98:0/2005871975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:59.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc4637fe640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc45c0a33b0 0x7fc45c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc472d9d640 1 --2- 172.21.15.98:0/2005871975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5c84a0 0x561c0d5c8880 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fc46c005fd0 tx=0x7fc46c00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7600c2d5ffed9525 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:59.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc47259c640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc45c0a33b0 0x7fc45c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc4637fe640 1 -- 172.21.15.98:0/1380451929 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7fc4640cf850 con 0x7fc474101dd0 2024-09-19T08:12:59.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc460ff9640 1 -- 172.21.15.98:0/2005871975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46c018680 con 0x561c0d5c84a0 2024-09-19T08:12:59.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc460ff9640 1 -- 172.21.15.98:0/2005871975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc46c018d00 con 0x561c0d5c84a0 2024-09-19T08:12:59.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.395+0000 7fc460ff9640 1 -- 172.21.15.98:0/2005871975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46c00e3b0 con 0x561c0d5c84a0 2024-09-19T08:12:59.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.396+0000 7fc47b330740 1 -- 172.21.15.98:0/2005871975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5c84a0 msgr2=0x561c0d5c8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:59.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.396+0000 7fc47259c640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc45c0a33b0 0x7fc45c0a5870 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc4640cd050 tx=0x7fc4640048a0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:59.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.396+0000 7fc47b330740 1 --2- 172.21.15.98:0/2005871975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5c84a0 0x561c0d5c8880 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fc46c005fd0 tx=0x7fc46c00d710 comp rx=0 tx=0).stop 2024-09-19T08:12:59.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.396+0000 7fc47b330740 1 -- 172.21.15.98:0/2005871975 shutdown_connections 2024-09-19T08:12:59.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.396+0000 7fc47b330740 1 --2- 172.21.15.98:0/2005871975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5d3680 0x561c0d5d5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.396+0000 7fc47b330740 1 --2- 172.21.15.98:0/2005871975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.396+0000 7fc47b330740 1 --2- 172.21.15.98:0/2005871975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5c84a0 0x561c0d5c8880 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.396+0000 7fc47b330740 1 -- 172.21.15.98:0/2005871975 >> 172.21.15.98:0/2005871975 conn(0x561c0d5c4a30 msgr2=0x561c0d5c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:59.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.396+0000 7fc47b330740 1 -- 172.21.15.98:0/2005871975 shutdown_connections 2024-09-19T08:12:59.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.396+0000 7fc47b330740 1 -- 172.21.15.98:0/2005871975 wait complete. 2024-09-19T08:12:59.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.397+0000 7fc47b330740 1 -- start start 2024-09-19T08:12:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.397+0000 7fc47b330740 1 Processor -- start 2024-09-19T08:12:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.397+0000 7fc47b330740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c84a0 0x561c0d5d3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.397+0000 7fc47b330740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.397+0000 7fc47b330740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5d3680 0x561c0d5d0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.397+0000 7fc47b330740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561c0d5d6250 con 0x561c0d5d3680 2024-09-19T08:12:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.397+0000 7fc47b330740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561c0d4e9050 con 0x561c0d5c84a0 2024-09-19T08:12:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.397+0000 7fc47b330740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561c0d5d63d0 con 0x561c0d5c8dc0 2024-09-19T08:12:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.397+0000 7fc47b330740 10 client.0 _ll_register_callbacks cb 0x561c0d529160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.397+0000 7fc472d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5d3680 0x561c0d5d0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.397+0000 7fc472d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5d3680 0x561c0d5d0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42882/0 (socket says 172.21.15.98:42882) 2024-09-19T08:12:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.397+0000 7fc472d9d640 1 -- 172.21.15.98:0/568010376 learned_addr learned my addr 172.21.15.98:0/568010376 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.397+0000 7fc471d9b640 1 --2- 172.21.15.98:0/568010376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc472d9d640 1 --2- 172.21.15.98:0/568010376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c84a0 0x561c0d5d3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc472d9d640 1 -- 172.21.15.98:0/568010376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5c8dc0 msgr2=0x561c0d5d02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:59.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc472d9d640 1 --2- 172.21.15.98:0/568010376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc472d9d640 1 -- 172.21.15.98:0/568010376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c84a0 msgr2=0x561c0d5d3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:59.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc472d9d640 1 --2- 172.21.15.98:0/568010376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c84a0 0x561c0d5d3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc472d9d640 1 -- 172.21.15.98:0/568010376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c0d29f3c0 con 0x561c0d5d3680 2024-09-19T08:12:59.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc472d9d640 1 --2- 172.21.15.98:0/568010376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5d3680 0x561c0d5d0840 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fc46c000f80 tx=0x7fc46c00fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b26118c63fc1810b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:59.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc459ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:59.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc459ffb640 1 -- 172.21.15.98:0/568010376 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46c004a40 con 0x561c0d5d3680 2024-09-19T08:12:59.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc459ffb640 1 -- 172.21.15.98:0/568010376 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7fc46c004be0 con 0x561c0d5d3680 2024-09-19T08:12:59.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc47b330740 1 -- 172.21.15.98:0/568010376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc46c018d00 con 0x561c0d5d3680 2024-09-19T08:12:59.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc459ffb640 1 -- 172.21.15.98:0/568010376 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc46c004de0 con 0x561c0d5d3680 2024-09-19T08:12:59.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc47b330740 10 client.17706 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:59.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc47b330740 1 -- 172.21.15.98:0/568010376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561c0d5e1c10 con 0x561c0d5d3680 2024-09-19T08:12:59.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.398+0000 7fc459ffb640 1 -- 172.21.15.98:0/568010376 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc46c018d00 con 0x561c0d5d3680 2024-09-19T08:12:59.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.399+0000 7fc459ffb640 1 -- 172.21.15.98:0/568010376 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc46c00ecc0 con 0x561c0d5d3680 2024-09-19T08:12:59.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.399+0000 7fc47b330740 10 client.17706 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:59.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.399+0000 7fc47b330740 10 client.17706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:12:59.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.399+0000 7fc47b330740 20 client.17706 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:59.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.399+0000 7fc47b330740 1 --2- 172.21.15.98:0/568010376 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c0d5e30d0 0x561c0d5e54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.399+0000 7fc47b330740 4 client.17706 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:12:59.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.399+0000 7fc47b330740 1 -- 172.21.15.98:0/568010376 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c0d53b850 con 0x561c0d5e30d0 2024-09-19T08:12:59.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.399+0000 7fc471d9b640 1 --2- 172.21.15.98:0/568010376 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c0d5e30d0 0x561c0d5e54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.399+0000 7fc471d9b640 1 --2- 172.21.15.98:0/568010376 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c0d5e30d0 0x561c0d5e54c0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27f260071334034c server_cookie=18234a7e09525cd5 in_seq=0 out_seq=0 2024-09-19T08:12:59.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.399+0000 7fc459ffb640 10 client.17706 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:59.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.400+0000 7fc459ffb640 1 -- 172.21.15.98:0/568010376 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561c0d53b850 con 0x561c0d5e30d0 2024-09-19T08:12:59.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.400+0000 7fc459ffb640 10 client.17706 handle_command_reply: tid=0 2024-09-19T08:12:59.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.400+0000 7fc47b330740 1 client.17706 shutdown 2024-09-19T08:12:59.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.400+0000 7fc47b330740 1 -- 172.21.15.98:0/568010376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5d3680 msgr2=0x561c0d5d0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:59.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.400+0000 7fc47b330740 1 --2- 172.21.15.98:0/568010376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5d3680 0x561c0d5d0840 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fc46c000f80 tx=0x7fc46c00fa80 comp rx=0 tx=0).stop 2024-09-19T08:12:59.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.400+0000 7fc47b330740 1 -- 172.21.15.98:0/568010376 shutdown_connections 2024-09-19T08:12:59.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.400+0000 7fc47b330740 1 --2- 172.21.15.98:0/568010376 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c0d5e30d0 0x561c0d5e54c0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.400+0000 7fc47b330740 1 --2- 172.21.15.98:0/568010376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5d3680 0x561c0d5d0840 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.400+0000 7fc47b330740 1 --2- 172.21.15.98:0/568010376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.400+0000 7fc47b330740 1 --2- 172.21.15.98:0/568010376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c84a0 0x561c0d5d3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.400+0000 7fc47b330740 1 -- 172.21.15.98:0/568010376 >> 172.21.15.98:0/568010376 conn(0x561c0d5c4a30 msgr2=0x561c0d5c3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:59.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.400+0000 7fc459ffb640 0 client.17706 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:59.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.400+0000 7fc47b330740 1 -- 172.21.15.98:0/568010376 shutdown_connections 2024-09-19T08:12:59.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.400+0000 7fc47b330740 1 -- 172.21.15.98:0/568010376 wait complete. 2024-09-19T08:12:59.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.401+0000 7fc47b330740 20 client.17706 trim_cache size 0 max 16384 2024-09-19T08:12:59.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.417+0000 7fc47b330740 1 Processor -- start 2024-09-19T08:12:59.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.417+0000 7fc47b330740 1 -- start start 2024-09-19T08:12:59.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.417+0000 7fc47b330740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5c84a0 0x561c0d5df4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.417+0000 7fc47b330740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c8dc0 0x561c0d5eabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.417+0000 7fc47b330740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5d3680 0x561c0d5eb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.417+0000 7fc47b330740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4741af9f0 con 0x561c0d5d3680 2024-09-19T08:12:59.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.417+0000 7fc47b330740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc4741b1530 con 0x561c0d5c8dc0 2024-09-19T08:12:59.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.417+0000 7fc47b330740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc4741b00c0 con 0x561c0d5c84a0 2024-09-19T08:12:59.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.418+0000 7fc472d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5d3680 0x561c0d5eb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.418+0000 7fc472d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5d3680 0x561c0d5eb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42896/0 (socket says 172.21.15.98:42896) 2024-09-19T08:12:59.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.418+0000 7fc472d9d640 1 -- 172.21.15.98:0/243947370 learned_addr learned my addr 172.21.15.98:0/243947370 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:59.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.418+0000 7fc471d9b640 1 --2- 172.21.15.98:0/243947370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c8dc0 0x561c0d5eabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.418+0000 7fc472d9d640 1 -- 172.21.15.98:0/243947370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5c84a0 msgr2=0x561c0d5df4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:59.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.418+0000 7fc472d9d640 1 --2- 172.21.15.98:0/243947370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5c84a0 0x561c0d5df4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.418+0000 7fc472d9d640 1 -- 172.21.15.98:0/243947370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c8dc0 msgr2=0x561c0d5eabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:59.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.418+0000 7fc472d9d640 1 --2- 172.21.15.98:0/243947370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c8dc0 0x561c0d5eabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.418+0000 7fc472d9d640 1 -- 172.21.15.98:0/243947370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c0d5c8bf0 con 0x561c0d5d3680 2024-09-19T08:12:59.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.418+0000 7fc472d9d640 1 --2- 172.21.15.98:0/243947370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5d3680 0x561c0d5eb120 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc46c002230 tx=0x7fc46c030770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acc59a0c8ef5ab86 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:59.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.418+0000 7fc45bfff640 1 -- 172.21.15.98:0/243947370 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46c01c070 con 0x561c0d5d3680 2024-09-19T08:12:59.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.418+0000 7fc45bfff640 1 -- 172.21.15.98:0/243947370 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc46c018d00 con 0x561c0d5d3680 2024-09-19T08:12:59.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.418+0000 7fc45bfff640 1 -- 172.21.15.98:0/243947370 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46c00e4e0 con 0x561c0d5d3680 2024-09-19T08:12:59.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.419+0000 7fc47b330740 1 -- 172.21.15.98:0/243947370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5d3680 msgr2=0x561c0d5eb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:59.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.419+0000 7fc47b330740 1 --2- 172.21.15.98:0/243947370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5d3680 0x561c0d5eb120 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc46c002230 tx=0x7fc46c030770 comp rx=0 tx=0).stop 2024-09-19T08:12:59.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.419+0000 7fc47b330740 1 -- 172.21.15.98:0/243947370 shutdown_connections 2024-09-19T08:12:59.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.419+0000 7fc47b330740 1 --2- 172.21.15.98:0/243947370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5d3680 0x561c0d5eb120 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.419+0000 7fc47b330740 1 --2- 172.21.15.98:0/243947370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5c8dc0 0x561c0d5eabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.419+0000 7fc47b330740 1 --2- 172.21.15.98:0/243947370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5c84a0 0x561c0d5df4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.419+0000 7fc47b330740 1 -- 172.21.15.98:0/243947370 >> 172.21.15.98:0/243947370 conn(0x561c0d5c4a30 msgr2=0x561c0d5cb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:59.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.419+0000 7fc47b330740 1 -- 172.21.15.98:0/243947370 shutdown_connections 2024-09-19T08:12:59.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.419+0000 7fc47b330740 1 -- 172.21.15.98:0/243947370 wait complete. 2024-09-19T08:12:59.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.419+0000 7fc47b330740 1 -- start start 2024-09-19T08:12:59.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.419+0000 7fc47b330740 1 Processor -- start 2024-09-19T08:12:59.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc47b330740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5c84a0 0x561c0d5d65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc47b330740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc47b330740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5d3680 0x561c0d5d7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc47b330740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4741b0e60 con 0x561c0d5c8dc0 2024-09-19T08:12:59.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc47b330740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc4741b0790 con 0x561c0d5d3680 2024-09-19T08:12:59.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc47b330740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc4741ad9f0 con 0x561c0d5c84a0 2024-09-19T08:12:59.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc47b330740 10 client.0 _ll_register_callbacks cb 0x7fc46c008bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:12:59.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc471d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc471d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42904/0 (socket says 172.21.15.98:42904) 2024-09-19T08:12:59.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc471d9b640 1 -- 172.21.15.98:0/1556986003 learned_addr learned my addr 172.21.15.98:0/1556986003 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:12:59.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc472d9d640 1 --2- 172.21.15.98:0/1556986003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5c84a0 0x561c0d5d65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc472d9d640 1 --2- 172.21.15.98:0/1556986003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5d3680 0x561c0d5d7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc471d9b640 1 -- 172.21.15.98:0/1556986003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5c84a0 msgr2=0x561c0d5d65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:59.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc471d9b640 1 --2- 172.21.15.98:0/1556986003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5c84a0 0x561c0d5d65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc471d9b640 1 -- 172.21.15.98:0/1556986003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5d3680 msgr2=0x561c0d5d7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:12:59.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc471d9b640 1 --2- 172.21.15.98:0/1556986003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5d3680 0x561c0d5d7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc471d9b640 1 -- 172.21.15.98:0/1556986003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c0d64be40 con 0x561c0d5c8dc0 2024-09-19T08:12:59.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc472d9d640 1 --2- 172.21.15.98:0/1556986003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5d3680 0x561c0d5d7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:12:59.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc471d9b640 1 --2- 172.21.15.98:0/1556986003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d6b20 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc468009230 tx=0x7fc468004300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8410fee15f59230c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:12:59.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc459ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:12:59.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc459ffb640 1 -- 172.21.15.98:0/1556986003 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc468040030 con 0x561c0d5c8dc0 2024-09-19T08:12:59.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc459ffb640 1 -- 172.21.15.98:0/1556986003 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7fc468005040 con 0x561c0d5c8dc0 2024-09-19T08:12:59.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.420+0000 7fc47b330740 1 -- 172.21.15.98:0/1556986003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc46c018d00 con 0x561c0d5c8dc0 2024-09-19T08:12:59.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.421+0000 7fc459ffb640 1 -- 172.21.15.98:0/1556986003 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc468050050 con 0x561c0d5c8dc0 2024-09-19T08:12:59.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.421+0000 7fc47b330740 10 client.17712 fetch_fsmap learned FSMap version 32 2024-09-19T08:12:59.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.421+0000 7fc47b330740 1 -- 172.21.15.98:0/1556986003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561c0d64d150 con 0x561c0d5c8dc0 2024-09-19T08:12:59.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.421+0000 7fc459ffb640 1 -- 172.21.15.98:0/1556986003 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc46c018d00 con 0x561c0d5c8dc0 2024-09-19T08:12:59.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.421+0000 7fc459ffb640 1 -- 172.21.15.98:0/1556986003 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc46804b030 con 0x561c0d5c8dc0 2024-09-19T08:12:59.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.421+0000 7fc47b330740 10 client.17712 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:12:59.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.421+0000 7fc47b330740 10 client.17712 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:12:59.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.421+0000 7fc47b330740 20 client.17712 populate_metadata read hostname 'smithi098' 2024-09-19T08:12:59.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.421+0000 7fc47b330740 1 --2- 172.21.15.98:0/1556986003 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c0d64d900 0x561c0d64fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:12:59.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.421+0000 7fc47b330740 4 client.17712 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:12:59.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.421+0000 7fc47b330740 1 -- 172.21.15.98:0/1556986003 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561c0d535d60 con 0x561c0d64d900 2024-09-19T08:12:59.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.421+0000 7fc472d9d640 1 --2- 172.21.15.98:0/1556986003 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c0d64d900 0x561c0d64fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:12:59.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc459ffb640 10 client.17712 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:59.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc472d9d640 1 --2- 172.21.15.98:0/1556986003 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c0d64d900 0x561c0d64fd20 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b5a4d57fdafb754 server_cookie=eea1be1f3f0e98df in_seq=0 out_seq=0 2024-09-19T08:12:59.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc459ffb640 1 -- 172.21.15.98:0/1556986003 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561c0d535d60 con 0x561c0d64d900 2024-09-19T08:12:59.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc459ffb640 10 client.17712 handle_command_reply: tid=0 2024-09-19T08:12:59.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc47b330740 1 client.17712 shutdown 2024-09-19T08:12:59.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc47b330740 1 -- 172.21.15.98:0/1556986003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5c8dc0 msgr2=0x561c0d5d6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:59.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc47b330740 1 --2- 172.21.15.98:0/1556986003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d6b20 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc468009230 tx=0x7fc468004300 comp rx=0 tx=0).stop 2024-09-19T08:12:59.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc47b330740 1 -- 172.21.15.98:0/1556986003 shutdown_connections 2024-09-19T08:12:59.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc47b330740 1 --2- 172.21.15.98:0/1556986003 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x561c0d64d900 0x561c0d64fd20 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc47b330740 1 --2- 172.21.15.98:0/1556986003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561c0d5d3680 0x561c0d5d7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc47b330740 1 --2- 172.21.15.98:0/1556986003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561c0d5c8dc0 0x561c0d5d6b20 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc47b330740 1 --2- 172.21.15.98:0/1556986003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c0d5c84a0 0x561c0d5d65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc47b330740 1 -- 172.21.15.98:0/1556986003 >> 172.21.15.98:0/1556986003 conn(0x561c0d5c4a30 msgr2=0x561c0d5d1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:59.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc459ffb640 0 client.17712 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:12:59.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc47b330740 1 -- 172.21.15.98:0/1556986003 shutdown_connections 2024-09-19T08:12:59.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.422+0000 7fc47b330740 1 -- 172.21.15.98:0/1556986003 wait complete. 2024-09-19T08:12:59.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.423+0000 7fc47b330740 20 client.17712 trim_cache size 0 max 16384 2024-09-19T08:12:59.428 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:12:59.428 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:12:59.428 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:12:59.428 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:12:59.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.423+0000 7fc478f57640 1 -- 172.21.15.98:0/1380451929 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc45c0a33b0 msgr2=0x7fc45c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:59.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.423+0000 7fc478f57640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc45c0a33b0 0x7fc45c0a5870 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc4640cd050 tx=0x7fc4640048a0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.423+0000 7fc478f57640 1 -- 172.21.15.98:0/1380451929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc474101dd0 msgr2=0x7fc4740744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:12:59.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.423+0000 7fc478f57640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc474101dd0 0x7fc4740744f0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fc46400d770 tx=0x7fc46400dc30 comp rx=0 tx=0).stop 2024-09-19T08:12:59.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.423+0000 7fc478f57640 1 -- 172.21.15.98:0/1380451929 shutdown_connections 2024-09-19T08:12:59.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.423+0000 7fc478f57640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fc45c0a33b0 0x7fc45c0a5870 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.423+0000 7fc478f57640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc474106f00 0x7fc474074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.423+0000 7fc478f57640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc474102780 0x7fc474074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.423+0000 7fc478f57640 1 --2- 172.21.15.98:0/1380451929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc474101dd0 0x7fc4740744f0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:12:59.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.423+0000 7fc478f57640 1 -- 172.21.15.98:0/1380451929 >> 172.21.15.98:0/1380451929 conn(0x7fc4740fd8b0 msgr2=0x7fc4740febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:12:59.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.423+0000 7fc478f57640 1 -- 172.21.15.98:0/1380451929 shutdown_connections 2024-09-19T08:12:59.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:12:59.423+0000 7fc478f57640 1 -- 172.21.15.98:0/1380451929 wait complete. 2024-09-19T08:13:00.240 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:12:59 smithi098 ceph-mon[31333]: pgmap v308: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:13:00.242 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:13:00.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:13:00.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:12:59 smithi162 ceph-mon[35260]: pgmap v308: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:13:00.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:12:59 smithi139 ceph-mon[34874]: pgmap v308: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:13:00.493 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:01.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.997+0000 7f1bd6946640 1 -- 172.21.15.98:0/4214969799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0102ef0 msgr2=0x7f1bd010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:01.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.997+0000 7f1bd6946640 1 --2- 172.21.15.98:0/4214969799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0102ef0 0x7f1bd010da40 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009ca0 tx=0x7f1bc802a950 comp rx=0 tx=0).stop 2024-09-19T08:13:01.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.997+0000 7f1bd6946640 1 -- 172.21.15.98:0/4214969799 shutdown_connections 2024-09-19T08:13:01.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.997+0000 7f1bd6946640 1 --2- 172.21.15.98:0/4214969799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0102ef0 0x7f1bd010da40 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:01.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.997+0000 7f1bd6946640 1 --2- 172.21.15.98:0/4214969799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1bd0102570 0x7f1bd01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:01.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.997+0000 7f1bd6946640 1 --2- 172.21.15.98:0/4214969799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bd0100e20 0x7f1bd0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:01.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.997+0000 7f1bd6946640 1 -- 172.21.15.98:0/4214969799 >> 172.21.15.98:0/4214969799 conn(0x7f1bd00fc7d0 msgr2=0x7f1bd00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:01.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.997+0000 7f1bd6946640 1 -- 172.21.15.98:0/4214969799 shutdown_connections 2024-09-19T08:13:01.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.997+0000 7f1bd6946640 1 -- 172.21.15.98:0/4214969799 wait complete. 2024-09-19T08:13:01.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.998+0000 7f1bd6946640 1 Processor -- start 2024-09-19T08:13:01.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.998+0000 7f1bd6946640 1 -- start start 2024-09-19T08:13:01.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.998+0000 7f1bd6946640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1bd0100e20 0x7f1bd0197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:01.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.998+0000 7f1bd6946640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bd0102570 0x7f1bd0197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:01.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.998+0000 7f1bd6946640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0102ef0 0x7f1bd019bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:01.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.998+0000 7f1bd6946640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd0110f80 con 0x7f1bd0102ef0 2024-09-19T08:13:01.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.998+0000 7f1bd6946640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd0110e00 con 0x7f1bd0102570 2024-09-19T08:13:01.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.998+0000 7f1bd6946640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd0111100 con 0x7f1bd0100e20 2024-09-19T08:13:01.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.998+0000 7f1bd4ee3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0102ef0 0x7f1bd019bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:01.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.998+0000 7f1bd4ee3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0102ef0 0x7f1bd019bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42932/0 (socket says 172.21.15.98:42932) 2024-09-19T08:13:01.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.998+0000 7f1bd4ee3640 1 -- 172.21.15.98:0/2036814312 learned_addr learned my addr 172.21.15.98:0/2036814312 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:01.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.998+0000 7f1bcffff640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1bd0100e20 0x7f1bd0197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:01.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.998+0000 7f1bcf7fe640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bd0102570 0x7f1bd0197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:01.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.999+0000 7f1bd4ee3640 1 -- 172.21.15.98:0/2036814312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1bd0100e20 msgr2=0x7f1bd0197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:01.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.999+0000 7f1bd4ee3640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1bd0100e20 0x7f1bd0197320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:01.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.999+0000 7f1bd4ee3640 1 -- 172.21.15.98:0/2036814312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bd0102570 msgr2=0x7f1bd0197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:01.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.999+0000 7f1bd4ee3640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bd0102570 0x7f1bd0197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:01.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.999+0000 7f1bd4ee3640 1 -- 172.21.15.98:0/2036814312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd019c2d0 con 0x7f1bd0102ef0 2024-09-19T08:13:01.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.999+0000 7f1bcf7fe640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bd0102570 0x7f1bd0197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:01.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.999+0000 7f1bd4ee3640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0102ef0 0x7f1bd019bbf0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f1bc802ae70 tx=0x7f1bc802f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:01.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.999+0000 7f1bcd7fa640 1 -- 172.21.15.98:0/2036814312 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc8034040 con 0x7f1bd0102ef0 2024-09-19T08:13:01.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.999+0000 7f1bcd7fa640 1 -- 172.21.15.98:0/2036814312 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bc8030920 con 0x7f1bd0102ef0 2024-09-19T08:13:01.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.999+0000 7f1bd6946640 1 -- 172.21.15.98:0/2036814312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd019c560 con 0x7f1bd0102ef0 2024-09-19T08:13:01.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.999+0000 7f1bcd7fa640 1 -- 172.21.15.98:0/2036814312 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc80047f0 con 0x7f1bd0102ef0 2024-09-19T08:13:01.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:00.999+0000 7f1bd6946640 1 -- 172.21.15.98:0/2036814312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd01a3ea0 con 0x7f1bd0102ef0 2024-09-19T08:13:01.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.000+0000 7f1bd6946640 1 -- 172.21.15.98:0/2036814312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd0101200 con 0x7f1bd0102ef0 2024-09-19T08:13:01.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.001+0000 7f1bcd7fa640 1 -- 172.21.15.98:0/2036814312 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f1bc8037020 con 0x7f1bd0102ef0 2024-09-19T08:13:01.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.002+0000 7f1bcd7fa640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f1bb80a33b0 0x7f1bb80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:01.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.002+0000 7f1bcd7fa640 1 -- 172.21.15.98:0/2036814312 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f1bc80f0b60 con 0x7f1bd0102ef0 2024-09-19T08:13:01.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.002+0000 7f1bcffff640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f1bb80a33b0 0x7f1bb80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:01.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.002+0000 7f1bcffff640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f1bb80a33b0 0x7f1bb80a5870 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1bc000abe0 tx=0x7f1bc000a2d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:01.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.006+0000 7f1bcd7fa640 1 -- 172.21.15.98:0/2036814312 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f1bc80bd650 con 0x7f1bd0102ef0 2024-09-19T08:13:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.211+0000 7f1bd6946640 1 -- 172.21.15.98:0/2036814312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1bd01a4070 con 0x7f1bd0102ef0 2024-09-19T08:13:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.211+0000 7f1bcd7fa640 1 -- 172.21.15.98:0/2036814312 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11983 (secure 0 0 0) 0x7f1bc80bcf50 con 0x7f1bd0102ef0 2024-09-19T08:13:01.212 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:13:01.212 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"btime":"2024-09-19T08:08:26:985720+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:08:26.555658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:13:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2024-09-19T08:13:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.213+0000 7f1bd6946640 1 -- 172.21.15.98:0/2036814312 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f1bb80a33b0 msgr2=0x7f1bb80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.213+0000 7f1bd6946640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f1bb80a33b0 0x7f1bb80a5870 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1bc000abe0 tx=0x7f1bc000a2d0 comp rx=0 tx=0).stop 2024-09-19T08:13:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.213+0000 7f1bd6946640 1 -- 172.21.15.98:0/2036814312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0102ef0 msgr2=0x7f1bd019bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.213+0000 7f1bd6946640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0102ef0 0x7f1bd019bbf0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f1bc802ae70 tx=0x7f1bc802f5d0 comp rx=0 tx=0).stop 2024-09-19T08:13:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.214+0000 7f1bd6946640 1 -- 172.21.15.98:0/2036814312 shutdown_connections 2024-09-19T08:13:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.214+0000 7f1bd6946640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f1bb80a33b0 0x7f1bb80a5870 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.214+0000 7f1bd6946640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0102ef0 0x7f1bd019bbf0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.214+0000 7f1bd6946640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bd0102570 0x7f1bd0197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.214+0000 7f1bd6946640 1 --2- 172.21.15.98:0/2036814312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1bd0100e20 0x7f1bd0197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.214+0000 7f1bd6946640 1 -- 172.21.15.98:0/2036814312 >> 172.21.15.98:0/2036814312 conn(0x7f1bd00fc7d0 msgr2=0x7f1bd01067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.214+0000 7f1bd6946640 1 -- 172.21.15.98:0/2036814312 shutdown_connections 2024-09-19T08:13:01.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:01.214+0000 7f1bd6946640 1 -- 172.21.15.98:0/2036814312 wait complete. 2024-09-19T08:13:01.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:13:02.083 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:02.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:01 smithi098 ceph-mon[31333]: pgmap v309: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:02.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:01 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2036814312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:13:02.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:01 smithi162 ceph-mon[35260]: pgmap v309: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:02.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:01 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2036814312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:13:02.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:01 smithi139 ceph-mon[34874]: pgmap v309: 97 pgs: 97 active+clean; 7.3 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:02.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:01 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2036814312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:13:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.570+0000 7f2aef283640 1 -- 172.21.15.98:0/3362521698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae8106a80 msgr2=0x7f2ae8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.570+0000 7f2aef283640 1 --2- 172.21.15.98:0/3362521698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae8106a80 0x7f2ae8106e60 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009930 tx=0x7f2ae402a420 comp rx=0 tx=0).stop 2024-09-19T08:13:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.570+0000 7f2aef283640 1 -- 172.21.15.98:0/3362521698 shutdown_connections 2024-09-19T08:13:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.570+0000 7f2aef283640 1 --2- 172.21.15.98:0/3362521698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ae8101420 0x7f2ae810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.570+0000 7f2aef283640 1 --2- 172.21.15.98:0/3362521698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ae8100a80 0x7f2ae8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.570+0000 7f2aef283640 1 --2- 172.21.15.98:0/3362521698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae8106a80 0x7f2ae8106e60 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.570+0000 7f2aef283640 1 -- 172.21.15.98:0/3362521698 >> 172.21.15.98:0/3362521698 conn(0x7f2ae80fc7a0 msgr2=0x7f2ae80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:02.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.570+0000 7f2aef283640 1 -- 172.21.15.98:0/3362521698 shutdown_connections 2024-09-19T08:13:02.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.570+0000 7f2aef283640 1 -- 172.21.15.98:0/3362521698 wait complete. 2024-09-19T08:13:02.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.571+0000 7f2aef283640 1 Processor -- start 2024-09-19T08:13:02.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.571+0000 7f2aef283640 1 -- start start 2024-09-19T08:13:02.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.571+0000 7f2aef283640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ae8100a80 0x7f2ae8199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.571+0000 7f2aef283640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ae8101420 0x7f2ae8199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.571+0000 7f2aef283640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae8106a80 0x7f2ae81a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.571+0000 7f2aef283640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae8110e80 con 0x7f2ae8106a80 2024-09-19T08:13:02.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.571+0000 7f2aef283640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae8110d00 con 0x7f2ae8100a80 2024-09-19T08:13:02.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.571+0000 7f2aef283640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae8111000 con 0x7f2ae8101420 2024-09-19T08:13:02.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aed820640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae8106a80 0x7f2ae81a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aed820640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae8106a80 0x7f2ae81a4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42948/0 (socket says 172.21.15.98:42948) 2024-09-19T08:13:02.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aed820640 1 -- 172.21.15.98:0/1965795934 learned_addr learned my addr 172.21.15.98:0/1965795934 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:02.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aed01f640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ae8100a80 0x7f2ae8199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aec81e640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ae8101420 0x7f2ae8199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aed820640 1 -- 172.21.15.98:0/1965795934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ae8101420 msgr2=0x7f2ae8199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:02.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aed820640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ae8101420 0x7f2ae8199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aed820640 1 -- 172.21.15.98:0/1965795934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ae8100a80 msgr2=0x7f2ae8199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:02.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aed820640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ae8100a80 0x7f2ae8199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aed820640 1 -- 172.21.15.98:0/1965795934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae81a48d0 con 0x7f2ae8106a80 2024-09-19T08:13:02.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aed01f640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ae8100a80 0x7f2ae8199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:02.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aec81e640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ae8101420 0x7f2ae8199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:02.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aed820640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae8106a80 0x7f2ae81a4390 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f2ae000eaa0 tx=0x7f2ae000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:02.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2ade7fc640 1 -- 172.21.15.98:0/1965795934 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae000cdb0 con 0x7f2ae8106a80 2024-09-19T08:13:02.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aef283640 1 -- 172.21.15.98:0/1965795934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae81a4aa0 con 0x7f2ae8106a80 2024-09-19T08:13:02.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2ade7fc640 1 -- 172.21.15.98:0/1965795934 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ae0005020 con 0x7f2ae8106a80 2024-09-19T08:13:02.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2ade7fc640 1 -- 172.21.15.98:0/1965795934 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae00116b0 con 0x7f2ae8106a80 2024-09-19T08:13:02.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.572+0000 7f2aef283640 1 -- 172.21.15.98:0/1965795934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae81a4fb0 con 0x7f2ae8106a80 2024-09-19T08:13:02.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.574+0000 7f2ade7fc640 1 -- 172.21.15.98:0/1965795934 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f2ae0005560 con 0x7f2ae8106a80 2024-09-19T08:13:02.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.575+0000 7f2af165c740 1 Processor -- start 2024-09-19T08:13:02.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.575+0000 7f2af165c740 1 -- start start 2024-09-19T08:13:02.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.575+0000 7f2af165c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc24682b40 0x55cc24681090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.575+0000 7f2af165c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc246815d0 0x55cc24681a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.575+0000 7f2af165c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc2468c370 0x55cc2468e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.575+0000 7f2af165c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cc245e72f0 con 0x55cc2468c370 2024-09-19T08:13:02.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.575+0000 7f2af165c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cc2458ecf0 con 0x55cc24682b40 2024-09-19T08:13:02.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.575+0000 7f2af165c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cc245e32b0 con 0x55cc246815d0 2024-09-19T08:13:02.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.575+0000 7f2ade7fc640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2ad40a3430 0x7f2ad40a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.575+0000 7f2aec81e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc2468c370 0x55cc2468e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.575+0000 7f2aec81e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc2468c370 0x55cc2468e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42950/0 (socket says 172.21.15.98:42950) 2024-09-19T08:13:02.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.575+0000 7f2aec81e640 1 -- 172.21.15.98:0/3353444420 learned_addr learned my addr 172.21.15.98:0/3353444420 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:02.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.576+0000 7f2aed820640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2ad40a3430 0x7f2ad40a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.576+0000 7f2aec81e640 1 --2- 172.21.15.98:0/3353444420 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc24682b40 0x55cc24681090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.576+0000 7f2aed01f640 1 --2- 172.21.15.98:0/3353444420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc246815d0 0x55cc24681a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.576+0000 7f2ade7fc640 1 -- 172.21.15.98:0/1965795934 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f2ae00cff30 con 0x7f2ae8106a80 2024-09-19T08:13:02.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.576+0000 7f2aec81e640 1 -- 172.21.15.98:0/3353444420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc246815d0 msgr2=0x55cc24681a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:02.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.576+0000 7f2aec81e640 1 --2- 172.21.15.98:0/3353444420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc246815d0 0x55cc24681a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.576+0000 7f2aec81e640 1 -- 172.21.15.98:0/3353444420 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc24682b40 msgr2=0x55cc24681090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:02.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.576+0000 7f2aec81e640 1 --2- 172.21.15.98:0/3353444420 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc24682b40 0x55cc24681090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.576+0000 7f2aec81e640 1 -- 172.21.15.98:0/3353444420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc245733a0 con 0x55cc2468c370 2024-09-19T08:13:02.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.576+0000 7f2aed820640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2ad40a3430 0x7f2ad40a58f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00cd730 tx=0x7f2ae006df90 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:02.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.576+0000 7f2aec81e640 1 --2- 172.21.15.98:0/3353444420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc2468c370 0x55cc2468e760 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8014a30 tx=0x7f2ad800c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c2fb6271016021 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:02.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.576+0000 7f2ad37fe640 1 -- 172.21.15.98:0/3353444420 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad80142d0 con 0x55cc2468c370 2024-09-19T08:13:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.576+0000 7f2ad37fe640 1 -- 172.21.15.98:0/3353444420 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ad8005500 con 0x55cc2468c370 2024-09-19T08:13:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.576+0000 7f2ad37fe640 1 -- 172.21.15.98:0/3353444420 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad80047a0 con 0x55cc2468c370 2024-09-19T08:13:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.577+0000 7f2af165c740 1 -- 172.21.15.98:0/3353444420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc2468c370 msgr2=0x55cc2468e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.577+0000 7f2af165c740 1 --2- 172.21.15.98:0/3353444420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc2468c370 0x55cc2468e760 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8014a30 tx=0x7f2ad800c570 comp rx=0 tx=0).stop 2024-09-19T08:13:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.577+0000 7f2af165c740 1 -- 172.21.15.98:0/3353444420 shutdown_connections 2024-09-19T08:13:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.577+0000 7f2af165c740 1 --2- 172.21.15.98:0/3353444420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc2468c370 0x55cc2468e760 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.577+0000 7f2af165c740 1 --2- 172.21.15.98:0/3353444420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc246815d0 0x55cc24681a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.577+0000 7f2af165c740 1 --2- 172.21.15.98:0/3353444420 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc24682b40 0x55cc24681090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.577+0000 7f2af165c740 1 -- 172.21.15.98:0/3353444420 >> 172.21.15.98:0/3353444420 conn(0x55cc2464a310 msgr2=0x55cc2467dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.577+0000 7f2af165c740 1 -- 172.21.15.98:0/3353444420 shutdown_connections 2024-09-19T08:13:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.577+0000 7f2af165c740 1 -- 172.21.15.98:0/3353444420 wait complete. 2024-09-19T08:13:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.577+0000 7f2af165c740 1 -- start start 2024-09-19T08:13:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.577+0000 7f2af165c740 1 Processor -- start 2024-09-19T08:13:02.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2af165c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc246815d0 0x55cc2468bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2af165c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc24682b40 0x55cc24688fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2af165c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc2468c370 0x55cc24689570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2af165c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cc2468f010 con 0x55cc24682b40 2024-09-19T08:13:02.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2af165c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cc245a2060 con 0x55cc246815d0 2024-09-19T08:13:02.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2af165c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cc2468f190 con 0x55cc2468c370 2024-09-19T08:13:02.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2af165c740 10 client.0 _ll_register_callbacks cb 0x55cc242f58c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:02.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2aed01f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc24682b40 0x55cc24688fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2aed01f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc24682b40 0x55cc24688fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42954/0 (socket says 172.21.15.98:42954) 2024-09-19T08:13:02.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2aed01f640 1 -- 172.21.15.98:0/2417159890 learned_addr learned my addr 172.21.15.98:0/2417159890 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:02.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2aec81e640 1 --2- 172.21.15.98:0/2417159890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc246815d0 0x55cc2468bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2aec81e640 1 --2- 172.21.15.98:0/2417159890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc2468c370 0x55cc24689570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2aed01f640 1 -- 172.21.15.98:0/2417159890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc2468c370 msgr2=0x55cc24689570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2aed01f640 1 --2- 172.21.15.98:0/2417159890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc2468c370 0x55cc24689570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2aed01f640 1 -- 172.21.15.98:0/2417159890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc246815d0 msgr2=0x55cc2468bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2aed01f640 1 --2- 172.21.15.98:0/2417159890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc246815d0 0x55cc2468bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2aed01f640 1 -- 172.21.15.98:0/2417159890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc243583c0 con 0x55cc24682b40 2024-09-19T08:13:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2aec81e640 1 --2- 172.21.15.98:0/2417159890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc246815d0 0x55cc2468bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2aed01f640 1 --2- 172.21.15.98:0/2417159890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc24682b40 0x55cc24688fd0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009930 tx=0x7f2ae4031860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa51eac246d857a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.578+0000 7f2ad0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.579+0000 7f2ad0ff9640 1 -- 172.21.15.98:0/2417159890 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4035040 con 0x55cc24682b40 2024-09-19T08:13:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.579+0000 7f2ad0ff9640 1 -- 172.21.15.98:0/2417159890 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f2ae4004240 con 0x55cc24682b40 2024-09-19T08:13:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.579+0000 7f2af165c740 1 -- 172.21.15.98:0/2417159890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ad8005500 con 0x55cc24682b40 2024-09-19T08:13:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.579+0000 7f2ad0ff9640 1 -- 172.21.15.98:0/2417159890 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2ae4004440 con 0x55cc24682b40 2024-09-19T08:13:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.579+0000 7f2af165c740 10 client.17730 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.579+0000 7f2af165c740 1 -- 172.21.15.98:0/2417159890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cc2469aa90 con 0x55cc24682b40 2024-09-19T08:13:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.579+0000 7f2ad0ff9640 1 -- 172.21.15.98:0/2417159890 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ad8005500 con 0x55cc24682b40 2024-09-19T08:13:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.579+0000 7f2ad0ff9640 1 -- 172.21.15.98:0/2417159890 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2ae4004b50 con 0x55cc24682b40 2024-09-19T08:13:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.579+0000 7f2af165c740 10 client.17730 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.579+0000 7f2af165c740 10 client.17730 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:13:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.579+0000 7f2af165c740 20 client.17730 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.579+0000 7f2af165c740 1 --2- 172.21.15.98:0/2417159890 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55cc2469bef0 0x55cc2469e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.579+0000 7f2af165c740 4 client.17730 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:13:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.579+0000 7f2af165c740 1 -- 172.21.15.98:0/2417159890 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc245f4860 con 0x55cc2469bef0 2024-09-19T08:13:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.580+0000 7f2aec81e640 1 --2- 172.21.15.98:0/2417159890 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55cc2469bef0 0x55cc2469e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.580+0000 7f2aec81e640 1 --2- 172.21.15.98:0/2417159890 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55cc2469bef0 0x55cc2469e310 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7cbc75b07eb51eb6 server_cookie=ba5f1e988c29adf1 in_seq=0 out_seq=0 2024-09-19T08:13:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.580+0000 7f2ad0ff9640 10 client.17730 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:13:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2ad0ff9640 1 -- 172.21.15.98:0/2417159890 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cc245f4860 con 0x55cc2469bef0 2024-09-19T08:13:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2ad0ff9640 10 client.17730 handle_command_reply: tid=0 2024-09-19T08:13:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2af165c740 1 client.17730 shutdown 2024-09-19T08:13:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2af165c740 1 -- 172.21.15.98:0/2417159890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc24682b40 msgr2=0x55cc24688fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2af165c740 1 --2- 172.21.15.98:0/2417159890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc24682b40 0x55cc24688fd0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009930 tx=0x7f2ae4031860 comp rx=0 tx=0).stop 2024-09-19T08:13:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2af165c740 1 -- 172.21.15.98:0/2417159890 shutdown_connections 2024-09-19T08:13:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2af165c740 1 --2- 172.21.15.98:0/2417159890 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55cc2469bef0 0x55cc2469e310 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2af165c740 1 --2- 172.21.15.98:0/2417159890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc2468c370 0x55cc24689570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2af165c740 1 --2- 172.21.15.98:0/2417159890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc24682b40 0x55cc24688fd0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2af165c740 1 --2- 172.21.15.98:0/2417159890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc246815d0 0x55cc2468bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2ad0ff9640 0 client.17730 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:13:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2af165c740 1 -- 172.21.15.98:0/2417159890 >> 172.21.15.98:0/2417159890 conn(0x55cc2464a310 msgr2=0x55cc24683960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:02.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2af165c740 1 -- 172.21.15.98:0/2417159890 shutdown_connections 2024-09-19T08:13:02.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2af165c740 1 -- 172.21.15.98:0/2417159890 wait complete. 2024-09-19T08:13:02.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.581+0000 7f2af165c740 20 client.17730 trim_cache size 0 max 16384 2024-09-19T08:13:02.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2af165c740 1 Processor -- start 2024-09-19T08:13:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2af165c740 1 -- start start 2024-09-19T08:13:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2af165c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc246815d0 0x55cc24688fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2af165c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc24682b40 0x55cc246982e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2af165c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc2468c370 0x55cc24694830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2af165c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae81ad910 con 0x55cc24682b40 2024-09-19T08:13:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2af165c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae81af450 con 0x55cc246815d0 2024-09-19T08:13:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2af165c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae81adfe0 con 0x55cc2468c370 2024-09-19T08:13:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2aed01f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc24682b40 0x55cc246982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2aed01f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc24682b40 0x55cc246982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42958/0 (socket says 172.21.15.98:42958) 2024-09-19T08:13:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2aed01f640 1 -- 172.21.15.98:0/410435288 learned_addr learned my addr 172.21.15.98:0/410435288 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2aec81e640 1 --2- 172.21.15.98:0/410435288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc2468c370 0x55cc24694830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2aec81e640 1 --2- 172.21.15.98:0/410435288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc246815d0 0x55cc24688fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2aed01f640 1 -- 172.21.15.98:0/410435288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc2468c370 msgr2=0x55cc24694830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:02.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2aed01f640 1 --2- 172.21.15.98:0/410435288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc2468c370 0x55cc24694830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2aed01f640 1 -- 172.21.15.98:0/410435288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc246815d0 msgr2=0x55cc24688fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:02.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2aed01f640 1 --2- 172.21.15.98:0/410435288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc246815d0 0x55cc24688fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.594+0000 7f2aed01f640 1 -- 172.21.15.98:0/410435288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc24682760 con 0x55cc24682b40 2024-09-19T08:13:02.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.595+0000 7f2aec81e640 1 --2- 172.21.15.98:0/410435288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc246815d0 0x55cc24688fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:02.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.595+0000 7f2aed01f640 1 --2- 172.21.15.98:0/410435288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc24682b40 0x55cc246982e0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4004f20 tx=0x7f2ae40317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c88cfd271e4f807a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:02.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.595+0000 7f2ad2ffd640 1 -- 172.21.15.98:0/410435288 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4000e60 con 0x55cc24682b40 2024-09-19T08:13:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.595+0000 7f2ad2ffd640 1 -- 172.21.15.98:0/410435288 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ae40046c0 con 0x55cc24682b40 2024-09-19T08:13:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.595+0000 7f2ad2ffd640 1 -- 172.21.15.98:0/410435288 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4035070 con 0x55cc24682b40 2024-09-19T08:13:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.595+0000 7f2af165c740 1 -- 172.21.15.98:0/410435288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc24682b40 msgr2=0x55cc246982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.595+0000 7f2af165c740 1 --2- 172.21.15.98:0/410435288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc24682b40 0x55cc246982e0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4004f20 tx=0x7f2ae40317c0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.595+0000 7f2af165c740 1 -- 172.21.15.98:0/410435288 shutdown_connections 2024-09-19T08:13:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 1 --2- 172.21.15.98:0/410435288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc2468c370 0x55cc24694830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 1 --2- 172.21.15.98:0/410435288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc24682b40 0x55cc246982e0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 1 --2- 172.21.15.98:0/410435288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc246815d0 0x55cc24688fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 1 -- 172.21.15.98:0/410435288 >> 172.21.15.98:0/410435288 conn(0x55cc2464a310 msgr2=0x55cc24688890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 1 -- 172.21.15.98:0/410435288 shutdown_connections 2024-09-19T08:13:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 1 -- 172.21.15.98:0/410435288 wait complete. 2024-09-19T08:13:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 1 -- start start 2024-09-19T08:13:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 1 Processor -- start 2024-09-19T08:13:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc246815d0 0x55cc246feb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc24682b40 0x55cc246ff0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc2468c370 0x55cc24709bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae81aed80 con 0x55cc2468c370 2024-09-19T08:13:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae81ae6b0 con 0x55cc24682b40 2024-09-19T08:13:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae81ab910 con 0x55cc246815d0 2024-09-19T08:13:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2af165c740 10 client.0 _ll_register_callbacks cb 0x7f2ae4005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.596+0000 7f2aec81e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc2468c370 0x55cc24709bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2aec81e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc2468c370 0x55cc24709bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42966/0 (socket says 172.21.15.98:42966) 2024-09-19T08:13:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2aec81e640 1 -- 172.21.15.98:0/443631610 learned_addr learned my addr 172.21.15.98:0/443631610 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2aed01f640 1 --2- 172.21.15.98:0/443631610 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc24682b40 0x55cc246ff0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2aec81e640 1 --2- 172.21.15.98:0/443631610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc246815d0 0x55cc246feb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2aec81e640 1 -- 172.21.15.98:0/443631610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc246815d0 msgr2=0x55cc246feb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2aec81e640 1 --2- 172.21.15.98:0/443631610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc246815d0 0x55cc246feb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2aec81e640 1 -- 172.21.15.98:0/443631610 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc24682b40 msgr2=0x55cc246ff0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:02.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2aec81e640 1 --2- 172.21.15.98:0/443631610 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc24682b40 0x55cc246ff0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2aec81e640 1 -- 172.21.15.98:0/443631610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc246ff940 con 0x55cc2468c370 2024-09-19T08:13:02.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2aec81e640 1 --2- 172.21.15.98:0/443631610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc2468c370 0x55cc24709bf0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x55cc24682070 tx=0x7f2ae4009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f8499b95728bc1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:02.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2ad0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:02.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2ad0ff9640 1 -- 172.21.15.98:0/443631610 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad802a040 con 0x55cc2468c370 2024-09-19T08:13:02.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2ad0ff9640 1 -- 172.21.15.98:0/443631610 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f2ad800ce00 con 0x55cc2468c370 2024-09-19T08:13:02.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2af165c740 1 -- 172.21.15.98:0/443631610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ae40046c0 con 0x55cc2468c370 2024-09-19T08:13:02.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2ad0ff9640 1 -- 172.21.15.98:0/443631610 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2ad800fd80 con 0x55cc2468c370 2024-09-19T08:13:02.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2af165c740 10 client.17736 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:02.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2af165c740 1 -- 172.21.15.98:0/443631610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cc2470b170 con 0x55cc2468c370 2024-09-19T08:13:02.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.597+0000 7f2ad0ff9640 1 -- 172.21.15.98:0/443631610 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ae40046c0 con 0x55cc2468c370 2024-09-19T08:13:02.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.598+0000 7f2ad0ff9640 1 -- 172.21.15.98:0/443631610 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2ad8004680 con 0x55cc2468c370 2024-09-19T08:13:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.598+0000 7f2af165c740 10 client.17736 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.598+0000 7f2af165c740 10 client.17736 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:13:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.598+0000 7f2af165c740 20 client.17736 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.598+0000 7f2af165c740 1 --2- 172.21.15.98:0/443631610 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55cc2470b690 0x55cc2470da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.598+0000 7f2af165c740 4 client.17736 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:13:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.598+0000 7f2af165c740 1 -- 172.21.15.98:0/443631610 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cc245eed70 con 0x55cc2470b690 2024-09-19T08:13:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.598+0000 7f2aed01f640 1 --2- 172.21.15.98:0/443631610 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55cc2470b690 0x55cc2470da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.598+0000 7f2ad0ff9640 10 client.17736 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:13:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.598+0000 7f2aed01f640 1 --2- 172.21.15.98:0/443631610 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55cc2470b690 0x55cc2470da80 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f94b539f9b94de1a server_cookie=500fb02c2bb0c970 in_seq=0 out_seq=0 2024-09-19T08:13:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.598+0000 7f2ad0ff9640 1 -- 172.21.15.98:0/443631610 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cc245eed70 con 0x55cc2470b690 2024-09-19T08:13:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.598+0000 7f2ad0ff9640 10 client.17736 handle_command_reply: tid=0 2024-09-19T08:13:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.598+0000 7f2af165c740 1 client.17736 shutdown 2024-09-19T08:13:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2af165c740 1 -- 172.21.15.98:0/443631610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc2468c370 msgr2=0x55cc24709bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2af165c740 1 --2- 172.21.15.98:0/443631610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc2468c370 0x55cc24709bf0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x55cc24682070 tx=0x7f2ae4009630 comp rx=0 tx=0).stop 2024-09-19T08:13:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2af165c740 1 -- 172.21.15.98:0/443631610 shutdown_connections 2024-09-19T08:13:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2af165c740 1 --2- 172.21.15.98:0/443631610 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55cc2470b690 0x55cc2470da80 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2af165c740 1 --2- 172.21.15.98:0/443631610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc2468c370 0x55cc24709bf0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2af165c740 1 --2- 172.21.15.98:0/443631610 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cc24682b40 0x55cc246ff0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2af165c740 1 --2- 172.21.15.98:0/443631610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc246815d0 0x55cc246feb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2af165c740 1 -- 172.21.15.98:0/443631610 >> 172.21.15.98:0/443631610 conn(0x55cc2464a310 msgr2=0x55cc2468a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2ad0ff9640 0 client.17736 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:13:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2af165c740 1 -- 172.21.15.98:0/443631610 shutdown_connections 2024-09-19T08:13:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2af165c740 1 -- 172.21.15.98:0/443631610 wait complete. 2024-09-19T08:13:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2af165c740 20 client.17736 trim_cache size 0 max 16384 2024-09-19T08:13:02.610 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:13:02.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2aef283640 1 -- 172.21.15.98:0/1965795934 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2ad40a3430 msgr2=0x7f2ad40a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:02.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2aef283640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2ad40a3430 0x7f2ad40a58f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00cd730 tx=0x7f2ae006df90 comp rx=0 tx=0).stop 2024-09-19T08:13:02.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2aef283640 1 -- 172.21.15.98:0/1965795934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae8106a80 msgr2=0x7f2ae81a4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:02.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2aef283640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae8106a80 0x7f2ae81a4390 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f2ae000eaa0 tx=0x7f2ae000ef60 comp rx=0 tx=0).stop 2024-09-19T08:13:02.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2aef283640 1 -- 172.21.15.98:0/1965795934 shutdown_connections 2024-09-19T08:13:02.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2aef283640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2ad40a3430 0x7f2ad40a58f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2aef283640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae8106a80 0x7f2ae81a4390 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2aef283640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ae8101420 0x7f2ae8199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.599+0000 7f2aef283640 1 --2- 172.21.15.98:0/1965795934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ae8100a80 0x7f2ae8199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:02.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.600+0000 7f2aef283640 1 -- 172.21.15.98:0/1965795934 >> 172.21.15.98:0/1965795934 conn(0x7f2ae80fc7a0 msgr2=0x7f2ae80fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:02.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.600+0000 7f2aef283640 1 -- 172.21.15.98:0/1965795934 shutdown_connections 2024-09-19T08:13:02.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:02.600+0000 7f2aef283640 1 -- 172.21.15.98:0/1965795934 wait complete. 2024-09-19T08:13:03.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:13:03.561 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:04.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.059+0000 7ffb947ab640 1 -- 172.21.15.98:0/2264044240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c100e20 msgr2=0x7ffb8c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:04.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.059+0000 7ffb947ab640 1 --2- 172.21.15.98:0/2264044240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c100e20 0x7ffb8c101200 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7ffb84009900 tx=0x7ffb8402a480 comp rx=0 tx=0).stop 2024-09-19T08:13:04.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 -- 172.21.15.98:0/2264044240 shutdown_connections 2024-09-19T08:13:04.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 --2- 172.21.15.98:0/2264044240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb8c102ef0 0x7ffb8c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 --2- 172.21.15.98:0/2264044240 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffb8c102570 0x7ffb8c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 --2- 172.21.15.98:0/2264044240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c100e20 0x7ffb8c101200 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 -- 172.21.15.98:0/2264044240 >> 172.21.15.98:0/2264044240 conn(0x7ffb8c0fc7d0 msgr2=0x7ffb8c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:04.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 -- 172.21.15.98:0/2264044240 shutdown_connections 2024-09-19T08:13:04.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 -- 172.21.15.98:0/2264044240 wait complete. 2024-09-19T08:13:04.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 Processor -- start 2024-09-19T08:13:04.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 -- start start 2024-09-19T08:13:04.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb8c100e20 0x7ffb8c199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffb8c102570 0x7ffb8c199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c102ef0 0x7ffb8c1a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c110fa0 con 0x7ffb8c102ef0 2024-09-19T08:13:04.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c110e20 con 0x7ffb8c102570 2024-09-19T08:13:04.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb947ab640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c111120 con 0x7ffb8c100e20 2024-09-19T08:13:04.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.060+0000 7ffb92d48640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c102ef0 0x7ffb8c1a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb92d48640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c102ef0 0x7ffb8c1a43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42982/0 (socket says 172.21.15.98:42982) 2024-09-19T08:13:04.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb92d48640 1 -- 172.21.15.98:0/203859125 learned_addr learned my addr 172.21.15.98:0/203859125 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:04.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb91d46640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffb8c102570 0x7ffb8c199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb92547640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb8c100e20 0x7ffb8c199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb92d48640 1 -- 172.21.15.98:0/203859125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb8c100e20 msgr2=0x7ffb8c199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:04.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb92d48640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb8c100e20 0x7ffb8c199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb92d48640 1 -- 172.21.15.98:0/203859125 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffb8c102570 msgr2=0x7ffb8c199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:04.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb92d48640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffb8c102570 0x7ffb8c199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb92d48640 1 -- 172.21.15.98:0/203859125 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb8c1a48f0 con 0x7ffb8c102ef0 2024-09-19T08:13:04.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb92547640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb8c100e20 0x7ffb8c199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:04.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb92d48640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c102ef0 0x7ffb8c1a43b0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00b970 tx=0x7ffb7c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:04.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb8b7fe640 1 -- 172.21.15.98:0/203859125 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c019070 con 0x7ffb8c102ef0 2024-09-19T08:13:04.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb8b7fe640 1 -- 172.21.15.98:0/203859125 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb7c014440 con 0x7ffb8c102ef0 2024-09-19T08:13:04.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb947ab640 1 -- 172.21.15.98:0/203859125 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb8c1a4ac0 con 0x7ffb8c102ef0 2024-09-19T08:13:04.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb8b7fe640 1 -- 172.21.15.98:0/203859125 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c0075d0 con 0x7ffb8c102ef0 2024-09-19T08:13:04.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.061+0000 7ffb947ab640 1 -- 172.21.15.98:0/203859125 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb8c1a4f80 con 0x7ffb8c102ef0 2024-09-19T08:13:04.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.063+0000 7ffb8b7fe640 1 -- 172.21.15.98:0/203859125 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7ffb7c01c070 con 0x7ffb8c102ef0 2024-09-19T08:13:04.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.064+0000 7ffb96b84740 1 Processor -- start 2024-09-19T08:13:04.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.064+0000 7ffb96b84740 1 -- start start 2024-09-19T08:13:04.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.064+0000 7ffb96b84740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fbf9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.064+0000 7ffb96b84740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fbf95d0 0x563b5fbf9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.064+0000 7ffb96b84740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 0x563b5fc06760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.064+0000 7ffb96b84740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563b5fb5f2f0 con 0x563b5fbfab40 2024-09-19T08:13:04.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.064+0000 7ffb96b84740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563b5fb06cf0 con 0x563b5fbf95d0 2024-09-19T08:13:04.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.064+0000 7ffb96b84740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563b5fb5b2b0 con 0x563b5fc04370 2024-09-19T08:13:04.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.064+0000 7ffb91d46640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fbf9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.064+0000 7ffb91d46640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fbf9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42984/0 (socket says 172.21.15.98:42984) 2024-09-19T08:13:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb91d46640 1 -- 172.21.15.98:0/3305224617 learned_addr learned my addr 172.21.15.98:0/3305224617 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb92547640 1 --2- 172.21.15.98:0/3305224617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fbf95d0 0x563b5fbf9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb91d46640 1 --2- 172.21.15.98:0/3305224617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 0x563b5fc06760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb8b7fe640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ffb740a33b0 0x7ffb740a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb91d46640 1 -- 172.21.15.98:0/3305224617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 msgr2=0x563b5fc06760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb91d46640 1 --2- 172.21.15.98:0/3305224617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 0x563b5fc06760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb91d46640 1 -- 172.21.15.98:0/3305224617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fbf95d0 msgr2=0x563b5fbf9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb91d46640 1 --2- 172.21.15.98:0/3305224617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fbf95d0 0x563b5fbf9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb91d46640 1 -- 172.21.15.98:0/3305224617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b5faeb3a0 con 0x563b5fbfab40 2024-09-19T08:13:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb92d48640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ffb740a33b0 0x7ffb740a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb91d46640 1 --2- 172.21.15.98:0/3305224617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fbf9090 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7ffb80013560 tx=0x7ffb8000a800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dfff045e3f10fb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb8b7fe640 1 -- 172.21.15.98:0/203859125 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7ffb7c0cf2a0 con 0x7ffb8c102ef0 2024-09-19T08:13:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb88ff9640 1 -- 172.21.15.98:0/3305224617 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb80019030 con 0x563b5fbfab40 2024-09-19T08:13:04.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb88ff9640 1 -- 172.21.15.98:0/3305224617 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb800110d0 con 0x563b5fbfab40 2024-09-19T08:13:04.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.065+0000 7ffb88ff9640 1 -- 172.21.15.98:0/3305224617 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8000f7f0 con 0x563b5fbfab40 2024-09-19T08:13:04.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.066+0000 7ffb96b84740 1 -- 172.21.15.98:0/3305224617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 msgr2=0x563b5fbf9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:04.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.066+0000 7ffb92d48640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ffb740a33b0 0x7ffb740a5870 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c09b330 tx=0x7ffb7c06d3b0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:04.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.066+0000 7ffb96b84740 1 --2- 172.21.15.98:0/3305224617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fbf9090 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7ffb80013560 tx=0x7ffb8000a800 comp rx=0 tx=0).stop 2024-09-19T08:13:04.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.066+0000 7ffb96b84740 1 -- 172.21.15.98:0/3305224617 shutdown_connections 2024-09-19T08:13:04.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.066+0000 7ffb96b84740 1 --2- 172.21.15.98:0/3305224617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 0x563b5fc06760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.066+0000 7ffb96b84740 1 --2- 172.21.15.98:0/3305224617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fbf95d0 0x563b5fbf9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.066+0000 7ffb96b84740 1 --2- 172.21.15.98:0/3305224617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fbf9090 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.066+0000 7ffb96b84740 1 -- 172.21.15.98:0/3305224617 >> 172.21.15.98:0/3305224617 conn(0x563b5fbc2310 msgr2=0x563b5fbf5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:04.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.066+0000 7ffb96b84740 1 -- 172.21.15.98:0/3305224617 shutdown_connections 2024-09-19T08:13:04.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.066+0000 7ffb96b84740 1 -- 172.21.15.98:0/3305224617 wait complete. 2024-09-19T08:13:04.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.066+0000 7ffb96b84740 1 -- start start 2024-09-19T08:13:04.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.066+0000 7ffb96b84740 1 Processor -- start 2024-09-19T08:13:04.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb96b84740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fbf95d0 0x563b5fc03d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb96b84740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fc00fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb96b84740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fc04370 0x563b5fc01570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb96b84740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563b5fc07070 con 0x563b5fbfab40 2024-09-19T08:13:04.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb96b84740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563b5fb1a060 con 0x563b5fc04370 2024-09-19T08:13:04.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb96b84740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563b5fc071f0 con 0x563b5fbf95d0 2024-09-19T08:13:04.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb96b84740 10 client.0 _ll_register_callbacks cb 0x563b5f86d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:04.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb92547640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fc00fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb92547640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fc00fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42996/0 (socket says 172.21.15.98:42996) 2024-09-19T08:13:04.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb92547640 1 -- 172.21.15.98:0/689686859 learned_addr learned my addr 172.21.15.98:0/689686859 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:04.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb91d46640 1 --2- 172.21.15.98:0/689686859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fbf95d0 0x563b5fc03d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb91d46640 1 --2- 172.21.15.98:0/689686859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fc04370 0x563b5fc01570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb92547640 1 -- 172.21.15.98:0/689686859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fbf95d0 msgr2=0x563b5fc03d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb92547640 1 --2- 172.21.15.98:0/689686859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fbf95d0 0x563b5fc03d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb92547640 1 -- 172.21.15.98:0/689686859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fc04370 msgr2=0x563b5fc01570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb92547640 1 --2- 172.21.15.98:0/689686859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fc04370 0x563b5fc01570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb92547640 1 -- 172.21.15.98:0/689686859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b5f8d03c0 con 0x563b5fbfab40 2024-09-19T08:13:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb91d46640 1 --2- 172.21.15.98:0/689686859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fc04370 0x563b5fc01570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb92547640 1 --2- 172.21.15.98:0/689686859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fc00fd0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ffb8402a9a0 tx=0x7ffb840044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a844067b75c7fb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb71ffb640 1 -- 172.21.15.98:0/689686859 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb84035040 con 0x563b5fbfab40 2024-09-19T08:13:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb71ffb640 1 -- 172.21.15.98:0/689686859 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7ffb84003b40 con 0x563b5fbfab40 2024-09-19T08:13:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.067+0000 7ffb96b84740 1 -- 172.21.15.98:0/689686859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb800110d0 con 0x563b5fbfab40 2024-09-19T08:13:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.068+0000 7ffb71ffb640 1 -- 172.21.15.98:0/689686859 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffb84003d40 con 0x563b5fbfab40 2024-09-19T08:13:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.068+0000 7ffb96b84740 10 client.17748 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.068+0000 7ffb96b84740 1 -- 172.21.15.98:0/689686859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563b5fc12aa0 con 0x563b5fbfab40 2024-09-19T08:13:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.068+0000 7ffb71ffb640 1 -- 172.21.15.98:0/689686859 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb800110d0 con 0x563b5fbfab40 2024-09-19T08:13:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.068+0000 7ffb71ffb640 1 -- 172.21.15.98:0/689686859 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ffb8404e6b0 con 0x563b5fbfab40 2024-09-19T08:13:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.068+0000 7ffb96b84740 10 client.17748 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.068+0000 7ffb96b84740 10 client.17748 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:13:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.068+0000 7ffb96b84740 20 client.17748 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.068+0000 7ffb96b84740 1 --2- 172.21.15.98:0/689686859 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x563b5fc13eb0 0x563b5fc162f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.068+0000 7ffb96b84740 4 client.17748 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:13:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.068+0000 7ffb96b84740 1 -- 172.21.15.98:0/689686859 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b5fb6c860 con 0x563b5fc13eb0 2024-09-19T08:13:04.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.069+0000 7ffb91d46640 1 --2- 172.21.15.98:0/689686859 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x563b5fc13eb0 0x563b5fc162f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.069+0000 7ffb91d46640 1 --2- 172.21.15.98:0/689686859 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x563b5fc13eb0 0x563b5fc162f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e4c6df9365e8964b server_cookie=4f33285b12719324 in_seq=0 out_seq=0 2024-09-19T08:13:04.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.069+0000 7ffb71ffb640 10 client.17748 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:13:04.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb71ffb640 1 -- 172.21.15.98:0/689686859 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563b5fb6c860 con 0x563b5fc13eb0 2024-09-19T08:13:04.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb71ffb640 10 client.17748 handle_command_reply: tid=0 2024-09-19T08:13:04.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb96b84740 1 client.17748 shutdown 2024-09-19T08:13:04.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb96b84740 1 -- 172.21.15.98:0/689686859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 msgr2=0x563b5fc00fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:04.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb96b84740 1 --2- 172.21.15.98:0/689686859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fc00fd0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ffb8402a9a0 tx=0x7ffb840044d0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb96b84740 1 -- 172.21.15.98:0/689686859 shutdown_connections 2024-09-19T08:13:04.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb96b84740 1 --2- 172.21.15.98:0/689686859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fc04370 0x563b5fc01570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb96b84740 1 --2- 172.21.15.98:0/689686859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fc00fd0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb96b84740 1 --2- 172.21.15.98:0/689686859 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x563b5fc13eb0 0x563b5fc162f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb96b84740 1 --2- 172.21.15.98:0/689686859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fbf95d0 0x563b5fc03d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb96b84740 1 -- 172.21.15.98:0/689686859 >> 172.21.15.98:0/689686859 conn(0x563b5fbc2310 msgr2=0x563b5fbfb930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:04.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb71ffb640 0 client.17748 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:13:04.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb96b84740 1 -- 172.21.15.98:0/689686859 shutdown_connections 2024-09-19T08:13:04.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb96b84740 1 -- 172.21.15.98:0/689686859 wait complete. 2024-09-19T08:13:04.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.070+0000 7ffb96b84740 20 client.17748 trim_cache size 0 max 16384 2024-09-19T08:13:04.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb96b84740 1 Processor -- start 2024-09-19T08:13:04.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb96b84740 1 -- start start 2024-09-19T08:13:04.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb96b84740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbf95d0 0x563b5fc10310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb96b84740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fbfab40 0x563b5fc1bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb96b84740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 0x563b5fc1c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb96b84740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c1ad840 con 0x563b5fbf95d0 2024-09-19T08:13:04.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb96b84740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c1af380 con 0x563b5fbfab40 2024-09-19T08:13:04.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb96b84740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c1adf10 con 0x563b5fc04370 2024-09-19T08:13:04.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb91d46640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbf95d0 0x563b5fc10310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb91d46640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbf95d0 0x563b5fc10310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43006/0 (socket says 172.21.15.98:43006) 2024-09-19T08:13:04.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb91d46640 1 -- 172.21.15.98:0/3190100423 learned_addr learned my addr 172.21.15.98:0/3190100423 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:04.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb91d46640 1 -- 172.21.15.98:0/3190100423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 msgr2=0x563b5fc1c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:04.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb91d46640 1 --2- 172.21.15.98:0/3190100423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 0x563b5fc1c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb91d46640 1 -- 172.21.15.98:0/3190100423 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fbfab40 msgr2=0x563b5fc1bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:04.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb91d46640 1 --2- 172.21.15.98:0/3190100423 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fbfab40 0x563b5fc1bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.083+0000 7ffb91d46640 1 -- 172.21.15.98:0/3190100423 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b5fbf6960 con 0x563b5fbf95d0 2024-09-19T08:13:04.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.084+0000 7ffb91d46640 1 --2- 172.21.15.98:0/3190100423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbf95d0 0x563b5fc10310 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7ffb80012010 tx=0x7ffb80011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e7b1b01ae7ca90 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:04.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.084+0000 7ffb73fff640 1 -- 172.21.15.98:0/3190100423 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb80030070 con 0x563b5fbf95d0 2024-09-19T08:13:04.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.084+0000 7ffb73fff640 1 -- 172.21.15.98:0/3190100423 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb8000ad40 con 0x563b5fbf95d0 2024-09-19T08:13:04.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.084+0000 7ffb73fff640 1 -- 172.21.15.98:0/3190100423 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb80019040 con 0x563b5fbf95d0 2024-09-19T08:13:04.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.084+0000 7ffb96b84740 1 -- 172.21.15.98:0/3190100423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbf95d0 msgr2=0x563b5fc10310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:04.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.084+0000 7ffb96b84740 1 --2- 172.21.15.98:0/3190100423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbf95d0 0x563b5fc10310 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7ffb80012010 tx=0x7ffb80011d20 comp rx=0 tx=0).stop 2024-09-19T08:13:04.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.084+0000 7ffb96b84740 1 -- 172.21.15.98:0/3190100423 shutdown_connections 2024-09-19T08:13:04.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.084+0000 7ffb96b84740 1 --2- 172.21.15.98:0/3190100423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 0x563b5fc1c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.084+0000 7ffb96b84740 1 --2- 172.21.15.98:0/3190100423 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fbfab40 0x563b5fc1bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.084+0000 7ffb96b84740 1 --2- 172.21.15.98:0/3190100423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbf95d0 0x563b5fc10310 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.084+0000 7ffb96b84740 1 -- 172.21.15.98:0/3190100423 >> 172.21.15.98:0/3190100423 conn(0x563b5fbc2310 msgr2=0x563b5fc000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:04.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.084+0000 7ffb96b84740 1 -- 172.21.15.98:0/3190100423 shutdown_connections 2024-09-19T08:13:04.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.084+0000 7ffb96b84740 1 -- 172.21.15.98:0/3190100423 wait complete. 2024-09-19T08:13:04.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb96b84740 1 -- start start 2024-09-19T08:13:04.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb96b84740 1 Processor -- start 2024-09-19T08:13:04.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb96b84740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fbf95d0 0x563b5fc08bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb96b84740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fc090f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb96b84740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 0x563b5fc0c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb96b84740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c1aecb0 con 0x563b5fbfab40 2024-09-19T08:13:04.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb96b84740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c1ae5e0 con 0x563b5fbf95d0 2024-09-19T08:13:04.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb96b84740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c1ab840 con 0x563b5fc04370 2024-09-19T08:13:04.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb96b84740 10 client.0 _ll_register_callbacks cb 0x7ffb84004160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:04.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb92547640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fc090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb92547640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fc090f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43016/0 (socket says 172.21.15.98:43016) 2024-09-19T08:13:04.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb92547640 1 -- 172.21.15.98:0/1334332435 learned_addr learned my addr 172.21.15.98:0/1334332435 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:04.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb91d46640 1 --2- 172.21.15.98:0/1334332435 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fbf95d0 0x563b5fc08bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb91d46640 1 --2- 172.21.15.98:0/1334332435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 0x563b5fc0c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb92547640 1 -- 172.21.15.98:0/1334332435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 msgr2=0x563b5fc0c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:04.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb92547640 1 --2- 172.21.15.98:0/1334332435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 0x563b5fc0c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb92547640 1 -- 172.21.15.98:0/1334332435 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fbf95d0 msgr2=0x563b5fc08bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:04.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.085+0000 7ffb92547640 1 --2- 172.21.15.98:0/1334332435 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fbf95d0 0x563b5fc08bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb92547640 1 -- 172.21.15.98:0/1334332435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b5fbfc120 con 0x563b5fbfab40 2024-09-19T08:13:04.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb91d46640 1 --2- 172.21.15.98:0/1334332435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 0x563b5fc0c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:04.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb92547640 1 --2- 172.21.15.98:0/1334332435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fc090f0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7ffb84006010 tx=0x7ffb840040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3413cb97aee81902 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:04.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:04.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb71ffb640 1 -- 172.21.15.98:0/1334332435 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb84035050 con 0x563b5fbfab40 2024-09-19T08:13:04.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb71ffb640 1 -- 172.21.15.98:0/1334332435 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7ffb8404e680 con 0x563b5fbfab40 2024-09-19T08:13:04.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb96b84740 1 -- 172.21.15.98:0/1334332435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb8000ad40 con 0x563b5fbfab40 2024-09-19T08:13:04.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb71ffb640 1 -- 172.21.15.98:0/1334332435 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffb8404e880 con 0x563b5fbfab40 2024-09-19T08:13:04.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb96b84740 10 client.17754 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:04.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb96b84740 1 -- 172.21.15.98:0/1334332435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563b5fc7f510 con 0x563b5fbfab40 2024-09-19T08:13:04.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb71ffb640 1 -- 172.21.15.98:0/1334332435 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb8000ad40 con 0x563b5fbfab40 2024-09-19T08:13:04.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb71ffb640 1 -- 172.21.15.98:0/1334332435 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ffb8403d520 con 0x563b5fbfab40 2024-09-19T08:13:04.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb96b84740 10 client.17754 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:04.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb96b84740 10 client.17754 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:13:04.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb96b84740 20 client.17754 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:04.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb96b84740 1 --2- 172.21.15.98:0/1334332435 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x563b5fc7fc00 0x563b5fc82020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:04.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb96b84740 4 client.17754 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:13:04.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.086+0000 7ffb96b84740 1 -- 172.21.15.98:0/1334332435 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563b5fb66d70 con 0x563b5fc7fc00 2024-09-19T08:13:04.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb91d46640 1 --2- 172.21.15.98:0/1334332435 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x563b5fc7fc00 0x563b5fc82020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:04.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb91d46640 1 --2- 172.21.15.98:0/1334332435 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x563b5fc7fc00 0x563b5fc82020 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5cff2921f3cfafd1 server_cookie=a1d3eac7b868a75a in_seq=0 out_seq=0 2024-09-19T08:13:04.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb71ffb640 10 client.17754 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:13:04.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb71ffb640 1 -- 172.21.15.98:0/1334332435 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563b5fb66d70 con 0x563b5fc7fc00 2024-09-19T08:13:04.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb71ffb640 10 client.17754 handle_command_reply: tid=0 2024-09-19T08:13:04.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb96b84740 1 client.17754 shutdown 2024-09-19T08:13:04.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb96b84740 1 -- 172.21.15.98:0/1334332435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 msgr2=0x563b5fc090f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:04.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb96b84740 1 --2- 172.21.15.98:0/1334332435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fc090f0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7ffb84006010 tx=0x7ffb840040f0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb96b84740 1 -- 172.21.15.98:0/1334332435 shutdown_connections 2024-09-19T08:13:04.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb96b84740 1 --2- 172.21.15.98:0/1334332435 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x563b5fc7fc00 0x563b5fc82020 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb96b84740 1 --2- 172.21.15.98:0/1334332435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b5fc04370 0x563b5fc0c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb96b84740 1 --2- 172.21.15.98:0/1334332435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b5fbfab40 0x563b5fc090f0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb96b84740 1 --2- 172.21.15.98:0/1334332435 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b5fbf95d0 0x563b5fc08bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb96b84740 1 -- 172.21.15.98:0/1334332435 >> 172.21.15.98:0/1334332435 conn(0x563b5fbc2310 msgr2=0x563b5fbf44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:04.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.087+0000 7ffb71ffb640 0 client.17754 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:13:04.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb96b84740 1 -- 172.21.15.98:0/1334332435 shutdown_connections 2024-09-19T08:13:04.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb96b84740 1 -- 172.21.15.98:0/1334332435 wait complete. 2024-09-19T08:13:04.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb96b84740 20 client.17754 trim_cache size 0 max 16384 2024-09-19T08:13:04.101 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:13:04.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb947ab640 1 -- 172.21.15.98:0/203859125 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ffb740a33b0 msgr2=0x7ffb740a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:04.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb947ab640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ffb740a33b0 0x7ffb740a5870 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c09b330 tx=0x7ffb7c06d3b0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb947ab640 1 -- 172.21.15.98:0/203859125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c102ef0 msgr2=0x7ffb8c1a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:04.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb947ab640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c102ef0 0x7ffb8c1a43b0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00b970 tx=0x7ffb7c00bd70 comp rx=0 tx=0).stop 2024-09-19T08:13:04.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb947ab640 1 -- 172.21.15.98:0/203859125 shutdown_connections 2024-09-19T08:13:04.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb947ab640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7ffb740a33b0 0x7ffb740a5870 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb947ab640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8c102ef0 0x7ffb8c1a43b0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb947ab640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffb8c102570 0x7ffb8c199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb947ab640 1 --2- 172.21.15.98:0/203859125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb8c100e20 0x7ffb8c199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:04.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb947ab640 1 -- 172.21.15.98:0/203859125 >> 172.21.15.98:0/203859125 conn(0x7ffb8c0fc7d0 msgr2=0x7ffb8c106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:04.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb947ab640 1 -- 172.21.15.98:0/203859125 shutdown_connections 2024-09-19T08:13:04.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:04.088+0000 7ffb947ab640 1 -- 172.21.15.98:0/203859125 wait complete. 2024-09-19T08:13:04.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:03 smithi162 ceph-mon[35260]: pgmap v310: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:13:04.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:03 smithi139 ceph-mon[34874]: pgmap v310: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:13:04.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:03 smithi098 ceph-mon[31333]: pgmap v310: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:13:04.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:13:05.120 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:05.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.583+0000 7fd454922640 1 -- 172.21.15.98:0/1698009037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd450100e30 msgr2=0x7fd450104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:05.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.583+0000 7fd454922640 1 --2- 172.21.15.98:0/1698009037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd450100e30 0x7fd450104e80 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fd444009900 tx=0x7fd44402a410 comp rx=0 tx=0).stop 2024-09-19T08:13:05.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.584+0000 7fd454922640 1 -- 172.21.15.98:0/1698009037 shutdown_connections 2024-09-19T08:13:05.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.584+0000 7fd454922640 1 --2- 172.21.15.98:0/1698009037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4501055b0 0x7fd45010d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.584+0000 7fd454922640 1 --2- 172.21.15.98:0/1698009037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd450100e30 0x7fd450104e80 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.584+0000 7fd454922640 1 --2- 172.21.15.98:0/1698009037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd450100480 0x7fd450100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.584+0000 7fd454922640 1 -- 172.21.15.98:0/1698009037 >> 172.21.15.98:0/1698009037 conn(0x7fd450077210 msgr2=0x7fd4500fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:05.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.584+0000 7fd454922640 1 -- 172.21.15.98:0/1698009037 shutdown_connections 2024-09-19T08:13:05.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.584+0000 7fd454922640 1 -- 172.21.15.98:0/1698009037 wait complete. 2024-09-19T08:13:05.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.584+0000 7fd454922640 1 Processor -- start 2024-09-19T08:13:05.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd454922640 1 -- start start 2024-09-19T08:13:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd454922640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd450100480 0x7fd450197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd454922640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd450100e30 0x7fd450197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd454922640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4501055b0 0x7fd45019bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd454922640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd450110630 con 0x7fd4501055b0 2024-09-19T08:13:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd454922640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501104b0 con 0x7fd450100e30 2024-09-19T08:13:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd454922640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501107b0 con 0x7fd450100480 2024-09-19T08:13:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd44ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4501055b0 0x7fd45019bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd44ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4501055b0 0x7fd45019bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43032/0 (socket says 172.21.15.98:43032) 2024-09-19T08:13:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd44ed9d640 1 -- 172.21.15.98:0/1455679002 learned_addr learned my addr 172.21.15.98:0/1455679002 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd44dd9b640 1 --2- 172.21.15.98:0/1455679002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd450100e30 0x7fd450197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd44e59c640 1 --2- 172.21.15.98:0/1455679002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd450100480 0x7fd450197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd44ed9d640 1 -- 172.21.15.98:0/1455679002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd450100480 msgr2=0x7fd450197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd44ed9d640 1 --2- 172.21.15.98:0/1455679002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd450100480 0x7fd450197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd44ed9d640 1 -- 172.21.15.98:0/1455679002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd450100e30 msgr2=0x7fd450197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd44ed9d640 1 --2- 172.21.15.98:0/1455679002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd450100e30 0x7fd450197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd44ed9d640 1 -- 172.21.15.98:0/1455679002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45019c310 con 0x7fd4501055b0 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.585+0000 7fd44ed9d640 1 --2- 172.21.15.98:0/1455679002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4501055b0 0x7fd45019bc30 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fd44800b970 tx=0x7fd44800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.586+0000 7fd43f7fe640 1 -- 172.21.15.98:0/1455679002 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448019070 con 0x7fd4501055b0 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.586+0000 7fd43f7fe640 1 -- 172.21.15.98:0/1455679002 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd448014440 con 0x7fd4501055b0 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.586+0000 7fd454922640 1 -- 172.21.15.98:0/1455679002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd45019c600 con 0x7fd4501055b0 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.586+0000 7fd43f7fe640 1 -- 172.21.15.98:0/1455679002 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4480075d0 con 0x7fd4501055b0 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.586+0000 7fd454922640 1 -- 172.21.15.98:0/1455679002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4501967b0 con 0x7fd4501055b0 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.588+0000 7fd43f7fe640 1 -- 172.21.15.98:0/1455679002 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fd44801c070 con 0x7fd4501055b0 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.588+0000 7fd456cfb740 1 Processor -- start 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.588+0000 7fd456cfb740 1 -- start start 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd456cfb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86bb40 0x55bcda86a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd456cfb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda86a5d0 0x55bcda86aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd456cfb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda875370 0x55bcda877760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd456cfb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bcda7d02f0 con 0x55bcda875370 2024-09-19T08:13:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd456cfb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bcda777cf0 con 0x55bcda86bb40 2024-09-19T08:13:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd456cfb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bcda7cc2b0 con 0x55bcda86a5d0 2024-09-19T08:13:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd43f7fe640 1 --2- 172.21.15.98:0/1455679002 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd4380a33b0 0x7fd4380a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd44dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda875370 0x55bcda877760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd44dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda875370 0x55bcda877760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43042/0 (socket says 172.21.15.98:43042) 2024-09-19T08:13:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd44dd9b640 1 -- 172.21.15.98:0/1848331803 learned_addr learned my addr 172.21.15.98:0/1848331803 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd44dd9b640 1 --2- 172.21.15.98:0/1848331803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86bb40 0x55bcda86a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd44ed9d640 1 --2- 172.21.15.98:0/1455679002 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd4380a33b0 0x7fd4380a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd44e59c640 1 --2- 172.21.15.98:0/1848331803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda86a5d0 0x55bcda86aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd43f7fe640 1 -- 172.21.15.98:0/1455679002 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7fd4480cf1b0 con 0x7fd4501055b0 2024-09-19T08:13:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd44dd9b640 1 -- 172.21.15.98:0/1848331803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda86a5d0 msgr2=0x55bcda86aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd44dd9b640 1 --2- 172.21.15.98:0/1848331803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda86a5d0 0x55bcda86aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd44dd9b640 1 -- 172.21.15.98:0/1848331803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86bb40 msgr2=0x55bcda86a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd44dd9b640 1 --2- 172.21.15.98:0/1848331803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86bb40 0x55bcda86a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd44dd9b640 1 -- 172.21.15.98:0/1848331803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcda75c3a0 con 0x55bcda875370 2024-09-19T08:13:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.589+0000 7fd44dd9b640 1 --2- 172.21.15.98:0/1848331803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda875370 0x55bcda877760 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fd444009900 tx=0x7fd4440314d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902949b7ccaad69b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.590+0000 7fd44ed9d640 1 --2- 172.21.15.98:0/1455679002 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd4380a33b0 0x7fd4380a5870 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd44800bea0 tx=0x7fd44800e040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.590+0000 7fd43cff9640 1 -- 172.21.15.98:0/1848331803 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd444035040 con 0x55bcda875370 2024-09-19T08:13:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.590+0000 7fd43cff9640 1 -- 172.21.15.98:0/1848331803 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd444031a10 con 0x55bcda875370 2024-09-19T08:13:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.590+0000 7fd43cff9640 1 -- 172.21.15.98:0/1848331803 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4440386d0 con 0x55bcda875370 2024-09-19T08:13:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.590+0000 7fd456cfb740 1 -- 172.21.15.98:0/1848331803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda875370 msgr2=0x55bcda877760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.590+0000 7fd456cfb740 1 --2- 172.21.15.98:0/1848331803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda875370 0x55bcda877760 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fd444009900 tx=0x7fd4440314d0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.590+0000 7fd456cfb740 1 -- 172.21.15.98:0/1848331803 shutdown_connections 2024-09-19T08:13:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.590+0000 7fd456cfb740 1 --2- 172.21.15.98:0/1848331803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda875370 0x55bcda877760 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.590+0000 7fd456cfb740 1 --2- 172.21.15.98:0/1848331803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda86a5d0 0x55bcda86aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.590+0000 7fd456cfb740 1 --2- 172.21.15.98:0/1848331803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86bb40 0x55bcda86a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.590+0000 7fd456cfb740 1 -- 172.21.15.98:0/1848331803 >> 172.21.15.98:0/1848331803 conn(0x55bcda833310 msgr2=0x55bcda866c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.590+0000 7fd456cfb740 1 -- 172.21.15.98:0/1848331803 shutdown_connections 2024-09-19T08:13:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd456cfb740 1 -- 172.21.15.98:0/1848331803 wait complete. 2024-09-19T08:13:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd456cfb740 1 -- start start 2024-09-19T08:13:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd456cfb740 1 Processor -- start 2024-09-19T08:13:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd456cfb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86a5d0 0x55bcda874db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd456cfb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86bb40 0x55bcda872060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd456cfb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 0x55bcda8727f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd456cfb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bcda878010 con 0x55bcda86a5d0 2024-09-19T08:13:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd456cfb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bcda78b060 con 0x55bcda86bb40 2024-09-19T08:13:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd456cfb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bcda878190 con 0x55bcda875370 2024-09-19T08:13:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd456cfb740 10 client.0 _ll_register_callbacks cb 0x55bcda4de8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd44dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86a5d0 0x55bcda874db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd44dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86a5d0 0x55bcda874db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43054/0 (socket says 172.21.15.98:43054) 2024-09-19T08:13:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd44dd9b640 1 -- 172.21.15.98:0/563340187 learned_addr learned my addr 172.21.15.98:0/563340187 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd44dd9b640 1 --2- 172.21.15.98:0/563340187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 0x55bcda8727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd44dd9b640 1 -- 172.21.15.98:0/563340187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 msgr2=0x55bcda8727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd44dd9b640 1 --2- 172.21.15.98:0/563340187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 0x55bcda8727f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd44dd9b640 1 -- 172.21.15.98:0/563340187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86bb40 msgr2=0x55bcda872060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd44dd9b640 1 --2- 172.21.15.98:0/563340187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86bb40 0x55bcda872060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.591+0000 7fd44dd9b640 1 -- 172.21.15.98:0/563340187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcda5413c0 con 0x55bcda86a5d0 2024-09-19T08:13:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd44dd9b640 1 --2- 172.21.15.98:0/563340187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86a5d0 0x55bcda874db0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fd44402a450 tx=0x7fd444003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecef37a9e6a5da50 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd435ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd435ffb640 1 -- 172.21.15.98:0/563340187 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd444035070 con 0x55bcda86a5d0 2024-09-19T08:13:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd435ffb640 1 -- 172.21.15.98:0/563340187 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7fd444003a00 con 0x55bcda86a5d0 2024-09-19T08:13:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd456cfb740 1 -- 172.21.15.98:0/563340187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bcda7dd860 con 0x55bcda86a5d0 2024-09-19T08:13:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd435ffb640 1 -- 172.21.15.98:0/563340187 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd444003e80 con 0x55bcda86a5d0 2024-09-19T08:13:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd456cfb740 10 client.17766 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd456cfb740 1 -- 172.21.15.98:0/563340187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bcda878010 con 0x55bcda86a5d0 2024-09-19T08:13:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd435ffb640 1 -- 172.21.15.98:0/563340187 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bcda7dd860 con 0x55bcda86a5d0 2024-09-19T08:13:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd435ffb640 1 -- 172.21.15.98:0/563340187 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd444038b10 con 0x55bcda86a5d0 2024-09-19T08:13:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd456cfb740 10 client.17766 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd456cfb740 10 client.17766 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:13:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd456cfb740 20 client.17766 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd456cfb740 1 --2- 172.21.15.98:0/563340187 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55bcda884c80 0x55bcda887070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd456cfb740 4 client.17766 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:13:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.592+0000 7fd456cfb740 1 -- 172.21.15.98:0/563340187 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcda7d7d70 con 0x55bcda884c80 2024-09-19T08:13:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.593+0000 7fd44e59c640 1 --2- 172.21.15.98:0/563340187 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55bcda884c80 0x55bcda887070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.593+0000 7fd44e59c640 1 --2- 172.21.15.98:0/563340187 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55bcda884c80 0x55bcda887070 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=802678cebaf2e62f server_cookie=36eb3f79c17c82e6 in_seq=0 out_seq=0 2024-09-19T08:13:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.593+0000 7fd435ffb640 10 client.17766 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:13:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd435ffb640 1 -- 172.21.15.98:0/563340187 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bcda7d7d70 con 0x55bcda884c80 2024-09-19T08:13:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd435ffb640 10 client.17766 handle_command_reply: tid=0 2024-09-19T08:13:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd456cfb740 1 client.17766 shutdown 2024-09-19T08:13:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd456cfb740 1 -- 172.21.15.98:0/563340187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86a5d0 msgr2=0x55bcda874db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd456cfb740 1 --2- 172.21.15.98:0/563340187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86a5d0 0x55bcda874db0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fd44402a450 tx=0x7fd444003ce0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd456cfb740 1 -- 172.21.15.98:0/563340187 shutdown_connections 2024-09-19T08:13:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd456cfb740 1 --2- 172.21.15.98:0/563340187 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55bcda884c80 0x55bcda887070 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd456cfb740 1 --2- 172.21.15.98:0/563340187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 0x55bcda8727f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd456cfb740 1 --2- 172.21.15.98:0/563340187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86bb40 0x55bcda872060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd435ffb640 0 client.17766 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:13:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd456cfb740 1 --2- 172.21.15.98:0/563340187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86a5d0 0x55bcda874db0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd456cfb740 1 -- 172.21.15.98:0/563340187 >> 172.21.15.98:0/563340187 conn(0x55bcda833310 msgr2=0x55bcda86c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd456cfb740 1 -- 172.21.15.98:0/563340187 shutdown_connections 2024-09-19T08:13:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd456cfb740 1 -- 172.21.15.98:0/563340187 wait complete. 2024-09-19T08:13:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.594+0000 7fd456cfb740 20 client.17766 trim_cache size 0 max 16384 2024-09-19T08:13:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd456cfb740 1 Processor -- start 2024-09-19T08:13:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd456cfb740 1 -- start start 2024-09-19T08:13:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd456cfb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86a5d0 0x55bcda87d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd456cfb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86bb40 0x55bcda87dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd456cfb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 0x55bcda86d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd456cfb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501ab5d0 con 0x55bcda86bb40 2024-09-19T08:13:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd456cfb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501ad110 con 0x55bcda86a5d0 2024-09-19T08:13:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd456cfb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501abca0 con 0x55bcda875370 2024-09-19T08:13:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd44e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86bb40 0x55bcda87dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd44e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86bb40 0x55bcda87dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43064/0 (socket says 172.21.15.98:43064) 2024-09-19T08:13:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd44e59c640 1 -- 172.21.15.98:0/184061541 learned_addr learned my addr 172.21.15.98:0/184061541 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd44e59c640 1 -- 172.21.15.98:0/184061541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 msgr2=0x55bcda86d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd44dd9b640 1 --2- 172.21.15.98:0/184061541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 0x55bcda86d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd44e59c640 1 --2- 172.21.15.98:0/184061541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 0x55bcda86d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd44e59c640 1 -- 172.21.15.98:0/184061541 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86a5d0 msgr2=0x55bcda87d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd44e59c640 1 --2- 172.21.15.98:0/184061541 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86a5d0 0x55bcda87d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd44e59c640 1 -- 172.21.15.98:0/184061541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcda86b300 con 0x55bcda86bb40 2024-09-19T08:13:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd44dd9b640 1 --2- 172.21.15.98:0/184061541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 0x55bcda86d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.607+0000 7fd44e59c640 1 --2- 172.21.15.98:0/184061541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86bb40 0x55bcda87dd40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fd44000ac90 tx=0x7fd440012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec7f54e3859967 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.608+0000 7fd437fff640 1 -- 172.21.15.98:0/184061541 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd440021030 con 0x55bcda86bb40 2024-09-19T08:13:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.608+0000 7fd437fff640 1 -- 172.21.15.98:0/184061541 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd440012e80 con 0x55bcda86bb40 2024-09-19T08:13:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.608+0000 7fd437fff640 1 -- 172.21.15.98:0/184061541 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd440010720 con 0x55bcda86bb40 2024-09-19T08:13:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.608+0000 7fd456cfb740 1 -- 172.21.15.98:0/184061541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86bb40 msgr2=0x55bcda87dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.608+0000 7fd456cfb740 1 --2- 172.21.15.98:0/184061541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86bb40 0x55bcda87dd40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fd44000ac90 tx=0x7fd440012820 comp rx=0 tx=0).stop 2024-09-19T08:13:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.608+0000 7fd456cfb740 1 -- 172.21.15.98:0/184061541 shutdown_connections 2024-09-19T08:13:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.608+0000 7fd456cfb740 1 --2- 172.21.15.98:0/184061541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 0x55bcda86d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.608+0000 7fd456cfb740 1 --2- 172.21.15.98:0/184061541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86bb40 0x55bcda87dd40 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.608+0000 7fd456cfb740 1 --2- 172.21.15.98:0/184061541 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86a5d0 0x55bcda87d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.608+0000 7fd456cfb740 1 -- 172.21.15.98:0/184061541 >> 172.21.15.98:0/184061541 conn(0x55bcda833310 msgr2=0x55bcda865d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.608+0000 7fd456cfb740 1 -- 172.21.15.98:0/184061541 shutdown_connections 2024-09-19T08:13:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.608+0000 7fd456cfb740 1 -- 172.21.15.98:0/184061541 wait complete. 2024-09-19T08:13:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.608+0000 7fd456cfb740 1 -- start start 2024-09-19T08:13:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd456cfb740 1 Processor -- start 2024-09-19T08:13:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd456cfb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86a5d0 0x55bcda88c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd456cfb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86bb40 0x55bcda88cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd456cfb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 0x55bcda8ead60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd456cfb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501aca40 con 0x55bcda86a5d0 2024-09-19T08:13:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd456cfb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501ac370 con 0x55bcda86bb40 2024-09-19T08:13:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd456cfb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501a95d0 con 0x55bcda875370 2024-09-19T08:13:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd456cfb740 10 client.0 _ll_register_callbacks cb 0x7fd444032e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd44dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86a5d0 0x55bcda88c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd44dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86a5d0 0x55bcda88c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43076/0 (socket says 172.21.15.98:43076) 2024-09-19T08:13:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd44dd9b640 1 -- 172.21.15.98:0/2556732666 learned_addr learned my addr 172.21.15.98:0/2556732666 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd44e59c640 1 --2- 172.21.15.98:0/2556732666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86bb40 0x55bcda88cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd44dd9b640 1 -- 172.21.15.98:0/2556732666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 msgr2=0x55bcda8ead60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd44dd9b640 1 --2- 172.21.15.98:0/2556732666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 0x55bcda8ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd44dd9b640 1 -- 172.21.15.98:0/2556732666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86bb40 msgr2=0x55bcda88cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd44dd9b640 1 --2- 172.21.15.98:0/2556732666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86bb40 0x55bcda88cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd44dd9b640 1 -- 172.21.15.98:0/2556732666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcda8eb400 con 0x55bcda86a5d0 2024-09-19T08:13:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd44dd9b640 1 --2- 172.21.15.98:0/2556732666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86a5d0 0x55bcda88c610 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fd44402a930 tx=0x7fd4440523e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444cf878a166c8fd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd435ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd435ffb640 1 -- 172.21.15.98:0/2556732666 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd444061020 con 0x55bcda86a5d0 2024-09-19T08:13:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd435ffb640 1 -- 172.21.15.98:0/2556732666 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7fd444003800 con 0x55bcda86a5d0 2024-09-19T08:13:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.609+0000 7fd456cfb740 1 -- 172.21.15.98:0/2556732666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bcda754170 con 0x55bcda86a5d0 2024-09-19T08:13:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.610+0000 7fd435ffb640 1 -- 172.21.15.98:0/2556732666 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd44406c050 con 0x55bcda86a5d0 2024-09-19T08:13:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.610+0000 7fd456cfb740 10 client.17772 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.610+0000 7fd456cfb740 1 -- 172.21.15.98:0/2556732666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bcda8f3900 con 0x55bcda86a5d0 2024-09-19T08:13:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.610+0000 7fd435ffb640 1 -- 172.21.15.98:0/2556732666 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bcda754170 con 0x55bcda86a5d0 2024-09-19T08:13:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.610+0000 7fd435ffb640 1 -- 172.21.15.98:0/2556732666 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd4440047a0 con 0x55bcda86a5d0 2024-09-19T08:13:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.610+0000 7fd456cfb740 10 client.17772 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.610+0000 7fd456cfb740 10 client.17772 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:13:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.610+0000 7fd456cfb740 20 client.17772 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.610+0000 7fd456cfb740 1 --2- 172.21.15.98:0/2556732666 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55bcda8f4050 0x55bcda8f6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.610+0000 7fd456cfb740 4 client.17772 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:13:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.610+0000 7fd456cfb740 1 -- 172.21.15.98:0/2556732666 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bcda7a19b0 con 0x55bcda8f4050 2024-09-19T08:13:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.610+0000 7fd44e59c640 1 --2- 172.21.15.98:0/2556732666 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55bcda8f4050 0x55bcda8f6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd44e59c640 1 --2- 172.21.15.98:0/2556732666 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55bcda8f4050 0x55bcda8f6470 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b3c5c64e2ba80223 server_cookie=67bb15d2b84242f in_seq=0 out_seq=0 2024-09-19T08:13:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd435ffb640 10 client.17772 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:13:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd435ffb640 1 -- 172.21.15.98:0/2556732666 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bcda7a19b0 con 0x55bcda8f4050 2024-09-19T08:13:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd435ffb640 10 client.17772 handle_command_reply: tid=0 2024-09-19T08:13:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd456cfb740 1 client.17772 shutdown 2024-09-19T08:13:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd456cfb740 1 -- 172.21.15.98:0/2556732666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86a5d0 msgr2=0x55bcda88c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd456cfb740 1 --2- 172.21.15.98:0/2556732666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86a5d0 0x55bcda88c610 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fd44402a930 tx=0x7fd4440523e0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd456cfb740 1 -- 172.21.15.98:0/2556732666 shutdown_connections 2024-09-19T08:13:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd456cfb740 1 --2- 172.21.15.98:0/2556732666 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x55bcda8f4050 0x55bcda8f6470 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd435ffb640 0 client.17772 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:13:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd456cfb740 1 --2- 172.21.15.98:0/2556732666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcda875370 0x55bcda8ead60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd456cfb740 1 --2- 172.21.15.98:0/2556732666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bcda86bb40 0x55bcda88cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd456cfb740 1 --2- 172.21.15.98:0/2556732666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bcda86a5d0 0x55bcda88c610 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd456cfb740 1 -- 172.21.15.98:0/2556732666 >> 172.21.15.98:0/2556732666 conn(0x55bcda833310 msgr2=0x55bcda870800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd456cfb740 1 -- 172.21.15.98:0/2556732666 shutdown_connections 2024-09-19T08:13:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd456cfb740 1 -- 172.21.15.98:0/2556732666 wait complete. 2024-09-19T08:13:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.611+0000 7fd456cfb740 20 client.17772 trim_cache size 0 max 16384 2024-09-19T08:13:05.637 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:13:05.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.612+0000 7fd454922640 1 -- 172.21.15.98:0/1455679002 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd4380a33b0 msgr2=0x7fd4380a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:05.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.612+0000 7fd454922640 1 --2- 172.21.15.98:0/1455679002 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd4380a33b0 0x7fd4380a5870 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd44800bea0 tx=0x7fd44800e040 comp rx=0 tx=0).stop 2024-09-19T08:13:05.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.612+0000 7fd454922640 1 -- 172.21.15.98:0/1455679002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4501055b0 msgr2=0x7fd45019bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:05.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.612+0000 7fd454922640 1 --2- 172.21.15.98:0/1455679002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4501055b0 0x7fd45019bc30 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fd44800b970 tx=0x7fd44800bd70 comp rx=0 tx=0).stop 2024-09-19T08:13:05.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.612+0000 7fd454922640 1 -- 172.21.15.98:0/1455679002 shutdown_connections 2024-09-19T08:13:05.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.612+0000 7fd454922640 1 --2- 172.21.15.98:0/1455679002 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd4380a33b0 0x7fd4380a5870 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.612+0000 7fd454922640 1 --2- 172.21.15.98:0/1455679002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4501055b0 0x7fd45019bc30 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.612+0000 7fd454922640 1 --2- 172.21.15.98:0/1455679002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd450100e30 0x7fd450197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.612+0000 7fd454922640 1 --2- 172.21.15.98:0/1455679002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd450100480 0x7fd450197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.612+0000 7fd454922640 1 -- 172.21.15.98:0/1455679002 >> 172.21.15.98:0/1455679002 conn(0x7fd450077210 msgr2=0x7fd450103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.612+0000 7fd454922640 1 -- 172.21.15.98:0/1455679002 shutdown_connections 2024-09-19T08:13:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:05.612+0000 7fd454922640 1 -- 172.21.15.98:0/1455679002 wait complete. 2024-09-19T08:13:06.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:05 smithi098 ceph-mon[31333]: pgmap v311: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:13:06.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:13:06.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:05 smithi162 ceph-mon[35260]: pgmap v311: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:13:06.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:05 smithi139 ceph-mon[34874]: pgmap v311: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:13:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.103+0000 7f02d426a640 1 -- 172.21.15.98:0/508183525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc100a80 msgr2=0x7f02cc100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.103+0000 7f02d426a640 1 --2- 172.21.15.98:0/508183525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc100a80 0x7f02cc100ee0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009930 tx=0x7f02bc02a420 comp rx=0 tx=0).stop 2024-09-19T08:13:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.103+0000 7f02d426a640 1 -- 172.21.15.98:0/508183525 shutdown_connections 2024-09-19T08:13:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.103+0000 7f02d426a640 1 --2- 172.21.15.98:0/508183525 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f02cc101420 0x7f02cc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.103+0000 7f02d426a640 1 --2- 172.21.15.98:0/508183525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc100a80 0x7f02cc100ee0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.103+0000 7f02d426a640 1 --2- 172.21.15.98:0/508183525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02cc106a80 0x7f02cc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.103+0000 7f02d426a640 1 -- 172.21.15.98:0/508183525 >> 172.21.15.98:0/508183525 conn(0x7f02cc0fc7a0 msgr2=0x7f02cc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.103+0000 7f02d426a640 1 -- 172.21.15.98:0/508183525 shutdown_connections 2024-09-19T08:13:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.103+0000 7f02d426a640 1 -- 172.21.15.98:0/508183525 wait complete. 2024-09-19T08:13:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.103+0000 7f02d426a640 1 Processor -- start 2024-09-19T08:13:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.103+0000 7f02d426a640 1 -- start start 2024-09-19T08:13:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d426a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f02cc100a80 0x7f02cc199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d426a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc101420 0x7f02cc1997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d426a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02cc106a80 0x7f02cc1a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d426a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc110e80 con 0x7f02cc101420 2024-09-19T08:13:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d426a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc110d00 con 0x7f02cc100a80 2024-09-19T08:13:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d426a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc111000 con 0x7f02cc106a80 2024-09-19T08:13:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d1805640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc101420 0x7f02cc1997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d1805640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc101420 0x7f02cc1997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48766/0 (socket says 172.21.15.98:48766) 2024-09-19T08:13:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d1805640 1 -- 172.21.15.98:0/3405435557 learned_addr learned my addr 172.21.15.98:0/3405435557 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d2807640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02cc106a80 0x7f02cc1a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d2006640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f02cc100a80 0x7f02cc199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d1805640 1 -- 172.21.15.98:0/3405435557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02cc106a80 msgr2=0x7f02cc1a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d1805640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02cc106a80 0x7f02cc1a4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d1805640 1 -- 172.21.15.98:0/3405435557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f02cc100a80 msgr2=0x7f02cc199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d1805640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f02cc100a80 0x7f02cc199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d1805640 1 -- 172.21.15.98:0/3405435557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02cc1a4840 con 0x7f02cc101420 2024-09-19T08:13:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d2006640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f02cc100a80 0x7f02cc199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d1805640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc101420 0x7f02cc1997c0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009230 tx=0x7f02bc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02c37fe640 1 -- 172.21.15.98:0/3405435557 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc03f030 con 0x7f02cc101420 2024-09-19T08:13:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02c37fe640 1 -- 172.21.15.98:0/3405435557 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02bc003ad0 con 0x7f02cc101420 2024-09-19T08:13:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02d426a640 1 -- 172.21.15.98:0/3405435557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02cc1a4a10 con 0x7f02cc101420 2024-09-19T08:13:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.104+0000 7f02c37fe640 1 -- 172.21.15.98:0/3405435557 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc035070 con 0x7f02cc101420 2024-09-19T08:13:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.105+0000 7f02d426a640 1 -- 172.21.15.98:0/3405435557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02cc1a4e60 con 0x7f02cc101420 2024-09-19T08:13:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.107+0000 7f02c37fe640 1 -- 172.21.15.98:0/3405435557 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f02bc042030 con 0x7f02cc101420 2024-09-19T08:13:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.108+0000 7f02d6643740 1 Processor -- start 2024-09-19T08:13:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.108+0000 7f02d6643740 1 -- start start 2024-09-19T08:13:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.108+0000 7f02d6643740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f717b40 0x558f8f716090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.108+0000 7f02d6643740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f7165d0 0x558f8f716a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.108+0000 7f02d6643740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f725780 0x558f8f727b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.108+0000 7f02d6643740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558f8f67c2f0 con 0x558f8f725780 2024-09-19T08:13:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.108+0000 7f02d6643740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558f8f623cf0 con 0x558f8f7165d0 2024-09-19T08:13:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.108+0000 7f02d6643740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558f8f6782b0 con 0x558f8f717b40 2024-09-19T08:13:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.108+0000 7f02d1805640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f725780 0x558f8f727b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.108+0000 7f02d1805640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f725780 0x558f8f727b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48780/0 (socket says 172.21.15.98:48780) 2024-09-19T08:13:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.108+0000 7f02d1805640 1 -- 172.21.15.98:0/4274546821 learned_addr learned my addr 172.21.15.98:0/4274546821 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.108+0000 7f02d2006640 1 --2- 172.21.15.98:0/4274546821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f7165d0 0x558f8f716a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.108+0000 7f02d2807640 1 --2- 172.21.15.98:0/4274546821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f717b40 0x558f8f716090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.108+0000 7f02c37fe640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f02b80a33b0 0x7f02b80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.109+0000 7f02d1805640 1 -- 172.21.15.98:0/4274546821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f717b40 msgr2=0x558f8f716090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.109+0000 7f02d1805640 1 --2- 172.21.15.98:0/4274546821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f717b40 0x558f8f716090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.109+0000 7f02d1805640 1 -- 172.21.15.98:0/4274546821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f7165d0 msgr2=0x558f8f716a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.109+0000 7f02d1805640 1 --2- 172.21.15.98:0/4274546821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f7165d0 0x558f8f716a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.109+0000 7f02d1805640 1 -- 172.21.15.98:0/4274546821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f8f6083a0 con 0x558f8f725780 2024-09-19T08:13:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.109+0000 7f02d2006640 1 --2- 172.21.15.98:0/4274546821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f7165d0 0x558f8f716a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.109+0000 7f02d2807640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f02b80a33b0 0x7f02b80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.109+0000 7f02c37fe640 1 -- 172.21.15.98:0/3405435557 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f02bc0f0230 con 0x7f02cc101420 2024-09-19T08:13:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.109+0000 7f02d1805640 1 --2- 172.21.15.98:0/4274546821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f725780 0x558f8f727b70 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0ed920 tx=0x7f02bc0e72e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a29b2f9c6a3b4b13 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.109+0000 7f02c0dea640 1 -- 172.21.15.98:0/4274546821 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc049030 con 0x558f8f725780 2024-09-19T08:13:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.109+0000 7f02d2807640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f02b80a33b0 0x7f02b80a5870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f02c400ac40 tx=0x7f02c40046c0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.109+0000 7f02c0dea640 1 -- 172.21.15.98:0/4274546821 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02bc0bcc10 con 0x558f8f725780 2024-09-19T08:13:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.109+0000 7f02c0dea640 1 -- 172.21.15.98:0/4274546821 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc0bc050 con 0x558f8f725780 2024-09-19T08:13:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.110+0000 7f02d6643740 1 -- 172.21.15.98:0/4274546821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f725780 msgr2=0x558f8f727b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.110+0000 7f02d6643740 1 --2- 172.21.15.98:0/4274546821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f725780 0x558f8f727b70 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0ed920 tx=0x7f02bc0e72e0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.110+0000 7f02d6643740 1 -- 172.21.15.98:0/4274546821 shutdown_connections 2024-09-19T08:13:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.110+0000 7f02d6643740 1 --2- 172.21.15.98:0/4274546821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f725780 0x558f8f727b70 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.110+0000 7f02d6643740 1 --2- 172.21.15.98:0/4274546821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f7165d0 0x558f8f716a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.110+0000 7f02d6643740 1 --2- 172.21.15.98:0/4274546821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f717b40 0x558f8f716090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.110+0000 7f02d6643740 1 -- 172.21.15.98:0/4274546821 >> 172.21.15.98:0/4274546821 conn(0x558f8f6df310 msgr2=0x558f8f712c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.110+0000 7f02d6643740 1 -- 172.21.15.98:0/4274546821 shutdown_connections 2024-09-19T08:13:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.110+0000 7f02d6643740 1 -- 172.21.15.98:0/4274546821 wait complete. 2024-09-19T08:13:07.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.110+0000 7f02d6643740 1 -- start start 2024-09-19T08:13:07.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.110+0000 7f02d6643740 1 Processor -- start 2024-09-19T08:13:07.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d6643740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f7165d0 0x558f8f725100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d6643740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f717b40 0x558f8f7223b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d6643740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 0x558f8f722930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d6643740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558f8f67c2f0 con 0x558f8f7165d0 2024-09-19T08:13:07.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d6643740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558f8f637060 con 0x558f8f717b40 2024-09-19T08:13:07.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d6643740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558f8f728240 con 0x558f8f725780 2024-09-19T08:13:07.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d6643740 10 client.0 _ll_register_callbacks cb 0x558f8f38a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:07.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d2006640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f7165d0 0x558f8f725100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d2006640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f7165d0 0x558f8f725100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48786/0 (socket says 172.21.15.98:48786) 2024-09-19T08:13:07.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d2006640 1 -- 172.21.15.98:0/3929411617 learned_addr learned my addr 172.21.15.98:0/3929411617 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:07.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d2807640 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 0x558f8f722930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d1805640 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f717b40 0x558f8f7223b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d2006640 1 -- 172.21.15.98:0/3929411617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 msgr2=0x558f8f722930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:07.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d2006640 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 0x558f8f722930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d2006640 1 -- 172.21.15.98:0/3929411617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f717b40 msgr2=0x558f8f7223b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:07.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d2006640 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f717b40 0x558f8f7223b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d2006640 1 -- 172.21.15.98:0/3929411617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f8f3ed3c0 con 0x558f8f7165d0 2024-09-19T08:13:07.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d1805640 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f717b40 0x558f8f7223b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:07.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d2807640 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 0x558f8f722930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:07.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d2006640 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f7165d0 0x558f8f725100 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f02c8012720 tx=0x7f02c8012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b1a71b4ddecf5c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:07.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:07.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02b5ffb640 1 -- 172.21.15.98:0/3929411617 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c801f060 con 0x558f8f7165d0 2024-09-19T08:13:07.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02b5ffb640 1 -- 172.21.15.98:0/3929411617 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f02c8010d60 con 0x558f8f7165d0 2024-09-19T08:13:07.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02d6643740 1 -- 172.21.15.98:0/3929411617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02bc0bcc10 con 0x558f8f7165d0 2024-09-19T08:13:07.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.111+0000 7f02b5ffb640 1 -- 172.21.15.98:0/3929411617 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f02c8023410 con 0x558f8f7165d0 2024-09-19T08:13:07.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.112+0000 7f02d6643740 10 client.17784 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:07.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.112+0000 7f02d6643740 1 -- 172.21.15.98:0/3929411617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558f8f733b50 con 0x558f8f7165d0 2024-09-19T08:13:07.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.112+0000 7f02b5ffb640 1 -- 172.21.15.98:0/3929411617 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02bc0bcc10 con 0x558f8f7165d0 2024-09-19T08:13:07.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.112+0000 7f02b5ffb640 1 -- 172.21.15.98:0/3929411617 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f02c8019030 con 0x558f8f7165d0 2024-09-19T08:13:07.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.112+0000 7f02d6643740 10 client.17784 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:07.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.112+0000 7f02d6643740 10 client.17784 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:13:07.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.112+0000 7f02d6643740 20 client.17784 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:07.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.112+0000 7f02d6643740 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x558f8f735010 0x558f8f737490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.112+0000 7f02d6643740 4 client.17784 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:13:07.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.112+0000 7f02d6643740 1 -- 172.21.15.98:0/3929411617 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f8f689860 con 0x558f8f735010 2024-09-19T08:13:07.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.112+0000 7f02d1805640 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x558f8f735010 0x558f8f737490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.113+0000 7f02d1805640 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x558f8f735010 0x558f8f737490 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9fc897c655a8e252 server_cookie=97397ddea209a47a in_seq=0 out_seq=0 2024-09-19T08:13:07.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.113+0000 7f02b5ffb640 10 client.17784 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:13:07.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.113+0000 7f02b5ffb640 1 -- 172.21.15.98:0/3929411617 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558f8f689860 con 0x558f8f735010 2024-09-19T08:13:07.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.113+0000 7f02b5ffb640 10 client.17784 handle_command_reply: tid=0 2024-09-19T08:13:07.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.113+0000 7f02d6643740 1 client.17784 shutdown 2024-09-19T08:13:07.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.113+0000 7f02d6643740 1 -- 172.21.15.98:0/3929411617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f7165d0 msgr2=0x558f8f725100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:07.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.113+0000 7f02d6643740 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f7165d0 0x558f8f725100 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f02c8012720 tx=0x7f02c8012be0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.113+0000 7f02d6643740 1 -- 172.21.15.98:0/3929411617 shutdown_connections 2024-09-19T08:13:07.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.113+0000 7f02d6643740 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x558f8f735010 0x558f8f737490 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.113+0000 7f02d6643740 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 0x558f8f722930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.113+0000 7f02d6643740 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f717b40 0x558f8f7223b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.113+0000 7f02b5ffb640 0 client.17784 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:13:07.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.114+0000 7f02d6643740 1 --2- 172.21.15.98:0/3929411617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f7165d0 0x558f8f725100 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.114+0000 7f02d6643740 1 -- 172.21.15.98:0/3929411617 >> 172.21.15.98:0/3929411617 conn(0x558f8f6df310 msgr2=0x558f8f718960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:07.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.114+0000 7f02d6643740 1 -- 172.21.15.98:0/3929411617 shutdown_connections 2024-09-19T08:13:07.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.114+0000 7f02d6643740 1 -- 172.21.15.98:0/3929411617 wait complete. 2024-09-19T08:13:07.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.114+0000 7f02d6643740 20 client.17784 trim_cache size 0 max 16384 2024-09-19T08:13:07.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.126+0000 7f02d6643740 1 Processor -- start 2024-09-19T08:13:07.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d6643740 1 -- start start 2024-09-19T08:13:07.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d6643740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f7165d0 0x558f8f731360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d6643740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f717b40 0x558f8f7213b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d6643740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 0x558f8f73cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d6643740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc1ab660 con 0x558f8f7165d0 2024-09-19T08:13:07.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d6643740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc1ad1a0 con 0x558f8f717b40 2024-09-19T08:13:07.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d6643740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc1abd30 con 0x558f8f725780 2024-09-19T08:13:07.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d2006640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f7165d0 0x558f8f731360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d2006640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f7165d0 0x558f8f731360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48790/0 (socket says 172.21.15.98:48790) 2024-09-19T08:13:07.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d2006640 1 -- 172.21.15.98:0/1109609917 learned_addr learned my addr 172.21.15.98:0/1109609917 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:07.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d1805640 1 --2- 172.21.15.98:0/1109609917 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f717b40 0x558f8f7213b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d2807640 1 --2- 172.21.15.98:0/1109609917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 0x558f8f73cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d2006640 1 -- 172.21.15.98:0/1109609917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 msgr2=0x558f8f73cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:07.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d2006640 1 --2- 172.21.15.98:0/1109609917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 0x558f8f73cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d2006640 1 -- 172.21.15.98:0/1109609917 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f717b40 msgr2=0x558f8f7213b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:07.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d2006640 1 --2- 172.21.15.98:0/1109609917 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f717b40 0x558f8f7213b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d2006640 1 -- 172.21.15.98:0/1109609917 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f8f71dfd0 con 0x558f8f7165d0 2024-09-19T08:13:07.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d1805640 1 --2- 172.21.15.98:0/1109609917 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f717b40 0x558f8f7213b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:07.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d2807640 1 --2- 172.21.15.98:0/1109609917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 0x558f8f73cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:07.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02d2006640 1 --2- 172.21.15.98:0/1109609917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f7165d0 0x558f8f731360 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f02c800a010 tx=0x7f02c80099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60efcb84bab455d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:07.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02b7fff640 1 -- 172.21.15.98:0/1109609917 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c8012070 con 0x558f8f7165d0 2024-09-19T08:13:07.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.127+0000 7f02b7fff640 1 -- 172.21.15.98:0/1109609917 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02c8012210 con 0x558f8f7165d0 2024-09-19T08:13:07.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.128+0000 7f02b7fff640 1 -- 172.21.15.98:0/1109609917 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c8023910 con 0x558f8f7165d0 2024-09-19T08:13:07.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.128+0000 7f02d6643740 1 -- 172.21.15.98:0/1109609917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f7165d0 msgr2=0x558f8f731360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:07.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.128+0000 7f02d6643740 1 --2- 172.21.15.98:0/1109609917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f7165d0 0x558f8f731360 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f02c800a010 tx=0x7f02c80099d0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.128+0000 7f02d6643740 1 -- 172.21.15.98:0/1109609917 shutdown_connections 2024-09-19T08:13:07.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.128+0000 7f02d6643740 1 --2- 172.21.15.98:0/1109609917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 0x558f8f73cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.128+0000 7f02d6643740 1 --2- 172.21.15.98:0/1109609917 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f717b40 0x558f8f7213b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.128+0000 7f02d6643740 1 --2- 172.21.15.98:0/1109609917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f7165d0 0x558f8f731360 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.128+0000 7f02d6643740 1 -- 172.21.15.98:0/1109609917 >> 172.21.15.98:0/1109609917 conn(0x558f8f6df310 msgr2=0x558f8f720ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:07.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.128+0000 7f02d6643740 1 -- 172.21.15.98:0/1109609917 shutdown_connections 2024-09-19T08:13:07.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.128+0000 7f02d6643740 1 -- 172.21.15.98:0/1109609917 wait complete. 2024-09-19T08:13:07.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.128+0000 7f02d6643740 1 -- start start 2024-09-19T08:13:07.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d6643740 1 Processor -- start 2024-09-19T08:13:07.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d6643740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f7165d0 0x558f8f71f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d6643740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f717b40 0x558f8f71f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d6643740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 0x558f8f71fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d6643740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc1acad0 con 0x558f8f717b40 2024-09-19T08:13:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d6643740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc1ac400 con 0x558f8f7165d0 2024-09-19T08:13:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d6643740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc1a9660 con 0x558f8f725780 2024-09-19T08:13:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d6643740 10 client.0 _ll_register_callbacks cb 0x7f02c80096e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:07.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d1805640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f717b40 0x558f8f71f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d1805640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f717b40 0x558f8f71f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48800/0 (socket says 172.21.15.98:48800) 2024-09-19T08:13:07.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d1805640 1 -- 172.21.15.98:0/2244423601 learned_addr learned my addr 172.21.15.98:0/2244423601 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:07.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d2006640 1 --2- 172.21.15.98:0/2244423601 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f7165d0 0x558f8f71f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d2807640 1 --2- 172.21.15.98:0/2244423601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 0x558f8f71fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d1805640 1 -- 172.21.15.98:0/2244423601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 msgr2=0x558f8f71fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:07.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d1805640 1 --2- 172.21.15.98:0/2244423601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 0x558f8f71fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d1805640 1 -- 172.21.15.98:0/2244423601 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f7165d0 msgr2=0x558f8f71f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:07.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d1805640 1 --2- 172.21.15.98:0/2244423601 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f7165d0 0x558f8f71f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d1805640 1 -- 172.21.15.98:0/2244423601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f8f79ed60 con 0x558f8f717b40 2024-09-19T08:13:07.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02d1805640 1 --2- 172.21.15.98:0/2244423601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f717b40 0x558f8f71f9b0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f02bc03d040 tx=0x7f02bc02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a760b40ade1f0ff server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:07.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:07.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.129+0000 7f02b5ffb640 1 -- 172.21.15.98:0/2244423601 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc049040 con 0x558f8f717b40 2024-09-19T08:13:07.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.130+0000 7f02b5ffb640 1 -- 172.21.15.98:0/2244423601 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f02bc03b460 con 0x558f8f717b40 2024-09-19T08:13:07.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.130+0000 7f02d6643740 1 -- 172.21.15.98:0/2244423601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02c8012210 con 0x558f8f717b40 2024-09-19T08:13:07.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.130+0000 7f02b5ffb640 1 -- 172.21.15.98:0/2244423601 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f02bc0bbe70 con 0x558f8f717b40 2024-09-19T08:13:07.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.130+0000 7f02d6643740 10 client.17790 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:07.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.130+0000 7f02d6643740 1 -- 172.21.15.98:0/2244423601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558f8f79fff0 con 0x558f8f717b40 2024-09-19T08:13:07.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.130+0000 7f02b5ffb640 1 -- 172.21.15.98:0/2244423601 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02c8012210 con 0x558f8f717b40 2024-09-19T08:13:07.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.130+0000 7f02b5ffb640 1 -- 172.21.15.98:0/2244423601 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f02bc0edc00 con 0x558f8f717b40 2024-09-19T08:13:07.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.130+0000 7f02d6643740 10 client.17790 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:07.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.130+0000 7f02d6643740 10 client.17790 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:13:07.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.130+0000 7f02d6643740 20 client.17790 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:07.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.130+0000 7f02d6643740 1 --2- 172.21.15.98:0/2244423601 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x558f8f735010 0x558f8f7a2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:07.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.130+0000 7f02d6643740 4 client.17790 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:13:07.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.130+0000 7f02d6643740 1 -- 172.21.15.98:0/2244423601 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558f8f683d70 con 0x558f8f735010 2024-09-19T08:13:07.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02d2006640 1 --2- 172.21.15.98:0/2244423601 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x558f8f735010 0x558f8f7a2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:07.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02d2006640 1 --2- 172.21.15.98:0/2244423601 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x558f8f735010 0x558f8f7a2480 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4a42384dd9f58351 server_cookie=7f21ab607a6fb60b in_seq=0 out_seq=0 2024-09-19T08:13:07.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02b5ffb640 10 client.17790 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:13:07.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02b5ffb640 1 -- 172.21.15.98:0/2244423601 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558f8f683d70 con 0x558f8f735010 2024-09-19T08:13:07.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02b5ffb640 10 client.17790 handle_command_reply: tid=0 2024-09-19T08:13:07.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02d6643740 1 client.17790 shutdown 2024-09-19T08:13:07.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02d6643740 1 -- 172.21.15.98:0/2244423601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f717b40 msgr2=0x558f8f71f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:07.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02d6643740 1 --2- 172.21.15.98:0/2244423601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f717b40 0x558f8f71f9b0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f02bc03d040 tx=0x7f02bc02a420 comp rx=0 tx=0).stop 2024-09-19T08:13:07.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02d6643740 1 -- 172.21.15.98:0/2244423601 shutdown_connections 2024-09-19T08:13:07.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02d6643740 1 --2- 172.21.15.98:0/2244423601 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x558f8f735010 0x558f8f7a2480 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02d6643740 1 --2- 172.21.15.98:0/2244423601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f8f725780 0x558f8f71fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02d6643740 1 --2- 172.21.15.98:0/2244423601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f8f717b40 0x558f8f71f9b0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02d6643740 1 --2- 172.21.15.98:0/2244423601 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f8f7165d0 0x558f8f71f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02d6643740 1 -- 172.21.15.98:0/2244423601 >> 172.21.15.98:0/2244423601 conn(0x558f8f6df310 msgr2=0x558f8f7214c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:07.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02b5ffb640 0 client.17790 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:13:07.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02d6643740 1 -- 172.21.15.98:0/2244423601 shutdown_connections 2024-09-19T08:13:07.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02d6643740 1 -- 172.21.15.98:0/2244423601 wait complete. 2024-09-19T08:13:07.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.131+0000 7f02d6643740 20 client.17790 trim_cache size 0 max 16384 2024-09-19T08:13:07.159 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:13:07.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.132+0000 7f02d426a640 1 -- 172.21.15.98:0/3405435557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f02b80a33b0 msgr2=0x7f02b80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:07.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.132+0000 7f02d426a640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f02b80a33b0 0x7f02b80a5870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f02c400ac40 tx=0x7f02c40046c0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.132+0000 7f02d426a640 1 -- 172.21.15.98:0/3405435557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc101420 msgr2=0x7f02cc1997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:07.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.132+0000 7f02d426a640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc101420 0x7f02cc1997c0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009230 tx=0x7f02bc004270 comp rx=0 tx=0).stop 2024-09-19T08:13:07.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.132+0000 7f02d426a640 1 -- 172.21.15.98:0/3405435557 shutdown_connections 2024-09-19T08:13:07.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.132+0000 7f02d426a640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f02b80a33b0 0x7f02b80a5870 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.132+0000 7f02d426a640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02cc106a80 0x7f02cc1a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.132+0000 7f02d426a640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc101420 0x7f02cc1997c0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.132+0000 7f02d426a640 1 --2- 172.21.15.98:0/3405435557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f02cc100a80 0x7f02cc199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:07.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.132+0000 7f02d426a640 1 -- 172.21.15.98:0/3405435557 >> 172.21.15.98:0/3405435557 conn(0x7f02cc0fc7a0 msgr2=0x7f02cc0fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:07.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.132+0000 7f02d426a640 1 -- 172.21.15.98:0/3405435557 shutdown_connections 2024-09-19T08:13:07.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:07.132+0000 7f02d426a640 1 -- 172.21.15.98:0/3405435557 wait complete. 2024-09-19T08:13:07.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:13:08.061 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:08.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:07 smithi098 ceph-mon[31333]: pgmap v312: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-09-19T08:13:08.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:07 smithi162 ceph-mon[35260]: pgmap v312: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-09-19T08:13:08.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:07 smithi139 ceph-mon[34874]: pgmap v312: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-09-19T08:13:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.503+0000 7f4ca64c5640 1 -- 172.21.15.98:0/223390503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca010c180 msgr2=0x7f4ca010e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.503+0000 7f4ca64c5640 1 --2- 172.21.15.98:0/223390503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca010c180 0x7f4ca010e600 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f4c98009ca0 tx=0x7f4c9802a950 comp rx=0 tx=0).stop 2024-09-19T08:13:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.503+0000 7f4ca64c5640 1 -- 172.21.15.98:0/223390503 shutdown_connections 2024-09-19T08:13:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.503+0000 7f4ca64c5640 1 --2- 172.21.15.98:0/223390503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca010c180 0x7f4ca010e600 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.503+0000 7f4ca64c5640 1 --2- 172.21.15.98:0/223390503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ca01036c0 0x7f4ca010bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.503+0000 7f4ca64c5640 1 --2- 172.21.15.98:0/223390503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ca0102da0 0x7f4ca0103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.503+0000 7f4ca64c5640 1 -- 172.21.15.98:0/223390503 >> 172.21.15.98:0/223390503 conn(0x7f4ca00fc7a0 msgr2=0x7f4ca00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.503+0000 7f4ca64c5640 1 -- 172.21.15.98:0/223390503 shutdown_connections 2024-09-19T08:13:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.503+0000 7f4ca64c5640 1 -- 172.21.15.98:0/223390503 wait complete. 2024-09-19T08:13:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.503+0000 7f4ca64c5640 1 Processor -- start 2024-09-19T08:13:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.503+0000 7f4ca64c5640 1 -- start start 2024-09-19T08:13:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca64c5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ca0102da0 0x7f4ca0198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca64c5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ca01036c0 0x7f4ca01988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca64c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca010c180 0x7f4ca019cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca64c5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca0111fa0 con 0x7f4ca010c180 2024-09-19T08:13:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca64c5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca0111e20 con 0x7f4ca01036c0 2024-09-19T08:13:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca64c5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca0112120 con 0x7f4ca0102da0 2024-09-19T08:13:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca4a62640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca010c180 0x7f4ca019cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca4a62640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca010c180 0x7f4ca019cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48818/0 (socket says 172.21.15.98:48818) 2024-09-19T08:13:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca4a62640 1 -- 172.21.15.98:0/2612626313 learned_addr learned my addr 172.21.15.98:0/2612626313 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4c9ffff640 1 --2- 172.21.15.98:0/2612626313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ca0102da0 0x7f4ca0198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4c9f7fe640 1 --2- 172.21.15.98:0/2612626313 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ca01036c0 0x7f4ca01988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca4a62640 1 -- 172.21.15.98:0/2612626313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ca0102da0 msgr2=0x7f4ca0198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca4a62640 1 --2- 172.21.15.98:0/2612626313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ca0102da0 0x7f4ca0198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca4a62640 1 -- 172.21.15.98:0/2612626313 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ca01036c0 msgr2=0x7f4ca01988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca4a62640 1 --2- 172.21.15.98:0/2612626313 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ca01036c0 0x7f4ca01988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca4a62640 1 -- 172.21.15.98:0/2612626313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca019d340 con 0x7f4ca010c180 2024-09-19T08:13:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.504+0000 7f4ca4a62640 1 --2- 172.21.15.98:0/2612626313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca010c180 0x7f4ca019cc60 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f4c98006010 tx=0x7f4c9802f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.505+0000 7f4c9d7fa640 1 -- 172.21.15.98:0/2612626313 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c98034040 con 0x7f4ca010c180 2024-09-19T08:13:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.505+0000 7f4c9d7fa640 1 -- 172.21.15.98:0/2612626313 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c98003e00 con 0x7f4ca010c180 2024-09-19T08:13:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.505+0000 7f4ca64c5640 1 -- 172.21.15.98:0/2612626313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca019d5d0 con 0x7f4ca010c180 2024-09-19T08:13:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.505+0000 7f4c9d7fa640 1 -- 172.21.15.98:0/2612626313 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c98045480 con 0x7f4ca010c180 2024-09-19T08:13:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.505+0000 7f4ca64c5640 1 -- 172.21.15.98:0/2612626313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca01a4f10 con 0x7f4ca010c180 2024-09-19T08:13:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.507+0000 7f4c9d7fa640 1 -- 172.21.15.98:0/2612626313 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f4c98037020 con 0x7f4ca010c180 2024-09-19T08:13:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.507+0000 7f4ca889e740 1 Processor -- start 2024-09-19T08:13:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.507+0000 7f4ca889e740 1 -- start start 2024-09-19T08:13:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.507+0000 7f4ca889e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c50b40 0x557461c4f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.507+0000 7f4ca889e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c4f5d0 0x557461c4fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.507+0000 7f4ca889e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c5a370 0x557461c5c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.507+0000 7f4ca889e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557461bb52f0 con 0x557461c5a370 2024-09-19T08:13:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.507+0000 7f4ca889e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557461b5ccf0 con 0x557461c50b40 2024-09-19T08:13:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.507+0000 7f4ca889e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557461bb12b0 con 0x557461c4f5d0 2024-09-19T08:13:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c9f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c5a370 0x557461c5c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c9f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c5a370 0x557461c5c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48820/0 (socket says 172.21.15.98:48820) 2024-09-19T08:13:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/4140219030 learned_addr learned my addr 172.21.15.98:0/4140219030 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c9d7fa640 1 --2- 172.21.15.98:0/2612626313 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4c8c0a33b0 0x7f4c8c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c9ffff640 1 --2- 172.21.15.98:0/4140219030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c4f5d0 0x557461c4fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c9f7fe640 1 --2- 172.21.15.98:0/4140219030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c50b40 0x557461c4f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4ca4a62640 1 --2- 172.21.15.98:0/2612626313 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4c8c0a33b0 0x7f4c8c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/4140219030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c4f5d0 msgr2=0x557461c4fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c9f7fe640 1 --2- 172.21.15.98:0/4140219030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c4f5d0 0x557461c4fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/4140219030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c50b40 msgr2=0x557461c4f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c9f7fe640 1 --2- 172.21.15.98:0/4140219030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c50b40 0x557461c4f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/4140219030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557461b413a0 con 0x557461c5a370 2024-09-19T08:13:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c9d7fa640 1 -- 172.21.15.98:0/2612626313 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f4c980f0c90 con 0x7f4ca010c180 2024-09-19T08:13:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c9f7fe640 1 --2- 172.21.15.98:0/4140219030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c5a370 0x557461c5c760 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f4c880115e0 tx=0x7f4c8800a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2359ed273c5a29d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4ca4a62640 1 --2- 172.21.15.98:0/2612626313 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4c8c0a33b0 0x7f4c8c0a5870 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4c980b94e0 tx=0x7f4c980e7e50 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c967fc640 1 -- 172.21.15.98:0/4140219030 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88005220 con 0x557461c5a370 2024-09-19T08:13:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c967fc640 1 -- 172.21.15.98:0/4140219030 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c88004c80 con 0x557461c5a370 2024-09-19T08:13:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.508+0000 7f4c967fc640 1 -- 172.21.15.98:0/4140219030 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8800fc50 con 0x557461c5a370 2024-09-19T08:13:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.509+0000 7f4ca889e740 1 -- 172.21.15.98:0/4140219030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c5a370 msgr2=0x557461c5c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.509+0000 7f4ca889e740 1 --2- 172.21.15.98:0/4140219030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c5a370 0x557461c5c760 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f4c880115e0 tx=0x7f4c8800a810 comp rx=0 tx=0).stop 2024-09-19T08:13:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.509+0000 7f4ca889e740 1 -- 172.21.15.98:0/4140219030 shutdown_connections 2024-09-19T08:13:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.509+0000 7f4ca889e740 1 --2- 172.21.15.98:0/4140219030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c5a370 0x557461c5c760 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.509+0000 7f4ca889e740 1 --2- 172.21.15.98:0/4140219030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c4f5d0 0x557461c4fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.509+0000 7f4ca889e740 1 --2- 172.21.15.98:0/4140219030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c50b40 0x557461c4f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.509+0000 7f4ca889e740 1 -- 172.21.15.98:0/4140219030 >> 172.21.15.98:0/4140219030 conn(0x557461c18310 msgr2=0x557461c4bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.509+0000 7f4ca889e740 1 -- 172.21.15.98:0/4140219030 shutdown_connections 2024-09-19T08:13:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.509+0000 7f4ca889e740 1 -- 172.21.15.98:0/4140219030 wait complete. 2024-09-19T08:13:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.509+0000 7f4ca889e740 1 -- start start 2024-09-19T08:13:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4ca889e740 1 Processor -- start 2024-09-19T08:13:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4ca889e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c59db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4ca889e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c50b40 0x557461c57060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4ca889e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c5a370 0x557461c577f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4ca889e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557461c61480 con 0x557461c4f5d0 2024-09-19T08:13:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4ca889e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557461b70060 con 0x557461c5a370 2024-09-19T08:13:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4ca889e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557461c61600 con 0x557461c50b40 2024-09-19T08:13:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4ca889e740 10 client.0 _ll_register_callbacks cb 0x5574618c38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4c9f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c59db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4c9f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c59db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48834/0 (socket says 172.21.15.98:48834) 2024-09-19T08:13:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/1639221551 learned_addr learned my addr 172.21.15.98:0/1639221551 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:08.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4c9ffff640 1 --2- 172.21.15.98:0/1639221551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c50b40 0x557461c57060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/1639221551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c50b40 msgr2=0x557461c57060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:08.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4c9f7fe640 1 --2- 172.21.15.98:0/1639221551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c50b40 0x557461c57060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/1639221551 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c5a370 msgr2=0x557461c577f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:08.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4c9f7fe640 1 --2- 172.21.15.98:0/1639221551 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c5a370 0x557461c577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.510+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/1639221551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574619263c0 con 0x557461c4f5d0 2024-09-19T08:13:08.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4c9f7fe640 1 --2- 172.21.15.98:0/1639221551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c59db0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800ae80 tx=0x7f4c8800a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e2e7c0ee30749f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:08.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4c7f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:08.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4c7f7fe640 1 -- 172.21.15.98:0/1639221551 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88029020 con 0x557461c4f5d0 2024-09-19T08:13:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4c7f7fe640 1 -- 172.21.15.98:0/1639221551 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f4c880048d0 con 0x557461c4f5d0 2024-09-19T08:13:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4ca889e740 1 -- 172.21.15.98:0/1639221551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c88004c80 con 0x557461c4f5d0 2024-09-19T08:13:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4c7f7fe640 1 -- 172.21.15.98:0/1639221551 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4c88034050 con 0x557461c4f5d0 2024-09-19T08:13:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4ca889e740 10 client.17802 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4ca889e740 1 -- 172.21.15.98:0/1639221551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557461c6cf70 con 0x557461c4f5d0 2024-09-19T08:13:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4c7f7fe640 1 -- 172.21.15.98:0/1639221551 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c88004c80 con 0x557461c4f5d0 2024-09-19T08:13:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4c7f7fe640 1 -- 172.21.15.98:0/1639221551 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4c8801b070 con 0x557461c4f5d0 2024-09-19T08:13:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4ca889e740 10 client.17802 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4ca889e740 10 client.17802 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:13:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4ca889e740 20 client.17802 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4ca889e740 1 --2- 172.21.15.98:0/1639221551 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x557461c6e2c0 0x557461c706b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4ca889e740 4 client.17802 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:13:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.511+0000 7f4ca889e740 1 -- 172.21.15.98:0/1639221551 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557461bc2860 con 0x557461c6e2c0 2024-09-19T08:13:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.512+0000 7f4c9ffff640 1 --2- 172.21.15.98:0/1639221551 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x557461c6e2c0 0x557461c706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.512+0000 7f4c7f7fe640 10 client.17802 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.512+0000 7f4c9ffff640 1 --2- 172.21.15.98:0/1639221551 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x557461c6e2c0 0x557461c706b0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fefb652a5973d578 server_cookie=8341ef04ae60b6fb in_seq=0 out_seq=0 2024-09-19T08:13:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4c7f7fe640 1 -- 172.21.15.98:0/1639221551 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557461bc2860 con 0x557461c6e2c0 2024-09-19T08:13:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4c7f7fe640 10 client.17802 handle_command_reply: tid=0 2024-09-19T08:13:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4ca889e740 1 client.17802 shutdown 2024-09-19T08:13:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4ca889e740 1 -- 172.21.15.98:0/1639221551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 msgr2=0x557461c59db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4ca889e740 1 --2- 172.21.15.98:0/1639221551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c59db0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800ae80 tx=0x7f4c8800a550 comp rx=0 tx=0).stop 2024-09-19T08:13:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4ca889e740 1 -- 172.21.15.98:0/1639221551 shutdown_connections 2024-09-19T08:13:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4ca889e740 1 --2- 172.21.15.98:0/1639221551 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x557461c6e2c0 0x557461c706b0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4ca889e740 1 --2- 172.21.15.98:0/1639221551 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c5a370 0x557461c577f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4c7f7fe640 0 client.17802 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4ca889e740 1 --2- 172.21.15.98:0/1639221551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c50b40 0x557461c57060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4ca889e740 1 --2- 172.21.15.98:0/1639221551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c59db0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4ca889e740 1 -- 172.21.15.98:0/1639221551 >> 172.21.15.98:0/1639221551 conn(0x557461c18310 msgr2=0x557461c51960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4ca889e740 1 -- 172.21.15.98:0/1639221551 shutdown_connections 2024-09-19T08:13:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4ca889e740 1 -- 172.21.15.98:0/1639221551 wait complete. 2024-09-19T08:13:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.513+0000 7f4ca889e740 20 client.17802 trim_cache size 0 max 16384 2024-09-19T08:13:08.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.526+0000 7f4ca889e740 1 Processor -- start 2024-09-19T08:13:08.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.526+0000 7f4ca889e740 1 -- start start 2024-09-19T08:13:08.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.526+0000 7f4ca889e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c56040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.526+0000 7f4ca889e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c50b40 0x557461c75e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.526+0000 7f4ca889e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c5a370 0x557461c76380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.526+0000 7f4ca889e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca01ab570 con 0x557461c4f5d0 2024-09-19T08:13:08.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.526+0000 7f4ca889e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca01ad0b0 con 0x557461c5a370 2024-09-19T08:13:08.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.526+0000 7f4ca889e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca01abc40 con 0x557461c50b40 2024-09-19T08:13:08.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.526+0000 7f4c9f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c56040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.526+0000 7f4c9f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c56040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48848/0 (socket says 172.21.15.98:48848) 2024-09-19T08:13:08.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.526+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/3990705939 learned_addr learned my addr 172.21.15.98:0/3990705939 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:08.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.527+0000 7f4c9ffff640 1 --2- 172.21.15.98:0/3990705939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c50b40 0x557461c75e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.527+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/3990705939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c50b40 msgr2=0x557461c75e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:08.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.527+0000 7f4c9f7fe640 1 --2- 172.21.15.98:0/3990705939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c50b40 0x557461c75e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.527+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/3990705939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c5a370 msgr2=0x557461c76380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:08.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.527+0000 7f4c9f7fe640 1 --2- 172.21.15.98:0/3990705939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c5a370 0x557461c76380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.527+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/3990705939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557461c555b0 con 0x557461c4f5d0 2024-09-19T08:13:08.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.527+0000 7f4c9ffff640 1 --2- 172.21.15.98:0/3990705939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c50b40 0x557461c75e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:08.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.527+0000 7f4c9f7fe640 1 --2- 172.21.15.98:0/3990705939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c56040 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800f780 tx=0x7f4c88005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a61d82af1fab9f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:08.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.527+0000 7f4c95ffb640 1 -- 172.21.15.98:0/3990705939 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88029070 con 0x557461c4f5d0 2024-09-19T08:13:08.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.527+0000 7f4c95ffb640 1 -- 172.21.15.98:0/3990705939 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c8800a2d0 con 0x557461c4f5d0 2024-09-19T08:13:08.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.527+0000 7f4c95ffb640 1 -- 172.21.15.98:0/3990705939 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88004b30 con 0x557461c4f5d0 2024-09-19T08:13:08.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.527+0000 7f4ca889e740 1 -- 172.21.15.98:0/3990705939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 msgr2=0x557461c56040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.527+0000 7f4ca889e740 1 --2- 172.21.15.98:0/3990705939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c56040 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800f780 tx=0x7f4c88005b10 comp rx=0 tx=0).stop 2024-09-19T08:13:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 -- 172.21.15.98:0/3990705939 shutdown_connections 2024-09-19T08:13:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 --2- 172.21.15.98:0/3990705939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c5a370 0x557461c76380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 --2- 172.21.15.98:0/3990705939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c50b40 0x557461c75e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 --2- 172.21.15.98:0/3990705939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c56040 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 -- 172.21.15.98:0/3990705939 >> 172.21.15.98:0/3990705939 conn(0x557461c18310 msgr2=0x557461c55930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 -- 172.21.15.98:0/3990705939 shutdown_connections 2024-09-19T08:13:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 -- 172.21.15.98:0/3990705939 wait complete. 2024-09-19T08:13:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 -- start start 2024-09-19T08:13:08.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 Processor -- start 2024-09-19T08:13:08.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c66e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c50b40 0x557461c62140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c5a370 0x557461c62680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca01ac9e0 con 0x557461c4f5d0 2024-09-19T08:13:08.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca01ac310 con 0x557461c50b40 2024-09-19T08:13:08.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca01a9570 con 0x557461c5a370 2024-09-19T08:13:08.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4ca889e740 10 client.0 _ll_register_callbacks cb 0x7f4c900093c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:08.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.528+0000 7f4c9f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c66e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4c9f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c66e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48856/0 (socket says 172.21.15.98:48856) 2024-09-19T08:13:08.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/3723827831 learned_addr learned my addr 172.21.15.98:0/3723827831 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:08.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4c9ffff640 1 --2- 172.21.15.98:0/3723827831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c50b40 0x557461c62140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/3723827831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c5a370 msgr2=0x557461c62680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4c9f7fe640 1 --2- 172.21.15.98:0/3723827831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c5a370 0x557461c62680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/3723827831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c50b40 msgr2=0x557461c62140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4c9f7fe640 1 --2- 172.21.15.98:0/3723827831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c50b40 0x557461c62140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4c9f7fe640 1 -- 172.21.15.98:0/3723827831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557461c62cc0 con 0x557461c4f5d0 2024-09-19T08:13:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4c9ffff640 1 --2- 172.21.15.98:0/3723827831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c50b40 0x557461c62140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4c9f7fe640 1 --2- 172.21.15.98:0/3723827831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c66e10 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f4c88004690 tx=0x7f4c88011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92abb7d644f3c7f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4c7f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4c7f7fe640 1 -- 172.21.15.98:0/3723827831 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88029070 con 0x557461c4f5d0 2024-09-19T08:13:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4c7f7fe640 1 -- 172.21.15.98:0/3723827831 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f4c880258a0 con 0x557461c4f5d0 2024-09-19T08:13:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4ca889e740 1 -- 172.21.15.98:0/3723827831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c8800a2d0 con 0x557461c4f5d0 2024-09-19T08:13:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4c7f7fe640 1 -- 172.21.15.98:0/3723827831 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4c880048d0 con 0x557461c4f5d0 2024-09-19T08:13:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4ca889e740 10 client.17808 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:08.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.529+0000 7f4ca889e740 1 -- 172.21.15.98:0/3723827831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557461cd8bb0 con 0x557461c4f5d0 2024-09-19T08:13:08.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.530+0000 7f4c7f7fe640 1 -- 172.21.15.98:0/3723827831 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c8800a2d0 con 0x557461c4f5d0 2024-09-19T08:13:08.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.530+0000 7f4c7f7fe640 1 -- 172.21.15.98:0/3723827831 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4c8801b040 con 0x557461c4f5d0 2024-09-19T08:13:08.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.530+0000 7f4ca889e740 10 client.17808 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:08.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.530+0000 7f4ca889e740 10 client.17808 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:13:08.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.530+0000 7f4ca889e740 20 client.17808 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:08.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.530+0000 7f4ca889e740 1 --2- 172.21.15.98:0/3723827831 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x557461cd9210 0x557461cdb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:08.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.530+0000 7f4ca889e740 4 client.17808 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:13:08.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.530+0000 7f4ca889e740 1 -- 172.21.15.98:0/3723827831 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557461bbcd70 con 0x557461cd9210 2024-09-19T08:13:08.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.530+0000 7f4c9ffff640 1 --2- 172.21.15.98:0/3723827831 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x557461cd9210 0x557461cdb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:08.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.530+0000 7f4c9ffff640 1 --2- 172.21.15.98:0/3723827831 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x557461cd9210 0x557461cdb630 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc2cdb97d9591026 server_cookie=9162ca8d1437146e in_seq=0 out_seq=0 2024-09-19T08:13:08.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.530+0000 7f4c7f7fe640 10 client.17808 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:08.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4c7f7fe640 1 -- 172.21.15.98:0/3723827831 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557461bbcd70 con 0x557461cd9210 2024-09-19T08:13:08.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4c7f7fe640 10 client.17808 handle_command_reply: tid=0 2024-09-19T08:13:08.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4ca889e740 1 client.17808 shutdown 2024-09-19T08:13:08.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4ca889e740 1 -- 172.21.15.98:0/3723827831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 msgr2=0x557461c66e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:08.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4ca889e740 1 --2- 172.21.15.98:0/3723827831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c66e10 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f4c88004690 tx=0x7f4c88011440 comp rx=0 tx=0).stop 2024-09-19T08:13:08.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4ca889e740 1 -- 172.21.15.98:0/3723827831 shutdown_connections 2024-09-19T08:13:08.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4ca889e740 1 --2- 172.21.15.98:0/3723827831 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x557461cd9210 0x557461cdb630 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4ca889e740 1 --2- 172.21.15.98:0/3723827831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557461c5a370 0x557461c62680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4ca889e740 1 --2- 172.21.15.98:0/3723827831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557461c50b40 0x557461c62140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4ca889e740 1 --2- 172.21.15.98:0/3723827831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557461c4f5d0 0x557461c66e10 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4ca889e740 1 -- 172.21.15.98:0/3723827831 >> 172.21.15.98:0/3723827831 conn(0x557461c18310 msgr2=0x557461c56150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:08.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4c7f7fe640 0 client.17808 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:08.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4ca889e740 1 -- 172.21.15.98:0/3723827831 shutdown_connections 2024-09-19T08:13:08.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4ca889e740 1 -- 172.21.15.98:0/3723827831 wait complete. 2024-09-19T08:13:08.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.531+0000 7f4ca889e740 20 client.17808 trim_cache size 0 max 16384 2024-09-19T08:13:08.539 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:13:08.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.532+0000 7f4ca64c5640 1 -- 172.21.15.98:0/2612626313 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4c8c0a33b0 msgr2=0x7f4c8c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:08.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.532+0000 7f4ca64c5640 1 --2- 172.21.15.98:0/2612626313 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4c8c0a33b0 0x7f4c8c0a5870 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4c980b94e0 tx=0x7f4c980e7e50 comp rx=0 tx=0).stop 2024-09-19T08:13:08.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.532+0000 7f4ca64c5640 1 -- 172.21.15.98:0/2612626313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca010c180 msgr2=0x7f4ca019cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:08.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.532+0000 7f4ca64c5640 1 --2- 172.21.15.98:0/2612626313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca010c180 0x7f4ca019cc60 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f4c98006010 tx=0x7f4c9802f710 comp rx=0 tx=0).stop 2024-09-19T08:13:08.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.532+0000 7f4ca64c5640 1 -- 172.21.15.98:0/2612626313 shutdown_connections 2024-09-19T08:13:08.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.532+0000 7f4ca64c5640 1 --2- 172.21.15.98:0/2612626313 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f4c8c0a33b0 0x7f4c8c0a5870 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.532+0000 7f4ca64c5640 1 --2- 172.21.15.98:0/2612626313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca010c180 0x7f4ca019cc60 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.532+0000 7f4ca64c5640 1 --2- 172.21.15.98:0/2612626313 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ca01036c0 0x7f4ca01988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.532+0000 7f4ca64c5640 1 --2- 172.21.15.98:0/2612626313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ca0102da0 0x7f4ca0198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:08.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.532+0000 7f4ca64c5640 1 -- 172.21.15.98:0/2612626313 >> 172.21.15.98:0/2612626313 conn(0x7f4ca00fc7a0 msgr2=0x7f4ca00fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:08.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.532+0000 7f4ca64c5640 1 -- 172.21.15.98:0/2612626313 shutdown_connections 2024-09-19T08:13:08.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:08.532+0000 7f4ca64c5640 1 -- 172.21.15.98:0/2612626313 wait complete. 2024-09-19T08:13:10.316 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-19T08:13:10.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-19T08:13:10.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:09 smithi098 ceph-mon[31333]: pgmap v313: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:13:10.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:09 smithi162 ceph-mon[35260]: pgmap v313: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:13:10.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:09 smithi139 ceph-mon[34874]: pgmap v313: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:13:10.576 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.089+0000 7fb3b385d640 1 -- 172.21.15.98:0/1086564645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac102790 msgr2=0x7fb3ac1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.089+0000 7fb3b385d640 1 --2- 172.21.15.98:0/1086564645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac102790 0x7fb3ac1067e0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0009900 tx=0x7fb3a002a410 comp rx=0 tx=0).stop 2024-09-19T08:13:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.089+0000 7fb3b385d640 1 -- 172.21.15.98:0/1086564645 shutdown_connections 2024-09-19T08:13:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.089+0000 7fb3b385d640 1 --2- 172.21.15.98:0/1086564645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3ac106f10 0x7fb3ac10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.089+0000 7fb3b385d640 1 --2- 172.21.15.98:0/1086564645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac102790 0x7fb3ac1067e0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.089+0000 7fb3b385d640 1 --2- 172.21.15.98:0/1086564645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3ac101de0 0x7fb3ac1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.089+0000 7fb3b385d640 1 -- 172.21.15.98:0/1086564645 >> 172.21.15.98:0/1086564645 conn(0x7fb3ac0fd880 msgr2=0x7fb3ac0ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.089+0000 7fb3b385d640 1 -- 172.21.15.98:0/1086564645 shutdown_connections 2024-09-19T08:13:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.089+0000 7fb3b385d640 1 -- 172.21.15.98:0/1086564645 wait complete. 2024-09-19T08:13:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.090+0000 7fb3b385d640 1 Processor -- start 2024-09-19T08:13:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.090+0000 7fb3b385d640 1 -- start start 2024-09-19T08:13:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.090+0000 7fb3b385d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3ac101de0 0x7fb3ac0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.090+0000 7fb3b385d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac102790 0x7fb3ac074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.090+0000 7fb3b385d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3ac106f10 0x7fb3ac074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.090+0000 7fb3b385d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac111f70 con 0x7fb3ac102790 2024-09-19T08:13:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.090+0000 7fb3b385d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac111df0 con 0x7fb3ac106f10 2024-09-19T08:13:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.090+0000 7fb3b385d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1120f0 con 0x7fb3ac101de0 2024-09-19T08:13:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.090+0000 7fb3b0df8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac102790 0x7fb3ac074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.090+0000 7fb3b0df8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac102790 0x7fb3ac074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48864/0 (socket says 172.21.15.98:48864) 2024-09-19T08:13:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.090+0000 7fb3b0df8640 1 -- 172.21.15.98:0/4016887498 learned_addr learned my addr 172.21.15.98:0/4016887498 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.090+0000 7fb3b15f9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3ac101de0 0x7fb3ac0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.091+0000 7fb3b1dfa640 1 --2- 172.21.15.98:0/4016887498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3ac106f10 0x7fb3ac074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.091+0000 7fb3b0df8640 1 -- 172.21.15.98:0/4016887498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3ac101de0 msgr2=0x7fb3ac0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.091+0000 7fb3b0df8640 1 --2- 172.21.15.98:0/4016887498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3ac101de0 0x7fb3ac0744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.091+0000 7fb3b0df8640 1 -- 172.21.15.98:0/4016887498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3ac106f10 msgr2=0x7fb3ac074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.091+0000 7fb3b0df8640 1 --2- 172.21.15.98:0/4016887498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3ac106f10 0x7fb3ac074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.091+0000 7fb3b0df8640 1 -- 172.21.15.98:0/4016887498 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ac1a68f0 con 0x7fb3ac102790 2024-09-19T08:13:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.091+0000 7fb3b1dfa640 1 --2- 172.21.15.98:0/4016887498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3ac106f10 0x7fb3ac074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.091+0000 7fb3b0df8640 1 --2- 172.21.15.98:0/4016887498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac102790 0x7fb3ac074a30 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0006010 tx=0x7fb3a00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.091+0000 7fb39e7fc640 1 -- 172.21.15.98:0/4016887498 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a003f030 con 0x7fb3ac102790 2024-09-19T08:13:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.091+0000 7fb3b385d640 1 -- 172.21.15.98:0/4016887498 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3ac1a6b80 con 0x7fb3ac102790 2024-09-19T08:13:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.091+0000 7fb39e7fc640 1 -- 172.21.15.98:0/4016887498 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3a0003ce0 con 0x7fb3ac102790 2024-09-19T08:13:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.091+0000 7fb39e7fc640 1 -- 172.21.15.98:0/4016887498 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0035070 con 0x7fb3ac102790 2024-09-19T08:13:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.091+0000 7fb3b385d640 1 -- 172.21.15.98:0/4016887498 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3ac1a7060 con 0x7fb3ac102790 2024-09-19T08:13:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.094+0000 7fb39e7fc640 1 -- 172.21.15.98:0/4016887498 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fb3a0042030 con 0x7fb3ac102790 2024-09-19T08:13:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.094+0000 7fb3b5c36740 1 Processor -- start 2024-09-19T08:13:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b5c36740 1 -- start start 2024-09-19T08:13:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b5c36740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2524b60 0x55e9f2524f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b5c36740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f2525480 0x55e9f252f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b5c36740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f252fd40 0x55e9f2532130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b5c36740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e9f248a280 con 0x55e9f252fd40 2024-09-19T08:13:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b5c36740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e9f2431b10 con 0x55e9f2524b60 2024-09-19T08:13:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b5c36740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e9f2486240 con 0x55e9f2525480 2024-09-19T08:13:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b0df8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f252fd40 0x55e9f2532130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b15f9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f2525480 0x55e9f252f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b0df8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f252fd40 0x55e9f2532130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48876/0 (socket says 172.21.15.98:48876) 2024-09-19T08:13:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b1dfa640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2524b60 0x55e9f2524f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b0df8640 1 -- 172.21.15.98:0/4217640085 learned_addr learned my addr 172.21.15.98:0/4217640085 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b15f9640 1 -- 172.21.15.98:0/4217640085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2524b60 msgr2=0x55e9f2524f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b15f9640 1 --2- 172.21.15.98:0/4217640085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2524b60 0x55e9f2524f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b15f9640 1 -- 172.21.15.98:0/4217640085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f252fd40 msgr2=0x55e9f2532130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b15f9640 1 --2- 172.21.15.98:0/4217640085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f252fd40 0x55e9f2532130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb3b15f9640 1 -- 172.21.15.98:0/4217640085 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9f243e1f0 con 0x55e9f2525480 2024-09-19T08:13:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.095+0000 7fb39e7fc640 1 --2- 172.21.15.98:0/4016887498 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb3980a33b0 0x7fb3980a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.096+0000 7fb3b15f9640 1 --2- 172.21.15.98:0/4217640085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f2525480 0x55e9f252f800 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8005fd0 tx=0x7fb3a80107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b2cd30aeaf451c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.096+0000 7fb3b1dfa640 1 --2- 172.21.15.98:0/4016887498 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb3980a33b0 0x7fb3980a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.096+0000 7fb39e7fc640 1 -- 172.21.15.98:0/4016887498 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8565+0+0 (secure 0 0 0) 0x7fb3a00f0260 con 0x7fb3ac102790 2024-09-19T08:13:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.096+0000 7fb3977fe640 1 -- 172.21.15.98:0/4217640085 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a801c480 con 0x55e9f2525480 2024-09-19T08:13:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.096+0000 7fb3977fe640 1 -- 172.21.15.98:0/4217640085 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3a801cb00 con 0x55e9f2525480 2024-09-19T08:13:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.096+0000 7fb3977fe640 1 -- 172.21.15.98:0/4217640085 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a8011c30 con 0x55e9f2525480 2024-09-19T08:13:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.097+0000 7fb3b5c36740 1 -- 172.21.15.98:0/4217640085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f2525480 msgr2=0x55e9f252f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.097+0000 7fb3b1dfa640 1 --2- 172.21.15.98:0/4016887498 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb3980a33b0 0x7fb3980a5870 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4005fd0 tx=0x7fb3a4009780 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.097+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/4217640085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f2525480 0x55e9f252f800 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8005fd0 tx=0x7fb3a80107a0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.097+0000 7fb3b5c36740 1 -- 172.21.15.98:0/4217640085 shutdown_connections 2024-09-19T08:13:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.097+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/4217640085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f252fd40 0x55e9f2532130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.097+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/4217640085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f2525480 0x55e9f252f800 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.097+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/4217640085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2524b60 0x55e9f2524f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.097+0000 7fb3b5c36740 1 -- 172.21.15.98:0/4217640085 >> 172.21.15.98:0/4217640085 conn(0x55e9f25210f0 msgr2=0x55e9f25214d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:11.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.097+0000 7fb3b5c36740 1 -- 172.21.15.98:0/4217640085 shutdown_connections 2024-09-19T08:13:11.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.097+0000 7fb3b5c36740 1 -- 172.21.15.98:0/4217640085 wait complete. 2024-09-19T08:13:11.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.097+0000 7fb3b5c36740 1 -- start start 2024-09-19T08:13:11.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.097+0000 7fb3b5c36740 1 Processor -- start 2024-09-19T08:13:11.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b5c36740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2524b60 0x55e9f252f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b5c36740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2525480 0x55e9f252cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b5c36740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f252fd40 0x55e9f252d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b5c36740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e9f2532910 con 0x55e9f2524b60 2024-09-19T08:13:11.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b5c36740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e9f2444f10 con 0x55e9f2525480 2024-09-19T08:13:11.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b5c36740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e9f2532a90 con 0x55e9f252fd40 2024-09-19T08:13:11.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b15f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2524b60 0x55e9f252f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b5c36740 10 client.0 _ll_register_callbacks cb 0x55e9f240da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:11.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b15f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2524b60 0x55e9f252f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48892/0 (socket says 172.21.15.98:48892) 2024-09-19T08:13:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b15f9640 1 -- 172.21.15.98:0/3807615448 learned_addr learned my addr 172.21.15.98:0/3807615448 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b0df8640 1 --2- 172.21.15.98:0/3807615448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2525480 0x55e9f252cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b1dfa640 1 --2- 172.21.15.98:0/3807615448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f252fd40 0x55e9f252d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b15f9640 1 -- 172.21.15.98:0/3807615448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f252fd40 msgr2=0x55e9f252d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b15f9640 1 --2- 172.21.15.98:0/3807615448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f252fd40 0x55e9f252d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b15f9640 1 -- 172.21.15.98:0/3807615448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2525480 msgr2=0x55e9f252cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b15f9640 1 --2- 172.21.15.98:0/3807615448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2525480 0x55e9f252cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b15f9640 1 -- 172.21.15.98:0/3807615448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9f21fb6a0 con 0x55e9f2524b60 2024-09-19T08:13:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb3b15f9640 1 --2- 172.21.15.98:0/3807615448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2524b60 0x55e9f252f800 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fb3a801cec0 tx=0x7fb3a801cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd2a9b27fbbc764 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.098+0000 7fb394ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.099+0000 7fb394ff9640 1 -- 172.21.15.98:0/3807615448 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a802b060 con 0x55e9f2524b60 2024-09-19T08:13:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.099+0000 7fb394ff9640 1 -- 172.21.15.98:0/3807615448 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8565+0+0 (secure 0 0 0) 0x7fb3a801c480 con 0x55e9f2524b60 2024-09-19T08:13:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.099+0000 7fb3b5c36740 1 -- 172.21.15.98:0/3807615448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3a801cb00 con 0x55e9f2524b60 2024-09-19T08:13:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.099+0000 7fb394ff9640 1 -- 172.21.15.98:0/3807615448 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb3a801c680 con 0x55e9f2524b60 2024-09-19T08:13:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.099+0000 7fb3b5c36740 10 client.17820 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.099+0000 7fb3b5c36740 1 -- 172.21.15.98:0/3807615448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e9f253e470 con 0x55e9f2524b60 2024-09-19T08:13:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.099+0000 7fb394ff9640 1 -- 172.21.15.98:0/3807615448 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3a801cb00 con 0x55e9f2524b60 2024-09-19T08:13:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.099+0000 7fb394ff9640 1 -- 172.21.15.98:0/3807615448 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb3a802e580 con 0x55e9f2524b60 2024-09-19T08:13:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.099+0000 7fb3b5c36740 10 client.17820 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.099+0000 7fb3b5c36740 10 client.17820 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:13:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.099+0000 7fb3b5c36740 20 client.17820 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.099+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/3807615448 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e9f253f930 0x55e9f2541d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.100+0000 7fb3b5c36740 4 client.17820 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:13:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.100+0000 7fb3b5c36740 1 -- 172.21.15.98:0/3807615448 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9f24977f0 con 0x55e9f253f930 2024-09-19T08:13:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.100+0000 7fb3b0df8640 1 --2- 172.21.15.98:0/3807615448 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e9f253f930 0x55e9f2541d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.100+0000 7fb394ff9640 10 client.17820 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.100+0000 7fb3b0df8640 1 --2- 172.21.15.98:0/3807615448 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e9f253f930 0x55e9f2541d20 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d96ad4600391f341 server_cookie=c7fd01f44725ca28 in_seq=0 out_seq=0 2024-09-19T08:13:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb394ff9640 1 -- 172.21.15.98:0/3807615448 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e9f24977f0 con 0x55e9f253f930 2024-09-19T08:13:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb394ff9640 10 client.17820 handle_command_reply: tid=0 2024-09-19T08:13:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb3b5c36740 1 client.17820 shutdown 2024-09-19T08:13:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb3b5c36740 1 -- 172.21.15.98:0/3807615448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2524b60 msgr2=0x55e9f252f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/3807615448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2524b60 0x55e9f252f800 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fb3a801cec0 tx=0x7fb3a801cef0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb3b5c36740 1 -- 172.21.15.98:0/3807615448 shutdown_connections 2024-09-19T08:13:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/3807615448 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e9f253f930 0x55e9f2541d20 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/3807615448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f252fd40 0x55e9f252d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/3807615448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2525480 0x55e9f252cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/3807615448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2524b60 0x55e9f252f800 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb3b5c36740 1 -- 172.21.15.98:0/3807615448 >> 172.21.15.98:0/3807615448 conn(0x55e9f25210f0 msgr2=0x55e9f2526dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb394ff9640 0 client.17820 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:11.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb3b5c36740 1 -- 172.21.15.98:0/3807615448 shutdown_connections 2024-09-19T08:13:11.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb3b5c36740 1 -- 172.21.15.98:0/3807615448 wait complete. 2024-09-19T08:13:11.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.101+0000 7fb3b5c36740 20 client.17820 trim_cache size 0 max 16384 2024-09-19T08:13:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.118+0000 7fb3b5c36740 1 Processor -- start 2024-09-19T08:13:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.118+0000 7fb3b5c36740 1 -- start start 2024-09-19T08:13:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b5c36740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2524b60 0x55e9f252e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b5c36740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2525480 0x55e9f2547320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b5c36740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f252fd40 0x55e9f2547860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b5c36740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1afab0 con 0x55e9f2525480 2024-09-19T08:13:11.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b5c36740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1b15f0 con 0x55e9f2524b60 2024-09-19T08:13:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b5c36740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1b0180 con 0x55e9f252fd40 2024-09-19T08:13:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b0df8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2525480 0x55e9f2547320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b0df8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2525480 0x55e9f2547320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48896/0 (socket says 172.21.15.98:48896) 2024-09-19T08:13:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b0df8640 1 -- 172.21.15.98:0/2247869821 learned_addr learned my addr 172.21.15.98:0/2247869821 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b15f9640 1 --2- 172.21.15.98:0/2247869821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2524b60 0x55e9f252e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b1dfa640 1 --2- 172.21.15.98:0/2247869821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f252fd40 0x55e9f2547860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b0df8640 1 -- 172.21.15.98:0/2247869821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f252fd40 msgr2=0x55e9f2547860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b0df8640 1 --2- 172.21.15.98:0/2247869821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f252fd40 0x55e9f2547860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b0df8640 1 -- 172.21.15.98:0/2247869821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2524b60 msgr2=0x55e9f252e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b0df8640 1 --2- 172.21.15.98:0/2247869821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2524b60 0x55e9f252e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b0df8640 1 -- 172.21.15.98:0/2247869821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9f2520f20 con 0x55e9f2525480 2024-09-19T08:13:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b1dfa640 1 --2- 172.21.15.98:0/2247869821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f252fd40 0x55e9f2547860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb3b0df8640 1 --2- 172.21.15.98:0/2247869821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2525480 0x55e9f2547320 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00bc560 tx=0x7fb3a00f0620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39681a025d678a68 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:11.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb396ffd640 1 -- 172.21.15.98:0/2247869821 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0049040 con 0x55e9f2525480 2024-09-19T08:13:11.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb396ffd640 1 -- 172.21.15.98:0/2247869821 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3a003b580 con 0x55e9f2525480 2024-09-19T08:13:11.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.119+0000 7fb396ffd640 1 -- 172.21.15.98:0/2247869821 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a00bccd0 con 0x55e9f2525480 2024-09-19T08:13:11.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.120+0000 7fb3b5c36740 1 -- 172.21.15.98:0/2247869821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2525480 msgr2=0x55e9f2547320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:11.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.120+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/2247869821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2525480 0x55e9f2547320 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00bc560 tx=0x7fb3a00f0620 comp rx=0 tx=0).stop 2024-09-19T08:13:11.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.120+0000 7fb3b5c36740 1 -- 172.21.15.98:0/2247869821 shutdown_connections 2024-09-19T08:13:11.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.120+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/2247869821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f252fd40 0x55e9f2547860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.120+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/2247869821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2525480 0x55e9f2547320 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.120+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/2247869821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f2524b60 0x55e9f252e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.120+0000 7fb3b5c36740 1 -- 172.21.15.98:0/2247869821 >> 172.21.15.98:0/2247869821 conn(0x55e9f25210f0 msgr2=0x55e9f252e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:11.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.120+0000 7fb3b5c36740 1 -- 172.21.15.98:0/2247869821 shutdown_connections 2024-09-19T08:13:11.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.120+0000 7fb3b5c36740 1 -- 172.21.15.98:0/2247869821 wait complete. 2024-09-19T08:13:11.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.120+0000 7fb3b5c36740 1 -- start start 2024-09-19T08:13:11.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.120+0000 7fb3b5c36740 1 Processor -- start 2024-09-19T08:13:11.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b5c36740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2524b60 0x55e9f2530eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b5c36740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f2525480 0x55e9f25313f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b5c36740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f252fd40 0x55e9f2531930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b5c36740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1b0f20 con 0x55e9f2524b60 2024-09-19T08:13:11.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b5c36740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1b0850 con 0x55e9f252fd40 2024-09-19T08:13:11.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b5c36740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1adab0 con 0x55e9f2525480 2024-09-19T08:13:11.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b5c36740 10 client.0 _ll_register_callbacks cb 0x7fb3a00f3c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:11.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b15f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2524b60 0x55e9f2530eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b15f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2524b60 0x55e9f2530eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48898/0 (socket says 172.21.15.98:48898) 2024-09-19T08:13:11.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b15f9640 1 -- 172.21.15.98:0/2276245514 learned_addr learned my addr 172.21.15.98:0/2276245514 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:11.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b0df8640 1 --2- 172.21.15.98:0/2276245514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f2525480 0x55e9f25313f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b1dfa640 1 --2- 172.21.15.98:0/2276245514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f252fd40 0x55e9f2531930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b15f9640 1 -- 172.21.15.98:0/2276245514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f2525480 msgr2=0x55e9f25313f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b15f9640 1 --2- 172.21.15.98:0/2276245514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f2525480 0x55e9f25313f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b15f9640 1 -- 172.21.15.98:0/2276245514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f252fd40 msgr2=0x55e9f2531930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b15f9640 1 --2- 172.21.15.98:0/2276245514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f252fd40 0x55e9f2531930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b15f9640 1 -- 172.21.15.98:0/2276245514 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9f2536260 con 0x55e9f2524b60 2024-09-19T08:13:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.121+0000 7fb3b1dfa640 1 --2- 172.21.15.98:0/2276245514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f252fd40 0x55e9f2531930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb3b15f9640 1 --2- 172.21.15.98:0/2276245514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2524b60 0x55e9f2530eb0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8000c00 tx=0x7fb3a800bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959bfc1391173d24 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb394ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb394ff9640 1 -- 172.21.15.98:0/2276245514 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a801c820 con 0x55e9f2524b60 2024-09-19T08:13:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb394ff9640 1 -- 172.21.15.98:0/2276245514 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8565+0+0 (secure 0 0 0) 0x7fb3a801c9c0 con 0x55e9f2524b60 2024-09-19T08:13:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb3b5c36740 1 -- 172.21.15.98:0/2276245514 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e9f2542110 con 0x55e9f2524b60 2024-09-19T08:13:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb394ff9640 1 -- 172.21.15.98:0/2276245514 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb3a802e3e0 con 0x55e9f2524b60 2024-09-19T08:13:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb3b5c36740 10 client.17826 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb3b5c36740 1 -- 172.21.15.98:0/2276245514 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e9f25aa530 con 0x55e9f2524b60 2024-09-19T08:13:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb394ff9640 1 -- 172.21.15.98:0/2276245514 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e9f2542110 con 0x55e9f2524b60 2024-09-19T08:13:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb394ff9640 1 -- 172.21.15.98:0/2276245514 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb3a8011a40 con 0x55e9f2524b60 2024-09-19T08:13:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb3b5c36740 10 client.17826 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb3b5c36740 10 client.17826 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:13:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb3b5c36740 20 client.17826 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/2276245514 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e9f25aac10 0x55e9f25ad030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb3b5c36740 4 client.17826 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-19T08:13:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.122+0000 7fb3b5c36740 1 -- 172.21.15.98:0/2276245514 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e9f2491d00 con 0x55e9f25aac10 2024-09-19T08:13:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.123+0000 7fb3b0df8640 1 --2- 172.21.15.98:0/2276245514 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e9f25aac10 0x55e9f25ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.123+0000 7fb3b0df8640 1 --2- 172.21.15.98:0/2276245514 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e9f25aac10 0x55e9f25ad030 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae995acf71cf2ec4 server_cookie=e2e5b34da12a37c7 in_seq=0 out_seq=0 2024-09-19T08:13:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.123+0000 7fb394ff9640 10 client.17826 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:11.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.124+0000 7fb394ff9640 1 -- 172.21.15.98:0/2276245514 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e9f2491d00 con 0x55e9f25aac10 2024-09-19T08:13:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.124+0000 7fb394ff9640 10 client.17826 handle_command_reply: tid=0 2024-09-19T08:13:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.124+0000 7fb3b5c36740 1 client.17826 shutdown 2024-09-19T08:13:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.124+0000 7fb3b5c36740 1 -- 172.21.15.98:0/2276245514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2524b60 msgr2=0x55e9f2530eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.124+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/2276245514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2524b60 0x55e9f2530eb0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8000c00 tx=0x7fb3a800bb20 comp rx=0 tx=0).stop 2024-09-19T08:13:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.124+0000 7fb3b5c36740 1 -- 172.21.15.98:0/2276245514 shutdown_connections 2024-09-19T08:13:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.124+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/2276245514 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55e9f25aac10 0x55e9f25ad030 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/2276245514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e9f252fd40 0x55e9f2531930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb394ff9640 0 client.17826 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/2276245514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f2525480 0x55e9f25313f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b5c36740 1 --2- 172.21.15.98:0/2276245514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9f2524b60 0x55e9f2530eb0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b5c36740 1 -- 172.21.15.98:0/2276245514 >> 172.21.15.98:0/2276245514 conn(0x55e9f25210f0 msgr2=0x55e9f2520270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b5c36740 1 -- 172.21.15.98:0/2276245514 shutdown_connections 2024-09-19T08:13:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b5c36740 1 -- 172.21.15.98:0/2276245514 wait complete. 2024-09-19T08:13:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b5c36740 20 client.17826 trim_cache size 0 max 16384 2024-09-19T08:13:11.130 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:13:11.130 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-09-19T08:13:11.130 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "fc3b494f-8edf-45b6-b770-a2bd83af103b", 2024-09-19T08:13:11.130 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-09-19T08:13:11.130 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:13:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b385d640 1 -- 172.21.15.98:0/4016887498 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb3980a33b0 msgr2=0x7fb3980a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b385d640 1 --2- 172.21.15.98:0/4016887498 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb3980a33b0 0x7fb3980a5870 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4005fd0 tx=0x7fb3a4009780 comp rx=0 tx=0).stop 2024-09-19T08:13:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b385d640 1 -- 172.21.15.98:0/4016887498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac102790 msgr2=0x7fb3ac074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b385d640 1 --2- 172.21.15.98:0/4016887498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac102790 0x7fb3ac074a30 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0006010 tx=0x7fb3a00042f0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b385d640 1 -- 172.21.15.98:0/4016887498 shutdown_connections 2024-09-19T08:13:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b385d640 1 --2- 172.21.15.98:0/4016887498 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb3980a33b0 0x7fb3980a5870 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b385d640 1 --2- 172.21.15.98:0/4016887498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3ac106f10 0x7fb3ac074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b385d640 1 --2- 172.21.15.98:0/4016887498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac102790 0x7fb3ac074a30 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b385d640 1 --2- 172.21.15.98:0/4016887498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3ac101de0 0x7fb3ac0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.125+0000 7fb3b385d640 1 -- 172.21.15.98:0/4016887498 >> 172.21.15.98:0/4016887498 conn(0x7fb3ac0fd880 msgr2=0x7fb3ac0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.126+0000 7fb3b385d640 1 -- 172.21.15.98:0/4016887498 shutdown_connections 2024-09-19T08:13:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.126+0000 7fb3b385d640 1 -- 172.21.15.98:0/4016887498 wait complete. 2024-09-19T08:13:11.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:13:11.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-09-19T08:13:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.969+0000 7f08431e8640 1 -- 172.21.15.98:0/191740822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1069c0 msgr2=0x7f083c106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.969+0000 7f08431e8640 1 --2- 172.21.15.98:0/191740822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1069c0 0x7f083c106da0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f0838009930 tx=0x7f083802a420 comp rx=0 tx=0).stop 2024-09-19T08:13:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.969+0000 7f08431e8640 1 -- 172.21.15.98:0/191740822 shutdown_connections 2024-09-19T08:13:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.969+0000 7f08431e8640 1 --2- 172.21.15.98:0/191740822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f083c101350 0x7f083c10d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.969+0000 7f08431e8640 1 --2- 172.21.15.98:0/191740822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f083c1009b0 0x7f083c100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.969+0000 7f08431e8640 1 --2- 172.21.15.98:0/191740822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1069c0 0x7f083c106da0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.969+0000 7f08431e8640 1 -- 172.21.15.98:0/191740822 >> 172.21.15.98:0/191740822 conn(0x7f083c077990 msgr2=0x7f083c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.969+0000 7f08431e8640 1 -- 172.21.15.98:0/191740822 shutdown_connections 2024-09-19T08:13:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.969+0000 7f08431e8640 1 -- 172.21.15.98:0/191740822 wait complete. 2024-09-19T08:13:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.969+0000 7f08431e8640 1 Processor -- start 2024-09-19T08:13:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f08431e8640 1 -- start start 2024-09-19T08:13:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f08431e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1009b0 0x7f083c197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f08431e8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f083c101350 0x7f083c1977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f08431e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f083c1069c0 0x7f083c19bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f08431e8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f083c110dc0 con 0x7f083c1009b0 2024-09-19T08:13:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f08431e8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f083c110c40 con 0x7f083c101350 2024-09-19T08:13:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f08431e8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f083c110f40 con 0x7f083c1069c0 2024-09-19T08:13:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f0840f84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1009b0 0x7f083c197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f0840f84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1009b0 0x7f083c197260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48908/0 (socket says 172.21.15.98:48908) 2024-09-19T08:13:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f0840f84640 1 -- 172.21.15.98:0/2258333715 learned_addr learned my addr 172.21.15.98:0/2258333715 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f0841785640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f083c1069c0 0x7f083c19bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f0833fff640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f083c101350 0x7f083c1977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f0840f84640 1 -- 172.21.15.98:0/2258333715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f083c1069c0 msgr2=0x7f083c19bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f0840f84640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f083c1069c0 0x7f083c19bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f0840f84640 1 -- 172.21.15.98:0/2258333715 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f083c101350 msgr2=0x7f083c1977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f0840f84640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f083c101350 0x7f083c1977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f0840f84640 1 -- 172.21.15.98:0/2258333715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083c19c250 con 0x7f083c1009b0 2024-09-19T08:13:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.970+0000 7f0833fff640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f083c101350 0x7f083c1977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.971+0000 7f0840f84640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1009b0 0x7f083c197260 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f0838009230 tx=0x7f0838004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.971+0000 7f0831ffb640 1 -- 172.21.15.98:0/2258333715 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083803f030 con 0x7f083c1009b0 2024-09-19T08:13:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.971+0000 7f0831ffb640 1 -- 172.21.15.98:0/2258333715 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0838003ad0 con 0x7f083c1009b0 2024-09-19T08:13:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.971+0000 7f08431e8640 1 -- 172.21.15.98:0/2258333715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f083c19c4e0 con 0x7f083c1009b0 2024-09-19T08:13:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.971+0000 7f0831ffb640 1 -- 172.21.15.98:0/2258333715 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0838035070 con 0x7f083c1009b0 2024-09-19T08:13:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.971+0000 7f08431e8640 1 -- 172.21.15.98:0/2258333715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f083c1a3dd0 con 0x7f083c1009b0 2024-09-19T08:13:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.972+0000 7f08431e8640 1 -- 172.21.15.98:0/2258333715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f083c100e10 con 0x7f083c1009b0 2024-09-19T08:13:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.974+0000 7f0831ffb640 1 -- 172.21.15.98:0/2258333715 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f0838042030 con 0x7f083c1009b0 2024-09-19T08:13:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.974+0000 7f0831ffb640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08240a34e0 0x7f08240a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.975+0000 7f0833fff640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08240a34e0 0x7f08240a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.975+0000 7f0831ffb640 1 -- 172.21.15.98:0/2258333715 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8565+0+0 (secure 0 0 0) 0x7f08380f00f0 con 0x7f083c1009b0 2024-09-19T08:13:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.975+0000 7f0833fff640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08240a34e0 0x7f08240a59a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f083c0fff40 tx=0x7f082c00b5f0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:11.979+0000 7f0831ffb640 1 -- 172.21.15.98:0/2258333715 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f08380bcaa0 con 0x7f083c1009b0 2024-09-19T08:13:12.132 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:12.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.199+0000 7f08431e8640 1 -- 172.21.15.98:0/2258333715 --> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f083c108f90 con 0x7f08240a34e0 2024-09-19T08:13:12.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.206+0000 7f0831ffb640 1 -- 172.21.15.98:0/2258333715 <== mgr.15252 v2:172.21.15.98:6800/1961446727 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f082c00fb90 con 0x7f08240a34e0 2024-09-19T08:13:12.206 INFO:teuthology.orchestra.run.smithi098.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-09-19T00:00:00", "created": "2024-09-19T08:03:10", "first": "2024-09-19T08:04:00", "last": "2024-09-19T08:13:00", "last_pruned": "2024-09-19T08:13:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-09-19T08:13:12.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.208+0000 7f08431e8640 1 -- 172.21.15.98:0/2258333715 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08240a34e0 msgr2=0x7f08240a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.208+0000 7f08431e8640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08240a34e0 0x7f08240a59a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f083c0fff40 tx=0x7f082c00b5f0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.208+0000 7f08431e8640 1 -- 172.21.15.98:0/2258333715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1009b0 msgr2=0x7f083c197260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.208+0000 7f08431e8640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1009b0 0x7f083c197260 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f0838009230 tx=0x7f0838004270 comp rx=0 tx=0).stop 2024-09-19T08:13:12.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.209+0000 7f08431e8640 1 -- 172.21.15.98:0/2258333715 shutdown_connections 2024-09-19T08:13:12.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.209+0000 7f08431e8640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f08240a34e0 0x7f08240a59a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.209+0000 7f08431e8640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f083c1069c0 0x7f083c19bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.209+0000 7f08431e8640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f083c101350 0x7f083c1977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.209+0000 7f08431e8640 1 --2- 172.21.15.98:0/2258333715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1009b0 0x7f083c197260 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.209+0000 7f08431e8640 1 -- 172.21.15.98:0/2258333715 >> 172.21.15.98:0/2258333715 conn(0x7f083c077990 msgr2=0x7f083c103910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:12.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.209+0000 7f08431e8640 1 -- 172.21.15.98:0/2258333715 shutdown_connections 2024-09-19T08:13:12.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.209+0000 7f08431e8640 1 -- 172.21.15.98:0/2258333715 wait complete. 2024-09-19T08:13:12.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-09-19T08:13:12.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:11 smithi098 ceph-mon[31333]: pgmap v314: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:13:12.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:11 smithi098 ceph-mon[31333]: osdmap e88: 12 total, 12 up, 12 in 2024-09-19T08:13:12.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:11 smithi098 ceph-mon[31333]: scrub queued for path: / 2024-09-19T08:13:12.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:11 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:13:12.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:11 smithi098 ceph-mon[31333]: scrub summary: active paths [/] 2024-09-19T08:13:12.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:11 smithi098 ceph-mon[31333]: scrub summary: idle+waiting paths [/] 2024-09-19T08:13:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.322+0000 7fa7c85b7640 1 -- 172.21.15.98:0/3665299801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c0102cc0 msgr2=0x7fa7c01030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.322+0000 7fa7c85b7640 1 --2- 172.21.15.98:0/3665299801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c0102cc0 0x7fa7c01030a0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009ca0 tx=0x7fa7b002a950 comp rx=0 tx=0).stop 2024-09-19T08:13:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 -- 172.21.15.98:0/3665299801 shutdown_connections 2024-09-19T08:13:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 --2- 172.21.15.98:0/3665299801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7c010c110 0x7fa7c010e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 --2- 172.21.15.98:0/3665299801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7c01035e0 0x7fa7c010bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 --2- 172.21.15.98:0/3665299801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c0102cc0 0x7fa7c01030a0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 -- 172.21.15.98:0/3665299801 >> 172.21.15.98:0/3665299801 conn(0x7fa7c00fc6b0 msgr2=0x7fa7c00fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 -- 172.21.15.98:0/3665299801 shutdown_connections 2024-09-19T08:13:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 -- 172.21.15.98:0/3665299801 wait complete. 2024-09-19T08:13:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 Processor -- start 2024-09-19T08:13:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 -- start start 2024-09-19T08:13:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c0102cc0 0x7fa7c0198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7c01035e0 0x7fa7c0198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7c010c110 0x7fa7c019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c0111ef0 con 0x7fa7c0102cc0 2024-09-19T08:13:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c0111d70 con 0x7fa7c01035e0 2024-09-19T08:13:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c85b7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c0112070 con 0x7fa7c010c110 2024-09-19T08:13:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c6353640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c0102cc0 0x7fa7c0198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c6353640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c0102cc0 0x7fa7c0198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48922/0 (socket says 172.21.15.98:48922) 2024-09-19T08:13:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.323+0000 7fa7c6353640 1 -- 172.21.15.98:0/66901837 learned_addr learned my addr 172.21.15.98:0/66901837 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.324+0000 7fa7c6353640 1 -- 172.21.15.98:0/66901837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7c010c110 msgr2=0x7fa7c019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.324+0000 7fa7c6353640 1 --2- 172.21.15.98:0/66901837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7c010c110 0x7fa7c019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.324+0000 7fa7c6353640 1 -- 172.21.15.98:0/66901837 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7c01035e0 msgr2=0x7fa7c0198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.324+0000 7fa7c6353640 1 --2- 172.21.15.98:0/66901837 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7c01035e0 0x7fa7c0198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.324+0000 7fa7c6353640 1 -- 172.21.15.98:0/66901837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c019d430 con 0x7fa7c0102cc0 2024-09-19T08:13:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.324+0000 7fa7c6353640 1 --2- 172.21.15.98:0/66901837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c0102cc0 0x7fa7c0198440 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0005cc0 tx=0x7fa7b002f630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.324+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/66901837 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b0034040 con 0x7fa7c0102cc0 2024-09-19T08:13:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.324+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/66901837 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7b0030920 con 0x7fa7c0102cc0 2024-09-19T08:13:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.324+0000 7fa7c85b7640 1 -- 172.21.15.98:0/66901837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7c019d6c0 con 0x7fa7c0102cc0 2024-09-19T08:13:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.324+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/66901837 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b0004830 con 0x7fa7c0102cc0 2024-09-19T08:13:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.324+0000 7fa7c85b7640 1 -- 172.21.15.98:0/66901837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7c01a4fb0 con 0x7fa7c0102cc0 2024-09-19T08:13:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.325+0000 7fa7c85b7640 1 -- 172.21.15.98:0/66901837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7c01030a0 con 0x7fa7c0102cc0 2024-09-19T08:13:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.326+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/66901837 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fa7b0037020 con 0x7fa7c0102cc0 2024-09-19T08:13:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.327+0000 7fa7bf7fe640 1 --2- 172.21.15.98:0/66901837 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa7a80a3460 0x7fa7a80a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.327+0000 7fa7c5b52640 1 --2- 172.21.15.98:0/66901837 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa7a80a3460 0x7fa7a80a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.327+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/66901837 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8565+0+0 (secure 0 0 0) 0x7fa7b00f0d90 con 0x7fa7c0102cc0 2024-09-19T08:13:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.327+0000 7fa7c5b52640 1 --2- 172.21.15.98:0/66901837 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa7a80a3460 0x7fa7a80a5920 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4005fd0 tx=0x7fa7b4005ee0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.330+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/66901837 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fa7b00bd850 con 0x7fa7c0102cc0 2024-09-19T08:13:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:11 smithi162 ceph-mon[35260]: pgmap v314: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:13:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:11 smithi162 ceph-mon[35260]: osdmap e88: 12 total, 12 up, 12 in 2024-09-19T08:13:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:11 smithi162 ceph-mon[35260]: scrub queued for path: / 2024-09-19T08:13:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:11 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:13:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:11 smithi162 ceph-mon[35260]: scrub summary: active paths [/] 2024-09-19T08:13:12.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:11 smithi162 ceph-mon[35260]: scrub summary: idle+waiting paths [/] 2024-09-19T08:13:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:11 smithi139 ceph-mon[34874]: pgmap v314: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:13:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:11 smithi139 ceph-mon[34874]: osdmap e88: 12 total, 12 up, 12 in 2024-09-19T08:13:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:11 smithi139 ceph-mon[34874]: scrub queued for path: / 2024-09-19T08:13:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:11 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:13:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:11 smithi139 ceph-mon[34874]: scrub summary: active paths [/] 2024-09-19T08:13:12.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:11 smithi139 ceph-mon[34874]: scrub summary: idle+waiting paths [/] 2024-09-19T08:13:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.525+0000 7fa7c85b7640 1 -- 172.21.15.98:0/66901837 --> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fa7c0062630 con 0x7fa7a80a3460 2024-09-19T08:13:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.533+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/66901837 <== mgr.15252 v2:172.21.15.98:6800/1961446727 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fa7c0062630 con 0x7fa7a80a3460 2024-09-19T08:13:12.533 INFO:teuthology.orchestra.run.smithi098.stdout:/ 1m 6m3h 2024-09-19T08:13:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.535+0000 7fa7c85b7640 1 -- 172.21.15.98:0/66901837 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa7a80a3460 msgr2=0x7fa7a80a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.535+0000 7fa7c85b7640 1 --2- 172.21.15.98:0/66901837 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa7a80a3460 0x7fa7a80a5920 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4005fd0 tx=0x7fa7b4005ee0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.535+0000 7fa7c85b7640 1 -- 172.21.15.98:0/66901837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c0102cc0 msgr2=0x7fa7c0198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.535+0000 7fa7c85b7640 1 --2- 172.21.15.98:0/66901837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c0102cc0 0x7fa7c0198440 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0005cc0 tx=0x7fa7b002f630 comp rx=0 tx=0).stop 2024-09-19T08:13:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.535+0000 7fa7c85b7640 1 -- 172.21.15.98:0/66901837 shutdown_connections 2024-09-19T08:13:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.535+0000 7fa7c85b7640 1 --2- 172.21.15.98:0/66901837 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa7a80a3460 0x7fa7a80a5920 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.535+0000 7fa7c85b7640 1 --2- 172.21.15.98:0/66901837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7c010c110 0x7fa7c019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.535+0000 7fa7c85b7640 1 --2- 172.21.15.98:0/66901837 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7c01035e0 0x7fa7c0198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.535+0000 7fa7c85b7640 1 --2- 172.21.15.98:0/66901837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c0102cc0 0x7fa7c0198440 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.535+0000 7fa7c85b7640 1 -- 172.21.15.98:0/66901837 >> 172.21.15.98:0/66901837 conn(0x7fa7c00fc6b0 msgr2=0x7fa7c00fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.535+0000 7fa7c85b7640 1 -- 172.21.15.98:0/66901837 shutdown_connections 2024-09-19T08:13:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.535+0000 7fa7c85b7640 1 -- 172.21.15.98:0/66901837 wait complete. 2024-09-19T08:13:12.546 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-09-19T08:13:12.557 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-09-19T08:13:12.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.656+0000 7f6a3b115640 1 -- 172.21.15.98:0/4253809770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a34102da0 msgr2=0x7f6a34103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.656+0000 7f6a3b115640 1 --2- 172.21.15.98:0/4253809770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a34102da0 0x7f6a34103180 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f6a24009900 tx=0x7f6a2402a480 comp rx=0 tx=0).stop 2024-09-19T08:13:12.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 -- 172.21.15.98:0/4253809770 shutdown_connections 2024-09-19T08:13:12.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 --2- 172.21.15.98:0/4253809770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a3410c180 0x7f6a3410e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 --2- 172.21.15.98:0/4253809770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a341036c0 0x7f6a3410bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 --2- 172.21.15.98:0/4253809770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a34102da0 0x7f6a34103180 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 -- 172.21.15.98:0/4253809770 >> 172.21.15.98:0/4253809770 conn(0x7f6a340fc7a0 msgr2=0x7f6a340febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:12.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 -- 172.21.15.98:0/4253809770 shutdown_connections 2024-09-19T08:13:12.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 -- 172.21.15.98:0/4253809770 wait complete. 2024-09-19T08:13:12.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 Processor -- start 2024-09-19T08:13:12.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 -- start start 2024-09-19T08:13:12.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a34102da0 0x7f6a341983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a341036c0 0x7f6a341988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a3410c180 0x7f6a3419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a34111f60 con 0x7f6a341036c0 2024-09-19T08:13:12.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6a34111de0 con 0x7f6a3410c180 2024-09-19T08:13:12.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.657+0000 7f6a3b115640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341120e0 con 0x7f6a34102da0 2024-09-19T08:13:12.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a341036c0 0x7f6a341988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a341036c0 0x7f6a341988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48946/0 (socket says 172.21.15.98:48946) 2024-09-19T08:13:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a33fff640 1 -- 172.21.15.98:0/1182005539 learned_addr learned my addr 172.21.15.98:0/1182005539 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a38eb1640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a34102da0 0x7f6a341983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a396b2640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a3410c180 0x7f6a3419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a33fff640 1 -- 172.21.15.98:0/1182005539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a34102da0 msgr2=0x7f6a341983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a33fff640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a34102da0 0x7f6a341983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a33fff640 1 -- 172.21.15.98:0/1182005539 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a3410c180 msgr2=0x7f6a3419cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a33fff640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a3410c180 0x7f6a3419cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a33fff640 1 -- 172.21.15.98:0/1182005539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3419d350 con 0x7f6a341036c0 2024-09-19T08:13:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a396b2640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a3410c180 0x7f6a3419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a33fff640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a341036c0 0x7f6a341988e0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f6a2800d950 tx=0x7f6a2800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a31ffb640 1 -- 172.21.15.98:0/1182005539 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a28017030 con 0x7f6a341036c0 2024-09-19T08:13:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a3b115640 1 -- 172.21.15.98:0/1182005539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a3419d640 con 0x7f6a341036c0 2024-09-19T08:13:12.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a31ffb640 1 -- 172.21.15.98:0/1182005539 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a28004020 con 0x7f6a341036c0 2024-09-19T08:13:12.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.658+0000 7f6a31ffb640 1 -- 172.21.15.98:0/1182005539 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a280047a0 con 0x7f6a341036c0 2024-09-19T08:13:12.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.659+0000 7f6a3b115640 1 -- 172.21.15.98:0/1182005539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a341a4f20 con 0x7f6a341036c0 2024-09-19T08:13:12.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.661+0000 7f6a31ffb640 1 -- 172.21.15.98:0/1182005539 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f6a280041d0 con 0x7f6a341036c0 2024-09-19T08:13:12.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.661+0000 7f6a3d4ee740 1 Processor -- start 2024-09-19T08:13:12.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a3d4ee740 1 -- start start 2024-09-19T08:13:12.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a3d4ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e89c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a3d4ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e8a1d0 0x562a21e94550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a3d4ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 0x562a21e96e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a3d4ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562a21deb2e0 con 0x562a21e898b0 2024-09-19T08:13:12.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a3d4ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x562a21d92ce0 con 0x562a21e8a1d0 2024-09-19T08:13:12.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a3d4ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562a21de72a0 con 0x562a21e94a90 2024-09-19T08:13:12.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a396b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e89c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a396b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e89c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48960/0 (socket says 172.21.15.98:48960) 2024-09-19T08:13:12.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a396b2640 1 -- 172.21.15.98:0/1951341463 learned_addr learned my addr 172.21.15.98:0/1951341463 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a38eb1640 1 --2- 172.21.15.98:0/1951341463 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e8a1d0 0x562a21e94550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a31ffb640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6a1c0a33b0 0x7f6a1c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a33fff640 1 --2- 172.21.15.98:0/1951341463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 0x562a21e96e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a396b2640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6a1c0a33b0 0x7f6a1c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a396b2640 1 -- 172.21.15.98:0/1951341463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 msgr2=0x562a21e96e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a396b2640 1 --2- 172.21.15.98:0/1951341463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 0x562a21e96e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a396b2640 1 -- 172.21.15.98:0/1951341463 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e8a1d0 msgr2=0x562a21e94550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a396b2640 1 --2- 172.21.15.98:0/1951341463 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e8a1d0 0x562a21e94550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a396b2640 1 -- 172.21.15.98:0/1951341463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a21d77440 con 0x562a21e898b0 2024-09-19T08:13:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.662+0000 7f6a31ffb640 1 -- 172.21.15.98:0/1182005539 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8565+0+0 (secure 0 0 0) 0x7f6a280cf7f0 con 0x7f6a341036c0 2024-09-19T08:13:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.663+0000 7f6a396b2640 1 --2- 172.21.15.98:0/1951341463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e89c90 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0149a0 tx=0x7f6a2c014e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4416e46c658297a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.663+0000 7f6a1affd640 1 -- 172.21.15.98:0/1951341463 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c021070 con 0x562a21e898b0 2024-09-19T08:13:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.663+0000 7f6a1affd640 1 -- 172.21.15.98:0/1951341463 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a2c013460 con 0x562a21e898b0 2024-09-19T08:13:12.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.663+0000 7f6a1affd640 1 -- 172.21.15.98:0/1951341463 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c00e650 con 0x562a21e898b0 2024-09-19T08:13:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.663+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/1951341463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 msgr2=0x562a21e89c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.663+0000 7f6a396b2640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6a1c0a33b0 0x7f6a1c0a5870 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c01e000 tx=0x7f6a2c019000 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.663+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/1951341463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e89c90 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0149a0 tx=0x7f6a2c014e60 comp rx=0 tx=0).stop 2024-09-19T08:13:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/1951341463 shutdown_connections 2024-09-19T08:13:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/1951341463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 0x562a21e96e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/1951341463 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e8a1d0 0x562a21e94550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/1951341463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e89c90 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/1951341463 >> 172.21.15.98:0/1951341463 conn(0x562a21e85e40 msgr2=0x562a21e86220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/1951341463 shutdown_connections 2024-09-19T08:13:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/1951341463 wait complete. 2024-09-19T08:13:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 -- start start 2024-09-19T08:13:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 Processor -- start 2024-09-19T08:13:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e94410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e8a1d0 0x562a21e916c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e94a90 0x562a21e91c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562a21e97660 con 0x562a21e898b0 2024-09-19T08:13:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x562a21da6050 con 0x562a21e94a90 2024-09-19T08:13:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562a21e977e0 con 0x562a21e8a1d0 2024-09-19T08:13:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.664+0000 7f6a3d4ee740 10 client.0 _ll_register_callbacks cb 0x562a21de6160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a38eb1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e94410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a38eb1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e94410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48974/0 (socket says 172.21.15.98:48974) 2024-09-19T08:13:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a38eb1640 1 -- 172.21.15.98:0/3330712712 learned_addr learned my addr 172.21.15.98:0/3330712712 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a396b2640 1 --2- 172.21.15.98:0/3330712712 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e94a90 0x562a21e91c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a33fff640 1 --2- 172.21.15.98:0/3330712712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e8a1d0 0x562a21e916c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a38eb1640 1 -- 172.21.15.98:0/3330712712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e8a1d0 msgr2=0x562a21e916c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a38eb1640 1 --2- 172.21.15.98:0/3330712712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e8a1d0 0x562a21e916c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a38eb1640 1 -- 172.21.15.98:0/3330712712 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e94a90 msgr2=0x562a21e91c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a38eb1640 1 --2- 172.21.15.98:0/3330712712 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e94a90 0x562a21e91c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a38eb1640 1 -- 172.21.15.98:0/3330712712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a21b5c3c0 con 0x562a21e898b0 2024-09-19T08:13:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a396b2640 1 --2- 172.21.15.98:0/3330712712 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e94a90 0x562a21e91c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a38eb1640 1 --2- 172.21.15.98:0/3330712712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e94410 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6a24009230 tx=0x7f6a240315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf3e1e14b19333a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a13fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a13fff640 1 -- 172.21.15.98:0/3330712712 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a24040070 con 0x562a21e898b0 2024-09-19T08:13:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a13fff640 1 -- 172.21.15.98:0/3330712712 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8565+0+0 (secure 0 0 0) 0x7f6a24004040 con 0x562a21e898b0 2024-09-19T08:13:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/3330712712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a2c013460 con 0x562a21e898b0 2024-09-19T08:13:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.665+0000 7f6a13fff640 1 -- 172.21.15.98:0/3330712712 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6a24035040 con 0x562a21e898b0 2024-09-19T08:13:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.666+0000 7f6a3d4ee740 10 client.17850 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.666+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/3330712712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562a21ea3050 con 0x562a21e898b0 2024-09-19T08:13:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.666+0000 7f6a13fff640 1 -- 172.21.15.98:0/3330712712 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a2c013460 con 0x562a21e898b0 2024-09-19T08:13:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.666+0000 7f6a13fff640 1 -- 172.21.15.98:0/3330712712 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6a2404b070 con 0x562a21e898b0 2024-09-19T08:13:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.666+0000 7f6a3d4ee740 10 client.17850 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.666+0000 7f6a3d4ee740 10 client.17850 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:13:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.666+0000 7f6a3d4ee740 20 client.17850 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.666+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/3330712712 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x562a21ea4510 0x562a21ea6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.666+0000 7f6a3d4ee740 4 client.17850 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:13:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.666+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/3330712712 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a21df8850 con 0x562a21ea4510 2024-09-19T08:13:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.666+0000 7f6a33fff640 1 --2- 172.21.15.98:0/3330712712 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x562a21ea4510 0x562a21ea6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.667+0000 7f6a13fff640 10 client.17850 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.667+0000 7f6a33fff640 1 --2- 172.21.15.98:0/3330712712 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x562a21ea4510 0x562a21ea6920 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d9766e5a76ce77f server_cookie=a4f7a2a167d01bf1 in_seq=0 out_seq=0 2024-09-19T08:13:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.667+0000 7f6a13fff640 1 -- 172.21.15.98:0/3330712712 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562a21df8850 con 0x562a21ea4510 2024-09-19T08:13:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.667+0000 7f6a13fff640 10 client.17850 handle_command_reply: tid=0 2024-09-19T08:13:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.667+0000 7f6a3d4ee740 1 client.17850 shutdown 2024-09-19T08:13:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.668+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/3330712712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 msgr2=0x562a21e94410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.668+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/3330712712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e94410 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6a24009230 tx=0x7f6a240315b0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.668+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/3330712712 shutdown_connections 2024-09-19T08:13:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.668+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/3330712712 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x562a21ea4510 0x562a21ea6920 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.668+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/3330712712 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e94a90 0x562a21e91c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.668+0000 7f6a13fff640 0 client.17850 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.668+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/3330712712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e8a1d0 0x562a21e916c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.668+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/3330712712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e94410 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.668+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/3330712712 >> 172.21.15.98:0/3330712712 conn(0x562a21e85e40 msgr2=0x562a21e80b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.668+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/3330712712 shutdown_connections 2024-09-19T08:13:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.668+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/3330712712 wait complete. 2024-09-19T08:13:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.668+0000 7f6a3d4ee740 20 client.17850 trim_cache size 0 max 16384 2024-09-19T08:13:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.684+0000 7f6a3d4ee740 1 Processor -- start 2024-09-19T08:13:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.684+0000 7f6a3d4ee740 1 -- start start 2024-09-19T08:13:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a3d4ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e8ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a3d4ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e8a1d0 0x562a21eac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a3d4ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 0x562a21eac730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a3d4ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341ab710 con 0x562a21e898b0 2024-09-19T08:13:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a3d4ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341ad250 con 0x562a21e8a1d0 2024-09-19T08:13:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a3d4ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341abde0 con 0x562a21e94a90 2024-09-19T08:13:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a38eb1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e8ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a38eb1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e8ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48982/0 (socket says 172.21.15.98:48982) 2024-09-19T08:13:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a38eb1640 1 -- 172.21.15.98:0/3389146603 learned_addr learned my addr 172.21.15.98:0/3389146603 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a396b2640 1 --2- 172.21.15.98:0/3389146603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 0x562a21eac730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a33fff640 1 --2- 172.21.15.98:0/3389146603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e8a1d0 0x562a21eac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a38eb1640 1 -- 172.21.15.98:0/3389146603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 msgr2=0x562a21eac730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a38eb1640 1 --2- 172.21.15.98:0/3389146603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 0x562a21eac730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a38eb1640 1 -- 172.21.15.98:0/3389146603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e8a1d0 msgr2=0x562a21eac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a38eb1640 1 --2- 172.21.15.98:0/3389146603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e8a1d0 0x562a21eac1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a38eb1640 1 -- 172.21.15.98:0/3389146603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a21e90670 con 0x562a21e898b0 2024-09-19T08:13:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a396b2640 1 --2- 172.21.15.98:0/3389146603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 0x562a21eac730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a33fff640 1 --2- 172.21.15.98:0/3389146603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e8a1d0 0x562a21eac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a38eb1640 1 --2- 172.21.15.98:0/3389146603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e8ff90 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f6a2402a9a0 tx=0x7f6a2402a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=822fceea98b6205e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a1a7fc640 1 -- 172.21.15.98:0/3389146603 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a24051030 con 0x562a21e898b0 2024-09-19T08:13:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a1a7fc640 1 -- 172.21.15.98:0/3389146603 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a240319b0 con 0x562a21e898b0 2024-09-19T08:13:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.685+0000 7f6a1a7fc640 1 -- 172.21.15.98:0/3389146603 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a24000e60 con 0x562a21e898b0 2024-09-19T08:13:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.686+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/3389146603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 msgr2=0x562a21e8ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.686+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/3389146603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e8ff90 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f6a2402a9a0 tx=0x7f6a2402a480 comp rx=0 tx=0).stop 2024-09-19T08:13:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.686+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/3389146603 shutdown_connections 2024-09-19T08:13:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.686+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/3389146603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 0x562a21eac730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.686+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/3389146603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e8a1d0 0x562a21eac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.686+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/3389146603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e898b0 0x562a21e8ff90 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.686+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/3389146603 >> 172.21.15.98:0/3389146603 conn(0x562a21e85e40 msgr2=0x562a21e80b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.686+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/3389146603 shutdown_connections 2024-09-19T08:13:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.686+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/3389146603 wait complete. 2024-09-19T08:13:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.686+0000 7f6a3d4ee740 1 -- start start 2024-09-19T08:13:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.687+0000 7f6a3d4ee740 1 Processor -- start 2024-09-19T08:13:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.687+0000 7f6a3d4ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e898b0 0x562a21e9d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.687+0000 7f6a3d4ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e8a1d0 0x562a21e9a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.687+0000 7f6a3d4ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 0x562a21e9abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.687+0000 7f6a3d4ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341acb80 con 0x562a21e8a1d0 2024-09-19T08:13:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.687+0000 7f6a3d4ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341ac4b0 con 0x562a21e898b0 2024-09-19T08:13:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.687+0000 7f6a3d4ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341a9710 con 0x562a21e94a90 2024-09-19T08:13:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.687+0000 7f6a3d4ee740 10 client.0 _ll_register_callbacks cb 0x7f6a24009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.687+0000 7f6a33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e8a1d0 0x562a21e9a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.687+0000 7f6a33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e8a1d0 0x562a21e9a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48992/0 (socket says 172.21.15.98:48992) 2024-09-19T08:13:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.687+0000 7f6a33fff640 1 -- 172.21.15.98:0/1940457813 learned_addr learned my addr 172.21.15.98:0/1940457813 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.687+0000 7f6a38eb1640 1 --2- 172.21.15.98:0/1940457813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e898b0 0x562a21e9d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.687+0000 7f6a396b2640 1 --2- 172.21.15.98:0/1940457813 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 0x562a21e9abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a33fff640 1 -- 172.21.15.98:0/1940457813 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 msgr2=0x562a21e9abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a33fff640 1 --2- 172.21.15.98:0/1940457813 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 0x562a21e9abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a33fff640 1 -- 172.21.15.98:0/1940457813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e898b0 msgr2=0x562a21e9d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a33fff640 1 --2- 172.21.15.98:0/1940457813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e898b0 0x562a21e9d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a33fff640 1 -- 172.21.15.98:0/1940457813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a21e9b420 con 0x562a21e8a1d0 2024-09-19T08:13:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a38eb1640 1 --2- 172.21.15.98:0/1940457813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e898b0 0x562a21e9d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a33fff640 1 --2- 172.21.15.98:0/1940457813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e8a1d0 0x562a21e9a620 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f6a2809b7a0 tx=0x7f6a2809bb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed58f2259fa2f4ef server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a13fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a13fff640 1 -- 172.21.15.98:0/1940457813 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a28098110 con 0x562a21e8a1d0 2024-09-19T08:13:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a13fff640 1 -- 172.21.15.98:0/1940457813 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8565+0+0 (secure 0 0 0) 0x7f6a280860b0 con 0x562a21e8a1d0 2024-09-19T08:13:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/1940457813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a240319b0 con 0x562a21e8a1d0 2024-09-19T08:13:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a13fff640 1 -- 172.21.15.98:0/1940457813 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6a280982b0 con 0x562a21e8a1d0 2024-09-19T08:13:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a3d4ee740 10 client.17856 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/1940457813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562a21f0f250 con 0x562a21e8a1d0 2024-09-19T08:13:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a13fff640 1 -- 172.21.15.98:0/1940457813 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a240319b0 con 0x562a21e8a1d0 2024-09-19T08:13:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a13fff640 1 -- 172.21.15.98:0/1940457813 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6a280ba210 con 0x562a21e8a1d0 2024-09-19T08:13:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a3d4ee740 10 client.17856 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a3d4ee740 10 client.17856 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:13:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a3d4ee740 20 client.17856 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/1940457813 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x562a21f0f970 0x562a21f11d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a3d4ee740 4 client.17856 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:13:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.688+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/1940457813 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562a21df2d60 con 0x562a21f0f970 2024-09-19T08:13:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.689+0000 7f6a38eb1640 1 --2- 172.21.15.98:0/1940457813 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x562a21f0f970 0x562a21f11d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.689+0000 7f6a13fff640 10 client.17856 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.689+0000 7f6a38eb1640 1 --2- 172.21.15.98:0/1940457813 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x562a21f0f970 0x562a21f11d60 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=480efb157ed1e53a server_cookie=17c92ca7d2013a9d in_seq=0 out_seq=0 2024-09-19T08:13:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.689+0000 7f6a13fff640 1 -- 172.21.15.98:0/1940457813 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562a21df2d60 con 0x562a21f0f970 2024-09-19T08:13:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.689+0000 7f6a13fff640 10 client.17856 handle_command_reply: tid=0 2024-09-19T08:13:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.689+0000 7f6a3d4ee740 1 client.17856 shutdown 2024-09-19T08:13:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.689+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/1940457813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e8a1d0 msgr2=0x562a21e9a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.689+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/1940457813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e8a1d0 0x562a21e9a620 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f6a2809b7a0 tx=0x7f6a2809bb60 comp rx=0 tx=0).stop 2024-09-19T08:13:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/1940457813 shutdown_connections 2024-09-19T08:13:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/1940457813 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x562a21f0f970 0x562a21f11d60 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/1940457813 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a21e94a90 0x562a21e9abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/1940457813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562a21e8a1d0 0x562a21e9a620 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3d4ee740 1 --2- 172.21.15.98:0/1940457813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562a21e898b0 0x562a21e9d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/1940457813 >> 172.21.15.98:0/1940457813 conn(0x562a21e85e40 msgr2=0x562a21e92f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a13fff640 0 client.17856 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/1940457813 shutdown_connections 2024-09-19T08:13:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3d4ee740 1 -- 172.21.15.98:0/1940457813 wait complete. 2024-09-19T08:13:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3d4ee740 20 client.17856 trim_cache size 0 max 16384 2024-09-19T08:13:12.696 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:13:12.697 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-19T08:13:12.697 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-09-19T08:13:12.697 INFO:teuthology.orchestra.run.smithi098.stdout: "fc3b494f-8edf-45b6-b770-a2bd83af103b": { 2024-09-19T08:13:12.697 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-09-19T08:13:12.697 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "fc3b494f-8edf-45b6-b770-a2bd83af103b", 2024-09-19T08:13:12.697 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-09-19T08:13:12.698 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:12.698 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:12.698 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:13:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3b115640 1 -- 172.21.15.98:0/1182005539 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6a1c0a33b0 msgr2=0x7f6a1c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3b115640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6a1c0a33b0 0x7f6a1c0a5870 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c01e000 tx=0x7f6a2c019000 comp rx=0 tx=0).stop 2024-09-19T08:13:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3b115640 1 -- 172.21.15.98:0/1182005539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a341036c0 msgr2=0x7f6a341988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3b115640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a341036c0 0x7f6a341988e0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f6a2800d950 tx=0x7f6a2800de10 comp rx=0 tx=0).stop 2024-09-19T08:13:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3b115640 1 -- 172.21.15.98:0/1182005539 shutdown_connections 2024-09-19T08:13:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3b115640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f6a1c0a33b0 0x7f6a1c0a5870 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3b115640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a3410c180 0x7f6a3419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3b115640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a341036c0 0x7f6a341988e0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3b115640 1 --2- 172.21.15.98:0/1182005539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a34102da0 0x7f6a341983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.690+0000 7f6a3b115640 1 -- 172.21.15.98:0/1182005539 >> 172.21.15.98:0/1182005539 conn(0x7f6a340fc7a0 msgr2=0x7f6a340fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.691+0000 7f6a3b115640 1 -- 172.21.15.98:0/1182005539 shutdown_connections 2024-09-19T08:13:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:12.691+0000 7f6a3b115640 1 -- 172.21.15.98:0/1182005539 wait complete. 2024-09-19T08:13:13.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:12 smithi098 ceph-mon[31333]: from='client.17832 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:13:13.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:12 smithi098 ceph-mon[31333]: from='client.17838 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:13:13.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:12 smithi162 ceph-mon[35260]: from='client.17832 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:13:13.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:12 smithi162 ceph-mon[35260]: from='client.17838 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:13:13.392 INFO:tasks.cephfs.filesystem:scrub status for tag:fc3b494f-8edf-45b6-b770-a2bd83af103b - {'path': '/', 'tag': 'fc3b494f-8edf-45b6-b770-a2bd83af103b', 'options': 'recursive,force'} 2024-09-19T08:13:13.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:12 smithi139 ceph-mon[34874]: from='client.17832 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:13:13.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:12 smithi139 ceph-mon[34874]: from='client.17838 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T08:13:14.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:14 smithi139 ceph-mon[34874]: pgmap v316: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 204 B/s wr, 13 op/s 2024-09-19T08:13:14.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:13 smithi162 ceph-mon[35260]: pgmap v316: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 204 B/s wr, 13 op/s 2024-09-19T08:13:14.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:13 smithi098 ceph-mon[31333]: pgmap v316: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 204 B/s wr, 13 op/s 2024-09-19T08:13:16.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:16 smithi098 ceph-mon[31333]: pgmap v317: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 204 B/s wr, 13 op/s 2024-09-19T08:13:16.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:16 smithi098 ceph-mon[31333]: scrub summary: idle 2024-09-19T08:13:16.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:16 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:13:16.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:16 smithi098 ceph-mon[31333]: osdmap e89: 12 total, 12 up, 12 in 2024-09-19T08:13:16.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:16 smithi162 ceph-mon[35260]: pgmap v317: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 204 B/s wr, 13 op/s 2024-09-19T08:13:16.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:16 smithi162 ceph-mon[35260]: scrub summary: idle 2024-09-19T08:13:16.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:16 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:13:16.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:16 smithi162 ceph-mon[35260]: osdmap e89: 12 total, 12 up, 12 in 2024-09-19T08:13:16.359 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:16 smithi139 ceph-mon[34874]: pgmap v317: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 204 B/s wr, 13 op/s 2024-09-19T08:13:16.359 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:16 smithi139 ceph-mon[34874]: scrub summary: idle 2024-09-19T08:13:16.360 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:16 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:13:16.360 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:16 smithi139 ceph-mon[34874]: osdmap e89: 12 total, 12 up, 12 in 2024-09-19T08:13:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: pgmap v319: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 255 B/s wr, 15 op/s 2024-09-19T08:13:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi098 to 176.0M 2024-09-19T08:13:18.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi162 to 176.0M 2024-09-19T08:13:18.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: Adjusting osd_memory_target on smithi139 to 176.0M 2024-09-19T08:13:18.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: Unable to set osd_memory_target on smithi098 to 184576716: error parsing value: Value '184576716' is below minimum 939524096 2024-09-19T08:13:18.373 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: Unable to set osd_memory_target on smithi139 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-19T08:13:18.373 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: Unable to set osd_memory_target on smithi162 to 184591052: error parsing value: Value '184591052' is below minimum 939524096 2024-09-19T08:13:18.373 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:13:18.373 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:13:18.373 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: pgmap v319: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 255 B/s wr, 15 op/s 2024-09-19T08:13:18.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi098 to 176.0M 2024-09-19T08:13:18.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi162 to 176.0M 2024-09-19T08:13:18.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: Adjusting osd_memory_target on smithi139 to 176.0M 2024-09-19T08:13:18.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: Unable to set osd_memory_target on smithi098 to 184576716: error parsing value: Value '184576716' is below minimum 939524096 2024-09-19T08:13:18.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: Unable to set osd_memory_target on smithi139 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-19T08:13:18.430 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: Unable to set osd_memory_target on smithi162 to 184591052: error parsing value: Value '184591052' is below minimum 939524096 2024-09-19T08:13:18.430 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:13:18.430 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:13:18.430 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: pgmap v319: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 255 B/s wr, 15 op/s 2024-09-19T08:13:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:18.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:13:18.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi098 to 176.0M 2024-09-19T08:13:18.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi162 to 176.0M 2024-09-19T08:13:18.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: Adjusting osd_memory_target on smithi139 to 176.0M 2024-09-19T08:13:18.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: Unable to set osd_memory_target on smithi098 to 184576716: error parsing value: Value '184576716' is below minimum 939524096 2024-09-19T08:13:18.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: Unable to set osd_memory_target on smithi139 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-19T08:13:18.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: Unable to set osd_memory_target on smithi162 to 184591052: error parsing value: Value '184591052' is below minimum 939524096 2024-09-19T08:13:18.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:13:18.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:13:18.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:13:20.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:20 smithi162 ceph-mon[35260]: pgmap v320: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 255 B/s wr, 16 op/s 2024-09-19T08:13:20.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:20 smithi139 ceph-mon[34874]: pgmap v320: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 255 B/s wr, 16 op/s 2024-09-19T08:13:20.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:20 smithi098 ceph-mon[31333]: pgmap v320: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 255 B/s wr, 16 op/s 2024-09-19T08:13:22.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:22 smithi162 ceph-mon[35260]: pgmap v321: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 205 B/s wr, 13 op/s 2024-09-19T08:13:22.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:22 smithi139 ceph-mon[34874]: pgmap v321: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 205 B/s wr, 13 op/s 2024-09-19T08:13:22.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:22 smithi098 ceph-mon[31333]: pgmap v321: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 205 B/s wr, 13 op/s 2024-09-19T08:13:24.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:24 smithi162 ceph-mon[35260]: pgmap v322: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:13:24.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:24 smithi139 ceph-mon[34874]: pgmap v322: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:13:24.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:24 smithi098 ceph-mon[31333]: pgmap v322: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:13:26.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:26 smithi162 ceph-mon[35260]: pgmap v323: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:13:26.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:26 smithi139 ceph-mon[34874]: pgmap v323: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:13:26.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:26 smithi098 ceph-mon[31333]: pgmap v323: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-19T08:13:28.374 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:28 smithi162 ceph-mon[35260]: pgmap v324: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 92 B/s wr, 12 op/s 2024-09-19T08:13:28.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:28 smithi139 ceph-mon[34874]: pgmap v324: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 92 B/s wr, 12 op/s 2024-09-19T08:13:28.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:28 smithi098 ceph-mon[31333]: pgmap v324: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 92 B/s wr, 12 op/s 2024-09-19T08:13:30.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:30 smithi162 ceph-mon[35260]: pgmap v325: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:13:30.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:30 smithi139 ceph-mon[34874]: pgmap v325: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:13:30.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:30 smithi098 ceph-mon[31333]: pgmap v325: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:13:32.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:32 smithi162 ceph-mon[35260]: pgmap v326: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:32.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:32 smithi139 ceph-mon[34874]: pgmap v326: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:32.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:32 smithi098 ceph-mon[31333]: pgmap v326: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:34.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:34 smithi162 ceph-mon[35260]: pgmap v327: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:13:34.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:34 smithi139 ceph-mon[34874]: pgmap v327: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:13:34.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:34 smithi098 ceph-mon[31333]: pgmap v327: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:13:36.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:36 smithi162 ceph-mon[35260]: pgmap v328: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:36.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:36 smithi139 ceph-mon[34874]: pgmap v328: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:36.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:36 smithi098 ceph-mon[31333]: pgmap v328: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:38.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:38 smithi162 ceph-mon[35260]: pgmap v329: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:13:38.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:38 smithi139 ceph-mon[34874]: pgmap v329: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:13:38.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:38 smithi098 ceph-mon[31333]: pgmap v329: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:13:40.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:40 smithi162 ceph-mon[35260]: pgmap v330: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:13:40.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:40 smithi139 ceph-mon[34874]: pgmap v330: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:13:40.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:40 smithi098 ceph-mon[31333]: pgmap v330: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:13:43.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:42 smithi139 ceph-mon[34874]: pgmap v331: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:43.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:42 smithi098 ceph-mon[31333]: pgmap v331: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:43.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:42 smithi162 ceph-mon[35260]: pgmap v331: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:43.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-19T08:13:43.651 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:44.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.104+0000 7f96829e9640 1 -- 172.21.15.98:0/561601729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c100a80 msgr2=0x7f967c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:44.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.104+0000 7f96829e9640 1 --2- 172.21.15.98:0/561601729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c100a80 0x7f967c100ee0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f9664009930 tx=0x7f966402a420 comp rx=0 tx=0).stop 2024-09-19T08:13:44.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.104+0000 7f96829e9640 1 -- 172.21.15.98:0/561601729 shutdown_connections 2024-09-19T08:13:44.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.104+0000 7f96829e9640 1 --2- 172.21.15.98:0/561601729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f967c101420 0x7f967c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.104+0000 7f96829e9640 1 --2- 172.21.15.98:0/561601729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c100a80 0x7f967c100ee0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.104+0000 7f96829e9640 1 --2- 172.21.15.98:0/561601729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f967c106a80 0x7f967c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.104+0000 7f96829e9640 1 -- 172.21.15.98:0/561601729 >> 172.21.15.98:0/561601729 conn(0x7f967c0fc7a0 msgr2=0x7f967c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.104+0000 7f96829e9640 1 -- 172.21.15.98:0/561601729 shutdown_connections 2024-09-19T08:13:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.104+0000 7f96829e9640 1 -- 172.21.15.98:0/561601729 wait complete. 2024-09-19T08:13:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f96829e9640 1 Processor -- start 2024-09-19T08:13:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f96829e9640 1 -- start start 2024-09-19T08:13:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f96829e9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f967c100a80 0x7f967c1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f96829e9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f967c101420 0x7f967c198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f96829e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c106a80 0x7f967c19ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f96829e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f967c110e80 con 0x7f967c106a80 2024-09-19T08:13:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f96829e9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f967c110d00 con 0x7f967c101420 2024-09-19T08:13:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f96829e9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f967c111000 con 0x7f967c100a80 2024-09-19T08:13:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f9680f86640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c106a80 0x7f967c19ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f9680f86640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c106a80 0x7f967c19ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48364/0 (socket says 172.21.15.98:48364) 2024-09-19T08:13:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f9680f86640 1 -- 172.21.15.98:0/1330781119 learned_addr learned my addr 172.21.15.98:0/1330781119 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f967bfff640 1 --2- 172.21.15.98:0/1330781119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f967c100a80 0x7f967c1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f9673fff640 1 --2- 172.21.15.98:0/1330781119 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f967c101420 0x7f967c198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f9680f86640 1 -- 172.21.15.98:0/1330781119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f967c100a80 msgr2=0x7f967c1983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f9680f86640 1 --2- 172.21.15.98:0/1330781119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f967c100a80 0x7f967c1983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f9680f86640 1 -- 172.21.15.98:0/1330781119 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f967c101420 msgr2=0x7f967c198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f9680f86640 1 --2- 172.21.15.98:0/1330781119 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f967c101420 0x7f967c198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f9680f86640 1 -- 172.21.15.98:0/1330781119 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f967c19d390 con 0x7f967c106a80 2024-09-19T08:13:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.105+0000 7f9680f86640 1 --2- 172.21.15.98:0/1330781119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c106a80 0x7f967c19ccb0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f967400eaa0 tx=0x7f967400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.106+0000 7f9679ffb640 1 -- 172.21.15.98:0/1330781119 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967400cdb0 con 0x7f967c106a80 2024-09-19T08:13:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.106+0000 7f9679ffb640 1 -- 172.21.15.98:0/1330781119 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9674005020 con 0x7f967c106a80 2024-09-19T08:13:44.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.106+0000 7f9679ffb640 1 -- 172.21.15.98:0/1330781119 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96740116b0 con 0x7f967c106a80 2024-09-19T08:13:44.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.106+0000 7f96829e9640 1 -- 172.21.15.98:0/1330781119 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f967c19d680 con 0x7f967c106a80 2024-09-19T08:13:44.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.106+0000 7f96829e9640 1 -- 172.21.15.98:0/1330781119 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f967c1a4f60 con 0x7f967c106a80 2024-09-19T08:13:44.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.108+0000 7f9679ffb640 1 -- 172.21.15.98:0/1330781119 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f9674005560 con 0x7f967c106a80 2024-09-19T08:13:44.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.108+0000 7f9684dc2740 1 Processor -- start 2024-09-19T08:13:44.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.108+0000 7f9684dc2740 1 -- start start 2024-09-19T08:13:44.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f9684dc2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba02254a0 0x55cba0225880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f9684dc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0225dc0 0x55cba0230140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f9684dc2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba0230680 0x55cba0232a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f9684dc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cba018b2e0 con 0x55cba0225dc0 2024-09-19T08:13:44.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f9684dc2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cba0132ce0 con 0x55cba0230680 2024-09-19T08:13:44.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f9684dc2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cba01872a0 con 0x55cba02254a0 2024-09-19T08:13:44.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f9679ffb640 1 --2- 172.21.15.98:0/1330781119 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f96680a3430 0x7f96680a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f967bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0225dc0 0x55cba0230140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f967bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0225dc0 0x55cba0230140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48370/0 (socket says 172.21.15.98:48370) 2024-09-19T08:13:44.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f967bfff640 1 -- 172.21.15.98:0/3088465572 learned_addr learned my addr 172.21.15.98:0/3088465572 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:44.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f9680f86640 1 --2- 172.21.15.98:0/1330781119 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f96680a3430 0x7f96680a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f9673fff640 1 --2- 172.21.15.98:0/3088465572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba02254a0 0x55cba0225880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f9673fff640 1 --2- 172.21.15.98:0/3088465572 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba0230680 0x55cba0232a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f9679ffb640 1 -- 172.21.15.98:0/1330781119 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f96740cfe40 con 0x7f967c106a80 2024-09-19T08:13:44.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f967bfff640 1 -- 172.21.15.98:0/3088465572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba02254a0 msgr2=0x55cba0225880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:44.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f967bfff640 1 --2- 172.21.15.98:0/3088465572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba02254a0 0x55cba0225880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f967bfff640 1 -- 172.21.15.98:0/3088465572 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba0230680 msgr2=0x55cba0232a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:44.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f967bfff640 1 --2- 172.21.15.98:0/3088465572 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba0230680 0x55cba0232a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f967bfff640 1 -- 172.21.15.98:0/3088465572 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cba0117440 con 0x55cba0225dc0 2024-09-19T08:13:44.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f9680f86640 1 --2- 172.21.15.98:0/1330781119 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f96680a3430 0x7f96680a58f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9674000f80 tx=0x7f967406df50 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:44.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f967bfff640 1 --2- 172.21.15.98:0/3088465572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0225dc0 0x55cba0230140 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f966c00bc90 tx=0x7f966c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac91c6adfad13680 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:44.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.109+0000 7f96727fc640 1 -- 172.21.15.98:0/3088465572 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c01c480 con 0x55cba0225dc0 2024-09-19T08:13:44.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.110+0000 7f96727fc640 1 -- 172.21.15.98:0/3088465572 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f966c01cb00 con 0x55cba0225dc0 2024-09-19T08:13:44.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.110+0000 7f96727fc640 1 -- 172.21.15.98:0/3088465572 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c011c60 con 0x55cba0225dc0 2024-09-19T08:13:44.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.110+0000 7f9684dc2740 1 -- 172.21.15.98:0/3088465572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0225dc0 msgr2=0x55cba0230140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:44.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.110+0000 7f9684dc2740 1 --2- 172.21.15.98:0/3088465572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0225dc0 0x55cba0230140 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f966c00bc90 tx=0x7f966c0107a0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.110+0000 7f9684dc2740 1 -- 172.21.15.98:0/3088465572 shutdown_connections 2024-09-19T08:13:44.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.110+0000 7f9684dc2740 1 --2- 172.21.15.98:0/3088465572 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba0230680 0x55cba0232a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.110+0000 7f9684dc2740 1 --2- 172.21.15.98:0/3088465572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0225dc0 0x55cba0230140 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.110+0000 7f9684dc2740 1 --2- 172.21.15.98:0/3088465572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba02254a0 0x55cba0225880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.110+0000 7f9684dc2740 1 -- 172.21.15.98:0/3088465572 >> 172.21.15.98:0/3088465572 conn(0x55cba0221a30 msgr2=0x55cba0221e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:44.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.110+0000 7f9684dc2740 1 -- 172.21.15.98:0/3088465572 shutdown_connections 2024-09-19T08:13:44.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.110+0000 7f9684dc2740 1 -- 172.21.15.98:0/3088465572 wait complete. 2024-09-19T08:13:44.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.111+0000 7f9684dc2740 1 -- start start 2024-09-19T08:13:44.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.111+0000 7f9684dc2740 1 Processor -- start 2024-09-19T08:13:44.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.111+0000 7f9684dc2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba02254a0 0x55cba022ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.111+0000 7f9684dc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0225dc0 0x55cba022d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.111+0000 7f9684dc2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0230680 0x55cba022d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.111+0000 7f9684dc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cba0233250 con 0x55cba0225dc0 2024-09-19T08:13:44.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.111+0000 7f9684dc2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cba0146050 con 0x55cba02254a0 2024-09-19T08:13:44.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.111+0000 7f9684dc2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cba02333d0 con 0x55cba0230680 2024-09-19T08:13:44.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.111+0000 7f9684dc2740 10 client.0 _ll_register_callbacks cb 0x55cba0186160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:44.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.111+0000 7f967bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0225dc0 0x55cba022d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.111+0000 7f967bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0225dc0 0x55cba022d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48386/0 (socket says 172.21.15.98:48386) 2024-09-19T08:13:44.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.111+0000 7f967bfff640 1 -- 172.21.15.98:0/660714258 learned_addr learned my addr 172.21.15.98:0/660714258 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:44.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.111+0000 7f9673fff640 1 --2- 172.21.15.98:0/660714258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0230680 0x55cba022d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f9673fff640 1 --2- 172.21.15.98:0/660714258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba02254a0 0x55cba022ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f967bfff640 1 -- 172.21.15.98:0/660714258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0230680 msgr2=0x55cba022d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:44.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f967bfff640 1 --2- 172.21.15.98:0/660714258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0230680 0x55cba022d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f967bfff640 1 -- 172.21.15.98:0/660714258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba02254a0 msgr2=0x55cba022ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:44.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f967bfff640 1 --2- 172.21.15.98:0/660714258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba02254a0 0x55cba022ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f967bfff640 1 -- 172.21.15.98:0/660714258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb9fefc3c0 con 0x55cba0225dc0 2024-09-19T08:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f9673fff640 1 --2- 172.21.15.98:0/660714258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0230680 0x55cba022d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f967bfff640 1 --2- 172.21.15.98:0/660714258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0225dc0 0x55cba022d210 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f966c010ca0 tx=0x7f966c012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b0bf050ad6daa48 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f965b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f965b7fe640 1 -- 172.21.15.98:0/660714258 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c004020 con 0x55cba0225dc0 2024-09-19T08:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f965b7fe640 1 -- 172.21.15.98:0/660714258 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f966c012e00 con 0x55cba0225dc0 2024-09-19T08:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f9684dc2740 1 -- 172.21.15.98:0/660714258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f966c01cb00 con 0x55cba0225dc0 2024-09-19T08:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f965b7fe640 1 -- 172.21.15.98:0/660714258 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f966c0041c0 con 0x55cba0225dc0 2024-09-19T08:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f9684dc2740 10 client.17868 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f9684dc2740 1 -- 172.21.15.98:0/660714258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cba023ebd0 con 0x55cba0225dc0 2024-09-19T08:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f965b7fe640 1 -- 172.21.15.98:0/660714258 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f966c01cb00 con 0x55cba0225dc0 2024-09-19T08:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.112+0000 7f965b7fe640 1 -- 172.21.15.98:0/660714258 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f966c0052a0 con 0x55cba0225dc0 2024-09-19T08:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.113+0000 7f9684dc2740 10 client.17868 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.113+0000 7f9684dc2740 10 client.17868 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:13:44.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.113+0000 7f9684dc2740 20 client.17868 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:44.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.113+0000 7f9684dc2740 1 --2- 172.21.15.98:0/660714258 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55cba0240030 0x55cba0242420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.113+0000 7f9684dc2740 4 client.17868 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:13:44.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.113+0000 7f9684dc2740 1 -- 172.21.15.98:0/660714258 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cba0198850 con 0x55cba0240030 2024-09-19T08:13:44.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.113+0000 7f9673fff640 1 --2- 172.21.15.98:0/660714258 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55cba0240030 0x55cba0242420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.114+0000 7f9673fff640 1 --2- 172.21.15.98:0/660714258 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55cba0240030 0x55cba0242420 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cb257994e61a06e server_cookie=41f4940d3876bd02 in_seq=0 out_seq=0 2024-09-19T08:13:44.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.114+0000 7f965b7fe640 10 client.17868 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:44.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f965b7fe640 1 -- 172.21.15.98:0/660714258 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cba0198850 con 0x55cba0240030 2024-09-19T08:13:44.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f965b7fe640 10 client.17868 handle_command_reply: tid=0 2024-09-19T08:13:44.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f9684dc2740 1 client.17868 shutdown 2024-09-19T08:13:44.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f9684dc2740 1 -- 172.21.15.98:0/660714258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0225dc0 msgr2=0x55cba022d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:44.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f9684dc2740 1 --2- 172.21.15.98:0/660714258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0225dc0 0x55cba022d210 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f966c010ca0 tx=0x7f966c012be0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f9684dc2740 1 -- 172.21.15.98:0/660714258 shutdown_connections 2024-09-19T08:13:44.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f9684dc2740 1 --2- 172.21.15.98:0/660714258 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55cba0240030 0x55cba0242420 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f9684dc2740 1 --2- 172.21.15.98:0/660714258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0230680 0x55cba022d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f9684dc2740 1 --2- 172.21.15.98:0/660714258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0225dc0 0x55cba022d210 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f965b7fe640 0 client.17868 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:44.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f9684dc2740 1 --2- 172.21.15.98:0/660714258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba02254a0 0x55cba022ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f9684dc2740 1 -- 172.21.15.98:0/660714258 >> 172.21.15.98:0/660714258 conn(0x55cba0221a30 msgr2=0x55cba0227710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:44.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f9684dc2740 1 -- 172.21.15.98:0/660714258 shutdown_connections 2024-09-19T08:13:44.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f9684dc2740 1 -- 172.21.15.98:0/660714258 wait complete. 2024-09-19T08:13:44.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.115+0000 7f9684dc2740 20 client.17868 trim_cache size 0 max 16384 2024-09-19T08:13:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9684dc2740 1 Processor -- start 2024-09-19T08:13:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9684dc2740 1 -- start start 2024-09-19T08:13:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9684dc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba02254a0 0x55cba0247be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9684dc2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba0225dc0 0x55cba0248120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9684dc2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0230680 0x55cba022a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9684dc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f967c1ad950 con 0x55cba02254a0 2024-09-19T08:13:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9684dc2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f967c1af490 con 0x55cba0225dc0 2024-09-19T08:13:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9684dc2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f967c1ae020 con 0x55cba0230680 2024-09-19T08:13:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9673fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba02254a0 0x55cba0247be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9673fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba02254a0 0x55cba0247be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48390/0 (socket says 172.21.15.98:48390) 2024-09-19T08:13:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9673fff640 1 -- 172.21.15.98:0/513010411 learned_addr learned my addr 172.21.15.98:0/513010411 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f967bfff640 1 --2- 172.21.15.98:0/513010411 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba0225dc0 0x55cba0248120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9673fff640 1 -- 172.21.15.98:0/513010411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0230680 msgr2=0x55cba022a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9673fff640 1 --2- 172.21.15.98:0/513010411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0230680 0x55cba022a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9673fff640 1 -- 172.21.15.98:0/513010411 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba0225dc0 msgr2=0x55cba0248120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9673fff640 1 --2- 172.21.15.98:0/513010411 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba0225dc0 0x55cba0248120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.132+0000 7f9673fff640 1 -- 172.21.15.98:0/513010411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cba0242790 con 0x55cba02254a0 2024-09-19T08:13:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.133+0000 7f9673fff640 1 --2- 172.21.15.98:0/513010411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba02254a0 0x55cba0247be0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f96640096c0 tx=0x7f96640058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743857716d0a1ac4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.133+0000 7f9671ffb640 1 -- 172.21.15.98:0/513010411 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966404e020 con 0x55cba02254a0 2024-09-19T08:13:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.133+0000 7f9671ffb640 1 -- 172.21.15.98:0/513010411 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9664048040 con 0x55cba02254a0 2024-09-19T08:13:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.133+0000 7f9671ffb640 1 -- 172.21.15.98:0/513010411 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9664035040 con 0x55cba02254a0 2024-09-19T08:13:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.133+0000 7f9684dc2740 1 -- 172.21.15.98:0/513010411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba02254a0 msgr2=0x55cba0247be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.133+0000 7f9684dc2740 1 --2- 172.21.15.98:0/513010411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba02254a0 0x55cba0247be0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f96640096c0 tx=0x7f96640058d0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.133+0000 7f9684dc2740 1 -- 172.21.15.98:0/513010411 shutdown_connections 2024-09-19T08:13:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.133+0000 7f9684dc2740 1 --2- 172.21.15.98:0/513010411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0230680 0x55cba022a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.133+0000 7f9684dc2740 1 --2- 172.21.15.98:0/513010411 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba0225dc0 0x55cba0248120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.133+0000 7f9684dc2740 1 --2- 172.21.15.98:0/513010411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba02254a0 0x55cba0247be0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.134+0000 7f9684dc2740 1 -- 172.21.15.98:0/513010411 >> 172.21.15.98:0/513010411 conn(0x55cba0221a30 msgr2=0x55cba022c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.134+0000 7f9684dc2740 1 -- 172.21.15.98:0/513010411 shutdown_connections 2024-09-19T08:13:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.134+0000 7f9684dc2740 1 -- 172.21.15.98:0/513010411 wait complete. 2024-09-19T08:13:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.134+0000 7f9684dc2740 1 -- start start 2024-09-19T08:13:44.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.134+0000 7f9684dc2740 1 Processor -- start 2024-09-19T08:13:44.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.134+0000 7f9684dc2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba02254a0 0x55cba0238ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.134+0000 7f9684dc2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0225dc0 0x55cba0235e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.134+0000 7f9684dc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0230680 0x55cba02363b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.134+0000 7f9684dc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f967c1aedc0 con 0x55cba0230680 2024-09-19T08:13:44.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.134+0000 7f9684dc2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f967c1ae6f0 con 0x55cba02254a0 2024-09-19T08:13:44.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.134+0000 7f9684dc2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f967c1ab950 con 0x55cba0225dc0 2024-09-19T08:13:44.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.134+0000 7f9684dc2740 10 client.0 _ll_register_callbacks cb 0x7f96640314a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:44.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f9673fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0230680 0x55cba02363b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f9673fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0230680 0x55cba02363b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48394/0 (socket says 172.21.15.98:48394) 2024-09-19T08:13:44.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f9673fff640 1 -- 172.21.15.98:0/2442668376 learned_addr learned my addr 172.21.15.98:0/2442668376 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:44.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f967bfff640 1 --2- 172.21.15.98:0/2442668376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0225dc0 0x55cba0235e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f9673fff640 1 --2- 172.21.15.98:0/2442668376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba02254a0 0x55cba0238ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f9673fff640 1 -- 172.21.15.98:0/2442668376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0225dc0 msgr2=0x55cba0235e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:44.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f9673fff640 1 --2- 172.21.15.98:0/2442668376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0225dc0 0x55cba0235e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f9673fff640 1 -- 172.21.15.98:0/2442668376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba02254a0 msgr2=0x55cba0238ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:44.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f9673fff640 1 --2- 172.21.15.98:0/2442668376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba02254a0 0x55cba0238ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f9673fff640 1 -- 172.21.15.98:0/2442668376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cba02368f0 con 0x55cba0230680 2024-09-19T08:13:44.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f9673fff640 1 --2- 172.21.15.98:0/2442668376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0230680 0x55cba02363b0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f9664000c00 tx=0x7f9664003b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca7a70812f10799 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:44.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f965b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:44.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f965b7fe640 1 -- 172.21.15.98:0/2442668376 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9664058070 con 0x55cba0230680 2024-09-19T08:13:44.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f965b7fe640 1 -- 172.21.15.98:0/2442668376 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f9664004040 con 0x55cba0230680 2024-09-19T08:13:44.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f9684dc2740 1 -- 172.21.15.98:0/2442668376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9664048040 con 0x55cba0230680 2024-09-19T08:13:44.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f965b7fe640 1 -- 172.21.15.98:0/2442668376 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f966404e030 con 0x55cba0230680 2024-09-19T08:13:44.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f9684dc2740 10 client.17874 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:44.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.135+0000 7f9684dc2740 1 -- 172.21.15.98:0/2442668376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cba02aa900 con 0x55cba0230680 2024-09-19T08:13:44.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.136+0000 7f965b7fe640 1 -- 172.21.15.98:0/2442668376 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9664048040 con 0x55cba0230680 2024-09-19T08:13:44.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.136+0000 7f965b7fe640 1 -- 172.21.15.98:0/2442668376 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9664004890 con 0x55cba0230680 2024-09-19T08:13:44.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.136+0000 7f9684dc2740 10 client.17874 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:44.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.136+0000 7f9684dc2740 10 client.17874 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:13:44.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.136+0000 7f9684dc2740 20 client.17874 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:44.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.136+0000 7f9684dc2740 1 --2- 172.21.15.98:0/2442668376 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55cba02aaf70 0x55cba02ad390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:44.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.136+0000 7f9684dc2740 4 client.17874 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-19T08:13:44.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.136+0000 7f9684dc2740 1 -- 172.21.15.98:0/2442668376 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cba0192d60 con 0x55cba02aaf70 2024-09-19T08:13:44.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.136+0000 7f967bfff640 1 --2- 172.21.15.98:0/2442668376 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55cba02aaf70 0x55cba02ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.136+0000 7f965b7fe640 10 client.17874 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.136+0000 7f967bfff640 1 --2- 172.21.15.98:0/2442668376 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55cba02aaf70 0x55cba02ad390 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2d35a1658355938 server_cookie=eb3c230bc554c740 in_seq=0 out_seq=0 2024-09-19T08:13:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f965b7fe640 1 -- 172.21.15.98:0/2442668376 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cba0192d60 con 0x55cba02aaf70 2024-09-19T08:13:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f965b7fe640 10 client.17874 handle_command_reply: tid=0 2024-09-19T08:13:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f9684dc2740 1 client.17874 shutdown 2024-09-19T08:13:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f9684dc2740 1 -- 172.21.15.98:0/2442668376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0230680 msgr2=0x55cba02363b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f9684dc2740 1 --2- 172.21.15.98:0/2442668376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0230680 0x55cba02363b0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f9664000c00 tx=0x7f9664003b70 comp rx=0 tx=0).stop 2024-09-19T08:13:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f9684dc2740 1 -- 172.21.15.98:0/2442668376 shutdown_connections 2024-09-19T08:13:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f9684dc2740 1 --2- 172.21.15.98:0/2442668376 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x55cba02aaf70 0x55cba02ad390 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f9684dc2740 1 --2- 172.21.15.98:0/2442668376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cba0230680 0x55cba02363b0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f9684dc2740 1 --2- 172.21.15.98:0/2442668376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cba0225dc0 0x55cba0235e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f965b7fe640 0 client.17874 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:44.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f9684dc2740 1 --2- 172.21.15.98:0/2442668376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cba02254a0 0x55cba0238ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f9684dc2740 1 -- 172.21.15.98:0/2442668376 >> 172.21.15.98:0/2442668376 conn(0x55cba0221a30 msgr2=0x55cba0220350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:44.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f9684dc2740 1 -- 172.21.15.98:0/2442668376 shutdown_connections 2024-09-19T08:13:44.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f9684dc2740 1 -- 172.21.15.98:0/2442668376 wait complete. 2024-09-19T08:13:44.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f9684dc2740 20 client.17874 trim_cache size 0 max 16384 2024-09-19T08:13:44.144 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:13:44.144 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-09-19T08:13:44.145 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-09-19T08:13:44.145 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:13:44.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f96829e9640 1 -- 172.21.15.98:0/1330781119 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f96680a3430 msgr2=0x7f96680a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:44.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f96829e9640 1 --2- 172.21.15.98:0/1330781119 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f96680a3430 0x7f96680a58f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9674000f80 tx=0x7f967406df50 comp rx=0 tx=0).stop 2024-09-19T08:13:44.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f96829e9640 1 -- 172.21.15.98:0/1330781119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c106a80 msgr2=0x7f967c19ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:44.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.137+0000 7f96829e9640 1 --2- 172.21.15.98:0/1330781119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c106a80 0x7f967c19ccb0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f967400eaa0 tx=0x7f967400ef60 comp rx=0 tx=0).stop 2024-09-19T08:13:44.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.138+0000 7f96829e9640 1 -- 172.21.15.98:0/1330781119 shutdown_connections 2024-09-19T08:13:44.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.138+0000 7f96829e9640 1 --2- 172.21.15.98:0/1330781119 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f96680a3430 0x7f96680a58f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.138+0000 7f96829e9640 1 --2- 172.21.15.98:0/1330781119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f967c106a80 0x7f967c19ccb0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.138+0000 7f96829e9640 1 --2- 172.21.15.98:0/1330781119 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f967c101420 0x7f967c198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.138+0000 7f96829e9640 1 --2- 172.21.15.98:0/1330781119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f967c100a80 0x7f967c1983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:44.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.138+0000 7f96829e9640 1 -- 172.21.15.98:0/1330781119 >> 172.21.15.98:0/1330781119 conn(0x7f967c0fc7a0 msgr2=0x7f967c06aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:44.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.138+0000 7f96829e9640 1 -- 172.21.15.98:0/1330781119 shutdown_connections 2024-09-19T08:13:44.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:44.138+0000 7f96829e9640 1 -- 172.21.15.98:0/1330781119 wait complete. 2024-09-19T08:13:44.956 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-19T08:13:44.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:13:45.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:44 smithi139 ceph-mon[34874]: pgmap v332: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:13:45.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:44 smithi098 ceph-mon[31333]: pgmap v332: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:13:45.202 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:45.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:44 smithi162 ceph-mon[35260]: pgmap v332: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:13:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.643+0000 7f945e298640 1 -- 172.21.15.98:0/548300669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9458106a80 msgr2=0x7f9458106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.643+0000 7f945e298640 1 --2- 172.21.15.98:0/548300669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9458106a80 0x7f9458106e60 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f9448009930 tx=0x7f944802a420 comp rx=0 tx=0).stop 2024-09-19T08:13:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.643+0000 7f945e298640 1 -- 172.21.15.98:0/548300669 shutdown_connections 2024-09-19T08:13:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.643+0000 7f945e298640 1 --2- 172.21.15.98:0/548300669 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9458101420 0x7f945810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.643+0000 7f945e298640 1 --2- 172.21.15.98:0/548300669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9458100a80 0x7f9458100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.643+0000 7f945e298640 1 --2- 172.21.15.98:0/548300669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9458106a80 0x7f9458106e60 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.643+0000 7f945e298640 1 -- 172.21.15.98:0/548300669 >> 172.21.15.98:0/548300669 conn(0x7f94580fc7a0 msgr2=0x7f94580febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.643+0000 7f945e298640 1 -- 172.21.15.98:0/548300669 shutdown_connections 2024-09-19T08:13:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.643+0000 7f945e298640 1 -- 172.21.15.98:0/548300669 wait complete. 2024-09-19T08:13:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.645+0000 7f945e298640 1 Processor -- start 2024-09-19T08:13:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.645+0000 7f945e298640 1 -- start start 2024-09-19T08:13:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.645+0000 7f945e298640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9458100a80 0x7f94581992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.645+0000 7f945e298640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9458101420 0x7f9458199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.645+0000 7f945e298640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9458106a80 0x7f94581a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.645+0000 7f945e298640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9458110e80 con 0x7f9458101420 2024-09-19T08:13:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.645+0000 7f945e298640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9458110d00 con 0x7f9458100a80 2024-09-19T08:13:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.645+0000 7f945e298640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9458111000 con 0x7f9458106a80 2024-09-19T08:13:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.645+0000 7f94577fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9458101420 0x7f9458199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.645+0000 7f94577fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9458101420 0x7f9458199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48408/0 (socket says 172.21.15.98:48408) 2024-09-19T08:13:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.645+0000 7f94577fe640 1 -- 172.21.15.98:0/318443106 learned_addr learned my addr 172.21.15.98:0/318443106 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.645+0000 7f9457fff640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9458100a80 0x7f94581992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.646+0000 7f945c835640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9458106a80 0x7f94581a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.646+0000 7f94577fe640 1 -- 172.21.15.98:0/318443106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9458106a80 msgr2=0x7f94581a4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.646+0000 7f94577fe640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9458106a80 0x7f94581a4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.646+0000 7f94577fe640 1 -- 172.21.15.98:0/318443106 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9458100a80 msgr2=0x7f94581992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.646+0000 7f94577fe640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9458100a80 0x7f94581992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:45.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.646+0000 7f94577fe640 1 -- 172.21.15.98:0/318443106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94581a4890 con 0x7f9458101420 2024-09-19T08:13:45.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.646+0000 7f945c835640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9458106a80 0x7f94581a4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:45.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.646+0000 7f94577fe640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9458101420 0x7f9458199810 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f944000c970 tx=0x7f944000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:45.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.646+0000 7f94557fa640 1 -- 172.21.15.98:0/318443106 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9440007bb0 con 0x7f9458101420 2024-09-19T08:13:45.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.646+0000 7f94557fa640 1 -- 172.21.15.98:0/318443106 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9440007d50 con 0x7f9458101420 2024-09-19T08:13:45.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.646+0000 7f945e298640 1 -- 172.21.15.98:0/318443106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94581a4a60 con 0x7f9458101420 2024-09-19T08:13:45.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.646+0000 7f94557fa640 1 -- 172.21.15.98:0/318443106 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944000f450 con 0x7f9458101420 2024-09-19T08:13:45.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.646+0000 7f945e298640 1 -- 172.21.15.98:0/318443106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94581a4f20 con 0x7f9458101420 2024-09-19T08:13:45.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.647+0000 7f945e298640 1 -- 172.21.15.98:0/318443106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9458100ee0 con 0x7f9458101420 2024-09-19T08:13:45.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.648+0000 7f94557fa640 1 -- 172.21.15.98:0/318443106 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f9440005020 con 0x7f9458101420 2024-09-19T08:13:45.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.649+0000 7f94557fa640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94440a33b0 0x7f94440a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:45.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.649+0000 7f9457fff640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94440a33b0 0x7f94440a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:45.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.649+0000 7f94557fa640 1 -- 172.21.15.98:0/318443106 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f94400cfb00 con 0x7f9458101420 2024-09-19T08:13:45.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.650+0000 7f9457fff640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94440a33b0 0x7f94440a5870 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9448009900 tx=0x7f94480058d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:45.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.654+0000 7f94557fa640 1 -- 172.21.15.98:0/318443106 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f9440014030 con 0x7f9458101420 2024-09-19T08:13:45.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.854+0000 7f945e298640 1 -- 172.21.15.98:0/318443106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9458108ff0 con 0x7f9458101420 2024-09-19T08:13:45.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.855+0000 7f94557fa640 1 -- 172.21.15.98:0/318443106 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11983 (secure 0 0 0) 0x7f944009c4e0 con 0x7f9458101420 2024-09-19T08:13:45.856 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:13:45.856 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"btime":"2024-09-19T08:08:26:985720+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:08:26.555658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:13:45.857 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2024-09-19T08:13:45.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.857+0000 7f945e298640 1 -- 172.21.15.98:0/318443106 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94440a33b0 msgr2=0x7f94440a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:45.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.857+0000 7f945e298640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94440a33b0 0x7f94440a5870 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9448009900 tx=0x7f94480058d0 comp rx=0 tx=0).stop 2024-09-19T08:13:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.858+0000 7f945e298640 1 -- 172.21.15.98:0/318443106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9458101420 msgr2=0x7f9458199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.858+0000 7f945e298640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9458101420 0x7f9458199810 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f944000c970 tx=0x7f944000ce30 comp rx=0 tx=0).stop 2024-09-19T08:13:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.858+0000 7f945e298640 1 -- 172.21.15.98:0/318443106 shutdown_connections 2024-09-19T08:13:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.858+0000 7f945e298640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f94440a33b0 0x7f94440a5870 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.858+0000 7f945e298640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9458106a80 0x7f94581a4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.858+0000 7f945e298640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9458101420 0x7f9458199810 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.858+0000 7f945e298640 1 --2- 172.21.15.98:0/318443106 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9458100a80 0x7f94581992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.858+0000 7f945e298640 1 -- 172.21.15.98:0/318443106 >> 172.21.15.98:0/318443106 conn(0x7f94580fc7a0 msgr2=0x7f94580fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.858+0000 7f945e298640 1 -- 172.21.15.98:0/318443106 shutdown_connections 2024-09-19T08:13:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:45.858+0000 7f945e298640 1 -- 172.21.15.98:0/318443106 wait complete. 2024-09-19T08:13:45.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:45 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/318443106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:13:46.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:45 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/318443106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:13:46.271 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:45 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/318443106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:13:46.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-19T08:13:46.694 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:47.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:46 smithi098 ceph-mon[31333]: pgmap v333: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 -- 172.21.15.98:0/1391370847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78a010c170 msgr2=0x7f78a010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 --2- 172.21.15.98:0/1391370847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78a010c170 0x7f78a010e610 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f7898009ca0 tx=0x7f789802a950 comp rx=0 tx=0).stop 2024-09-19T08:13:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 -- 172.21.15.98:0/1391370847 shutdown_connections 2024-09-19T08:13:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 --2- 172.21.15.98:0/1391370847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78a010c170 0x7f78a010e610 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 --2- 172.21.15.98:0/1391370847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a0103660 0x7f78a010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 --2- 172.21.15.98:0/1391370847 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78a0102d40 0x7f78a0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 -- 172.21.15.98:0/1391370847 >> 172.21.15.98:0/1391370847 conn(0x7f78a00fc7a0 msgr2=0x7f78a00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 -- 172.21.15.98:0/1391370847 shutdown_connections 2024-09-19T08:13:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 -- 172.21.15.98:0/1391370847 wait complete. 2024-09-19T08:13:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 Processor -- start 2024-09-19T08:13:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 -- start start 2024-09-19T08:13:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a0102d40 0x7f78a0198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78a0103660 0x7f78a0198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78a010c170 0x7f78a019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.171+0000 7f78a710b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78a0111fb0 con 0x7f78a0103660 2024-09-19T08:13:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f78a710b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78a0111e30 con 0x7f78a010c170 2024-09-19T08:13:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f78a710b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78a0112130 con 0x7f78a0102d40 2024-09-19T08:13:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f789ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78a0103660 0x7f78a0198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f789ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78a0103660 0x7f78a0198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37606/0 (socket says 172.21.15.98:37606) 2024-09-19T08:13:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f789ffff640 1 -- 172.21.15.98:0/83093308 learned_addr learned my addr 172.21.15.98:0/83093308 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f78a4ea7640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a0102d40 0x7f78a0198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f78a56a8640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78a010c170 0x7f78a019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f789ffff640 1 -- 172.21.15.98:0/83093308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a0102d40 msgr2=0x7f78a0198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f789ffff640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a0102d40 0x7f78a0198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f789ffff640 1 -- 172.21.15.98:0/83093308 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78a010c170 msgr2=0x7f78a019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f789ffff640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78a010c170 0x7f78a019cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f789ffff640 1 -- 172.21.15.98:0/83093308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a019d3e0 con 0x7f78a0103660 2024-09-19T08:13:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f78a4ea7640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a0102d40 0x7f78a0198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f789ffff640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78a0103660 0x7f78a0198970 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f788c00b3f0 tx=0x7f788c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f789dffb640 1 -- 172.21.15.98:0/83093308 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c015020 con 0x7f78a0103660 2024-09-19T08:13:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f789dffb640 1 -- 172.21.15.98:0/83093308 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f788c011d10 con 0x7f78a0103660 2024-09-19T08:13:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f789dffb640 1 -- 172.21.15.98:0/83093308 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c00f2e0 con 0x7f78a0103660 2024-09-19T08:13:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.172+0000 7f78a710b640 1 -- 172.21.15.98:0/83093308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a019d6d0 con 0x7f78a0103660 2024-09-19T08:13:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.173+0000 7f78a710b640 1 -- 172.21.15.98:0/83093308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a01a4fb0 con 0x7f78a0103660 2024-09-19T08:13:47.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:46 smithi139 ceph-mon[34874]: pgmap v333: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.175+0000 7f789dffb640 1 -- 172.21.15.98:0/83093308 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f788c018020 con 0x7f78a0103660 2024-09-19T08:13:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.175+0000 7f78a94e4740 1 Processor -- start 2024-09-19T08:13:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.175+0000 7f78a94e4740 1 -- start start 2024-09-19T08:13:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.175+0000 7f78a94e4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e792168b40 0x55e792167090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.175+0000 7f78a94e4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e7921675d0 0x55e792167a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.175+0000 7f78a94e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 0x55e792174760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.175+0000 7f78a94e4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e7920cd2f0 con 0x55e792168b40 2024-09-19T08:13:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.175+0000 7f78a94e4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e792074cf0 con 0x55e7921675d0 2024-09-19T08:13:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.175+0000 7f78a94e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e7920c92b0 con 0x55e792172370 2024-09-19T08:13:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.175+0000 7f789dffb640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f78900a33b0 0x7f78900a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.175+0000 7f78a56a8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e792168b40 0x55e792167090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.175+0000 7f78a56a8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e792168b40 0x55e792167090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37612/0 (socket says 172.21.15.98:37612) 2024-09-19T08:13:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.175+0000 7f78a56a8640 1 -- 172.21.15.98:0/3591218145 learned_addr learned my addr 172.21.15.98:0/3591218145 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f78a4ea7640 1 --2- 172.21.15.98:0/3591218145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e7921675d0 0x55e792167a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f78a56a8640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f78900a33b0 0x7f78900a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f789dffb640 1 -- 172.21.15.98:0/83093308 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f788c0cf550 con 0x7f78a0103660 2024-09-19T08:13:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f789ffff640 1 --2- 172.21.15.98:0/3591218145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 0x55e792174760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f78a56a8640 1 -- 172.21.15.98:0/3591218145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 msgr2=0x55e792174760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f78a56a8640 1 --2- 172.21.15.98:0/3591218145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 0x55e792174760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f78a56a8640 1 -- 172.21.15.98:0/3591218145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e7921675d0 msgr2=0x55e792167a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f78a56a8640 1 --2- 172.21.15.98:0/3591218145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e7921675d0 0x55e792167a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f78a56a8640 1 -- 172.21.15.98:0/3591218145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7920593a0 con 0x55e792168b40 2024-09-19T08:13:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f78a4ea7640 1 --2- 172.21.15.98:0/3591218145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e7921675d0 0x55e792167a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T08:13:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f78a56a8640 1 --2- 172.21.15.98:0/3591218145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e792168b40 0x55e792167090 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f7898030040 tx=0x7f789802f680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14c986ef228a6bc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f78a56a8640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f78900a33b0 0x7f78900a5870 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f789802f6f0 tx=0x7f7898041000 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f788affd640 1 -- 172.21.15.98:0/3591218145 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7898034040 con 0x55e792168b40 2024-09-19T08:13:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f788affd640 1 -- 172.21.15.98:0/3591218145 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7898003e30 con 0x55e792168b40 2024-09-19T08:13:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.176+0000 7f788affd640 1 -- 172.21.15.98:0/3591218145 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7898004780 con 0x55e792168b40 2024-09-19T08:13:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.177+0000 7f78a94e4740 1 -- 172.21.15.98:0/3591218145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e792168b40 msgr2=0x55e792167090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.177+0000 7f78a94e4740 1 --2- 172.21.15.98:0/3591218145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e792168b40 0x55e792167090 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f7898030040 tx=0x7f789802f680 comp rx=0 tx=0).stop 2024-09-19T08:13:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.177+0000 7f78a94e4740 1 -- 172.21.15.98:0/3591218145 shutdown_connections 2024-09-19T08:13:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.177+0000 7f78a94e4740 1 --2- 172.21.15.98:0/3591218145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 0x55e792174760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.177+0000 7f78a94e4740 1 --2- 172.21.15.98:0/3591218145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e7921675d0 0x55e792167a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.177+0000 7f78a94e4740 1 --2- 172.21.15.98:0/3591218145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e792168b40 0x55e792167090 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.177+0000 7f78a94e4740 1 -- 172.21.15.98:0/3591218145 >> 172.21.15.98:0/3591218145 conn(0x55e792130310 msgr2=0x55e792163c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.177+0000 7f78a94e4740 1 -- 172.21.15.98:0/3591218145 shutdown_connections 2024-09-19T08:13:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.177+0000 7f78a94e4740 1 -- 172.21.15.98:0/3591218145 wait complete. 2024-09-19T08:13:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.177+0000 7f78a94e4740 1 -- start start 2024-09-19T08:13:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a94e4740 1 Processor -- start 2024-09-19T08:13:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a94e4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e792171e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a94e4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792168b40 0x55e79216f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a94e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 0x55e79216f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a94e4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e792175010 con 0x55e7921675d0 2024-09-19T08:13:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a94e4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e792088060 con 0x55e792168b40 2024-09-19T08:13:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a94e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e792175190 con 0x55e792172370 2024-09-19T08:13:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a94e4740 10 client.0 _ll_register_callbacks cb 0x55e791ddb8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a4ea7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e792171e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a4ea7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e792171e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37624/0 (socket says 172.21.15.98:37624) 2024-09-19T08:13:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a4ea7640 1 -- 172.21.15.98:0/2127233901 learned_addr learned my addr 172.21.15.98:0/2127233901 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a56a8640 1 --2- 172.21.15.98:0/2127233901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 0x55e79216f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f789ffff640 1 --2- 172.21.15.98:0/2127233901 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792168b40 0x55e79216f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a4ea7640 1 -- 172.21.15.98:0/2127233901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 msgr2=0x55e79216f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a4ea7640 1 --2- 172.21.15.98:0/2127233901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 0x55e79216f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a4ea7640 1 -- 172.21.15.98:0/2127233901 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792168b40 msgr2=0x55e79216f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a4ea7640 1 --2- 172.21.15.98:0/2127233901 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792168b40 0x55e79216f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f78a4ea7640 1 -- 172.21.15.98:0/2127233901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e791e3e3c0 con 0x55e7921675d0 2024-09-19T08:13:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.178+0000 7f789ffff640 1 --2- 172.21.15.98:0/2127233901 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792168b40 0x55e79216f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f78a4ea7640 1 --2- 172.21.15.98:0/2127233901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e792171e00 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f7894010770 tx=0x7f7894010c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90e2f5b6058e1ac server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f787ffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f787ffff640 1 -- 172.21.15.98:0/2127233901 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789401d060 con 0x55e7921675d0 2024-09-19T08:13:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f787ffff640 1 -- 172.21.15.98:0/2127233901 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f789400eda0 con 0x55e7921675d0 2024-09-19T08:13:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f78a94e4740 1 -- 172.21.15.98:0/2127233901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7898003e30 con 0x55e7921675d0 2024-09-19T08:13:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f787ffff640 1 -- 172.21.15.98:0/2127233901 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7894018430 con 0x55e7921675d0 2024-09-19T08:13:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f78a94e4740 10 client.17892 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f78a94e4740 1 -- 172.21.15.98:0/2127233901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e792180bc0 con 0x55e7921675d0 2024-09-19T08:13:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f787ffff640 1 -- 172.21.15.98:0/2127233901 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7898003e30 con 0x55e7921675d0 2024-09-19T08:13:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f787ffff640 1 -- 172.21.15.98:0/2127233901 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7894020790 con 0x55e7921675d0 2024-09-19T08:13:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f78a94e4740 10 client.17892 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f78a94e4740 10 client.17892 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:13:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f78a94e4740 20 client.17892 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f78a94e4740 1 --2- 172.21.15.98:0/2127233901 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e792182080 0x55e7921844d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f78a94e4740 4 client.17892 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:13:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.179+0000 7f78a94e4740 1 -- 172.21.15.98:0/2127233901 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7920da860 con 0x55e792182080 2024-09-19T08:13:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.180+0000 7f789ffff640 1 --2- 172.21.15.98:0/2127233901 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e792182080 0x55e7921844d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.180+0000 7f789ffff640 1 --2- 172.21.15.98:0/2127233901 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e792182080 0x55e7921844d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3175eb68238cfa18 server_cookie=950be5a6abf73bb4 in_seq=0 out_seq=0 2024-09-19T08:13:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.180+0000 7f787ffff640 10 client.17892 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:13:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f787ffff640 1 -- 172.21.15.98:0/2127233901 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e7920da860 con 0x55e792182080 2024-09-19T08:13:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f787ffff640 10 client.17892 handle_command_reply: tid=0 2024-09-19T08:13:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f78a94e4740 1 client.17892 shutdown 2024-09-19T08:13:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f78a94e4740 1 -- 172.21.15.98:0/2127233901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 msgr2=0x55e792171e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f78a94e4740 1 --2- 172.21.15.98:0/2127233901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e792171e00 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f7894010770 tx=0x7f7894010c30 comp rx=0 tx=0).stop 2024-09-19T08:13:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f78a94e4740 1 -- 172.21.15.98:0/2127233901 shutdown_connections 2024-09-19T08:13:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f78a94e4740 1 --2- 172.21.15.98:0/2127233901 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e792182080 0x55e7921844d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f78a94e4740 1 --2- 172.21.15.98:0/2127233901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 0x55e79216f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f78a94e4740 1 --2- 172.21.15.98:0/2127233901 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792168b40 0x55e79216f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f787ffff640 0 client.17892 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:13:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f78a94e4740 1 --2- 172.21.15.98:0/2127233901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e792171e00 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f78a94e4740 1 -- 172.21.15.98:0/2127233901 >> 172.21.15.98:0/2127233901 conn(0x55e792130310 msgr2=0x55e792169960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f78a94e4740 1 -- 172.21.15.98:0/2127233901 shutdown_connections 2024-09-19T08:13:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f78a94e4740 1 -- 172.21.15.98:0/2127233901 wait complete. 2024-09-19T08:13:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.181+0000 7f78a94e4740 20 client.17892 trim_cache size 0 max 16384 2024-09-19T08:13:47.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a94e4740 1 Processor -- start 2024-09-19T08:13:47.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a94e4740 1 -- start start 2024-09-19T08:13:47.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a94e4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e792162c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a94e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792168b40 0x55e792163150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a94e4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792172370 0x55e79216a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a94e4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78a01ad8c0 con 0x55e7921675d0 2024-09-19T08:13:47.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a94e4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78a01af400 con 0x55e792172370 2024-09-19T08:13:47.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a94e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78a01adf90 con 0x55e792168b40 2024-09-19T08:13:47.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a4ea7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e792162c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a4ea7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e792162c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37632/0 (socket says 172.21.15.98:37632) 2024-09-19T08:13:47.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a4ea7640 1 -- 172.21.15.98:0/2224602407 learned_addr learned my addr 172.21.15.98:0/2224602407 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:47.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f789ffff640 1 --2- 172.21.15.98:0/2224602407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792168b40 0x55e792163150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a4ea7640 1 -- 172.21.15.98:0/2224602407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792168b40 msgr2=0x55e792163150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:47.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a4ea7640 1 --2- 172.21.15.98:0/2224602407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792168b40 0x55e792163150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a4ea7640 1 -- 172.21.15.98:0/2224602407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792172370 msgr2=0x55e79216a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:47.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a4ea7640 1 --2- 172.21.15.98:0/2224602407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792172370 0x55e79216a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.194+0000 7f78a4ea7640 1 -- 172.21.15.98:0/2224602407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e792184b50 con 0x55e7921675d0 2024-09-19T08:13:47.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.195+0000 7f78a4ea7640 1 --2- 172.21.15.98:0/2224602407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e792162c10 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f789400bc80 tx=0x7f7894005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a34092fcbbe46722 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:47.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.195+0000 7f788a7fc640 1 -- 172.21.15.98:0/2224602407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7894005df0 con 0x55e7921675d0 2024-09-19T08:13:47.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.195+0000 7f788a7fc640 1 -- 172.21.15.98:0/2224602407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7894030070 con 0x55e7921675d0 2024-09-19T08:13:47.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.195+0000 7f788a7fc640 1 -- 172.21.15.98:0/2224602407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78940189c0 con 0x55e7921675d0 2024-09-19T08:13:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.195+0000 7f78a94e4740 1 -- 172.21.15.98:0/2224602407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 msgr2=0x55e792162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.195+0000 7f78a94e4740 1 --2- 172.21.15.98:0/2224602407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e792162c10 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f789400bc80 tx=0x7f7894005bc0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.195+0000 7f78a94e4740 1 -- 172.21.15.98:0/2224602407 shutdown_connections 2024-09-19T08:13:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.195+0000 7f78a94e4740 1 --2- 172.21.15.98:0/2224602407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792172370 0x55e79216a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.195+0000 7f78a94e4740 1 --2- 172.21.15.98:0/2224602407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792168b40 0x55e792163150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.195+0000 7f78a94e4740 1 --2- 172.21.15.98:0/2224602407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e792162c10 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.195+0000 7f78a94e4740 1 -- 172.21.15.98:0/2224602407 >> 172.21.15.98:0/2224602407 conn(0x55e792130310 msgr2=0x55e792162500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:47.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.196+0000 7f78a94e4740 1 -- 172.21.15.98:0/2224602407 shutdown_connections 2024-09-19T08:13:47.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.196+0000 7f78a94e4740 1 -- 172.21.15.98:0/2224602407 wait complete. 2024-09-19T08:13:47.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.196+0000 7f78a94e4740 1 -- start start 2024-09-19T08:13:47.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.196+0000 7f78a94e4740 1 Processor -- start 2024-09-19T08:13:47.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.196+0000 7f78a94e4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e79216c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.196+0000 7f78a94e4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792168b40 0x55e79216c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.196+0000 7f78a94e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 0x55e79216cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.196+0000 7f78a94e4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78a01aed30 con 0x55e7921675d0 2024-09-19T08:13:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.196+0000 7f78a94e4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78a01ae660 con 0x55e792168b40 2024-09-19T08:13:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.196+0000 7f78a94e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78a01ab8c0 con 0x55e792172370 2024-09-19T08:13:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.196+0000 7f78a4ea7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e79216c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f78a4ea7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e79216c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37648/0 (socket says 172.21.15.98:37648) 2024-09-19T08:13:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f78a4ea7640 1 -- 172.21.15.98:0/1559810875 learned_addr learned my addr 172.21.15.98:0/1559810875 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f78a94e4740 10 client.0 _ll_register_callbacks cb 0x7f789400b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f789ffff640 1 --2- 172.21.15.98:0/1559810875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792168b40 0x55e79216c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f78a56a8640 1 --2- 172.21.15.98:0/1559810875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 0x55e79216cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f78a4ea7640 1 -- 172.21.15.98:0/1559810875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 msgr2=0x55e79216cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f78a4ea7640 1 --2- 172.21.15.98:0/1559810875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 0x55e79216cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f78a4ea7640 1 -- 172.21.15.98:0/1559810875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792168b40 msgr2=0x55e79216c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f78a4ea7640 1 --2- 172.21.15.98:0/1559810875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792168b40 0x55e79216c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f78a4ea7640 1 -- 172.21.15.98:0/1559810875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7921ebd70 con 0x55e7921675d0 2024-09-19T08:13:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f78a56a8640 1 --2- 172.21.15.98:0/1559810875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 0x55e79216cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f78a4ea7640 1 --2- 172.21.15.98:0/1559810875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e79216c0e0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f7894020d70 tx=0x7f789400bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb63239545e14ba7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f787ffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f787ffff640 1 -- 172.21.15.98:0/1559810875 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789401d040 con 0x55e7921675d0 2024-09-19T08:13:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f787ffff640 1 -- 172.21.15.98:0/1559810875 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f7894010040 con 0x55e7921675d0 2024-09-19T08:13:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f78a94e4740 1 -- 172.21.15.98:0/1559810875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7894030070 con 0x55e7921675d0 2024-09-19T08:13:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f787ffff640 1 -- 172.21.15.98:0/1559810875 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7894041050 con 0x55e7921675d0 2024-09-19T08:13:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f78a94e4740 10 client.17898 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.197+0000 7f78a94e4740 1 -- 172.21.15.98:0/1559810875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e7921ed010 con 0x55e7921675d0 2024-09-19T08:13:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.198+0000 7f787ffff640 1 -- 172.21.15.98:0/1559810875 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7894030070 con 0x55e7921675d0 2024-09-19T08:13:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.198+0000 7f787ffff640 1 -- 172.21.15.98:0/1559810875 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f789402a070 con 0x55e7921675d0 2024-09-19T08:13:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.198+0000 7f78a94e4740 10 client.17898 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.198+0000 7f78a94e4740 10 client.17898 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-09-19T08:13:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.198+0000 7f78a94e4740 20 client.17898 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.198+0000 7f78a94e4740 1 --2- 172.21.15.98:0/1559810875 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e7921ed610 0x55e7921efa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.198+0000 7f78a94e4740 4 client.17898 mds_command: new command op to 24376 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:13:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.198+0000 7f78a94e4740 1 -- 172.21.15.98:0/1559810875 --> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e7920d4d70 con 0x55e7921ed610 2024-09-19T08:13:47.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.198+0000 7f789ffff640 1 --2- 172.21.15.98:0/1559810875 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e7921ed610 0x55e7921efa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:47.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.198+0000 7f787ffff640 10 client.17898 ms_handle_connect on v2:172.21.15.162:6832/3959364141 2024-09-19T08:13:47.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.198+0000 7f789ffff640 1 --2- 172.21.15.98:0/1559810875 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e7921ed610 0x55e7921efa30 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f48d81e99a65d6fb server_cookie=42385475f78632 in_seq=0 out_seq=0 2024-09-19T08:13:47.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f787ffff640 1 -- 172.21.15.98:0/1559810875 <== mds.4 v2:172.21.15.162:6832/3959364141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e7920d4d70 con 0x55e7921ed610 2024-09-19T08:13:47.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f787ffff640 10 client.17898 handle_command_reply: tid=0 2024-09-19T08:13:47.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f78a94e4740 1 client.17898 shutdown 2024-09-19T08:13:47.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f78a94e4740 1 -- 172.21.15.98:0/1559810875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 msgr2=0x55e79216c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:47.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f78a94e4740 1 --2- 172.21.15.98:0/1559810875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e79216c0e0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f7894020d70 tx=0x7f789400bcb0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f78a94e4740 1 -- 172.21.15.98:0/1559810875 shutdown_connections 2024-09-19T08:13:47.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f78a94e4740 1 --2- 172.21.15.98:0/1559810875 >> [v2:172.21.15.162:6832/3959364141,v1:172.21.15.162:6833/3959364141] conn(0x55e7921ed610 0x55e7921efa30 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f78a94e4740 1 --2- 172.21.15.98:0/1559810875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e792172370 0x55e79216cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f78a94e4740 1 --2- 172.21.15.98:0/1559810875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e792168b40 0x55e79216c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f787ffff640 0 client.17898 ms_handle_reset on v2:172.21.15.162:6832/3959364141 2024-09-19T08:13:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f78a94e4740 1 --2- 172.21.15.98:0/1559810875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7921675d0 0x55e79216c0e0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f78a94e4740 1 -- 172.21.15.98:0/1559810875 >> 172.21.15.98:0/1559810875 conn(0x55e792130310 msgr2=0x55e792162d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f78a94e4740 1 -- 172.21.15.98:0/1559810875 shutdown_connections 2024-09-19T08:13:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f78a94e4740 1 -- 172.21.15.98:0/1559810875 wait complete. 2024-09-19T08:13:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.199+0000 7f78a94e4740 20 client.17898 trim_cache size 0 max 16384 2024-09-19T08:13:47.211 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:13:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.200+0000 7f78a710b640 1 -- 172.21.15.98:0/83093308 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f78900a33b0 msgr2=0x7f78900a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.200+0000 7f78a710b640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f78900a33b0 0x7f78900a5870 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f789802f6f0 tx=0x7f7898041000 comp rx=0 tx=0).stop 2024-09-19T08:13:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.200+0000 7f78a710b640 1 -- 172.21.15.98:0/83093308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78a0103660 msgr2=0x7f78a0198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:47.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.200+0000 7f78a710b640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78a0103660 0x7f78a0198970 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f788c00b3f0 tx=0x7f788c00b8b0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.200+0000 7f78a710b640 1 -- 172.21.15.98:0/83093308 shutdown_connections 2024-09-19T08:13:47.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.200+0000 7f78a710b640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f78900a33b0 0x7f78900a5870 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.200+0000 7f78a710b640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78a010c170 0x7f78a019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.200+0000 7f78a710b640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78a0103660 0x7f78a0198970 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.200+0000 7f78a710b640 1 --2- 172.21.15.98:0/83093308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a0102d40 0x7f78a0198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:47.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.200+0000 7f78a710b640 1 -- 172.21.15.98:0/83093308 >> 172.21.15.98:0/83093308 conn(0x7f78a00fc7a0 msgr2=0x7f78a00fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:47.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.200+0000 7f78a710b640 1 -- 172.21.15.98:0/83093308 shutdown_connections 2024-09-19T08:13:47.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:47.200+0000 7f78a710b640 1 -- 172.21.15.98:0/83093308 wait complete. 2024-09-19T08:13:47.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:46 smithi162 ceph-mon[35260]: pgmap v333: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:47.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-19T08:13:48.127 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.562+0000 7fe1d32ec640 1 -- 172.21.15.98:0/4078311086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1cc101420 msgr2=0x7fe1cc10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.562+0000 7fe1d32ec640 1 --2- 172.21.15.98:0/4078311086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1cc101420 0x7fe1cc10d950 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4009cd0 tx=0x7fe1c402a8e0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.562+0000 7fe1d32ec640 1 -- 172.21.15.98:0/4078311086 shutdown_connections 2024-09-19T08:13:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.562+0000 7fe1d32ec640 1 --2- 172.21.15.98:0/4078311086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1cc101420 0x7fe1cc10d950 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.562+0000 7fe1d32ec640 1 --2- 172.21.15.98:0/4078311086 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1cc100a80 0x7fe1cc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.562+0000 7fe1d32ec640 1 --2- 172.21.15.98:0/4078311086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1cc106a80 0x7fe1cc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.562+0000 7fe1d32ec640 1 -- 172.21.15.98:0/4078311086 >> 172.21.15.98:0/4078311086 conn(0x7fe1cc0fc7a0 msgr2=0x7fe1cc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.562+0000 7fe1d32ec640 1 -- 172.21.15.98:0/4078311086 shutdown_connections 2024-09-19T08:13:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.562+0000 7fe1d32ec640 1 -- 172.21.15.98:0/4078311086 wait complete. 2024-09-19T08:13:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d32ec640 1 Processor -- start 2024-09-19T08:13:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d32ec640 1 -- start start 2024-09-19T08:13:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d32ec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1cc100a80 0x7fe1cc198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d32ec640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1cc101420 0x7fe1cc198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d32ec640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1cc106a80 0x7fe1cc19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d32ec640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc110f20 con 0x7fe1cc100a80 2024-09-19T08:13:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d32ec640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc110da0 con 0x7fe1cc101420 2024-09-19T08:13:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d32ec640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc1110a0 con 0x7fe1cc106a80 2024-09-19T08:13:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d1088640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1cc100a80 0x7fe1cc198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d1088640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1cc100a80 0x7fe1cc198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37658/0 (socket says 172.21.15.98:37658) 2024-09-19T08:13:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d1088640 1 -- 172.21.15.98:0/1845480041 learned_addr learned my addr 172.21.15.98:0/1845480041 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d0887640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1cc101420 0x7fe1cc198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d1889640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1cc106a80 0x7fe1cc19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d1088640 1 -- 172.21.15.98:0/1845480041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1cc106a80 msgr2=0x7fe1cc19ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.563+0000 7fe1d1088640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1cc106a80 0x7fe1cc19ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.564+0000 7fe1d1088640 1 -- 172.21.15.98:0/1845480041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1cc101420 msgr2=0x7fe1cc198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.564+0000 7fe1d1088640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1cc101420 0x7fe1cc198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.564+0000 7fe1d1088640 1 -- 172.21.15.98:0/1845480041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1cc19d520 con 0x7fe1cc100a80 2024-09-19T08:13:48.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.564+0000 7fe1d1889640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1cc106a80 0x7fe1cc19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:48.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.564+0000 7fe1d0887640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1cc101420 0x7fe1cc198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:48.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.564+0000 7fe1d1088640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1cc100a80 0x7fe1cc198570 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fe1c800b4a0 tx=0x7fe1c800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:48.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.564+0000 7fe1c27fc640 1 -- 172.21.15.98:0/1845480041 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8014070 con 0x7fe1cc100a80 2024-09-19T08:13:48.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.564+0000 7fe1c27fc640 1 -- 172.21.15.98:0/1845480041 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1c8010af0 con 0x7fe1cc100a80 2024-09-19T08:13:48.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.564+0000 7fe1d32ec640 1 -- 172.21.15.98:0/1845480041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1cc19d810 con 0x7fe1cc100a80 2024-09-19T08:13:48.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.564+0000 7fe1c27fc640 1 -- 172.21.15.98:0/1845480041 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8017d10 con 0x7fe1cc100a80 2024-09-19T08:13:48.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.564+0000 7fe1d32ec640 1 -- 172.21.15.98:0/1845480041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1cc1a50f0 con 0x7fe1cc100a80 2024-09-19T08:13:48.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.566+0000 7fe1c27fc640 1 -- 172.21.15.98:0/1845480041 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fe1c8017070 con 0x7fe1cc100a80 2024-09-19T08:13:48.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.567+0000 7fe1d56c5740 1 Processor -- start 2024-09-19T08:13:48.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.567+0000 7fe1d56c5740 1 -- start start 2024-09-19T08:13:48.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.567+0000 7fe1c27fc640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1b80a33b0 0x7fe1b80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.567+0000 7fe1d56c5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fc1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.567+0000 7fe1d56c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc15d0 0x556d79fc1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.567+0000 7fe1d56c5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 0x556d79fce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.567+0000 7fe1d56c5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556d79f272f0 con 0x556d79fc2b40 2024-09-19T08:13:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.567+0000 7fe1d56c5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556d79ececf0 con 0x556d79fcc370 2024-09-19T08:13:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.567+0000 7fe1d56c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556d79f232b0 con 0x556d79fc15d0 2024-09-19T08:13:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.567+0000 7fe1d1889640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1b80a33b0 0x7fe1b80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.567+0000 7fe1d1088640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fc1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.567+0000 7fe1d1088640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fc1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37670/0 (socket says 172.21.15.98:37670) 2024-09-19T08:13:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.567+0000 7fe1d1088640 1 -- 172.21.15.98:0/709711152 learned_addr learned my addr 172.21.15.98:0/709711152 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.567+0000 7fe1d0887640 1 --2- 172.21.15.98:0/709711152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc15d0 0x556d79fc1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.568+0000 7fe1d1088640 1 -- 172.21.15.98:0/709711152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc15d0 msgr2=0x556d79fc1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.568+0000 7fe1d1088640 1 --2- 172.21.15.98:0/709711152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc15d0 0x556d79fc1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.568+0000 7fe1d1088640 1 -- 172.21.15.98:0/709711152 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 msgr2=0x556d79fce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.568+0000 7fe1d1088640 1 --2- 172.21.15.98:0/709711152 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 0x556d79fce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.568+0000 7fe1d1088640 1 -- 172.21.15.98:0/709711152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d79eb33a0 con 0x556d79fc2b40 2024-09-19T08:13:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.568+0000 7fe1c27fc640 1 -- 172.21.15.98:0/1845480041 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7fe1c80cf0f0 con 0x7fe1cc100a80 2024-09-19T08:13:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.568+0000 7fe1d1889640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1b80a33b0 0x7fe1b80a5870 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4006010 tx=0x7fe1c4009800 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.568+0000 7fe1d1088640 1 --2- 172.21.15.98:0/709711152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fc1090 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80a3bc0 tx=0x7fe1c800bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1088aa36b6d3514 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.568+0000 7fe1b77fe640 1 -- 172.21.15.98:0/709711152 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c801e070 con 0x556d79fc2b40 2024-09-19T08:13:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.568+0000 7fe1b77fe640 1 -- 172.21.15.98:0/709711152 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1c809bb80 con 0x556d79fc2b40 2024-09-19T08:13:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.568+0000 7fe1b77fe640 1 -- 172.21.15.98:0/709711152 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8097af0 con 0x556d79fc2b40 2024-09-19T08:13:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.569+0000 7fe1d56c5740 1 -- 172.21.15.98:0/709711152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 msgr2=0x556d79fc1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.569+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/709711152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fc1090 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80a3bc0 tx=0x7fe1c800bf90 comp rx=0 tx=0).stop 2024-09-19T08:13:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.569+0000 7fe1d56c5740 1 -- 172.21.15.98:0/709711152 shutdown_connections 2024-09-19T08:13:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.569+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/709711152 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 0x556d79fce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.569+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/709711152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc15d0 0x556d79fc1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.569+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/709711152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fc1090 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.569+0000 7fe1d56c5740 1 -- 172.21.15.98:0/709711152 >> 172.21.15.98:0/709711152 conn(0x556d79f8a310 msgr2=0x556d79fbdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.569+0000 7fe1d56c5740 1 -- 172.21.15.98:0/709711152 shutdown_connections 2024-09-19T08:13:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.569+0000 7fe1d56c5740 1 -- 172.21.15.98:0/709711152 wait complete. 2024-09-19T08:13:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.569+0000 7fe1d56c5740 1 -- start start 2024-09-19T08:13:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.569+0000 7fe1d56c5740 1 Processor -- start 2024-09-19T08:13:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d56c5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fc15d0 0x556d79fcbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d56c5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fc9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d56c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fcc370 0x556d79fc97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d56c5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556d79fcf070 con 0x556d79fc2b40 2024-09-19T08:13:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d56c5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556d79ee2060 con 0x556d79fc15d0 2024-09-19T08:13:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d56c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556d79fcf1f0 con 0x556d79fcc370 2024-09-19T08:13:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d56c5740 10 client.0 _ll_register_callbacks cb 0x556d79c358c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d0887640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fc9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d0887640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fc9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37674/0 (socket says 172.21.15.98:37674) 2024-09-19T08:13:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d0887640 1 -- 172.21.15.98:0/822767453 learned_addr learned my addr 172.21.15.98:0/822767453 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d1088640 1 --2- 172.21.15.98:0/822767453 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fc15d0 0x556d79fcbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d1889640 1 --2- 172.21.15.98:0/822767453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fcc370 0x556d79fc97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d0887640 1 -- 172.21.15.98:0/822767453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fcc370 msgr2=0x556d79fc97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d0887640 1 --2- 172.21.15.98:0/822767453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fcc370 0x556d79fc97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d0887640 1 -- 172.21.15.98:0/822767453 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fc15d0 msgr2=0x556d79fcbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d0887640 1 --2- 172.21.15.98:0/822767453 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fc15d0 0x556d79fcbda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d0887640 1 -- 172.21.15.98:0/822767453 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d79c983c0 con 0x556d79fc2b40 2024-09-19T08:13:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.570+0000 7fe1d1088640 1 --2- 172.21.15.98:0/822767453 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fc15d0 0x556d79fcbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1d0887640 1 --2- 172.21.15.98:0/822767453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fc9050 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc00f4c0 tx=0x7fe1bc00f980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d098aaf46e6e38 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1b4ff9640 1 -- 172.21.15.98:0/822767453 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc013850 con 0x556d79fc2b40 2024-09-19T08:13:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1b4ff9640 1 -- 172.21.15.98:0/822767453 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7fe1bc0139f0 con 0x556d79fc2b40 2024-09-19T08:13:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1d56c5740 1 -- 172.21.15.98:0/822767453 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1c809bb80 con 0x556d79fc2b40 2024-09-19T08:13:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1b4ff9640 1 -- 172.21.15.98:0/822767453 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe1bc013bf0 con 0x556d79fc2b40 2024-09-19T08:13:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1d56c5740 10 client.17910 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1d56c5740 1 -- 172.21.15.98:0/822767453 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556d79fdabb0 con 0x556d79fc2b40 2024-09-19T08:13:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1b4ff9640 1 -- 172.21.15.98:0/822767453 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1c809bb80 con 0x556d79fc2b40 2024-09-19T08:13:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1b4ff9640 1 -- 172.21.15.98:0/822767453 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe1bc0113e0 con 0x556d79fc2b40 2024-09-19T08:13:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1d56c5740 10 client.17910 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1d56c5740 10 client.17910 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:13:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1d56c5740 20 client.17910 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/822767453 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556d79fdc070 0x556d79fde4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1d56c5740 4 client.17910 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:13:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.571+0000 7fe1d56c5740 1 -- 172.21.15.98:0/822767453 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d79f34860 con 0x556d79fdc070 2024-09-19T08:13:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.572+0000 7fe1d1088640 1 --2- 172.21.15.98:0/822767453 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556d79fdc070 0x556d79fde4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.572+0000 7fe1d1088640 1 --2- 172.21.15.98:0/822767453 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556d79fdc070 0x556d79fde4e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=75e390e806f05c3b server_cookie=f02268128537554d in_seq=0 out_seq=0 2024-09-19T08:13:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.572+0000 7fe1b4ff9640 10 client.17910 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:13:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1b4ff9640 1 -- 172.21.15.98:0/822767453 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556d79f34860 con 0x556d79fdc070 2024-09-19T08:13:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1b4ff9640 10 client.17910 handle_command_reply: tid=0 2024-09-19T08:13:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1d56c5740 1 client.17910 shutdown 2024-09-19T08:13:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1d56c5740 1 -- 172.21.15.98:0/822767453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 msgr2=0x556d79fc9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/822767453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fc9050 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc00f4c0 tx=0x7fe1bc00f980 comp rx=0 tx=0).stop 2024-09-19T08:13:48.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1d56c5740 1 -- 172.21.15.98:0/822767453 shutdown_connections 2024-09-19T08:13:48.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/822767453 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556d79fdc070 0x556d79fde4e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/822767453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fcc370 0x556d79fc97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/822767453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fc9050 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/822767453 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fc15d0 0x556d79fcbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1d56c5740 1 -- 172.21.15.98:0/822767453 >> 172.21.15.98:0/822767453 conn(0x556d79f8a310 msgr2=0x556d79fc3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:48.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1b4ff9640 0 client.17910 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:13:48.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1d56c5740 1 -- 172.21.15.98:0/822767453 shutdown_connections 2024-09-19T08:13:48.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1d56c5740 1 -- 172.21.15.98:0/822767453 wait complete. 2024-09-19T08:13:48.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.573+0000 7fe1d56c5740 20 client.17910 trim_cache size 0 max 16384 2024-09-19T08:13:48.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.586+0000 7fe1d56c5740 1 Processor -- start 2024-09-19T08:13:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.586+0000 7fe1d56c5740 1 -- start start 2024-09-19T08:13:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.586+0000 7fe1d56c5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc15d0 0x556d79fbd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.586+0000 7fe1d56c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc2b40 0x556d79fbdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.586+0000 7fe1d56c5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 0x556d79fe3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.586+0000 7fe1d56c5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc1ab850 con 0x556d79fc15d0 2024-09-19T08:13:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.586+0000 7fe1d56c5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc1ad390 con 0x556d79fcc370 2024-09-19T08:13:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.586+0000 7fe1d56c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc1abf20 con 0x556d79fc2b40 2024-09-19T08:13:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.586+0000 7fe1d1088640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc15d0 0x556d79fbd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.586+0000 7fe1d1088640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc15d0 0x556d79fbd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37686/0 (socket says 172.21.15.98:37686) 2024-09-19T08:13:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.586+0000 7fe1d1088640 1 -- 172.21.15.98:0/3945308324 learned_addr learned my addr 172.21.15.98:0/3945308324 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.587+0000 7fe1d1088640 1 -- 172.21.15.98:0/3945308324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc2b40 msgr2=0x556d79fbdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.587+0000 7fe1d1088640 1 --2- 172.21.15.98:0/3945308324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc2b40 0x556d79fbdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.587+0000 7fe1d1088640 1 -- 172.21.15.98:0/3945308324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 msgr2=0x556d79fe3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.587+0000 7fe1d1889640 1 --2- 172.21.15.98:0/3945308324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 0x556d79fe3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.587+0000 7fe1d1088640 1 --2- 172.21.15.98:0/3945308324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 0x556d79fe3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.587+0000 7fe1d1088640 1 -- 172.21.15.98:0/3945308324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d79fc9b50 con 0x556d79fc15d0 2024-09-19T08:13:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.587+0000 7fe1d1088640 1 --2- 172.21.15.98:0/3945308324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc15d0 0x556d79fbd700 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fe1c809b640 tx=0x7fe1c8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4880000f750943f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.587+0000 7fe1b6ffd640 1 -- 172.21.15.98:0/3945308324 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8079070 con 0x556d79fc15d0 2024-09-19T08:13:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.587+0000 7fe1b6ffd640 1 -- 172.21.15.98:0/3945308324 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1c8010ca0 con 0x556d79fc15d0 2024-09-19T08:13:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.587+0000 7fe1b6ffd640 1 -- 172.21.15.98:0/3945308324 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c801e040 con 0x556d79fc15d0 2024-09-19T08:13:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 -- 172.21.15.98:0/3945308324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc15d0 msgr2=0x556d79fbd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/3945308324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc15d0 0x556d79fbd700 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fe1c809b640 tx=0x7fe1c8000bc0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 -- 172.21.15.98:0/3945308324 shutdown_connections 2024-09-19T08:13:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/3945308324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 0x556d79fe3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/3945308324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc2b40 0x556d79fbdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/3945308324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc15d0 0x556d79fbd700 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 -- 172.21.15.98:0/3945308324 >> 172.21.15.98:0/3945308324 conn(0x556d79f8a310 msgr2=0x556d79fc4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 -- 172.21.15.98:0/3945308324 shutdown_connections 2024-09-19T08:13:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 -- 172.21.15.98:0/3945308324 wait complete. 2024-09-19T08:13:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 -- start start 2024-09-19T08:13:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 Processor -- start 2024-09-19T08:13:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc15d0 0x556d79fd4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fcffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 0x556d79fd0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc1accc0 con 0x556d79fc2b40 2024-09-19T08:13:48.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.588+0000 7fe1d56c5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc1ac5f0 con 0x556d79fcc370 2024-09-19T08:13:48.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d56c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc1a9850 con 0x556d79fc15d0 2024-09-19T08:13:48.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d56c5740 10 client.0 _ll_register_callbacks cb 0x7fe1bc00a510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:48.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d0887640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fcffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d0887640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fcffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37696/0 (socket says 172.21.15.98:37696) 2024-09-19T08:13:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d0887640 1 -- 172.21.15.98:0/2106008678 learned_addr learned my addr 172.21.15.98:0/2106008678 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d0887640 1 -- 172.21.15.98:0/2106008678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc15d0 msgr2=0x556d79fd4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d1889640 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 0x556d79fd0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d1088640 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc15d0 0x556d79fd4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d0887640 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc15d0 0x556d79fd4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d0887640 1 -- 172.21.15.98:0/2106008678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 msgr2=0x556d79fd0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:48.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d0887640 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 0x556d79fd0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d0887640 1 -- 172.21.15.98:0/2106008678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d79fd0d90 con 0x556d79fc2b40 2024-09-19T08:13:48.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d1088640 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc15d0 0x556d79fd4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:48.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d1889640 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 0x556d79fd0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:48.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d0887640 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fcffc0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc001eb0 tx=0x7fe1bc00ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc5e37bfe9deeb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:48.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:48.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1b4ff9640 1 -- 172.21.15.98:0/2106008678 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc028070 con 0x556d79fc2b40 2024-09-19T08:13:48.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1b4ff9640 1 -- 172.21.15.98:0/2106008678 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7fe1bc013850 con 0x556d79fc2b40 2024-09-19T08:13:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d56c5740 1 -- 172.21.15.98:0/2106008678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1c8010ca0 con 0x556d79fc2b40 2024-09-19T08:13:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1b4ff9640 1 -- 172.21.15.98:0/2106008678 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe1bc013a50 con 0x556d79fc2b40 2024-09-19T08:13:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.589+0000 7fe1d56c5740 10 client.17916 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.590+0000 7fe1d56c5740 1 -- 172.21.15.98:0/2106008678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556d7a046e90 con 0x556d79fc2b40 2024-09-19T08:13:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.590+0000 7fe1b4ff9640 1 -- 172.21.15.98:0/2106008678 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1c8010ca0 con 0x556d79fc2b40 2024-09-19T08:13:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.590+0000 7fe1b4ff9640 1 -- 172.21.15.98:0/2106008678 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe1bc0064e0 con 0x556d79fc2b40 2024-09-19T08:13:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.590+0000 7fe1d56c5740 10 client.17916 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.590+0000 7fe1d56c5740 10 client.17916 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-19T08:13:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.590+0000 7fe1d56c5740 20 client.17916 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.590+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556d7a047610 0x556d7a049a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.590+0000 7fe1d56c5740 4 client.17916 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:13:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.590+0000 7fe1d56c5740 1 -- 172.21.15.98:0/2106008678 --> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556d79f2ed70 con 0x556d7a047610 2024-09-19T08:13:48.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.590+0000 7fe1d1088640 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556d7a047610 0x556d7a049a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:48.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1b4ff9640 10 client.17916 ms_handle_connect on v2:172.21.15.162:6834/1149502202 2024-09-19T08:13:48.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1d1088640 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556d7a047610 0x556d7a049a30 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c7d659c373e85a7c server_cookie=d6892e6db66d2259 in_seq=0 out_seq=0 2024-09-19T08:13:48.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1b4ff9640 1 -- 172.21.15.98:0/2106008678 <== mds.3 v2:172.21.15.162:6834/1149502202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556d79f2ed70 con 0x556d7a047610 2024-09-19T08:13:48.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1b4ff9640 10 client.17916 handle_command_reply: tid=0 2024-09-19T08:13:48.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1d56c5740 1 client.17916 shutdown 2024-09-19T08:13:48.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1d56c5740 1 -- 172.21.15.98:0/2106008678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 msgr2=0x556d79fcffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:48.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fcffc0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc001eb0 tx=0x7fe1bc00ae90 comp rx=0 tx=0).stop 2024-09-19T08:13:48.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1d56c5740 1 -- 172.21.15.98:0/2106008678 shutdown_connections 2024-09-19T08:13:48.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d79fcc370 0x556d79fd0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556d79fc2b40 0x556d79fcffc0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.162:6834/1149502202,v1:172.21.15.162:6835/1149502202] conn(0x556d7a047610 0x556d7a049a30 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1b4ff9640 0 client.17916 ms_handle_reset on v2:172.21.15.162:6834/1149502202 2024-09-19T08:13:48.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1d56c5740 1 --2- 172.21.15.98:0/2106008678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d79fc15d0 0x556d79fd4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1d56c5740 1 -- 172.21.15.98:0/2106008678 >> 172.21.15.98:0/2106008678 conn(0x556d79f8a310 msgr2=0x556d79fca8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:48.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1d56c5740 1 -- 172.21.15.98:0/2106008678 shutdown_connections 2024-09-19T08:13:48.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1d56c5740 1 -- 172.21.15.98:0/2106008678 wait complete. 2024-09-19T08:13:48.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.591+0000 7fe1d56c5740 20 client.17916 trim_cache size 0 max 16384 2024-09-19T08:13:48.610 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:13:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.592+0000 7fe1d32ec640 1 -- 172.21.15.98:0/1845480041 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1b80a33b0 msgr2=0x7fe1b80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.592+0000 7fe1d32ec640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1b80a33b0 0x7fe1b80a5870 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4006010 tx=0x7fe1c4009800 comp rx=0 tx=0).stop 2024-09-19T08:13:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.592+0000 7fe1d32ec640 1 -- 172.21.15.98:0/1845480041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1cc100a80 msgr2=0x7fe1cc198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.592+0000 7fe1d32ec640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1cc100a80 0x7fe1cc198570 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fe1c800b4a0 tx=0x7fe1c800b960 comp rx=0 tx=0).stop 2024-09-19T08:13:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.592+0000 7fe1d32ec640 1 -- 172.21.15.98:0/1845480041 shutdown_connections 2024-09-19T08:13:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.592+0000 7fe1d32ec640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1b80a33b0 0x7fe1b80a5870 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.592+0000 7fe1d32ec640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1cc106a80 0x7fe1cc19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.592+0000 7fe1d32ec640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1cc101420 0x7fe1cc198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.592+0000 7fe1d32ec640 1 --2- 172.21.15.98:0/1845480041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1cc100a80 0x7fe1cc198570 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.592+0000 7fe1d32ec640 1 -- 172.21.15.98:0/1845480041 >> 172.21.15.98:0/1845480041 conn(0x7fe1cc0fc7a0 msgr2=0x7fe1cc06aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.592+0000 7fe1d32ec640 1 -- 172.21.15.98:0/1845480041 shutdown_connections 2024-09-19T08:13:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:48.592+0000 7fe1d32ec640 1 -- 172.21.15.98:0/1845480041 wait complete. 2024-09-19T08:13:49.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:48 smithi139 ceph-mon[34874]: pgmap v334: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:13:49.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:48 smithi098 ceph-mon[31333]: pgmap v334: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:13:49.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-19T08:13:49.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:48 smithi162 ceph-mon[35260]: pgmap v334: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:13:49.600 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:50.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.106+0000 7f2ce98a5640 1 -- 172.21.15.98:0/1347755798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ce4102ef0 msgr2=0x7f2ce410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:50.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.106+0000 7f2ce98a5640 1 --2- 172.21.15.98:0/1347755798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ce4102ef0 0x7f2ce410da40 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009900 tx=0x7f2cdc02a480 comp rx=0 tx=0).stop 2024-09-19T08:13:50.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.106+0000 7f2ce98a5640 1 -- 172.21.15.98:0/1347755798 shutdown_connections 2024-09-19T08:13:50.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.106+0000 7f2ce98a5640 1 --2- 172.21.15.98:0/1347755798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ce4102ef0 0x7f2ce410da40 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.106+0000 7f2ce98a5640 1 --2- 172.21.15.98:0/1347755798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ce4102570 0x7f2ce41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.106+0000 7f2ce98a5640 1 --2- 172.21.15.98:0/1347755798 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ce4100e20 0x7f2ce4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.106+0000 7f2ce98a5640 1 -- 172.21.15.98:0/1347755798 >> 172.21.15.98:0/1347755798 conn(0x7f2ce40fc7d0 msgr2=0x7f2ce40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:50.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.106+0000 7f2ce98a5640 1 -- 172.21.15.98:0/1347755798 shutdown_connections 2024-09-19T08:13:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.106+0000 7f2ce98a5640 1 -- 172.21.15.98:0/1347755798 wait complete. 2024-09-19T08:13:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.106+0000 7f2ce98a5640 1 Processor -- start 2024-09-19T08:13:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce98a5640 1 -- start start 2024-09-19T08:13:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce98a5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ce4100e20 0x7f2ce40fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce98a5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ce4102570 0x7f2ce40fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce98a5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ce4102ef0 0x7f2ce40fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce98a5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce4110f80 con 0x7f2ce4102570 2024-09-19T08:13:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce98a5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce4110e00 con 0x7f2ce4100e20 2024-09-19T08:13:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce98a5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce4111100 con 0x7f2ce4102ef0 2024-09-19T08:13:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ce4102570 0x7f2ce40fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ce4102570 0x7f2ce40fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37708/0 (socket says 172.21.15.98:37708) 2024-09-19T08:13:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce27fc640 1 -- 172.21.15.98:0/1452462538 learned_addr learned my addr 172.21.15.98:0/1452462538 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce37fe640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ce4102ef0 0x7f2ce40fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce2ffd640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ce4100e20 0x7f2ce40fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce27fc640 1 -- 172.21.15.98:0/1452462538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ce4102ef0 msgr2=0x7f2ce40fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce27fc640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ce4102ef0 0x7f2ce40fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce27fc640 1 -- 172.21.15.98:0/1452462538 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ce4100e20 msgr2=0x7f2ce40fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce27fc640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ce4100e20 0x7f2ce40fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce27fc640 1 -- 172.21.15.98:0/1452462538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce41a6990 con 0x7f2ce4102570 2024-09-19T08:13:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce2ffd640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ce4100e20 0x7f2ce40fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.107+0000 7f2ce27fc640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ce4102570 0x7f2ce40fd1c0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00d950 tx=0x7f2ccc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.108+0000 7f2ce88a3640 1 -- 172.21.15.98:0/1452462538 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc017070 con 0x7f2ce4102570 2024-09-19T08:13:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.108+0000 7f2ce98a5640 1 -- 172.21.15.98:0/1452462538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ce41a6c80 con 0x7f2ce4102570 2024-09-19T08:13:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.108+0000 7f2ce88a3640 1 -- 172.21.15.98:0/1452462538 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ccc009c30 con 0x7f2ce4102570 2024-09-19T08:13:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.108+0000 7f2ce88a3640 1 -- 172.21.15.98:0/1452462538 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc00f6b0 con 0x7f2ce4102570 2024-09-19T08:13:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.108+0000 7f2ce98a5640 1 -- 172.21.15.98:0/1452462538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ce41a71c0 con 0x7f2ce4102570 2024-09-19T08:13:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.110+0000 7f2cebc7e740 1 Processor -- start 2024-09-19T08:13:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.110+0000 7f2ce88a3640 1 -- 172.21.15.98:0/1452462538 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f2ccc0109d0 con 0x7f2ce4102570 2024-09-19T08:13:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.110+0000 7f2cebc7e740 1 -- start start 2024-09-19T08:13:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.110+0000 7f2cebc7e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.110+0000 7f2cebc7e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 0x5593609c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.110+0000 7f2cebc7e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609cc370 0x5593609ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.110+0000 7f2cebc7e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5593609272f0 con 0x5593609c2b40 2024-09-19T08:13:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.110+0000 7f2cebc7e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5593608cecf0 con 0x5593609cc370 2024-09-19T08:13:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.110+0000 7f2cebc7e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5593609232b0 con 0x5593609c15d0 2024-09-19T08:13:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.110+0000 7f2ce37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2ce37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37720/0 (socket says 172.21.15.98:37720) 2024-09-19T08:13:50.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2ce37fe640 1 -- 172.21.15.98:0/1209062234 learned_addr learned my addr 172.21.15.98:0/1209062234 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:50.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2ce2ffd640 1 --2- 172.21.15.98:0/1209062234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 0x5593609c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2ce37fe640 1 -- 172.21.15.98:0/1209062234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 msgr2=0x5593609c1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:50.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2ce37fe640 1 --2- 172.21.15.98:0/1209062234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 0x5593609c1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2ce27fc640 1 --2- 172.21.15.98:0/1209062234 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609cc370 0x5593609ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2ce37fe640 1 -- 172.21.15.98:0/1209062234 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609cc370 msgr2=0x5593609ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:50.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2ce37fe640 1 --2- 172.21.15.98:0/1209062234 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609cc370 0x5593609ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2ce37fe640 1 -- 172.21.15.98:0/1209062234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593608b33a0 con 0x5593609c2b40 2024-09-19T08:13:50.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2ce27fc640 1 --2- 172.21.15.98:0/1209062234 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609cc370 0x5593609ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:50.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2ce37fe640 1 --2- 172.21.15.98:0/1209062234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c1090 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc033000 tx=0x7f2cdc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4905de171c83dad server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:50.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2cd9ffb640 1 -- 172.21.15.98:0/1209062234 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc035030 con 0x5593609c2b40 2024-09-19T08:13:50.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2cd9ffb640 1 -- 172.21.15.98:0/1209062234 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cdc040070 con 0x5593609c2b40 2024-09-19T08:13:50.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2cd9ffb640 1 -- 172.21.15.98:0/1209062234 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc031a40 con 0x5593609c2b40 2024-09-19T08:13:50.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2ce88a3640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2cd00a33b0 0x7f2cd00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.111+0000 7f2ce2ffd640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2cd00a33b0 0x7f2cd00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.112+0000 7f2ce88a3640 1 -- 172.21.15.98:0/1452462538 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f2ccc0cf8b0 con 0x7f2ce4102570 2024-09-19T08:13:50.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.112+0000 7f2cebc7e740 1 -- 172.21.15.98:0/1209062234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 msgr2=0x5593609c1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:50.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.112+0000 7f2ce2ffd640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2cd00a33b0 0x7f2cd00a5870 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x5593609beac0 tx=0x7f2cd4009420 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:50.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.112+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/1209062234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c1090 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc033000 tx=0x7f2cdc004040 comp rx=0 tx=0).stop 2024-09-19T08:13:50.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.112+0000 7f2cebc7e740 1 -- 172.21.15.98:0/1209062234 shutdown_connections 2024-09-19T08:13:50.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.112+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/1209062234 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609cc370 0x5593609ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.112+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/1209062234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 0x5593609c1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.112+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/1209062234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c1090 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.112+0000 7f2cebc7e740 1 -- 172.21.15.98:0/1209062234 >> 172.21.15.98:0/1209062234 conn(0x55936098a310 msgr2=0x5593609bdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.112+0000 7f2cebc7e740 1 -- 172.21.15.98:0/1209062234 shutdown_connections 2024-09-19T08:13:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.112+0000 7f2cebc7e740 1 -- 172.21.15.98:0/1209062234 wait complete. 2024-09-19T08:13:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.112+0000 7f2cebc7e740 1 -- start start 2024-09-19T08:13:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2cebc7e740 1 Processor -- start 2024-09-19T08:13:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2cebc7e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 0x5593609cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2cebc7e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609c2b40 0x5593609c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2cebc7e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609cc370 0x5593609c9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2cebc7e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5593609cf010 con 0x5593609cc370 2024-09-19T08:13:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2cebc7e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5593608e2060 con 0x5593609c2b40 2024-09-19T08:13:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2cebc7e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5593609cf190 con 0x5593609c15d0 2024-09-19T08:13:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2cebc7e740 10 client.0 _ll_register_callbacks cb 0x5593606358c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2ce37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609cc370 0x5593609c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2ce37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609cc370 0x5593609c9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37726/0 (socket says 172.21.15.98:37726) 2024-09-19T08:13:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2ce37fe640 1 -- 172.21.15.98:0/3725451326 learned_addr learned my addr 172.21.15.98:0/3725451326 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2ce2ffd640 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 0x5593609cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2ce27fc640 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609c2b40 0x5593609c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2ce37fe640 1 -- 172.21.15.98:0/3725451326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 msgr2=0x5593609cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2ce37fe640 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 0x5593609cbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2ce37fe640 1 -- 172.21.15.98:0/3725451326 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609c2b40 msgr2=0x5593609c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2ce37fe640 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609c2b40 0x5593609c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2ce37fe640 1 -- 172.21.15.98:0/3725451326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593606983c0 con 0x5593609cc370 2024-09-19T08:13:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2ce27fc640 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609c2b40 0x5593609c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2ce2ffd640 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 0x5593609cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2ce37fe640 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609cc370 0x5593609c9560 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009850 tx=0x7f2cdc004520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32282ee08f9b2524 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.113+0000 7f2cc2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2cc2ffd640 1 -- 172.21.15.98:0/3725451326 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc035070 con 0x5593609cc370 2024-09-19T08:13:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2cebc7e740 1 -- 172.21.15.98:0/3725451326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cdc040070 con 0x5593609cc370 2024-09-19T08:13:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2cc2ffd640 1 -- 172.21.15.98:0/3725451326 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f2cdc003ac0 con 0x5593609cc370 2024-09-19T08:13:50.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2cc2ffd640 1 -- 172.21.15.98:0/3725451326 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2cdc057050 con 0x5593609cc370 2024-09-19T08:13:50.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2cebc7e740 10 client.17928 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:50.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2cebc7e740 1 -- 172.21.15.98:0/3725451326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5593609daa00 con 0x5593609cc370 2024-09-19T08:13:50.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2cc2ffd640 1 -- 172.21.15.98:0/3725451326 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cdc040070 con 0x5593609cc370 2024-09-19T08:13:50.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2cc2ffd640 1 -- 172.21.15.98:0/3725451326 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2cdc0046c0 con 0x5593609cc370 2024-09-19T08:13:50.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2cebc7e740 10 client.17928 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:50.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2cebc7e740 10 client.17928 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:13:50.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2cebc7e740 20 client.17928 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:50.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5593609dbd50 0x5593609de140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2cebc7e740 4 client.17928 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:13:50.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2cebc7e740 1 -- 172.21.15.98:0/3725451326 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559360934860 con 0x5593609dbd50 2024-09-19T08:13:50.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.114+0000 7f2ce2ffd640 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5593609dbd50 0x5593609de140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.115+0000 7f2ce2ffd640 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5593609dbd50 0x5593609de140 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=40c101437b70ef62 server_cookie=3ce11843c87c9167 in_seq=0 out_seq=0 2024-09-19T08:13:50.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.115+0000 7f2cc2ffd640 10 client.17928 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:13:50.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cc2ffd640 1 -- 172.21.15.98:0/3725451326 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559360934860 con 0x5593609dbd50 2024-09-19T08:13:50.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cc2ffd640 10 client.17928 handle_command_reply: tid=0 2024-09-19T08:13:50.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cebc7e740 1 client.17928 shutdown 2024-09-19T08:13:50.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cebc7e740 1 -- 172.21.15.98:0/3725451326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609cc370 msgr2=0x5593609c9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:50.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609cc370 0x5593609c9560 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009850 tx=0x7f2cdc004520 comp rx=0 tx=0).stop 2024-09-19T08:13:50.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cebc7e740 1 -- 172.21.15.98:0/3725451326 shutdown_connections 2024-09-19T08:13:50.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x5593609dbd50 0x5593609de140 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609cc370 0x5593609c9560 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609c2b40 0x5593609c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/3725451326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 0x5593609cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cc2ffd640 0 client.17928 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:13:50.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cebc7e740 1 -- 172.21.15.98:0/3725451326 >> 172.21.15.98:0/3725451326 conn(0x55936098a310 msgr2=0x5593609c3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:50.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cebc7e740 1 -- 172.21.15.98:0/3725451326 shutdown_connections 2024-09-19T08:13:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cebc7e740 1 -- 172.21.15.98:0/3725451326 wait complete. 2024-09-19T08:13:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.116+0000 7f2cebc7e740 20 client.17928 trim_cache size 0 max 16384 2024-09-19T08:13:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2cebc7e740 1 Processor -- start 2024-09-19T08:13:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2cebc7e740 1 -- start start 2024-09-19T08:13:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2cebc7e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 0x5593609d82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2cebc7e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2cebc7e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609cc370 0x5593609c4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2cebc7e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce41afa50 con 0x5593609c2b40 2024-09-19T08:13:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2cebc7e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce41b1590 con 0x5593609cc370 2024-09-19T08:13:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2cebc7e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce41b0120 con 0x5593609c15d0 2024-09-19T08:13:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2ce27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2ce27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37738/0 (socket says 172.21.15.98:37738) 2024-09-19T08:13:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2ce27fc640 1 -- 172.21.15.98:0/3181974795 learned_addr learned my addr 172.21.15.98:0/3181974795 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2ce2ffd640 1 --2- 172.21.15.98:0/3181974795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 0x5593609d82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2ce27fc640 1 -- 172.21.15.98:0/3181974795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 msgr2=0x5593609d82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2ce27fc640 1 --2- 172.21.15.98:0/3181974795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 0x5593609d82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2ce27fc640 1 -- 172.21.15.98:0/3181974795 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609cc370 msgr2=0x5593609c4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2ce27fc640 1 --2- 172.21.15.98:0/3181974795 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609cc370 0x5593609c4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2ce27fc640 1 -- 172.21.15.98:0/3181974795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593609c7510 con 0x5593609c2b40 2024-09-19T08:13:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2ce2ffd640 1 --2- 172.21.15.98:0/3181974795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 0x5593609d82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2ce27fc640 1 --2- 172.21.15.98:0/3181974795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c4120 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc009610 tx=0x7f2ccc09b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75be9c491314491f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2cd97fa640 1 -- 172.21.15.98:0/3181974795 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc020040 con 0x5593609c2b40 2024-09-19T08:13:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2cd97fa640 1 -- 172.21.15.98:0/3181974795 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ccc09bb90 con 0x5593609c2b40 2024-09-19T08:13:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.129+0000 7f2cd97fa640 1 -- 172.21.15.98:0/3181974795 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc0860c0 con 0x5593609c2b40 2024-09-19T08:13:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.130+0000 7f2cebc7e740 1 -- 172.21.15.98:0/3181974795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 msgr2=0x5593609c4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.130+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/3181974795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c4120 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc009610 tx=0x7f2ccc09b5b0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.130+0000 7f2cebc7e740 1 -- 172.21.15.98:0/3181974795 shutdown_connections 2024-09-19T08:13:50.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.130+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/3181974795 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609cc370 0x5593609c4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.130+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/3181974795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c4120 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.130+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/3181974795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609c15d0 0x5593609d82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.130+0000 7f2cebc7e740 1 -- 172.21.15.98:0/3181974795 >> 172.21.15.98:0/3181974795 conn(0x55936098a310 msgr2=0x5593609cb040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:50.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.130+0000 7f2cebc7e740 1 -- 172.21.15.98:0/3181974795 shutdown_connections 2024-09-19T08:13:50.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.130+0000 7f2cebc7e740 1 -- 172.21.15.98:0/3181974795 wait complete. 2024-09-19T08:13:50.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.130+0000 7f2cebc7e740 1 -- start start 2024-09-19T08:13:50.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2cebc7e740 1 Processor -- start 2024-09-19T08:13:50.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2cebc7e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609c15d0 0x5593609d4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2cebc7e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2cebc7e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609cc370 0x5593609c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2cebc7e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce41b0ec0 con 0x5593609c2b40 2024-09-19T08:13:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2cebc7e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce41b07f0 con 0x5593609c15d0 2024-09-19T08:13:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2cebc7e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce41ada50 con 0x5593609cc370 2024-09-19T08:13:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2cebc7e740 10 client.0 _ll_register_callbacks cb 0x7f2cdc005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2ce27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2ce27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37754/0 (socket says 172.21.15.98:37754) 2024-09-19T08:13:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2ce27fc640 1 -- 172.21.15.98:0/365133468 learned_addr learned my addr 172.21.15.98:0/365133468 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2ce2ffd640 1 --2- 172.21.15.98:0/365133468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609c15d0 0x5593609d4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2ce27fc640 1 -- 172.21.15.98:0/365133468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609cc370 msgr2=0x5593609c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:50.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2ce37fe640 1 --2- 172.21.15.98:0/365133468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609cc370 0x5593609c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2ce27fc640 1 --2- 172.21.15.98:0/365133468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609cc370 0x5593609c61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2ce27fc640 1 -- 172.21.15.98:0/365133468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609c15d0 msgr2=0x5593609d4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:50.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2ce27fc640 1 --2- 172.21.15.98:0/365133468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609c15d0 0x5593609d4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.131+0000 7f2ce27fc640 1 -- 172.21.15.98:0/365133468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593609c6a00 con 0x5593609c2b40 2024-09-19T08:13:50.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2ce27fc640 1 --2- 172.21.15.98:0/365133468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c5c90 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0cd0b0 tx=0x7f2ccc0cd4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c315fb9aa41e22c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:50.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cc2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:50.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cc2ffd640 1 -- 172.21.15.98:0/365133468 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc0b9060 con 0x5593609c2b40 2024-09-19T08:13:50.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cc2ffd640 1 -- 172.21.15.98:0/365133468 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f2ccc086050 con 0x5593609c2b40 2024-09-19T08:13:50.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cebc7e740 1 -- 172.21.15.98:0/365133468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ccc09bb90 con 0x5593609c2b40 2024-09-19T08:13:50.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cc2ffd640 1 -- 172.21.15.98:0/365133468 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2ccc020030 con 0x5593609c2b40 2024-09-19T08:13:50.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cebc7e740 10 client.17934 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:50.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cebc7e740 1 -- 172.21.15.98:0/365133468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559360a46650 con 0x5593609c2b40 2024-09-19T08:13:50.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cc2ffd640 1 -- 172.21.15.98:0/365133468 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ccc09bb90 con 0x5593609c2b40 2024-09-19T08:13:50.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cc2ffd640 1 -- 172.21.15.98:0/365133468 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2ccc0978e0 con 0x5593609c2b40 2024-09-19T08:13:50.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cebc7e740 10 client.17934 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:50.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cebc7e740 10 client.17934 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-09-19T08:13:50.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cebc7e740 20 client.17934 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:50.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/365133468 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x559360a46cc0 0x559360a490e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:50.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cebc7e740 4 client.17934 mds_command: new command op to 24437 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:13:50.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.132+0000 7f2cebc7e740 1 -- 172.21.15.98:0/365133468 --> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55936092ed70 con 0x559360a46cc0 2024-09-19T08:13:50.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2ce2ffd640 1 --2- 172.21.15.98:0/365133468 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x559360a46cc0 0x559360a490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:50.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2ce2ffd640 1 --2- 172.21.15.98:0/365133468 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x559360a46cc0 0x559360a490e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=49b51bcf7b968aaf server_cookie=fe6f1cafe7d031ff in_seq=0 out_seq=0 2024-09-19T08:13:50.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2cc2ffd640 10 client.17934 ms_handle_connect on v2:172.21.15.139:6834/1275838513 2024-09-19T08:13:50.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2cc2ffd640 1 -- 172.21.15.98:0/365133468 <== mds.2 v2:172.21.15.139:6834/1275838513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55936092ed70 con 0x559360a46cc0 2024-09-19T08:13:50.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2cc2ffd640 10 client.17934 handle_command_reply: tid=0 2024-09-19T08:13:50.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2cebc7e740 1 client.17934 shutdown 2024-09-19T08:13:50.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2cebc7e740 1 -- 172.21.15.98:0/365133468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 msgr2=0x5593609c5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:50.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/365133468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c5c90 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0cd0b0 tx=0x7f2ccc0cd4b0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2cebc7e740 1 -- 172.21.15.98:0/365133468 shutdown_connections 2024-09-19T08:13:50.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/365133468 >> [v2:172.21.15.139:6834/1275838513,v1:172.21.15.139:6835/1275838513] conn(0x559360a46cc0 0x559360a490e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/365133468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593609cc370 0x5593609c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/365133468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593609c2b40 0x5593609c5c90 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2cebc7e740 1 --2- 172.21.15.98:0/365133468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593609c15d0 0x5593609d4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2cc2ffd640 0 client.17934 ms_handle_reset on v2:172.21.15.139:6834/1275838513 2024-09-19T08:13:50.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2cebc7e740 1 -- 172.21.15.98:0/365133468 >> 172.21.15.98:0/365133468 conn(0x55936098a310 msgr2=0x5593609c7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:50.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.133+0000 7f2cebc7e740 1 -- 172.21.15.98:0/365133468 shutdown_connections 2024-09-19T08:13:50.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.134+0000 7f2cebc7e740 1 -- 172.21.15.98:0/365133468 wait complete. 2024-09-19T08:13:50.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.134+0000 7f2cebc7e740 20 client.17934 trim_cache size 0 max 16384 2024-09-19T08:13:50.158 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:13:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.134+0000 7f2ce98a5640 1 -- 172.21.15.98:0/1452462538 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2cd00a33b0 msgr2=0x7f2cd00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.134+0000 7f2ce98a5640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2cd00a33b0 0x7f2cd00a5870 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x5593609beac0 tx=0x7f2cd4009420 comp rx=0 tx=0).stop 2024-09-19T08:13:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.134+0000 7f2ce98a5640 1 -- 172.21.15.98:0/1452462538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ce4102570 msgr2=0x7f2ce40fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.134+0000 7f2ce98a5640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ce4102570 0x7f2ce40fd1c0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00d950 tx=0x7f2ccc00de10 comp rx=0 tx=0).stop 2024-09-19T08:13:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.134+0000 7f2ce98a5640 1 -- 172.21.15.98:0/1452462538 shutdown_connections 2024-09-19T08:13:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.134+0000 7f2ce98a5640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f2cd00a33b0 0x7f2cd00a5870 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.134+0000 7f2ce98a5640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ce4102ef0 0x7f2ce40fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.134+0000 7f2ce98a5640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ce4102570 0x7f2ce40fd1c0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.134+0000 7f2ce98a5640 1 --2- 172.21.15.98:0/1452462538 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ce4100e20 0x7f2ce40fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.134+0000 7f2ce98a5640 1 -- 172.21.15.98:0/1452462538 >> 172.21.15.98:0/1452462538 conn(0x7f2ce40fc7d0 msgr2=0x7f2ce406a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.134+0000 7f2ce98a5640 1 -- 172.21.15.98:0/1452462538 shutdown_connections 2024-09-19T08:13:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:50.134+0000 7f2ce98a5640 1 -- 172.21.15.98:0/1452462538 wait complete. 2024-09-19T08:13:50.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-19T08:13:51.109 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:51.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:50 smithi139 ceph-mon[34874]: pgmap v335: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:13:51.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:50 smithi098 ceph-mon[31333]: pgmap v335: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:13:51.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:50 smithi162 ceph-mon[35260]: pgmap v335: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:13:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.565+0000 7f5905b59640 1 -- 172.21.15.98:0/2242761790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5900106a80 msgr2=0x7f5900106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.565+0000 7f5905b59640 1 --2- 172.21.15.98:0/2242761790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5900106a80 0x7f5900106e60 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009930 tx=0x7f58f002a420 comp rx=0 tx=0).stop 2024-09-19T08:13:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.565+0000 7f5905b59640 1 -- 172.21.15.98:0/2242761790 shutdown_connections 2024-09-19T08:13:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.565+0000 7f5905b59640 1 --2- 172.21.15.98:0/2242761790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5900101420 0x7f590010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.565+0000 7f5905b59640 1 --2- 172.21.15.98:0/2242761790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5900100a80 0x7f5900100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.565+0000 7f5905b59640 1 --2- 172.21.15.98:0/2242761790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5900106a80 0x7f5900106e60 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.565+0000 7f5905b59640 1 -- 172.21.15.98:0/2242761790 >> 172.21.15.98:0/2242761790 conn(0x7f59000fc7a0 msgr2=0x7f59000febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.565+0000 7f5905b59640 1 -- 172.21.15.98:0/2242761790 shutdown_connections 2024-09-19T08:13:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.565+0000 7f5905b59640 1 -- 172.21.15.98:0/2242761790 wait complete. 2024-09-19T08:13:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.566+0000 7f5905b59640 1 Processor -- start 2024-09-19T08:13:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.566+0000 7f5905b59640 1 -- start start 2024-09-19T08:13:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.566+0000 7f5905b59640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5900100a80 0x7f5900102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.566+0000 7f5905b59640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5900101420 0x7f5900102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.566+0000 7f5905b59640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5900106a80 0x7f5900103520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.566+0000 7f5905b59640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5900110e80 con 0x7f5900100a80 2024-09-19T08:13:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.566+0000 7f5905b59640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5900110d00 con 0x7f5900106a80 2024-09-19T08:13:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.566+0000 7f5905b59640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5900111000 con 0x7f5900101420 2024-09-19T08:13:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.566+0000 7f58ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5900100a80 0x7f5900102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.566+0000 7f58ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5900100a80 0x7f5900102aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37774/0 (socket says 172.21.15.98:37774) 2024-09-19T08:13:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.566+0000 7f58ff7fe640 1 -- 172.21.15.98:0/3447955982 learned_addr learned my addr 172.21.15.98:0/3447955982 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.566+0000 7f58feffd640 1 --2- 172.21.15.98:0/3447955982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5900101420 0x7f5900102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.566+0000 7f58fffff640 1 --2- 172.21.15.98:0/3447955982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5900106a80 0x7f5900103520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.567+0000 7f58ff7fe640 1 -- 172.21.15.98:0/3447955982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5900101420 msgr2=0x7f5900102fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.567+0000 7f58ff7fe640 1 --2- 172.21.15.98:0/3447955982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5900101420 0x7f5900102fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.567+0000 7f58ff7fe640 1 -- 172.21.15.98:0/3447955982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5900106a80 msgr2=0x7f5900103520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.567+0000 7f58ff7fe640 1 --2- 172.21.15.98:0/3447955982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5900106a80 0x7f5900103520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.567+0000 7f58ff7fe640 1 -- 172.21.15.98:0/3447955982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59001a6900 con 0x7f5900100a80 2024-09-19T08:13:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.567+0000 7f58ff7fe640 1 --2- 172.21.15.98:0/3447955982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5900100a80 0x7f5900102aa0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009230 tx=0x7f58f0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.567+0000 7f58fcff9640 1 -- 172.21.15.98:0/3447955982 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f003f030 con 0x7f5900100a80 2024-09-19T08:13:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.567+0000 7f5905b59640 1 -- 172.21.15.98:0/3447955982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59001a6b90 con 0x7f5900100a80 2024-09-19T08:13:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.567+0000 7f58fcff9640 1 -- 172.21.15.98:0/3447955982 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f0003ad0 con 0x7f5900100a80 2024-09-19T08:13:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.567+0000 7f58fcff9640 1 -- 172.21.15.98:0/3447955982 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0035070 con 0x7f5900100a80 2024-09-19T08:13:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.567+0000 7f5905b59640 1 -- 172.21.15.98:0/3447955982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59001a7070 con 0x7f5900100a80 2024-09-19T08:13:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.569+0000 7f58fcff9640 1 -- 172.21.15.98:0/3447955982 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f58f0042030 con 0x7f5900100a80 2024-09-19T08:13:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f5907f32740 1 Processor -- start 2024-09-19T08:13:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f5907f32740 1 -- start start 2024-09-19T08:13:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f5907f32740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 0x55683683d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f5907f32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683d5d0 0x55683683da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f5907f32740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x55683684a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f5907f32740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5568367a32f0 con 0x556836848370 2024-09-19T08:13:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f5907f32740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55683674acf0 con 0x55683683eb40 2024-09-19T08:13:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f5907f32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55683679f2b0 con 0x55683683d5d0 2024-09-19T08:13:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f58fcff9640 1 --2- 172.21.15.98:0/3447955982 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58e80a33b0 0x7f58e80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f58feffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x55683684a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f58feffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x55683684a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37784/0 (socket says 172.21.15.98:37784) 2024-09-19T08:13:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f58fffff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 0x55683683d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f58feffd640 1 -- 172.21.15.98:0/329374103 learned_addr learned my addr 172.21.15.98:0/329374103 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f58ff7fe640 1 --2- 172.21.15.98:0/329374103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683d5d0 0x55683683da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f58fffff640 1 --2- 172.21.15.98:0/3447955982 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58e80a33b0 0x7f58e80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f58fcff9640 1 -- 172.21.15.98:0/3447955982 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f58f00f00d0 con 0x7f5900100a80 2024-09-19T08:13:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f58feffd640 1 -- 172.21.15.98:0/329374103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683d5d0 msgr2=0x55683683da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f58feffd640 1 --2- 172.21.15.98:0/329374103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683d5d0 0x55683683da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f58feffd640 1 -- 172.21.15.98:0/329374103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 msgr2=0x55683683d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f58feffd640 1 --2- 172.21.15.98:0/329374103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 0x55683683d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f58feffd640 1 -- 172.21.15.98:0/329374103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55683672f3a0 con 0x556836848370 2024-09-19T08:13:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.570+0000 7f58ff7fe640 1 --2- 172.21.15.98:0/329374103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683d5d0 0x55683683da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.571+0000 7f58feffd640 1 --2- 172.21.15.98:0/329374103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x55683684a760 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f58f400abf0 tx=0x7f58f400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986fb5fe3a6e3ef9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.571+0000 7f58fffff640 1 --2- 172.21.15.98:0/3447955982 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58e80a33b0 0x7f58e80a5870 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f58f8009fd0 tx=0x7f58f8009600 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.571+0000 7f58e5ffb640 1 -- 172.21.15.98:0/329374103 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f401d070 con 0x556836848370 2024-09-19T08:13:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.571+0000 7f58e5ffb640 1 -- 172.21.15.98:0/329374103 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f4018b20 con 0x556836848370 2024-09-19T08:13:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.571+0000 7f58e5ffb640 1 -- 172.21.15.98:0/329374103 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f4010a50 con 0x556836848370 2024-09-19T08:13:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.571+0000 7f5907f32740 1 -- 172.21.15.98:0/329374103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 msgr2=0x55683684a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.571+0000 7f5907f32740 1 --2- 172.21.15.98:0/329374103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x55683684a760 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f58f400abf0 tx=0x7f58f400f790 comp rx=0 tx=0).stop 2024-09-19T08:13:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.572+0000 7f5907f32740 1 -- 172.21.15.98:0/329374103 shutdown_connections 2024-09-19T08:13:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.572+0000 7f5907f32740 1 --2- 172.21.15.98:0/329374103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x55683684a760 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.572+0000 7f5907f32740 1 --2- 172.21.15.98:0/329374103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683d5d0 0x55683683da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.572+0000 7f5907f32740 1 --2- 172.21.15.98:0/329374103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 0x55683683d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.572+0000 7f5907f32740 1 -- 172.21.15.98:0/329374103 >> 172.21.15.98:0/329374103 conn(0x556836806310 msgr2=0x556836839c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.572+0000 7f5907f32740 1 -- 172.21.15.98:0/329374103 shutdown_connections 2024-09-19T08:13:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.572+0000 7f5907f32740 1 -- 172.21.15.98:0/329374103 wait complete. 2024-09-19T08:13:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.572+0000 7f5907f32740 1 -- start start 2024-09-19T08:13:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.572+0000 7f5907f32740 1 Processor -- start 2024-09-19T08:13:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.572+0000 7f5907f32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683d5d0 0x556836847d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f5907f32740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 0x556836844fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f5907f32740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x556836845570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f5907f32740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55683684b070 con 0x556836848370 2024-09-19T08:13:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f5907f32740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55683675e060 con 0x55683683eb40 2024-09-19T08:13:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f5907f32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55683684b1f0 con 0x55683683d5d0 2024-09-19T08:13:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f5907f32740 10 client.0 _ll_register_callbacks cb 0x5568364b18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f58fffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x556836845570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f58fffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x556836845570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37786/0 (socket says 172.21.15.98:37786) 2024-09-19T08:13:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f58fffff640 1 -- 172.21.15.98:0/1998830117 learned_addr learned my addr 172.21.15.98:0/1998830117 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f58feffd640 1 --2- 172.21.15.98:0/1998830117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 0x556836844fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f58ff7fe640 1 --2- 172.21.15.98:0/1998830117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683d5d0 0x556836847d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f58fffff640 1 -- 172.21.15.98:0/1998830117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683d5d0 msgr2=0x556836847d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f58fffff640 1 --2- 172.21.15.98:0/1998830117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683d5d0 0x556836847d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f58fffff640 1 -- 172.21.15.98:0/1998830117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 msgr2=0x556836844fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f58fffff640 1 --2- 172.21.15.98:0/1998830117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 0x556836844fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f58fffff640 1 -- 172.21.15.98:0/1998830117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568365143c0 con 0x556836848370 2024-09-19T08:13:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f58fffff640 1 --2- 172.21.15.98:0/1998830117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x556836845570 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f58f800ab90 tx=0x7f58f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6309fd97bda98220 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f58deffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f58deffd640 1 -- 172.21.15.98:0/1998830117 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f801ebb0 con 0x556836848370 2024-09-19T08:13:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f5907f32740 1 -- 172.21.15.98:0/1998830117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58f4018b20 con 0x556836848370 2024-09-19T08:13:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.573+0000 7f58deffd640 1 -- 172.21.15.98:0/1998830117 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f58f801ed50 con 0x556836848370 2024-09-19T08:13:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.574+0000 7f58deffd640 1 -- 172.21.15.98:0/1998830117 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f58f8004b20 con 0x556836848370 2024-09-19T08:13:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.574+0000 7f5907f32740 10 client.17946 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.574+0000 7f5907f32740 1 -- 172.21.15.98:0/1998830117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556836856b10 con 0x556836848370 2024-09-19T08:13:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.574+0000 7f58deffd640 1 -- 172.21.15.98:0/1998830117 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f4018b20 con 0x556836848370 2024-09-19T08:13:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.574+0000 7f58deffd640 1 -- 172.21.15.98:0/1998830117 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f58f800f040 con 0x556836848370 2024-09-19T08:13:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.574+0000 7f5907f32740 10 client.17946 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.574+0000 7f5907f32740 10 client.17946 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:13:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.574+0000 7f5907f32740 20 client.17946 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.574+0000 7f5907f32740 1 --2- 172.21.15.98:0/1998830117 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x556836857fd0 0x55683685a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.574+0000 7f5907f32740 4 client.17946 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:13:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.574+0000 7f5907f32740 1 -- 172.21.15.98:0/1998830117 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568367b0860 con 0x556836857fd0 2024-09-19T08:13:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.574+0000 7f58ff7fe640 1 --2- 172.21.15.98:0/1998830117 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x556836857fd0 0x55683685a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.575+0000 7f58ff7fe640 1 --2- 172.21.15.98:0/1998830117 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x556836857fd0 0x55683685a470 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c88ca051eeb3abe8 server_cookie=668579426fa3c621 in_seq=0 out_seq=0 2024-09-19T08:13:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.575+0000 7f58deffd640 10 client.17946 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:13:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.575+0000 7f58deffd640 1 -- 172.21.15.98:0/1998830117 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5568367b0860 con 0x556836857fd0 2024-09-19T08:13:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.575+0000 7f58deffd640 10 client.17946 handle_command_reply: tid=0 2024-09-19T08:13:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.576+0000 7f5907f32740 1 client.17946 shutdown 2024-09-19T08:13:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.576+0000 7f5907f32740 1 -- 172.21.15.98:0/1998830117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 msgr2=0x556836845570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.576+0000 7f5907f32740 1 --2- 172.21.15.98:0/1998830117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x556836845570 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f58f800ab90 tx=0x7f58f8004a40 comp rx=0 tx=0).stop 2024-09-19T08:13:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.576+0000 7f5907f32740 1 -- 172.21.15.98:0/1998830117 shutdown_connections 2024-09-19T08:13:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.576+0000 7f5907f32740 1 --2- 172.21.15.98:0/1998830117 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x556836857fd0 0x55683685a470 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.576+0000 7f5907f32740 1 --2- 172.21.15.98:0/1998830117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x556836845570 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.576+0000 7f5907f32740 1 --2- 172.21.15.98:0/1998830117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 0x556836844fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.576+0000 7f5907f32740 1 --2- 172.21.15.98:0/1998830117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683d5d0 0x556836847d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.576+0000 7f58deffd640 0 client.17946 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:13:51.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.576+0000 7f5907f32740 1 -- 172.21.15.98:0/1998830117 >> 172.21.15.98:0/1998830117 conn(0x556836806310 msgr2=0x55683683f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:51.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.576+0000 7f5907f32740 1 -- 172.21.15.98:0/1998830117 shutdown_connections 2024-09-19T08:13:51.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.576+0000 7f5907f32740 1 -- 172.21.15.98:0/1998830117 wait complete. 2024-09-19T08:13:51.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.576+0000 7f5907f32740 20 client.17946 trim_cache size 0 max 16384 2024-09-19T08:13:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f5907f32740 1 Processor -- start 2024-09-19T08:13:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f5907f32740 1 -- start start 2024-09-19T08:13:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f5907f32740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55683683d5d0 0x556836854310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f5907f32740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 0x5568368438a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f5907f32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556836848370 0x556836843de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f5907f32740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59001afa10 con 0x55683683d5d0 2024-09-19T08:13:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f5907f32740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f59001b1550 con 0x55683683eb40 2024-09-19T08:13:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f5907f32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59001b00e0 con 0x556836848370 2024-09-19T08:13:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f58ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55683683d5d0 0x556836854310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f58ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55683683d5d0 0x556836854310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37790/0 (socket says 172.21.15.98:37790) 2024-09-19T08:13:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f58ff7fe640 1 -- 172.21.15.98:0/2271415621 learned_addr learned my addr 172.21.15.98:0/2271415621 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f58fffff640 1 --2- 172.21.15.98:0/2271415621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556836848370 0x556836843de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f58ff7fe640 1 -- 172.21.15.98:0/2271415621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556836848370 msgr2=0x556836843de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f58ff7fe640 1 --2- 172.21.15.98:0/2271415621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556836848370 0x556836843de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f58ff7fe640 1 -- 172.21.15.98:0/2271415621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 msgr2=0x5568368438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f58feffd640 1 --2- 172.21.15.98:0/2271415621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 0x5568368438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f58ff7fe640 1 --2- 172.21.15.98:0/2271415621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 0x5568368438a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f58ff7fe640 1 -- 172.21.15.98:0/2271415621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55683683a960 con 0x55683683d5d0 2024-09-19T08:13:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f58feffd640 1 --2- 172.21.15.98:0/2271415621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 0x5568368438a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.589+0000 7f58ff7fe640 1 --2- 172.21.15.98:0/2271415621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55683683d5d0 0x556836854310 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f58f00b8ed0 tx=0x7f58f0049770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd227c451192efe9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.590+0000 7f58e57fa640 1 -- 172.21.15.98:0/2271415621 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0049040 con 0x55683683d5d0 2024-09-19T08:13:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.590+0000 7f58e57fa640 1 -- 172.21.15.98:0/2271415621 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f00c0800 con 0x55683683d5d0 2024-09-19T08:13:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.590+0000 7f58e57fa640 1 -- 172.21.15.98:0/2271415621 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f002fdf0 con 0x55683683d5d0 2024-09-19T08:13:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.590+0000 7f5907f32740 1 -- 172.21.15.98:0/2271415621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55683683d5d0 msgr2=0x556836854310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.590+0000 7f5907f32740 1 --2- 172.21.15.98:0/2271415621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55683683d5d0 0x556836854310 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f58f00b8ed0 tx=0x7f58f0049770 comp rx=0 tx=0).stop 2024-09-19T08:13:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.590+0000 7f5907f32740 1 -- 172.21.15.98:0/2271415621 shutdown_connections 2024-09-19T08:13:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.590+0000 7f5907f32740 1 --2- 172.21.15.98:0/2271415621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556836848370 0x556836843de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.590+0000 7f5907f32740 1 --2- 172.21.15.98:0/2271415621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683eb40 0x5568368438a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.590+0000 7f5907f32740 1 --2- 172.21.15.98:0/2271415621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55683683d5d0 0x556836854310 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.590+0000 7f5907f32740 1 -- 172.21.15.98:0/2271415621 >> 172.21.15.98:0/2271415621 conn(0x556836806310 msgr2=0x556836844f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.590+0000 7f5907f32740 1 -- 172.21.15.98:0/2271415621 shutdown_connections 2024-09-19T08:13:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.590+0000 7f5907f32740 1 -- 172.21.15.98:0/2271415621 wait complete. 2024-09-19T08:13:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.591+0000 7f5907f32740 1 -- start start 2024-09-19T08:13:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.591+0000 7f5907f32740 1 Processor -- start 2024-09-19T08:13:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.591+0000 7f5907f32740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683d5d0 0x556836850a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.591+0000 7f5907f32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683eb40 0x55683684bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.591+0000 7f5907f32740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x55683684c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.591+0000 7f5907f32740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59001b0e80 con 0x556836848370 2024-09-19T08:13:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.591+0000 7f5907f32740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f59001b07b0 con 0x55683683d5d0 2024-09-19T08:13:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.591+0000 7f5907f32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59001ada10 con 0x55683683eb40 2024-09-19T08:13:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.591+0000 7f5907f32740 10 client.0 _ll_register_callbacks cb 0x7f58f8012760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.591+0000 7f58fffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x55683684c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.591+0000 7f58fffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x55683684c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37806/0 (socket says 172.21.15.98:37806) 2024-09-19T08:13:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.591+0000 7f58fffff640 1 -- 172.21.15.98:0/3411241833 learned_addr learned my addr 172.21.15.98:0/3411241833 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.591+0000 7f58ff7fe640 1 --2- 172.21.15.98:0/3411241833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683d5d0 0x556836850a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f58feffd640 1 --2- 172.21.15.98:0/3411241833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683eb40 0x55683684bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f58fffff640 1 -- 172.21.15.98:0/3411241833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683eb40 msgr2=0x55683684bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f58fffff640 1 --2- 172.21.15.98:0/3411241833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683eb40 0x55683684bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f58fffff640 1 -- 172.21.15.98:0/3411241833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683d5d0 msgr2=0x556836850a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f58fffff640 1 --2- 172.21.15.98:0/3411241833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683d5d0 0x556836850a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f58fffff640 1 -- 172.21.15.98:0/3411241833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55683684c920 con 0x556836848370 2024-09-19T08:13:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f58fffff640 1 --2- 172.21.15.98:0/3411241833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x55683684c2b0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f58f8006010 tx=0x7f58f8009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410ec1875b4d4f5a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f58deffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f58deffd640 1 -- 172.21.15.98:0/3411241833 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f801e880 con 0x556836848370 2024-09-19T08:13:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f58deffd640 1 -- 172.21.15.98:0/3411241833 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f58f801edc0 con 0x556836848370 2024-09-19T08:13:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f5907f32740 1 -- 172.21.15.98:0/3411241833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58f00c0800 con 0x556836848370 2024-09-19T08:13:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f58deffd640 1 -- 172.21.15.98:0/3411241833 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f58f8004cc0 con 0x556836848370 2024-09-19T08:13:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f5907f32740 10 client.17952 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f5907f32740 1 -- 172.21.15.98:0/3411241833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5568368c2b40 con 0x556836848370 2024-09-19T08:13:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.592+0000 7f58deffd640 1 -- 172.21.15.98:0/3411241833 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f00c0800 con 0x556836848370 2024-09-19T08:13:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.593+0000 7f58deffd640 1 -- 172.21.15.98:0/3411241833 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f58f8031760 con 0x556836848370 2024-09-19T08:13:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.593+0000 7f5907f32740 10 client.17952 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.593+0000 7f5907f32740 10 client.17952 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-19T08:13:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.593+0000 7f5907f32740 20 client.17952 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.593+0000 7f5907f32740 1 --2- 172.21.15.98:0/3411241833 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5568368c31f0 0x5568368c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.593+0000 7f5907f32740 4 client.17952 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:13:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.593+0000 7f5907f32740 1 -- 172.21.15.98:0/3411241833 --> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5568367aad70 con 0x5568368c31f0 2024-09-19T08:13:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.593+0000 7f58ff7fe640 1 --2- 172.21.15.98:0/3411241833 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5568368c31f0 0x5568368c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.593+0000 7f58deffd640 10 client.17952 ms_handle_connect on v2:172.21.15.139:6836/1885641254 2024-09-19T08:13:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.593+0000 7f58ff7fe640 1 --2- 172.21.15.98:0/3411241833 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5568368c31f0 0x5568368c55e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ea594cdc7028cfc2 server_cookie=ba4ba2850e09d5ba in_seq=0 out_seq=0 2024-09-19T08:13:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.593+0000 7f58deffd640 1 -- 172.21.15.98:0/3411241833 <== mds.1 v2:172.21.15.139:6836/1885641254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5568367aad70 con 0x5568368c31f0 2024-09-19T08:13:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.593+0000 7f58deffd640 10 client.17952 handle_command_reply: tid=0 2024-09-19T08:13:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.593+0000 7f5907f32740 1 client.17952 shutdown 2024-09-19T08:13:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5907f32740 1 -- 172.21.15.98:0/3411241833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 msgr2=0x55683684c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5907f32740 1 --2- 172.21.15.98:0/3411241833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x55683684c2b0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f58f8006010 tx=0x7f58f8009870 comp rx=0 tx=0).stop 2024-09-19T08:13:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5907f32740 1 -- 172.21.15.98:0/3411241833 shutdown_connections 2024-09-19T08:13:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5907f32740 1 --2- 172.21.15.98:0/3411241833 >> [v2:172.21.15.139:6836/1885641254,v1:172.21.15.139:6837/1885641254] conn(0x5568368c31f0 0x5568368c55e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5907f32740 1 --2- 172.21.15.98:0/3411241833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556836848370 0x55683684c2b0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f58deffd640 0 client.17952 ms_handle_reset on v2:172.21.15.139:6836/1885641254 2024-09-19T08:13:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5907f32740 1 --2- 172.21.15.98:0/3411241833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55683683eb40 0x55683684bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5907f32740 1 --2- 172.21.15.98:0/3411241833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55683683d5d0 0x556836850a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5907f32740 1 -- 172.21.15.98:0/3411241833 >> 172.21.15.98:0/3411241833 conn(0x556836806310 msgr2=0x5568368437e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5907f32740 1 -- 172.21.15.98:0/3411241833 shutdown_connections 2024-09-19T08:13:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5907f32740 1 -- 172.21.15.98:0/3411241833 wait complete. 2024-09-19T08:13:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5907f32740 20 client.17952 trim_cache size 0 max 16384 2024-09-19T08:13:51.602 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:13:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5905b59640 1 -- 172.21.15.98:0/3447955982 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58e80a33b0 msgr2=0x7f58e80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5905b59640 1 --2- 172.21.15.98:0/3447955982 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58e80a33b0 0x7f58e80a5870 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f58f8009fd0 tx=0x7f58f8009600 comp rx=0 tx=0).stop 2024-09-19T08:13:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5905b59640 1 -- 172.21.15.98:0/3447955982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5900100a80 msgr2=0x7f5900102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5905b59640 1 --2- 172.21.15.98:0/3447955982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5900100a80 0x7f5900102aa0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009230 tx=0x7f58f0004270 comp rx=0 tx=0).stop 2024-09-19T08:13:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5905b59640 1 -- 172.21.15.98:0/3447955982 shutdown_connections 2024-09-19T08:13:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5905b59640 1 --2- 172.21.15.98:0/3447955982 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f58e80a33b0 0x7f58e80a5870 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5905b59640 1 --2- 172.21.15.98:0/3447955982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5900106a80 0x7f5900103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5905b59640 1 --2- 172.21.15.98:0/3447955982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5900101420 0x7f5900102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5905b59640 1 --2- 172.21.15.98:0/3447955982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5900100a80 0x7f5900102aa0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5905b59640 1 -- 172.21.15.98:0/3447955982 >> 172.21.15.98:0/3447955982 conn(0x7f59000fc7a0 msgr2=0x7f5900104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5905b59640 1 -- 172.21.15.98:0/3447955982 shutdown_connections 2024-09-19T08:13:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:51.594+0000 7f5905b59640 1 -- 172.21.15.98:0/3447955982 wait complete. 2024-09-19T08:13:52.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-19T08:13:52.612 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:53.011 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:52 smithi098 ceph-mon[31333]: pgmap v336: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:53.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.092+0000 7fe11e3b8640 1 -- 172.21.15.98:0/1079235804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe118101430 msgr2=0x7fe11810d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:53.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.092+0000 7fe11e3b8640 1 --2- 172.21.15.98:0/1079235804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe118101430 0x7fe11810d960 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fe110009cd0 tx=0x7fe11002a8e0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 -- 172.21.15.98:0/1079235804 shutdown_connections 2024-09-19T08:13:53.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 --2- 172.21.15.98:0/1079235804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe118101430 0x7fe11810d960 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 --2- 172.21.15.98:0/1079235804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe118100a90 0x7fe118100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 --2- 172.21.15.98:0/1079235804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe118106a90 0x7fe118106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 -- 172.21.15.98:0/1079235804 >> 172.21.15.98:0/1079235804 conn(0x7fe1180fc7d0 msgr2=0x7fe1180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:53.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 -- 172.21.15.98:0/1079235804 shutdown_connections 2024-09-19T08:13:53.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 -- 172.21.15.98:0/1079235804 wait complete. 2024-09-19T08:13:53.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 Processor -- start 2024-09-19T08:13:53.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 -- start start 2024-09-19T08:13:53.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe118100a90 0x7fe118198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe118101430 0x7fe1181989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe118106a90 0x7fe11819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe118110ed0 con 0x7fe118106a90 2024-09-19T08:13:53.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe118110d50 con 0x7fe118100a90 2024-09-19T08:13:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.093+0000 7fe11e3b8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe118111050 con 0x7fe118101430 2024-09-19T08:13:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe11c955640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe118106a90 0x7fe11819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe11c955640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe118106a90 0x7fe11819cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37812/0 (socket says 172.21.15.98:37812) 2024-09-19T08:13:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe11c955640 1 -- 172.21.15.98:0/1300060470 learned_addr learned my addr 172.21.15.98:0/1300060470 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe10ffff640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe118101430 0x7fe1181989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe117fff640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe118100a90 0x7fe118198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe11c955640 1 -- 172.21.15.98:0/1300060470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe118101430 msgr2=0x7fe1181989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe11c955640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe118101430 0x7fe1181989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe11c955640 1 -- 172.21.15.98:0/1300060470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe118100a90 msgr2=0x7fe118198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe11c955640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe118100a90 0x7fe118198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe11c955640 1 -- 172.21.15.98:0/1300060470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11819d420 con 0x7fe118106a90 2024-09-19T08:13:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe117fff640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe118100a90 0x7fe118198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe11c955640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe118106a90 0x7fe11819cd40 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe110005cc0 tx=0x7fe110003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe115ffb640 1 -- 172.21.15.98:0/1300060470 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe110004040 con 0x7fe118106a90 2024-09-19T08:13:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe11e3b8640 1 -- 172.21.15.98:0/1300060470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11819d6b0 con 0x7fe118106a90 2024-09-19T08:13:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe115ffb640 1 -- 172.21.15.98:0/1300060470 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1100046c0 con 0x7fe118106a90 2024-09-19T08:13:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe115ffb640 1 -- 172.21.15.98:0/1300060470 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe110046480 con 0x7fe118106a90 2024-09-19T08:13:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.094+0000 7fe11e3b8640 1 -- 172.21.15.98:0/1300060470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1181a4ff0 con 0x7fe118106a90 2024-09-19T08:13:53.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.097+0000 7fe115ffb640 1 -- 172.21.15.98:0/1300060470 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fe1100305a0 con 0x7fe118106a90 2024-09-19T08:13:53.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.097+0000 7fe120791740 1 Processor -- start 2024-09-19T08:13:53.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.097+0000 7fe120791740 1 -- start start 2024-09-19T08:13:53.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe120791740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84ce0f50 0x560f84cdf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe120791740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cdf9e0 0x560f84cdfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe120791740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84cea780 0x560f84cecb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe120791740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560f84c412f0 con 0x560f84cea780 2024-09-19T08:13:53.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe120791740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560f84be8cf0 con 0x560f84ce0f50 2024-09-19T08:13:53.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe120791740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560f84c3d2b0 con 0x560f84cdf9e0 2024-09-19T08:13:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe10ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84cea780 0x560f84cecb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe10ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84cea780 0x560f84cecb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37826/0 (socket says 172.21.15.98:37826) 2024-09-19T08:13:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe10ffff640 1 -- 172.21.15.98:0/1676487339 learned_addr learned my addr 172.21.15.98:0/1676487339 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe117fff640 1 --2- 172.21.15.98:0/1676487339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cdf9e0 0x560f84cdfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe115ffb640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1040a33b0 0x7fe1040a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe10ffff640 1 --2- 172.21.15.98:0/1676487339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84ce0f50 0x560f84cdf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe10ffff640 1 -- 172.21.15.98:0/1676487339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cdf9e0 msgr2=0x560f84cdfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe10ffff640 1 --2- 172.21.15.98:0/1676487339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cdf9e0 0x560f84cdfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe10ffff640 1 -- 172.21.15.98:0/1676487339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84ce0f50 msgr2=0x560f84cdf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe10ffff640 1 --2- 172.21.15.98:0/1676487339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84ce0f50 0x560f84cdf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe10ffff640 1 -- 172.21.15.98:0/1676487339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f84bcd3a0 con 0x560f84cea780 2024-09-19T08:13:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe117fff640 1 --2- 172.21.15.98:0/1676487339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cdf9e0 0x560f84cdfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe11c955640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1040a33b0 0x7fe1040a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe115ffb640 1 -- 172.21.15.98:0/1300060470 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7fe11002e030 con 0x7fe118106a90 2024-09-19T08:13:53.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe10ffff640 1 --2- 172.21.15.98:0/1676487339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84cea780 0x560f84cecb70 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fe1000125a0 tx=0x7fe10000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a8a6ec182c565f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:53.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.098+0000 7fe10e7fc640 1 -- 172.21.15.98:0/1676487339 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100004250 con 0x560f84cea780 2024-09-19T08:13:53.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.099+0000 7fe10e7fc640 1 -- 172.21.15.98:0/1676487339 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1000043f0 con 0x560f84cea780 2024-09-19T08:13:53.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.099+0000 7fe10e7fc640 1 -- 172.21.15.98:0/1676487339 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100004e40 con 0x560f84cea780 2024-09-19T08:13:53.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.099+0000 7fe120791740 1 -- 172.21.15.98:0/1676487339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84cea780 msgr2=0x560f84cecb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:53.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.099+0000 7fe120791740 1 --2- 172.21.15.98:0/1676487339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84cea780 0x560f84cecb70 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fe1000125a0 tx=0x7fe10000b870 comp rx=0 tx=0).stop 2024-09-19T08:13:53.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.099+0000 7fe11c955640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1040a33b0 0x7fe1040a5870 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe1100ef180 tx=0x7fe1100a8040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:53.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.099+0000 7fe120791740 1 -- 172.21.15.98:0/1676487339 shutdown_connections 2024-09-19T08:13:53.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.099+0000 7fe120791740 1 --2- 172.21.15.98:0/1676487339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84cea780 0x560f84cecb70 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.099+0000 7fe120791740 1 --2- 172.21.15.98:0/1676487339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cdf9e0 0x560f84cdfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.099+0000 7fe120791740 1 --2- 172.21.15.98:0/1676487339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84ce0f50 0x560f84cdf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.099+0000 7fe120791740 1 -- 172.21.15.98:0/1676487339 >> 172.21.15.98:0/1676487339 conn(0x560f84ca4310 msgr2=0x560f84cdc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.099+0000 7fe120791740 1 -- 172.21.15.98:0/1676487339 shutdown_connections 2024-09-19T08:13:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.099+0000 7fe120791740 1 -- 172.21.15.98:0/1676487339 wait complete. 2024-09-19T08:13:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.100+0000 7fe120791740 1 -- start start 2024-09-19T08:13:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.100+0000 7fe120791740 1 Processor -- start 2024-09-19T08:13:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.100+0000 7fe120791740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cdf9e0 0x560f84cea200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.100+0000 7fe120791740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84ce0f50 0x560f84ce74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.100+0000 7fe120791740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cea780 0x560f84ce7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.100+0000 7fe120791740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560f84ced480 con 0x560f84ce0f50 2024-09-19T08:13:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.100+0000 7fe120791740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560f84bfc060 con 0x560f84cea780 2024-09-19T08:13:53.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.100+0000 7fe120791740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560f84ced600 con 0x560f84cdf9e0 2024-09-19T08:13:53.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.100+0000 7fe120791740 10 client.0 _ll_register_callbacks cb 0x560f8494f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.100+0000 7fe117fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84ce0f50 0x560f84ce74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.100+0000 7fe117fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84ce0f50 0x560f84ce74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37836/0 (socket says 172.21.15.98:37836) 2024-09-19T08:13:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.100+0000 7fe117fff640 1 -- 172.21.15.98:0/1399625993 learned_addr learned my addr 172.21.15.98:0/1399625993 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.100+0000 7fe10ffff640 1 --2- 172.21.15.98:0/1399625993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cdf9e0 0x560f84cea200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe10ffff640 1 --2- 172.21.15.98:0/1399625993 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cea780 0x560f84ce7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe117fff640 1 -- 172.21.15.98:0/1399625993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cdf9e0 msgr2=0x560f84cea200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe117fff640 1 --2- 172.21.15.98:0/1399625993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cdf9e0 0x560f84cea200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe117fff640 1 -- 172.21.15.98:0/1399625993 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cea780 msgr2=0x560f84ce7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe117fff640 1 --2- 172.21.15.98:0/1399625993 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cea780 0x560f84ce7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe117fff640 1 -- 172.21.15.98:0/1399625993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f849b23c0 con 0x560f84ce0f50 2024-09-19T08:13:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe10ffff640 1 --2- 172.21.15.98:0/1399625993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cdf9e0 0x560f84cea200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:53.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe117fff640 1 --2- 172.21.15.98:0/1399625993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84ce0f50 0x560f84ce74b0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe10800df00 tx=0x7fe10800f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce679b92ae9295c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe0f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe0f77fe640 1 -- 172.21.15.98:0/1399625993 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe108018070 con 0x560f84ce0f50 2024-09-19T08:13:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe0f77fe640 1 -- 172.21.15.98:0/1399625993 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7fe108014680 con 0x560f84ce0f50 2024-09-19T08:13:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe120791740 1 -- 172.21.15.98:0/1399625993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1000043f0 con 0x560f84ce0f50 2024-09-19T08:13:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe0f77fe640 1 -- 172.21.15.98:0/1399625993 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe108024050 con 0x560f84ce0f50 2024-09-19T08:13:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe120791740 10 client.17964 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe120791740 1 -- 172.21.15.98:0/1399625993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560f84cf9060 con 0x560f84ce0f50 2024-09-19T08:13:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe0f77fe640 1 -- 172.21.15.98:0/1399625993 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1000043f0 con 0x560f84ce0f50 2024-09-19T08:13:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.101+0000 7fe0f77fe640 1 -- 172.21.15.98:0/1399625993 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe108013b70 con 0x560f84ce0f50 2024-09-19T08:13:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.102+0000 7fe120791740 10 client.17964 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.102+0000 7fe120791740 10 client.17964 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:13:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.102+0000 7fe120791740 20 client.17964 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.102+0000 7fe120791740 1 --2- 172.21.15.98:0/1399625993 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560f84cfa4c0 0x560f84cfc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.102+0000 7fe120791740 4 client.17964 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T08:13:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.102+0000 7fe120791740 1 -- 172.21.15.98:0/1399625993 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f84c4e860 con 0x560f84cfa4c0 2024-09-19T08:13:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.102+0000 7fe10ffff640 1 --2- 172.21.15.98:0/1399625993 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560f84cfa4c0 0x560f84cfc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.102+0000 7fe0f77fe640 10 client.17964 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.102+0000 7fe10ffff640 1 --2- 172.21.15.98:0/1399625993 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560f84cfa4c0 0x560f84cfc920 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be79786737bee3e server_cookie=706ecd55b31bd6c9 in_seq=0 out_seq=0 2024-09-19T08:13:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.103+0000 7fe0f77fe640 1 -- 172.21.15.98:0/1399625993 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560f84c4e860 con 0x560f84cfa4c0 2024-09-19T08:13:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.103+0000 7fe0f77fe640 10 client.17964 handle_command_reply: tid=0 2024-09-19T08:13:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.103+0000 7fe120791740 1 client.17964 shutdown 2024-09-19T08:13:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.103+0000 7fe120791740 1 -- 172.21.15.98:0/1399625993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84ce0f50 msgr2=0x560f84ce74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.103+0000 7fe120791740 1 --2- 172.21.15.98:0/1399625993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84ce0f50 0x560f84ce74b0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe10800df00 tx=0x7fe10800f5b0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.103+0000 7fe120791740 1 -- 172.21.15.98:0/1399625993 shutdown_connections 2024-09-19T08:13:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.103+0000 7fe120791740 1 --2- 172.21.15.98:0/1399625993 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560f84cfa4c0 0x560f84cfc920 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.103+0000 7fe120791740 1 --2- 172.21.15.98:0/1399625993 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cea780 0x560f84ce7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.103+0000 7fe120791740 1 --2- 172.21.15.98:0/1399625993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84ce0f50 0x560f84ce74b0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.103+0000 7fe120791740 1 --2- 172.21.15.98:0/1399625993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cdf9e0 0x560f84cea200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.103+0000 7fe120791740 1 -- 172.21.15.98:0/1399625993 >> 172.21.15.98:0/1399625993 conn(0x560f84ca4310 msgr2=0x560f84ce1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.103+0000 7fe0f77fe640 0 client.17964 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.104+0000 7fe120791740 1 -- 172.21.15.98:0/1399625993 shutdown_connections 2024-09-19T08:13:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.104+0000 7fe120791740 1 -- 172.21.15.98:0/1399625993 wait complete. 2024-09-19T08:13:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.104+0000 7fe120791740 20 client.17964 trim_cache size 0 max 16384 2024-09-19T08:13:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.116+0000 7fe120791740 1 Processor -- start 2024-09-19T08:13:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.116+0000 7fe120791740 1 -- start start 2024-09-19T08:13:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.116+0000 7fe120791740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cdf9e0 0x560f84cf6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.116+0000 7fe120791740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84ce0f50 0x560f84d021f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.116+0000 7fe120791740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cea780 0x560f84d02730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.116+0000 7fe120791740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181ab590 con 0x560f84ce0f50 2024-09-19T08:13:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.116+0000 7fe120791740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181ad0d0 con 0x560f84cdf9e0 2024-09-19T08:13:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.116+0000 7fe120791740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181abc60 con 0x560f84cea780 2024-09-19T08:13:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.117+0000 7fe117fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84ce0f50 0x560f84d021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.117+0000 7fe117fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84ce0f50 0x560f84d021f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37846/0 (socket says 172.21.15.98:37846) 2024-09-19T08:13:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.117+0000 7fe117fff640 1 -- 172.21.15.98:0/223490912 learned_addr learned my addr 172.21.15.98:0/223490912 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.117+0000 7fe10ffff640 1 --2- 172.21.15.98:0/223490912 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cdf9e0 0x560f84cf6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.117+0000 7fe117fff640 1 -- 172.21.15.98:0/223490912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cea780 msgr2=0x560f84d02730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.117+0000 7fe117fff640 1 --2- 172.21.15.98:0/223490912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cea780 0x560f84d02730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.117+0000 7fe117fff640 1 -- 172.21.15.98:0/223490912 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cdf9e0 msgr2=0x560f84cf6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.117+0000 7fe117fff640 1 --2- 172.21.15.98:0/223490912 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cdf9e0 0x560f84cf6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.117+0000 7fe117fff640 1 -- 172.21.15.98:0/223490912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f84cdcd70 con 0x560f84ce0f50 2024-09-19T08:13:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.117+0000 7fe117fff640 1 --2- 172.21.15.98:0/223490912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84ce0f50 0x560f84d021f0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe108014040 tx=0x7fe108004660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9e4fad888a98b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.117+0000 7fe10dffb640 1 -- 172.21.15.98:0/223490912 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe108018040 con 0x560f84ce0f50 2024-09-19T08:13:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.117+0000 7fe10dffb640 1 -- 172.21.15.98:0/223490912 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe10801c520 con 0x560f84ce0f50 2024-09-19T08:13:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.117+0000 7fe10dffb640 1 -- 172.21.15.98:0/223490912 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe108006020 con 0x560f84ce0f50 2024-09-19T08:13:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.118+0000 7fe120791740 1 -- 172.21.15.98:0/223490912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84ce0f50 msgr2=0x560f84d021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.118+0000 7fe120791740 1 --2- 172.21.15.98:0/223490912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84ce0f50 0x560f84d021f0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe108014040 tx=0x7fe108004660 comp rx=0 tx=0).stop 2024-09-19T08:13:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.118+0000 7fe120791740 1 -- 172.21.15.98:0/223490912 shutdown_connections 2024-09-19T08:13:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.118+0000 7fe120791740 1 --2- 172.21.15.98:0/223490912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84cea780 0x560f84d02730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.118+0000 7fe120791740 1 --2- 172.21.15.98:0/223490912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84ce0f50 0x560f84d021f0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.118+0000 7fe120791740 1 --2- 172.21.15.98:0/223490912 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cdf9e0 0x560f84cf6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.118+0000 7fe120791740 1 -- 172.21.15.98:0/223490912 >> 172.21.15.98:0/223490912 conn(0x560f84ca4310 msgr2=0x560f84ce9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.118+0000 7fe120791740 1 -- 172.21.15.98:0/223490912 shutdown_connections 2024-09-19T08:13:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.118+0000 7fe120791740 1 -- 172.21.15.98:0/223490912 wait complete. 2024-09-19T08:13:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.118+0000 7fe120791740 1 -- start start 2024-09-19T08:13:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.118+0000 7fe120791740 1 Processor -- start 2024-09-19T08:13:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.118+0000 7fe120791740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cdf9e0 0x560f84cee9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.118+0000 7fe120791740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84ce0f50 0x560f84ceef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe120791740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84cea780 0x560f84d641c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe120791740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181aca00 con 0x560f84cea780 2024-09-19T08:13:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe120791740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181ac330 con 0x560f84cdf9e0 2024-09-19T08:13:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe120791740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181a9590 con 0x560f84ce0f50 2024-09-19T08:13:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe120791740 10 client.0 _ll_register_callbacks cb 0x7fe10800d180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T08:13:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe10ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84cea780 0x560f84d641c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe10ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84cea780 0x560f84d641c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37854/0 (socket says 172.21.15.98:37854) 2024-09-19T08:13:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe10ffff640 1 -- 172.21.15.98:0/125007556 learned_addr learned my addr 172.21.15.98:0/125007556 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe117fff640 1 --2- 172.21.15.98:0/125007556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84ce0f50 0x560f84ceef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe10ffff640 1 --2- 172.21.15.98:0/125007556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cdf9e0 0x560f84cee9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe10ffff640 1 -- 172.21.15.98:0/125007556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84ce0f50 msgr2=0x560f84ceef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe10ffff640 1 --2- 172.21.15.98:0/125007556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84ce0f50 0x560f84ceef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe10ffff640 1 -- 172.21.15.98:0/125007556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cdf9e0 msgr2=0x560f84cee9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T08:13:53.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe10ffff640 1 --2- 172.21.15.98:0/125007556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cdf9e0 0x560f84cee9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe10ffff640 1 -- 172.21.15.98:0/125007556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f84d64700 con 0x560f84cea780 2024-09-19T08:13:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe10ffff640 1 --2- 172.21.15.98:0/125007556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84cea780 0x560f84d641c0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe100012540 tx=0x7fe100014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74a4e22597644f3b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe0f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-09-19T08:13:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe0f77fe640 1 -- 172.21.15.98:0/125007556 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100033050 con 0x560f84cea780 2024-09-19T08:13:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe0f77fe640 1 -- 172.21.15.98:0/125007556 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7fe100029040 con 0x560f84cea780 2024-09-19T08:13:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe120791740 1 -- 172.21.15.98:0/125007556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe10801c520 con 0x560f84cea780 2024-09-19T08:13:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.119+0000 7fe0f77fe640 1 -- 172.21.15.98:0/125007556 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe10003f050 con 0x560f84cea780 2024-09-19T08:13:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.120+0000 7fe120791740 10 client.17970 fetch_fsmap learned FSMap version 32 2024-09-19T08:13:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.120+0000 7fe120791740 1 -- 172.21.15.98:0/125007556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560f84d65850 con 0x560f84cea780 2024-09-19T08:13:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.120+0000 7fe0f77fe640 1 -- 172.21.15.98:0/125007556 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe10801c520 con 0x560f84cea780 2024-09-19T08:13:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.120+0000 7fe0f77fe640 1 -- 172.21.15.98:0/125007556 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe100037020 con 0x560f84cea780 2024-09-19T08:13:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.120+0000 7fe120791740 10 client.17970 fetch_fsmap finished waiting for FSMap version 32 2024-09-19T08:13:53.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.120+0000 7fe120791740 10 client.17970 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-19T08:13:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.120+0000 7fe120791740 20 client.17970 populate_metadata read hostname 'smithi098' 2024-09-19T08:13:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.120+0000 7fe120791740 1 --2- 172.21.15.98:0/125007556 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560f84d65cc0 0x560f84d680b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.120+0000 7fe120791740 4 client.17970 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-19T08:13:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.120+0000 7fe120791740 1 -- 172.21.15.98:0/125007556 --> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f84c48d70 con 0x560f84d65cc0 2024-09-19T08:13:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.120+0000 7fe117fff640 1 --2- 172.21.15.98:0/125007556 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560f84d65cc0 0x560f84d680b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.120+0000 7fe117fff640 1 --2- 172.21.15.98:0/125007556 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560f84d65cc0 0x560f84d680b0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2e99b1841e829e3 server_cookie=debf9fd0f22f9a63 in_seq=0 out_seq=0 2024-09-19T08:13:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.120+0000 7fe0f77fe640 10 client.17970 ms_handle_connect on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe0f77fe640 1 -- 172.21.15.98:0/125007556 <== mds.0 v2:172.21.15.139:6838/1001534713 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f84c48d70 con 0x560f84d65cc0 2024-09-19T08:13:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe0f77fe640 10 client.17970 handle_command_reply: tid=0 2024-09-19T08:13:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe120791740 1 client.17970 shutdown 2024-09-19T08:13:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe120791740 1 -- 172.21.15.98:0/125007556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84cea780 msgr2=0x560f84d641c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe120791740 1 --2- 172.21.15.98:0/125007556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84cea780 0x560f84d641c0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe100012540 tx=0x7fe100014040 comp rx=0 tx=0).stop 2024-09-19T08:13:53.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe120791740 1 -- 172.21.15.98:0/125007556 shutdown_connections 2024-09-19T08:13:53.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe120791740 1 --2- 172.21.15.98:0/125007556 >> [v2:172.21.15.139:6838/1001534713,v1:172.21.15.139:6839/1001534713] conn(0x560f84d65cc0 0x560f84d680b0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe120791740 1 --2- 172.21.15.98:0/125007556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f84cea780 0x560f84d641c0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe120791740 1 --2- 172.21.15.98:0/125007556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f84ce0f50 0x560f84ceef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe120791740 1 --2- 172.21.15.98:0/125007556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f84cdf9e0 0x560f84cee9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe120791740 1 -- 172.21.15.98:0/125007556 >> 172.21.15.98:0/125007556 conn(0x560f84ca4310 msgr2=0x560f84ce8d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:53.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe0f77fe640 0 client.17970 ms_handle_reset on v2:172.21.15.139:6838/1001534713 2024-09-19T08:13:53.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe120791740 1 -- 172.21.15.98:0/125007556 shutdown_connections 2024-09-19T08:13:53.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe120791740 1 -- 172.21.15.98:0/125007556 wait complete. 2024-09-19T08:13:53.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.121+0000 7fe120791740 20 client.17970 trim_cache size 0 max 16384 2024-09-19T08:13:53.128 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-09-19T08:13:53.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.122+0000 7fe11e3b8640 1 -- 172.21.15.98:0/1300060470 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1040a33b0 msgr2=0x7fe1040a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:53.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.122+0000 7fe11e3b8640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1040a33b0 0x7fe1040a5870 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe1100ef180 tx=0x7fe1100a8040 comp rx=0 tx=0).stop 2024-09-19T08:13:53.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.122+0000 7fe11e3b8640 1 -- 172.21.15.98:0/1300060470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe118106a90 msgr2=0x7fe11819cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:53.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.122+0000 7fe11e3b8640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe118106a90 0x7fe11819cd40 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe110005cc0 tx=0x7fe110003970 comp rx=0 tx=0).stop 2024-09-19T08:13:53.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.122+0000 7fe11e3b8640 1 -- 172.21.15.98:0/1300060470 shutdown_connections 2024-09-19T08:13:53.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.122+0000 7fe11e3b8640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fe1040a33b0 0x7fe1040a5870 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.122+0000 7fe11e3b8640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe118106a90 0x7fe11819cd40 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.122+0000 7fe11e3b8640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe118101430 0x7fe1181989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.122+0000 7fe11e3b8640 1 --2- 172.21.15.98:0/1300060470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe118100a90 0x7fe118198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.122+0000 7fe11e3b8640 1 -- 172.21.15.98:0/1300060470 >> 172.21.15.98:0/1300060470 conn(0x7fe1180fc7d0 msgr2=0x7fe11806a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:53.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.122+0000 7fe11e3b8640 1 -- 172.21.15.98:0/1300060470 shutdown_connections 2024-09-19T08:13:53.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.122+0000 7fe11e3b8640 1 -- 172.21.15.98:0/1300060470 wait complete. 2024-09-19T08:13:53.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:52 smithi139 ceph-mon[34874]: pgmap v336: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:53.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:52 smithi162 ceph-mon[35260]: pgmap v336: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:53.824 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-09-19T08:13:53.825 INFO:tasks.fwd_scrub:done joining 2024-09-19T08:13:53.825 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-09-19T08:13:53.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T08:13:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.944+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3105706494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34ac106990 msgr2=0x7f34ac106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.944+0000 7f34b2c7b640 1 --2- 172.21.15.98:0/3105706494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34ac106990 0x7f34ac106d70 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f34a4009930 tx=0x7f34a402a420 comp rx=0 tx=0).stop 2024-09-19T08:13:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.944+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3105706494 shutdown_connections 2024-09-19T08:13:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.944+0000 7f34b2c7b640 1 --2- 172.21.15.98:0/3105706494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34ac101320 0x7f34ac10d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.944+0000 7f34b2c7b640 1 --2- 172.21.15.98:0/3105706494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34ac100980 0x7f34ac100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.944+0000 7f34b2c7b640 1 --2- 172.21.15.98:0/3105706494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34ac106990 0x7f34ac106d70 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.944+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3105706494 >> 172.21.15.98:0/3105706494 conn(0x7f34ac077a10 msgr2=0x7f34ac0fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.944+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3105706494 shutdown_connections 2024-09-19T08:13:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.944+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3105706494 wait complete. 2024-09-19T08:13:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34b2c7b640 1 Processor -- start 2024-09-19T08:13:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34b2c7b640 1 -- start start 2024-09-19T08:13:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34b2c7b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34ac100980 0x7f34ac199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34b2c7b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34ac101320 0x7f34ac199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34b2c7b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34ac106990 0x7f34ac1a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34b2c7b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac110dd0 con 0x7f34ac101320 2024-09-19T08:13:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34b2c7b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac110c50 con 0x7f34ac106990 2024-09-19T08:13:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34b2c7b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac110f50 con 0x7f34ac100980 2024-09-19T08:13:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34abfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34ac101320 0x7f34ac199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34abfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34ac101320 0x7f34ac199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37868/0 (socket says 172.21.15.98:37868) 2024-09-19T08:13:53.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34abfff640 1 -- 172.21.15.98:0/3355813384 learned_addr learned my addr 172.21.15.98:0/3355813384 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34b1218640 1 --2- 172.21.15.98:0/3355813384 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34ac106990 0x7f34ac1a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34b0a17640 1 --2- 172.21.15.98:0/3355813384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34ac100980 0x7f34ac199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34abfff640 1 -- 172.21.15.98:0/3355813384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34ac100980 msgr2=0x7f34ac199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34abfff640 1 --2- 172.21.15.98:0/3355813384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34ac100980 0x7f34ac199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34abfff640 1 -- 172.21.15.98:0/3355813384 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34ac106990 msgr2=0x7f34ac1a4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34abfff640 1 --2- 172.21.15.98:0/3355813384 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34ac106990 0x7f34ac1a4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34abfff640 1 -- 172.21.15.98:0/3355813384 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34ac1a48c0 con 0x7f34ac101320 2024-09-19T08:13:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.945+0000 7f34abfff640 1 --2- 172.21.15.98:0/3355813384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34ac101320 0x7f34ac199840 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f349c00eab0 tx=0x7f349c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.946+0000 7f34a9ffb640 1 -- 172.21.15.98:0/3355813384 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349c00ce40 con 0x7f34ac101320 2024-09-19T08:13:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.946+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3355813384 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34ac1a4a90 con 0x7f34ac101320 2024-09-19T08:13:53.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.946+0000 7f34a9ffb640 1 -- 172.21.15.98:0/3355813384 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f349c011950 con 0x7f34ac101320 2024-09-19T08:13:53.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.946+0000 7f34a9ffb640 1 -- 172.21.15.98:0/3355813384 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349c010ae0 con 0x7f34ac101320 2024-09-19T08:13:53.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.946+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3355813384 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34ac1a5000 con 0x7f34ac101320 2024-09-19T08:13:53.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.947+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3355813384 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34ac100de0 con 0x7f34ac101320 2024-09-19T08:13:53.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.948+0000 7f34a9ffb640 1 -- 172.21.15.98:0/3355813384 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f349c011b00 con 0x7f34ac101320 2024-09-19T08:13:53.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.948+0000 7f34a9ffb640 1 --2- 172.21.15.98:0/3355813384 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f34980a33c0 0x7f34980a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:53.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.949+0000 7f34a9ffb640 1 -- 172.21.15.98:0/3355813384 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f349c0d04d0 con 0x7f34ac101320 2024-09-19T08:13:53.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.949+0000 7f34b0a17640 1 --2- 172.21.15.98:0/3355813384 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f34980a33c0 0x7f34980a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:53.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.949+0000 7f34b0a17640 1 --2- 172.21.15.98:0/3355813384 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f34980a33c0 0x7f34980a5880 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f34a4000c00 tx=0x7f34a4005a20 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:53.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:53.953+0000 7f34a9ffb640 1 -- 172.21.15.98:0/3355813384 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f349c016030 con 0x7f34ac101320 2024-09-19T08:13:54.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.120+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3355813384 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f34ac108f20 con 0x7f34ac101320 2024-09-19T08:13:54.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.122+0000 7f34a9ffb640 1 -- 172.21.15.98:0/3355813384 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+23218 (secure 0 0 0) 0x7f349c09ceb0 con 0x7f34ac101320 2024-09-19T08:13:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:13:54.122 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","created":"2024-09-19T07:53:39.127398+0000","modified":"2024-09-19T08:13:15.869049+0000","last_up_change":"2024-09-19T08:00:22.467777+0000","last_in_change":"2024-09-19T07:59:58.538492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:56:43.773892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T08:01:07.693796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T08:01:09.518028+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":88,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d01ce6-165d-493d-ae98-d8ad9e5d4134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6803","nonce":1719360533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6805","nonce":1719360533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6809","nonce":1719360533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6807","nonce":1719360533}]},"public_addr":"172.21.15.98:6803/1719360533","cluster_addr":"172.21.15.98:6805/1719360533","heartbeat_back_addr":"172.21.15.98:6809/1719360533","heartbeat_front_addr":"172.21.15.98:6807/1719360533","state":["exists","up"]},{"osd":1,"uuid":"34724bd5-0738-4473-b087-c09031c1ee2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036962626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036962626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036962626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036962626}]},"public_addr":"172.21.15.139:6801/4036962626","cluster_addr":"172.21.15.139:6803/4036962626","heartbeat_back_addr":"172.21.15.139:6807/4036962626","heartbeat_front_addr":"172.21.15.139:6805/4036962626","state":["exists","up"]},{"osd":2,"uuid":"16985ebe-0e32-42ee-bded-b7838e290afc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6801","nonce":2811412891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6803","nonce":2811412891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6807","nonce":2811412891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6805","nonce":2811412891}]},"public_addr":"172.21.15.162:6801/2811412891","cluster_addr":"172.21.15.162:6803/2811412891","heartbeat_back_addr":"172.21.15.162:6807/2811412891","heartbeat_front_addr":"172.21.15.162:6805/2811412891","state":["exists","up"]},{"osd":3,"uuid":"b140e466-3119-49cb-aec2-ba6ca4d10d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6811","nonce":1383392014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6813","nonce":1383392014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6817","nonce":1383392014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6815","nonce":1383392014}]},"public_addr":"172.21.15.98:6811/1383392014","cluster_addr":"172.21.15.98:6813/1383392014","heartbeat_back_addr":"172.21.15.98:6817/1383392014","heartbeat_front_addr":"172.21.15.98:6815/1383392014","state":["exists","up"]},{"osd":4,"uuid":"b4412813-df57-4ac6-8fb7-f57b69c74768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6809","nonce":116453520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6811","nonce":116453520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6815","nonce":116453520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6813","nonce":116453520}]},"public_addr":"172.21.15.139:6809/116453520","cluster_addr":"172.21.15.139:6811/116453520","heartbeat_back_addr":"172.21.15.139:6815/116453520","heartbeat_front_addr":"172.21.15.139:6813/116453520","state":["exists","up"]},{"osd":5,"uuid":"d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6809","nonce":3769530297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6811","nonce":3769530297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6815","nonce":3769530297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6813","nonce":3769530297}]},"public_addr":"172.21.15.162:6809/3769530297","cluster_addr":"172.21.15.162:6811/3769530297","heartbeat_back_addr":"172.21.15.162:6815/3769530297","heartbeat_front_addr":"172.21.15.162:6813/3769530297","state":["exists","up"]},{"osd":6,"uuid":"ba1c1590-6aea-40dc-90c8-e21b88896909","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6819","nonce":692373267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6821","nonce":692373267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6825","nonce":692373267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6823","nonce":692373267}]},"public_addr":"172.21.15.98:6819/692373267","cluster_addr":"172.21.15.98:6821/692373267","heartbeat_back_addr":"172.21.15.98:6825/692373267","heartbeat_front_addr":"172.21.15.98:6823/692373267","state":["exists","up"]},{"osd":7,"uuid":"eb9093ad-a068-4a63-ab87-0b50522cbc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6817","nonce":1963811650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6819","nonce":1963811650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6823","nonce":1963811650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6821","nonce":1963811650}]},"public_addr":"172.21.15.139:6817/1963811650","cluster_addr":"172.21.15.139:6819/1963811650","heartbeat_back_addr":"172.21.15.139:6823/1963811650","heartbeat_front_addr":"172.21.15.139:6821/1963811650","state":["exists","up"]},{"osd":8,"uuid":"31ca330a-fc45-466d-81c2-ef4386b3cc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6817","nonce":3607966458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6819","nonce":3607966458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6823","nonce":3607966458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6821","nonce":3607966458}]},"public_addr":"172.21.15.162:6817/3607966458","cluster_addr":"172.21.15.162:6819/3607966458","heartbeat_back_addr":"172.21.15.162:6823/3607966458","heartbeat_front_addr":"172.21.15.162:6821/3607966458","state":["exists","up"]},{"osd":9,"uuid":"9c4f216f-aef1-4367-9c58-b913b0c4f7db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6827","nonce":2050391244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6829","nonce":2050391244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6833","nonce":2050391244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6831","nonce":2050391244}]},"public_addr":"172.21.15.98:6827/2050391244","cluster_addr":"172.21.15.98:6829/2050391244","heartbeat_back_addr":"172.21.15.98:6833/2050391244","heartbeat_front_addr":"172.21.15.98:6831/2050391244","state":["exists","up"]},{"osd":10,"uuid":"03cc7df6-84ae-4d9a-a3d7-74c91e1744e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6825","nonce":2336761639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6827","nonce":2336761639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6831","nonce":2336761639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6829","nonce":2336761639}]},"public_addr":"172.21.15.139:6825/2336761639","cluster_addr":"172.21.15.139:6827/2336761639","heartbeat_back_addr":"172.21.15.139:6831/2336761639","heartbeat_front_addr":"172.21.15.139:6829/2336761639","state":["exists","up"]},{"osd":11,"uuid":"29e95498-4952-46d4-8fa6-6d8e2b862111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6825","nonce":1231011554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6827","nonce":1231011554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6831","nonce":1231011554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6829","nonce":1231011554}]},"public_addr":"172.21.15.162:6825/1231011554","cluster_addr":"172.21.15.162:6827/1231011554","heartbeat_back_addr":"172.21.15.162:6831/1231011554","heartbeat_front_addr":"172.21.15.162:6829/1231011554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:55:49.995244+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:15.655543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:40.331165+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:04.688881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:29.174966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:53.663161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:17.407551+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:42.584296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:06.317219+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:31.078625+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:55.689269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T08:00:19.659748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1},{"from":7,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1252058007":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/921776066":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/336479555":"2024-09-20T08:03:00.753287+0000","172.21.15.98:6800/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6800/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/1497156658":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6801/470997812":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/1238193420":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6800/470997812":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/3838590070":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3842627255":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3866031018":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3194864054":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3067331617":"2024-09-20T08:03:00.753287+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T08:13:54.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.124+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3355813384 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f34980a33c0 msgr2=0x7f34980a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.124+0000 7f34b2c7b640 1 --2- 172.21.15.98:0/3355813384 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f34980a33c0 0x7f34980a5880 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f34a4000c00 tx=0x7f34a4005a20 comp rx=0 tx=0).stop 2024-09-19T08:13:54.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.124+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3355813384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34ac101320 msgr2=0x7f34ac199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.124+0000 7f34b2c7b640 1 --2- 172.21.15.98:0/3355813384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34ac101320 0x7f34ac199840 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f349c00eab0 tx=0x7f349c00ef70 comp rx=0 tx=0).stop 2024-09-19T08:13:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.124+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3355813384 shutdown_connections 2024-09-19T08:13:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.124+0000 7f34b2c7b640 1 --2- 172.21.15.98:0/3355813384 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f34980a33c0 0x7f34980a5880 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.124+0000 7f34b2c7b640 1 --2- 172.21.15.98:0/3355813384 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34ac106990 0x7f34ac1a4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.124+0000 7f34b2c7b640 1 --2- 172.21.15.98:0/3355813384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34ac101320 0x7f34ac199840 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.124+0000 7f34b2c7b640 1 --2- 172.21.15.98:0/3355813384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34ac100980 0x7f34ac199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.124+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3355813384 >> 172.21.15.98:0/3355813384 conn(0x7f34ac077a10 msgr2=0x7f34ac0ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.124+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3355813384 shutdown_connections 2024-09-19T08:13:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.124+0000 7f34b2c7b640 1 -- 172.21.15.98:0/3355813384 wait complete. 2024-09-19T08:13:54.136 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-19T07:56:43.773892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-19T08:01:07.693796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-19T08:01:09.518028+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 88, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-19T08:13:54.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-19T08:13:54.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 -- 172.21.15.98:0/3174108579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56680fffd0 msgr2=0x7f5668100430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 --2- 172.21.15.98:0/3174108579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56680fffd0 0x7f5668100430 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f5658009900 tx=0x7f565802a410 comp rx=0 tx=0).stop 2024-09-19T08:13:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 -- 172.21.15.98:0/3174108579 shutdown_connections 2024-09-19T08:13:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 --2- 172.21.15.98:0/3174108579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5668100970 0x7f566810e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 --2- 172.21.15.98:0/3174108579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56680fffd0 0x7f5668100430 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 --2- 172.21.15.98:0/3174108579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5668106ab0 0x7f5668106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 -- 172.21.15.98:0/3174108579 >> 172.21.15.98:0/3174108579 conn(0x7f5668077170 msgr2=0x7f56680fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 -- 172.21.15.98:0/3174108579 shutdown_connections 2024-09-19T08:13:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 -- 172.21.15.98:0/3174108579 wait complete. 2024-09-19T08:13:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 Processor -- start 2024-09-19T08:13:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 -- start start 2024-09-19T08:13:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56680fffd0 0x7f5668198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5668100970 0x7f5668198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5668106ab0 0x7f566819ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5668111fc0 con 0x7f5668100970 2024-09-19T08:13:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5668111e40 con 0x7f5668106ab0 2024-09-19T08:13:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.239+0000 7f566f35f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5668112140 con 0x7f56680fffd0 2024-09-19T08:13:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f566c8fa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5668100970 0x7f5668198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f566c8fa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5668100970 0x7f5668198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37884/0 (socket says 172.21.15.98:37884) 2024-09-19T08:13:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f566c8fa640 1 -- 172.21.15.98:0/3210257826 learned_addr learned my addr 172.21.15.98:0/3210257826 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f566d0fb640 1 --2- 172.21.15.98:0/3210257826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56680fffd0 0x7f5668198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f566c8fa640 1 -- 172.21.15.98:0/3210257826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56680fffd0 msgr2=0x7f5668198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f566c8fa640 1 --2- 172.21.15.98:0/3210257826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56680fffd0 0x7f5668198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f566c8fa640 1 -- 172.21.15.98:0/3210257826 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5668106ab0 msgr2=0x7f566819ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f566c8fa640 1 --2- 172.21.15.98:0/3210257826 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5668106ab0 0x7f566819ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f566c8fa640 1 -- 172.21.15.98:0/3210257826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566819d540 con 0x7f5668100970 2024-09-19T08:13:54.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f566d0fb640 1 --2- 172.21.15.98:0/3210257826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56680fffd0 0x7f5668198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:54.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f566c8fa640 1 --2- 172.21.15.98:0/3210257826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5668100970 0x7f5668198a90 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f5658006010 tx=0x7f56580042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:54.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f565e7fc640 1 -- 172.21.15.98:0/3210257826 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565803f030 con 0x7f5668100970 2024-09-19T08:13:54.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f566f35f640 1 -- 172.21.15.98:0/3210257826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f566819d7d0 con 0x7f5668100970 2024-09-19T08:13:54.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f565e7fc640 1 -- 172.21.15.98:0/3210257826 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5658003ce0 con 0x7f5668100970 2024-09-19T08:13:54.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f565e7fc640 1 -- 172.21.15.98:0/3210257826 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5658035070 con 0x7f5668100970 2024-09-19T08:13:54.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.240+0000 7f566f35f640 1 -- 172.21.15.98:0/3210257826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56681a50c0 con 0x7f5668100970 2024-09-19T08:13:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.241+0000 7f566f35f640 1 -- 172.21.15.98:0/3210257826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5668100430 con 0x7f5668100970 2024-09-19T08:13:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.242+0000 7f565e7fc640 1 -- 172.21.15.98:0/3210257826 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f5658042030 con 0x7f5668100970 2024-09-19T08:13:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.243+0000 7f565e7fc640 1 --2- 172.21.15.98:0/3210257826 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f56540a33c0 0x7f56540a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.243+0000 7f566d0fb640 1 --2- 172.21.15.98:0/3210257826 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f56540a33c0 0x7f56540a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.243+0000 7f565e7fc640 1 -- 172.21.15.98:0/3210257826 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f56580f0570 con 0x7f5668100970 2024-09-19T08:13:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.243+0000 7f566d0fb640 1 --2- 172.21.15.98:0/3210257826 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f56540a33c0 0x7f56540a5880 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5664005fd0 tx=0x7f56640094d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:54.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.247+0000 7f565e7fc640 1 -- 172.21.15.98:0/3210257826 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f56580bd010 con 0x7f5668100970 2024-09-19T08:13:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.415+0000 7f566f35f640 1 -- 172.21.15.98:0/3210257826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5668109f90 con 0x7f5668100970 2024-09-19T08:13:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.416+0000 7f565e7fc640 1 -- 172.21.15.98:0/3210257826 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f56580bc910 con 0x7f5668100970 2024-09-19T08:13:54.416 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-09-19T08:13:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.418+0000 7f566f35f640 1 -- 172.21.15.98:0/3210257826 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f56540a33c0 msgr2=0x7f56540a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.418+0000 7f566f35f640 1 --2- 172.21.15.98:0/3210257826 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f56540a33c0 0x7f56540a5880 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5664005fd0 tx=0x7f56640094d0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.418+0000 7f566f35f640 1 -- 172.21.15.98:0/3210257826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5668100970 msgr2=0x7f5668198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.418+0000 7f566f35f640 1 --2- 172.21.15.98:0/3210257826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5668100970 0x7f5668198a90 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f5658006010 tx=0x7f56580042f0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.418+0000 7f566f35f640 1 -- 172.21.15.98:0/3210257826 shutdown_connections 2024-09-19T08:13:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.418+0000 7f566f35f640 1 --2- 172.21.15.98:0/3210257826 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f56540a33c0 0x7f56540a5880 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.418+0000 7f566f35f640 1 --2- 172.21.15.98:0/3210257826 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5668106ab0 0x7f566819ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.418+0000 7f566f35f640 1 --2- 172.21.15.98:0/3210257826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5668100970 0x7f5668198a90 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.418+0000 7f566f35f640 1 --2- 172.21.15.98:0/3210257826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56680fffd0 0x7f5668198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.418+0000 7f566f35f640 1 -- 172.21.15.98:0/3210257826 >> 172.21.15.98:0/3210257826 conn(0x7f5668077170 msgr2=0x7f5668104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:54.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.418+0000 7f566f35f640 1 -- 172.21.15.98:0/3210257826 shutdown_connections 2024-09-19T08:13:54.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.418+0000 7f566f35f640 1 -- 172.21.15.98:0/3210257826 wait complete. 2024-09-19T08:13:54.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-19T08:13:54.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.533+0000 7fa8456f1640 1 -- 172.21.15.98:0/1930279714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa84010c140 msgr2=0x7fa84010e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.533+0000 7fa8456f1640 1 --2- 172.21.15.98:0/1930279714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa84010c140 0x7fa84010e5c0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fa838009900 tx=0x7fa83802a480 comp rx=0 tx=0).stop 2024-09-19T08:13:54.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.533+0000 7fa8456f1640 1 -- 172.21.15.98:0/1930279714 shutdown_connections 2024-09-19T08:13:54.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.533+0000 7fa8456f1640 1 --2- 172.21.15.98:0/1930279714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa84010c140 0x7fa84010e5c0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.533+0000 7fa8456f1640 1 --2- 172.21.15.98:0/1930279714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa840103690 0x7fa84010bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.533+0000 7fa8456f1640 1 --2- 172.21.15.98:0/1930279714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa840102d70 0x7fa840103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.533+0000 7fa8456f1640 1 -- 172.21.15.98:0/1930279714 >> 172.21.15.98:0/1930279714 conn(0x7fa8400fc740 msgr2=0x7fa8400feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:54.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.533+0000 7fa8456f1640 1 -- 172.21.15.98:0/1930279714 shutdown_connections 2024-09-19T08:13:54.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.533+0000 7fa8456f1640 1 -- 172.21.15.98:0/1930279714 wait complete. 2024-09-19T08:13:54.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.533+0000 7fa8456f1640 1 Processor -- start 2024-09-19T08:13:54.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa8456f1640 1 -- start start 2024-09-19T08:13:54.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa8456f1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa840102d70 0x7fa8401009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:54.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa8456f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa840103690 0x7fa840100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:54.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa8456f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa84010c140 0x7fa840101440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:54.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa8456f1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa840111ee0 con 0x7fa840103690 2024-09-19T08:13:54.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa8456f1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa840111d60 con 0x7fa840102d70 2024-09-19T08:13:54.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa8456f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa840112060 con 0x7fa84010c140 2024-09-19T08:13:54.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa83e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa840103690 0x7fa840100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:54.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa83e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa840103690 0x7fa840100f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37896/0 (socket says 172.21.15.98:37896) 2024-09-19T08:13:54.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa83e7fc640 1 -- 172.21.15.98:0/3677127544 learned_addr learned my addr 172.21.15.98:0/3677127544 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:54.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa83effd640 1 --2- 172.21.15.98:0/3677127544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa840102d70 0x7fa8401009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:54.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa83f7fe640 1 --2- 172.21.15.98:0/3677127544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa84010c140 0x7fa840101440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:54.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa83e7fc640 1 -- 172.21.15.98:0/3677127544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa84010c140 msgr2=0x7fa840101440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa83e7fc640 1 --2- 172.21.15.98:0/3677127544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa84010c140 0x7fa840101440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa83e7fc640 1 -- 172.21.15.98:0/3677127544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa840102d70 msgr2=0x7fa8401009c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa83e7fc640 1 --2- 172.21.15.98:0/3677127544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa840102d70 0x7fa8401009c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa83e7fc640 1 -- 172.21.15.98:0/3677127544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8401a6890 con 0x7fa840103690 2024-09-19T08:13:54.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.534+0000 7fa83e7fc640 1 --2- 172.21.15.98:0/3677127544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa840103690 0x7fa840100f00 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fa83400c890 tx=0x7fa83400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:54.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.535+0000 7fa827fff640 1 -- 172.21.15.98:0/3677127544 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa834004220 con 0x7fa840103690 2024-09-19T08:13:54.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.535+0000 7fa827fff640 1 -- 172.21.15.98:0/3677127544 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8340043c0 con 0x7fa840103690 2024-09-19T08:13:54.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.535+0000 7fa8456f1640 1 -- 172.21.15.98:0/3677127544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8401a6b80 con 0x7fa840103690 2024-09-19T08:13:54.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.535+0000 7fa827fff640 1 -- 172.21.15.98:0/3677127544 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8340049e0 con 0x7fa840103690 2024-09-19T08:13:54.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.535+0000 7fa8456f1640 1 -- 172.21.15.98:0/3677127544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8401a70d0 con 0x7fa840103690 2024-09-19T08:13:54.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.536+0000 7fa8456f1640 1 -- 172.21.15.98:0/3677127544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa840103150 con 0x7fa840103690 2024-09-19T08:13:54.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.537+0000 7fa827fff640 1 -- 172.21.15.98:0/3677127544 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fa8340079b0 con 0x7fa840103690 2024-09-19T08:13:54.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.538+0000 7fa827fff640 1 --2- 172.21.15.98:0/3677127544 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa8280a33c0 0x7fa8280a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:54.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.538+0000 7fa83effd640 1 --2- 172.21.15.98:0/3677127544 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa8280a33c0 0x7fa8280a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:54.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.538+0000 7fa827fff640 1 -- 172.21.15.98:0/3677127544 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7fa8340cfd40 con 0x7fa840103690 2024-09-19T08:13:54.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.538+0000 7fa83effd640 1 --2- 172.21.15.98:0/3677127544 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa8280a33c0 0x7fa8280a5880 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa830007ec0 tx=0x7fa8300075b0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:54.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.542+0000 7fa827fff640 1 -- 172.21.15.98:0/3677127544 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fa834014030 con 0x7fa840103690 2024-09-19T08:13:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.711+0000 7fa8456f1640 1 -- 172.21.15.98:0/3677127544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa8401a7300 con 0x7fa840103690 2024-09-19T08:13:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.711+0000 7fa827fff640 1 -- 172.21.15.98:0/3677127544 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fa83409c720 con 0x7fa840103690 2024-09-19T08:13:54.711 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-09-19T08:13:54.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.713+0000 7fa8456f1640 1 -- 172.21.15.98:0/3677127544 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa8280a33c0 msgr2=0x7fa8280a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.713+0000 7fa8456f1640 1 --2- 172.21.15.98:0/3677127544 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa8280a33c0 0x7fa8280a5880 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa830007ec0 tx=0x7fa8300075b0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.713+0000 7fa8456f1640 1 -- 172.21.15.98:0/3677127544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa840103690 msgr2=0x7fa840100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.713+0000 7fa8456f1640 1 --2- 172.21.15.98:0/3677127544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa840103690 0x7fa840100f00 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fa83400c890 tx=0x7fa83400cd50 comp rx=0 tx=0).stop 2024-09-19T08:13:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.713+0000 7fa8456f1640 1 -- 172.21.15.98:0/3677127544 shutdown_connections 2024-09-19T08:13:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.714+0000 7fa8456f1640 1 --2- 172.21.15.98:0/3677127544 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa8280a33c0 0x7fa8280a5880 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.714+0000 7fa8456f1640 1 --2- 172.21.15.98:0/3677127544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa84010c140 0x7fa840101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.714+0000 7fa8456f1640 1 --2- 172.21.15.98:0/3677127544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa840103690 0x7fa840100f00 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.714+0000 7fa8456f1640 1 --2- 172.21.15.98:0/3677127544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa840102d70 0x7fa8401009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.714+0000 7fa8456f1640 1 -- 172.21.15.98:0/3677127544 >> 172.21.15.98:0/3677127544 conn(0x7fa8400fc740 msgr2=0x7fa8400fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.714+0000 7fa8456f1640 1 -- 172.21.15.98:0/3677127544 shutdown_connections 2024-09-19T08:13:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.714+0000 7fa8456f1640 1 -- 172.21.15.98:0/3677127544 wait complete. 2024-09-19T08:13:54.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-19T08:13:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.827+0000 7fb15782e640 1 -- 172.21.15.98:0/377795456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb150103600 msgr2=0x7fb15010bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.827+0000 7fb15782e640 1 --2- 172.21.15.98:0/377795456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb150103600 0x7fb15010bae0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fb140009900 tx=0x7fb14002a480 comp rx=0 tx=0).stop 2024-09-19T08:13:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.827+0000 7fb15782e640 1 -- 172.21.15.98:0/377795456 shutdown_connections 2024-09-19T08:13:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.827+0000 7fb15782e640 1 --2- 172.21.15.98:0/377795456 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb15010c0b0 0x7fb15010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.827+0000 7fb15782e640 1 --2- 172.21.15.98:0/377795456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb150103600 0x7fb15010bae0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.827+0000 7fb15782e640 1 --2- 172.21.15.98:0/377795456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb150102ce0 0x7fb1501030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.827+0000 7fb15782e640 1 -- 172.21.15.98:0/377795456 >> 172.21.15.98:0/377795456 conn(0x7fb1500fc690 msgr2=0x7fb1500feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.827+0000 7fb15782e640 1 -- 172.21.15.98:0/377795456 shutdown_connections 2024-09-19T08:13:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.827+0000 7fb15782e640 1 -- 172.21.15.98:0/377795456 wait complete. 2024-09-19T08:13:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb15782e640 1 Processor -- start 2024-09-19T08:13:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb15782e640 1 -- start start 2024-09-19T08:13:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb15782e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb150102ce0 0x7fb1501984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb15782e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb150103600 0x7fb1501989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb15782e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb15010c0b0 0x7fb15019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb15782e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb150111ef0 con 0x7fb150103600 2024-09-19T08:13:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb15782e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb150111d70 con 0x7fb150102ce0 2024-09-19T08:13:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb15782e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb150112070 con 0x7fb15010c0b0 2024-09-19T08:13:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb154dc9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb150103600 0x7fb1501989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb154dc9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb150103600 0x7fb1501989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37902/0 (socket says 172.21.15.98:37902) 2024-09-19T08:13:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb154dc9640 1 -- 172.21.15.98:0/3744113815 learned_addr learned my addr 172.21.15.98:0/3744113815 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb155dcb640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb15010c0b0 0x7fb15019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb1555ca640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb150102ce0 0x7fb1501984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb154dc9640 1 -- 172.21.15.98:0/3744113815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb15010c0b0 msgr2=0x7fb15019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb154dc9640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb15010c0b0 0x7fb15019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb154dc9640 1 -- 172.21.15.98:0/3744113815 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb150102ce0 msgr2=0x7fb1501984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb154dc9640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb150102ce0 0x7fb1501984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb154dc9640 1 -- 172.21.15.98:0/3744113815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb15019d490 con 0x7fb150103600 2024-09-19T08:13:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb1555ca640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb150102ce0 0x7fb1501984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:13:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb154dc9640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb150103600 0x7fb1501989e0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fb140006010 tx=0x7fb1400042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.828+0000 7fb1467fc640 1 -- 172.21.15.98:0/3744113815 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb14003f030 con 0x7fb150103600 2024-09-19T08:13:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.829+0000 7fb15782e640 1 -- 172.21.15.98:0/3744113815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb15019d720 con 0x7fb150103600 2024-09-19T08:13:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.829+0000 7fb1467fc640 1 -- 172.21.15.98:0/3744113815 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb140003ce0 con 0x7fb150103600 2024-09-19T08:13:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.829+0000 7fb1467fc640 1 -- 172.21.15.98:0/3744113815 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb140035070 con 0x7fb150103600 2024-09-19T08:13:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.829+0000 7fb15782e640 1 -- 172.21.15.98:0/3744113815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1501a5010 con 0x7fb150103600 2024-09-19T08:13:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.830+0000 7fb15782e640 1 -- 172.21.15.98:0/3744113815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1501030c0 con 0x7fb150103600 2024-09-19T08:13:54.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.831+0000 7fb1467fc640 1 -- 172.21.15.98:0/3744113815 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fb140042030 con 0x7fb150103600 2024-09-19T08:13:54.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.831+0000 7fb1467fc640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb13c0a33c0 0x7fb13c0a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:54.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.831+0000 7fb1555ca640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb13c0a33c0 0x7fb13c0a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:54.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.831+0000 7fb1467fc640 1 -- 172.21.15.98:0/3744113815 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7fb1400f0540 con 0x7fb150103600 2024-09-19T08:13:54.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.832+0000 7fb1555ca640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb13c0a33c0 0x7fb13c0a5880 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb14c006fd0 tx=0x7fb14c008040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:54.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:54.835+0000 7fb1467fc640 1 -- 172.21.15.98:0/3744113815 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb1400bcfe0 con 0x7fb150103600 2024-09-19T08:13:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.002+0000 7fb15782e640 1 -- 172.21.15.98:0/3744113815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb150107130 con 0x7fb150103600 2024-09-19T08:13:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.003+0000 7fb1467fc640 1 -- 172.21.15.98:0/3744113815 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fb1400bc8e0 con 0x7fb150103600 2024-09-19T08:13:55.003 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-09-19T08:13:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.005+0000 7fb15782e640 1 -- 172.21.15.98:0/3744113815 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb13c0a33c0 msgr2=0x7fb13c0a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.005+0000 7fb15782e640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb13c0a33c0 0x7fb13c0a5880 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb14c006fd0 tx=0x7fb14c008040 comp rx=0 tx=0).stop 2024-09-19T08:13:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.005+0000 7fb15782e640 1 -- 172.21.15.98:0/3744113815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb150103600 msgr2=0x7fb1501989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.005+0000 7fb15782e640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb150103600 0x7fb1501989e0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fb140006010 tx=0x7fb1400042f0 comp rx=0 tx=0).stop 2024-09-19T08:13:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.005+0000 7fb15782e640 1 -- 172.21.15.98:0/3744113815 shutdown_connections 2024-09-19T08:13:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.005+0000 7fb15782e640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb13c0a33c0 0x7fb13c0a5880 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.005+0000 7fb15782e640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb15010c0b0 0x7fb15019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.005+0000 7fb15782e640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb150103600 0x7fb1501989e0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.005+0000 7fb15782e640 1 --2- 172.21.15.98:0/3744113815 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb150102ce0 0x7fb1501984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.005+0000 7fb15782e640 1 -- 172.21.15.98:0/3744113815 >> 172.21.15.98:0/3744113815 conn(0x7fb1500fc690 msgr2=0x7fb1500fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.005+0000 7fb15782e640 1 -- 172.21.15.98:0/3744113815 shutdown_connections 2024-09-19T08:13:55.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.005+0000 7fb15782e640 1 -- 172.21.15.98:0/3744113815 wait complete. 2024-09-19T08:13:55.015 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:54 smithi098 ceph-mon[31333]: pgmap v337: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:13:55.015 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:54 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3355813384' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:13:55.015 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:54 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3210257826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-19T08:13:55.015 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:54 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3677127544' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-19T08:13:55.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-09-19T08:13:55.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.116+0000 7f42f51cf640 1 -- 172.21.15.98:0/3048969790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42f01003e0 msgr2=0x7f42f01007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:55.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.116+0000 7f42f51cf640 1 --2- 172.21.15.98:0/3048969790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42f01003e0 0x7f42f01007c0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f42e0009900 tx=0x7f42e002a410 comp rx=0 tx=0).stop 2024-09-19T08:13:55.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.116+0000 7f42f51cf640 1 -- 172.21.15.98:0/3048969790 shutdown_connections 2024-09-19T08:13:55.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.116+0000 7f42f51cf640 1 --2- 172.21.15.98:0/3048969790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42f01054e0 0x7f42f010d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:55.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.116+0000 7f42f51cf640 1 --2- 172.21.15.98:0/3048969790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f42f0100d90 0x7f42f0104d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:55.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.116+0000 7f42f51cf640 1 --2- 172.21.15.98:0/3048969790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42f01003e0 0x7f42f01007c0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:55.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.116+0000 7f42f51cf640 1 -- 172.21.15.98:0/3048969790 >> 172.21.15.98:0/3048969790 conn(0x7f42f00771a0 msgr2=0x7f42f00fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:55.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.116+0000 7f42f51cf640 1 -- 172.21.15.98:0/3048969790 shutdown_connections 2024-09-19T08:13:55.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.116+0000 7f42f51cf640 1 -- 172.21.15.98:0/3048969790 wait complete. 2024-09-19T08:13:55.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.117+0000 7f42f51cf640 1 Processor -- start 2024-09-19T08:13:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.117+0000 7f42f51cf640 1 -- start start 2024-09-19T08:13:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.117+0000 7f42f51cf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42f01003e0 0x7f42f0198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.117+0000 7f42f51cf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42f01054e0 0x7f42f01989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.117+0000 7f42f51cf640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f42f019cd20 0x7f42f019d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:55.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.117+0000 7f42f51cf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42f010fcc0 con 0x7f42f01003e0 2024-09-19T08:13:55.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.117+0000 7f42f51cf640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f42f010fb40 con 0x7f42f019cd20 2024-09-19T08:13:55.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.117+0000 7f42f51cf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f42f010fe40 con 0x7f42f01054e0 2024-09-19T08:13:55.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42eed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42f01003e0 0x7f42f0198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:55.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42eed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42f01003e0 0x7f42f0198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37914/0 (socket says 172.21.15.98:37914) 2024-09-19T08:13:55.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42eed9d640 1 -- 172.21.15.98:0/2152295873 learned_addr learned my addr 172.21.15.98:0/2152295873 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:13:55.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42e659c640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42f01054e0 0x7f42f01989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:55.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42eed9d640 1 -- 172.21.15.98:0/2152295873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42f01054e0 msgr2=0x7f42f01989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:55.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42eed9d640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42f01054e0 0x7f42f01989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:55.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42ef59e640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f42f019cd20 0x7f42f019d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:55.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42eed9d640 1 -- 172.21.15.98:0/2152295873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f42f019cd20 msgr2=0x7f42f019d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:55.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42eed9d640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f42f019cd20 0x7f42f019d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:55.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42eed9d640 1 -- 172.21.15.98:0/2152295873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f019d810 con 0x7f42f01003e0 2024-09-19T08:13:55.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42e659c640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42f01054e0 0x7f42f01989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:13:55.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42eed9d640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42f01003e0 0x7f42f0198460 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f42e0006010 tx=0x7f42e00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:55.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42ecd99640 1 -- 172.21.15.98:0/2152295873 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e003f030 con 0x7f42f01003e0 2024-09-19T08:13:55.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42ecd99640 1 -- 172.21.15.98:0/2152295873 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42e0003ce0 con 0x7f42f01003e0 2024-09-19T08:13:55.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42f51cf640 1 -- 172.21.15.98:0/2152295873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f019daa0 con 0x7f42f01003e0 2024-09-19T08:13:55.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42ecd99640 1 -- 172.21.15.98:0/2152295873 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e0035070 con 0x7f42f01003e0 2024-09-19T08:13:55.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.118+0000 7f42f51cf640 1 -- 172.21.15.98:0/2152295873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f01a5260 con 0x7f42f01003e0 2024-09-19T08:13:55.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.119+0000 7f42f51cf640 1 -- 172.21.15.98:0/2152295873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42f01007c0 con 0x7f42f01003e0 2024-09-19T08:13:55.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.121+0000 7f42ecd99640 1 -- 172.21.15.98:0/2152295873 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f42e0042030 con 0x7f42f01003e0 2024-09-19T08:13:55.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.121+0000 7f42ecd99640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f42dc0a3440 0x7f42dc0a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:13:55.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.121+0000 7f42e659c640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f42dc0a3440 0x7f42dc0a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:13:55.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.121+0000 7f42ecd99640 1 -- 172.21.15.98:0/2152295873 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8521+0+0 (secure 0 0 0) 0x7f42e00f05b0 con 0x7f42f01003e0 2024-09-19T08:13:55.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.122+0000 7f42e659c640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f42dc0a3440 0x7f42dc0a5900 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f42f00ff710 tx=0x7f42d800b6f0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:13:55.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.124+0000 7f42ecd99640 1 -- 172.21.15.98:0/2152295873 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f42e00bd050 con 0x7f42f01003e0 2024-09-19T08:13:55.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:54 smithi139 ceph-mon[34874]: pgmap v337: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:13:55.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:54 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3355813384' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:13:55.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:54 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3210257826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-19T08:13:55.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:54 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3677127544' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-19T08:13:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.338+0000 7f42f51cf640 1 -- 172.21.15.98:0/2152295873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f42f0100d90 con 0x7f42f01003e0 2024-09-19T08:13:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.345+0000 7f42ecd99640 1 -- 172.21.15.98:0/2152295873 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v24) ==== 81+0+742873 (secure 0 0 0) 0x7f42e00bc950 con 0x7f42f01003e0 2024-09-19T08:13:55.354 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:13:55.354 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:13:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 24, 2024-09-19T08:13:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "active_gid": 15252, 2024-09-19T08:13:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "active_name": "x", 2024-09-19T08:13:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "active_addrs": { 2024-09-19T08:13:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-19T08:13:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-19T08:13:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6800", 2024-09-19T08:13:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1961446727 2024-09-19T08:13:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-19T08:13:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6801", 2024-09-19T08:13:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1961446727 2024-09-19T08:13:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-19T08:13:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "active_addr": "172.21.15.98:6801/1961446727", 2024-09-19T08:13:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "active_change": "2024-09-19T08:03:00.753479+0000", 2024-09-19T08:13:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "active_mgr_features": 4540701547738038271, 2024-09-19T08:13:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "available": true, 2024-09-19T08:13:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "standbys": [ 2024-09-19T08:13:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 24754, 2024-09-19T08:13:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "z", 2024-09-19T08:13:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_features": 4540701547738038271, 2024-09-19T08:13:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-09-19T08:13:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-09-19T08:13:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-09-19T08:13:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-09-19T08:13:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-09-19T08:13:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-09-19T08:13:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-09-19T08:13:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-09-19T08:13:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-09-19T08:13:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-09-19T08:13:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-09-19T08:13:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-09-19T08:13:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-09-19T08:13:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-09-19T08:13:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-09-19T08:13:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-09-19T08:13:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-09-19T08:13:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-09-19T08:13:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-09-19T08:13:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-09-19T08:13:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-09-19T08:13:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-09-19T08:13:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-09-19T08:13:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-09-19T08:13:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-09-19T08:13:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-19T08:13:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-09-19T08:13:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-09-19T08:13:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-09-19T08:13:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-09-19T08:13:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-09-19T08:13:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-09-19T08:13:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-09-19T08:13:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-09-19T08:13:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-09-19T08:13:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-09-19T08:13:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-09-19T08:13:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-19T08:13:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-09-19T08:13:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-09-19T08:13:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-09-19T08:13:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-09-19T08:13:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-19T08:13:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-19T08:13:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-09-19T08:13:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-09-19T08:13:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-09-19T08:13:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-09-19T08:13:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-09-19T08:13:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-19T08:13:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-09-19T08:13:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-09-19T08:13:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-09-19T08:13:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-19T08:13:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-19T08:13:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-09-19T08:13:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-09-19T08:13:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-09-19T08:13:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-09-19T08:13:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-09-19T08:13:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-09-19T08:13:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-19T08:13:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-09-19T08:13:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-09-19T08:13:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-09-19T08:13:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-09-19T08:13:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-19T08:13:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-09-19T08:13:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-09-19T08:13:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-09-19T08:13:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-09-19T08:13:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-19T08:13:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-19T08:13:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-09-19T08:13:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-09-19T08:13:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-19T08:13:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-09-19T08:13:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-09-19T08:13:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-09-19T08:13:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-09-19T08:13:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-09-19T08:13:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-09-19T08:13:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-09-19T08:13:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-09-19T08:13:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-09-19T08:13:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-09-19T08:13:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-09-19T08:13:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-19T08:13:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-19T08:13:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-09-19T08:13:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-09-19T08:13:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-09-19T08:13:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-19T08:13:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-19T08:13:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-09-19T08:13:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-09-19T08:13:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-19T08:13:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-09-19T08:13:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-09-19T08:13:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-09-19T08:13:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-09-19T08:13:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-19T08:13:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-09-19T08:13:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-09-19T08:13:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-09-19T08:13:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-19T08:13:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-09-19T08:13:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-09-19T08:13:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-09-19T08:13:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-19T08:13:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-09-19T08:13:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-09-19T08:13:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-19T08:13:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-19T08:13:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-09-19T08:13:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-09-19T08:13:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-19T08:13:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-09-19T08:13:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-09-19T08:13:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-09-19T08:13:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-19T08:13:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-09-19T08:13:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-09-19T08:13:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-09-19T08:13:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-09-19T08:13:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-09-19T08:13:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-19T08:13:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-09-19T08:13:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-09-19T08:13:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-19T08:13:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-09-19T08:13:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-09-19T08:13:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-19T08:13:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-09-19T08:13:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-09-19T08:13:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-19T08:13:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-09-19T08:13:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-09-19T08:13:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-19T08:13:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-09-19T08:13:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-09-19T08:13:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-09-19T08:13:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-19T08:13:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-09-19T08:13:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-09-19T08:13:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-09-19T08:13:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-19T08:13:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-09-19T08:13:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-09-19T08:13:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-09-19T08:13:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-19T08:13:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-09-19T08:13:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-09-19T08:13:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-09-19T08:13:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-19T08:13:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-09-19T08:13:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-09-19T08:13:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-09-19T08:13:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-19T08:13:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-09-19T08:13:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-09-19T08:13:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-09-19T08:13:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-19T08:13:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-09-19T08:13:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-09-19T08:13:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-09-19T08:13:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-19T08:13:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-09-19T08:13:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-09-19T08:13:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-09-19T08:13:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-19T08:13:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-09-19T08:13:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-09-19T08:13:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-09-19T08:13:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-19T08:13:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-09-19T08:13:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-09-19T08:13:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-09-19T08:13:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-19T08:13:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-09-19T08:13:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-09-19T08:13:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-09-19T08:13:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-19T08:13:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-09-19T08:13:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-09-19T08:13:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-09-19T08:13:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-19T08:13:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-09-19T08:13:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-09-19T08:13:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-09-19T08:13:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-19T08:13:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-09-19T08:13:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_samba": { 2024-09-19T08:13:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_samba", 2024-09-19T08:13:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-19T08:13:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Samba/SMB container image", 2024-09-19T08:13:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-09-19T08:13:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-09-19T08:13:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-19T08:13:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-09-19T08:13:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-09-19T08:13:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-09-19T08:13:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-09-19T08:13:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-09-19T08:13:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-09-19T08:13:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-19T08:13:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-09-19T08:13:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-09-19T08:13:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-09-19T08:13:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-19T08:13:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-09-19T08:13:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-09-19T08:13:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-09-19T08:13:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-19T08:13:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-09-19T08:13:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-09-19T08:13:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-09-19T08:13:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-09-19T08:13:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-09-19T08:13:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-09-19T08:13:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-19T08:13:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-09-19T08:13:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-09-19T08:13:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-09-19T08:13:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "grafana_dashboards_path": { 2024-09-19T08:13:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "grafana_dashboards_path", 2024-09-19T08:13:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-19T08:13:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-19T08:13:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-09-19T08:13:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-09-19T08:13:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-09-19T08:13:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-09-19T08:13:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-09-19T08:13:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-19T08:13:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-09-19T08:13:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-09-19T08:13:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-19T08:13:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-09-19T08:13:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-09-19T08:13:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-19T08:13:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-19T08:13:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-19T08:13:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-19T08:13:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-19T08:13:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-19T08:13:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-19T08:13:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-09-19T08:13:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-19T08:13:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-09-19T08:13:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-09-19T08:13:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-09-19T08:13:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-09-19T08:13:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-09-19T08:13:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-19T08:13:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-09-19T08:13:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-09-19T08:13:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-09-19T08:13:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-09-19T08:13:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-09-19T08:13:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-09-19T08:13:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-09-19T08:13:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-09-19T08:13:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-09-19T08:13:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-09-19T08:13:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-09-19T08:13:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-09-19T08:13:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-19T08:13:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-09-19T08:13:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-09-19T08:13:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-19T08:13:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-19T08:13:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-09-19T08:13:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-09-19T08:13:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-19T08:13:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-09-19T08:13:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-09-19T08:13:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-19T08:13:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-09-19T08:13:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-09-19T08:13:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-19T08:13:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-09-19T08:13:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-09-19T08:13:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-19T08:13:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-09-19T08:13:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-09-19T08:13:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-19T08:13:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-09-19T08:13:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-09-19T08:13:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-09-19T08:13:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-09-19T08:13:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-09-19T08:13:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-09-19T08:13:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-19T08:13:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-09-19T08:13:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-09-19T08:13:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-19T08:13:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-09-19T08:13:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-09-19T08:13:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-19T08:13:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-09-19T08:13:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-09-19T08:13:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-09-19T08:13:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-09-19T08:13:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-09-19T08:13:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-19T08:13:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.462 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-09-19T08:13:55.462 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-09-19T08:13:55.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.462 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.462 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.462 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.462 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.462 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.462 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.462 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-19T08:13:55.462 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.462 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.462 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-09-19T08:13:55.463 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.463 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.463 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.463 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.463 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.463 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.464 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.464 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.464 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.464 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.464 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.464 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.464 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.464 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.464 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.464 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.464 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.464 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.464 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.465 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.465 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.465 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.465 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.465 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.465 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.465 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.465 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.466 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.466 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.466 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.466 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.466 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.466 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.466 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.466 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.466 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.467 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.467 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.467 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.467 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.467 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.467 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.467 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.467 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.467 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.467 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.467 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.467 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.467 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.468 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.468 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.468 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.468 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.468 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.468 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.468 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.468 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.468 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.468 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.469 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-09-19T08:13:55.469 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-09-19T08:13:55.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.469 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.469 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.469 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-09-19T08:13:55.469 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.469 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.469 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.469 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-19T08:13:55.469 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.469 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.470 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.470 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-09-19T08:13:55.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-09-19T08:13:55.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.470 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.470 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.470 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-09-19T08:13:55.470 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.470 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.470 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.470 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-19T08:13:55.470 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.471 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.471 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-09-19T08:13:55.471 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.471 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.471 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.471 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-19T08:13:55.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-19T08:13:55.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.471 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.472 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.472 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:55.472 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.472 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.472 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.472 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.472 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.472 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.472 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.472 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-09-19T08:13:55.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-19T08:13:55.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.473 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.473 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.473 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.473 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.473 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.473 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.473 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.473 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.473 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.473 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.473 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-19T08:13:55.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-19T08:13:55.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.474 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.474 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.474 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.474 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.474 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.474 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.474 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.474 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.474 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.474 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.474 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-09-19T08:13:55.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-09-19T08:13:55.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.475 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.475 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.475 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.475 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.475 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.475 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.475 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.475 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.475 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.475 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.476 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-19T08:13:55.476 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-19T08:13:55.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.476 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.476 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.476 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.476 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.476 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.476 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.476 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.476 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.476 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.476 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.477 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-09-19T08:13:55.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-19T08:13:55.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.477 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.477 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.477 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.477 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.477 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.477 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.477 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.477 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.478 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.478 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.478 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-19T08:13:55.478 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-19T08:13:55.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.478 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.478 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.478 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.478 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.478 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.478 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.479 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.479 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.479 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-19T08:13:55.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-19T08:13:55.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.479 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.479 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.479 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.479 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.479 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.479 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.480 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.480 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.480 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.480 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.480 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-19T08:13:55.480 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-19T08:13:55.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.480 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.480 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.480 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.480 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.481 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.481 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.481 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.481 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.481 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.481 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.481 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-19T08:13:55.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-19T08:13:55.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.481 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.481 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.481 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.482 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.482 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.482 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.482 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.482 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.482 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.482 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.482 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-19T08:13:55.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-19T08:13:55.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.482 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.482 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.483 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.483 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.483 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.483 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.483 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.483 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.483 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.483 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.483 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-19T08:13:55.483 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-19T08:13:55.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.483 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.484 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.484 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.484 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.484 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.484 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.484 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.484 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.484 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.484 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.484 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-19T08:13:55.484 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-19T08:13:55.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.485 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.485 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.485 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.485 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.485 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.485 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.485 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.485 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.485 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.485 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.485 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-19T08:13:55.486 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-19T08:13:55.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.486 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.486 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.486 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.486 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.486 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.486 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.486 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.486 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.486 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.487 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-09-19T08:13:55.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-19T08:13:55.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.487 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.487 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.487 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-09-19T08:13:55.487 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.487 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.487 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.487 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.487 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.487 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.487 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.488 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-19T08:13:55.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-19T08:13:55.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.488 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.488 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.488 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.488 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.488 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.488 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.488 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.488 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.488 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.489 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.489 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-09-19T08:13:55.489 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-09-19T08:13:55.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.489 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.489 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.489 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.489 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.489 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.489 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.489 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.490 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.490 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.490 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-09-19T08:13:55.490 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-19T08:13:55.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.490 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.490 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.490 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-09-19T08:13:55.490 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.490 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.490 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.490 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-19T08:13:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-19T08:13:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.492 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.492 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.492 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.492 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-19T08:13:55.492 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-19T08:13:55.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.492 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.492 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.492 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.492 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.492 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.493 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.493 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.493 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.493 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.493 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.493 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-19T08:13:55.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-19T08:13:55.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.493 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.493 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.493 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.493 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.494 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.494 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.494 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.494 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.494 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.494 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.494 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-19T08:13:55.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-19T08:13:55.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.494 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.494 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.495 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.495 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.495 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.495 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.495 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.495 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.495 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.495 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-09-19T08:13:55.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-19T08:13:55.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.495 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.496 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.496 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.496 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.496 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.496 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.496 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.496 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.496 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.496 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.496 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-19T08:13:55.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-19T08:13:55.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.497 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.497 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.497 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.497 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.497 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.497 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.497 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.497 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.497 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.497 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.497 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-19T08:13:55.497 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-19T08:13:55.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.498 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.498 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.498 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.498 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.498 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.498 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.498 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.498 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.498 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.498 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.498 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-19T08:13:55.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-19T08:13:55.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.499 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.499 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.499 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.499 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.499 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.499 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.499 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.499 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.500 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-19T08:13:55.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-19T08:13:55.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.500 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.500 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.500 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.500 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.500 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.500 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.500 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.500 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.500 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.500 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.501 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-19T08:13:55.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-19T08:13:55.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.501 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.501 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.501 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.501 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.501 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.501 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.501 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.501 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.501 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.501 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.502 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-19T08:13:55.502 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-19T08:13:55.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.502 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.502 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.502 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.502 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.502 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.502 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.502 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.502 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.503 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.503 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-19T08:13:55.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-19T08:13:55.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.503 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.503 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.503 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.503 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.503 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.504 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.504 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.504 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.504 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.504 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.504 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-19T08:13:55.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-19T08:13:55.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.504 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.504 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-09-19T08:13:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-19T08:13:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.506 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.506 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.506 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.506 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.506 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.506 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.506 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.506 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-19T08:13:55.506 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-19T08:13:55.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.507 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.507 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.507 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-19T08:13:55.507 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.507 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.507 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.507 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.507 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.507 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.507 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.507 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-19T08:13:55.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-19T08:13:55.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.508 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.508 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.508 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:55.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.508 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.508 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.508 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.508 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.508 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.508 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.508 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-19T08:13:55.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-19T08:13:55.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.509 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.509 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-09-19T08:13:55.509 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.509 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.509 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.509 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.509 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.509 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.510 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-19T08:13:55.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-19T08:13:55.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.510 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.510 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.510 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-09-19T08:13:55.510 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.510 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.510 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.510 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.510 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.510 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.510 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.511 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-09-19T08:13:55.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-19T08:13:55.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.511 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.511 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.511 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.511 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.511 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.511 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.511 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.511 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.511 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.511 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.512 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-19T08:13:55.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-19T08:13:55.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.512 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.512 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.512 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-09-19T08:13:55.512 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.512 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.512 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.512 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.512 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.512 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.512 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.513 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-09-19T08:13:55.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-19T08:13:55.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.513 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.513 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.513 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.513 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.513 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.513 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.513 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.513 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.513 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.514 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.514 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-09-19T08:13:55.514 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-19T08:13:55.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.514 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.514 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.514 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.514 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.514 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.514 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.514 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.514 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.515 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.515 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.515 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-19T08:13:55.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-19T08:13:55.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.515 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.515 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.515 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:55.515 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.515 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.515 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.515 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.516 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.516 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.516 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.516 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-19T08:13:55.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-19T08:13:55.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.516 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.516 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.516 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:55.516 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.516 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.516 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.516 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.517 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.517 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.517 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.517 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-19T08:13:55.517 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-19T08:13:55.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.517 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.517 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:55.517 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.517 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.517 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.517 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.518 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.518 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.518 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.518 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-09-19T08:13:55.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-09-19T08:13:55.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.518 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.518 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.518 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.518 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.518 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.518 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.518 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.519 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.519 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.519 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.519 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-09-19T08:13:55.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-09-19T08:13:55.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.519 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.519 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.519 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.519 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.519 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.519 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.520 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.520 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.520 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.520 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.520 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-09-19T08:13:55.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-09-19T08:13:55.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.520 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.520 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.520 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.521 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.521 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.521 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-09-19T08:13:55.521 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.521 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.521 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.521 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-09-19T08:13:55.521 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-09-19T08:13:55.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.521 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.521 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.521 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-09-19T08:13:55.522 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.522 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.522 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.522 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.522 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.522 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.522 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.522 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-09-19T08:13:55.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-09-19T08:13:55.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.522 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.522 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.522 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.523 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.523 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.523 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.523 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.523 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.523 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.523 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.523 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.523 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.523 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.524 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.524 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.524 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.524 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.524 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.524 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.524 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.524 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.524 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.524 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.524 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.524 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.524 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.524 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.525 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.525 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.525 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.525 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.525 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.525 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.525 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.525 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.525 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.525 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.526 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.526 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.526 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.526 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.526 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.526 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.526 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.526 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.526 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.526 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.526 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.526 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.527 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.527 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.527 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.527 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.527 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.527 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.527 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.527 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.527 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.527 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.527 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.528 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.528 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.528 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.528 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.528 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.528 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.528 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.528 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.528 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.528 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-09-19T08:13:55.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-09-19T08:13:55.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.529 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.529 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.529 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.529 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.529 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.529 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.529 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-09-19T08:13:55.529 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.529 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.529 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.529 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-09-19T08:13:55.529 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-09-19T08:13:55.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.530 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.530 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.530 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.530 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.530 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.530 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.530 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.530 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.530 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.530 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.530 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-09-19T08:13:55.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-09-19T08:13:55.530 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.531 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.531 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.531 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-09-19T08:13:55.531 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.531 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.531 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.531 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.531 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.531 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.531 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.531 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-09-19T08:13:55.531 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-09-19T08:13:55.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.532 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.532 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-09-19T08:13:55.532 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.532 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.532 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.532 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.532 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.532 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.532 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.533 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-09-19T08:13:55.533 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-09-19T08:13:55.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.533 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.533 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.533 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.533 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.533 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.533 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.533 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.533 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.533 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.533 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.534 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-09-19T08:13:55.534 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-09-19T08:13:55.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.534 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.534 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.534 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-09-19T08:13:55.534 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.534 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.534 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.534 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.534 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.535 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.535 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.535 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-09-19T08:13:55.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-09-19T08:13:55.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.535 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.535 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.535 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-09-19T08:13:55.535 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.535 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.535 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.535 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.536 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-09-19T08:13:55.536 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.536 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.536 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.536 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.536 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.536 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-09-19T08:13:55.536 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-09-19T08:13:55.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.536 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.536 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.537 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-09-19T08:13:55.537 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-09-19T08:13:55.537 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-09-19T08:13:55.537 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.537 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.537 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.537 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.537 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.537 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-09-19T08:13:55.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-09-19T08:13:55.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.538 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.538 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.538 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.538 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.538 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.538 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.538 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.538 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.538 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-09-19T08:13:55.539 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.539 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.539 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.539 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-09-19T08:13:55.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-09-19T08:13:55.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.539 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.539 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.539 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.540 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.540 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.540 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.540 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-09-19T08:13:55.540 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.540 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.540 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.540 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.540 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.541 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.541 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.541 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.541 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.541 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.541 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.541 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.541 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.541 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.541 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.541 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.541 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.541 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.542 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.542 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.542 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.542 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.542 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.542 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.542 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.542 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.542 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.542 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.542 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.543 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.543 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.543 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.543 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.543 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.543 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.543 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.543 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.543 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.543 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.543 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.544 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.544 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.544 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.544 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.544 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.544 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.544 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.544 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.544 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.544 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.544 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.544 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-09-19T08:13:55.546 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-09-19T08:13:55.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.546 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.546 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.546 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-09-19T08:13:55.546 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.546 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.546 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.546 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-19T08:13:55.546 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.546 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.546 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.546 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-09-19T08:13:55.547 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-09-19T08:13:55.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.547 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.547 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.547 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-09-19T08:13:55.547 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.547 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.547 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.547 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-19T08:13:55.547 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.547 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.547 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.547 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-09-19T08:13:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-09-19T08:13:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-09-19T08:13:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-09-19T08:13:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.549 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-09-19T08:13:55.549 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-09-19T08:13:55.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.549 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.549 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.549 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-09-19T08:13:55.549 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.549 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.549 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.549 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-19T08:13:55.549 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.549 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.550 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.550 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-09-19T08:13:55.550 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-09-19T08:13:55.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.550 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.550 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.550 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.550 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.550 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.550 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.550 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-19T08:13:55.550 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.550 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.551 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.551 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:55.551 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:55.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.551 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.551 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.551 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:55.551 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.551 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.551 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.551 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-09-19T08:13:55.551 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.551 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.552 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.552 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-09-19T08:13:55.552 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-09-19T08:13:55.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.552 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.552 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.552 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-09-19T08:13:55.552 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.552 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.552 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.552 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-19T08:13:55.552 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.552 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.553 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.553 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.553 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-09-19T08:13:55.553 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.553 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.553 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.553 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.553 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.554 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.555 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.555 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.555 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.555 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.555 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.555 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.555 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.555 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.555 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.555 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.555 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.556 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.556 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.556 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.556 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.556 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.556 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.556 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.556 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.556 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.556 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.556 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.556 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.556 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.557 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.557 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.557 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.557 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.557 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.557 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.557 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.557 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.557 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.557 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.557 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-09-19T08:13:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-09-19T08:13:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.559 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.559 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-09-19T08:13:55.559 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.559 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.559 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.559 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.559 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.559 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.559 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.559 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-09-19T08:13:55.559 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-09-19T08:13:55.559 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.559 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.560 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.560 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-09-19T08:13:55.560 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.560 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.560 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.560 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.560 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.560 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.560 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.560 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:55.561 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:55.561 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.561 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.561 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.561 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:55.561 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.561 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.561 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.561 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.561 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.561 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.561 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.561 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.562 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.562 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-09-19T08:13:55.562 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-09-19T08:13:55.562 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-09-19T08:13:55.562 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.562 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-09-19T08:13:55.562 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-09-19T08:13:55.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.562 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.562 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.562 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-09-19T08:13:55.562 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.563 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.563 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.563 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-19T08:13:55.563 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.563 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.563 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.563 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-09-19T08:13:55.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-09-19T08:13:55.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.563 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.563 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.564 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-09-19T08:13:55.564 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.564 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.564 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.564 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-19T08:13:55.564 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.564 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.564 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.564 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-09-19T08:13:55.564 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-09-19T08:13:55.564 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.565 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.565 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.565 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.565 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.565 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.565 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.565 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-09-19T08:13:55.565 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.565 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.565 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.565 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:55.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-09-19T08:13:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-09-19T08:13:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-19T08:13:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.567 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.567 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.567 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.567 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.567 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.567 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.567 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.567 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.567 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.567 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.567 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.567 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.568 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.568 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.568 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.568 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.568 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.568 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.568 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.568 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.568 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.568 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.569 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.569 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.569 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.569 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.569 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.569 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.569 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.569 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.569 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.569 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.569 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.569 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.570 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.570 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.570 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.570 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.570 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.570 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.570 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.570 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.570 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.570 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.570 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.570 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.570 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.570 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.571 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.571 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.571 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.571 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.571 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.571 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.571 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.571 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.571 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.571 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.571 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.571 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.572 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.572 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.572 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-09-19T08:13:55.572 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-09-19T08:13:55.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.572 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.572 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.572 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.572 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.572 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.572 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.572 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-09-19T08:13:55.573 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.573 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.573 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.573 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-09-19T08:13:55.573 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-09-19T08:13:55.573 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.573 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.573 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.573 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-09-19T08:13:55.573 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.573 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.573 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.574 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-09-19T08:13:55.574 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.574 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.574 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.574 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-09-19T08:13:55.574 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-09-19T08:13:55.574 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.574 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.574 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.574 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-09-19T08:13:55.574 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.574 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.575 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.575 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-19T08:13:55.575 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.575 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.575 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.575 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-09-19T08:13:55.575 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-09-19T08:13:55.575 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.575 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.575 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.575 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:55.575 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-09-19T08:13:55.575 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-09-19T08:13:55.576 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.576 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-19T08:13:55.576 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.576 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.576 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.576 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-09-19T08:13:55.576 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-09-19T08:13:55.576 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.576 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.576 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.576 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.576 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.577 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.577 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.577 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-09-19T08:13:55.577 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.577 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.577 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.577 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-09-19T08:13:55.577 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-09-19T08:13:55.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.577 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.577 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.577 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-09-19T08:13:55.578 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.578 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.578 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.578 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-19T08:13:55.578 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.578 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.578 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.578 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.578 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.578 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-09-19T08:13:55.578 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-09-19T08:13:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-09-19T08:13:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-09-19T08:13:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-09-19T08:13:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-09-19T08:13:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-19T08:13:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-09-19T08:13:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-09-19T08:13:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-09-19T08:13:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-19T08:13:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-09-19T08:13:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-09-19T08:13:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-09-19T08:13:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-09-19T08:13:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-09-19T08:13:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-19T08:13:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.601 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.601 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.601 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.601 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.601 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.601 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.601 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.601 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.601 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.601 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.601 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.601 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.601 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.601 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.602 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.602 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.602 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.602 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.602 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.602 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.602 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.602 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.602 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.602 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.602 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.602 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.603 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.603 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.603 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.603 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.603 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.603 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.603 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.603 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.603 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.603 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.603 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.604 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.604 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.604 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.604 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.604 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.604 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.604 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.604 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.604 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.604 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.604 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.605 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.605 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.605 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.605 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.605 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.605 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.605 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.605 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.605 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.605 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-09-19T08:13:55.605 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-09-19T08:13:55.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.606 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.606 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.606 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.606 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.606 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.606 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.606 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-09-19T08:13:55.606 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-19T08:13:55.606 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.606 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.606 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-09-19T08:13:55.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-09-19T08:13:55.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.607 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.607 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.607 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-09-19T08:13:55.607 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.607 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.607 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.607 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-09-19T08:13:55.607 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.607 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.607 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.607 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-09-19T08:13:55.608 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-09-19T08:13:55.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.608 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.608 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.608 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-09-19T08:13:55.608 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.608 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.608 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.608 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-09-19T08:13:55.608 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.608 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.608 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.609 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-09-19T08:13:55.609 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-09-19T08:13:55.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.609 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.609 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.609 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.609 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.609 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.609 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.609 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-09-19T08:13:55.609 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.609 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.609 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.610 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-09-19T08:13:55.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-09-19T08:13:55.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.610 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.610 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.610 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-09-19T08:13:55.610 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.610 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.610 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.610 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-19T08:13:55.610 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-19T08:13:55.610 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.610 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.611 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-09-19T08:13:55.611 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.611 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.611 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.611 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.611 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.611 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.611 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.612 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.612 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.612 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.612 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.612 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.612 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.612 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.612 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.612 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.612 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.612 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.612 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.612 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.613 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.613 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.613 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.613 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.613 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.613 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.613 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.613 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.613 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.613 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.614 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.614 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.614 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.614 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.614 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.614 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.614 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.614 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.614 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.614 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.614 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.614 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.615 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.615 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.615 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.615 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.615 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.615 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.615 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.615 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.615 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.615 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.615 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.615 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-09-19T08:13:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.619 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.619 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.619 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.619 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.619 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.619 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.619 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.619 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.619 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.619 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.619 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.619 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.620 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.620 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.620 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.620 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.620 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.620 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.620 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.620 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.620 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.620 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.620 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.620 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.620 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.620 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.621 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.621 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.621 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.621 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.621 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.621 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.621 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.621 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.621 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.621 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.622 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.622 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.622 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.622 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-09-19T08:13:55.622 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.622 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.622 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.622 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.623 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.623 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.623 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.623 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.623 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.623 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.623 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.623 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.623 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.623 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.623 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.623 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.624 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.624 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.624 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.624 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.624 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.624 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.624 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.624 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.624 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.624 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.625 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.625 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.625 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.625 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.625 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.625 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.625 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.625 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.625 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.625 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.626 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.626 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.626 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.626 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.626 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.626 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.626 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.626 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.626 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.626 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.626 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.626 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.627 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.627 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.627 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.627 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.627 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.627 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.627 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.627 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.627 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.627 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.627 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.628 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.628 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.628 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.628 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.628 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.628 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-09-19T08:13:55.628 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.628 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.628 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.629 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-09-19T08:13:55.629 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-09-19T08:13:55.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.629 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.629 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.629 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.629 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.629 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.629 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.629 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-19T08:13:55.629 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.629 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.629 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.630 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.630 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.630 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.630 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.630 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.630 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.630 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.630 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.630 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.630 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.631 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.631 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.631 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.631 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.631 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.631 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.631 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.631 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.631 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.631 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.632 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.632 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.632 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.632 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.632 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.632 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.632 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.632 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.632 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.632 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.632 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.633 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.633 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.633 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.633 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.633 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.633 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.633 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.633 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.633 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.633 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.633 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.633 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.634 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.634 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.634 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.634 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.634 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.634 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.634 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.634 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.634 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.634 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.634 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.635 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.635 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.635 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.635 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.635 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.635 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.635 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-09-19T08:13:55.635 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-09-19T08:13:55.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.635 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.635 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.635 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.636 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.636 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.636 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.636 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-09-19T08:13:55.636 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-09-19T08:13:55.636 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-09-19T08:13:55.636 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.636 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-09-19T08:13:55.636 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.636 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.636 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.637 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-09-19T08:13:55.637 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.637 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.637 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.637 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.637 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.637 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.638 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.638 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.638 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.638 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.638 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.638 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.638 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.638 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.638 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.638 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.638 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.638 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.639 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.639 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.639 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.639 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.639 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.639 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.639 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.639 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.639 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.639 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.639 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.640 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.640 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.640 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.640 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.640 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.640 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.640 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.640 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.640 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.640 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.640 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.640 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.641 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.641 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.641 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.641 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.641 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.641 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.641 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.641 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.641 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.641 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.641 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.641 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.642 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.642 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.642 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.642 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.642 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.642 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.642 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.642 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.642 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.642 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.642 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.642 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.643 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-09-19T08:13:55.643 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.643 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.643 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.643 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.643 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.643 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.643 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.643 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.644 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.644 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.644 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.644 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.644 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.644 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.644 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.644 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.644 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.644 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.644 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.644 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.644 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.645 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.645 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.645 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.645 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.645 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.645 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.645 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.645 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.645 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.645 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.645 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.646 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.646 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.646 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.646 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.646 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.646 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.646 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.646 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.646 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.646 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.647 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.647 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.647 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.647 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.647 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.647 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.647 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.647 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.647 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.647 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.647 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.647 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.647 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.648 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.648 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.648 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.648 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.648 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.648 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.648 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.648 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.648 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.648 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.648 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.649 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.649 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.649 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-09-19T08:13:55.649 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.649 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.649 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.649 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.650 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.650 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.650 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.650 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.650 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.650 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.650 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.650 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.650 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.650 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.650 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.651 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.651 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.651 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.651 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.651 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.651 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.651 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.651 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.651 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.651 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.651 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.652 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.652 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.652 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.652 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.652 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.652 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.652 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.652 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.652 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.652 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.652 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.653 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.653 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.653 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.653 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.653 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.653 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.653 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.653 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.653 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.653 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.653 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.653 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.654 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.654 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.654 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.654 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.654 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.654 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.654 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.654 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.654 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.654 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.654 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.655 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.655 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.655 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.655 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.655 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.655 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:55.655 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:55.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.655 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.655 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.656 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:55.656 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.656 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.656 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.656 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.656 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.656 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.656 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.656 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-09-19T08:13:55.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-09-19T08:13:55.657 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:55.657 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.657 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.657 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-09-19T08:13:55.657 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-09-19T08:13:55.657 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.657 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.657 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-09-19T08:13:55.657 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-19T08:13:55.657 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.657 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.657 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.658 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.658 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-09-19T08:13:55.658 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.658 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.658 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.658 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-09-19T08:13:55.658 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-09-19T08:13:55.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.658 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.658 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.658 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.659 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.659 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.659 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.659 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-19T08:13:55.659 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.659 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.659 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.659 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-09-19T08:13:55.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-09-19T08:13:55.659 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.659 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.659 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.660 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.660 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.660 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.660 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.660 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.660 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.660 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.660 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.660 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.660 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.660 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.661 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.661 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.661 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.661 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.661 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.661 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.661 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.661 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.661 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.661 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.661 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.661 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.661 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-09-19T08:13:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-09-19T08:13:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-09-19T08:13:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-09-19T08:13:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-09-19T08:13:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-09-19T08:13:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-19T08:13:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-19T08:13:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-09-19T08:13:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-09-19T08:13:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-19T08:13:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-19T08:13:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-09-19T08:13:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-09-19T08:13:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-09-19T08:13:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-09-19T08:13:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-09-19T08:13:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-09-19T08:13:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-09-19T08:13:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-19T08:13:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-09-19T08:13:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-09-19T08:13:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-09-19T08:13:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-19T08:13:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-09-19T08:13:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-09-19T08:13:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-09-19T08:13:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-09-19T08:13:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-09-19T08:13:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-09-19T08:13:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-09-19T08:13:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-09-19T08:13:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-09-19T08:13:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-09-19T08:13:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-09-19T08:13:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-09-19T08:13:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-09-19T08:13:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-09-19T08:13:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.692 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.692 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.692 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.692 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-09-19T08:13:55.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-09-19T08:13:55.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.692 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.692 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.692 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:55.692 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.692 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.693 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.693 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.693 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.693 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.693 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.693 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-09-19T08:13:55.693 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-09-19T08:13:55.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.693 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.693 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.693 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.694 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.694 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.694 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.694 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.694 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.694 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.694 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.694 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-09-19T08:13:55.694 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-09-19T08:13:55.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.694 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.694 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.695 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.695 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.695 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.695 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.695 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.695 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.695 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.695 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.695 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.695 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.695 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-09-19T08:13:55.696 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.696 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.696 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.696 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-09-19T08:13:55.696 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-09-19T08:13:55.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.696 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.696 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.696 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.696 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.696 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.696 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.697 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.697 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.697 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.697 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.697 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-09-19T08:13:55.697 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-09-19T08:13:55.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.697 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.697 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.697 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.697 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.697 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.698 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.698 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.698 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.698 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.698 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.698 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.698 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.699 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.699 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.699 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.699 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.699 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.699 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.699 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.699 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.699 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.699 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.699 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.700 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.700 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.700 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.700 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.700 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.700 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.700 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.700 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.700 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.700 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.701 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.701 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.701 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.701 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.701 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.701 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.701 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.701 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.701 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.701 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.701 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.702 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.702 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.702 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.702 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.702 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.702 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.702 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.702 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.702 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.702 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.702 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.702 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.703 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.703 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.703 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.703 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.703 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.703 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.703 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.703 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.703 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.703 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.704 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.704 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.704 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.704 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.704 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.704 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-09-19T08:13:55.704 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-09-19T08:13:55.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.704 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.704 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.704 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.704 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.705 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.705 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.705 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.705 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.705 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.705 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.705 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-09-19T08:13:55.705 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-09-19T08:13:55.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.705 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.705 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.705 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.706 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.706 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.706 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.706 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.706 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.706 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.706 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.706 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.706 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-09-19T08:13:55.706 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.707 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.707 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.707 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.707 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.707 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.707 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.707 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.707 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.707 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.707 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.707 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.707 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.708 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.708 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.708 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.708 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.708 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.708 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.708 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.708 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.708 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.708 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.709 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.709 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.709 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.709 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.709 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.709 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.709 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.709 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.709 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.709 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.710 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.710 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.710 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.710 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.710 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.710 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.710 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.710 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.710 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.710 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.710 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.711 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.711 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.711 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.711 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.711 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.711 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.711 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.711 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.711 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.711 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.711 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.712 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.712 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.712 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.712 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.712 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.712 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.712 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.712 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.712 INFO:teuthology.orchestra.run.smithi098.stdout: "secondary_zone_period_retry_limit": { 2024-09-19T08:13:55.712 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-19T08:13:55.712 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.713 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.713 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.713 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:55.713 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.713 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.713 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.713 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-19T08:13:55.713 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.713 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.713 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-09-19T08:13:55.714 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.714 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.714 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.714 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.714 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.714 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.714 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.715 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.715 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.715 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.715 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.715 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.715 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.715 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.715 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.715 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.715 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.715 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.715 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.715 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.716 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.716 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.716 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.716 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.716 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.716 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.716 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.716 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.717 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.717 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.717 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.717 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.717 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.717 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.717 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.717 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.717 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.717 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.717 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.718 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.718 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.718 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.718 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.718 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.718 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.718 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.718 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.718 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.718 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.718 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.718 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.718 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.719 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.719 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.719 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.719 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.719 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.719 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.719 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.719 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.719 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.719 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.719 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.719 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.720 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-09-19T08:13:55.720 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-09-19T08:13:55.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.720 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.720 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.720 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-09-19T08:13:55.720 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.720 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.720 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.720 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-19T08:13:55.720 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.720 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.720 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.721 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.721 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-09-19T08:13:55.721 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.721 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.721 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.721 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.721 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.721 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.721 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.721 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.722 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.722 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.722 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.722 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.722 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.722 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.722 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.722 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.722 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.722 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.722 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.722 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.722 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.723 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.723 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.723 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.723 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.723 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.723 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.723 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.723 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.723 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.723 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.724 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.724 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.724 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.724 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.724 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.724 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.724 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.724 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.724 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.724 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.724 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.725 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.725 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.725 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.725 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.725 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.725 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.725 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.725 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.725 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.725 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.725 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.725 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.726 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.726 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.726 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.726 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.726 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.726 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.726 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.726 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.726 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.726 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.726 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.727 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-09-19T08:13:55.727 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-09-19T08:13:55.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.727 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.727 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.727 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.727 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.727 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.727 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.727 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.727 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.728 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.728 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.728 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-09-19T08:13:55.728 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-09-19T08:13:55.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.728 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.728 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.728 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-09-19T08:13:55.728 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.728 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.728 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.728 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.729 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.729 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.729 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.729 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-09-19T08:13:55.729 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-09-19T08:13:55.729 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.729 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.729 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.729 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.729 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.730 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.730 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.730 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.730 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.730 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.730 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.730 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-09-19T08:13:55.730 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-09-19T08:13:55.730 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.730 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.730 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.730 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.731 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.731 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.731 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.731 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.731 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.731 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.731 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.731 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-09-19T08:13:55.731 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-09-19T08:13:55.731 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:55.731 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.732 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.732 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.732 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.732 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.732 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.732 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.732 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.732 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.732 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.732 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-09-19T08:13:55.732 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-09-19T08:13:55.733 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.733 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.733 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.733 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.733 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.733 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.733 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.733 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.733 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.733 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.733 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.733 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-09-19T08:13:55.734 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-09-19T08:13:55.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.734 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.734 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.734 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.734 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.734 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.734 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.734 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.734 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.734 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.734 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.735 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-09-19T08:13:55.735 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-09-19T08:13:55.735 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.735 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.735 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.735 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.735 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.735 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.735 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.735 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.735 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.735 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.736 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.736 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-09-19T08:13:55.736 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-09-19T08:13:55.736 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.736 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.736 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.736 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.736 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-09-19T08:13:55.736 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-09-19T08:13:55.736 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.736 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.737 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.737 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.737 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.737 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-09-19T08:13:55.737 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-09-19T08:13:55.737 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.737 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.737 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.737 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.737 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.737 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.738 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.738 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.738 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.738 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.738 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.738 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-09-19T08:13:55.738 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-09-19T08:13:55.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.738 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.738 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.738 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-09-19T08:13:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-09-19T08:13:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.740 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.740 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.740 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.740 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.740 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.740 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.740 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.740 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.740 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.740 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-09-19T08:13:55.740 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.741 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.741 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.741 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-09-19T08:13:55.741 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-09-19T08:13:55.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.741 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.741 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.741 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.741 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.741 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.741 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.741 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-09-19T08:13:55.741 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.742 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.742 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.742 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-09-19T08:13:55.742 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-09-19T08:13:55.742 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.742 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.742 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.742 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.742 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.742 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.742 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.742 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-09-19T08:13:55.743 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.743 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.743 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.743 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.743 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.743 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.743 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.743 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.743 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.743 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.743 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.744 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.744 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.744 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.744 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.744 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.744 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.744 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.744 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.744 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.744 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.745 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.745 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.745 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.745 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.745 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.745 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.745 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.745 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.745 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.745 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.745 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.745 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.745 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.746 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.746 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.746 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.746 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.746 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.746 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.746 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.746 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.746 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.746 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.747 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.747 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.747 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.747 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.747 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.747 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.747 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.747 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.747 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.747 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.747 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.747 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.748 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.748 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.748 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.748 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.748 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.748 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.748 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.748 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.748 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.748 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.748 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.748 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.749 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-09-19T08:13:55.749 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.749 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.749 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.749 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.749 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.749 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.749 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.749 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.749 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.749 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.749 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.750 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.750 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.750 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.750 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.750 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.750 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.750 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.750 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.750 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.750 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.750 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.750 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.751 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.751 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.751 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.751 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.751 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.751 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.751 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.751 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.751 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.751 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.751 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.751 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.752 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.752 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.752 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.752 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.752 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.752 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.752 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.752 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.752 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.752 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.752 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.753 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.753 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.753 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.753 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.753 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.753 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.753 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.753 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.753 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.753 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.753 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.753 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.754 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.754 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.754 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.754 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.754 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.754 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.754 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.754 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.754 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.754 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.754 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.754 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.755 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-09-19T08:13:55.755 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.755 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.755 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.755 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.755 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.755 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.755 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.755 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.755 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.755 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.755 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.756 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.756 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.756 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.756 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.756 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.756 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.756 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.756 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.756 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.756 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.756 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.756 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.757 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.757 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.757 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.757 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.757 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.757 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.757 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.757 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.757 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.757 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.757 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.758 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.758 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.758 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.758 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.758 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.758 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.758 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.758 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.758 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.758 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.758 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.759 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.759 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.759 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.759 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.759 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.759 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.759 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.759 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.759 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.759 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.759 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.759 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.760 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.760 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.760 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.760 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.760 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.760 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.760 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.760 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.760 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.760 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.760 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.761 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-09-19T08:13:55.761 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.761 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.761 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.761 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-09-19T08:13:55.761 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-09-19T08:13:55.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.761 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.761 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.762 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-19T08:13:55.762 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.762 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.762 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.762 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.762 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.762 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.762 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.762 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:55.762 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:55.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.763 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.763 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.763 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-09-19T08:13:55.763 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.763 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.763 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.763 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.763 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.763 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.763 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.763 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.763 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.764 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.764 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.764 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.764 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.764 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.764 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.764 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.764 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.764 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.764 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.764 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.765 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.765 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.765 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.765 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.765 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.765 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.765 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.765 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.765 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.765 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.765 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.765 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.766 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.766 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.766 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.766 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.766 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.766 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.766 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.766 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.766 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.766 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.766 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.767 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.767 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.767 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.767 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.767 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.767 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.767 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.767 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.767 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.767 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.767 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.767 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.768 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.768 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.768 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.768 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.768 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.768 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.768 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.768 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.768 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.768 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-09-19T08:13:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-09-19T08:13:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-09-19T08:13:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-19T08:13:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-09-19T08:13:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-09-19T08:13:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-19T08:13:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-09-19T08:13:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-09-19T08:13:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-19T08:13:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-09-19T08:13:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-09-19T08:13:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-19T08:13:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-09-19T08:13:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-09-19T08:13:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-19T08:13:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-09-19T08:13:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-09-19T08:13:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-09-19T08:13:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-09-19T08:13:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-09-19T08:13:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-09-19T08:13:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-19T08:13:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-09-19T08:13:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-09-19T08:13:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-09-19T08:13:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-09-19T08:13:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-09-19T08:13:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-09-19T08:13:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-09-19T08:13:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-09-19T08:13:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-09-19T08:13:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-09-19T08:13:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-09-19T08:13:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-09-19T08:13:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-09-19T08:13:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-09-19T08:13:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-09-19T08:13:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-09-19T08:13:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-09-19T08:13:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-19T08:13:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-09-19T08:13:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.798 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.798 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.798 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-09-19T08:13:55.798 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.798 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.798 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.798 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.798 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.798 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.799 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.799 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.799 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.799 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.799 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.799 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.799 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.799 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.799 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.799 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.799 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.799 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.799 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.800 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.800 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.800 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.800 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.800 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.800 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.800 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.800 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.800 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.800 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.800 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.800 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-09-19T08:13:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-09-19T08:13:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-09-19T08:13:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-19T08:13:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-09-19T08:13:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-09-19T08:13:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-09-19T08:13:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-09-19T08:13:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-09-19T08:13:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-19T08:13:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-09-19T08:13:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-09-19T08:13:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-19T08:13:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-09-19T08:13:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-09-19T08:13:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-09-19T08:13:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-09-19T08:13:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-09-19T08:13:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-09-19T08:13:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-09-19T08:13:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-09-19T08:13:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-09-19T08:13:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-09-19T08:13:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-09-19T08:13:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-09-19T08:13:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-09-19T08:13:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-19T08:13:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-19T08:13:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 24770, 2024-09-19T08:13:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "y", 2024-09-19T08:13:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_features": 4540701547738038271, 2024-09-19T08:13:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-09-19T08:13:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-09-19T08:13:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-09-19T08:13:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-09-19T08:13:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-09-19T08:13:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-09-19T08:13:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-09-19T08:13:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-09-19T08:13:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-09-19T08:13:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-09-19T08:13:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-09-19T08:13:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-09-19T08:13:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-09-19T08:13:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-09-19T08:13:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-09-19T08:13:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-09-19T08:13:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-09-19T08:13:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-09-19T08:13:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-09-19T08:13:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-09-19T08:13:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-09-19T08:13:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-09-19T08:13:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-09-19T08:13:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-09-19T08:13:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-09-19T08:13:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-19T08:13:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-09-19T08:13:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-09-19T08:13:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-09-19T08:13:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-09-19T08:13:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-09-19T08:13:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-09-19T08:13:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-09-19T08:13:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-09-19T08:13:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-09-19T08:13:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-09-19T08:13:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-09-19T08:13:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-19T08:13:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-09-19T08:13:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-09-19T08:13:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-09-19T08:13:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-09-19T08:13:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-19T08:13:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-19T08:13:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-09-19T08:13:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-09-19T08:13:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-09-19T08:13:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-09-19T08:13:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-09-19T08:13:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-19T08:13:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-09-19T08:13:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-09-19T08:13:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-09-19T08:13:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-19T08:13:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-19T08:13:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-09-19T08:13:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-09-19T08:13:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-09-19T08:13:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-09-19T08:13:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-09-19T08:13:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-09-19T08:13:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-19T08:13:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-09-19T08:13:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-09-19T08:13:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-09-19T08:13:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-09-19T08:13:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-19T08:13:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-09-19T08:13:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-09-19T08:13:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-09-19T08:13:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-09-19T08:13:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-19T08:13:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-19T08:13:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-09-19T08:13:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-09-19T08:13:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-09-19T08:13:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-09-19T08:13:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-09-19T08:13:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-19T08:13:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-19T08:13:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-09-19T08:13:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-09-19T08:13:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-09-19T08:13:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-19T08:13:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-19T08:13:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-09-19T08:13:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-09-19T08:13:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-19T08:13:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-19T08:13:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-09-19T08:13:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-09-19T08:13:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-09-19T08:13:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-19T08:13:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-19T08:13:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-09-19T08:13:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-09-19T08:13:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-19T08:13:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-09-19T08:13:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-09-19T08:13:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-09-19T08:13:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-19T08:13:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-09-19T08:13:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-09-19T08:13:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-09-19T08:13:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-09-19T08:13:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-09-19T08:13:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-19T08:13:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-09-19T08:13:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-09-19T08:13:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-19T08:13:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-09-19T08:13:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-09-19T08:13:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-19T08:13:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-09-19T08:13:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-09-19T08:13:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-19T08:13:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-09-19T08:13:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-09-19T08:13:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-19T08:13:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-09-19T08:13:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-09-19T08:13:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-09-19T08:13:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-19T08:13:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-09-19T08:13:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-09-19T08:13:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-09-19T08:13:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-19T08:13:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-09-19T08:13:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-09-19T08:13:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-09-19T08:13:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-19T08:13:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-09-19T08:13:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-09-19T08:13:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-09-19T08:13:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-19T08:13:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-09-19T08:13:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-09-19T08:13:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-09-19T08:13:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-19T08:13:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-09-19T08:13:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-09-19T08:13:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-09-19T08:13:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-19T08:13:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-09-19T08:13:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-09-19T08:13:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-09-19T08:13:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-19T08:13:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-09-19T08:13:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-09-19T08:13:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-09-19T08:13:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-19T08:13:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-09-19T08:13:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-09-19T08:13:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-09-19T08:13:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-19T08:13:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-09-19T08:13:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-09-19T08:13:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-09-19T08:13:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-19T08:13:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-09-19T08:13:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-09-19T08:13:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-09-19T08:13:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-19T08:13:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-09-19T08:13:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-09-19T08:13:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-09-19T08:13:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-19T08:13:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-09-19T08:13:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_samba": { 2024-09-19T08:13:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_samba", 2024-09-19T08:13:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-19T08:13:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Samba/SMB container image", 2024-09-19T08:13:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-09-19T08:13:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-09-19T08:13:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-19T08:13:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-09-19T08:13:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-09-19T08:13:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-09-19T08:13:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-09-19T08:13:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-09-19T08:13:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-09-19T08:13:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-19T08:13:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-09-19T08:13:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-09-19T08:13:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-09-19T08:13:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-19T08:13:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-09-19T08:13:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-09-19T08:13:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-09-19T08:13:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-19T08:13:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-09-19T08:13:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-09-19T08:13:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-09-19T08:13:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-09-19T08:13:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-09-19T08:13:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-09-19T08:13:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-19T08:13:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-09-19T08:13:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-09-19T08:13:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-09-19T08:13:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "grafana_dashboards_path": { 2024-09-19T08:13:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "grafana_dashboards_path", 2024-09-19T08:13:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-19T08:13:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-19T08:13:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-09-19T08:13:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-09-19T08:13:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-09-19T08:13:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-09-19T08:13:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-09-19T08:13:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-19T08:13:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-09-19T08:13:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-09-19T08:13:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-19T08:13:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-09-19T08:13:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-09-19T08:13:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-19T08:13:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-19T08:13:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-19T08:13:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-19T08:13:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-19T08:13:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-19T08:13:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-09-19T08:13:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-19T08:13:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-09-19T08:13:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-09-19T08:13:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-09-19T08:13:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-09-19T08:13:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-09-19T08:13:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-19T08:13:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-09-19T08:13:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-09-19T08:13:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-09-19T08:13:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-09-19T08:13:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-09-19T08:13:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-09-19T08:13:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-09-19T08:13:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-09-19T08:13:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-09-19T08:13:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-09-19T08:13:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-09-19T08:13:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-09-19T08:13:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-19T08:13:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-09-19T08:13:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-09-19T08:13:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-19T08:13:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-19T08:13:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-09-19T08:13:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-09-19T08:13:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-19T08:13:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-09-19T08:13:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-09-19T08:13:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-19T08:13:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-09-19T08:13:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-09-19T08:13:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-19T08:13:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-09-19T08:13:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-09-19T08:13:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-19T08:13:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-09-19T08:13:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-09-19T08:13:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-19T08:13:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-09-19T08:13:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-09-19T08:13:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-09-19T08:13:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-09-19T08:13:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-09-19T08:13:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-09-19T08:13:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-19T08:13:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-09-19T08:13:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-09-19T08:13:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-19T08:13:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-09-19T08:13:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-09-19T08:13:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-19T08:13:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-09-19T08:13:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-09-19T08:13:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-09-19T08:13:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-09-19T08:13:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-09-19T08:13:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-19T08:13:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-09-19T08:13:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-09-19T08:13:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-19T08:13:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-09-19T08:13:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-09-19T08:13:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-09-19T08:13:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-09-19T08:13:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-19T08:13:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-09-19T08:13:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-19T08:13:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-19T08:13:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-09-19T08:13:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-19T08:13:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-19T08:13:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-19T08:13:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-09-19T08:13:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-19T08:13:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-19T08:13:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-19T08:13:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-19T08:13:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-19T08:13:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-19T08:13:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-19T08:13:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-19T08:13:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-19T08:13:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-19T08:13:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-19T08:13:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-19T08:13:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-19T08:13:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-19T08:13:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-19T08:13:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-19T08:13:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-19T08:13:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-09-19T08:13:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-19T08:13:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-09-19T08:13:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-19T08:13:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-19T08:13:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-09-19T08:13:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-09-19T08:13:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-09-19T08:13:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-19T08:13:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-09-19T08:13:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-19T08:13:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-19T08:13:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-19T08:13:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-19T08:13:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-19T08:13:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-19T08:13:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-19T08:13:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-19T08:13:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-09-19T08:13:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-19T08:13:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-19T08:13:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-19T08:13:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-19T08:13:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-19T08:13:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-19T08:13:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-19T08:13:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.961 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.961 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.961 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.961 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.961 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.961 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.961 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-19T08:13:55.961 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-19T08:13:55.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.961 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.961 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.961 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-19T08:13:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-19T08:13:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-19T08:13:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-19T08:13:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-19T08:13:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-19T08:13:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-09-19T08:13:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-19T08:13:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-19T08:13:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-19T08:13:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-19T08:13:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-19T08:13:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-19T08:13:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-09-19T08:13:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-19T08:13:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-19T08:13:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-09-19T08:13:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-09-19T08:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-19T08:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-19T08:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-19T08:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-09-19T08:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-09-19T08:13:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-19T08:13:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-09-19T08:13:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-19T08:13:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-19T08:13:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-19T08:13:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-19T08:13:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-19T08:13:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-19T08:13:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-19T08:13:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-09-19T08:13:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-09-19T08:13:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-09-19T08:13:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-09-19T08:13:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-09-19T08:13:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-09-19T08:13:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-09-19T08:13:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-09-19T08:13:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-09-19T08:13:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-09-19T08:13:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-09-19T08:13:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-09-19T08:13:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-09-19T08:13:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-09-19T08:13:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-09-19T08:13:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-09-19T08:13:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-09-19T08:13:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-09-19T08:13:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-09-19T08:13:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-09-19T08:13:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-09-19T08:13:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-09-19T08:13:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-09-19T08:13:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-09-19T08:13:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-09-19T08:13:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-09-19T08:13:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-09-19T08:13:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-09-19T08:13:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-09-19T08:13:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-09-19T08:13:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-09-19T08:13:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-09-19T08:13:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-09-19T08:13:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-09-19T08:13:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-09-19T08:13:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-09-19T08:13:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-09-19T08:13:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-09-19T08:13:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-09-19T08:13:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-09-19T08:13:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-09-19T08:13:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-09-19T08:13:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-09-19T08:13:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-09-19T08:13:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-09-19T08:13:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-09-19T08:13:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-19T08:13:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-09-19T08:13:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-09-19T08:13:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-09-19T08:13:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-19T08:13:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-09-19T08:13:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-09-19T08:13:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-09-19T08:13:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-09-19T08:13:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-09-19T08:13:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-09-19T08:13:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-09-19T08:13:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-19T08:13:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-09-19T08:13:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-09-19T08:13:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-19T08:13:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-09-19T08:13:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-09-19T08:13:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-09-19T08:13:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-09-19T08:13:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-19T08:13:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-09-19T08:13:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-09-19T08:13:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-09-19T08:13:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-09-19T08:13:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-09-19T08:13:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-09-19T08:13:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-09-19T08:13:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-09-19T08:13:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-09-19T08:13:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-09-19T08:13:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-09-19T08:13:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-09-19T08:13:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-09-19T08:13:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-19T08:13:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-09-19T08:13:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-09-19T08:13:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-09-19T08:13:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-19T08:13:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-09-19T08:13:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-09-19T08:13:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-09-19T08:13:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-09-19T08:13:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-09-19T08:13:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-19T08:13:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.037 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.037 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.037 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.037 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.037 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.038 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.038 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.038 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.038 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.038 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.038 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.038 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.038 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-09-19T08:13:56.038 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-09-19T08:13:56.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.038 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.039 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.039 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.039 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.039 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.039 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.039 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-09-19T08:13:56.039 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.039 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.039 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.039 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-09-19T08:13:56.039 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-09-19T08:13:56.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.040 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.040 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.040 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-09-19T08:13:56.040 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.040 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.040 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.040 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-09-19T08:13:56.040 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.040 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.040 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.040 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-09-19T08:13:56.041 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-09-19T08:13:56.041 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.041 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.041 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.041 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-09-19T08:13:56.041 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.041 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.041 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.041 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-19T08:13:56.041 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.041 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.041 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.042 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-09-19T08:13:56.042 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-09-19T08:13:56.042 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.042 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.042 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.042 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:56.042 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-09-19T08:13:56.042 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-09-19T08:13:56.042 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.042 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-19T08:13:56.042 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.042 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.043 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.043 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-09-19T08:13:56.043 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-09-19T08:13:56.043 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.043 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.043 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.043 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.043 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.043 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.043 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.043 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-09-19T08:13:56.043 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.044 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.044 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.044 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-09-19T08:13:56.044 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-09-19T08:13:56.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.045 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.045 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.045 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-09-19T08:13:56.045 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.045 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.045 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.045 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-19T08:13:56.045 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.046 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.046 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.046 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.046 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.046 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-09-19T08:13:56.046 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.046 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.046 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.046 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.046 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.046 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.046 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.047 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.047 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.047 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.047 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.047 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.047 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.047 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.047 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.047 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.047 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.047 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.047 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.047 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.047 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.048 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.048 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.048 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.048 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.048 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.048 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.048 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.048 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.048 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.048 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.048 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.048 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.049 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.049 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.049 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.049 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.049 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.049 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.049 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.049 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.049 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.049 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.049 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.049 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.049 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.050 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.050 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.050 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.050 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.050 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.050 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.050 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.050 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.050 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.050 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.050 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.050 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.050 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.051 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.051 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.051 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.051 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.051 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.051 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.051 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.051 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.051 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.051 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-09-19T08:13:56.052 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.052 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.052 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.052 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.052 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.052 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.052 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.052 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.052 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.053 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.053 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.053 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.053 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.053 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.053 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.053 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.053 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.053 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.053 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.053 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.053 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.054 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.054 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.054 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.054 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.054 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.054 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.054 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.054 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.054 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.054 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.054 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.054 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.055 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.055 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.055 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.055 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.055 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.055 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.055 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.055 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.055 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.055 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.056 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.056 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.056 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.056 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.056 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.056 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.056 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.056 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.056 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.056 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.056 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.056 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.056 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.057 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.057 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.057 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.057 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.057 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.057 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.057 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.057 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.057 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.057 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.057 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.057 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.058 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-09-19T08:13:56.058 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.058 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.058 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.058 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-09-19T08:13:56.058 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-09-19T08:13:56.058 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.058 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.058 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.058 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-09-19T08:13:56.058 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.058 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.058 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.058 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-19T08:13:56.059 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.059 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.059 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.059 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-09-19T08:13:56.059 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-09-19T08:13:56.059 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.059 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.059 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.059 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:56.059 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-09-19T08:13:56.059 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.059 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-19T08:13:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-09-19T08:13:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-09-19T08:13:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-09-19T08:13:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-09-19T08:13:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-09-19T08:13:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-19T08:13:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-09-19T08:13:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-09-19T08:13:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.074 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-09-19T08:13:56.074 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-19T08:13:56.074 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.074 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.074 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-09-19T08:13:56.074 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-09-19T08:13:56.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.074 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.074 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.075 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-09-19T08:13:56.075 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.075 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.075 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.075 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-09-19T08:13:56.075 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.075 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.075 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.075 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-09-19T08:13:56.075 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-09-19T08:13:56.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.075 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.076 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.076 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-09-19T08:13:56.076 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.076 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.076 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.076 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-09-19T08:13:56.076 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.076 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.076 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.076 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-09-19T08:13:56.076 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-09-19T08:13:56.076 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.076 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.077 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.077 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.077 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.077 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.077 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.077 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-09-19T08:13:56.077 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.077 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.077 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.078 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-09-19T08:13:56.078 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-09-19T08:13:56.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.078 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.078 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.079 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-09-19T08:13:56.079 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.079 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.079 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.079 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-19T08:13:56.079 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-19T08:13:56.079 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.080 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.080 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.080 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.080 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-09-19T08:13:56.080 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.080 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.080 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.080 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.081 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.081 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.081 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.081 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.081 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.081 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.081 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.081 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.081 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.081 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.081 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.081 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.082 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.082 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.082 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.082 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.082 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.082 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.082 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.082 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.082 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.082 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.082 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.082 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.083 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.083 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.083 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.083 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.083 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.083 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.083 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.083 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.083 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.083 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.083 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.084 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.084 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.084 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.084 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.084 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.084 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.084 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.084 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.084 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.084 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.084 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.084 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.084 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.085 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.085 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.085 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.085 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.085 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.085 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.085 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.085 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.085 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.085 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.085 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.086 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.086 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-09-19T08:13:56.086 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.086 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.086 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.086 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.087 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.087 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.087 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.087 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.087 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.087 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.087 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.087 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.087 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.087 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.087 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.087 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.087 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.087 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.088 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.088 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.088 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.088 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.088 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.088 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.088 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.088 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.088 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.088 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.089 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.089 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.089 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.089 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.089 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.089 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.089 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.089 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.089 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.089 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.089 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.089 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.090 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.090 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.090 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.090 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.090 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.090 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.090 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.090 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.090 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.090 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.090 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.090 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.091 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.091 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.091 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.091 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.091 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.091 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.091 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.091 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.091 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.091 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.092 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.092 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.092 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.092 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.092 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-09-19T08:13:56.092 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.092 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.092 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.092 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.092 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.092 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.093 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.093 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.093 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.093 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.093 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.093 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.093 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.093 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.093 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.093 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.093 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.093 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.093 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.094 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.094 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.094 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.094 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.094 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.094 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.094 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.095 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.095 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.095 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.095 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.095 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.095 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.095 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.095 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.095 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.095 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.095 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.096 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.096 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.096 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.096 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.096 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.096 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.096 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.096 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.096 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.096 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.096 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.096 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.097 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.097 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.097 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.097 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.097 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.097 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.097 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.097 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.097 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.098 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.098 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.098 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.098 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.098 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.098 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.098 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.098 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-09-19T08:13:56.098 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.098 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.099 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.099 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-09-19T08:13:56.099 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-09-19T08:13:56.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.099 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.099 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.099 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.099 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.099 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.099 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.099 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-19T08:13:56.099 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.100 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.100 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.100 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.100 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.100 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.100 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.100 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.100 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.100 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.100 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.100 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.100 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.101 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.101 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.101 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.101 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.101 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.101 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.101 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.101 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.101 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.101 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.101 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.101 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.102 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.102 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.102 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.102 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.102 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.102 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.102 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.102 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.102 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.102 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-09-19T08:13:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-09-19T08:13:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-09-19T08:13:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-09-19T08:13:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-09-19T08:13:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-09-19T08:13:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-09-19T08:13:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-09-19T08:13:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-09-19T08:13:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-09-19T08:13:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-09-19T08:13:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-09-19T08:13:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-09-19T08:13:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-09-19T08:13:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-19T08:13:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-09-19T08:13:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-09-19T08:13:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-09-19T08:13:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-19T08:13:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-09-19T08:13:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-09-19T08:13:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-09-19T08:13:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-09-19T08:13:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-09-19T08:13:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-09-19T08:13:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-09-19T08:13:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-09-19T08:13:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-09-19T08:13:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-09-19T08:13:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-09-19T08:13:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-09-19T08:13:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-19T08:13:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-19T08:13:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-09-19T08:13:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-09-19T08:13:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-19T08:13:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-19T08:13:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-09-19T08:13:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-09-19T08:13:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-09-19T08:13:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-09-19T08:13:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-09-19T08:13:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-09-19T08:13:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-09-19T08:13:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-19T08:13:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-09-19T08:13:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-09-19T08:13:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-09-19T08:13:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-19T08:13:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-09-19T08:13:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-09-19T08:13:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-09-19T08:13:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-09-19T08:13:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-09-19T08:13:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-09-19T08:13:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-09-19T08:13:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-09-19T08:13:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-09-19T08:13:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-09-19T08:13:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-09-19T08:13:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-09-19T08:13:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-09-19T08:13:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-09-19T08:13:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-09-19T08:13:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-09-19T08:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-09-19T08:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-09-19T08:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-09-19T08:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-09-19T08:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-09-19T08:13:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-09-19T08:13:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-09-19T08:13:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-09-19T08:13:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-09-19T08:13:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-09-19T08:13:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-09-19T08:13:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.178 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.178 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.178 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.178 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.178 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.178 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-09-19T08:13:56.178 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-09-19T08:13:56.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.178 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.178 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.178 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.178 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.179 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.180 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.180 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.180 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.180 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.180 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.180 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.180 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.180 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.180 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.180 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.180 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.180 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.180 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.181 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.181 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.181 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.181 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.181 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.181 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.181 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.181 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.181 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.181 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.182 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.182 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.182 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.182 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.182 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.182 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.182 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.182 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.182 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.182 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.182 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.183 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.183 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.183 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.183 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.183 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.183 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.183 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.183 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.183 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.183 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.183 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.183 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.183 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.184 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.184 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.184 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.184 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.184 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.184 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.184 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.184 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.184 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.184 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.184 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.184 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.184 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.185 INFO:teuthology.orchestra.run.smithi098.stdout: "secondary_zone_period_retry_limit": { 2024-09-19T08:13:56.185 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-19T08:13:56.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.185 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.185 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.185 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:56.185 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.185 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.185 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.185 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-19T08:13:56.185 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.185 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.186 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.186 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.186 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.186 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-09-19T08:13:56.186 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.186 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.186 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.186 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.186 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.186 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.187 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.187 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.187 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.187 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.187 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.187 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.187 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.187 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.187 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.187 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.187 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.187 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.187 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.187 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.188 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.188 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.188 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.188 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.188 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.188 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.188 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.188 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.188 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.188 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.188 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.188 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.189 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.189 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.189 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.189 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.189 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.189 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.189 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.189 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.189 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.189 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.189 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.189 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.189 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.190 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.190 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.190 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.190 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.190 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.190 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.190 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.190 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.190 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.190 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.190 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.190 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.191 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.191 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.191 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.191 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.191 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.191 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.191 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.191 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.191 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.191 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.191 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-09-19T08:13:56.191 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-09-19T08:13:56.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.192 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.192 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.192 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-09-19T08:13:56.192 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.192 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.192 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.192 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-19T08:13:56.192 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.192 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.192 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.192 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.192 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.193 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-09-19T08:13:56.193 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.193 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.193 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.193 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.193 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.193 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.193 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.193 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.193 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.193 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.193 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.193 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.194 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.194 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.194 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.194 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.194 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.194 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.194 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.194 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.194 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.194 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.194 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.194 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.195 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.195 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.195 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.195 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.195 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.195 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.195 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.195 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.195 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.195 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.195 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.195 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.195 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.196 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.196 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.196 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.196 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.196 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.196 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.196 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.196 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.196 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.196 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.196 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.196 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.196 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.196 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.198 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.198 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.198 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.198 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-09-19T08:13:56.198 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-09-19T08:13:56.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.198 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.198 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.198 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.198 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.198 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.198 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.198 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.199 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.200 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.200 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.200 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-09-19T08:13:56.200 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-09-19T08:13:56.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.200 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.200 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.200 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.200 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.200 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.200 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.200 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-09-19T08:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-09-19T08:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.202 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.202 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.202 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.202 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-09-19T08:13:56.202 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-09-19T08:13:56.202 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:56.202 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.202 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.202 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.202 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.202 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.202 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.202 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.203 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.203 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.203 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.203 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-09-19T08:13:56.203 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-09-19T08:13:56.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.203 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.203 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.203 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.203 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.203 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.203 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.203 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.204 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.204 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.204 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.204 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-09-19T08:13:56.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-09-19T08:13:56.204 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.204 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.204 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.204 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.204 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.204 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.204 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.205 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.205 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.205 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.205 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.205 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-09-19T08:13:56.205 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-09-19T08:13:56.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.205 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.205 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.205 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.205 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.205 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.205 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.206 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.206 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.206 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.206 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.206 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-09-19T08:13:56.206 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-09-19T08:13:56.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.206 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.206 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.206 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.206 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-09-19T08:13:56.206 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-09-19T08:13:56.207 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.207 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.207 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.207 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.207 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.207 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-09-19T08:13:56.207 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-09-19T08:13:56.207 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.207 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.207 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.207 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.207 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.207 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-09-19T08:13:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-09-19T08:13:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.208 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-09-19T08:13:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-09-19T08:13:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-09-19T08:13:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-09-19T08:13:56.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-09-19T08:13:56.211 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.211 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.211 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.211 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.211 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.211 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.211 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.211 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-09-19T08:13:56.211 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.211 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.211 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.212 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-09-19T08:13:56.212 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-09-19T08:13:56.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.212 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.212 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.212 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.212 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.212 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.212 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.212 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-09-19T08:13:56.212 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.212 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.212 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.213 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.213 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.213 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.213 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.213 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.213 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.213 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.213 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.213 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.213 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.213 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.214 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.214 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.214 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.214 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.214 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.216 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.216 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.216 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.216 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.216 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.216 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.216 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.217 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.217 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.217 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.217 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.217 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.217 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.217 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.217 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.217 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.217 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.217 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.218 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.218 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.218 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.218 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.218 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.218 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.218 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.218 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.218 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.218 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.218 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.218 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.218 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.218 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.219 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.219 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.219 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.219 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.219 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.219 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.219 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.219 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.219 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.219 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.219 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.220 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.220 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.220 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.220 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.220 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-09-19T08:13:56.220 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.220 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.220 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.220 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.220 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.222 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.222 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.222 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.222 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.222 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.222 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.222 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.222 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.222 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.222 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.222 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.222 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.223 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.223 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.223 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.223 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.223 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.223 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.223 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.223 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.223 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.223 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.223 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.224 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.224 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.224 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.224 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.224 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.224 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.224 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.224 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.224 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.224 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.224 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.224 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-09-19T08:13:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-09-19T08:13:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-09-19T08:13:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-09-19T08:13:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-19T08:13:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-09-19T08:13:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-09-19T08:13:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-09-19T08:13:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-09-19T08:13:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-19T08:13:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-09-19T08:13:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-09-19T08:13:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-19T08:13:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-09-19T08:13:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-09-19T08:13:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-19T08:13:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-09-19T08:13:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-09-19T08:13:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-09-19T08:13:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-09-19T08:13:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-09-19T08:13:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-09-19T08:13:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-19T08:13:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-09-19T08:13:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-09-19T08:13:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-09-19T08:13:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-09-19T08:13:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-09-19T08:13:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-09-19T08:13:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-09-19T08:13:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-09-19T08:13:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-09-19T08:13:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-09-19T08:13:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-09-19T08:13:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-09-19T08:13:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-09-19T08:13:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-09-19T08:13:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-09-19T08:13:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-09-19T08:13:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-09-19T08:13:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-19T08:13:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-09-19T08:13:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-09-19T08:13:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-09-19T08:13:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-09-19T08:13:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-09-19T08:13:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-19T08:13:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-09-19T08:13:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-09-19T08:13:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-09-19T08:13:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-09-19T08:13:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-09-19T08:13:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-19T08:13:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-09-19T08:13:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-09-19T08:13:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-19T08:13:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-09-19T08:13:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-09-19T08:13:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-09-19T08:13:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-09-19T08:13:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-09-19T08:13:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-09-19T08:13:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-09-19T08:13:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-09-19T08:13:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-09-19T08:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-09-19T08:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-09-19T08:13:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-09-19T08:13:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-09-19T08:13:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-19T08:13:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-19T08:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "modules": [ 2024-09-19T08:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-09-19T08:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "dashboard", 2024-09-19T08:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "iostat", 2024-09-19T08:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "nfs", 2024-09-19T08:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "restful", 2024-09-19T08:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_schedule" 2024-09-19T08:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-09-19T08:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-09-19T08:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-09-19T08:13:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-09-19T08:13:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-09-19T08:13:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-09-19T08:13:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-09-19T08:13:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-09-19T08:13:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-09-19T08:13:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-09-19T08:13:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-09-19T08:13:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-09-19T08:13:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-09-19T08:13:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-09-19T08:13:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-09-19T08:13:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-09-19T08:13:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-09-19T08:13:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-09-19T08:13:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-09-19T08:13:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-09-19T08:13:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-09-19T08:13:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-09-19T08:13:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-09-19T08:13:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-09-19T08:13:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-09-19T08:13:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-19T08:13:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-09-19T08:13:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-09-19T08:13:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-09-19T08:13:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-09-19T08:13:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-09-19T08:13:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-09-19T08:13:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-09-19T08:13:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-09-19T08:13:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-09-19T08:13:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-09-19T08:13:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-09-19T08:13:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-19T08:13:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-09-19T08:13:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-09-19T08:13:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-09-19T08:13:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-09-19T08:13:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-19T08:13:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-19T08:13:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-09-19T08:13:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-09-19T08:13:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-09-19T08:13:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-09-19T08:13:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-09-19T08:13:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-19T08:13:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-09-19T08:13:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-09-19T08:13:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-09-19T08:13:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-09-19T08:13:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-09-19T08:13:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-09-19T08:13:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-19T08:13:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-09-19T08:13:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-09-19T08:13:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-09-19T08:13:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-09-19T08:13:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-19T08:13:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-09-19T08:13:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-09-19T08:13:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-09-19T08:13:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-09-19T08:13:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-19T08:13:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-19T08:13:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.315 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.315 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.315 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.315 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.315 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.315 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.315 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.315 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.315 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.315 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.315 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.315 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.315 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.315 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.316 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.316 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.316 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.316 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.316 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.316 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.316 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.316 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.316 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.317 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.317 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.317 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.317 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.317 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.317 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.317 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.317 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.317 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.317 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.318 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.318 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.318 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.318 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.318 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.318 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.318 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.318 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.318 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.318 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.318 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.318 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.319 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.319 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.319 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.319 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.319 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.319 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.319 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.319 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.319 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.319 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.319 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-09-19T08:13:56.319 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-09-19T08:13:56.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:56.320 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.320 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.320 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:56.320 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.320 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.320 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.320 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-19T08:13:56.320 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.320 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.320 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.320 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-09-19T08:13:56.321 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-09-19T08:13:56.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.321 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.321 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.321 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-09-19T08:13:56.321 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.321 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.321 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.321 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-09-19T08:13:56.321 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-09-19T08:13:56.321 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-09-19T08:13:56.321 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-09-19T08:13:56.321 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-09-19T08:13:56.322 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.322 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-09-19T08:13:56.322 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.322 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.322 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.322 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-09-19T08:13:56.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-09-19T08:13:56.322 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.322 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.322 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.322 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.322 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.322 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.323 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.323 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-19T08:13:56.323 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.323 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.323 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.323 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:56.323 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:56.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.323 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.323 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.323 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:56.323 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.323 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.324 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.324 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-19T08:13:56.324 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.324 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.324 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.324 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-09-19T08:13:56.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-09-19T08:13:56.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.324 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.324 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.324 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:56.324 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-09-19T08:13:56.325 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.325 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.325 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-19T08:13:56.325 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-19T08:13:56.325 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.325 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.325 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-09-19T08:13:56.325 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-09-19T08:13:56.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:56.325 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.325 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.325 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:56.325 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-19T08:13:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-09-19T08:13:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.327 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-09-19T08:13:56.327 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-09-19T08:13:56.327 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:56.327 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.327 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.327 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-09-19T08:13:56.327 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.327 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.327 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.327 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-19T08:13:56.327 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.327 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.327 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.328 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-09-19T08:13:56.328 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-09-19T08:13:56.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.328 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.328 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.328 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-09-19T08:13:56.328 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.328 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.328 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.328 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-19T08:13:56.328 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.328 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.328 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.329 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-09-19T08:13:56.329 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-09-19T08:13:56.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.329 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.329 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.329 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-09-19T08:13:56.329 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.329 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.329 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.329 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-19T08:13:56.329 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.329 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.330 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.330 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-09-19T08:13:56.330 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-09-19T08:13:56.330 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.330 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.330 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.330 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.330 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.330 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.330 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.330 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-19T08:13:56.330 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-19T08:13:56.330 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.331 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.331 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-09-19T08:13:56.331 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-09-19T08:13:56.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.331 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.331 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.331 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:56.331 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.331 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.331 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.331 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-19T08:13:56.331 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.331 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.332 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.332 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-09-19T08:13:56.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-09-19T08:13:56.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:56.332 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.332 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.332 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-09-19T08:13:56.332 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.332 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.332 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.332 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-19T08:13:56.332 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.332 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.333 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.333 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-09-19T08:13:56.333 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-09-19T08:13:56.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.333 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.333 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.333 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.333 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.333 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.333 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.333 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-09-19T08:13:56.333 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-09-19T08:13:56.333 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-09-19T08:13:56.334 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.334 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-19T08:13:56.334 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.334 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.334 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.334 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-09-19T08:13:56.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-09-19T08:13:56.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.334 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.334 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.334 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.334 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.334 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.335 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.335 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-19T08:13:56.335 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.335 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.335 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.335 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-09-19T08:13:56.335 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-09-19T08:13:56.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.335 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.335 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.335 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.335 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-19T08:13:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-09-19T08:13:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-09-19T08:13:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.337 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.337 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.337 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-19T08:13:56.337 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.337 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.337 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.337 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-09-19T08:13:56.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-09-19T08:13:56.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.337 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.337 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.337 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-19T08:13:56.338 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.338 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.338 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.338 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-09-19T08:13:56.338 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.338 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.338 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.338 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-09-19T08:13:56.338 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-09-19T08:13:56.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.338 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.338 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.338 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-19T08:13:56.339 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.339 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.339 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.339 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-09-19T08:13:56.339 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.339 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.339 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.339 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-09-19T08:13:56.339 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-09-19T08:13:56.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.339 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.339 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.340 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-19T08:13:56.340 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.340 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.340 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.340 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-09-19T08:13:56.340 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.340 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.340 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.340 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-09-19T08:13:56.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-09-19T08:13:56.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.340 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.340 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.341 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-19T08:13:56.341 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.341 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.341 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.341 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-09-19T08:13:56.341 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.341 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.341 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.341 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-09-19T08:13:56.341 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-09-19T08:13:56.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.341 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.342 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.342 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-19T08:13:56.342 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.342 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.342 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.342 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-09-19T08:13:56.342 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.342 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.342 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.342 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-09-19T08:13:56.342 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-09-19T08:13:56.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.342 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.343 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.343 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-19T08:13:56.343 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.343 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.343 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.343 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-09-19T08:13:56.343 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.343 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.343 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.343 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-09-19T08:13:56.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-09-19T08:13:56.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.343 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.344 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.344 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-19T08:13:56.344 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.344 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.344 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.344 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-09-19T08:13:56.344 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.344 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.344 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.344 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-09-19T08:13:56.344 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-09-19T08:13:56.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.345 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.345 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.345 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-19T08:13:56.345 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.345 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.345 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.345 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-09-19T08:13:56.345 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.345 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.345 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.345 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-09-19T08:13:56.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-09-19T08:13:56.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.346 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.346 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.346 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-19T08:13:56.346 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.346 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.346 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.346 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-09-19T08:13:56.346 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.346 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.346 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.346 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-09-19T08:13:56.346 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-09-19T08:13:56.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.347 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.347 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.347 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-19T08:13:56.347 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.347 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.347 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.347 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-09-19T08:13:56.347 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.347 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.347 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.347 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-09-19T08:13:56.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-09-19T08:13:56.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.348 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.348 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-19T08:13:56.348 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.348 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.348 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.348 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-09-19T08:13:56.348 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.348 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.348 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.349 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-09-19T08:13:56.349 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-09-19T08:13:56.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.349 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.349 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.349 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-19T08:13:56.349 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.349 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.349 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.349 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-09-19T08:13:56.349 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.349 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.349 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.350 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-09-19T08:13:56.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-09-19T08:13:56.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.351 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.351 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.351 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-19T08:13:56.351 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.351 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.351 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.351 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-09-19T08:13:56.351 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.351 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.351 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.352 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_samba": { 2024-09-19T08:13:56.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_samba", 2024-09-19T08:13:56.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.352 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.352 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.352 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-19T08:13:56.352 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.352 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.352 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.352 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Samba/SMB container image", 2024-09-19T08:13:56.352 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.352 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.352 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-09-19T08:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-09-19T08:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-19T08:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-09-19T08:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-09-19T08:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-09-19T08:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-09-19T08:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-09-19T08:13:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-09-19T08:13:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-19T08:13:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.356 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-09-19T08:13:56.356 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-09-19T08:13:56.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.356 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.356 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.356 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-09-19T08:13:56.356 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.356 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.356 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.356 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-19T08:13:56.356 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.356 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.356 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.357 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-09-19T08:13:56.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-09-19T08:13:56.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.357 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.357 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.357 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-09-19T08:13:56.357 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.357 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.357 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.357 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-19T08:13:56.357 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.357 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.358 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.358 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-09-19T08:13:56.358 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-09-19T08:13:56.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.358 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.358 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.358 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-09-19T08:13:56.358 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.358 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.358 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.358 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-09-19T08:13:56.358 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.359 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.359 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.359 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-09-19T08:13:56.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-09-19T08:13:56.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.359 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.359 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.359 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.359 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.359 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.359 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.359 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-19T08:13:56.359 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.360 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.360 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.360 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-09-19T08:13:56.360 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-09-19T08:13:56.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.360 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.360 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.360 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:56.360 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.360 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.360 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.360 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-09-19T08:13:56.360 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.361 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.361 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.361 INFO:teuthology.orchestra.run.smithi098.stdout: "grafana_dashboards_path": { 2024-09-19T08:13:56.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "grafana_dashboards_path", 2024-09-19T08:13:56.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.361 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.361 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-19T08:13:56.361 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.361 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.361 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.361 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-19T08:13:56.361 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.362 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.362 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.362 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-09-19T08:13:56.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-09-19T08:13:56.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.362 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.362 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.362 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:56.362 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.362 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.362 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.362 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-09-19T08:13:56.363 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.363 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.363 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.363 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-09-19T08:13:56.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-09-19T08:13:56.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.363 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.363 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.363 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.363 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.363 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.363 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-19T08:13:56.364 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.364 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.364 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.364 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-09-19T08:13:56.364 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-09-19T08:13:56.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.364 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.364 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.364 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.364 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.364 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.364 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.365 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-19T08:13:56.365 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.365 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.365 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.365 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.365 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.365 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.365 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.365 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.365 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.365 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-09-19T08:13:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-09-19T08:13:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-19T08:13:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-19T08:13:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-19T08:13:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-19T08:13:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-19T08:13:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-19T08:13:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-19T08:13:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-09-19T08:13:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-19T08:13:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-09-19T08:13:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-09-19T08:13:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-09-19T08:13:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-09-19T08:13:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-09-19T08:13:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-09-19T08:13:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-09-19T08:13:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-09-19T08:13:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-09-19T08:13:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-09-19T08:13:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-09-19T08:13:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-09-19T08:13:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-09-19T08:13:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-09-19T08:13:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-09-19T08:13:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-19T08:13:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-09-19T08:13:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-09-19T08:13:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-19T08:13:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-19T08:13:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-09-19T08:13:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-09-19T08:13:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-19T08:13:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-09-19T08:13:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-09-19T08:13:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-19T08:13:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-09-19T08:13:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-09-19T08:13:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-19T08:13:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-09-19T08:13:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-09-19T08:13:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-19T08:13:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-09-19T08:13:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-09-19T08:13:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-19T08:13:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-09-19T08:13:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-09-19T08:13:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-09-19T08:13:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-09-19T08:13:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-09-19T08:13:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-09-19T08:13:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-19T08:13:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-09-19T08:13:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-09-19T08:13:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-19T08:13:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-09-19T08:13:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-09-19T08:13:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-19T08:13:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-09-19T08:13:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-09-19T08:13:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-09-19T08:13:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-09-19T08:13:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-09-19T08:13:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-19T08:13:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-09-19T08:13:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-09-19T08:13:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-19T08:13:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-09-19T08:13:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-09-19T08:13:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-09-19T08:13:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-09-19T08:13:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-19T08:13:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-09-19T08:13:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-09-19T08:13:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-09-19T08:13:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-19T08:13:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-09-19T08:13:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-19T08:13:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-19T08:13:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-09-19T08:13:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-19T08:13:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-19T08:13:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-19T08:13:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-09-19T08:13:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-09-19T08:13:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-19T08:13:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-19T08:13:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-09-19T08:13:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-19T08:13:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-19T08:13:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-19T08:13:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-19T08:13:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-19T08:13:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-19T08:13:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-19T08:13:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-19T08:13:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-19T08:13:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-19T08:13:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-19T08:13:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-19T08:13:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-19T08:13:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-09-19T08:13:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-19T08:13:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-09-19T08:13:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-19T08:13:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-19T08:13:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-09-19T08:13:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-09-19T08:13:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-09-19T08:13:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-19T08:13:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-09-19T08:13:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-19T08:13:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-19T08:13:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-19T08:13:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-19T08:13:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-19T08:13:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-19T08:13:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-19T08:13:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-19T08:13:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-09-19T08:13:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-19T08:13:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-19T08:13:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-19T08:13:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-19T08:13:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-19T08:13:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-19T08:13:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-19T08:13:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-19T08:13:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-19T08:13:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-19T08:13:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-19T08:13:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-19T08:13:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-19T08:13:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-19T08:13:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-19T08:13:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-19T08:13:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-19T08:13:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-09-19T08:13:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-19T08:13:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-19T08:13:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-19T08:13:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-19T08:13:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-19T08:13:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-19T08:13:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-19T08:13:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-19T08:13:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-09-19T08:13:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-19T08:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-19T08:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-09-19T08:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-09-19T08:13:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-19T08:13:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-19T08:13:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-19T08:13:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-09-19T08:13:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-09-19T08:13:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-19T08:13:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-09-19T08:13:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-19T08:13:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-19T08:13:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-19T08:13:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-19T08:13:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-19T08:13:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-19T08:13:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-19T08:13:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-09-19T08:13:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-09-19T08:13:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-09-19T08:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-09-19T08:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-09-19T08:13:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-09-19T08:13:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-09-19T08:13:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-09-19T08:13:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-09-19T08:13:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-09-19T08:13:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-09-19T08:13:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-09-19T08:13:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-09-19T08:13:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-09-19T08:13:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-09-19T08:13:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-09-19T08:13:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-09-19T08:13:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-09-19T08:13:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-09-19T08:13:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-09-19T08:13:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-09-19T08:13:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-09-19T08:13:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-09-19T08:13:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-09-19T08:13:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-09-19T08:13:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-09-19T08:13:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-09-19T08:13:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-09-19T08:13:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-09-19T08:13:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-09-19T08:13:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-09-19T08:13:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-09-19T08:13:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-09-19T08:13:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-09-19T08:13:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-09-19T08:13:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-09-19T08:13:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-09-19T08:13:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-09-19T08:13:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-09-19T08:13:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-09-19T08:13:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-09-19T08:13:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-09-19T08:13:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-09-19T08:13:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-09-19T08:13:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-19T08:13:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-09-19T08:13:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-09-19T08:13:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-09-19T08:13:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-19T08:13:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-09-19T08:13:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-09-19T08:13:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-09-19T08:13:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-09-19T08:13:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-09-19T08:13:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-09-19T08:13:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-09-19T08:13:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-19T08:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-09-19T08:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-09-19T08:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-19T08:13:56.479 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.479 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.479 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:56.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:56.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.479 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.479 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.479 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:56.479 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.479 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.479 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.480 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-09-19T08:13:56.480 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.480 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.480 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.480 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-09-19T08:13:56.480 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-09-19T08:13:56.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.480 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.480 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.480 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-09-19T08:13:56.480 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.480 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.481 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.481 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-19T08:13:56.481 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.481 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.481 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-09-19T08:13:56.481 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.481 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.481 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.481 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.482 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.482 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.482 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.482 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.482 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.482 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.482 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.482 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.482 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.482 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.482 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.483 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.483 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.483 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.483 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.483 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.483 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.483 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.483 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.483 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.483 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.483 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.483 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.484 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.484 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.484 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.484 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.484 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.484 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.484 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.484 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.484 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.485 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.485 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.485 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.485 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.485 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.485 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.485 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.485 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.485 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.485 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.485 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.485 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.485 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.487 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.487 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.487 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.487 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.487 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-09-19T08:13:56.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-09-19T08:13:56.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.487 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.487 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.487 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-09-19T08:13:56.487 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.487 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.487 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.488 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.488 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.488 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.488 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.488 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-09-19T08:13:56.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-09-19T08:13:56.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.488 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.488 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.488 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-09-19T08:13:56.488 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.488 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-09-19T08:13:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.490 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.490 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.490 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.490 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.490 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.490 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.490 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.490 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.490 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-09-19T08:13:56.490 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-09-19T08:13:56.490 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-09-19T08:13:56.490 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.491 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-09-19T08:13:56.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-09-19T08:13:56.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.491 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.491 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.491 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-09-19T08:13:56.491 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.491 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.491 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.491 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-19T08:13:56.491 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.491 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.491 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.492 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-09-19T08:13:56.492 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-09-19T08:13:56.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.492 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.492 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.492 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-09-19T08:13:56.492 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.492 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.492 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.492 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-19T08:13:56.492 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.492 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.493 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.493 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-09-19T08:13:56.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-09-19T08:13:56.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.493 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.493 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.493 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.493 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.493 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.493 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.493 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-09-19T08:13:56.493 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.493 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.494 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.494 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:56.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:56.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.494 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.494 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-09-19T08:13:56.494 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-09-19T08:13:56.494 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.494 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.494 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-19T08:13:56.494 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.494 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.495 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.495 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.495 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.495 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.495 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.495 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.495 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.495 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.495 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.496 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.496 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.496 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.496 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.496 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.496 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.496 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.496 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.496 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.496 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.496 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.497 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.497 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.497 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.497 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.497 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.497 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.497 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.497 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.497 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.497 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.497 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.497 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.498 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.498 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.498 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.498 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.498 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.498 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.498 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.498 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.498 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.498 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.498 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.498 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.498 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.500 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.500 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.500 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.500 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.500 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-09-19T08:13:56.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-09-19T08:13:56.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.500 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.500 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.500 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.500 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.500 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.501 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.501 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-09-19T08:13:56.501 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.501 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.501 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.501 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-09-19T08:13:56.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-09-19T08:13:56.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.501 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.501 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.501 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-09-19T08:13:56.501 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.502 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.502 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.502 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-09-19T08:13:56.502 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.502 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.502 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.502 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-09-19T08:13:56.502 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-09-19T08:13:56.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.502 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.502 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.502 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-09-19T08:13:56.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.503 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.503 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.503 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-19T08:13:56.503 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.503 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.503 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.503 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-09-19T08:13:56.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-09-19T08:13:56.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.503 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.503 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.503 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-09-19T08:13:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-09-19T08:13:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-19T08:13:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-09-19T08:13:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-09-19T08:13:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.505 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.505 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.505 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.505 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-09-19T08:13:56.505 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.505 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.505 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.505 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-09-19T08:13:56.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-09-19T08:13:56.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.505 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.505 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.506 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-09-19T08:13:56.506 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.506 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.506 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.506 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-19T08:13:56.506 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.506 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.506 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.506 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-09-19T08:13:56.506 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.507 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.507 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.507 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.507 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.507 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.507 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.507 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.507 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.507 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.507 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.507 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.508 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.508 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.508 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.508 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.508 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.508 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.508 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.508 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.508 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.508 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.508 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.512 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.512 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.512 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.512 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.512 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-09-19T08:13:56.512 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.512 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.512 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.514 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.514 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.514 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.514 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.514 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.514 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.514 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.514 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.514 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.514 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.514 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.514 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.515 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.515 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.515 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.515 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.515 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.515 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.515 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.515 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.515 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.515 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.517 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.517 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.517 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.517 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.517 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.517 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.517 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.517 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.517 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.517 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.517 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-09-19T08:13:56.518 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.518 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.518 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.518 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-09-19T08:13:56.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-09-19T08:13:56.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.518 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.518 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.518 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-09-19T08:13:56.519 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.519 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.519 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.519 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-19T08:13:56.519 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.519 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.519 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.519 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-09-19T08:13:56.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-09-19T08:13:56.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.519 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.519 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.520 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:56.520 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-09-19T08:13:56.520 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.520 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.520 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-19T08:13:56.520 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.520 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.520 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.520 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.520 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.521 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.521 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.521 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.521 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.521 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.521 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.521 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.521 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.521 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.521 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.521 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.521 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.521 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.521 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.522 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.522 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.522 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.522 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.522 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.522 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.522 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.522 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.522 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.522 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.522 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-09-19T08:13:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-09-19T08:13:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-09-19T08:13:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-09-19T08:13:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-09-19T08:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-19T08:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-09-19T08:13:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-09-19T08:13:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.533 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.533 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.533 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.533 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.533 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.533 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.533 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-09-19T08:13:56.533 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-19T08:13:56.533 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.533 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.533 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-09-19T08:13:56.533 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-09-19T08:13:56.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.534 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.534 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.534 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-09-19T08:13:56.534 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.534 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.534 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.534 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-09-19T08:13:56.534 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.534 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.534 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.534 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-09-19T08:13:56.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-09-19T08:13:56.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.535 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.535 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.535 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-09-19T08:13:56.535 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.535 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.535 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.535 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-09-19T08:13:56.535 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.535 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.535 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.535 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-09-19T08:13:56.536 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-09-19T08:13:56.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.536 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.536 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.536 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.536 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.536 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.536 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.536 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-09-19T08:13:56.536 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.536 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.536 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.537 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-09-19T08:13:56.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-09-19T08:13:56.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.537 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.537 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.537 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-09-19T08:13:56.537 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.537 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.537 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.537 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-19T08:13:56.537 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-19T08:13:56.537 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.537 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.538 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-09-19T08:13:56.538 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.538 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.538 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.538 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.538 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.538 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.538 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.539 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.539 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.539 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.539 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.539 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.539 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.539 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.539 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.539 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.539 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.539 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.539 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.539 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.540 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.540 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.540 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.540 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.540 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.540 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.540 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.540 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.540 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.540 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.540 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.541 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.541 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.541 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.541 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.541 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.541 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.541 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.541 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.541 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.541 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.541 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.542 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.542 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.542 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.542 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.542 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.542 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.542 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.542 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.542 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.542 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.542 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.543 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.543 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.543 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.543 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.543 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.543 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.543 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.543 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.543 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-09-19T08:13:56.544 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.544 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.544 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.544 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.544 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.544 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.544 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.544 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.544 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.544 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.545 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.545 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.545 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.545 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.545 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.545 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.545 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.545 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.545 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.545 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.545 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.545 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.546 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.546 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.546 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.546 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.546 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.546 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.546 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.546 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.546 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.546 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.546 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.546 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.547 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.547 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.547 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.547 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.547 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.547 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.547 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.547 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.547 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.547 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.547 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.547 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.548 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.548 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.548 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.548 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.548 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.548 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.548 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.548 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.548 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.548 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.548 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.549 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.549 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.549 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.549 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.549 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.549 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.549 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.549 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-09-19T08:13:56.549 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.549 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.549 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.549 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.551 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.551 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.551 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.551 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.551 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.551 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.551 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.551 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.551 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.551 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.551 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.554 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.554 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.554 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.554 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.554 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.554 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.554 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.554 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.554 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.554 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.554 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.554 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.554 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-09-19T08:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-09-19T08:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-09-19T08:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.556 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.556 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.556 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-19T08:13:56.556 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.556 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.556 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.556 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.556 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.556 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.556 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.556 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.556 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.557 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.557 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.557 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.557 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.557 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.557 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.557 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.557 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.557 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.557 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.557 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.557 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.558 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.558 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.558 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.558 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.558 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.558 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.558 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.558 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.558 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.558 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.559 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.559 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.559 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.559 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.559 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.559 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.559 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.559 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.559 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.559 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.559 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.559 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.559 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.560 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.560 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.560 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.560 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.560 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.560 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.560 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.560 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.560 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.560 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.560 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.560 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.561 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.561 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.561 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.561 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.561 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.561 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.561 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.561 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.561 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.561 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-09-19T08:13:56.561 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-09-19T08:13:56.561 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.562 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.562 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.562 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.562 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.562 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.562 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.562 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-09-19T08:13:56.562 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-09-19T08:13:56.562 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-09-19T08:13:56.562 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.562 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-09-19T08:13:56.562 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.562 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.562 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.563 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.563 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-09-19T08:13:56.563 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.563 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.563 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.563 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.563 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.563 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.564 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.564 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.564 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.564 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.564 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.564 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.564 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.564 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.564 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.564 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.564 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.564 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.564 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.567 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.567 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.567 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.567 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.567 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.567 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.567 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.567 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.567 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.567 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.567 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.567 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.568 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.568 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.568 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.568 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.568 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.568 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.568 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.568 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.568 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.568 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.568 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-09-19T08:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-09-19T08:13:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-09-19T08:13:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-09-19T08:13:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-09-19T08:13:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-09-19T08:13:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-09-19T08:13:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-19T08:13:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-09-19T08:13:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-09-19T08:13:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-09-19T08:13:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-19T08:13:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-09-19T08:13:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-09-19T08:13:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-09-19T08:13:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-09-19T08:13:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-09-19T08:13:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-09-19T08:13:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-09-19T08:13:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-09-19T08:13:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-09-19T08:13:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-09-19T08:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-09-19T08:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-09-19T08:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-09-19T08:13:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-09-19T08:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-19T08:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-19T08:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.600 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-09-19T08:13:56.600 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-09-19T08:13:56.600 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.600 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.600 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.600 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.600 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.600 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.600 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.600 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.600 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.600 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.601 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.601 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-19T08:13:56.601 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-19T08:13:56.601 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.601 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.601 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.601 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-09-19T08:13:56.601 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.601 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.601 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.601 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.601 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.601 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.602 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.602 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-09-19T08:13:56.602 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-09-19T08:13:56.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:56.602 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.602 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.602 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-09-19T08:13:56.602 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.602 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.602 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.602 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.602 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.602 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.603 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.603 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-09-19T08:13:56.603 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-09-19T08:13:56.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.603 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.603 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.603 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-09-19T08:13:56.603 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.603 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.603 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.603 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-19T08:13:56.603 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.604 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.604 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.604 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-09-19T08:13:56.604 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-09-19T08:13:56.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.604 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.604 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.604 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-09-19T08:13:56.604 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.604 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.604 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.604 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-19T08:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-09-19T08:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-09-19T08:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-09-19T08:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.606 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.606 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.606 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.606 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-09-19T08:13:56.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-09-19T08:13:56.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.606 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.606 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.606 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-09-19T08:13:56.606 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.606 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.606 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.607 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-09-19T08:13:56.607 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-09-19T08:13:56.607 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.607 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.607 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.607 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.607 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.607 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-09-19T08:13:56.607 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-09-19T08:13:56.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.607 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.607 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.608 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-09-19T08:13:56.608 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-09-19T08:13:56.608 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-09-19T08:13:56.608 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.608 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.608 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.608 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.608 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.608 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.608 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.608 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-09-19T08:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.610 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.610 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.610 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.610 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.610 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.610 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.610 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.610 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.610 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.610 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.611 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.611 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.611 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.611 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.611 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.611 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.611 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.611 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.611 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.611 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.611 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.611 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.612 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.612 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.612 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.612 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.612 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.612 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.612 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.612 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.612 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.612 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.612 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.612 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.612 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.613 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.613 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.613 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.613 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.613 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.613 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.613 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.613 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.613 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.613 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.613 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.614 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.614 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.614 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.614 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.614 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-09-19T08:13:56.614 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-09-19T08:13:56.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.614 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.614 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.614 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-09-19T08:13:56.614 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.614 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.614 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.615 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.615 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.615 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.615 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.615 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-09-19T08:13:56.615 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-09-19T08:13:56.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.615 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.615 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.615 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.615 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.615 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.615 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.616 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.616 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.616 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.616 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-09-19T08:13:56.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-09-19T08:13:56.616 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.616 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.616 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.616 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.616 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.616 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.617 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.617 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.617 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.617 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.617 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.617 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-09-19T08:13:56.617 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.617 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.617 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.617 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-09-19T08:13:56.618 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-09-19T08:13:56.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.618 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.618 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.618 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.618 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.618 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.618 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.618 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.618 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.618 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.618 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.618 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-09-19T08:13:56.619 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-09-19T08:13:56.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.619 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.619 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.619 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.619 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.619 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.619 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.619 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.619 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.619 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.619 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.619 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-09-19T08:13:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-09-19T08:13:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-09-19T08:13:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-09-19T08:13:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-09-19T08:13:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "secondary_zone_period_retry_limit": { 2024-09-19T08:13:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-19T08:13:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-09-19T08:13:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-19T08:13:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-09-19T08:13:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-09-19T08:13:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-09-19T08:13:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-09-19T08:13:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-19T08:13:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-09-19T08:13:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.644 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.644 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.644 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.644 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.644 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.644 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.644 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.644 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.644 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.644 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.644 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.644 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.645 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.645 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.645 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.645 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.645 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.645 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.645 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.645 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.645 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.645 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.645 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.645 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.645 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.646 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.646 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.646 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.646 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.646 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.646 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.646 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.646 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.646 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.646 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.646 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.646 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.647 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-09-19T08:13:56.647 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-09-19T08:13:56.647 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.647 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.647 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.647 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.647 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.647 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.647 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.647 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.647 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.647 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.647 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.648 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-09-19T08:13:56.648 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-09-19T08:13:56.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.648 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.648 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.648 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-09-19T08:13:56.648 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.648 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.648 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.648 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.648 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.648 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.648 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.649 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-09-19T08:13:56.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-09-19T08:13:56.649 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.649 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.649 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.649 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.649 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.649 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.649 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.649 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.649 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.649 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.650 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.650 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-09-19T08:13:56.650 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-09-19T08:13:56.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.650 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.650 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.650 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.650 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.650 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.650 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.650 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.650 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.650 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.651 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.651 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-09-19T08:13:56.651 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-09-19T08:13:56.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-09-19T08:13:56.651 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.651 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.651 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.651 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.651 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.651 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.651 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.651 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.651 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.652 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.652 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-09-19T08:13:56.652 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-09-19T08:13:56.652 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.652 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.652 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.652 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.652 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.652 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.652 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.652 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.652 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.652 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.653 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.653 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-09-19T08:13:56.653 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-09-19T08:13:56.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.653 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.653 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.653 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.653 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.653 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.653 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.653 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.653 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.653 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.654 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.654 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-09-19T08:13:56.654 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-09-19T08:13:56.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.654 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.654 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.654 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.654 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.654 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.654 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.654 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.654 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.654 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.655 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.655 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-09-19T08:13:56.655 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-09-19T08:13:56.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.655 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.655 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.655 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.655 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-09-19T08:13:56.655 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-09-19T08:13:56.655 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.655 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.655 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.655 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.656 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.656 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-09-19T08:13:56.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-09-19T08:13:56.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.656 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.656 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.656 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.656 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.656 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.656 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.656 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.656 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.657 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.657 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.657 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-09-19T08:13:56.657 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-09-19T08:13:56.657 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.657 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.657 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.657 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.657 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.657 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.657 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.657 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.657 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.658 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.658 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.658 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-09-19T08:13:56.658 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-09-19T08:13:56.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.658 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.658 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.658 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.658 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.658 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.658 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.658 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.658 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.659 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.659 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.659 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.659 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-09-19T08:13:56.659 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.659 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.659 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.659 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-09-19T08:13:56.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-09-19T08:13:56.659 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.659 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.660 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.660 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.660 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.660 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.660 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.660 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-09-19T08:13:56.660 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.660 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.660 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.660 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-09-19T08:13:56.660 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-09-19T08:13:56.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.661 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.661 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.661 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.661 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.661 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.661 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.661 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-09-19T08:13:56.661 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.661 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.661 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.661 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.661 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.663 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.663 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.663 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.663 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.663 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.663 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.663 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.663 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.663 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.663 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.663 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.664 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.664 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.664 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.664 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.664 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.664 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.664 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.664 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.664 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.664 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.664 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.665 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.665 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.665 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.665 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.665 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.665 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.665 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.665 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.665 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.665 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.665 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.665 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.665 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.666 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.666 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.666 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.666 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.666 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.666 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.666 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.666 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.666 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.666 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.666 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.666 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-09-19T08:13:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.668 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.668 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.668 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.668 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.668 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.668 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.668 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.668 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.668 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.668 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.668 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.668 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.668 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.669 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.669 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.669 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.669 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.669 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.669 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.669 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.669 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.669 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.669 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.670 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.670 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.670 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.670 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.670 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.670 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.670 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.670 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.670 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.670 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.670 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.670 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.670 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.671 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.671 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.671 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.671 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.671 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.671 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.671 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.671 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.671 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.671 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.671 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.671 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.672 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.672 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.672 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.672 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.672 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.672 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.672 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.672 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.672 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.672 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-09-19T08:13:56.673 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.673 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.673 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.673 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.673 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.673 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.673 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.673 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.673 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.673 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.673 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.673 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.673 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.674 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.674 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.674 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.674 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.674 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.674 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.674 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.674 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.674 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.674 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.674 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.674 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.678 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.678 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.678 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.678 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-09-19T08:13:56.678 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.678 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.678 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.678 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-09-19T08:13:56.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-09-19T08:13:56.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.678 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-19T08:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.680 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.680 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-09-19T08:13:56.680 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.680 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.680 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.680 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.680 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.680 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.680 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.680 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.681 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.681 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.681 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.681 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.681 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.681 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.681 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.681 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.681 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.681 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.681 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.681 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.681 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.681 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.683 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.683 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.683 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.683 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.683 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.683 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.683 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.683 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.683 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.683 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.683 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.684 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.684 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.684 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.684 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.684 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.684 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.684 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.684 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.684 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.684 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.684 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.684 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.685 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.685 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.685 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.685 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.685 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.685 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.685 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.685 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.685 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.685 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.685 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.685 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-09-19T08:13:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-09-19T08:13:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-09-19T08:13:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-19T08:13:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-09-19T08:13:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-09-19T08:13:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.688 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.688 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-19T08:13:56.688 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.688 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.688 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.688 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-09-19T08:13:56.688 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-09-19T08:13:56.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.688 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.688 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.688 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.688 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.688 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.689 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.689 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-19T08:13:56.689 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.689 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.689 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.689 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-09-19T08:13:56.689 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-09-19T08:13:56.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.689 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.689 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.689 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.689 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-19T08:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-09-19T08:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-09-19T08:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.691 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.691 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.691 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-19T08:13:56.691 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.691 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.691 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.691 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-09-19T08:13:56.691 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-09-19T08:13:56.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.691 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.691 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.691 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.692 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.692 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.692 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.692 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.692 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.692 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.692 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.692 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-09-19T08:13:56.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-09-19T08:13:56.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.692 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.692 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.692 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-09-19T08:13:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-09-19T08:13:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-19T08:13:56.694 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.694 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.694 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.694 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.694 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.694 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.694 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.694 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-09-19T08:13:56.694 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-09-19T08:13:56.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.694 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.694 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.694 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.695 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.695 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.695 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.695 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.695 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.695 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.695 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.695 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:56.695 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:56.695 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.695 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.695 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.696 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-09-19T08:13:56.696 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-09-19T08:13:56.696 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.696 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.696 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.696 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.696 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.696 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.696 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-09-19T08:13:56.696 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-09-19T08:13:56.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.696 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.697 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.697 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-09-19T08:13:56.697 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.697 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.697 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.697 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.697 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.697 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.697 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.697 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-09-19T08:13:56.697 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-09-19T08:13:56.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.698 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.698 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.698 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.698 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.698 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.698 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.698 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.698 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.698 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.698 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.698 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-09-19T08:13:56.698 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-09-19T08:13:56.699 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.699 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.699 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.699 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.699 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.699 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.699 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.699 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.699 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.699 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.699 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.699 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.699 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.700 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.700 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.700 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.700 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.700 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.700 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.700 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.700 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.700 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.700 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.700 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.700 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.701 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.701 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.701 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.701 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.701 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.701 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.701 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.701 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.701 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.701 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.701 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.702 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.702 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.702 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.702 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.702 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.702 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.702 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.702 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.702 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.702 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.702 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.702 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.703 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.703 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.703 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.703 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.703 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.703 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.703 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.703 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.703 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.703 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.703 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.703 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.703 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.704 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.704 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.704 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.704 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.704 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.704 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.704 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.704 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.704 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.704 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.704 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.704 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.705 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-09-19T08:13:56.705 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-09-19T08:13:56.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.705 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.705 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.705 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.705 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.705 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.705 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.705 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.705 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.705 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.706 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.706 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-09-19T08:13:56.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-09-19T08:13:56.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.706 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.706 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.706 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.706 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.706 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.706 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.706 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.706 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.707 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.707 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.707 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-09-19T08:13:56.707 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-09-19T08:13:56.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.707 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.707 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.707 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-19T08:13:56.707 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.707 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.707 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.707 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.707 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.708 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.708 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.708 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.708 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-09-19T08:13:56.708 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.708 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.708 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.708 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.709 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.709 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.709 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.709 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.709 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.709 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.709 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.709 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.709 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.709 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.709 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.709 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.709 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.709 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-09-19T08:13:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-09-19T08:13:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-09-19T08:13:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-09-19T08:13:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-19T08:13:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-09-19T08:13:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-09-19T08:13:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-09-19T08:13:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-09-19T08:13:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-09-19T08:13:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-09-19T08:13:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-19T08:13:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-09-19T08:13:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-09-19T08:13:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-09-19T08:13:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-19T08:13:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-09-19T08:13:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-09-19T08:13:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-09-19T08:13:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-09-19T08:13:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-09-19T08:13:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-09-19T08:13:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-09-19T08:13:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-09-19T08:13:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-09-19T08:13:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-09-19T08:13:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-09-19T08:13:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-09-19T08:13:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-09-19T08:13:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-09-19T08:13:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-09-19T08:13:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-09-19T08:13:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-09-19T08:13:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-09-19T08:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-09-19T08:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-09-19T08:13:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-09-19T08:13:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-09-19T08:13:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-09-19T08:13:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-09-19T08:13:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-09-19T08:13:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-09-19T08:13:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-09-19T08:13:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-09-19T08:13:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-09-19T08:13:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-09-19T08:13:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-09-19T08:13:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-19T08:13:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-09-19T08:13:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-09-19T08:13:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-09-19T08:13:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-09-19T08:13:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-09-19T08:13:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-09-19T08:13:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-09-19T08:13:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-09-19T08:13:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-09-19T08:13:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-09-19T08:13:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-09-19T08:13:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-09-19T08:13:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-09-19T08:13:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-19T08:13:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-09-19T08:13:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-09-19T08:13:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-09-19T08:13:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-09-19T08:13:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-09-19T08:13:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-09-19T08:13:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-09-19T08:13:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "services": { 2024-09-19T08:13:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "dashboard": "https://172.21.15.98:8443/" 2024-09-19T08:13:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "always_on_modules": { 2024-09-19T08:13:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus": [ 2024-09-19T08:13:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-09-19T08:13:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-09-19T08:13:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-09-19T08:13:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-09-19T08:13:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-09-19T08:13:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-09-19T08:13:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-09-19T08:13:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-09-19T08:13:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-09-19T08:13:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-09-19T08:13:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific": [ 2024-09-19T08:13:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-09-19T08:13:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-09-19T08:13:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-09-19T08:13:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-09-19T08:13:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-09-19T08:13:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-09-19T08:13:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-09-19T08:13:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-09-19T08:13:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-09-19T08:13:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-09-19T08:13:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy": [ 2024-09-19T08:13:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-09-19T08:13:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-09-19T08:13:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-09-19T08:13:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-09-19T08:13:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-09-19T08:13:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-09-19T08:13:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-09-19T08:13:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-09-19T08:13:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-09-19T08:13:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-09-19T08:13:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "reef": [ 2024-09-19T08:13:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-09-19T08:13:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-09-19T08:13:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-09-19T08:13:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-09-19T08:13:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-09-19T08:13:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-09-19T08:13:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-09-19T08:13:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-09-19T08:13:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-09-19T08:13:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-09-19T08:13:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-19T08:13:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "squid": [ 2024-09-19T08:13:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-09-19T08:13:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-09-19T08:13:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-09-19T08:13:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-09-19T08:13:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-09-19T08:13:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-09-19T08:13:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-09-19T08:13:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-09-19T08:13:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-09-19T08:13:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-09-19T08:13:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-19T08:13:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "last_failure_osd_epoch": 81, 2024-09-19T08:13:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "active_clients": [ 2024-09-19T08:13:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-09-19T08:13:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-19T08:13:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-19T08:13:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-09-19T08:13:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3092825336 2024-09-19T08:13:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-19T08:13:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "libcephsqlite", 2024-09-19T08:13:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-19T08:13:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-19T08:13:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-09-19T08:13:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1480065576 2024-09-19T08:13:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-19T08:13:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-09-19T08:13:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-19T08:13:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-19T08:13:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-09-19T08:13:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1974442103 2024-09-19T08:13:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-19T08:13:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-09-19T08:13:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-19T08:13:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-19T08:13:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-09-19T08:13:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1240639074 2024-09-19T08:13:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-19T08:13:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-09-19T08:13:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-19T08:13:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-19T08:13:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-09-19T08:13:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2941739735 2024-09-19T08:13:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-19T08:13:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-09-19T08:13:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-19T08:13:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-19T08:13:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-19T08:13:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-09-19T08:13:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2105143026 2024-09-19T08:13:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-19T08:13:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-19T08:13:56.745 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:13:56.746 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:55 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3744113815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-19T08:13:56.746 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:55 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2152295873' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-19T08:13:56.747 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:54 smithi162 ceph-mon[35260]: pgmap v337: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-19T08:13:56.747 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:54 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3355813384' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:13:56.747 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:54 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3210257826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-19T08:13:56.747 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:54 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3677127544' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-19T08:13:56.747 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:55 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3744113815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-19T08:13:56.747 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:55 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2152295873' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-19T08:13:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.351+0000 7f42f51cf640 1 -- 172.21.15.98:0/2152295873 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f42dc0a3440 msgr2=0x7f42dc0a5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.351+0000 7f42f51cf640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f42dc0a3440 0x7f42dc0a5900 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f42f00ff710 tx=0x7f42d800b6f0 comp rx=0 tx=0).stop 2024-09-19T08:13:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.351+0000 7f42f51cf640 1 -- 172.21.15.98:0/2152295873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42f01003e0 msgr2=0x7f42f0198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:13:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.351+0000 7f42f51cf640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42f01003e0 0x7f42f0198460 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f42e0006010 tx=0x7f42e00042f0 comp rx=0 tx=0).stop 2024-09-19T08:13:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.351+0000 7f42f51cf640 1 -- 172.21.15.98:0/2152295873 shutdown_connections 2024-09-19T08:13:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.351+0000 7f42f51cf640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f42dc0a3440 0x7f42dc0a5900 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.351+0000 7f42f51cf640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f42f019cd20 0x7f42f019d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.351+0000 7f42f51cf640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42f01054e0 0x7f42f01989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.351+0000 7f42f51cf640 1 --2- 172.21.15.98:0/2152295873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42f01003e0 0x7f42f0198460 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:13:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.351+0000 7f42f51cf640 1 -- 172.21.15.98:0/2152295873 >> 172.21.15.98:0/2152295873 conn(0x7f42f00771a0 msgr2=0x7f42f0103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:13:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.351+0000 7f42f51cf640 1 -- 172.21.15.98:0/2152295873 shutdown_connections 2024-09-19T08:13:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:13:55.351+0000 7f42f51cf640 1 -- 172.21.15.98:0/2152295873 wait complete. 2024-09-19T08:13:56.805 DEBUG:tasks.check_counter:Getting stats from a 2024-09-19T08:13:56.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph daemon mds.a perf dump 2024-09-19T08:13:56.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:55 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3744113815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-19T08:13:56.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:55 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2152295873' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-19T08:13:57.067 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:57.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:56 smithi139 ceph-mon[34874]: pgmap v338: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:57.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:56 smithi098 ceph-mon[31333]: pgmap v338: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:57.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:56 smithi162 ceph-mon[35260]: pgmap v338: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:13:57.565 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:13:57.566 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-09-19T08:13:57.566 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1516, 2024-09-19T08:13:57.566 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 1516, 2024-09-19T08:13:57.566 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 456105, 2024-09-19T08:13:57.566 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 481265, 2024-09-19T08:13:57.566 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 6, 2024-09-19T08:13:57.566 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-09-19T08:13:57.566 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.145823500, 2024-09-19T08:13:57.566 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.046488555, 2024-09-19T08:13:57.566 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.040893973, 2024-09-19T08:13:57.567 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.046023031, 2024-09-19T08:13:57.567 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:13:57.567 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1515, 2024-09-19T08:13:57.567 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.059889637, 2024-09-19T08:13:57.567 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000039531 2024-09-19T08:13:57.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.567 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:13:57.567 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.567 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.567 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.567 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-09-19T08:13:57.568 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 448 2024-09-19T08:13:57.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.568 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-09-19T08:13:57.568 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 666, 2024-09-19T08:13:57.568 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 825, 2024-09-19T08:13:57.568 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 164316, 2024-09-19T08:13:57.568 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 659492, 2024-09-19T08:13:57.568 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 7, 2024-09-19T08:13:57.568 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 5, 2024-09-19T08:13:57.568 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.079147049, 2024-09-19T08:13:57.569 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.026160670, 2024-09-19T08:13:57.569 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.018894432, 2024-09-19T08:13:57.569 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.026877658, 2024-09-19T08:13:57.569 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:13:57.569 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 822, 2024-09-19T08:13:57.569 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.036485237, 2024-09-19T08:13:57.569 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000044385 2024-09-19T08:13:57.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.569 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:13:57.569 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.569 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.569 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.570 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-19T08:13:57.570 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 448272 2024-09-19T08:13:57.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.570 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-09-19T08:13:57.570 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 240, 2024-09-19T08:13:57.570 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 226, 2024-09-19T08:13:57.570 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1365596, 2024-09-19T08:13:57.570 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 131630, 2024-09-19T08:13:57.570 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 3, 2024-09-19T08:13:57.570 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-09-19T08:13:57.570 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.040153003, 2024-09-19T08:13:57.570 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.015275168, 2024-09-19T08:13:57.570 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.037203614, 2024-09-19T08:13:57.571 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000020714, 2024-09-19T08:13:57.571 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:13:57.571 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 225, 2024-09-19T08:13:57.571 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.012953593, 2024-09-19T08:13:57.571 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000057571 2024-09-19T08:13:57.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.571 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:13:57.571 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.571 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.571 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.571 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1365328, 2024-09-19T08:13:57.571 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 131312 2024-09-19T08:13:57.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-09-19T08:13:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-09-19T08:13:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-09-19T08:13:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-09-19T08:13:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-09-19T08:13:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-09-19T08:13:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 3440, 2024-09-19T08:13:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.098965620, 2024-09-19T08:13:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000028769 2024-09-19T08:13:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-09-19T08:13:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-09-19T08:13:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-09-19T08:13:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-09-19T08:13:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 0, 2024-09-19T08:13:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 0, 2024-09-19T08:13:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-09-19T08:13:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-09-19T08:13:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 0, 2024-09-19T08:13:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 0, 2024-09-19T08:13:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-09-19T08:13:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 0, 2024-09-19T08:13:57.575 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 0, 2024-09-19T08:13:57.575 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-09-19T08:13:57.575 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 0, 2024-09-19T08:13:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 0, 2024-09-19T08:13:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 0, 2024-09-19T08:13:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 0, 2024-09-19T08:13:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 0, 2024-09-19T08:13:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 0, 2024-09-19T08:13:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 0, 2024-09-19T08:13:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 0, 2024-09-19T08:13:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 1, 2024-09-19T08:13:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 0, 2024-09-19T08:13:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 0, 2024-09-19T08:13:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 0, 2024-09-19T08:13:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 0, 2024-09-19T08:13:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-09-19T08:13:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-09-19T08:13:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 0, 2024-09-19T08:13:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-09-19T08:13:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-09-19T08:13:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 0, 2024-09-19T08:13:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 0, 2024-09-19T08:13:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 0, 2024-09-19T08:13:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 0, 2024-09-19T08:13:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-09-19T08:13:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 0, 2024-09-19T08:13:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-09-19T08:13:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 0, 2024-09-19T08:13:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 0, 2024-09-19T08:13:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 0, 2024-09-19T08:13:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 0, 2024-09-19T08:13:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-09-19T08:13:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-09-19T08:13:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-09-19T08:13:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 0, 2024-09-19T08:13:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-09-19T08:13:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 0, 2024-09-19T08:13:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 0, 2024-09-19T08:13:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 0, 2024-09-19T08:13:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 0, 2024-09-19T08:13:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 0, 2024-09-19T08:13:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-09-19T08:13:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-19T08:13:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-19T08:13:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 0, 2024-09-19T08:13:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 0, 2024-09-19T08:13:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 0, 2024-09-19T08:13:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 0 2024-09-19T08:13:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-09-19T08:13:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 0, 2024-09-19T08:13:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 0, 2024-09-19T08:13:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 0, 2024-09-19T08:13:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 0, 2024-09-19T08:13:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 0, 2024-09-19T08:13:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 0, 2024-09-19T08:13:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-09-19T08:13:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-09-19T08:13:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 0, 2024-09-19T08:13:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 0, 2024-09-19T08:13:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-09-19T08:13:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-09-19T08:13:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-09-19T08:13:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-09-19T08:13:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-09-19T08:13:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-09-19T08:13:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-09-19T08:13:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 0, 2024-09-19T08:13:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 0, 2024-09-19T08:13:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-09-19T08:13:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 0, 2024-09-19T08:13:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-09-19T08:13:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 0, 2024-09-19T08:13:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-09-19T08:13:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-09-19T08:13:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-09-19T08:13:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-09-19T08:13:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 0, 2024-09-19T08:13:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-09-19T08:13:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-09-19T08:13:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 4, 2024-09-19T08:13:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-09-19T08:13:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-09-19T08:13:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 0, 2024-09-19T08:13:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-09-19T08:13:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-09-19T08:13:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 2, 2024-09-19T08:13:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 2, 2024-09-19T08:13:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-09-19T08:13:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-09-19T08:13:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 4194304, 2024-09-19T08:13:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 4195426, 2024-09-19T08:13:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 4195285, 2024-09-19T08:13:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-09-19T08:13:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 4 2024-09-19T08:13:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-09-19T08:13:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 3, 2024-09-19T08:13:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 3, 2024-09-19T08:13:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 0, 2024-09-19T08:13:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 1, 2024-09-19T08:13:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 1, 2024-09-19T08:13:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 0, 2024-09-19T08:13:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 0, 2024-09-19T08:13:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 0, 2024-09-19T08:13:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 0, 2024-09-19T08:13:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 0, 2024-09-19T08:13:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 0, 2024-09-19T08:13:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 0, 2024-09-19T08:13:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 48976, 2024-09-19T08:13:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 215328 2024-09-19T08:13:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-09-19T08:13:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 0, 2024-09-19T08:13:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 0, 2024-09-19T08:13:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 0, 2024-09-19T08:13:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 0, 2024-09-19T08:13:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 0, 2024-09-19T08:13:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-09-19T08:13:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-09-19T08:13:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-09-19T08:13:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-09-19T08:13:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-09-19T08:13:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-09-19T08:13:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-09-19T08:13:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-09-19T08:13:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-09-19T08:13:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-09-19T08:13:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-09-19T08:13:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-09-19T08:13:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-09-19T08:13:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-09-19T08:13:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-09-19T08:13:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-09-19T08:13:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-09-19T08:13:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-09-19T08:13:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.597 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.597 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.597 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-09-19T08:13:57.597 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.597 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.597 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.597 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-09-19T08:13:57.597 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.597 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.598 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.598 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-09-19T08:13:57.598 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.598 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.598 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.599 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-09-19T08:13:57.599 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.599 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.599 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.599 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-09-19T08:13:57.599 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.599 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.599 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.600 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-09-19T08:13:57.600 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.600 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.600 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.600 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-09-19T08:13:57.600 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.600 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.600 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.601 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-09-19T08:13:57.601 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.601 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.601 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.601 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-09-19T08:13:57.601 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.601 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.601 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.602 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-09-19T08:13:57.602 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.602 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.602 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.602 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-09-19T08:13:57.602 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.602 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.602 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.602 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-09-19T08:13:57.603 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-09-19T08:13:57.603 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-09-19T08:13:57.603 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.603 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.603 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.603 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:57.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.604 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-09-19T08:13:57.604 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 0, 2024-09-19T08:13:57.604 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 1, 2024-09-19T08:13:57.604 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 1, 2024-09-19T08:13:57.604 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 1, 2024-09-19T08:13:57.604 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-09-19T08:13:57.604 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 0, 2024-09-19T08:13:57.604 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 0, 2024-09-19T08:13:57.604 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 1019, 2024-09-19T08:13:57.604 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-09-19T08:13:57.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.605 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-09-19T08:13:57.605 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-09-19T08:13:57.605 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-09-19T08:13:57.605 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-09-19T08:13:57.605 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-09-19T08:13:57.605 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-09-19T08:13:57.605 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-09-19T08:13:57.605 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-19T08:13:57.605 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-09-19T08:13:57.606 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-19T08:13:57.606 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-09-19T08:13:57.606 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-09-19T08:13:57.606 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-09-19T08:13:57.606 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-19T08:13:57.606 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-09-19T08:13:57.606 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-09-19T08:13:57.606 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-09-19T08:13:57.606 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-09-19T08:13:57.606 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-09-19T08:13:57.606 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-19T08:13:57.606 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-09-19T08:13:57.607 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-19T08:13:57.607 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-09-19T08:13:57.607 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-09-19T08:13:57.607 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-09-19T08:13:57.607 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-09-19T08:13:57.607 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-09-19T08:13:57.607 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-19T08:13:57.607 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-09-19T08:13:57.607 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-09-19T08:13:57.607 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-09-19T08:13:57.607 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-09-19T08:13:57.608 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-09-19T08:13:57.608 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-09-19T08:13:57.608 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-09-19T08:13:57.608 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-09-19T08:13:57.608 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-09-19T08:13:57.608 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 189393, 2024-09-19T08:13:57.608 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 51, 2024-09-19T08:13:57.608 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-09-19T08:13:57.608 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-09-19T08:13:57.608 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-09-19T08:13:57.609 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-09-19T08:13:57.609 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-09-19T08:13:57.609 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-09-19T08:13:57.609 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-09-19T08:13:57.609 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-09-19T08:13:57.609 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 8136, 2024-09-19T08:13:57.609 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 180, 2024-09-19T08:13:57.609 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-09-19T08:13:57.609 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-09-19T08:13:57.609 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-09-19T08:13:57.609 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-09-19T08:13:57.609 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 8512, 2024-09-19T08:13:57.610 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 13, 2024-09-19T08:13:57.610 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-09-19T08:13:57.610 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-09-19T08:13:57.610 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-09-19T08:13:57.610 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-09-19T08:13:57.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.610 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-09-19T08:13:57.610 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-09-19T08:13:57.610 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-09-19T08:13:57.610 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 2177, 2024-09-19T08:13:57.610 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 21, 2024-09-19T08:13:57.611 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 3, 2024-09-19T08:13:57.611 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 2177, 2024-09-19T08:13:57.611 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-09-19T08:13:57.611 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2177, 2024-09-19T08:13:57.611 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 1.406982168, 2024-09-19T08:13:57.611 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000646294 2024-09-19T08:13:57.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.611 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-09-19T08:13:57.611 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-09-19T08:13:57.611 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2177, 2024-09-19T08:13:57.611 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 2178 2024-09-19T08:13:57.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.612 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 2177, 2024-09-19T08:13:57.612 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 2177, 2024-09-19T08:13:57.612 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 0, 2024-09-19T08:13:57.612 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-09-19T08:13:57.612 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-09-19T08:13:57.612 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 1415, 2024-09-19T08:13:57.612 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 0, 2024-09-19T08:13:57.612 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 760, 2024-09-19T08:13:57.612 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 0, 2024-09-19T08:13:57.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 0, 2024-09-19T08:13:57.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-09-19T08:13:57.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-09-19T08:13:57.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-09-19T08:13:57.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 0, 2024-09-19T08:13:57.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 0, 2024-09-19T08:13:57.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-09-19T08:13:57.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-09-19T08:13:57.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-09-19T08:13:57.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 1, 2024-09-19T08:13:57.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 0, 2024-09-19T08:13:57.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-09-19T08:13:57.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-09-19T08:13:57.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-09-19T08:13:57.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-09-19T08:13:57.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-09-19T08:13:57.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-09-19T08:13:57.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-09-19T08:13:57.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-09-19T08:13:57.615 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-09-19T08:13:57.615 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-09-19T08:13:57.615 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-09-19T08:13:57.615 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-09-19T08:13:57.615 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-09-19T08:13:57.615 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-09-19T08:13:57.615 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-09-19T08:13:57.615 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-09-19T08:13:57.615 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-09-19T08:13:57.615 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-09-19T08:13:57.615 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-09-19T08:13:57.616 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-09-19T08:13:57.616 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-09-19T08:13:57.616 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-09-19T08:13:57.616 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-09-19T08:13:57.616 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-09-19T08:13:57.616 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-09-19T08:13:57.616 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-09-19T08:13:57.616 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 89, 2024-09-19T08:13:57.616 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-09-19T08:13:57.616 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 11, 2024-09-19T08:13:57.616 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 5, 2024-09-19T08:13:57.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 5, 2024-09-19T08:13:57.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 0, 2024-09-19T08:13:57.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-09-19T08:13:57.617 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 0, 2024-09-19T08:13:57.617 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 2, 2024-09-19T08:13:57.617 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 0 2024-09-19T08:13:57.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.617 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-09-19T08:13:57.617 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 0, 2024-09-19T08:13:57.617 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-09-19T08:13:57.617 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-09-19T08:13:57.618 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-09-19T08:13:57.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.618 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-09-19T08:13:57.618 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 0 2024-09-19T08:13:57.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.618 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-09-19T08:13:57.618 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-09-19T08:13:57.618 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 0, 2024-09-19T08:13:57.618 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-09-19T08:13:57.618 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 0, 2024-09-19T08:13:57.618 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 0, 2024-09-19T08:13:57.619 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 0, 2024-09-19T08:13:57.619 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-09-19T08:13:57.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.619 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-19T08:13:57.619 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-09-19T08:13:57.619 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-09-19T08:13:57.619 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-09-19T08:13:57.619 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 2418, 2024-09-19T08:13:57.619 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1753100, 2024-09-19T08:13:57.619 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-09-19T08:13:57.620 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 2418, 2024-09-19T08:13:57.620 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-09-19T08:13:57.620 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-09-19T08:13:57.620 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 2418, 2024-09-19T08:13:57.620 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1753100, 2024-09-19T08:13:57.620 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-09-19T08:13:57.620 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.620 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.620 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:57.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.621 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-09-19T08:13:57.621 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-09-19T08:13:57.621 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-09-19T08:13:57.621 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-09-19T08:13:57.621 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-09-19T08:13:57.621 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-09-19T08:13:57.621 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-09-19T08:13:57.621 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-09-19T08:13:57.621 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 2177, 2024-09-19T08:13:57.622 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 1127, 2024-09-19T08:13:57.622 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 2177, 2024-09-19T08:13:57.622 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1127, 2024-09-19T08:13:57.622 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-09-19T08:13:57.622 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.622 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.622 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:57.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.622 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-09-19T08:13:57.622 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-09-19T08:13:57.623 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-09-19T08:13:57.623 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-09-19T08:13:57.623 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-09-19T08:13:57.623 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-09-19T08:13:57.623 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-09-19T08:13:57.623 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-09-19T08:13:57.623 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 2177, 2024-09-19T08:13:57.623 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 2177, 2024-09-19T08:13:57.623 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 2177, 2024-09-19T08:13:57.623 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 2177, 2024-09-19T08:13:57.623 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-09-19T08:13:57.624 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.624 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.624 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:57.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.624 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-09-19T08:13:57.624 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-09-19T08:13:57.624 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-09-19T08:13:57.624 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-09-19T08:13:57.625 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-09-19T08:13:57.625 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-09-19T08:13:57.625 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-09-19T08:13:57.625 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-09-19T08:13:57.625 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-09-19T08:13:57.625 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-09-19T08:13:57.625 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-09-19T08:13:57.625 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-09-19T08:13:57.625 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-09-19T08:13:57.625 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.625 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.625 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.626 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:57.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:57.626 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x562bf7ce4b40": { 2024-09-19T08:13:57.626 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-09-19T08:13:57.626 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-09-19T08:13:57.626 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-09-19T08:13:57.626 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-09-19T08:13:57.626 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-09-19T08:13:57.626 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-09-19T08:13:57.626 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-09-19T08:13:57.626 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-09-19T08:13:57.627 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-09-19T08:13:57.627 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-09-19T08:13:57.627 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-09-19T08:13:57.627 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-09-19T08:13:57.627 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:57.627 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:57.627 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:57.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:57.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:57.627 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:13:58.322 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-09-19T08:13:58.322 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-09-19T08:13:58.322 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-09-19T08:13:58.322 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-09-19T08:13:58.322 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-09-19T08:13:58.323 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-09-19T08:13:58.323 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-09-19T08:13:58.323 DEBUG:tasks.check_counter:Getting stats from d 2024-09-19T08:13:58.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph daemon mds.d perf dump 2024-09-19T08:13:58.567 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:13:58.947 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:13:58 smithi098 ceph-mon[31333]: pgmap v339: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:13:59.078 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:13:59.078 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-09-19T08:13:59.079 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-09-19T08:13:59.079 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-09-19T08:13:59.079 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-09-19T08:13:59.079 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-09-19T08:13:59.079 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-09-19T08:13:59.079 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-09-19T08:13:59.079 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001050781, 2024-09-19T08:13:59.079 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-09-19T08:13:59.079 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000171740, 2024-09-19T08:13:59.079 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-19T08:13:59.079 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:13:59.079 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:59.080 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:59.080 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:59.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:59.080 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:13:59.080 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:59.080 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:59.080 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:59.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:59.080 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-19T08:13:59.080 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-09-19T08:13:59.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:59.081 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-09-19T08:13:59.081 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 6, 2024-09-19T08:13:59.081 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 167, 2024-09-19T08:13:59.081 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 3392, 2024-09-19T08:13:59.081 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 186848, 2024-09-19T08:13:59.081 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-09-19T08:13:59.081 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-09-19T08:13:59.081 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.015436046, 2024-09-19T08:13:59.081 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.009397483, 2024-09-19T08:13:59.081 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.003687764, 2024-09-19T08:13:59.081 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-19T08:13:59.081 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:13:59.082 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 165, 2024-09-19T08:13:59.082 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.021990929, 2024-09-19T08:13:59.082 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000133278 2024-09-19T08:13:59.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:59.082 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:13:59.082 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:59.082 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:59.082 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:59.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:59.082 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-09-19T08:13:59.082 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 186848 2024-09-19T08:13:59.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:59.083 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-09-19T08:13:59.083 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 223, 2024-09-19T08:13:59.083 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 214, 2024-09-19T08:13:59.083 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1310560, 2024-09-19T08:13:59.083 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 131744, 2024-09-19T08:13:59.083 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-09-19T08:13:59.083 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-09-19T08:13:59.083 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.037997710, 2024-09-19T08:13:59.083 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.015351796, 2024-09-19T08:13:59.083 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.032484805, 2024-09-19T08:13:59.083 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-19T08:13:59.083 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:13:59.084 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 213, 2024-09-19T08:13:59.084 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.014098736, 2024-09-19T08:13:59.084 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000066191 2024-09-19T08:13:59.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:59.084 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:13:59.084 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:59.084 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:59.084 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:59.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:59.084 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1310560, 2024-09-19T08:13:59.084 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 131744 2024-09-19T08:13:59.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:59.084 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-09-19T08:13:59.085 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-09-19T08:13:59.085 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-09-19T08:13:59.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:59.085 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-09-19T08:13:59.085 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-09-19T08:13:59.085 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-09-19T08:13:59.085 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-09-19T08:13:59.085 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-09-19T08:13:59.085 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-09-19T08:13:59.085 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-09-19T08:13:59.086 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-19T08:13:59.086 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-09-19T08:13:59.086 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-19T08:13:59.086 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-09-19T08:13:59.086 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-09-19T08:13:59.086 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-09-19T08:13:59.086 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-19T08:13:59.086 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-09-19T08:13:59.086 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-09-19T08:13:59.086 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-09-19T08:13:59.086 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-09-19T08:13:59.086 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-09-19T08:13:59.087 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-19T08:13:59.087 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-09-19T08:13:59.087 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-19T08:13:59.087 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-09-19T08:13:59.087 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-09-19T08:13:59.087 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-09-19T08:13:59.087 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-09-19T08:13:59.087 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-09-19T08:13:59.087 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-19T08:13:59.087 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-09-19T08:13:59.087 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-09-19T08:13:59.087 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-09-19T08:13:59.087 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-09-19T08:13:59.088 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-09-19T08:13:59.088 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-09-19T08:13:59.088 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-09-19T08:13:59.088 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-09-19T08:13:59.088 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-09-19T08:13:59.088 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148977, 2024-09-19T08:13:59.088 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 30, 2024-09-19T08:13:59.088 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-09-19T08:13:59.088 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-09-19T08:13:59.088 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-09-19T08:13:59.088 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-09-19T08:13:59.088 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-09-19T08:13:59.088 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-09-19T08:13:59.089 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-09-19T08:13:59.089 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-09-19T08:13:59.089 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-09-19T08:13:59.089 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-09-19T08:13:59.089 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-09-19T08:13:59.089 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-09-19T08:13:59.089 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-09-19T08:13:59.089 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-09-19T08:13:59.089 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-09-19T08:13:59.089 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-09-19T08:13:59.089 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-09-19T08:13:59.089 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-09-19T08:13:59.090 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-09-19T08:13:59.090 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-09-19T08:13:59.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:13:59.090 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-19T08:13:59.090 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-09-19T08:13:59.090 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-09-19T08:13:59.090 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-09-19T08:13:59.090 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 225, 2024-09-19T08:13:59.090 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1279104, 2024-09-19T08:13:59.090 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-09-19T08:13:59.090 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 225, 2024-09-19T08:13:59.090 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-09-19T08:13:59.090 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-09-19T08:13:59.091 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 225, 2024-09-19T08:13:59.091 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1279104, 2024-09-19T08:13:59.091 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-09-19T08:13:59.091 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:13:59.091 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:13:59.091 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:13:59.091 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:59.091 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:13:59.091 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:13:59.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:13:58 smithi139 ceph-mon[34874]: pgmap v339: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:13:59.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:13:58 smithi162 ceph-mon[35260]: pgmap v339: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-19T08:13:59.755 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-09-19T08:13:59.755 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-09-19T08:13:59.756 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-09-19T08:13:59.756 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-09-19T08:13:59.756 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-09-19T08:13:59.756 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-09-19T08:13:59.756 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-09-19T08:13:59.756 DEBUG:tasks.check_counter:Getting stats from g 2024-09-19T08:13:59.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph daemon mds.g perf dump 2024-09-19T08:14:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:14:00.508 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:14:00.508 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-09-19T08:14:00.508 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1003, 2024-09-19T08:14:00.508 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 986, 2024-09-19T08:14:00.508 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1623722, 2024-09-19T08:14:00.508 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 373766, 2024-09-19T08:14:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-09-19T08:14:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-09-19T08:14:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.122612788, 2024-09-19T08:14:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.050802402, 2024-09-19T08:14:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.055367547, 2024-09-19T08:14:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.019234964, 2024-09-19T08:14:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 986, 2024-09-19T08:14:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.056179044, 2024-09-19T08:14:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000056976 2024-09-19T08:14:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1368944, 2024-09-19T08:14:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 133040 2024-09-19T08:14:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-09-19T08:14:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1421, 2024-09-19T08:14:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 1581, 2024-09-19T08:14:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 362692, 2024-09-19T08:14:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 899747, 2024-09-19T08:14:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 7, 2024-09-19T08:14:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 5, 2024-09-19T08:14:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.097456165, 2024-09-19T08:14:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.031762386, 2024-09-19T08:14:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.024778023, 2024-09-19T08:14:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.032647774, 2024-09-19T08:14:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1578, 2024-09-19T08:14:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.057227737, 2024-09-19T08:14:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000036265 2024-09-19T08:14:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-19T08:14:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 448448 2024-09-19T08:14:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-09-19T08:14:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 4, 2024-09-19T08:14:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 6, 2024-09-19T08:14:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1629, 2024-09-19T08:14:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 1284, 2024-09-19T08:14:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-09-19T08:14:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 4, 2024-09-19T08:14:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001178453, 2024-09-19T08:14:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000065358, 2024-09-19T08:14:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000933063, 2024-09-19T08:14:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000072578, 2024-09-19T08:14:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 4, 2024-09-19T08:14:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.002092563, 2024-09-19T08:14:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000523140 2024-09-19T08:14:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-19T08:14:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-09-19T08:14:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-09-19T08:14:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-09-19T08:14:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-09-19T08:14:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-09-19T08:14:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-09-19T08:14:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-09-19T08:14:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 3658, 2024-09-19T08:14:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.080613606, 2024-09-19T08:14:00.515 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000022037 2024-09-19T08:14:00.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:14:00.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.515 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-09-19T08:14:00.515 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-09-19T08:14:00.515 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-09-19T08:14:00.515 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.515 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.515 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:14:00.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.515 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-09-19T08:14:00.516 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 0, 2024-09-19T08:14:00.516 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 0, 2024-09-19T08:14:00.516 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-09-19T08:14:00.516 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.516 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.516 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.516 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-09-19T08:14:00.516 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 0, 2024-09-19T08:14:00.516 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 0, 2024-09-19T08:14:00.516 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-09-19T08:14:00.516 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 0, 2024-09-19T08:14:00.516 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 0, 2024-09-19T08:14:00.517 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-09-19T08:14:00.517 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 0, 2024-09-19T08:14:00.517 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 0, 2024-09-19T08:14:00.517 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 0, 2024-09-19T08:14:00.517 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 0, 2024-09-19T08:14:00.517 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 0, 2024-09-19T08:14:00.517 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 0, 2024-09-19T08:14:00.517 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 0, 2024-09-19T08:14:00.517 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 0, 2024-09-19T08:14:00.517 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 1, 2024-09-19T08:14:00.517 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 0, 2024-09-19T08:14:00.517 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 0, 2024-09-19T08:14:00.517 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 0, 2024-09-19T08:14:00.518 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 0, 2024-09-19T08:14:00.518 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-09-19T08:14:00.518 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-09-19T08:14:00.518 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 0, 2024-09-19T08:14:00.518 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-09-19T08:14:00.518 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-09-19T08:14:00.518 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 0, 2024-09-19T08:14:00.518 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 0, 2024-09-19T08:14:00.518 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 0, 2024-09-19T08:14:00.518 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 0, 2024-09-19T08:14:00.518 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-09-19T08:14:00.518 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 0, 2024-09-19T08:14:00.518 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-09-19T08:14:00.519 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 0, 2024-09-19T08:14:00.519 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 0, 2024-09-19T08:14:00.519 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 0, 2024-09-19T08:14:00.519 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 0, 2024-09-19T08:14:00.519 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-09-19T08:14:00.519 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-09-19T08:14:00.519 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-09-19T08:14:00.519 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 0, 2024-09-19T08:14:00.519 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-09-19T08:14:00.519 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 0, 2024-09-19T08:14:00.519 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 0, 2024-09-19T08:14:00.519 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 0, 2024-09-19T08:14:00.519 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 0, 2024-09-19T08:14:00.519 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 0, 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 0, 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 0, 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 0, 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 0 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 0, 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 0, 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 0, 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 0, 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 0, 2024-09-19T08:14:00.520 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 0, 2024-09-19T08:14:00.521 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-09-19T08:14:00.521 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-09-19T08:14:00.521 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 0, 2024-09-19T08:14:00.521 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 0, 2024-09-19T08:14:00.521 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-09-19T08:14:00.521 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-09-19T08:14:00.521 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-09-19T08:14:00.521 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-09-19T08:14:00.521 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-09-19T08:14:00.521 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-09-19T08:14:00.521 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-09-19T08:14:00.521 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 0, 2024-09-19T08:14:00.521 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 0, 2024-09-19T08:14:00.522 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-09-19T08:14:00.522 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 0, 2024-09-19T08:14:00.522 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-09-19T08:14:00.522 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 0, 2024-09-19T08:14:00.522 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-09-19T08:14:00.522 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-09-19T08:14:00.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.522 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-09-19T08:14:00.522 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-09-19T08:14:00.522 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 0, 2024-09-19T08:14:00.522 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-09-19T08:14:00.522 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-09-19T08:14:00.523 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 4, 2024-09-19T08:14:00.523 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-09-19T08:14:00.523 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-09-19T08:14:00.523 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 0, 2024-09-19T08:14:00.523 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-09-19T08:14:00.523 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-09-19T08:14:00.523 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 2, 2024-09-19T08:14:00.523 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 2, 2024-09-19T08:14:00.523 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-09-19T08:14:00.523 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-09-19T08:14:00.523 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 4194304, 2024-09-19T08:14:00.523 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 4195426, 2024-09-19T08:14:00.524 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 4195285, 2024-09-19T08:14:00.524 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-09-19T08:14:00.524 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.524 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.524 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.524 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 4 2024-09-19T08:14:00.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.524 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-09-19T08:14:00.524 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 3, 2024-09-19T08:14:00.524 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 3, 2024-09-19T08:14:00.524 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 0, 2024-09-19T08:14:00.524 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 1, 2024-09-19T08:14:00.525 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 1, 2024-09-19T08:14:00.525 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 0, 2024-09-19T08:14:00.525 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 0, 2024-09-19T08:14:00.525 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 0, 2024-09-19T08:14:00.525 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 0, 2024-09-19T08:14:00.525 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 0, 2024-09-19T08:14:00.525 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 0, 2024-09-19T08:14:00.525 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 0, 2024-09-19T08:14:00.525 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 48804, 2024-09-19T08:14:00.526 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 215328 2024-09-19T08:14:00.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.526 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-09-19T08:14:00.526 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 0, 2024-09-19T08:14:00.526 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 0, 2024-09-19T08:14:00.526 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 0, 2024-09-19T08:14:00.526 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 0, 2024-09-19T08:14:00.526 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 0, 2024-09-19T08:14:00.526 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-09-19T08:14:00.526 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.526 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.526 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.527 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-09-19T08:14:00.527 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.527 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.527 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.527 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-09-19T08:14:00.527 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.527 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.527 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.527 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-09-19T08:14:00.527 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.527 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.528 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.529 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.529 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-09-19T08:14:00.529 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.529 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.529 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.529 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-09-19T08:14:00.529 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.529 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.529 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.529 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-09-19T08:14:00.529 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.530 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.530 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.530 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-09-19T08:14:00.530 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.530 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.530 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.530 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-09-19T08:14:00.530 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.531 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.531 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.531 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-09-19T08:14:00.531 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.531 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.531 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.532 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-09-19T08:14:00.532 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.532 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.532 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.532 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-09-19T08:14:00.532 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.532 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.532 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.532 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-09-19T08:14:00.533 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.533 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.533 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.533 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-09-19T08:14:00.533 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.533 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.533 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.533 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-09-19T08:14:00.533 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.533 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.533 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.534 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-09-19T08:14:00.534 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.534 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.534 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.534 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-09-19T08:14:00.534 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.534 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.534 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.534 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-09-19T08:14:00.535 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.535 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.535 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.535 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-09-19T08:14:00.535 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.535 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.535 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.535 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-09-19T08:14:00.535 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.535 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.535 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.536 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-09-19T08:14:00.536 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.536 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.536 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.536 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-09-19T08:14:00.536 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.536 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.536 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.536 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-09-19T08:14:00.536 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.537 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.537 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.537 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-09-19T08:14:00.537 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.537 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.537 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.537 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-09-19T08:14:00.537 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.537 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.537 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.538 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-09-19T08:14:00.538 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.538 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.538 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.538 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-09-19T08:14:00.538 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-09-19T08:14:00.538 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-09-19T08:14:00.538 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.538 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.538 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:14:00.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.538 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-09-19T08:14:00.539 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 0, 2024-09-19T08:14:00.539 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 1, 2024-09-19T08:14:00.539 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 1, 2024-09-19T08:14:00.539 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 1, 2024-09-19T08:14:00.539 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-09-19T08:14:00.539 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 0, 2024-09-19T08:14:00.539 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 0, 2024-09-19T08:14:00.539 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 1116, 2024-09-19T08:14:00.539 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-09-19T08:14:00.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.539 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-09-19T08:14:00.539 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-09-19T08:14:00.539 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-09-19T08:14:00.539 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-09-19T08:14:00.540 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-09-19T08:14:00.540 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-09-19T08:14:00.540 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-09-19T08:14:00.540 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-19T08:14:00.540 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-09-19T08:14:00.540 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-19T08:14:00.540 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-09-19T08:14:00.540 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-09-19T08:14:00.541 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-09-19T08:14:00.541 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-19T08:14:00.541 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-09-19T08:14:00.541 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-09-19T08:14:00.541 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-09-19T08:14:00.541 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-09-19T08:14:00.541 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-09-19T08:14:00.541 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-19T08:14:00.541 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-09-19T08:14:00.541 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-19T08:14:00.541 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-09-19T08:14:00.541 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-09-19T08:14:00.541 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-09-19T08:14:00.542 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-09-19T08:14:00.542 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-09-19T08:14:00.542 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-19T08:14:00.542 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-09-19T08:14:00.542 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-09-19T08:14:00.542 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-09-19T08:14:00.542 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-09-19T08:14:00.542 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-09-19T08:14:00.542 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-09-19T08:14:00.542 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-09-19T08:14:00.542 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-09-19T08:14:00.542 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-09-19T08:14:00.542 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 197473, 2024-09-19T08:14:00.543 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 55, 2024-09-19T08:14:00.543 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-09-19T08:14:00.543 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-09-19T08:14:00.543 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-09-19T08:14:00.543 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-09-19T08:14:00.543 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-09-19T08:14:00.543 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-09-19T08:14:00.543 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-09-19T08:14:00.543 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-09-19T08:14:00.543 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 8136, 2024-09-19T08:14:00.543 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 180, 2024-09-19T08:14:00.543 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-09-19T08:14:00.543 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-09-19T08:14:00.544 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-09-19T08:14:00.544 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-09-19T08:14:00.544 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 8512, 2024-09-19T08:14:00.544 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 13, 2024-09-19T08:14:00.544 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-09-19T08:14:00.544 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-09-19T08:14:00.544 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-09-19T08:14:00.544 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-09-19T08:14:00.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.545 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-09-19T08:14:00.545 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-09-19T08:14:00.545 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-09-19T08:14:00.545 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 2180, 2024-09-19T08:14:00.545 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 21, 2024-09-19T08:14:00.545 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 3, 2024-09-19T08:14:00.545 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 2180, 2024-09-19T08:14:00.545 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-09-19T08:14:00.545 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2180, 2024-09-19T08:14:00.546 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 1.623979336, 2024-09-19T08:14:00.546 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000744944 2024-09-19T08:14:00.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.546 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-09-19T08:14:00.546 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-09-19T08:14:00.546 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2180, 2024-09-19T08:14:00.546 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 2181 2024-09-19T08:14:00.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.546 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 2180, 2024-09-19T08:14:00.546 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 2180, 2024-09-19T08:14:00.546 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 0, 2024-09-19T08:14:00.546 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-09-19T08:14:00.547 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-09-19T08:14:00.547 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 1415, 2024-09-19T08:14:00.547 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 0, 2024-09-19T08:14:00.547 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 763, 2024-09-19T08:14:00.547 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 0, 2024-09-19T08:14:00.547 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 0, 2024-09-19T08:14:00.547 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-09-19T08:14:00.547 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-09-19T08:14:00.547 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-09-19T08:14:00.547 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 0, 2024-09-19T08:14:00.547 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 0, 2024-09-19T08:14:00.547 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-09-19T08:14:00.547 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-09-19T08:14:00.548 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-09-19T08:14:00.548 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 1, 2024-09-19T08:14:00.548 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 0, 2024-09-19T08:14:00.548 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-09-19T08:14:00.548 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-09-19T08:14:00.548 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-09-19T08:14:00.548 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-09-19T08:14:00.548 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-09-19T08:14:00.548 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-09-19T08:14:00.548 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-09-19T08:14:00.548 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-09-19T08:14:00.549 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-09-19T08:14:00.549 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-09-19T08:14:00.549 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-09-19T08:14:00.549 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-09-19T08:14:00.549 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-09-19T08:14:00.549 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-09-19T08:14:00.549 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-09-19T08:14:00.549 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-09-19T08:14:00.549 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-09-19T08:14:00.549 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-09-19T08:14:00.549 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-09-19T08:14:00.549 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-09-19T08:14:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-09-19T08:14:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-09-19T08:14:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-09-19T08:14:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-09-19T08:14:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-09-19T08:14:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-09-19T08:14:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 89, 2024-09-19T08:14:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-09-19T08:14:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 11, 2024-09-19T08:14:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 6, 2024-09-19T08:14:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 6, 2024-09-19T08:14:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 0, 2024-09-19T08:14:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-09-19T08:14:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 0, 2024-09-19T08:14:00.551 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 2, 2024-09-19T08:14:00.551 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 0 2024-09-19T08:14:00.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.551 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-09-19T08:14:00.551 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 0, 2024-09-19T08:14:00.551 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-09-19T08:14:00.551 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-09-19T08:14:00.551 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-09-19T08:14:00.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.551 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-09-19T08:14:00.552 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 0 2024-09-19T08:14:00.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.552 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-09-19T08:14:00.552 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-09-19T08:14:00.552 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 0, 2024-09-19T08:14:00.552 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-09-19T08:14:00.552 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 0, 2024-09-19T08:14:00.552 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 0, 2024-09-19T08:14:00.552 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 0, 2024-09-19T08:14:00.552 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-09-19T08:14:00.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.552 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-19T08:14:00.552 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-09-19T08:14:00.553 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-09-19T08:14:00.553 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-09-19T08:14:00.553 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 2424, 2024-09-19T08:14:00.553 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1754482, 2024-09-19T08:14:00.553 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:00.553 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 2424, 2024-09-19T08:14:00.553 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-09-19T08:14:00.553 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-09-19T08:14:00.553 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 2424, 2024-09-19T08:14:00.553 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1754482, 2024-09-19T08:14:00.553 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-09-19T08:14:00.553 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.553 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.554 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.554 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:14:00.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.554 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-09-19T08:14:00.554 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-09-19T08:14:00.554 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-09-19T08:14:00.554 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-09-19T08:14:00.554 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-09-19T08:14:00.554 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-09-19T08:14:00.554 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:00.554 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:00.554 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 2180, 2024-09-19T08:14:00.554 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 1127, 2024-09-19T08:14:00.555 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 2180, 2024-09-19T08:14:00.555 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1127, 2024-09-19T08:14:00.555 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-09-19T08:14:00.555 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.555 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.555 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.555 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:14:00.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.555 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-09-19T08:14:00.555 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-09-19T08:14:00.555 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-09-19T08:14:00.555 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-09-19T08:14:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-09-19T08:14:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-09-19T08:14:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 2180, 2024-09-19T08:14:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 2180, 2024-09-19T08:14:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 2180, 2024-09-19T08:14:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 2180, 2024-09-19T08:14:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-09-19T08:14:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:14:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.557 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-09-19T08:14:00.557 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-09-19T08:14:00.557 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-09-19T08:14:00.557 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-09-19T08:14:00.557 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-09-19T08:14:00.557 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-09-19T08:14:00.557 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:00.557 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:00.557 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-09-19T08:14:00.557 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-09-19T08:14:00.557 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-09-19T08:14:00.558 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-09-19T08:14:00.558 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-09-19T08:14:00.558 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.558 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.558 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.558 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:14:00.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:00.558 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x55ce9cc0eb40": { 2024-09-19T08:14:00.558 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-09-19T08:14:00.558 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-09-19T08:14:00.558 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-09-19T08:14:00.558 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-09-19T08:14:00.558 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-09-19T08:14:00.559 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:00.559 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:00.559 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-09-19T08:14:00.559 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-09-19T08:14:00.559 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-09-19T08:14:00.559 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-09-19T08:14:00.559 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-09-19T08:14:00.559 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:00.559 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:00.559 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:00.559 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:14:00.559 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:14:00.559 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:14:01.129 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:00 smithi098 ceph-mon[31333]: pgmap v340: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:14:01.155 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-09-19T08:14:01.155 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-09-19T08:14:01.155 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-09-19T08:14:01.156 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-09-19T08:14:01.156 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-09-19T08:14:01.156 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-09-19T08:14:01.156 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-09-19T08:14:01.156 DEBUG:tasks.check_counter:Getting stats from j 2024-09-19T08:14:01.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph daemon mds.j perf dump 2024-09-19T08:14:01.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:00 smithi139 ceph-mon[34874]: pgmap v340: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:14:01.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:00 smithi162 ceph-mon[35260]: pgmap v340: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-19T08:14:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:14:01.900 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-19T08:14:01.901 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-09-19T08:14:01.901 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-09-19T08:14:01.901 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-09-19T08:14:01.901 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-09-19T08:14:01.901 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-09-19T08:14:01.901 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-09-19T08:14:01.901 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-09-19T08:14:01.901 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001078862, 2024-09-19T08:14:01.901 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-09-19T08:14:01.901 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000144220, 2024-09-19T08:14:01.902 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-19T08:14:01.902 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:01.902 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:01.902 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:01.902 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:01.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:01.902 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:01.902 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:01.902 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:01.902 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:01.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:01.902 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-19T08:14:01.903 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-09-19T08:14:01.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:01.903 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-09-19T08:14:01.903 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 229, 2024-09-19T08:14:01.903 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 218, 2024-09-19T08:14:01.903 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1314176, 2024-09-19T08:14:01.903 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 133472, 2024-09-19T08:14:01.903 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-09-19T08:14:01.903 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-09-19T08:14:01.903 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.043075769, 2024-09-19T08:14:01.903 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.016174106, 2024-09-19T08:14:01.903 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.037535951, 2024-09-19T08:14:01.904 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-19T08:14:01.904 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:01.904 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 218, 2024-09-19T08:14:01.904 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.016695838, 2024-09-19T08:14:01.904 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000076586 2024-09-19T08:14:01.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:01.904 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:01.904 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:01.904 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:01.904 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:01.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:01.904 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1314176, 2024-09-19T08:14:01.905 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 133472 2024-09-19T08:14:01.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:01.905 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-09-19T08:14:01.905 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 2, 2024-09-19T08:14:01.905 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 166, 2024-09-19T08:14:01.905 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 320, 2024-09-19T08:14:01.905 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 186576, 2024-09-19T08:14:01.905 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-09-19T08:14:01.905 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-09-19T08:14:01.905 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.013846796, 2024-09-19T08:14:01.905 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.009546863, 2024-09-19T08:14:01.905 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.002079788, 2024-09-19T08:14:01.905 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-19T08:14:01.906 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:01.906 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 163, 2024-09-19T08:14:01.906 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.015920613, 2024-09-19T08:14:01.906 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000097672 2024-09-19T08:14:01.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:01.906 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:01.906 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:01.906 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:01.906 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:01.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:01.906 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-19T08:14:01.906 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 186576 2024-09-19T08:14:01.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:01.907 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-09-19T08:14:01.907 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-09-19T08:14:01.907 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-09-19T08:14:01.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:01.907 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-09-19T08:14:01.907 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-09-19T08:14:01.907 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-09-19T08:14:01.907 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-09-19T08:14:01.907 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-09-19T08:14:01.907 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-09-19T08:14:01.907 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-09-19T08:14:01.907 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-19T08:14:01.907 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-09-19T08:14:01.908 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-19T08:14:01.908 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-09-19T08:14:01.908 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-09-19T08:14:01.908 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-09-19T08:14:01.908 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-19T08:14:01.908 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-09-19T08:14:01.908 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-09-19T08:14:01.908 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-09-19T08:14:01.908 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-09-19T08:14:01.908 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-09-19T08:14:01.908 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-19T08:14:01.908 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-09-19T08:14:01.908 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-19T08:14:01.909 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-09-19T08:14:01.909 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-09-19T08:14:01.909 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-09-19T08:14:01.909 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-09-19T08:14:01.909 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-09-19T08:14:01.909 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-19T08:14:01.909 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-09-19T08:14:01.909 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-09-19T08:14:01.909 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-09-19T08:14:01.909 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-09-19T08:14:01.909 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-09-19T08:14:01.909 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-09-19T08:14:01.909 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-09-19T08:14:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-09-19T08:14:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-09-19T08:14:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148995, 2024-09-19T08:14:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 31, 2024-09-19T08:14:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-09-19T08:14:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-09-19T08:14:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-09-19T08:14:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-09-19T08:14:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-09-19T08:14:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-09-19T08:14:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-09-19T08:14:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-09-19T08:14:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-09-19T08:14:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-09-19T08:14:01.911 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-09-19T08:14:01.911 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-09-19T08:14:01.911 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-09-19T08:14:01.911 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-09-19T08:14:01.911 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-09-19T08:14:01.911 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-09-19T08:14:01.911 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-09-19T08:14:01.911 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-09-19T08:14:01.911 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-09-19T08:14:01.911 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-09-19T08:14:01.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-19T08:14:01.911 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-19T08:14:01.911 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-09-19T08:14:01.912 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-09-19T08:14:01.912 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-09-19T08:14:01.912 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 227, 2024-09-19T08:14:01.912 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1279364, 2024-09-19T08:14:01.912 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:01.912 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 227, 2024-09-19T08:14:01.912 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-09-19T08:14:01.912 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-09-19T08:14:01.912 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 227, 2024-09-19T08:14:01.912 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1279364, 2024-09-19T08:14:01.913 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-09-19T08:14:01.913 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-09-19T08:14:01.913 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-09-19T08:14:01.913 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-09-19T08:14:01.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:14:01.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-19T08:14:01.913 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-19T08:14:02.627 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-09-19T08:14:02.627 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-09-19T08:14:02.627 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-09-19T08:14:02.627 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-09-19T08:14:02.627 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-09-19T08:14:02.627 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-09-19T08:14:02.627 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-09-19T08:14:02.627 DEBUG:tasks.check_counter:Getting stats from b 2024-09-19T08:14:02.628 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph daemon mds.b perf dump 2024-09-19T08:14:02.891 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T08:14:03.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:02 smithi139 ceph-mon[34874]: pgmap v341: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:14:03.191 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:02 smithi098 ceph-mon[31333]: pgmap v341: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:14:03.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:02 smithi162 ceph-mon[35260]: pgmap v341: 97 pgs: 97 active+clean; 7.3 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T08:14:03.478 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-19T08:14:03.479 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-0": { 2024-09-19T08:14:03.479 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 3857, 2024-09-19T08:14:03.479 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 3941, 2024-09-19T08:14:03.479 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 1452965, 2024-09-19T08:14:03.479 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 4762420, 2024-09-19T08:14:03.479 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 18, 2024-09-19T08:14:03.479 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 5, 2024-09-19T08:14:03.479 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.376323242, 2024-09-19T08:14:03.479 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.134072117, 2024-09-19T08:14:03.479 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.172751829, 2024-09-19T08:14:03.479 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.083459268, 2024-09-19T08:14:03.480 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:03.480 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 3942, 2024-09-19T08:14:03.480 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.710875924, 2024-09-19T08:14:03.480 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000180333 2024-09-19T08:14:03.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.480 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:03.480 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 4118, 2024-09-19T08:14:03.480 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.001076539, 2024-09-19T08:14:03.480 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000261 2024-09-19T08:14:03.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.480 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 58784, 2024-09-19T08:14:03.480 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 359216 2024-09-19T08:14:03.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.481 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { 2024-09-19T08:14:03.481 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1325, 2024-09-19T08:14:03.481 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 1133, 2024-09-19T08:14:03.481 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 1794885, 2024-09-19T08:14:03.481 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 496176, 2024-09-19T08:14:03.481 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 14, 2024-09-19T08:14:03.481 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 7, 2024-09-19T08:14:03.481 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.139649130, 2024-09-19T08:14:03.481 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.055356619, 2024-09-19T08:14:03.481 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.099283798, 2024-09-19T08:14:03.482 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.010757944, 2024-09-19T08:14:03.482 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:03.482 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1132, 2024-09-19T08:14:03.482 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.270462654, 2024-09-19T08:14:03.482 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000238924 2024-09-19T08:14:03.482 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.482 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:03.482 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1280, 2024-09-19T08:14:03.482 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000307490, 2024-09-19T08:14:03.482 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000240 2024-09-19T08:14:03.482 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.482 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 1458496, 2024-09-19T08:14:03.483 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 161856 2024-09-19T08:14:03.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.483 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-2": { 2024-09-19T08:14:03.483 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 2424, 2024-09-19T08:14:03.483 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 1563, 2024-09-19T08:14:03.483 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 753787, 2024-09-19T08:14:03.483 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 2238110, 2024-09-19T08:14:03.483 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 143, 2024-09-19T08:14:03.483 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 10, 2024-09-19T08:14:03.483 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.281853145, 2024-09-19T08:14:03.483 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.073956763, 2024-09-19T08:14:03.483 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.164770003, 2024-09-19T08:14:03.483 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.050564486, 2024-09-19T08:14:03.484 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:03.484 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1558, 2024-09-19T08:14:03.484 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.269788518, 2024-09-19T08:14:03.484 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000173163 2024-09-19T08:14:03.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.484 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:03.484 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 3337, 2024-09-19T08:14:03.484 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000667918, 2024-09-19T08:14:03.484 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000200 2024-09-19T08:14:03.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.484 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 409472, 2024-09-19T08:14:03.484 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 229888 2024-09-19T08:14:03.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.485 INFO:teuthology.orchestra.run.smithi139.stdout: "cct": { 2024-09-19T08:14:03.485 INFO:teuthology.orchestra.run.smithi139.stdout: "total_workers": 1, 2024-09-19T08:14:03.485 INFO:teuthology.orchestra.run.smithi139.stdout: "unhealthy_workers": 0 2024-09-19T08:14:03.485 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.485 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-MDSRank": { 2024-09-19T08:14:03.485 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-09-19T08:14:03.485 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-09-19T08:14:03.485 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 859, 2024-09-19T08:14:03.485 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.889349456, 2024-09-19T08:14:03.485 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.001035331 2024-09-19T08:14:03.485 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:03.485 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.485 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-PurgeQueue": { 2024-09-19T08:14:03.486 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-09-19T08:14:03.486 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-09-19T08:14:03.486 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2, 2024-09-19T08:14:03.486 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000008729, 2024-09-19T08:14:03.486 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000004364 2024-09-19T08:14:03.486 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:03.486 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.486 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": { 2024-09-19T08:14:03.486 INFO:teuthology.orchestra.run.smithi139.stdout: "request": 181, 2024-09-19T08:14:03.486 INFO:teuthology.orchestra.run.smithi139.stdout: "reply": 179, 2024-09-19T08:14:03.486 INFO:teuthology.orchestra.run.smithi139.stdout: "reply_latency": { 2024-09-19T08:14:03.486 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 179, 2024-09-19T08:14:03.486 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.239853354, 2024-09-19T08:14:03.487 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.001339962 2024-09-19T08:14:03.487 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.487 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_reply": 0, 2024-09-19T08:14:03.487 INFO:teuthology.orchestra.run.smithi139.stdout: "forward": 2, 2024-09-19T08:14:03.487 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_complete": 0, 2024-09-19T08:14:03.487 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_keys": 0, 2024-09-19T08:14:03.487 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_commit": 12, 2024-09-19T08:14:03.487 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_split": 0, 2024-09-19T08:14:03.487 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_merge": 0, 2024-09-19T08:14:03.487 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes": 37, 2024-09-19T08:14:03.487 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_top": 2, 2024-09-19T08:14:03.488 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_bottom": 13, 2024-09-19T08:14:03.488 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pin_tail": 22, 2024-09-19T08:14:03.488 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pinned": 33, 2024-09-19T08:14:03.488 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_expired": 15, 2024-09-19T08:14:03.488 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_with_caps": 2, 2024-09-19T08:14:03.488 INFO:teuthology.orchestra.run.smithi139.stdout: "caps": 3, 2024-09-19T08:14:03.488 INFO:teuthology.orchestra.run.smithi139.stdout: "subtrees": 3, 2024-09-19T08:14:03.488 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse": 254, 2024-09-19T08:14:03.488 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_hit": 230, 2024-09-19T08:14:03.488 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_forward": 1, 2024-09-19T08:14:03.488 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_discover": 0, 2024-09-19T08:14:03.488 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_dir_fetch": 0, 2024-09-19T08:14:03.488 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_remote_ino": 0, 2024-09-19T08:14:03.489 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_lock": 0, 2024-09-19T08:14:03.489 INFO:teuthology.orchestra.run.smithi139.stdout: "load_cent": 0, 2024-09-19T08:14:03.489 INFO:teuthology.orchestra.run.smithi139.stdout: "q": 0, 2024-09-19T08:14:03.489 INFO:teuthology.orchestra.run.smithi139.stdout: "exported": 3, 2024-09-19T08:14:03.489 INFO:teuthology.orchestra.run.smithi139.stdout: "exported_inodes": 11, 2024-09-19T08:14:03.489 INFO:teuthology.orchestra.run.smithi139.stdout: "imported": 3, 2024-09-19T08:14:03.489 INFO:teuthology.orchestra.run.smithi139.stdout: "imported_inodes": 13, 2024-09-19T08:14:03.489 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_dir_fetch": 0, 2024-09-19T08:14:03.489 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_backtrace_fetch": 173, 2024-09-19T08:14:03.489 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_peer_discover": 0, 2024-09-19T08:14:03.489 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rfiles": 4, 2024-09-19T08:14:03.489 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rbytes": 234, 2024-09-19T08:14:03.489 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rsnaps": 6, 2024-09-19T08:14:03.489 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_fetch": 173, 2024-09-19T08:14:03.490 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_set_tag": 0, 2024-09-19T08:14:03.490 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_repaired": 0, 2024-09-19T08:14:03.490 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_inotable_repaired": 0, 2024-09-19T08:14:03.490 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_inodes": 112, 2024-09-19T08:14:03.490 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_base_inodes": 15, 2024-09-19T08:14:03.490 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dirfrag_rstats": 97, 2024-09-19T08:14:03.490 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_file_inodes": 61, 2024-09-19T08:14:03.490 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_inode_file_caps": 2, 2024-09-19T08:14:03.490 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_revoke": 28, 2024-09-19T08:14:03.490 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_grant": 762, 2024-09-19T08:14:03.490 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_trunc": 0, 2024-09-19T08:14:03.490 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-19T08:14:03.491 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-19T08:14:03.491 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps": 1066, 2024-09-19T08:14:03.491 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps_dirty": 1028, 2024-09-19T08:14:03.491 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_cap_release": 4, 2024-09-19T08:14:03.491 INFO:teuthology.orchestra.run.smithi139.stdout: "process_request_cap_release": 25 2024-09-19T08:14:03.491 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.491 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_cache": { 2024-09-19T08:14:03.491 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update": 15, 2024-09-19T08:14:03.491 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update_receipt": 0, 2024-09-19T08:14:03.491 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_try_discover": 0, 2024-09-19T08:14:03.491 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_send_discover": 0, 2024-09-19T08:14:03.492 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_handle_discover": 39, 2024-09-19T08:14:03.492 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays": 2, 2024-09-19T08:14:03.492 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_delayed": 0, 2024-09-19T08:14:03.492 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_enqueuing": 0, 2024-09-19T08:14:03.492 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_created": 2, 2024-09-19T08:14:03.492 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_enqueued": 0, 2024-09-19T08:14:03.492 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_reintegrated": 0, 2024-09-19T08:14:03.492 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_migrated": 0, 2024-09-19T08:14:03.492 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_processing": 0, 2024-09-19T08:14:03.492 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_enqueued": 0, 2024-09-19T08:14:03.493 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_prioritized": 0, 2024-09-19T08:14:03.493 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_started": 0, 2024-09-19T08:14:03.493 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_completed": 0, 2024-09-19T08:14:03.493 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_path": 3, 2024-09-19T08:14:03.493 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_inode": 39, 2024-09-19T08:14:03.493 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_enqueue_scrub": 15, 2024-09-19T08:14:03.493 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_exportdir": 3, 2024-09-19T08:14:03.493 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_flush": 0, 2024-09-19T08:14:03.493 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragmentdir": 0, 2024-09-19T08:14:03.493 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragstats": 0, 2024-09-19T08:14:03.493 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_inodestats": 0 2024-09-19T08:14:03.493 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.494 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_log": { 2024-09-19T08:14:03.494 INFO:teuthology.orchestra.run.smithi139.stdout: "evlrg": 0, 2024-09-19T08:14:03.494 INFO:teuthology.orchestra.run.smithi139.stdout: "evadd": 1134, 2024-09-19T08:14:03.494 INFO:teuthology.orchestra.run.smithi139.stdout: "evex": 0, 2024-09-19T08:14:03.494 INFO:teuthology.orchestra.run.smithi139.stdout: "evtrm": 0, 2024-09-19T08:14:03.494 INFO:teuthology.orchestra.run.smithi139.stdout: "ev": 1134, 2024-09-19T08:14:03.494 INFO:teuthology.orchestra.run.smithi139.stdout: "evexg": 0, 2024-09-19T08:14:03.494 INFO:teuthology.orchestra.run.smithi139.stdout: "evexd": 0, 2024-09-19T08:14:03.494 INFO:teuthology.orchestra.run.smithi139.stdout: "segadd": 3, 2024-09-19T08:14:03.494 INFO:teuthology.orchestra.run.smithi139.stdout: "segex": 0, 2024-09-19T08:14:03.494 INFO:teuthology.orchestra.run.smithi139.stdout: "segtrm": 0, 2024-09-19T08:14:03.494 INFO:teuthology.orchestra.run.smithi139.stdout: "seg": 3, 2024-09-19T08:14:03.494 INFO:teuthology.orchestra.run.smithi139.stdout: "segmjr": 2, 2024-09-19T08:14:03.494 INFO:teuthology.orchestra.run.smithi139.stdout: "segexg": 0, 2024-09-19T08:14:03.495 INFO:teuthology.orchestra.run.smithi139.stdout: "segexd": 0, 2024-09-19T08:14:03.495 INFO:teuthology.orchestra.run.smithi139.stdout: "expos": 4194304, 2024-09-19T08:14:03.495 INFO:teuthology.orchestra.run.smithi139.stdout: "wrpos": 7245708, 2024-09-19T08:14:03.495 INFO:teuthology.orchestra.run.smithi139.stdout: "rdpos": 0, 2024-09-19T08:14:03.495 INFO:teuthology.orchestra.run.smithi139.stdout: "jlat": { 2024-09-19T08:14:03.495 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1105, 2024-09-19T08:14:03.495 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 70.611698971, 2024-09-19T08:14:03.495 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.063901990 2024-09-19T08:14:03.495 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.495 INFO:teuthology.orchestra.run.smithi139.stdout: "replayed": 0 2024-09-19T08:14:03.495 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.495 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_mem": { 2024-09-19T08:14:03.495 INFO:teuthology.orchestra.run.smithi139.stdout: "ino": 29, 2024-09-19T08:14:03.496 INFO:teuthology.orchestra.run.smithi139.stdout: "ino+": 49, 2024-09-19T08:14:03.496 INFO:teuthology.orchestra.run.smithi139.stdout: "ino-": 20, 2024-09-19T08:14:03.496 INFO:teuthology.orchestra.run.smithi139.stdout: "dir": 22, 2024-09-19T08:14:03.496 INFO:teuthology.orchestra.run.smithi139.stdout: "dir+": 22, 2024-09-19T08:14:03.496 INFO:teuthology.orchestra.run.smithi139.stdout: "dir-": 0, 2024-09-19T08:14:03.496 INFO:teuthology.orchestra.run.smithi139.stdout: "dn": 37, 2024-09-19T08:14:03.496 INFO:teuthology.orchestra.run.smithi139.stdout: "dn+": 52, 2024-09-19T08:14:03.496 INFO:teuthology.orchestra.run.smithi139.stdout: "dn-": 15, 2024-09-19T08:14:03.496 INFO:teuthology.orchestra.run.smithi139.stdout: "cap": 3, 2024-09-19T08:14:03.496 INFO:teuthology.orchestra.run.smithi139.stdout: "cap+": 25, 2024-09-19T08:14:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "cap-": 22, 2024-09-19T08:14:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "rss": 58428, 2024-09-19T08:14:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "heap": 231712 2024-09-19T08:14:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_server": { 2024-09-19T08:14:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_client_request": 239, 2024-09-19T08:14:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_server_request": 3, 2024-09-19T08:14:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_request": 181, 2024-09-19T08:14:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_session": 38, 2024-09-19T08:14:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_peer_request": 6, 2024-09-19T08:14:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "req_create_latency": { 2024-09-19T08:14:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 7, 2024-09-19T08:14:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.008133699, 2024-09-19T08:14:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.001161957 2024-09-19T08:14:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getattr_latency": { 2024-09-19T08:14:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 110, 2024-09-19T08:14:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.018654287, 2024-09-19T08:14:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000169584 2024-09-19T08:14:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getfilelock_latency": { 2024-09-19T08:14:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: "req_link_latency": { 2024-09-19T08:14:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookup_latency": { 2024-09-19T08:14:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 11, 2024-09-19T08:14:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.003566802, 2024-09-19T08:14:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000324254 2024-09-19T08:14:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookuphash_latency": { 2024-09-19T08:14:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupino_latency": { 2024-09-19T08:14:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1, 2024-09-19T08:14:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000190989, 2024-09-19T08:14:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000190989 2024-09-19T08:14:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupname_latency": { 2024-09-19T08:14:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupparent_latency": { 2024-09-19T08:14:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupsnap_latency": { 2024-09-19T08:14:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lssnap_latency": { 2024-09-19T08:14:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 11, 2024-09-19T08:14:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.001502131, 2024-09-19T08:14:03.502 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000136557 2024-09-19T08:14:03.502 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.502 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mkdir_latency": { 2024-09-19T08:14:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 8, 2024-09-19T08:14:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.014393988, 2024-09-19T08:14:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.001799248 2024-09-19T08:14:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mknod_latency": { 2024-09-19T08:14:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mksnap_latency": { 2024-09-19T08:14:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 11, 2024-09-19T08:14:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.129660857, 2024-09-19T08:14:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.011787350 2024-09-19T08:14:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: "req_open_latency": { 2024-09-19T08:14:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1, 2024-09-19T08:14:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000259000, 2024-09-19T08:14:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000259000 2024-09-19T08:14:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: "req_readdir_latency": { 2024-09-19T08:14:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1, 2024-09-19T08:14:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.004842522, 2024-09-19T08:14:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.004842522 2024-09-19T08:14:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rename_latency": { 2024-09-19T08:14:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2, 2024-09-19T08:14:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.002103159, 2024-09-19T08:14:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.001051579 2024-09-19T08:14:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: "req_renamesnap_latency": { 2024-09-19T08:14:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: "req_snapdiff_latency": { 2024-09-19T08:14:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmdir_latency": { 2024-09-19T08:14:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2, 2024-09-19T08:14:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.011203360, 2024-09-19T08:14:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.005601680 2024-09-19T08:14:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmsnap_latency": { 2024-09-19T08:14:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 5, 2024-09-19T08:14:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.023949537, 2024-09-19T08:14:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.004789907 2024-09-19T08:14:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmxattr_latency": { 2024-09-19T08:14:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setattr_latency": { 2024-09-19T08:14:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1, 2024-09-19T08:14:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000772323, 2024-09-19T08:14:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000772323 2024-09-19T08:14:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setdirlayout_latency": { 2024-09-19T08:14:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setfilelock_latency": { 2024-09-19T08:14:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setlayout_latency": { 2024-09-19T08:14:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setxattr_latency": { 2024-09-19T08:14:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 8, 2024-09-19T08:14:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.020620700, 2024-09-19T08:14:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.002577587 2024-09-19T08:14:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: "req_symlink_latency": { 2024-09-19T08:14:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: "req_unlink_latency": { 2024-09-19T08:14:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_revoke_eviction": 0, 2024-09-19T08:14:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_acquisition_throttle": 0, 2024-09-19T08:14:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getvxattr_latency": { 2024-09-19T08:14:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_sessions": { 2024-09-19T08:14:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: "session_count": 2, 2024-09-19T08:14:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: "session_add": 6, 2024-09-19T08:14:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: "session_remove": 4, 2024-09-19T08:14:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_open": 2, 2024-09-19T08:14:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_stale": 0, 2024-09-19T08:14:03.511 INFO:teuthology.orchestra.run.smithi139.stdout: "total_load": 27, 2024-09-19T08:14:03.511 INFO:teuthology.orchestra.run.smithi139.stdout: "average_load": 13, 2024-09-19T08:14:03.511 INFO:teuthology.orchestra.run.smithi139.stdout: "avg_session_uptime": 630, 2024-09-19T08:14:03.511 INFO:teuthology.orchestra.run.smithi139.stdout: "mdthresh_evicted": 0 2024-09-19T08:14:03.511 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.511 INFO:teuthology.orchestra.run.smithi139.stdout: "mempool": { 2024-09-19T08:14:03.511 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_bytes": 0, 2024-09-19T08:14:03.511 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_items": 0, 2024-09-19T08:14:03.511 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_bytes": 0, 2024-09-19T08:14:03.511 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_items": 0, 2024-09-19T08:14:03.511 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_bytes": 0, 2024-09-19T08:14:03.511 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_items": 0, 2024-09-19T08:14:03.512 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-19T08:14:03.512 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_items": 0, 2024-09-19T08:14:03.512 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-19T08:14:03.512 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_items": 0, 2024-09-19T08:14:03.512 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_bytes": 0, 2024-09-19T08:14:03.512 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_items": 0, 2024-09-19T08:14:03.512 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-19T08:14:03.512 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_items": 0, 2024-09-19T08:14:03.512 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_bytes": 0, 2024-09-19T08:14:03.512 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_items": 0, 2024-09-19T08:14:03.512 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_bytes": 0, 2024-09-19T08:14:03.512 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_items": 0, 2024-09-19T08:14:03.512 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-19T08:14:03.513 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_items": 0, 2024-09-19T08:14:03.513 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-19T08:14:03.513 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_items": 0, 2024-09-19T08:14:03.513 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_bytes": 0, 2024-09-19T08:14:03.513 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_items": 0, 2024-09-19T08:14:03.513 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_bytes": 0, 2024-09-19T08:14:03.513 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_items": 0, 2024-09-19T08:14:03.513 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-19T08:14:03.513 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_items": 0, 2024-09-19T08:14:03.513 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_bytes": 0, 2024-09-19T08:14:03.513 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_items": 0, 2024-09-19T08:14:03.513 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_bytes": 0, 2024-09-19T08:14:03.513 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_items": 0, 2024-09-19T08:14:03.514 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_bytes": 0, 2024-09-19T08:14:03.514 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_items": 0, 2024-09-19T08:14:03.514 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_bytes": 0, 2024-09-19T08:14:03.514 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_items": 0, 2024-09-19T08:14:03.514 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_bytes": 311291, 2024-09-19T08:14:03.514 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_items": 116, 2024-09-19T08:14:03.514 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_bytes": 0, 2024-09-19T08:14:03.514 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_items": 0, 2024-09-19T08:14:03.514 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_bytes": 0, 2024-09-19T08:14:03.514 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_items": 0, 2024-09-19T08:14:03.514 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_bytes": 0, 2024-09-19T08:14:03.514 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_items": 0, 2024-09-19T08:14:03.514 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_bytes": 0, 2024-09-19T08:14:03.515 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_items": 0, 2024-09-19T08:14:03.515 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_bytes": 8136, 2024-09-19T08:14:03.515 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_items": 180, 2024-09-19T08:14:03.515 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_bytes": 0, 2024-09-19T08:14:03.515 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_items": 0, 2024-09-19T08:14:03.515 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_bytes": 0, 2024-09-19T08:14:03.515 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, 2024-09-19T08:14:03.515 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 144685, 2024-09-19T08:14:03.515 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 1190, 2024-09-19T08:14:03.515 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_bytes": 0, 2024-09-19T08:14:03.515 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_items": 0, 2024-09-19T08:14:03.515 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_bytes": 0, 2024-09-19T08:14:03.515 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_items": 0 2024-09-19T08:14:03.515 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.516 INFO:teuthology.orchestra.run.smithi139.stdout: "objecter": { 2024-09-19T08:14:03.516 INFO:teuthology.orchestra.run.smithi139.stdout: "op_active": 0, 2024-09-19T08:14:03.516 INFO:teuthology.orchestra.run.smithi139.stdout: "op_laggy": 0, 2024-09-19T08:14:03.516 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send": 1893, 2024-09-19T08:14:03.516 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send_bytes": 3069776, 2024-09-19T08:14:03.516 INFO:teuthology.orchestra.run.smithi139.stdout: "op_resend": 37, 2024-09-19T08:14:03.516 INFO:teuthology.orchestra.run.smithi139.stdout: "op_reply": 1893, 2024-09-19T08:14:03.516 INFO:teuthology.orchestra.run.smithi139.stdout: "op_latency": { 2024-09-19T08:14:03.516 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1893, 2024-09-19T08:14:03.516 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 80.054278496, 2024-09-19T08:14:03.516 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.042289634 2024-09-19T08:14:03.516 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.516 INFO:teuthology.orchestra.run.smithi139.stdout: "op_inflight": 0, 2024-09-19T08:14:03.517 INFO:teuthology.orchestra.run.smithi139.stdout: "oplen_avg": { 2024-09-19T08:14:03.517 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1893, 2024-09-19T08:14:03.517 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 1945 2024-09-19T08:14:03.517 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.517 INFO:teuthology.orchestra.run.smithi139.stdout: "op": 1893, 2024-09-19T08:14:03.517 INFO:teuthology.orchestra.run.smithi139.stdout: "op_r": 203, 2024-09-19T08:14:03.517 INFO:teuthology.orchestra.run.smithi139.stdout: "op_w": 1690, 2024-09-19T08:14:03.517 INFO:teuthology.orchestra.run.smithi139.stdout: "op_rmw": 0, 2024-09-19T08:14:03.517 INFO:teuthology.orchestra.run.smithi139.stdout: "op_pg": 0, 2024-09-19T08:14:03.517 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_stat": 0, 2024-09-19T08:14:03.517 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_create": 11, 2024-09-19T08:14:03.517 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_read": 15, 2024-09-19T08:14:03.517 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_write": 1105, 2024-09-19T08:14:03.518 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writefull": 29, 2024-09-19T08:14:03.518 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writesame": 0, 2024-09-19T08:14:03.518 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_append": 0, 2024-09-19T08:14:03.518 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_zero": 0, 2024-09-19T08:14:03.518 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_truncate": 0, 2024-09-19T08:14:03.518 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_delete": 506, 2024-09-19T08:14:03.518 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_mapext": 0, 2024-09-19T08:14:03.518 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_sparse_read": 0, 2024-09-19T08:14:03.518 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_clonerange": 0, 2024-09-19T08:14:03.518 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_getxattr": 188, 2024-09-19T08:14:03.518 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_setxattr": 22, 2024-09-19T08:14:03.518 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_cmpxattr": 0, 2024-09-19T08:14:03.518 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_rmxattr": 0, 2024-09-19T08:14:03.519 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_resetxattrs": 0, 2024-09-19T08:14:03.519 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_call": 0, 2024-09-19T08:14:03.519 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_watch": 0, 2024-09-19T08:14:03.519 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_notify": 0, 2024-09-19T08:14:03.519 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_src_cmpxattr": 0, 2024-09-19T08:14:03.519 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls": 0, 2024-09-19T08:14:03.519 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls_filter": 0, 2024-09-19T08:14:03.519 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_other": 0, 2024-09-19T08:14:03.519 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_active": 0, 2024-09-19T08:14:03.519 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_send": 0, 2024-09-19T08:14:03.519 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_resend": 0, 2024-09-19T08:14:03.519 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_ping": 0, 2024-09-19T08:14:03.519 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_active": 0, 2024-09-19T08:14:03.519 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_send": 0, 2024-09-19T08:14:03.520 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_resend": 0, 2024-09-19T08:14:03.520 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_active": 0, 2024-09-19T08:14:03.520 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_send": 0, 2024-09-19T08:14:03.520 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_resend": 0, 2024-09-19T08:14:03.520 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_active": 0, 2024-09-19T08:14:03.520 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_send": 0, 2024-09-19T08:14:03.520 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_resend": 0, 2024-09-19T08:14:03.520 INFO:teuthology.orchestra.run.smithi139.stdout: "command_active": 0, 2024-09-19T08:14:03.520 INFO:teuthology.orchestra.run.smithi139.stdout: "command_send": 0, 2024-09-19T08:14:03.520 INFO:teuthology.orchestra.run.smithi139.stdout: "command_resend": 0, 2024-09-19T08:14:03.520 INFO:teuthology.orchestra.run.smithi139.stdout: "map_epoch": 89, 2024-09-19T08:14:03.521 INFO:teuthology.orchestra.run.smithi139.stdout: "map_full": 0, 2024-09-19T08:14:03.521 INFO:teuthology.orchestra.run.smithi139.stdout: "map_inc": 11, 2024-09-19T08:14:03.521 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_sessions": 12, 2024-09-19T08:14:03.521 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_open": 12, 2024-09-19T08:14:03.521 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_close": 0, 2024-09-19T08:14:03.521 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_laggy": 0, 2024-09-19T08:14:03.521 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_wr": 57, 2024-09-19T08:14:03.521 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_rd": 0, 2024-09-19T08:14:03.521 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_del": 12 2024-09-19T08:14:03.521 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.522 INFO:teuthology.orchestra.run.smithi139.stdout: "oft": { 2024-09-19T08:14:03.522 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_objs": 1, 2024-09-19T08:14:03.522 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_kv_pairs": 0, 2024-09-19T08:14:03.522 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_updates": 0, 2024-09-19T08:14:03.522 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_removes": 0 2024-09-19T08:14:03.522 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.522 INFO:teuthology.orchestra.run.smithi139.stdout: "osd-slow-ops": { 2024-09-19T08:14:03.522 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_ops_count": 42 2024-09-19T08:14:03.522 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.522 INFO:teuthology.orchestra.run.smithi139.stdout: "purge_queue": { 2024-09-19T08:14:03.522 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops": 0, 2024-09-19T08:14:03.522 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops_high_water": 0, 2024-09-19T08:14:03.522 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing": 0, 2024-09-19T08:14:03.523 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_high_water": 0, 2024-09-19T08:14:03.523 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed_ops": 0, 2024-09-19T08:14:03.523 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed": 0, 2024-09-19T08:14:03.523 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_item_in_journal": 0 2024-09-19T08:14:03.523 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.523 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-19T08:14:03.523 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:03.523 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-09-19T08:14:03.523 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:03.523 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 7602, 2024-09-19T08:14:03.523 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 3245198, 2024-09-19T08:14:03.523 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:03.523 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 7602, 2024-09-19T08:14:03.524 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-09-19T08:14:03.524 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-09-19T08:14:03.524 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 7602, 2024-09-19T08:14:03.524 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 3245198, 2024-09-19T08:14:03.524 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:03.524 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.524 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.524 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.524 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:03.524 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.524 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_bytes": { 2024-09-19T08:14:03.524 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:03.524 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-09-19T08:14:03.525 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:03.525 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-09-19T08:14:03.525 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-09-19T08:14:03.525 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:03.525 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:03.525 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 1893, 2024-09-19T08:14:03.525 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 3069776, 2024-09-19T08:14:03.526 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 1893, 2024-09-19T08:14:03.526 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 3069776, 2024-09-19T08:14:03.526 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:03.526 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.526 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.526 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.527 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:03.527 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.527 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_ops": { 2024-09-19T08:14:03.527 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:03.527 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 1024, 2024-09-19T08:14:03.527 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:03.528 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-09-19T08:14:03.528 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-09-19T08:14:03.528 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:03.528 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:03.528 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 1893, 2024-09-19T08:14:03.528 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 1893, 2024-09-19T08:14:03.529 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 1893, 2024-09-19T08:14:03.529 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 1893, 2024-09-19T08:14:03.529 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:03.529 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.529 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.529 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.529 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:03.529 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.530 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle": { 2024-09-19T08:14:03.530 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:03.530 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-09-19T08:14:03.530 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:03.530 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-09-19T08:14:03.530 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-09-19T08:14:03.530 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:03.531 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:03.531 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-09-19T08:14:03.531 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-09-19T08:14:03.531 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 0, 2024-09-19T08:14:03.531 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 0, 2024-09-19T08:14:03.531 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:03.532 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.532 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.532 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.532 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:03.532 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:03.532 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle-0x565470202960": { 2024-09-19T08:14:03.533 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 84, 2024-09-19T08:14:03.533 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-09-19T08:14:03.533 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:03.533 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 1134, 2024-09-19T08:14:03.533 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 3051404, 2024-09-19T08:14:03.534 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:03.534 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 1134, 2024-09-19T08:14:03.534 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-09-19T08:14:03.534 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-09-19T08:14:03.534 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 1105, 2024-09-19T08:14:03.534 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 3051320, 2024-09-19T08:14:03.534 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:03.534 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:03.534 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:03.534 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:03.534 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:03.535 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:03.535 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-19T08:14:04.307 INFO:tasks.check_counter:Daemon mds.b mds.exported=3 2024-09-19T08:14:04.307 INFO:tasks.check_counter:Daemon mds.b mds.imported=3 2024-09-19T08:14:04.307 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=15 2024-09-19T08:14:04.307 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-09-19T08:14:04.307 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-09-19T08:14:04.307 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-09-19T08:14:04.307 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-09-19T08:14:04.307 DEBUG:tasks.check_counter:Getting stats from e 2024-09-19T08:14:04.308 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph daemon mds.e perf dump 2024-09-19T08:14:04.552 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T08:14:05.026 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-19T08:14:05.026 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-0": { 2024-09-19T08:14:05.027 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 4393, 2024-09-19T08:14:05.027 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 4396, 2024-09-19T08:14:05.027 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 1099275, 2024-09-19T08:14:05.027 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 4822832, 2024-09-19T08:14:05.027 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 18, 2024-09-19T08:14:05.027 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 4, 2024-09-19T08:14:05.027 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.374807104, 2024-09-19T08:14:05.027 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.118372302, 2024-09-19T08:14:05.027 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.088091563, 2024-09-19T08:14:05.027 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.117568177, 2024-09-19T08:14:05.027 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:05.027 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 4394, 2024-09-19T08:14:05.028 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.239076852, 2024-09-19T08:14:05.028 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000054409 2024-09-19T08:14:05.028 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.028 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:05.028 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 19, 2024-09-19T08:14:05.028 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000003125, 2024-09-19T08:14:05.028 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000164 2024-09-19T08:14:05.028 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.028 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 928, 2024-09-19T08:14:05.028 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 976 2024-09-19T08:14:05.028 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.028 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { 2024-09-19T08:14:05.029 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1928, 2024-09-19T08:14:05.029 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 2043, 2024-09-19T08:14:05.029 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 1824640, 2024-09-19T08:14:05.029 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 1033074, 2024-09-19T08:14:05.029 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 8, 2024-09-19T08:14:05.029 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 7, 2024-09-19T08:14:05.029 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.147458701, 2024-09-19T08:14:05.029 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.048238519, 2024-09-19T08:14:05.029 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.069229543, 2024-09-19T08:14:05.029 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.036323793, 2024-09-19T08:14:05.029 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:05.029 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2042, 2024-09-19T08:14:05.029 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.095213066, 2024-09-19T08:14:05.030 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000046627 2024-09-19T08:14:05.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.030 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:05.030 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 5, 2024-09-19T08:14:05.030 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000835, 2024-09-19T08:14:05.030 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000167 2024-09-19T08:14:05.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.030 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 1400816, 2024-09-19T08:14:05.030 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 469424 2024-09-19T08:14:05.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.030 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-2": { 2024-09-19T08:14:05.030 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 4834, 2024-09-19T08:14:05.030 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 5822, 2024-09-19T08:14:05.031 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 1553880, 2024-09-19T08:14:05.031 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 2530171, 2024-09-19T08:14:05.031 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 41, 2024-09-19T08:14:05.031 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 8, 2024-09-19T08:14:05.031 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.348404570, 2024-09-19T08:14:05.031 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.135423049, 2024-09-19T08:14:05.031 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.283029469, 2024-09-19T08:14:05.031 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.038428635, 2024-09-19T08:14:05.031 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:05.031 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 5820, 2024-09-19T08:14:05.031 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.370973702, 2024-09-19T08:14:05.031 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000063741 2024-09-19T08:14:05.031 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.032 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:05.032 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 6341, 2024-09-19T08:14:05.032 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.001729373, 2024-09-19T08:14:05.032 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000272 2024-09-19T08:14:05.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.032 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 4160, 2024-09-19T08:14:05.032 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 125280 2024-09-19T08:14:05.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.032 INFO:teuthology.orchestra.run.smithi139.stdout: "cct": { 2024-09-19T08:14:05.032 INFO:teuthology.orchestra.run.smithi139.stdout: "total_workers": 1, 2024-09-19T08:14:05.032 INFO:teuthology.orchestra.run.smithi139.stdout: "unhealthy_workers": 0 2024-09-19T08:14:05.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.032 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-MDSRank": { 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 3401, 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.694162473, 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000204105 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-PurgeQueue": { 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 4364, 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.120528282, 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000027618 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:05.033 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.034 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": { 2024-09-19T08:14:05.034 INFO:teuthology.orchestra.run.smithi139.stdout: "request": 7, 2024-09-19T08:14:05.034 INFO:teuthology.orchestra.run.smithi139.stdout: "reply": 5, 2024-09-19T08:14:05.034 INFO:teuthology.orchestra.run.smithi139.stdout: "reply_latency": { 2024-09-19T08:14:05.034 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 5, 2024-09-19T08:14:05.034 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.008165309, 2024-09-19T08:14:05.034 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.001633061 2024-09-19T08:14:05.034 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.034 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_reply": 0, 2024-09-19T08:14:05.034 INFO:teuthology.orchestra.run.smithi139.stdout: "forward": 2, 2024-09-19T08:14:05.034 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_complete": 0, 2024-09-19T08:14:05.034 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_keys": 0, 2024-09-19T08:14:05.035 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_commit": 11, 2024-09-19T08:14:05.035 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_split": 0, 2024-09-19T08:14:05.035 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_merge": 0, 2024-09-19T08:14:05.035 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes": 15, 2024-09-19T08:14:05.035 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_top": 0, 2024-09-19T08:14:05.035 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_bottom": 2, 2024-09-19T08:14:05.035 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pin_tail": 13, 2024-09-19T08:14:05.035 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pinned": 13, 2024-09-19T08:14:05.035 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_expired": 15, 2024-09-19T08:14:05.035 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_with_caps": 0, 2024-09-19T08:14:05.035 INFO:teuthology.orchestra.run.smithi139.stdout: "caps": 0, 2024-09-19T08:14:05.035 INFO:teuthology.orchestra.run.smithi139.stdout: "subtrees": 3, 2024-09-19T08:14:05.035 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse": 26, 2024-09-19T08:14:05.036 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_hit": 16, 2024-09-19T08:14:05.036 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_forward": 2, 2024-09-19T08:14:05.036 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_discover": 4, 2024-09-19T08:14:05.036 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_dir_fetch": 0, 2024-09-19T08:14:05.036 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_remote_ino": 0, 2024-09-19T08:14:05.036 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_lock": 0, 2024-09-19T08:14:05.036 INFO:teuthology.orchestra.run.smithi139.stdout: "load_cent": 0, 2024-09-19T08:14:05.036 INFO:teuthology.orchestra.run.smithi139.stdout: "q": 0, 2024-09-19T08:14:05.036 INFO:teuthology.orchestra.run.smithi139.stdout: "exported": 3, 2024-09-19T08:14:05.036 INFO:teuthology.orchestra.run.smithi139.stdout: "exported_inodes": 13, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "imported": 3, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "imported_inodes": 11, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_dir_fetch": 0, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_backtrace_fetch": 2, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_peer_discover": 0, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rfiles": 4, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rbytes": 234, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rsnaps": 6, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_fetch": 2, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_set_tag": 0, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_repaired": 0, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_inotable_repaired": 0, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_inodes": 0, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_base_inodes": 0, 2024-09-19T08:14:05.037 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dirfrag_rstats": 0, 2024-09-19T08:14:05.038 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_file_inodes": 2, 2024-09-19T08:14:05.038 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_inode_file_caps": 0, 2024-09-19T08:14:05.038 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_revoke": 5, 2024-09-19T08:14:05.038 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_grant": 857, 2024-09-19T08:14:05.038 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_trunc": 0, 2024-09-19T08:14:05.038 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-19T08:14:05.038 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-19T08:14:05.038 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps": 2778, 2024-09-19T08:14:05.038 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps_dirty": 2772, 2024-09-19T08:14:05.038 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_cap_release": 3, 2024-09-19T08:14:05.038 INFO:teuthology.orchestra.run.smithi139.stdout: "process_request_cap_release": 5 2024-09-19T08:14:05.038 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.038 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_cache": { 2024-09-19T08:14:05.038 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update": 3, 2024-09-19T08:14:05.039 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update_receipt": 11, 2024-09-19T08:14:05.039 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_try_discover": 7, 2024-09-19T08:14:05.039 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_send_discover": 9, 2024-09-19T08:14:05.039 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_handle_discover": 3, 2024-09-19T08:14:05.039 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays": 0, 2024-09-19T08:14:05.039 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_delayed": 0, 2024-09-19T08:14:05.039 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_enqueuing": 0, 2024-09-19T08:14:05.039 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_created": 3, 2024-09-19T08:14:05.039 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_enqueued": 3, 2024-09-19T08:14:05.039 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_reintegrated": 0, 2024-09-19T08:14:05.039 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_migrated": 0, 2024-09-19T08:14:05.039 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_processing": 0, 2024-09-19T08:14:05.039 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_enqueued": 0, 2024-09-19T08:14:05.040 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_prioritized": 0, 2024-09-19T08:14:05.040 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_started": 0, 2024-09-19T08:14:05.040 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_completed": 0, 2024-09-19T08:14:05.040 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_path": 3, 2024-09-19T08:14:05.040 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_inode": 20, 2024-09-19T08:14:05.040 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_enqueue_scrub": 0, 2024-09-19T08:14:05.040 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_exportdir": 3, 2024-09-19T08:14:05.040 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_flush": 0, 2024-09-19T08:14:05.040 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragmentdir": 0, 2024-09-19T08:14:05.040 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragstats": 0, 2024-09-19T08:14:05.040 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_inodestats": 0 2024-09-19T08:14:05.040 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.040 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_log": { 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "evlrg": 0, 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "evadd": 2807, 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "evex": 0, 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "evtrm": 0, 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "ev": 2807, 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "evexg": 0, 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "evexd": 0, 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "segadd": 4, 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "segex": 0, 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "segtrm": 0, 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "seg": 4, 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "segmjr": 2, 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "segexg": 0, 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "segexd": 0, 2024-09-19T08:14:05.041 INFO:teuthology.orchestra.run.smithi139.stdout: "expos": 4194304, 2024-09-19T08:14:05.042 INFO:teuthology.orchestra.run.smithi139.stdout: "wrpos": 7515300, 2024-09-19T08:14:05.042 INFO:teuthology.orchestra.run.smithi139.stdout: "rdpos": 0, 2024-09-19T08:14:05.042 INFO:teuthology.orchestra.run.smithi139.stdout: "jlat": { 2024-09-19T08:14:05.042 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2776, 2024-09-19T08:14:05.042 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 161.272422932, 2024-09-19T08:14:05.042 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.058095253 2024-09-19T08:14:05.042 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.042 INFO:teuthology.orchestra.run.smithi139.stdout: "replayed": 0 2024-09-19T08:14:05.042 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.042 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_mem": { 2024-09-19T08:14:05.042 INFO:teuthology.orchestra.run.smithi139.stdout: "ino": 19, 2024-09-19T08:14:05.042 INFO:teuthology.orchestra.run.smithi139.stdout: "ino+": 29, 2024-09-19T08:14:05.042 INFO:teuthology.orchestra.run.smithi139.stdout: "ino-": 10, 2024-09-19T08:14:05.042 INFO:teuthology.orchestra.run.smithi139.stdout: "dir": 18, 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: "dir+": 23, 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: "dir-": 5, 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: "dn": 15, 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: "dn+": 33, 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: "dn-": 18, 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: "cap": 0, 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: "cap+": 4, 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: "cap-": 4, 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: "rss": 55588, 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: "heap": 223520 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_server": { 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_client_request": 13, 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_server_request": 0, 2024-09-19T08:14:05.043 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_request": 7, 2024-09-19T08:14:05.044 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_session": 19, 2024-09-19T08:14:05.044 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_peer_request": 3, 2024-09-19T08:14:05.044 INFO:teuthology.orchestra.run.smithi139.stdout: "req_create_latency": { 2024-09-19T08:14:05.044 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.044 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.044 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.044 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.044 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getattr_latency": { 2024-09-19T08:14:05.044 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.044 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.044 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.044 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.044 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getfilelock_latency": { 2024-09-19T08:14:05.044 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: "req_link_latency": { 2024-09-19T08:14:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookup_latency": { 2024-09-19T08:14:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1, 2024-09-19T08:14:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000502504, 2024-09-19T08:14:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000502504 2024-09-19T08:14:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookuphash_latency": { 2024-09-19T08:14:05.046 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.046 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.046 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.046 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.046 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupino_latency": { 2024-09-19T08:14:05.046 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.046 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.046 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.046 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.046 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupname_latency": { 2024-09-19T08:14:05.046 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.046 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.046 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.047 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.047 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupparent_latency": { 2024-09-19T08:14:05.047 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.047 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.047 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.047 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.047 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupsnap_latency": { 2024-09-19T08:14:05.047 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.047 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.047 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.047 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.047 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lssnap_latency": { 2024-09-19T08:14:05.047 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.047 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.048 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.048 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mkdir_latency": { 2024-09-19T08:14:05.048 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.048 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.048 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.049 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mknod_latency": { 2024-09-19T08:14:05.049 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.049 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.049 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.049 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mksnap_latency": { 2024-09-19T08:14:05.049 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.049 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.049 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.049 INFO:teuthology.orchestra.run.smithi139.stdout: "req_open_latency": { 2024-09-19T08:14:05.049 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1, 2024-09-19T08:14:05.050 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000856003, 2024-09-19T08:14:05.050 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000856003 2024-09-19T08:14:05.050 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.050 INFO:teuthology.orchestra.run.smithi139.stdout: "req_readdir_latency": { 2024-09-19T08:14:05.050 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.050 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.050 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.050 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.050 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rename_latency": { 2024-09-19T08:14:05.050 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.050 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.050 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.051 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.051 INFO:teuthology.orchestra.run.smithi139.stdout: "req_renamesnap_latency": { 2024-09-19T08:14:05.051 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.051 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.051 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.051 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.051 INFO:teuthology.orchestra.run.smithi139.stdout: "req_snapdiff_latency": { 2024-09-19T08:14:05.051 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.051 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.051 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.051 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.051 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmdir_latency": { 2024-09-19T08:14:05.051 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.051 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.052 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.052 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.052 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmsnap_latency": { 2024-09-19T08:14:05.052 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.052 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.052 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.052 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.052 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmxattr_latency": { 2024-09-19T08:14:05.052 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.052 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.052 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.052 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.053 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setattr_latency": { 2024-09-19T08:14:05.053 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.053 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.053 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.053 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setdirlayout_latency": { 2024-09-19T08:14:05.053 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.053 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.053 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.053 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setfilelock_latency": { 2024-09-19T08:14:05.053 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setlayout_latency": { 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setxattr_latency": { 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: "req_symlink_latency": { 2024-09-19T08:14:05.054 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.055 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.055 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.055 INFO:teuthology.orchestra.run.smithi139.stdout: "req_unlink_latency": { 2024-09-19T08:14:05.055 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 3, 2024-09-19T08:14:05.055 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.006806802, 2024-09-19T08:14:05.055 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.002268934 2024-09-19T08:14:05.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.055 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_revoke_eviction": 0, 2024-09-19T08:14:05.055 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_acquisition_throttle": 0, 2024-09-19T08:14:05.055 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getvxattr_latency": { 2024-09-19T08:14:05.055 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.055 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.056 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.056 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:05.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.056 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_sessions": { 2024-09-19T08:14:05.056 INFO:teuthology.orchestra.run.smithi139.stdout: "session_count": 1, 2024-09-19T08:14:05.056 INFO:teuthology.orchestra.run.smithi139.stdout: "session_add": 3, 2024-09-19T08:14:05.056 INFO:teuthology.orchestra.run.smithi139.stdout: "session_remove": 2, 2024-09-19T08:14:05.056 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_open": 1, 2024-09-19T08:14:05.056 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_stale": 0, 2024-09-19T08:14:05.056 INFO:teuthology.orchestra.run.smithi139.stdout: "total_load": 3, 2024-09-19T08:14:05.056 INFO:teuthology.orchestra.run.smithi139.stdout: "average_load": 3, 2024-09-19T08:14:05.056 INFO:teuthology.orchestra.run.smithi139.stdout: "avg_session_uptime": 549, 2024-09-19T08:14:05.056 INFO:teuthology.orchestra.run.smithi139.stdout: "mdthresh_evicted": 0 2024-09-19T08:14:05.057 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.057 INFO:teuthology.orchestra.run.smithi139.stdout: "mempool": { 2024-09-19T08:14:05.057 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_bytes": 0, 2024-09-19T08:14:05.057 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_items": 0, 2024-09-19T08:14:05.057 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_bytes": 0, 2024-09-19T08:14:05.057 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_items": 0, 2024-09-19T08:14:05.057 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_bytes": 0, 2024-09-19T08:14:05.057 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_items": 0, 2024-09-19T08:14:05.057 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-19T08:14:05.057 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_items": 0, 2024-09-19T08:14:05.057 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-19T08:14:05.057 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_items": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_bytes": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_items": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_items": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_bytes": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_items": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_bytes": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_items": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_items": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_items": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_bytes": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_items": 0, 2024-09-19T08:14:05.058 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_bytes": 0, 2024-09-19T08:14:05.059 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_items": 0, 2024-09-19T08:14:05.059 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-19T08:14:05.059 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_items": 0, 2024-09-19T08:14:05.059 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_bytes": 0, 2024-09-19T08:14:05.059 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_items": 0, 2024-09-19T08:14:05.059 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_bytes": 0, 2024-09-19T08:14:05.059 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_items": 0, 2024-09-19T08:14:05.059 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_bytes": 0, 2024-09-19T08:14:05.059 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_items": 0, 2024-09-19T08:14:05.059 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_bytes": 0, 2024-09-19T08:14:05.059 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_items": 0, 2024-09-19T08:14:05.059 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_bytes": 311527, 2024-09-19T08:14:05.059 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_items": 111, 2024-09-19T08:14:05.059 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_bytes": 0, 2024-09-19T08:14:05.060 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_items": 0, 2024-09-19T08:14:05.060 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_bytes": 0, 2024-09-19T08:14:05.060 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_items": 0, 2024-09-19T08:14:05.060 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_bytes": 0, 2024-09-19T08:14:05.060 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_items": 0, 2024-09-19T08:14:05.060 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_bytes": 0, 2024-09-19T08:14:05.060 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_items": 0, 2024-09-19T08:14:05.060 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_bytes": 8136, 2024-09-19T08:14:05.060 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_items": 180, 2024-09-19T08:14:05.060 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_bytes": 0, 2024-09-19T08:14:05.060 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_items": 0, 2024-09-19T08:14:05.060 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_bytes": 0, 2024-09-19T08:14:05.060 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, 2024-09-19T08:14:05.060 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 81317, 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 350, 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_bytes": 0, 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_items": 0, 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_bytes": 0, 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_items": 0 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: "objecter": { 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: "op_active": 0, 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: "op_laggy": 0, 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send": 7459, 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send_bytes": 3335014, 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: "op_resend": 33, 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: "op_reply": 7459, 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: "op_latency": { 2024-09-19T08:14:05.061 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 7459, 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 187.105977309, 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.025084592 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: "op_inflight": 0, 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: "oplen_avg": { 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 7459, 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 7497 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: "op": 7459, 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: "op_r": 5, 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: "op_w": 7454, 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: "op_rmw": 0, 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: "op_pg": 0, 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_stat": 0, 2024-09-19T08:14:05.062 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_create": 10, 2024-09-19T08:14:05.063 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_read": 3, 2024-09-19T08:14:05.063 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_write": 2779, 2024-09-19T08:14:05.063 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writefull": 17, 2024-09-19T08:14:05.063 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writesame": 0, 2024-09-19T08:14:05.063 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_append": 0, 2024-09-19T08:14:05.063 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_zero": 0, 2024-09-19T08:14:05.063 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_truncate": 0, 2024-09-19T08:14:05.063 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_delete": 4622, 2024-09-19T08:14:05.063 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_mapext": 0, 2024-09-19T08:14:05.063 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_sparse_read": 0, 2024-09-19T08:14:05.063 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_clonerange": 0, 2024-09-19T08:14:05.063 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_getxattr": 2, 2024-09-19T08:14:05.063 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_setxattr": 20, 2024-09-19T08:14:05.063 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_cmpxattr": 0, 2024-09-19T08:14:05.064 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_rmxattr": 0, 2024-09-19T08:14:05.064 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_resetxattrs": 0, 2024-09-19T08:14:05.064 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_call": 0, 2024-09-19T08:14:05.064 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_watch": 0, 2024-09-19T08:14:05.064 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_notify": 0, 2024-09-19T08:14:05.064 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_src_cmpxattr": 0, 2024-09-19T08:14:05.064 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls": 0, 2024-09-19T08:14:05.064 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls_filter": 0, 2024-09-19T08:14:05.064 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_other": 0, 2024-09-19T08:14:05.064 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_active": 0, 2024-09-19T08:14:05.064 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_send": 0, 2024-09-19T08:14:05.064 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_resend": 0, 2024-09-19T08:14:05.064 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_ping": 0, 2024-09-19T08:14:05.064 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_active": 0, 2024-09-19T08:14:05.065 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_send": 0, 2024-09-19T08:14:05.065 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_resend": 0, 2024-09-19T08:14:05.065 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_active": 0, 2024-09-19T08:14:05.065 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_send": 0, 2024-09-19T08:14:05.065 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_resend": 0, 2024-09-19T08:14:05.065 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_active": 0, 2024-09-19T08:14:05.065 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_send": 0, 2024-09-19T08:14:05.065 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_resend": 0, 2024-09-19T08:14:05.065 INFO:teuthology.orchestra.run.smithi139.stdout: "command_active": 0, 2024-09-19T08:14:05.065 INFO:teuthology.orchestra.run.smithi139.stdout: "command_send": 0, 2024-09-19T08:14:05.065 INFO:teuthology.orchestra.run.smithi139.stdout: "command_resend": 0, 2024-09-19T08:14:05.065 INFO:teuthology.orchestra.run.smithi139.stdout: "map_epoch": 89, 2024-09-19T08:14:05.065 INFO:teuthology.orchestra.run.smithi139.stdout: "map_full": 0, 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: "map_inc": 11, 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_sessions": 12, 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_open": 12, 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_close": 0, 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_laggy": 0, 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_wr": 37, 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_rd": 0, 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_del": 7 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: "oft": { 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_objs": 1, 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_kv_pairs": 0, 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_updates": 0, 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_removes": 0 2024-09-19T08:14:05.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: "osd-slow-ops": { 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_ops_count": 23 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: "purge_queue": { 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops": 0, 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops_high_water": 3584, 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing": 0, 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_high_water": 1, 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed_ops": 4610, 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed": 3, 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_item_in_journal": 0 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-09-19T08:14:05.067 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 11151, 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 3456377, 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 11151, 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 11151, 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 3456377, 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.068 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_bytes": { 2024-09-19T08:14:05.069 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:05.069 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-09-19T08:14:05.069 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:05.069 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-09-19T08:14:05.069 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-09-19T08:14:05.069 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:05.069 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:05.069 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 7459, 2024-09-19T08:14:05.069 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 3335333, 2024-09-19T08:14:05.069 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 7459, 2024-09-19T08:14:05.069 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 3335333, 2024-09-19T08:14:05.069 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:05.069 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.070 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.070 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.070 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:05.070 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.070 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_ops": { 2024-09-19T08:14:05.070 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:05.070 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 1024, 2024-09-19T08:14:05.070 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:05.070 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-09-19T08:14:05.070 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-09-19T08:14:05.070 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:05.070 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:05.070 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 7459, 2024-09-19T08:14:05.070 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 7459, 2024-09-19T08:14:05.071 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 7459, 2024-09-19T08:14:05.071 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 7459, 2024-09-19T08:14:05.071 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:05.071 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.071 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.071 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.071 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:05.071 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.071 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle": { 2024-09-19T08:14:05.071 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:05.071 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-09-19T08:14:05.071 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:05.071 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 3, 2024-09-19T08:14:05.071 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 319, 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 3, 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 3, 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 319, 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle-0x5604f10e8960": { 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:05.072 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-09-19T08:14:05.073 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:05.073 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 2807, 2024-09-19T08:14:05.073 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 3320996, 2024-09-19T08:14:05.073 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:05.073 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 2807, 2024-09-19T08:14:05.073 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-09-19T08:14:05.073 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-09-19T08:14:05.073 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 2776, 2024-09-19T08:14:05.073 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 3320996, 2024-09-19T08:14:05.073 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:05.073 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:05.073 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:05.074 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:05.074 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:05.074 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:05.074 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-19T08:14:05.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:04 smithi139 ceph-mon[34874]: pgmap v342: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:14:05.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:04 smithi098 ceph-mon[31333]: pgmap v342: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:14:05.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:04 smithi162 ceph-mon[35260]: pgmap v342: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-09-19T08:14:05.732 INFO:tasks.check_counter:Daemon mds.e mds.exported=3 2024-09-19T08:14:05.732 INFO:tasks.check_counter:Daemon mds.e mds.imported=3 2024-09-19T08:14:05.732 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=3 2024-09-19T08:14:05.732 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=11 2024-09-19T08:14:05.732 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-09-19T08:14:05.732 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-09-19T08:14:05.733 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-09-19T08:14:05.733 DEBUG:tasks.check_counter:Getting stats from h 2024-09-19T08:14:05.733 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph daemon mds.h perf dump 2024-09-19T08:14:05.991 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T08:14:06.247 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:05 smithi139 ceph-mon[34874]: pgmap v343: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:14:06.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:05 smithi162 ceph-mon[35260]: pgmap v343: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:14:06.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:05 smithi098 ceph-mon[31333]: pgmap v343: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:14:06.513 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-19T08:14:06.513 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-0": { 2024-09-19T08:14:06.513 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 805, 2024-09-19T08:14:06.513 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 994, 2024-09-19T08:14:06.513 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 1585576, 2024-09-19T08:14:06.513 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 869990, 2024-09-19T08:14:06.514 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 8, 2024-09-19T08:14:06.514 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 4, 2024-09-19T08:14:06.514 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.121063556, 2024-09-19T08:14:06.514 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.053563734, 2024-09-19T08:14:06.514 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.072026600, 2024-09-19T08:14:06.514 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.004607882, 2024-09-19T08:14:06.514 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:06.514 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 993, 2024-09-19T08:14:06.514 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.059043717, 2024-09-19T08:14:06.514 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000059459 2024-09-19T08:14:06.515 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.515 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:06.515 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1064, 2024-09-19T08:14:06.515 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000267866, 2024-09-19T08:14:06.515 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000251 2024-09-19T08:14:06.515 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.515 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 1403600, 2024-09-19T08:14:06.515 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 466048 2024-09-19T08:14:06.515 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.515 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { 2024-09-19T08:14:06.515 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 66, 2024-09-19T08:14:06.516 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 129, 2024-09-19T08:14:06.516 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 19264, 2024-09-19T08:14:06.516 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 489843, 2024-09-19T08:14:06.516 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 41, 2024-09-19T08:14:06.516 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 8, 2024-09-19T08:14:06.516 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.030860693, 2024-09-19T08:14:06.516 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.006284424, 2024-09-19T08:14:06.516 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.018313415, 2024-09-19T08:14:06.516 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.000237338, 2024-09-19T08:14:06.516 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:06.516 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 126, 2024-09-19T08:14:06.516 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.014465264, 2024-09-19T08:14:06.516 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000114803 2024-09-19T08:14:06.517 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.517 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:06.517 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 108, 2024-09-19T08:14:06.517 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000021546, 2024-09-19T08:14:06.517 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000199 2024-09-19T08:14:06.517 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.517 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-09-19T08:14:06.517 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 127984 2024-09-19T08:14:06.517 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.518 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-2": { 2024-09-19T08:14:06.518 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 55, 2024-09-19T08:14:06.518 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 89, 2024-09-19T08:14:06.518 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 15810, 2024-09-19T08:14:06.518 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 129458, 2024-09-19T08:14:06.518 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 16, 2024-09-19T08:14:06.518 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 5, 2024-09-19T08:14:06.518 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.014442560, 2024-09-19T08:14:06.518 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.002382763, 2024-09-19T08:14:06.518 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.011174293, 2024-09-19T08:14:06.518 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.000630995, 2024-09-19T08:14:06.518 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:06.519 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 88, 2024-09-19T08:14:06.519 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.031017826, 2024-09-19T08:14:06.519 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000352475 2024-09-19T08:14:06.519 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.519 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:06.519 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 89, 2024-09-19T08:14:06.519 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000028854, 2024-09-19T08:14:06.519 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000324 2024-09-19T08:14:06.519 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.519 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 9424, 2024-09-19T08:14:06.519 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 16704 2024-09-19T08:14:06.519 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.520 INFO:teuthology.orchestra.run.smithi139.stdout: "cct": { 2024-09-19T08:14:06.520 INFO:teuthology.orchestra.run.smithi139.stdout: "total_workers": 1, 2024-09-19T08:14:06.520 INFO:teuthology.orchestra.run.smithi139.stdout: "unhealthy_workers": 0 2024-09-19T08:14:06.520 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.520 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-MDSRank": { 2024-09-19T08:14:06.520 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-09-19T08:14:06.520 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-09-19T08:14:06.520 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 64, 2024-09-19T08:14:06.520 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.002039380, 2024-09-19T08:14:06.520 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000031865 2024-09-19T08:14:06.520 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:06.520 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.521 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-PurgeQueue": { 2024-09-19T08:14:06.521 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-09-19T08:14:06.521 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-09-19T08:14:06.521 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2, 2024-09-19T08:14:06.521 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000016626, 2024-09-19T08:14:06.521 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000008313 2024-09-19T08:14:06.521 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:06.521 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.521 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": { 2024-09-19T08:14:06.521 INFO:teuthology.orchestra.run.smithi139.stdout: "request": 10, 2024-09-19T08:14:06.521 INFO:teuthology.orchestra.run.smithi139.stdout: "reply": 6, 2024-09-19T08:14:06.521 INFO:teuthology.orchestra.run.smithi139.stdout: "reply_latency": { 2024-09-19T08:14:06.521 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 6, 2024-09-19T08:14:06.522 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000987148, 2024-09-19T08:14:06.522 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000164524 2024-09-19T08:14:06.522 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.522 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_reply": 0, 2024-09-19T08:14:06.522 INFO:teuthology.orchestra.run.smithi139.stdout: "forward": 4, 2024-09-19T08:14:06.522 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_complete": 0, 2024-09-19T08:14:06.522 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_keys": 0, 2024-09-19T08:14:06.522 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_commit": 11, 2024-09-19T08:14:06.522 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_split": 0, 2024-09-19T08:14:06.522 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_merge": 0, 2024-09-19T08:14:06.522 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes": 16, 2024-09-19T08:14:06.522 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_top": 0, 2024-09-19T08:14:06.522 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_bottom": 16, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pin_tail": 0, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pinned": 13, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_expired": 4, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_with_caps": 2, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "caps": 2, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "subtrees": 3, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse": 28, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_hit": 21, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_forward": 2, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_discover": 4, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_dir_fetch": 0, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_remote_ino": 0, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_lock": 0, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "load_cent": 0, 2024-09-19T08:14:06.523 INFO:teuthology.orchestra.run.smithi139.stdout: "q": 0, 2024-09-19T08:14:06.524 INFO:teuthology.orchestra.run.smithi139.stdout: "exported": 0, 2024-09-19T08:14:06.524 INFO:teuthology.orchestra.run.smithi139.stdout: "exported_inodes": 0, 2024-09-19T08:14:06.524 INFO:teuthology.orchestra.run.smithi139.stdout: "imported": 0, 2024-09-19T08:14:06.524 INFO:teuthology.orchestra.run.smithi139.stdout: "imported_inodes": 0, 2024-09-19T08:14:06.524 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_dir_fetch": 0, 2024-09-19T08:14:06.524 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_backtrace_fetch": 0, 2024-09-19T08:14:06.524 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_peer_discover": 0, 2024-09-19T08:14:06.524 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rfiles": 4, 2024-09-19T08:14:06.524 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rbytes": 234, 2024-09-19T08:14:06.524 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rsnaps": 6, 2024-09-19T08:14:06.525 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_fetch": 0, 2024-09-19T08:14:06.525 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_set_tag": 0, 2024-09-19T08:14:06.525 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_repaired": 0, 2024-09-19T08:14:06.525 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_inotable_repaired": 0, 2024-09-19T08:14:06.525 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_inodes": 0, 2024-09-19T08:14:06.525 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_base_inodes": 0, 2024-09-19T08:14:06.525 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dirfrag_rstats": 0, 2024-09-19T08:14:06.525 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_file_inodes": 0, 2024-09-19T08:14:06.525 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_inode_file_caps": 0, 2024-09-19T08:14:06.525 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_revoke": 4, 2024-09-19T08:14:06.525 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_grant": 0, 2024-09-19T08:14:06.525 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_trunc": 0, 2024-09-19T08:14:06.525 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-19T08:14:06.526 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-19T08:14:06.526 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps": 4, 2024-09-19T08:14:06.526 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps_dirty": 0, 2024-09-19T08:14:06.526 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_cap_release": 0, 2024-09-19T08:14:06.526 INFO:teuthology.orchestra.run.smithi139.stdout: "process_request_cap_release": 0 2024-09-19T08:14:06.526 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.526 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_cache": { 2024-09-19T08:14:06.526 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update": 0, 2024-09-19T08:14:06.526 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update_receipt": 14, 2024-09-19T08:14:06.526 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_try_discover": 9, 2024-09-19T08:14:06.526 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_send_discover": 11, 2024-09-19T08:14:06.526 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_handle_discover": 0, 2024-09-19T08:14:06.527 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays": 0, 2024-09-19T08:14:06.527 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_delayed": 0, 2024-09-19T08:14:06.527 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_enqueuing": 0, 2024-09-19T08:14:06.527 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_created": 0, 2024-09-19T08:14:06.527 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_enqueued": 0, 2024-09-19T08:14:06.527 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_reintegrated": 0, 2024-09-19T08:14:06.527 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_migrated": 0, 2024-09-19T08:14:06.527 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_processing": 0, 2024-09-19T08:14:06.527 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_enqueued": 0, 2024-09-19T08:14:06.527 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_prioritized": 0, 2024-09-19T08:14:06.527 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_started": 0, 2024-09-19T08:14:06.527 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_completed": 0, 2024-09-19T08:14:06.527 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_path": 3, 2024-09-19T08:14:06.528 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_inode": 17, 2024-09-19T08:14:06.528 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_enqueue_scrub": 0, 2024-09-19T08:14:06.528 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_exportdir": 0, 2024-09-19T08:14:06.528 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_flush": 0, 2024-09-19T08:14:06.528 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragmentdir": 0, 2024-09-19T08:14:06.528 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragstats": 0, 2024-09-19T08:14:06.528 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_inodestats": 0 2024-09-19T08:14:06.528 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.528 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_log": { 2024-09-19T08:14:06.528 INFO:teuthology.orchestra.run.smithi139.stdout: "evlrg": 0, 2024-09-19T08:14:06.528 INFO:teuthology.orchestra.run.smithi139.stdout: "evadd": 12, 2024-09-19T08:14:06.528 INFO:teuthology.orchestra.run.smithi139.stdout: "evex": 0, 2024-09-19T08:14:06.528 INFO:teuthology.orchestra.run.smithi139.stdout: "evtrm": 0, 2024-09-19T08:14:06.529 INFO:teuthology.orchestra.run.smithi139.stdout: "ev": 12, 2024-09-19T08:14:06.529 INFO:teuthology.orchestra.run.smithi139.stdout: "evexg": 0, 2024-09-19T08:14:06.529 INFO:teuthology.orchestra.run.smithi139.stdout: "evexd": 0, 2024-09-19T08:14:06.529 INFO:teuthology.orchestra.run.smithi139.stdout: "segadd": 2, 2024-09-19T08:14:06.529 INFO:teuthology.orchestra.run.smithi139.stdout: "segex": 0, 2024-09-19T08:14:06.529 INFO:teuthology.orchestra.run.smithi139.stdout: "segtrm": 0, 2024-09-19T08:14:06.529 INFO:teuthology.orchestra.run.smithi139.stdout: "seg": 2, 2024-09-19T08:14:06.529 INFO:teuthology.orchestra.run.smithi139.stdout: "segmjr": 2, 2024-09-19T08:14:06.529 INFO:teuthology.orchestra.run.smithi139.stdout: "segexg": 0, 2024-09-19T08:14:06.529 INFO:teuthology.orchestra.run.smithi139.stdout: "segexd": 0, 2024-09-19T08:14:06.529 INFO:teuthology.orchestra.run.smithi139.stdout: "expos": 4194304, 2024-09-19T08:14:06.529 INFO:teuthology.orchestra.run.smithi139.stdout: "wrpos": 4197729, 2024-09-19T08:14:06.529 INFO:teuthology.orchestra.run.smithi139.stdout: "rdpos": 0, 2024-09-19T08:14:06.530 INFO:teuthology.orchestra.run.smithi139.stdout: "jlat": { 2024-09-19T08:14:06.530 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 11, 2024-09-19T08:14:06.530 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.018466542, 2024-09-19T08:14:06.530 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.001678776 2024-09-19T08:14:06.530 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.530 INFO:teuthology.orchestra.run.smithi139.stdout: "replayed": 0 2024-09-19T08:14:06.530 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.530 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_mem": { 2024-09-19T08:14:06.530 INFO:teuthology.orchestra.run.smithi139.stdout: "ino": 20, 2024-09-19T08:14:06.530 INFO:teuthology.orchestra.run.smithi139.stdout: "ino+": 22, 2024-09-19T08:14:06.530 INFO:teuthology.orchestra.run.smithi139.stdout: "ino-": 2, 2024-09-19T08:14:06.530 INFO:teuthology.orchestra.run.smithi139.stdout: "dir": 19, 2024-09-19T08:14:06.530 INFO:teuthology.orchestra.run.smithi139.stdout: "dir+": 23, 2024-09-19T08:14:06.530 INFO:teuthology.orchestra.run.smithi139.stdout: "dir-": 4, 2024-09-19T08:14:06.531 INFO:teuthology.orchestra.run.smithi139.stdout: "dn": 16, 2024-09-19T08:14:06.531 INFO:teuthology.orchestra.run.smithi139.stdout: "dn+": 20, 2024-09-19T08:14:06.531 INFO:teuthology.orchestra.run.smithi139.stdout: "dn-": 4, 2024-09-19T08:14:06.531 INFO:teuthology.orchestra.run.smithi139.stdout: "cap": 2, 2024-09-19T08:14:06.531 INFO:teuthology.orchestra.run.smithi139.stdout: "cap+": 4, 2024-09-19T08:14:06.531 INFO:teuthology.orchestra.run.smithi139.stdout: "cap-": 2, 2024-09-19T08:14:06.531 INFO:teuthology.orchestra.run.smithi139.stdout: "rss": 50500, 2024-09-19T08:14:06.531 INFO:teuthology.orchestra.run.smithi139.stdout: "heap": 223520 2024-09-19T08:14:06.531 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.531 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_server": { 2024-09-19T08:14:06.531 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_client_request": 10, 2024-09-19T08:14:06.531 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_server_request": 0, 2024-09-19T08:14:06.532 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_request": 10, 2024-09-19T08:14:06.532 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_session": 34, 2024-09-19T08:14:06.532 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_peer_request": 0, 2024-09-19T08:14:06.532 INFO:teuthology.orchestra.run.smithi139.stdout: "req_create_latency": { 2024-09-19T08:14:06.532 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.532 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.532 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.532 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.532 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getattr_latency": { 2024-09-19T08:14:06.532 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 4, 2024-09-19T08:14:06.532 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000674081, 2024-09-19T08:14:06.532 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000168520 2024-09-19T08:14:06.533 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.533 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getfilelock_latency": { 2024-09-19T08:14:06.533 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.533 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.533 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.533 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.533 INFO:teuthology.orchestra.run.smithi139.stdout: "req_link_latency": { 2024-09-19T08:14:06.533 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.533 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.533 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.533 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.533 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookup_latency": { 2024-09-19T08:14:06.534 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1, 2024-09-19T08:14:06.534 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000134835, 2024-09-19T08:14:06.534 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000134835 2024-09-19T08:14:06.534 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.534 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookuphash_latency": { 2024-09-19T08:14:06.534 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.534 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.534 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.534 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.534 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupino_latency": { 2024-09-19T08:14:06.534 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1, 2024-09-19T08:14:06.534 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000178232, 2024-09-19T08:14:06.535 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000178232 2024-09-19T08:14:06.535 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.535 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupname_latency": { 2024-09-19T08:14:06.535 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.535 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.535 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.535 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.535 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupparent_latency": { 2024-09-19T08:14:06.535 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.535 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.535 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.535 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.535 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupsnap_latency": { 2024-09-19T08:14:06.536 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.536 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.536 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.536 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lssnap_latency": { 2024-09-19T08:14:06.536 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.536 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.536 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.536 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mkdir_latency": { 2024-09-19T08:14:06.537 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.537 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.537 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.537 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.537 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mknod_latency": { 2024-09-19T08:14:06.538 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.538 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.538 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.538 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.538 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mksnap_latency": { 2024-09-19T08:14:06.538 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.538 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.538 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.538 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.538 INFO:teuthology.orchestra.run.smithi139.stdout: "req_open_latency": { 2024-09-19T08:14:06.538 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.538 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.539 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.539 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.539 INFO:teuthology.orchestra.run.smithi139.stdout: "req_readdir_latency": { 2024-09-19T08:14:06.539 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.539 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.539 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.539 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.539 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rename_latency": { 2024-09-19T08:14:06.539 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.539 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.539 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.539 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.540 INFO:teuthology.orchestra.run.smithi139.stdout: "req_renamesnap_latency": { 2024-09-19T08:14:06.540 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.540 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.540 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.540 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.540 INFO:teuthology.orchestra.run.smithi139.stdout: "req_snapdiff_latency": { 2024-09-19T08:14:06.540 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.540 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.540 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.540 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.540 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmdir_latency": { 2024-09-19T08:14:06.540 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.540 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.540 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.541 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.541 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmsnap_latency": { 2024-09-19T08:14:06.541 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.541 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.541 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.541 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.541 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmxattr_latency": { 2024-09-19T08:14:06.541 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.541 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.541 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.541 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.541 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setattr_latency": { 2024-09-19T08:14:06.541 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.542 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.542 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.542 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.542 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setdirlayout_latency": { 2024-09-19T08:14:06.542 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.542 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.542 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.542 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.542 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setfilelock_latency": { 2024-09-19T08:14:06.542 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.542 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.542 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.543 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.543 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setlayout_latency": { 2024-09-19T08:14:06.543 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.543 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.543 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.543 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.543 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setxattr_latency": { 2024-09-19T08:14:06.543 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.543 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.544 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.544 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.544 INFO:teuthology.orchestra.run.smithi139.stdout: "req_symlink_latency": { 2024-09-19T08:14:06.544 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.544 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.544 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.544 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.544 INFO:teuthology.orchestra.run.smithi139.stdout: "req_unlink_latency": { 2024-09-19T08:14:06.544 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.544 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.545 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.545 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_revoke_eviction": 0, 2024-09-19T08:14:06.545 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_acquisition_throttle": 0, 2024-09-19T08:14:06.545 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getvxattr_latency": { 2024-09-19T08:14:06.545 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.545 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.545 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.545 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:06.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.545 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_sessions": { 2024-09-19T08:14:06.546 INFO:teuthology.orchestra.run.smithi139.stdout: "session_count": 2, 2024-09-19T08:14:06.546 INFO:teuthology.orchestra.run.smithi139.stdout: "session_add": 6, 2024-09-19T08:14:06.546 INFO:teuthology.orchestra.run.smithi139.stdout: "session_remove": 4, 2024-09-19T08:14:06.546 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_open": 2, 2024-09-19T08:14:06.546 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_stale": 0, 2024-09-19T08:14:06.546 INFO:teuthology.orchestra.run.smithi139.stdout: "total_load": 4, 2024-09-19T08:14:06.546 INFO:teuthology.orchestra.run.smithi139.stdout: "average_load": 0, 2024-09-19T08:14:06.546 INFO:teuthology.orchestra.run.smithi139.stdout: "avg_session_uptime": 633, 2024-09-19T08:14:06.546 INFO:teuthology.orchestra.run.smithi139.stdout: "mdthresh_evicted": 0 2024-09-19T08:14:06.546 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.546 INFO:teuthology.orchestra.run.smithi139.stdout: "mempool": { 2024-09-19T08:14:06.546 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_bytes": 0, 2024-09-19T08:14:06.547 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_items": 0, 2024-09-19T08:14:06.547 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_bytes": 0, 2024-09-19T08:14:06.547 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_items": 0, 2024-09-19T08:14:06.547 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_bytes": 0, 2024-09-19T08:14:06.547 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_items": 0, 2024-09-19T08:14:06.547 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-19T08:14:06.547 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_items": 0, 2024-09-19T08:14:06.547 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-19T08:14:06.547 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_items": 0, 2024-09-19T08:14:06.547 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_bytes": 0, 2024-09-19T08:14:06.547 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_items": 0, 2024-09-19T08:14:06.547 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-19T08:14:06.547 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_items": 0, 2024-09-19T08:14:06.547 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_bytes": 0, 2024-09-19T08:14:06.548 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_items": 0, 2024-09-19T08:14:06.548 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_bytes": 0, 2024-09-19T08:14:06.548 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_items": 0, 2024-09-19T08:14:06.548 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-19T08:14:06.548 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_items": 0, 2024-09-19T08:14:06.548 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-19T08:14:06.548 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_items": 0, 2024-09-19T08:14:06.548 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_bytes": 0, 2024-09-19T08:14:06.548 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_items": 0, 2024-09-19T08:14:06.548 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_bytes": 0, 2024-09-19T08:14:06.548 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_items": 0, 2024-09-19T08:14:06.548 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-19T08:14:06.548 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_items": 0, 2024-09-19T08:14:06.549 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_bytes": 0, 2024-09-19T08:14:06.549 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_items": 0, 2024-09-19T08:14:06.549 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_bytes": 0, 2024-09-19T08:14:06.549 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_items": 0, 2024-09-19T08:14:06.549 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_bytes": 0, 2024-09-19T08:14:06.549 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_items": 0, 2024-09-19T08:14:06.549 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_bytes": 0, 2024-09-19T08:14:06.549 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_items": 0, 2024-09-19T08:14:06.549 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_bytes": 270935, 2024-09-19T08:14:06.549 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_items": 96, 2024-09-19T08:14:06.549 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_bytes": 0, 2024-09-19T08:14:06.549 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_items": 0, 2024-09-19T08:14:06.550 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_bytes": 0, 2024-09-19T08:14:06.550 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_items": 0, 2024-09-19T08:14:06.550 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_bytes": 0, 2024-09-19T08:14:06.550 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_items": 0, 2024-09-19T08:14:06.550 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_bytes": 0, 2024-09-19T08:14:06.550 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_items": 0, 2024-09-19T08:14:06.550 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_bytes": 8136, 2024-09-19T08:14:06.550 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_items": 180, 2024-09-19T08:14:06.550 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_bytes": 0, 2024-09-19T08:14:06.550 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_items": 0, 2024-09-19T08:14:06.550 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_bytes": 0, 2024-09-19T08:14:06.550 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, 2024-09-19T08:14:06.550 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 82581, 2024-09-19T08:14:06.550 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 338, 2024-09-19T08:14:06.551 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_bytes": 0, 2024-09-19T08:14:06.551 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_items": 0, 2024-09-19T08:14:06.551 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_bytes": 0, 2024-09-19T08:14:06.551 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_items": 0 2024-09-19T08:14:06.551 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.551 INFO:teuthology.orchestra.run.smithi139.stdout: "objecter": { 2024-09-19T08:14:06.551 INFO:teuthology.orchestra.run.smithi139.stdout: "op_active": 0, 2024-09-19T08:14:06.551 INFO:teuthology.orchestra.run.smithi139.stdout: "op_laggy": 0, 2024-09-19T08:14:06.551 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send": 51, 2024-09-19T08:14:06.551 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send_bytes": 14168, 2024-09-19T08:14:06.551 INFO:teuthology.orchestra.run.smithi139.stdout: "op_resend": 33, 2024-09-19T08:14:06.551 INFO:teuthology.orchestra.run.smithi139.stdout: "op_reply": 51, 2024-09-19T08:14:06.551 INFO:teuthology.orchestra.run.smithi139.stdout: "op_latency": { 2024-09-19T08:14:06.551 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 51, 2024-09-19T08:14:06.552 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.231995046, 2024-09-19T08:14:06.552 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.004548922 2024-09-19T08:14:06.552 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.552 INFO:teuthology.orchestra.run.smithi139.stdout: "op_inflight": 0, 2024-09-19T08:14:06.552 INFO:teuthology.orchestra.run.smithi139.stdout: "oplen_avg": { 2024-09-19T08:14:06.552 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 51, 2024-09-19T08:14:06.552 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 76 2024-09-19T08:14:06.552 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.552 INFO:teuthology.orchestra.run.smithi139.stdout: "op": 51, 2024-09-19T08:14:06.552 INFO:teuthology.orchestra.run.smithi139.stdout: "op_r": 0, 2024-09-19T08:14:06.552 INFO:teuthology.orchestra.run.smithi139.stdout: "op_w": 51, 2024-09-19T08:14:06.552 INFO:teuthology.orchestra.run.smithi139.stdout: "op_rmw": 0, 2024-09-19T08:14:06.552 INFO:teuthology.orchestra.run.smithi139.stdout: "op_pg": 0, 2024-09-19T08:14:06.553 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_stat": 0, 2024-09-19T08:14:06.553 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_create": 10, 2024-09-19T08:14:06.553 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_read": 0, 2024-09-19T08:14:06.553 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_write": 11, 2024-09-19T08:14:06.553 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writefull": 8, 2024-09-19T08:14:06.553 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writesame": 0, 2024-09-19T08:14:06.553 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_append": 0, 2024-09-19T08:14:06.553 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_zero": 0, 2024-09-19T08:14:06.553 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_truncate": 0, 2024-09-19T08:14:06.553 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_delete": 6, 2024-09-19T08:14:06.553 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_mapext": 0, 2024-09-19T08:14:06.553 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_sparse_read": 0, 2024-09-19T08:14:06.554 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_clonerange": 0, 2024-09-19T08:14:06.554 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_getxattr": 0, 2024-09-19T08:14:06.554 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_setxattr": 20, 2024-09-19T08:14:06.554 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_cmpxattr": 0, 2024-09-19T08:14:06.554 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_rmxattr": 0, 2024-09-19T08:14:06.554 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_resetxattrs": 0, 2024-09-19T08:14:06.554 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_call": 0, 2024-09-19T08:14:06.554 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_watch": 0, 2024-09-19T08:14:06.554 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_notify": 0, 2024-09-19T08:14:06.554 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_src_cmpxattr": 0, 2024-09-19T08:14:06.554 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls": 0, 2024-09-19T08:14:06.554 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls_filter": 0, 2024-09-19T08:14:06.554 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_other": 0, 2024-09-19T08:14:06.555 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_active": 0, 2024-09-19T08:14:06.555 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_send": 0, 2024-09-19T08:14:06.555 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_resend": 0, 2024-09-19T08:14:06.555 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_ping": 0, 2024-09-19T08:14:06.555 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_active": 0, 2024-09-19T08:14:06.555 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_send": 0, 2024-09-19T08:14:06.555 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_resend": 0, 2024-09-19T08:14:06.555 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_active": 0, 2024-09-19T08:14:06.555 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_send": 0, 2024-09-19T08:14:06.555 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_resend": 0, 2024-09-19T08:14:06.555 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_active": 0, 2024-09-19T08:14:06.555 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_send": 0, 2024-09-19T08:14:06.555 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_resend": 0, 2024-09-19T08:14:06.556 INFO:teuthology.orchestra.run.smithi139.stdout: "command_active": 0, 2024-09-19T08:14:06.556 INFO:teuthology.orchestra.run.smithi139.stdout: "command_send": 0, 2024-09-19T08:14:06.556 INFO:teuthology.orchestra.run.smithi139.stdout: "command_resend": 0, 2024-09-19T08:14:06.556 INFO:teuthology.orchestra.run.smithi139.stdout: "map_epoch": 89, 2024-09-19T08:14:06.556 INFO:teuthology.orchestra.run.smithi139.stdout: "map_full": 0, 2024-09-19T08:14:06.556 INFO:teuthology.orchestra.run.smithi139.stdout: "map_inc": 11, 2024-09-19T08:14:06.556 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_sessions": 9, 2024-09-19T08:14:06.556 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_open": 9, 2024-09-19T08:14:06.556 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_close": 0, 2024-09-19T08:14:06.556 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_laggy": 0, 2024-09-19T08:14:06.556 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_wr": 20, 2024-09-19T08:14:06.556 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_rd": 0, 2024-09-19T08:14:06.557 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_del": 1 2024-09-19T08:14:06.557 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.557 INFO:teuthology.orchestra.run.smithi139.stdout: "oft": { 2024-09-19T08:14:06.557 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_objs": 1, 2024-09-19T08:14:06.557 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_kv_pairs": 0, 2024-09-19T08:14:06.557 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_updates": 0, 2024-09-19T08:14:06.557 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_removes": 0 2024-09-19T08:14:06.557 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.557 INFO:teuthology.orchestra.run.smithi139.stdout: "osd-slow-ops": { 2024-09-19T08:14:06.557 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_ops_count": 20 2024-09-19T08:14:06.557 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.557 INFO:teuthology.orchestra.run.smithi139.stdout: "purge_queue": { 2024-09-19T08:14:06.557 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops": 0, 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops_high_water": 0, 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing": 0, 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_high_water": 0, 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed_ops": 0, 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed": 0, 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_item_in_journal": 0 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 922, 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 1518782, 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:06.558 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 922, 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 922, 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 1518782, 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_bytes": { 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:06.559 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-09-19T08:14:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-09-19T08:14:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 51, 2024-09-19T08:14:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 14168, 2024-09-19T08:14:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 51, 2024-09-19T08:14:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 14168, 2024-09-19T08:14:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_ops": { 2024-09-19T08:14:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 1024, 2024-09-19T08:14:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-09-19T08:14:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-09-19T08:14:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 51, 2024-09-19T08:14:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 51, 2024-09-19T08:14:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 51, 2024-09-19T08:14:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 51, 2024-09-19T08:14:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:06.562 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.562 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.562 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.562 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:06.562 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.562 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle": { 2024-09-19T08:14:06.562 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:06.562 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-09-19T08:14:06.562 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:06.562 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-09-19T08:14:06.562 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-09-19T08:14:06.562 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:06.562 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:06.563 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-09-19T08:14:06.563 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-09-19T08:14:06.563 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 0, 2024-09-19T08:14:06.563 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 0, 2024-09-19T08:14:06.563 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:06.563 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.563 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.563 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.563 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:06.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:06.563 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle-0x559eb4e4e960": { 2024-09-19T08:14:06.563 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:06.564 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-09-19T08:14:06.564 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:06.564 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 12, 2024-09-19T08:14:06.564 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 3425, 2024-09-19T08:14:06.564 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:06.564 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 12, 2024-09-19T08:14:06.564 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-09-19T08:14:06.564 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-09-19T08:14:06.564 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 11, 2024-09-19T08:14:06.564 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 3425, 2024-09-19T08:14:06.564 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:06.564 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:06.565 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:06.565 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:06.565 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:06.565 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:06.565 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-19T08:14:07.292 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-09-19T08:14:07.293 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-09-19T08:14:07.293 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-09-19T08:14:07.293 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=14 2024-09-19T08:14:07.293 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-09-19T08:14:07.293 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-09-19T08:14:07.293 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-09-19T08:14:07.293 DEBUG:tasks.check_counter:Getting stats from k 2024-09-19T08:14:07.293 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph daemon mds.k perf dump 2024-09-19T08:14:07.537 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T08:14:08.060 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-19T08:14:08.060 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-0": { 2024-09-19T08:14:08.060 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 802, 2024-09-19T08:14:08.060 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 802, 2024-09-19T08:14:08.060 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 3533239, 2024-09-19T08:14:08.060 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 254276, 2024-09-19T08:14:08.060 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 6, 2024-09-19T08:14:08.060 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 2, 2024-09-19T08:14:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.077685120, 2024-09-19T08:14:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.030068879, 2024-09-19T08:14:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.020859816, 2024-09-19T08:14:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.018860541, 2024-09-19T08:14:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 801, 2024-09-19T08:14:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.036558680, 2024-09-19T08:14:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000045641 2024-09-19T08:14:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-09-19T08:14:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 448 2024-09-19T08:14:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { 2024-09-19T08:14:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 915, 2024-09-19T08:14:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 896, 2024-09-19T08:14:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 1556042, 2024-09-19T08:14:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 347593, 2024-09-19T08:14:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 4, 2024-09-19T08:14:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 3, 2024-09-19T08:14:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.101996600, 2024-09-19T08:14:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.034335376, 2024-09-19T08:14:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.052394030, 2024-09-19T08:14:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.022016889, 2024-09-19T08:14:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 895, 2024-09-19T08:14:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.038640583, 2024-09-19T08:14:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000043173 2024-09-19T08:14:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 1393168, 2024-09-19T08:14:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 137072 2024-09-19T08:14:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-2": { 2024-09-19T08:14:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 763, 2024-09-19T08:14:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 929, 2024-09-19T08:14:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 256016, 2024-09-19T08:14:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 694734, 2024-09-19T08:14:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 6, 2024-09-19T08:14:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 4, 2024-09-19T08:14:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.101380898, 2024-09-19T08:14:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.035737805, 2024-09-19T08:14:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.025140816, 2024-09-19T08:14:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.029383740, 2024-09-19T08:14:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 926, 2024-09-19T08:14:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.057370664, 2024-09-19T08:14:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000061955 2024-09-19T08:14:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-19T08:14:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 452736 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: "cct": { 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: "total_workers": 1, 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: "unhealthy_workers": 0 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-MDSRank": { 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 3703, 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.107968721, 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000029157 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-PurgeQueue": { 2024-09-19T08:14:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-09-19T08:14:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-09-19T08:14:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": { 2024-09-19T08:14:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "request": 0, 2024-09-19T08:14:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "reply": 0, 2024-09-19T08:14:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "reply_latency": { 2024-09-19T08:14:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_reply": 0, 2024-09-19T08:14:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "forward": 0, 2024-09-19T08:14:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_complete": 0, 2024-09-19T08:14:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_keys": 0, 2024-09-19T08:14:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_commit": 0, 2024-09-19T08:14:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_split": 0, 2024-09-19T08:14:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_merge": 0, 2024-09-19T08:14:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes": 5, 2024-09-19T08:14:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_top": 0, 2024-09-19T08:14:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_bottom": 5, 2024-09-19T08:14:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pin_tail": 0, 2024-09-19T08:14:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pinned": 5, 2024-09-19T08:14:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_expired": 0, 2024-09-19T08:14:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_with_caps": 0, 2024-09-19T08:14:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "caps": 0, 2024-09-19T08:14:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "subtrees": 1, 2024-09-19T08:14:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse": 0, 2024-09-19T08:14:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_hit": 0, 2024-09-19T08:14:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_forward": 0, 2024-09-19T08:14:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_discover": 0, 2024-09-19T08:14:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_dir_fetch": 0, 2024-09-19T08:14:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_remote_ino": 0, 2024-09-19T08:14:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_lock": 0, 2024-09-19T08:14:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "load_cent": 0, 2024-09-19T08:14:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "q": 0, 2024-09-19T08:14:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "exported": 0, 2024-09-19T08:14:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "exported_inodes": 0, 2024-09-19T08:14:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "imported": 0, 2024-09-19T08:14:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "imported_inodes": 0, 2024-09-19T08:14:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_dir_fetch": 0, 2024-09-19T08:14:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_backtrace_fetch": 0, 2024-09-19T08:14:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_peer_discover": 0, 2024-09-19T08:14:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rfiles": 0, 2024-09-19T08:14:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rbytes": 0, 2024-09-19T08:14:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rsnaps": 0, 2024-09-19T08:14:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_fetch": 0, 2024-09-19T08:14:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_set_tag": 0, 2024-09-19T08:14:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_repaired": 0, 2024-09-19T08:14:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_inotable_repaired": 0, 2024-09-19T08:14:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_inodes": 0, 2024-09-19T08:14:08.071 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_base_inodes": 0, 2024-09-19T08:14:08.071 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dirfrag_rstats": 0, 2024-09-19T08:14:08.071 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_file_inodes": 0, 2024-09-19T08:14:08.071 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_inode_file_caps": 0, 2024-09-19T08:14:08.071 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_revoke": 0, 2024-09-19T08:14:08.071 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_grant": 0, 2024-09-19T08:14:08.071 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_trunc": 0, 2024-09-19T08:14:08.071 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-19T08:14:08.071 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-19T08:14:08.071 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps": 0, 2024-09-19T08:14:08.071 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps_dirty": 0, 2024-09-19T08:14:08.071 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_cap_release": 0, 2024-09-19T08:14:08.071 INFO:teuthology.orchestra.run.smithi139.stdout: "process_request_cap_release": 0 2024-09-19T08:14:08.071 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_cache": { 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update": 0, 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update_receipt": 0, 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_try_discover": 0, 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_send_discover": 0, 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_handle_discover": 0, 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays": 0, 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_delayed": 0, 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_enqueuing": 0, 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_created": 0, 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_enqueued": 0, 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_reintegrated": 0, 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_migrated": 0, 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_processing": 0, 2024-09-19T08:14:08.072 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_enqueued": 0, 2024-09-19T08:14:08.073 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_prioritized": 0, 2024-09-19T08:14:08.073 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_started": 0, 2024-09-19T08:14:08.073 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_completed": 0, 2024-09-19T08:14:08.073 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_path": 0, 2024-09-19T08:14:08.073 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_inode": 0, 2024-09-19T08:14:08.073 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_enqueue_scrub": 0, 2024-09-19T08:14:08.073 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_exportdir": 0, 2024-09-19T08:14:08.073 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_flush": 0, 2024-09-19T08:14:08.076 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragmentdir": 0, 2024-09-19T08:14:08.076 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragstats": 0, 2024-09-19T08:14:08.076 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_inodestats": 0 2024-09-19T08:14:08.076 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.076 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_log": { 2024-09-19T08:14:08.076 INFO:teuthology.orchestra.run.smithi139.stdout: "evlrg": 0, 2024-09-19T08:14:08.077 INFO:teuthology.orchestra.run.smithi139.stdout: "evadd": 0, 2024-09-19T08:14:08.077 INFO:teuthology.orchestra.run.smithi139.stdout: "evex": 0, 2024-09-19T08:14:08.077 INFO:teuthology.orchestra.run.smithi139.stdout: "evtrm": 0, 2024-09-19T08:14:08.077 INFO:teuthology.orchestra.run.smithi139.stdout: "ev": 2807, 2024-09-19T08:14:08.077 INFO:teuthology.orchestra.run.smithi139.stdout: "evexg": 0, 2024-09-19T08:14:08.077 INFO:teuthology.orchestra.run.smithi139.stdout: "evexd": 0, 2024-09-19T08:14:08.077 INFO:teuthology.orchestra.run.smithi139.stdout: "segadd": 0, 2024-09-19T08:14:08.077 INFO:teuthology.orchestra.run.smithi139.stdout: "segex": 0, 2024-09-19T08:14:08.077 INFO:teuthology.orchestra.run.smithi139.stdout: "segtrm": 0, 2024-09-19T08:14:08.077 INFO:teuthology.orchestra.run.smithi139.stdout: "seg": 4, 2024-09-19T08:14:08.077 INFO:teuthology.orchestra.run.smithi139.stdout: "segmjr": 2, 2024-09-19T08:14:08.077 INFO:teuthology.orchestra.run.smithi139.stdout: "segexg": 0, 2024-09-19T08:14:08.077 INFO:teuthology.orchestra.run.smithi139.stdout: "segexd": 0, 2024-09-19T08:14:08.078 INFO:teuthology.orchestra.run.smithi139.stdout: "expos": 4194304, 2024-09-19T08:14:08.078 INFO:teuthology.orchestra.run.smithi139.stdout: "wrpos": 7515300, 2024-09-19T08:14:08.078 INFO:teuthology.orchestra.run.smithi139.stdout: "rdpos": 7514701, 2024-09-19T08:14:08.078 INFO:teuthology.orchestra.run.smithi139.stdout: "jlat": { 2024-09-19T08:14:08.078 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.078 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.078 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.078 INFO:teuthology.orchestra.run.smithi139.stdout: "replayed": 2807 2024-09-19T08:14:08.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.078 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_mem": { 2024-09-19T08:14:08.078 INFO:teuthology.orchestra.run.smithi139.stdout: "ino": 5, 2024-09-19T08:14:08.078 INFO:teuthology.orchestra.run.smithi139.stdout: "ino+": 30, 2024-09-19T08:14:08.079 INFO:teuthology.orchestra.run.smithi139.stdout: "ino-": 25, 2024-09-19T08:14:08.079 INFO:teuthology.orchestra.run.smithi139.stdout: "dir": 3, 2024-09-19T08:14:08.079 INFO:teuthology.orchestra.run.smithi139.stdout: "dir+": 24, 2024-09-19T08:14:08.079 INFO:teuthology.orchestra.run.smithi139.stdout: "dir-": 21, 2024-09-19T08:14:08.079 INFO:teuthology.orchestra.run.smithi139.stdout: "dn": 5, 2024-09-19T08:14:08.079 INFO:teuthology.orchestra.run.smithi139.stdout: "dn+": 36, 2024-09-19T08:14:08.079 INFO:teuthology.orchestra.run.smithi139.stdout: "dn-": 31, 2024-09-19T08:14:08.079 INFO:teuthology.orchestra.run.smithi139.stdout: "cap": 0, 2024-09-19T08:14:08.079 INFO:teuthology.orchestra.run.smithi139.stdout: "cap+": 0, 2024-09-19T08:14:08.079 INFO:teuthology.orchestra.run.smithi139.stdout: "cap-": 0, 2024-09-19T08:14:08.079 INFO:teuthology.orchestra.run.smithi139.stdout: "rss": 49640, 2024-09-19T08:14:08.079 INFO:teuthology.orchestra.run.smithi139.stdout: "heap": 215328 2024-09-19T08:14:08.079 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.079 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_server": { 2024-09-19T08:14:08.080 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_client_request": 0, 2024-09-19T08:14:08.080 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_server_request": 0, 2024-09-19T08:14:08.080 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_request": 0, 2024-09-19T08:14:08.080 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_session": 0, 2024-09-19T08:14:08.080 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_peer_request": 0, 2024-09-19T08:14:08.080 INFO:teuthology.orchestra.run.smithi139.stdout: "req_create_latency": { 2024-09-19T08:14:08.080 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.080 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.080 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.080 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.080 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getattr_latency": { 2024-09-19T08:14:08.080 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.080 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.081 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.081 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getfilelock_latency": { 2024-09-19T08:14:08.081 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.081 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.081 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.081 INFO:teuthology.orchestra.run.smithi139.stdout: "req_link_latency": { 2024-09-19T08:14:08.081 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.081 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.081 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.081 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookup_latency": { 2024-09-19T08:14:08.082 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.082 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.082 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.082 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.082 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookuphash_latency": { 2024-09-19T08:14:08.082 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.082 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.082 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.082 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.082 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupino_latency": { 2024-09-19T08:14:08.082 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.082 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.082 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.083 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.083 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupname_latency": { 2024-09-19T08:14:08.083 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.083 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.083 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.083 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.083 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupparent_latency": { 2024-09-19T08:14:08.083 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.083 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.083 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.083 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.083 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupsnap_latency": { 2024-09-19T08:14:08.083 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.084 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.084 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.084 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lssnap_latency": { 2024-09-19T08:14:08.084 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.084 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.084 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.084 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mkdir_latency": { 2024-09-19T08:14:08.084 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.085 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.085 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.085 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.085 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mknod_latency": { 2024-09-19T08:14:08.085 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.085 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.085 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.086 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.086 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mksnap_latency": { 2024-09-19T08:14:08.086 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.086 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.086 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.086 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.086 INFO:teuthology.orchestra.run.smithi139.stdout: "req_open_latency": { 2024-09-19T08:14:08.086 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.086 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.086 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.086 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.086 INFO:teuthology.orchestra.run.smithi139.stdout: "req_readdir_latency": { 2024-09-19T08:14:08.086 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.087 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.087 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.087 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rename_latency": { 2024-09-19T08:14:08.087 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.087 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.087 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.087 INFO:teuthology.orchestra.run.smithi139.stdout: "req_renamesnap_latency": { 2024-09-19T08:14:08.087 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.087 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.087 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.087 INFO:teuthology.orchestra.run.smithi139.stdout: "req_snapdiff_latency": { 2024-09-19T08:14:08.088 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.088 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.088 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.088 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmdir_latency": { 2024-09-19T08:14:08.088 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.088 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.088 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.088 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmsnap_latency": { 2024-09-19T08:14:08.088 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.088 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.089 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.089 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmxattr_latency": { 2024-09-19T08:14:08.089 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.089 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.089 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.089 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setattr_latency": { 2024-09-19T08:14:08.089 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.089 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.089 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.089 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setdirlayout_latency": { 2024-09-19T08:14:08.090 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.090 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.090 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.090 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setfilelock_latency": { 2024-09-19T08:14:08.090 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.090 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.090 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.090 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setlayout_latency": { 2024-09-19T08:14:08.090 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.090 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.090 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.091 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setxattr_latency": { 2024-09-19T08:14:08.091 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.091 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.091 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.091 INFO:teuthology.orchestra.run.smithi139.stdout: "req_symlink_latency": { 2024-09-19T08:14:08.091 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.091 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.091 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.091 INFO:teuthology.orchestra.run.smithi139.stdout: "req_unlink_latency": { 2024-09-19T08:14:08.091 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.092 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.092 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.092 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.092 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_revoke_eviction": 0, 2024-09-19T08:14:08.092 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_acquisition_throttle": 0, 2024-09-19T08:14:08.092 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getvxattr_latency": { 2024-09-19T08:14:08.092 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.092 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.092 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.092 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:08.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.093 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_sessions": { 2024-09-19T08:14:08.093 INFO:teuthology.orchestra.run.smithi139.stdout: "session_count": 1, 2024-09-19T08:14:08.093 INFO:teuthology.orchestra.run.smithi139.stdout: "session_add": 3, 2024-09-19T08:14:08.093 INFO:teuthology.orchestra.run.smithi139.stdout: "session_remove": 2, 2024-09-19T08:14:08.093 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_open": 1, 2024-09-19T08:14:08.093 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_stale": 0, 2024-09-19T08:14:08.093 INFO:teuthology.orchestra.run.smithi139.stdout: "total_load": 0, 2024-09-19T08:14:08.093 INFO:teuthology.orchestra.run.smithi139.stdout: "average_load": 0, 2024-09-19T08:14:08.093 INFO:teuthology.orchestra.run.smithi139.stdout: "avg_session_uptime": 1962, 2024-09-19T08:14:08.093 INFO:teuthology.orchestra.run.smithi139.stdout: "mdthresh_evicted": 0 2024-09-19T08:14:08.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.093 INFO:teuthology.orchestra.run.smithi139.stdout: "mempool": { 2024-09-19T08:14:08.093 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_bytes": 0, 2024-09-19T08:14:08.094 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_items": 0, 2024-09-19T08:14:08.094 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_bytes": 0, 2024-09-19T08:14:08.094 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_items": 0, 2024-09-19T08:14:08.094 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_bytes": 0, 2024-09-19T08:14:08.094 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_items": 0, 2024-09-19T08:14:08.094 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-19T08:14:08.094 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_items": 0, 2024-09-19T08:14:08.094 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-19T08:14:08.094 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_items": 0, 2024-09-19T08:14:08.094 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_bytes": 0, 2024-09-19T08:14:08.094 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_items": 0, 2024-09-19T08:14:08.094 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-19T08:14:08.094 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_items": 0, 2024-09-19T08:14:08.094 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_bytes": 0, 2024-09-19T08:14:08.095 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_items": 0, 2024-09-19T08:14:08.095 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_bytes": 0, 2024-09-19T08:14:08.095 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_items": 0, 2024-09-19T08:14:08.095 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-19T08:14:08.095 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_items": 0, 2024-09-19T08:14:08.095 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-19T08:14:08.095 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_items": 0, 2024-09-19T08:14:08.095 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_bytes": 0, 2024-09-19T08:14:08.095 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_items": 0, 2024-09-19T08:14:08.095 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_bytes": 0, 2024-09-19T08:14:08.095 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_items": 0, 2024-09-19T08:14:08.095 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-19T08:14:08.095 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_items": 0, 2024-09-19T08:14:08.095 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_bytes": 0, 2024-09-19T08:14:08.096 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_items": 0, 2024-09-19T08:14:08.096 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_bytes": 0, 2024-09-19T08:14:08.096 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_items": 0, 2024-09-19T08:14:08.096 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_bytes": 0, 2024-09-19T08:14:08.096 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_items": 0, 2024-09-19T08:14:08.096 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_bytes": 0, 2024-09-19T08:14:08.096 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_items": 0, 2024-09-19T08:14:08.096 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_bytes": 189383, 2024-09-19T08:14:08.096 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_items": 50, 2024-09-19T08:14:08.096 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_bytes": 0, 2024-09-19T08:14:08.096 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_items": 0, 2024-09-19T08:14:08.096 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_bytes": 0, 2024-09-19T08:14:08.097 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_items": 0, 2024-09-19T08:14:08.097 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_bytes": 0, 2024-09-19T08:14:08.097 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_items": 0, 2024-09-19T08:14:08.097 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_bytes": 0, 2024-09-19T08:14:08.097 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_items": 0, 2024-09-19T08:14:08.097 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_bytes": 8136, 2024-09-19T08:14:08.097 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_items": 180, 2024-09-19T08:14:08.097 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_bytes": 0, 2024-09-19T08:14:08.097 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_items": 0, 2024-09-19T08:14:08.097 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_bytes": 0, 2024-09-19T08:14:08.097 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, 2024-09-19T08:14:08.097 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 22336, 2024-09-19T08:14:08.097 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 66, 2024-09-19T08:14:08.098 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_bytes": 0, 2024-09-19T08:14:08.098 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_items": 0, 2024-09-19T08:14:08.098 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_bytes": 0, 2024-09-19T08:14:08.098 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_items": 0 2024-09-19T08:14:08.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.098 INFO:teuthology.orchestra.run.smithi139.stdout: "objecter": { 2024-09-19T08:14:08.098 INFO:teuthology.orchestra.run.smithi139.stdout: "op_active": 0, 2024-09-19T08:14:08.098 INFO:teuthology.orchestra.run.smithi139.stdout: "op_laggy": 0, 2024-09-19T08:14:08.098 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send": 2221, 2024-09-19T08:14:08.098 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send_bytes": 21, 2024-09-19T08:14:08.098 INFO:teuthology.orchestra.run.smithi139.stdout: "op_resend": 3, 2024-09-19T08:14:08.098 INFO:teuthology.orchestra.run.smithi139.stdout: "op_reply": 2221, 2024-09-19T08:14:08.098 INFO:teuthology.orchestra.run.smithi139.stdout: "op_latency": { 2024-09-19T08:14:08.098 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2221, 2024-09-19T08:14:08.099 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 6.241865753, 2024-09-19T08:14:08.099 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.002810385 2024-09-19T08:14:08.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.099 INFO:teuthology.orchestra.run.smithi139.stdout: "op_inflight": 0, 2024-09-19T08:14:08.099 INFO:teuthology.orchestra.run.smithi139.stdout: "oplen_avg": { 2024-09-19T08:14:08.099 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2221, 2024-09-19T08:14:08.099 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 2222 2024-09-19T08:14:08.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.099 INFO:teuthology.orchestra.run.smithi139.stdout: "op": 2221, 2024-09-19T08:14:08.099 INFO:teuthology.orchestra.run.smithi139.stdout: "op_r": 2221, 2024-09-19T08:14:08.099 INFO:teuthology.orchestra.run.smithi139.stdout: "op_w": 0, 2024-09-19T08:14:08.099 INFO:teuthology.orchestra.run.smithi139.stdout: "op_rmw": 0, 2024-09-19T08:14:08.099 INFO:teuthology.orchestra.run.smithi139.stdout: "op_pg": 0, 2024-09-19T08:14:08.100 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_stat": 1421, 2024-09-19T08:14:08.100 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_create": 0, 2024-09-19T08:14:08.100 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_read": 798, 2024-09-19T08:14:08.100 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_write": 0, 2024-09-19T08:14:08.100 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writefull": 0, 2024-09-19T08:14:08.100 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writesame": 0, 2024-09-19T08:14:08.100 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_append": 0, 2024-09-19T08:14:08.100 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_zero": 0, 2024-09-19T08:14:08.100 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_truncate": 0, 2024-09-19T08:14:08.100 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_delete": 0, 2024-09-19T08:14:08.100 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_mapext": 0, 2024-09-19T08:14:08.100 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_sparse_read": 0, 2024-09-19T08:14:08.100 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_clonerange": 0, 2024-09-19T08:14:08.100 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_getxattr": 1, 2024-09-19T08:14:08.101 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_setxattr": 0, 2024-09-19T08:14:08.101 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_cmpxattr": 0, 2024-09-19T08:14:08.101 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_rmxattr": 0, 2024-09-19T08:14:08.101 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_resetxattrs": 0, 2024-09-19T08:14:08.101 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_call": 0, 2024-09-19T08:14:08.101 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_watch": 0, 2024-09-19T08:14:08.101 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_notify": 0, 2024-09-19T08:14:08.101 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_src_cmpxattr": 0, 2024-09-19T08:14:08.101 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls": 0, 2024-09-19T08:14:08.101 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls_filter": 0, 2024-09-19T08:14:08.101 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_other": 0, 2024-09-19T08:14:08.101 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_active": 0, 2024-09-19T08:14:08.101 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_send": 0, 2024-09-19T08:14:08.102 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_resend": 0, 2024-09-19T08:14:08.102 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_ping": 0, 2024-09-19T08:14:08.102 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_active": 0, 2024-09-19T08:14:08.102 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_send": 0, 2024-09-19T08:14:08.102 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_resend": 0, 2024-09-19T08:14:08.102 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_active": 0, 2024-09-19T08:14:08.102 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_send": 0, 2024-09-19T08:14:08.102 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_resend": 0, 2024-09-19T08:14:08.102 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_active": 0, 2024-09-19T08:14:08.102 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_send": 0, 2024-09-19T08:14:08.102 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_resend": 0, 2024-09-19T08:14:08.102 INFO:teuthology.orchestra.run.smithi139.stdout: "command_active": 0, 2024-09-19T08:14:08.102 INFO:teuthology.orchestra.run.smithi139.stdout: "command_send": 0, 2024-09-19T08:14:08.102 INFO:teuthology.orchestra.run.smithi139.stdout: "command_resend": 0, 2024-09-19T08:14:08.103 INFO:teuthology.orchestra.run.smithi139.stdout: "map_epoch": 89, 2024-09-19T08:14:08.103 INFO:teuthology.orchestra.run.smithi139.stdout: "map_full": 0, 2024-09-19T08:14:08.103 INFO:teuthology.orchestra.run.smithi139.stdout: "map_inc": 11, 2024-09-19T08:14:08.103 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_sessions": 5, 2024-09-19T08:14:08.103 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_open": 5, 2024-09-19T08:14:08.103 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_close": 0, 2024-09-19T08:14:08.103 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_laggy": 0, 2024-09-19T08:14:08.103 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_wr": 0, 2024-09-19T08:14:08.103 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_rd": 2, 2024-09-19T08:14:08.103 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_del": 0 2024-09-19T08:14:08.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.103 INFO:teuthology.orchestra.run.smithi139.stdout: "oft": { 2024-09-19T08:14:08.103 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_objs": 0, 2024-09-19T08:14:08.104 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_kv_pairs": 0, 2024-09-19T08:14:08.104 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_updates": 0, 2024-09-19T08:14:08.104 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_removes": 0 2024-09-19T08:14:08.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.104 INFO:teuthology.orchestra.run.smithi139.stdout: "osd-slow-ops": { 2024-09-19T08:14:08.104 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_ops_count": 0 2024-09-19T08:14:08.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.104 INFO:teuthology.orchestra.run.smithi139.stdout: "purge_queue": { 2024-09-19T08:14:08.104 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops": 0, 2024-09-19T08:14:08.104 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops_high_water": 0, 2024-09-19T08:14:08.104 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing": 0, 2024-09-19T08:14:08.104 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_high_water": 0, 2024-09-19T08:14:08.104 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed_ops": 0, 2024-09-19T08:14:08.104 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed": 0, 2024-09-19T08:14:08.105 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_item_in_journal": 0 2024-09-19T08:14:08.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.105 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-19T08:14:08.105 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:08.105 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-09-19T08:14:08.105 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:08.105 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 2476, 2024-09-19T08:14:08.105 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 5106457, 2024-09-19T08:14:08.105 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:08.105 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 2476, 2024-09-19T08:14:08.105 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-09-19T08:14:08.105 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-09-19T08:14:08.105 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 2476, 2024-09-19T08:14:08.105 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 5106457, 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_bytes": { 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:08.106 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 2221, 2024-09-19T08:14:08.107 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 3321001, 2024-09-19T08:14:08.107 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 2221, 2024-09-19T08:14:08.107 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 3321001, 2024-09-19T08:14:08.107 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:08.107 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.107 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.107 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.107 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:08.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.107 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_ops": { 2024-09-19T08:14:08.107 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:08.107 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 1024, 2024-09-19T08:14:08.107 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:08.108 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-09-19T08:14:08.108 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-09-19T08:14:08.108 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:08.108 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:08.108 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 2221, 2024-09-19T08:14:08.108 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 2221, 2024-09-19T08:14:08.108 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 2221, 2024-09-19T08:14:08.108 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 2221, 2024-09-19T08:14:08.108 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:08.108 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.108 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.108 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.108 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:08.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.109 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle": { 2024-09-19T08:14:08.109 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:08.109 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-09-19T08:14:08.109 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:08.109 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-09-19T08:14:08.109 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-09-19T08:14:08.109 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:08.109 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:08.109 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-09-19T08:14:08.109 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-09-19T08:14:08.109 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 0, 2024-09-19T08:14:08.109 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 0, 2024-09-19T08:14:08.109 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:08.110 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.110 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.110 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.110 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:08.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-19T08:14:08.110 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle-0x55723b1eeb40": { 2024-09-19T08:14:08.110 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-09-19T08:14:08.110 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-09-19T08:14:08.110 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-09-19T08:14:08.110 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-09-19T08:14:08.110 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-09-19T08:14:08.110 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:08.111 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:08.111 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-09-19T08:14:08.111 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-09-19T08:14:08.111 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 0, 2024-09-19T08:14:08.111 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 0, 2024-09-19T08:14:08.111 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-09-19T08:14:08.111 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-09-19T08:14:08.111 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-09-19T08:14:08.111 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-09-19T08:14:08.111 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:08.111 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-19T08:14:08.111 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-19T08:14:08.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:07 smithi162 ceph-mon[35260]: pgmap v344: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-09-19T08:14:08.413 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:07 smithi139 ceph-mon[34874]: pgmap v344: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-09-19T08:14:08.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:07 smithi098 ceph-mon[31333]: pgmap v344: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-09-19T08:14:08.892 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-09-19T08:14:08.893 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-09-19T08:14:08.896 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-09-19T08:14:08.896 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-09-19T08:14:08.896 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-09-19T08:14:08.896 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-09-19T08:14:08.896 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-09-19T08:14:08.896 DEBUG:tasks.check_counter:Getting stats from c 2024-09-19T08:14:08.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph daemon mds.c perf dump 2024-09-19T08:14:09.160 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T08:14:09.840 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-19T08:14:09.840 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-09-19T08:14:09.840 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 2223, 2024-09-19T08:14:09.840 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 2223, 2024-09-19T08:14:09.840 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 634402, 2024-09-19T08:14:09.840 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 706097, 2024-09-19T08:14:09.840 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 6, 2024-09-19T08:14:09.840 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 2, 2024-09-19T08:14:09.840 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.202048552, 2024-09-19T08:14:09.840 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.065598926, 2024-09-19T08:14:09.840 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.053460337, 2024-09-19T08:14:09.841 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.067546517, 2024-09-19T08:14:09.841 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:09.841 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2222, 2024-09-19T08:14:09.841 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.091890578, 2024-09-19T08:14:09.841 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000041354 2024-09-19T08:14:09.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.841 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:09.841 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.841 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.841 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.841 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-09-19T08:14:09.842 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 448 2024-09-19T08:14:09.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.842 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-09-19T08:14:09.842 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 252, 2024-09-19T08:14:09.842 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 233, 2024-09-19T08:14:09.842 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 1392908, 2024-09-19T08:14:09.842 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 135850, 2024-09-19T08:14:09.842 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 3, 2024-09-19T08:14:09.842 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 2, 2024-09-19T08:14:09.842 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.042862569, 2024-09-19T08:14:09.842 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.016120083, 2024-09-19T08:14:09.842 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.039640363, 2024-09-19T08:14:09.842 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.000066640, 2024-09-19T08:14:09.843 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:09.843 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 232, 2024-09-19T08:14:09.843 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.015219470, 2024-09-19T08:14:09.843 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000065601 2024-09-19T08:14:09.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.843 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:09.843 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.843 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.843 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.843 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 1392352, 2024-09-19T08:14:09.843 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 135152 2024-09-19T08:14:09.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.844 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-09-19T08:14:09.844 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 3, 2024-09-19T08:14:09.844 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 166, 2024-09-19T08:14:09.844 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 588, 2024-09-19T08:14:09.844 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 452526, 2024-09-19T08:14:09.845 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 6, 2024-09-19T08:14:09.845 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 4, 2024-09-19T08:14:09.845 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.011050256, 2024-09-19T08:14:09.845 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.006389632, 2024-09-19T08:14:09.845 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.002561385, 2024-09-19T08:14:09.845 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.000045600, 2024-09-19T08:14:09.845 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:09.845 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 163, 2024-09-19T08:14:09.845 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.015337383, 2024-09-19T08:14:09.845 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000094094 2024-09-19T08:14:09.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.845 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:09.845 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.846 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.846 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.846 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-19T08:14:09.846 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 452208 2024-09-19T08:14:09.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.846 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-09-19T08:14:09.846 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-09-19T08:14:09.847 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-09-19T08:14:09.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.847 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-09-19T08:14:09.847 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-09-19T08:14:09.847 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-09-19T08:14:09.847 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3763, 2024-09-19T08:14:09.847 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.108406505, 2024-09-19T08:14:09.847 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000028808 2024-09-19T08:14:09.847 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:09.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.847 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-09-19T08:14:09.847 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-09-19T08:14:09.847 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-09-19T08:14:09.847 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.848 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.848 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.848 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:09.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.848 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-09-19T08:14:09.848 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 0, 2024-09-19T08:14:09.848 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 0, 2024-09-19T08:14:09.848 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-09-19T08:14:09.848 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.849 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.849 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.849 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-09-19T08:14:09.849 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 0, 2024-09-19T08:14:09.849 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 0, 2024-09-19T08:14:09.849 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-09-19T08:14:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 0, 2024-09-19T08:14:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 0, 2024-09-19T08:14:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-09-19T08:14:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 0, 2024-09-19T08:14:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 0, 2024-09-19T08:14:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 0, 2024-09-19T08:14:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 0, 2024-09-19T08:14:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 0, 2024-09-19T08:14:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 0, 2024-09-19T08:14:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 0, 2024-09-19T08:14:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 0, 2024-09-19T08:14:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 1, 2024-09-19T08:14:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 0, 2024-09-19T08:14:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 0, 2024-09-19T08:14:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 0, 2024-09-19T08:14:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 0, 2024-09-19T08:14:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 0, 2024-09-19T08:14:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-09-19T08:14:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 0, 2024-09-19T08:14:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 0, 2024-09-19T08:14:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-09-19T08:14:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 0, 2024-09-19T08:14:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 0, 2024-09-19T08:14:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 0, 2024-09-19T08:14:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 0, 2024-09-19T08:14:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-09-19T08:14:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 0, 2024-09-19T08:14:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-09-19T08:14:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 0, 2024-09-19T08:14:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 0, 2024-09-19T08:14:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 0, 2024-09-19T08:14:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 0, 2024-09-19T08:14:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-09-19T08:14:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-09-19T08:14:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-09-19T08:14:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-09-19T08:14:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-09-19T08:14:09.853 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-09-19T08:14:09.853 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 0, 2024-09-19T08:14:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 0, 2024-09-19T08:14:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 0, 2024-09-19T08:14:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 0, 2024-09-19T08:14:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-09-19T08:14:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-19T08:14:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-19T08:14:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 0, 2024-09-19T08:14:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 0, 2024-09-19T08:14:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 0, 2024-09-19T08:14:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 0 2024-09-19T08:14:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-09-19T08:14:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 0, 2024-09-19T08:14:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 0, 2024-09-19T08:14:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 0, 2024-09-19T08:14:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 0, 2024-09-19T08:14:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 0, 2024-09-19T08:14:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 0, 2024-09-19T08:14:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-09-19T08:14:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-09-19T08:14:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 0, 2024-09-19T08:14:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 0, 2024-09-19T08:14:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-09-19T08:14:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-09-19T08:14:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-09-19T08:14:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-09-19T08:14:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-09-19T08:14:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-09-19T08:14:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-09-19T08:14:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 0, 2024-09-19T08:14:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 0, 2024-09-19T08:14:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-09-19T08:14:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 0, 2024-09-19T08:14:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-09-19T08:14:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 0, 2024-09-19T08:14:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-09-19T08:14:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-09-19T08:14:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-09-19T08:14:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 0, 2024-09-19T08:14:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 0, 2024-09-19T08:14:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 0, 2024-09-19T08:14:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 0, 2024-09-19T08:14:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 12, 2024-09-19T08:14:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-09-19T08:14:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 0, 2024-09-19T08:14:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 0, 2024-09-19T08:14:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 0, 2024-09-19T08:14:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 0, 2024-09-19T08:14:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 2, 2024-09-19T08:14:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 2, 2024-09-19T08:14:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-09-19T08:14:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 0, 2024-09-19T08:14:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 4194304, 2024-09-19T08:14:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 4197729, 2024-09-19T08:14:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 4197588, 2024-09-19T08:14:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-09-19T08:14:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 12 2024-09-19T08:14:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-09-19T08:14:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 3, 2024-09-19T08:14:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 3, 2024-09-19T08:14:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 0, 2024-09-19T08:14:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 1, 2024-09-19T08:14:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 1, 2024-09-19T08:14:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 0, 2024-09-19T08:14:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 0, 2024-09-19T08:14:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 0, 2024-09-19T08:14:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 0, 2024-09-19T08:14:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 0, 2024-09-19T08:14:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 0, 2024-09-19T08:14:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 0, 2024-09-19T08:14:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 48492, 2024-09-19T08:14:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 215328 2024-09-19T08:14:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-09-19T08:14:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 0, 2024-09-19T08:14:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 0, 2024-09-19T08:14:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 0, 2024-09-19T08:14:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 0, 2024-09-19T08:14:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 0, 2024-09-19T08:14:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-09-19T08:14:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-09-19T08:14:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-09-19T08:14:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-09-19T08:14:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-09-19T08:14:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-09-19T08:14:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-09-19T08:14:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-09-19T08:14:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-09-19T08:14:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-09-19T08:14:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-09-19T08:14:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-09-19T08:14:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-09-19T08:14:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-09-19T08:14:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-09-19T08:14:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-09-19T08:14:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-09-19T08:14:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-09-19T08:14:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-09-19T08:14:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-09-19T08:14:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-09-19T08:14:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-09-19T08:14:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-09-19T08:14:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-09-19T08:14:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-09-19T08:14:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-09-19T08:14:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-09-19T08:14:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-09-19T08:14:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-09-19T08:14:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-09-19T08:14:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-09-19T08:14:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-09-19T08:14:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-09-19T08:14:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 2, 2024-09-19T08:14:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 6, 2024-09-19T08:14:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 4, 2024-09-19T08:14:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 5, 2024-09-19T08:14:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-09-19T08:14:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 0, 2024-09-19T08:14:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 0, 2024-09-19T08:14:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 3875, 2024-09-19T08:14:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-09-19T08:14:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-09-19T08:14:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-09-19T08:14:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-09-19T08:14:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-09-19T08:14:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-09-19T08:14:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-09-19T08:14:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-09-19T08:14:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-19T08:14:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-09-19T08:14:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-19T08:14:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-09-19T08:14:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-09-19T08:14:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-09-19T08:14:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-19T08:14:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-09-19T08:14:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-09-19T08:14:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-09-19T08:14:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-09-19T08:14:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-09-19T08:14:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-19T08:14:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-09-19T08:14:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-19T08:14:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-09-19T08:14:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-09-19T08:14:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-09-19T08:14:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-09-19T08:14:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-09-19T08:14:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-19T08:14:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-09-19T08:14:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-09-19T08:14:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-09-19T08:14:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-09-19T08:14:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-09-19T08:14:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-09-19T08:14:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-09-19T08:14:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-09-19T08:14:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-09-19T08:14:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 181343, 2024-09-19T08:14:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 47, 2024-09-19T08:14:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-09-19T08:14:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-09-19T08:14:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-09-19T08:14:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-09-19T08:14:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-09-19T08:14:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-09-19T08:14:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-09-19T08:14:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-09-19T08:14:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 8136, 2024-09-19T08:14:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 180, 2024-09-19T08:14:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-09-19T08:14:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-09-19T08:14:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-09-19T08:14:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-09-19T08:14:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 8512, 2024-09-19T08:14:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 13, 2024-09-19T08:14:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-09-19T08:14:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-09-19T08:14:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-09-19T08:14:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-09-19T08:14:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-09-19T08:14:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-09-19T08:14:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-09-19T08:14:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 2222, 2024-09-19T08:14:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 21, 2024-09-19T08:14:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 3, 2024-09-19T08:14:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 2222, 2024-09-19T08:14:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-09-19T08:14:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2222, 2024-09-19T08:14:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 1.488968697, 2024-09-19T08:14:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000670102 2024-09-19T08:14:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-09-19T08:14:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-09-19T08:14:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2222, 2024-09-19T08:14:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 2223 2024-09-19T08:14:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 2222, 2024-09-19T08:14:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 2222, 2024-09-19T08:14:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 0, 2024-09-19T08:14:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-09-19T08:14:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-09-19T08:14:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 1441, 2024-09-19T08:14:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 0, 2024-09-19T08:14:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 779, 2024-09-19T08:14:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 0, 2024-09-19T08:14:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 0, 2024-09-19T08:14:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-09-19T08:14:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-09-19T08:14:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-09-19T08:14:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 0, 2024-09-19T08:14:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 0, 2024-09-19T08:14:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-09-19T08:14:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-09-19T08:14:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-09-19T08:14:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 1, 2024-09-19T08:14:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 0, 2024-09-19T08:14:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-09-19T08:14:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-09-19T08:14:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-09-19T08:14:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-09-19T08:14:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-09-19T08:14:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-09-19T08:14:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-09-19T08:14:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-09-19T08:14:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-09-19T08:14:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-09-19T08:14:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-09-19T08:14:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-09-19T08:14:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-09-19T08:14:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-09-19T08:14:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-09-19T08:14:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-09-19T08:14:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-09-19T08:14:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-09-19T08:14:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-09-19T08:14:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-09-19T08:14:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-09-19T08:14:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-09-19T08:14:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-09-19T08:14:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-09-19T08:14:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-09-19T08:14:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-09-19T08:14:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 89, 2024-09-19T08:14:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-09-19T08:14:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 11, 2024-09-19T08:14:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 4, 2024-09-19T08:14:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 4, 2024-09-19T08:14:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 0, 2024-09-19T08:14:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-09-19T08:14:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 0, 2024-09-19T08:14:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 2, 2024-09-19T08:14:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 0 2024-09-19T08:14:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-09-19T08:14:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 0, 2024-09-19T08:14:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-09-19T08:14:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-09-19T08:14:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-09-19T08:14:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-09-19T08:14:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 0 2024-09-19T08:14:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-09-19T08:14:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-09-19T08:14:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 0, 2024-09-19T08:14:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-09-19T08:14:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 0, 2024-09-19T08:14:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 0, 2024-09-19T08:14:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 0, 2024-09-19T08:14:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-09-19T08:14:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-19T08:14:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-09-19T08:14:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 2474, 2024-09-19T08:14:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 1789394, 2024-09-19T08:14:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 2474, 2024-09-19T08:14:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-09-19T08:14:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-09-19T08:14:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 2474, 2024-09-19T08:14:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 1789394, 2024-09-19T08:14:09.889 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:09.889 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.889 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.889 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.889 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:09.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.889 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-09-19T08:14:09.889 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-09-19T08:14:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-09-19T08:14:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-09-19T08:14:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 2222, 2024-09-19T08:14:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 3430, 2024-09-19T08:14:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 2222, 2024-09-19T08:14:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 3430, 2024-09-19T08:14:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-09-19T08:14:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-09-19T08:14:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-09-19T08:14:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-09-19T08:14:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 2222, 2024-09-19T08:14:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 2222, 2024-09-19T08:14:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 2222, 2024-09-19T08:14:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 2222, 2024-09-19T08:14:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-09-19T08:14:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-09-19T08:14:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-09-19T08:14:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-09-19T08:14:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-09-19T08:14:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-09-19T08:14:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-09-19T08:14:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-09-19T08:14:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x55e117c20b40": { 2024-09-19T08:14:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-09-19T08:14:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-09-19T08:14:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-09-19T08:14:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-09-19T08:14:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-09-19T08:14:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-09-19T08:14:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-09-19T08:14:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:09.895 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-19T08:14:10.272 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:09 smithi162 ceph-mon[35260]: pgmap v345: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:14:10.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:09 smithi139 ceph-mon[34874]: pgmap v345: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:14:10.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:09 smithi098 ceph-mon[31333]: pgmap v345: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-09-19T08:14:10.753 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-09-19T08:14:10.753 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-09-19T08:14:10.753 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-09-19T08:14:10.753 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-09-19T08:14:10.753 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-09-19T08:14:10.753 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-09-19T08:14:10.753 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-09-19T08:14:10.753 DEBUG:tasks.check_counter:Getting stats from f 2024-09-19T08:14:10.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph daemon mds.f perf dump 2024-09-19T08:14:11.006 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T08:14:11.527 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-19T08:14:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-09-19T08:14:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 801, 2024-09-19T08:14:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 853, 2024-09-19T08:14:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 1578328, 2024-09-19T08:14:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 281922, 2024-09-19T08:14:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 7, 2024-09-19T08:14:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 2, 2024-09-19T08:14:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.106550968, 2024-09-19T08:14:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.040254241, 2024-09-19T08:14:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.072085482, 2024-09-19T08:14:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.004809951, 2024-09-19T08:14:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 854, 2024-09-19T08:14:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.056051203, 2024-09-19T08:14:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000065633 2024-09-19T08:14:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1067, 2024-09-19T08:14:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000287147, 2024-09-19T08:14:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000269 2024-09-19T08:14:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 1394544, 2024-09-19T08:14:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 141568 2024-09-19T08:14:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-09-19T08:14:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 42, 2024-09-19T08:14:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 206, 2024-09-19T08:14:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 7651, 2024-09-19T08:14:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 799882, 2024-09-19T08:14:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 36, 2024-09-19T08:14:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-09-19T08:14:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.022048396, 2024-09-19T08:14:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.007534979, 2024-09-19T08:14:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.011600302, 2024-09-19T08:14:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.000291546, 2024-09-19T08:14:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 203, 2024-09-19T08:14:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.023703528, 2024-09-19T08:14:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000116766 2024-09-19T08:14:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 28, 2024-09-19T08:14:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000002174, 2024-09-19T08:14:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000077 2024-09-19T08:14:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-19T08:14:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 452384 2024-09-19T08:14:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-09-19T08:14:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 33, 2024-09-19T08:14:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 35, 2024-09-19T08:14:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 16398, 2024-09-19T08:14:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 118197, 2024-09-19T08:14:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 16, 2024-09-19T08:14:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-09-19T08:14:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.009101522, 2024-09-19T08:14:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.000734444, 2024-09-19T08:14:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.007914791, 2024-09-19T08:14:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.000259774, 2024-09-19T08:14:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 32, 2024-09-19T08:14:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.019153464, 2024-09-19T08:14:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000598545 2024-09-19T08:14:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 24, 2024-09-19T08:14:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000003158, 2024-09-19T08:14:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000131 2024-09-19T08:14:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 4656, 2024-09-19T08:14:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 1888 2024-09-19T08:14:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-09-19T08:14:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-09-19T08:14:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-09-19T08:14:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-09-19T08:14:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-09-19T08:14:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-09-19T08:14:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 43, 2024-09-19T08:14:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.002281803, 2024-09-19T08:14:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000053065 2024-09-19T08:14:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-09-19T08:14:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-09-19T08:14:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-09-19T08:14:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2, 2024-09-19T08:14:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000012607, 2024-09-19T08:14:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000006303 2024-09-19T08:14:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-09-19T08:14:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 2, 2024-09-19T08:14:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 1, 2024-09-19T08:14:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-09-19T08:14:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1, 2024-09-19T08:14:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000176309, 2024-09-19T08:14:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000176309 2024-09-19T08:14:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-09-19T08:14:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 1, 2024-09-19T08:14:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 0, 2024-09-19T08:14:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-09-19T08:14:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 11, 2024-09-19T08:14:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 0, 2024-09-19T08:14:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-09-19T08:14:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 16, 2024-09-19T08:14:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 0, 2024-09-19T08:14:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 16, 2024-09-19T08:14:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 0, 2024-09-19T08:14:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 13, 2024-09-19T08:14:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 4, 2024-09-19T08:14:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 0, 2024-09-19T08:14:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 0, 2024-09-19T08:14:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 3, 2024-09-19T08:14:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 18, 2024-09-19T08:14:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 12, 2024-09-19T08:14:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 0, 2024-09-19T08:14:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 5, 2024-09-19T08:14:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 0, 2024-09-19T08:14:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-09-19T08:14:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 0, 2024-09-19T08:14:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 0, 2024-09-19T08:14:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-09-19T08:14:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 0, 2024-09-19T08:14:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 0, 2024-09-19T08:14:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 0, 2024-09-19T08:14:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 0, 2024-09-19T08:14:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-09-19T08:14:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 0, 2024-09-19T08:14:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-09-19T08:14:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 4, 2024-09-19T08:14:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 234, 2024-09-19T08:14:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 6, 2024-09-19T08:14:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 0, 2024-09-19T08:14:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-09-19T08:14:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-09-19T08:14:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-09-19T08:14:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-09-19T08:14:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-09-19T08:14:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-09-19T08:14:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 0, 2024-09-19T08:14:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 0, 2024-09-19T08:14:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 0, 2024-09-19T08:14:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 0, 2024-09-19T08:14:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-09-19T08:14:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-19T08:14:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-19T08:14:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 0, 2024-09-19T08:14:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 0, 2024-09-19T08:14:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 0, 2024-09-19T08:14:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 1 2024-09-19T08:14:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-09-19T08:14:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 0, 2024-09-19T08:14:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 12, 2024-09-19T08:14:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 8, 2024-09-19T08:14:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 11, 2024-09-19T08:14:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 0, 2024-09-19T08:14:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 0, 2024-09-19T08:14:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-09-19T08:14:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-09-19T08:14:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 0, 2024-09-19T08:14:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 0, 2024-09-19T08:14:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-09-19T08:14:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-09-19T08:14:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-09-19T08:14:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-09-19T08:14:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-09-19T08:14:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-09-19T08:14:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-09-19T08:14:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 3, 2024-09-19T08:14:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 17, 2024-09-19T08:14:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-09-19T08:14:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 0, 2024-09-19T08:14:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-09-19T08:14:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 0, 2024-09-19T08:14:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-09-19T08:14:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-09-19T08:14:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-09-19T08:14:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 0, 2024-09-19T08:14:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 4, 2024-09-19T08:14:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 0, 2024-09-19T08:14:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 0, 2024-09-19T08:14:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 4, 2024-09-19T08:14:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-09-19T08:14:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 0, 2024-09-19T08:14:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 2, 2024-09-19T08:14:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 0, 2024-09-19T08:14:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 0, 2024-09-19T08:14:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 2, 2024-09-19T08:14:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 2, 2024-09-19T08:14:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-09-19T08:14:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 0, 2024-09-19T08:14:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 4194304, 2024-09-19T08:14:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 4195426, 2024-09-19T08:14:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 0, 2024-09-19T08:14:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-09-19T08:14:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3, 2024-09-19T08:14:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.005239399, 2024-09-19T08:14:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.001746466 2024-09-19T08:14:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 0 2024-09-19T08:14:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-09-19T08:14:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 20, 2024-09-19T08:14:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 22, 2024-09-19T08:14:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 2, 2024-09-19T08:14:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 19, 2024-09-19T08:14:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 22, 2024-09-19T08:14:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 3, 2024-09-19T08:14:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 16, 2024-09-19T08:14:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 20, 2024-09-19T08:14:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 4, 2024-09-19T08:14:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 0, 2024-09-19T08:14:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 1, 2024-09-19T08:14:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 1, 2024-09-19T08:14:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 51300, 2024-09-19T08:14:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 223520 2024-09-19T08:14:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-09-19T08:14:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 3, 2024-09-19T08:14:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 0, 2024-09-19T08:14:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 2, 2024-09-19T08:14:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 4, 2024-09-19T08:14:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 0, 2024-09-19T08:14:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-09-19T08:14:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-09-19T08:14:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1, 2024-09-19T08:14:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000176309, 2024-09-19T08:14:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000176309 2024-09-19T08:14:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-09-19T08:14:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-09-19T08:14:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-09-19T08:14:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-09-19T08:14:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-09-19T08:14:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-09-19T08:14:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-09-19T08:14:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-09-19T08:14:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-09-19T08:14:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-09-19T08:14:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-09-19T08:14:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-09-19T08:14:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-09-19T08:14:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-09-19T08:14:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-09-19T08:14:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-09-19T08:14:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-09-19T08:14:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-09-19T08:14:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-09-19T08:14:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-09-19T08:14:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-09-19T08:14:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-09-19T08:14:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-09-19T08:14:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-09-19T08:14:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-09-19T08:14:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-09-19T08:14:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-09-19T08:14:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-09-19T08:14:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-09-19T08:14:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-09-19T08:14:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-09-19T08:14:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 0, 2024-09-19T08:14:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 1, 2024-09-19T08:14:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 1, 2024-09-19T08:14:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 0, 2024-09-19T08:14:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-09-19T08:14:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 1, 2024-09-19T08:14:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 1, 2024-09-19T08:14:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 89, 2024-09-19T08:14:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-09-19T08:14:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-09-19T08:14:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-09-19T08:14:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-09-19T08:14:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-09-19T08:14:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-09-19T08:14:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-09-19T08:14:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-09-19T08:14:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-19T08:14:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-09-19T08:14:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-19T08:14:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-09-19T08:14:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-09-19T08:14:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-09-19T08:14:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-19T08:14:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-09-19T08:14:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-09-19T08:14:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-09-19T08:14:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-09-19T08:14:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-09-19T08:14:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-19T08:14:11.561 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-09-19T08:14:11.561 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-19T08:14:11.564 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-09-19T08:14:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-09-19T08:14:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-09-19T08:14:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-09-19T08:14:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-09-19T08:14:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-19T08:14:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-09-19T08:14:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-09-19T08:14:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-09-19T08:14:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-09-19T08:14:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-09-19T08:14:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-09-19T08:14:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-09-19T08:14:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-09-19T08:14:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-09-19T08:14:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 229927, 2024-09-19T08:14:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 70, 2024-09-19T08:14:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-09-19T08:14:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-09-19T08:14:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-09-19T08:14:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-09-19T08:14:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-09-19T08:14:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-09-19T08:14:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-09-19T08:14:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-09-19T08:14:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 8136, 2024-09-19T08:14:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 180, 2024-09-19T08:14:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-09-19T08:14:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-09-19T08:14:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-09-19T08:14:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-09-19T08:14:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 81813, 2024-09-19T08:14:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 330, 2024-09-19T08:14:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-09-19T08:14:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-09-19T08:14:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-09-19T08:14:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-09-19T08:14:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-09-19T08:14:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-09-19T08:14:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-09-19T08:14:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 38, 2024-09-19T08:14:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 11334, 2024-09-19T08:14:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 33, 2024-09-19T08:14:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 38, 2024-09-19T08:14:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-09-19T08:14:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 38, 2024-09-19T08:14:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.199995845, 2024-09-19T08:14:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.005263048 2024-09-19T08:14:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-09-19T08:14:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-09-19T08:14:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 38, 2024-09-19T08:14:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 60 2024-09-19T08:14:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 38, 2024-09-19T08:14:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 0, 2024-09-19T08:14:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 38, 2024-09-19T08:14:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-09-19T08:14:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-09-19T08:14:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 0, 2024-09-19T08:14:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 10, 2024-09-19T08:14:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 0, 2024-09-19T08:14:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 3, 2024-09-19T08:14:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 6, 2024-09-19T08:14:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-09-19T08:14:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-09-19T08:14:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-09-19T08:14:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 0, 2024-09-19T08:14:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 6, 2024-09-19T08:14:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-09-19T08:14:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-09-19T08:14:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-09-19T08:14:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 0, 2024-09-19T08:14:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 20, 2024-09-19T08:14:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-09-19T08:14:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-09-19T08:14:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-09-19T08:14:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-09-19T08:14:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-09-19T08:14:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-09-19T08:14:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-09-19T08:14:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-09-19T08:14:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-09-19T08:14:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-09-19T08:14:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-09-19T08:14:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-09-19T08:14:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-09-19T08:14:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-09-19T08:14:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-09-19T08:14:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-09-19T08:14:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-09-19T08:14:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-09-19T08:14:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-09-19T08:14:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-09-19T08:14:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-09-19T08:14:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-09-19T08:14:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-09-19T08:14:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-09-19T08:14:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-09-19T08:14:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-09-19T08:14:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 90, 2024-09-19T08:14:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-09-19T08:14:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 12, 2024-09-19T08:14:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 8, 2024-09-19T08:14:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 8, 2024-09-19T08:14:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 0, 2024-09-19T08:14:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-09-19T08:14:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 14, 2024-09-19T08:14:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 0, 2024-09-19T08:14:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 1 2024-09-19T08:14:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-09-19T08:14:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 1, 2024-09-19T08:14:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-09-19T08:14:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-09-19T08:14:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-09-19T08:14:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-09-19T08:14:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 20 2024-09-19T08:14:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-09-19T08:14:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-09-19T08:14:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 0, 2024-09-19T08:14:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-09-19T08:14:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 0, 2024-09-19T08:14:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 0, 2024-09-19T08:14:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 0, 2024-09-19T08:14:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-09-19T08:14:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-19T08:14:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-09-19T08:14:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 872, 2024-09-19T08:14:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 1506260, 2024-09-19T08:14:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 872, 2024-09-19T08:14:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-09-19T08:14:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-09-19T08:14:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 872, 2024-09-19T08:14:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 1506260, 2024-09-19T08:14:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-09-19T08:14:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-09-19T08:14:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-09-19T08:14:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-09-19T08:14:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 38, 2024-09-19T08:14:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 11334, 2024-09-19T08:14:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 38, 2024-09-19T08:14:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 11334, 2024-09-19T08:14:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-09-19T08:14:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-09-19T08:14:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-09-19T08:14:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-09-19T08:14:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 38, 2024-09-19T08:14:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 38, 2024-09-19T08:14:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 38, 2024-09-19T08:14:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 38, 2024-09-19T08:14:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-09-19T08:14:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-09-19T08:14:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-09-19T08:14:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-09-19T08:14:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-09-19T08:14:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-09-19T08:14:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-09-19T08:14:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-09-19T08:14:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x559cc74fa960": { 2024-09-19T08:14:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-09-19T08:14:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 4, 2024-09-19T08:14:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 1122, 2024-09-19T08:14:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 4, 2024-09-19T08:14:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-09-19T08:14:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-09-19T08:14:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 3, 2024-09-19T08:14:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 1122, 2024-09-19T08:14:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:11.581 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-19T08:14:12.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:11 smithi162 ceph-mon[35260]: pgmap v346: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:14:12.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:11 smithi162 ceph-mon[35260]: osdmap e90: 12 total, 12 up, 12 in 2024-09-19T08:14:12.348 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-09-19T08:14:12.348 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-09-19T08:14:12.348 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-09-19T08:14:12.348 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=12 2024-09-19T08:14:12.348 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-09-19T08:14:12.348 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-09-19T08:14:12.348 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-09-19T08:14:12.348 DEBUG:tasks.check_counter:Getting stats from i 2024-09-19T08:14:12.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph daemon mds.i perf dump 2024-09-19T08:14:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:11 smithi139 ceph-mon[34874]: pgmap v346: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:14:12.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:11 smithi139 ceph-mon[34874]: osdmap e90: 12 total, 12 up, 12 in 2024-09-19T08:14:12.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:11 smithi098 ceph-mon[31333]: pgmap v346: 97 pgs: 97 active+clean; 7.3 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-09-19T08:14:12.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:11 smithi098 ceph-mon[31333]: osdmap e90: 12 total, 12 up, 12 in 2024-09-19T08:14:12.604 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T08:14:13.069 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-19T08:14:13.069 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-09-19T08:14:13.069 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 1624, 2024-09-19T08:14:13.069 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 1626, 2024-09-19T08:14:13.069 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 3533315, 2024-09-19T08:14:13.069 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 516442, 2024-09-19T08:14:13.069 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 6, 2024-09-19T08:14:13.069 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 2, 2024-09-19T08:14:13.069 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.158402792, 2024-09-19T08:14:13.070 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.050456037, 2024-09-19T08:14:13.070 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.044202377, 2024-09-19T08:14:13.070 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.048404917, 2024-09-19T08:14:13.070 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:13.070 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1624, 2024-09-19T08:14:13.070 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.065639554, 2024-09-19T08:14:13.070 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000040418 2024-09-19T08:14:13.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.070 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:13.070 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.071 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.071 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.071 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-19T08:14:13.071 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 0 2024-09-19T08:14:13.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.071 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-09-19T08:14:13.071 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 10, 2024-09-19T08:14:13.071 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 171, 2024-09-19T08:14:13.071 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 5048, 2024-09-19T08:14:13.071 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 455610, 2024-09-19T08:14:13.071 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 7, 2024-09-19T08:14:13.071 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-09-19T08:14:13.072 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.012867948, 2024-09-19T08:14:13.072 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.006549560, 2024-09-19T08:14:13.072 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.004408472, 2024-09-19T08:14:13.072 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.000159822, 2024-09-19T08:14:13.072 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:13.072 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 169, 2024-09-19T08:14:13.072 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.015434850, 2024-09-19T08:14:13.072 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000091330 2024-09-19T08:14:13.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.072 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:13.072 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.072 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.073 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.073 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-09-19T08:14:13.073 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 454272 2024-09-19T08:14:13.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.073 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-09-19T08:14:13.073 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 1016, 2024-09-19T08:14:13.073 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 997, 2024-09-19T08:14:13.073 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 1581481, 2024-09-19T08:14:13.073 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 378283, 2024-09-19T08:14:13.073 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 3, 2024-09-19T08:14:13.073 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 2, 2024-09-19T08:14:13.074 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.117643134, 2024-09-19T08:14:13.074 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.043774822, 2024-09-19T08:14:13.074 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.054339456, 2024-09-19T08:14:13.074 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.024257613, 2024-09-19T08:14:13.074 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:13.074 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 996, 2024-09-19T08:14:13.074 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.044387401, 2024-09-19T08:14:13.074 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000044565 2024-09-19T08:14:13.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.074 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:13.074 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.074 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.075 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.075 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 1393712, 2024-09-19T08:14:13.075 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 135648 2024-09-19T08:14:13.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.075 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-09-19T08:14:13.075 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-09-19T08:14:13.075 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-09-19T08:14:13.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.075 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-09-19T08:14:13.075 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-09-19T08:14:13.075 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-09-19T08:14:13.075 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3921, 2024-09-19T08:14:13.076 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.106293800, 2024-09-19T08:14:13.076 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000027108 2024-09-19T08:14:13.076 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:13.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.076 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-09-19T08:14:13.076 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-09-19T08:14:13.076 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-09-19T08:14:13.076 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.077 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.077 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:13.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.077 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-09-19T08:14:13.078 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 0, 2024-09-19T08:14:13.078 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 0, 2024-09-19T08:14:13.078 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-09-19T08:14:13.078 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.078 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.078 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.078 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-09-19T08:14:13.078 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 0, 2024-09-19T08:14:13.078 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 0, 2024-09-19T08:14:13.079 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-09-19T08:14:13.079 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 0, 2024-09-19T08:14:13.079 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 0, 2024-09-19T08:14:13.079 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-09-19T08:14:13.079 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 26, 2024-09-19T08:14:13.079 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 2, 2024-09-19T08:14:13.079 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 24, 2024-09-19T08:14:13.079 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 0, 2024-09-19T08:14:13.079 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 24, 2024-09-19T08:14:13.079 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 0, 2024-09-19T08:14:13.079 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 0, 2024-09-19T08:14:13.079 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 0, 2024-09-19T08:14:13.080 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 2, 2024-09-19T08:14:13.080 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 0, 2024-09-19T08:14:13.080 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 0, 2024-09-19T08:14:13.080 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 0, 2024-09-19T08:14:13.080 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 0, 2024-09-19T08:14:13.080 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 0, 2024-09-19T08:14:13.080 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-09-19T08:14:13.080 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 0, 2024-09-19T08:14:13.080 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 0, 2024-09-19T08:14:13.080 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-09-19T08:14:13.081 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 0, 2024-09-19T08:14:13.081 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 0, 2024-09-19T08:14:13.081 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 0, 2024-09-19T08:14:13.081 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 0, 2024-09-19T08:14:13.081 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-09-19T08:14:13.081 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 0, 2024-09-19T08:14:13.081 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-09-19T08:14:13.081 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 4, 2024-09-19T08:14:13.081 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 234, 2024-09-19T08:14:13.081 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 6, 2024-09-19T08:14:13.081 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 0, 2024-09-19T08:14:13.081 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-09-19T08:14:13.081 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-09-19T08:14:13.082 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-09-19T08:14:13.082 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-09-19T08:14:13.082 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-09-19T08:14:13.082 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-09-19T08:14:13.082 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 0, 2024-09-19T08:14:13.082 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 0, 2024-09-19T08:14:13.082 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 0, 2024-09-19T08:14:13.082 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 0, 2024-09-19T08:14:13.082 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-09-19T08:14:13.082 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-19T08:14:13.082 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-19T08:14:13.083 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 0, 2024-09-19T08:14:13.083 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 0, 2024-09-19T08:14:13.083 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 0, 2024-09-19T08:14:13.083 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 0 2024-09-19T08:14:13.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.083 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-09-19T08:14:13.083 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 0, 2024-09-19T08:14:13.083 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 0, 2024-09-19T08:14:13.083 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 0, 2024-09-19T08:14:13.083 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 0, 2024-09-19T08:14:13.084 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 0, 2024-09-19T08:14:13.084 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 0, 2024-09-19T08:14:13.084 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-09-19T08:14:13.084 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-09-19T08:14:13.084 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 0, 2024-09-19T08:14:13.084 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 0, 2024-09-19T08:14:13.084 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-09-19T08:14:13.084 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-09-19T08:14:13.084 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-09-19T08:14:13.084 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-09-19T08:14:13.084 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-09-19T08:14:13.085 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-09-19T08:14:13.085 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-09-19T08:14:13.085 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 0, 2024-09-19T08:14:13.085 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 0, 2024-09-19T08:14:13.085 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-09-19T08:14:13.085 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 0, 2024-09-19T08:14:13.085 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-09-19T08:14:13.085 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 0, 2024-09-19T08:14:13.085 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-09-19T08:14:13.085 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-09-19T08:14:13.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.085 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-09-19T08:14:13.085 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 0, 2024-09-19T08:14:13.086 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 0, 2024-09-19T08:14:13.086 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 0, 2024-09-19T08:14:13.086 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 0, 2024-09-19T08:14:13.086 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 1135, 2024-09-19T08:14:13.086 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-09-19T08:14:13.086 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 0, 2024-09-19T08:14:13.086 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 0, 2024-09-19T08:14:13.086 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 0, 2024-09-19T08:14:13.086 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 0, 2024-09-19T08:14:13.086 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 3, 2024-09-19T08:14:13.086 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 2, 2024-09-19T08:14:13.086 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-09-19T08:14:13.087 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 0, 2024-09-19T08:14:13.087 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 4194304, 2024-09-19T08:14:13.087 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 7245768, 2024-09-19T08:14:13.087 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 7245708, 2024-09-19T08:14:13.087 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-09-19T08:14:13.087 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.087 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.087 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.087 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 1135 2024-09-19T08:14:13.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.087 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-09-19T08:14:13.087 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 18, 2024-09-19T08:14:13.087 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 23, 2024-09-19T08:14:13.088 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 5, 2024-09-19T08:14:13.088 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 11, 2024-09-19T08:14:13.088 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 11, 2024-09-19T08:14:13.088 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 0, 2024-09-19T08:14:13.088 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 26, 2024-09-19T08:14:13.088 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 35, 2024-09-19T08:14:13.088 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 9, 2024-09-19T08:14:13.088 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 0, 2024-09-19T08:14:13.088 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 0, 2024-09-19T08:14:13.088 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 0, 2024-09-19T08:14:13.088 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 49228, 2024-09-19T08:14:13.088 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 215328 2024-09-19T08:14:13.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.089 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-09-19T08:14:13.089 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 0, 2024-09-19T08:14:13.089 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 0, 2024-09-19T08:14:13.089 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 0, 2024-09-19T08:14:13.089 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 0, 2024-09-19T08:14:13.089 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 0, 2024-09-19T08:14:13.089 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-09-19T08:14:13.089 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.089 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.089 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.089 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-09-19T08:14:13.089 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-09-19T08:14:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-09-19T08:14:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-09-19T08:14:13.091 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.091 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.091 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.091 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-09-19T08:14:13.091 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.091 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.091 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.091 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-09-19T08:14:13.091 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.091 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.091 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.092 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-09-19T08:14:13.092 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.092 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.092 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.092 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-09-19T08:14:13.092 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.092 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.092 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.092 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-09-19T08:14:13.092 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.093 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.093 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.093 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-09-19T08:14:13.093 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.093 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.093 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.094 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-09-19T08:14:13.094 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.095 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.095 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.095 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-09-19T08:14:13.096 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.096 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.096 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.096 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-09-19T08:14:13.096 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.097 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.097 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.097 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-09-19T08:14:13.097 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.097 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.097 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.097 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-09-19T08:14:13.097 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.098 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.098 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.098 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-09-19T08:14:13.098 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.099 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.099 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.099 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-09-19T08:14:13.099 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.099 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.099 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.099 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-09-19T08:14:13.099 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.099 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.100 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.100 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-09-19T08:14:13.100 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.100 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.100 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.100 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-09-19T08:14:13.100 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.100 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.100 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.101 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-09-19T08:14:13.101 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.101 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.101 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.101 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-09-19T08:14:13.101 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.101 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.101 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.101 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-09-19T08:14:13.101 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.102 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.102 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.102 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-09-19T08:14:13.102 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.102 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.102 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.102 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-09-19T08:14:13.102 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.102 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.102 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.102 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-09-19T08:14:13.103 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.103 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.103 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.103 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-09-19T08:14:13.103 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.103 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.103 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.103 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-09-19T08:14:13.104 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.104 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.104 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.104 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-09-19T08:14:13.104 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-09-19T08:14:13.104 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-09-19T08:14:13.104 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.104 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.104 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.104 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:13.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.104 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-09-19T08:14:13.105 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 2, 2024-09-19T08:14:13.105 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 6, 2024-09-19T08:14:13.105 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 4, 2024-09-19T08:14:13.105 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 5, 2024-09-19T08:14:13.105 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-09-19T08:14:13.105 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 0, 2024-09-19T08:14:13.105 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 0, 2024-09-19T08:14:13.105 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 3880, 2024-09-19T08:14:13.105 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-09-19T08:14:13.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.105 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-09-19T08:14:13.105 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-09-19T08:14:13.105 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-09-19T08:14:13.105 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-09-19T08:14:13.106 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-09-19T08:14:13.106 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-09-19T08:14:13.106 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-09-19T08:14:13.106 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-19T08:14:13.106 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-09-19T08:14:13.106 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-19T08:14:13.106 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-09-19T08:14:13.106 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-09-19T08:14:13.106 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-09-19T08:14:13.106 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-19T08:14:13.106 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-09-19T08:14:13.106 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-09-19T08:14:13.106 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-09-19T08:14:13.107 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-09-19T08:14:13.107 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-09-19T08:14:13.107 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-19T08:14:13.107 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-09-19T08:14:13.107 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-19T08:14:13.107 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-09-19T08:14:13.107 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-09-19T08:14:13.107 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-09-19T08:14:13.107 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-09-19T08:14:13.107 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-09-19T08:14:13.108 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-19T08:14:13.108 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-09-19T08:14:13.108 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-09-19T08:14:13.108 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-09-19T08:14:13.108 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-09-19T08:14:13.108 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-09-19T08:14:13.108 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-09-19T08:14:13.108 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-09-19T08:14:13.108 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-09-19T08:14:13.108 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-09-19T08:14:13.109 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 189726, 2024-09-19T08:14:13.109 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 60, 2024-09-19T08:14:13.109 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-09-19T08:14:13.109 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-09-19T08:14:13.109 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-09-19T08:14:13.109 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-09-19T08:14:13.109 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-09-19T08:14:13.109 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-09-19T08:14:13.109 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-09-19T08:14:13.109 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-09-19T08:14:13.110 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 8256, 2024-09-19T08:14:13.110 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 183, 2024-09-19T08:14:13.110 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-09-19T08:14:13.110 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-09-19T08:14:13.110 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-09-19T08:14:13.110 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-09-19T08:14:13.110 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 88645, 2024-09-19T08:14:13.111 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 634, 2024-09-19T08:14:13.111 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-09-19T08:14:13.111 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-09-19T08:14:13.111 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-09-19T08:14:13.111 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-09-19T08:14:13.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.112 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-09-19T08:14:13.112 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-09-19T08:14:13.112 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-09-19T08:14:13.112 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 2391, 2024-09-19T08:14:13.112 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 26, 2024-09-19T08:14:13.112 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 4, 2024-09-19T08:14:13.113 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 2391, 2024-09-19T08:14:13.113 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-09-19T08:14:13.113 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2391, 2024-09-19T08:14:13.113 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 1.202974683, 2024-09-19T08:14:13.113 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000503126 2024-09-19T08:14:13.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.114 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-09-19T08:14:13.114 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-09-19T08:14:13.114 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2391, 2024-09-19T08:14:13.114 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 2392 2024-09-19T08:14:13.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.114 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 2391, 2024-09-19T08:14:13.114 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 2391, 2024-09-19T08:14:13.115 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 0, 2024-09-19T08:14:13.115 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-09-19T08:14:13.115 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-09-19T08:14:13.115 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 1533, 2024-09-19T08:14:13.115 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 0, 2024-09-19T08:14:13.115 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 855, 2024-09-19T08:14:13.116 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 0, 2024-09-19T08:14:13.116 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 0, 2024-09-19T08:14:13.116 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-09-19T08:14:13.116 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-09-19T08:14:13.116 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-09-19T08:14:13.117 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 0, 2024-09-19T08:14:13.117 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 0, 2024-09-19T08:14:13.117 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-09-19T08:14:13.117 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-09-19T08:14:13.117 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-09-19T08:14:13.117 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 2, 2024-09-19T08:14:13.117 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 0, 2024-09-19T08:14:13.118 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-09-19T08:14:13.118 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-09-19T08:14:13.118 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-09-19T08:14:13.118 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-09-19T08:14:13.118 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-09-19T08:14:13.119 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-09-19T08:14:13.119 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-09-19T08:14:13.119 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-09-19T08:14:13.119 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-09-19T08:14:13.119 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-09-19T08:14:13.119 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-09-19T08:14:13.120 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-09-19T08:14:13.120 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-09-19T08:14:13.120 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-09-19T08:14:13.120 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-09-19T08:14:13.120 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-09-19T08:14:13.121 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-09-19T08:14:13.121 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-09-19T08:14:13.121 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-09-19T08:14:13.121 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-09-19T08:14:13.121 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-09-19T08:14:13.121 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-09-19T08:14:13.122 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-09-19T08:14:13.122 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-09-19T08:14:13.122 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-09-19T08:14:13.122 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-09-19T08:14:13.122 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 90, 2024-09-19T08:14:13.122 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-09-19T08:14:13.123 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 12, 2024-09-19T08:14:13.123 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 5, 2024-09-19T08:14:13.123 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 5, 2024-09-19T08:14:13.123 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 0, 2024-09-19T08:14:13.123 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-09-19T08:14:13.123 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 0, 2024-09-19T08:14:13.124 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 2, 2024-09-19T08:14:13.124 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 0 2024-09-19T08:14:13.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.124 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-09-19T08:14:13.124 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 0, 2024-09-19T08:14:13.124 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-09-19T08:14:13.125 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-09-19T08:14:13.125 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-09-19T08:14:13.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.125 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-09-19T08:14:13.125 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 0 2024-09-19T08:14:13.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.125 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-09-19T08:14:13.125 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-09-19T08:14:13.125 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 0, 2024-09-19T08:14:13.126 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-09-19T08:14:13.126 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 0, 2024-09-19T08:14:13.126 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 0, 2024-09-19T08:14:13.126 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 0, 2024-09-19T08:14:13.126 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-09-19T08:14:13.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.126 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-19T08:14:13.126 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:13.126 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-09-19T08:14:13.126 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:13.126 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 2646, 2024-09-19T08:14:13.126 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 4865711, 2024-09-19T08:14:13.126 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:13.127 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 2646, 2024-09-19T08:14:13.127 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-09-19T08:14:13.127 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-09-19T08:14:13.127 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 2646, 2024-09-19T08:14:13.127 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 4865711, 2024-09-19T08:14:13.127 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:13.127 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.127 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.127 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.127 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:13.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.127 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-09-19T08:14:13.127 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:13.127 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-09-19T08:14:13.128 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:13.128 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-09-19T08:14:13.128 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-09-19T08:14:13.128 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:13.128 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:13.128 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 2391, 2024-09-19T08:14:13.128 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 3051474, 2024-09-19T08:14:13.128 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 2391, 2024-09-19T08:14:13.128 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 3051474, 2024-09-19T08:14:13.128 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:13.128 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.128 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.128 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.129 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:13.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.129 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-09-19T08:14:13.129 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:13.129 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-09-19T08:14:13.129 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:13.129 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-09-19T08:14:13.129 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-09-19T08:14:13.129 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:13.129 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:13.129 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 2391, 2024-09-19T08:14:13.129 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 2391, 2024-09-19T08:14:13.129 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 2391, 2024-09-19T08:14:13.129 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 2391, 2024-09-19T08:14:13.130 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:13.130 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.130 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.130 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.130 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:13.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.130 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-09-19T08:14:13.130 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:13.130 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-09-19T08:14:13.130 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:13.130 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-09-19T08:14:13.130 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-09-19T08:14:13.130 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:13.131 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:13.131 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-09-19T08:14:13.131 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-09-19T08:14:13.131 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-09-19T08:14:13.131 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-09-19T08:14:13.131 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:13.131 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.131 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.131 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.131 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:13.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:13.131 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x55e688c5cb40": { 2024-09-19T08:14:13.132 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:13.132 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-09-19T08:14:13.132 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:13.132 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-09-19T08:14:13.132 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-09-19T08:14:13.132 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:13.132 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:13.132 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-09-19T08:14:13.132 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-09-19T08:14:13.132 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-09-19T08:14:13.132 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-09-19T08:14:13.132 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:13.132 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:13.132 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:13.133 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:13.133 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:13.133 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:13.133 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-19T08:14:13.689 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-09-19T08:14:13.689 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-09-19T08:14:13.689 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-09-19T08:14:13.689 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=0 2024-09-19T08:14:13.689 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-09-19T08:14:13.689 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-09-19T08:14:13.690 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-09-19T08:14:13.690 DEBUG:tasks.check_counter:Getting stats from l 2024-09-19T08:14:13.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph daemon mds.l perf dump 2024-09-19T08:14:13.939 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T08:14:14.307 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:13 smithi162 ceph-mon[35260]: pgmap v348: 97 pgs: 97 active+clean; 7.3 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:14:14.409 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-19T08:14:14.409 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-09-19T08:14:14.409 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 514, 2024-09-19T08:14:14.409 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 593, 2024-09-19T08:14:14.409 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 183689, 2024-09-19T08:14:14.409 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 141381, 2024-09-19T08:14:14.409 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 8, 2024-09-19T08:14:14.409 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 3, 2024-09-19T08:14:14.409 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.062526634, 2024-09-19T08:14:14.410 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.023010922, 2024-09-19T08:14:14.410 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.031186653, 2024-09-19T08:14:14.410 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.004966914, 2024-09-19T08:14:14.410 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:14.410 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 591, 2024-09-19T08:14:14.410 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.034925999, 2024-09-19T08:14:14.410 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000059096 2024-09-19T08:14:14.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.410 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:14.410 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1060, 2024-09-19T08:14:14.410 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000248453, 2024-09-19T08:14:14.410 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000234 2024-09-19T08:14:14.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.411 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-19T08:14:14.411 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 0 2024-09-19T08:14:14.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.411 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-09-19T08:14:14.411 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 26, 2024-09-19T08:14:14.411 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 195, 2024-09-19T08:14:14.411 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 12244, 2024-09-19T08:14:14.411 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 570552, 2024-09-19T08:14:14.411 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 19, 2024-09-19T08:14:14.412 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 4, 2024-09-19T08:14:14.412 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.015828970, 2024-09-19T08:14:14.412 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.007229262, 2024-09-19T08:14:14.412 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.006272164, 2024-09-19T08:14:14.412 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.000220020, 2024-09-19T08:14:14.412 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:14.412 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 192, 2024-09-19T08:14:14.412 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.030508733, 2024-09-19T08:14:14.412 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000158899 2024-09-19T08:14:14.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.412 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:14.413 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 21, 2024-09-19T08:14:14.413 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000004212, 2024-09-19T08:14:14.413 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000200 2024-09-19T08:14:14.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.413 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 1248, 2024-09-19T08:14:14.413 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 455680 2024-09-19T08:14:14.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.413 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-09-19T08:14:14.413 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 332, 2024-09-19T08:14:14.413 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 314, 2024-09-19T08:14:14.413 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 1396924, 2024-09-19T08:14:14.413 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 493440, 2024-09-19T08:14:14.413 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 33, 2024-09-19T08:14:14.414 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-09-19T08:14:14.414 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.060300465, 2024-09-19T08:14:14.414 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.018464246, 2024-09-19T08:14:14.414 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.053298081, 2024-09-19T08:14:14.414 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.000536353, 2024-09-19T08:14:14.414 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-09-19T08:14:14.414 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 314, 2024-09-19T08:14:14.414 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.035757391, 2024-09-19T08:14:14.414 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000113877 2024-09-19T08:14:14.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.414 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-09-19T08:14:14.414 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 28, 2024-09-19T08:14:14.414 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000002295, 2024-09-19T08:14:14.415 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000081 2024-09-19T08:14:14.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.415 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 1388800, 2024-09-19T08:14:14.415 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 144576 2024-09-19T08:14:14.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.415 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-09-19T08:14:14.415 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-09-19T08:14:14.415 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-09-19T08:14:14.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.416 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-09-19T08:14:14.416 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-09-19T08:14:14.416 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-09-19T08:14:14.416 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 41, 2024-09-19T08:14:14.416 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.001452554, 2024-09-19T08:14:14.416 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000035428 2024-09-19T08:14:14.417 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:14.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.417 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-09-19T08:14:14.417 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-09-19T08:14:14.417 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-09-19T08:14:14.417 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2, 2024-09-19T08:14:14.417 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000013576, 2024-09-19T08:14:14.417 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000006788 2024-09-19T08:14:14.417 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:14.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.418 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-09-19T08:14:14.418 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 0, 2024-09-19T08:14:14.418 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 0, 2024-09-19T08:14:14.418 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-09-19T08:14:14.418 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.418 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.418 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.418 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-09-19T08:14:14.418 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 0, 2024-09-19T08:14:14.418 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 0, 2024-09-19T08:14:14.419 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-09-19T08:14:14.419 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 11, 2024-09-19T08:14:14.419 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 0, 2024-09-19T08:14:14.419 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-09-19T08:14:14.420 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 16, 2024-09-19T08:14:14.420 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 0, 2024-09-19T08:14:14.420 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 16, 2024-09-19T08:14:14.420 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 0, 2024-09-19T08:14:14.420 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 13, 2024-09-19T08:14:14.420 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 4, 2024-09-19T08:14:14.420 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 0, 2024-09-19T08:14:14.420 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 0, 2024-09-19T08:14:14.420 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 3, 2024-09-19T08:14:14.420 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 15, 2024-09-19T08:14:14.420 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 11, 2024-09-19T08:14:14.420 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 0, 2024-09-19T08:14:14.421 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 4, 2024-09-19T08:14:14.421 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 0, 2024-09-19T08:14:14.421 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-09-19T08:14:14.421 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 0, 2024-09-19T08:14:14.421 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 0, 2024-09-19T08:14:14.421 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-09-19T08:14:14.421 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 0, 2024-09-19T08:14:14.421 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 0, 2024-09-19T08:14:14.421 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 0, 2024-09-19T08:14:14.421 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 0, 2024-09-19T08:14:14.421 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-09-19T08:14:14.421 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 0, 2024-09-19T08:14:14.422 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-09-19T08:14:14.422 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 4, 2024-09-19T08:14:14.422 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 234, 2024-09-19T08:14:14.422 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 6, 2024-09-19T08:14:14.422 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 0, 2024-09-19T08:14:14.422 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-09-19T08:14:14.422 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-09-19T08:14:14.422 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-09-19T08:14:14.422 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-09-19T08:14:14.422 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-09-19T08:14:14.422 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-09-19T08:14:14.422 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 0, 2024-09-19T08:14:14.422 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 0, 2024-09-19T08:14:14.423 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 0, 2024-09-19T08:14:14.423 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 0, 2024-09-19T08:14:14.423 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-09-19T08:14:14.423 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-19T08:14:14.423 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-19T08:14:14.423 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 0, 2024-09-19T08:14:14.423 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 0, 2024-09-19T08:14:14.423 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 0, 2024-09-19T08:14:14.423 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 0 2024-09-19T08:14:14.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.423 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-09-19T08:14:14.423 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 0, 2024-09-19T08:14:14.423 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 14, 2024-09-19T08:14:14.424 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 9, 2024-09-19T08:14:14.424 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 11, 2024-09-19T08:14:14.424 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 0, 2024-09-19T08:14:14.424 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 0, 2024-09-19T08:14:14.424 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-09-19T08:14:14.424 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-09-19T08:14:14.424 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 0, 2024-09-19T08:14:14.424 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 0, 2024-09-19T08:14:14.424 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-09-19T08:14:14.424 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-09-19T08:14:14.424 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-09-19T08:14:14.424 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-09-19T08:14:14.425 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-09-19T08:14:14.425 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-09-19T08:14:14.425 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-09-19T08:14:14.425 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 3, 2024-09-19T08:14:14.425 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 17, 2024-09-19T08:14:14.425 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-09-19T08:14:14.425 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 0, 2024-09-19T08:14:14.425 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-09-19T08:14:14.425 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 0, 2024-09-19T08:14:14.425 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-09-19T08:14:14.425 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-09-19T08:14:14.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.426 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-09-19T08:14:14.426 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 0, 2024-09-19T08:14:14.426 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 4, 2024-09-19T08:14:14.426 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 0, 2024-09-19T08:14:14.426 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 0, 2024-09-19T08:14:14.426 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 4, 2024-09-19T08:14:14.426 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-09-19T08:14:14.426 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 0, 2024-09-19T08:14:14.426 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 2, 2024-09-19T08:14:14.426 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 0, 2024-09-19T08:14:14.426 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 0, 2024-09-19T08:14:14.426 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 2, 2024-09-19T08:14:14.427 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 2, 2024-09-19T08:14:14.427 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-09-19T08:14:14.427 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 0, 2024-09-19T08:14:14.427 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 4194304, 2024-09-19T08:14:14.427 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 4195426, 2024-09-19T08:14:14.427 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 0, 2024-09-19T08:14:14.427 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-09-19T08:14:14.427 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3, 2024-09-19T08:14:14.427 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.005362651, 2024-09-19T08:14:14.427 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.001787550 2024-09-19T08:14:14.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.428 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 0 2024-09-19T08:14:14.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.428 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-09-19T08:14:14.428 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 20, 2024-09-19T08:14:14.428 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 22, 2024-09-19T08:14:14.428 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 2, 2024-09-19T08:14:14.428 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 19, 2024-09-19T08:14:14.428 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 23, 2024-09-19T08:14:14.428 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 4, 2024-09-19T08:14:14.428 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 16, 2024-09-19T08:14:14.428 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 20, 2024-09-19T08:14:14.429 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 4, 2024-09-19T08:14:14.429 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 0, 2024-09-19T08:14:14.429 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 0, 2024-09-19T08:14:14.429 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 0, 2024-09-19T08:14:14.429 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 50508, 2024-09-19T08:14:14.429 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 223520 2024-09-19T08:14:14.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.429 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-09-19T08:14:14.429 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 0, 2024-09-19T08:14:14.429 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 0, 2024-09-19T08:14:14.429 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 0, 2024-09-19T08:14:14.429 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 4, 2024-09-19T08:14:14.429 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 0, 2024-09-19T08:14:14.430 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-09-19T08:14:14.430 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.430 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.430 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.430 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-09-19T08:14:14.430 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.430 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.430 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.430 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-09-19T08:14:14.431 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.431 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.431 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.431 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-09-19T08:14:14.431 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.431 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.431 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.431 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-09-19T08:14:14.431 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.431 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.431 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.432 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-09-19T08:14:14.432 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.432 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.432 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.432 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-09-19T08:14:14.432 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.432 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.432 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.432 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-09-19T08:14:14.433 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.433 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.433 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.433 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-09-19T08:14:14.433 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.433 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.433 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.433 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-09-19T08:14:14.433 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.433 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.434 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.434 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-09-19T08:14:14.434 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.434 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.434 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.434 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-09-19T08:14:14.434 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.436 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:13 smithi098 ceph-mon[31333]: pgmap v348: 97 pgs: 97 active+clean; 7.3 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:14:14.436 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:13 smithi139 ceph-mon[34874]: pgmap v348: 97 pgs: 97 active+clean; 7.3 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:14:14.436 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.437 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.437 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-09-19T08:14:14.437 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.437 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.437 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.438 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-09-19T08:14:14.438 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.438 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.438 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.439 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-09-19T08:14:14.439 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.439 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.439 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.439 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-09-19T08:14:14.440 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.440 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.440 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.440 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-09-19T08:14:14.441 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.441 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.441 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.442 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-09-19T08:14:14.442 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.442 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.442 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.443 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-09-19T08:14:14.443 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.443 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.443 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.444 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-09-19T08:14:14.444 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.444 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.444 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.444 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-09-19T08:14:14.445 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.445 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.445 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.445 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-09-19T08:14:14.445 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.446 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.446 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.446 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-09-19T08:14:14.446 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.447 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.447 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.447 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-09-19T08:14:14.447 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.447 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.448 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.448 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-09-19T08:14:14.448 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.448 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.449 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.449 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-09-19T08:14:14.449 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.450 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.450 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.450 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-09-19T08:14:14.450 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.451 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.451 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.451 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-09-19T08:14:14.451 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.452 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.452 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.452 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-09-19T08:14:14.452 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.452 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.453 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.453 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-09-19T08:14:14.453 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-09-19T08:14:14.453 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-09-19T08:14:14.454 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.454 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.454 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.454 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:14.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.454 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-09-19T08:14:14.455 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 0, 2024-09-19T08:14:14.455 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 1, 2024-09-19T08:14:14.455 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 1, 2024-09-19T08:14:14.455 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 0, 2024-09-19T08:14:14.455 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-09-19T08:14:14.455 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 0, 2024-09-19T08:14:14.456 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 0, 2024-09-19T08:14:14.456 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 88, 2024-09-19T08:14:14.456 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-09-19T08:14:14.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.456 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-09-19T08:14:14.457 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-09-19T08:14:14.457 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-09-19T08:14:14.457 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-09-19T08:14:14.457 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-09-19T08:14:14.457 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-09-19T08:14:14.458 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-09-19T08:14:14.458 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-19T08:14:14.458 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-09-19T08:14:14.458 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-19T08:14:14.458 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-09-19T08:14:14.458 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-09-19T08:14:14.459 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-09-19T08:14:14.459 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-19T08:14:14.459 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-09-19T08:14:14.459 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-09-19T08:14:14.459 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-09-19T08:14:14.460 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-09-19T08:14:14.460 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-09-19T08:14:14.460 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-19T08:14:14.460 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-09-19T08:14:14.460 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-19T08:14:14.460 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-09-19T08:14:14.461 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-09-19T08:14:14.461 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-09-19T08:14:14.461 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-09-19T08:14:14.461 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-09-19T08:14:14.462 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-19T08:14:14.462 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-09-19T08:14:14.462 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-09-19T08:14:14.462 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-09-19T08:14:14.462 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-09-19T08:14:14.462 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-09-19T08:14:14.463 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-09-19T08:14:14.463 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-09-19T08:14:14.463 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-09-19T08:14:14.463 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-09-19T08:14:14.463 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 246071, 2024-09-19T08:14:14.463 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 78, 2024-09-19T08:14:14.464 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-09-19T08:14:14.464 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-09-19T08:14:14.464 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-09-19T08:14:14.464 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-09-19T08:14:14.464 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-09-19T08:14:14.464 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-09-19T08:14:14.465 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-09-19T08:14:14.465 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-09-19T08:14:14.465 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 8136, 2024-09-19T08:14:14.465 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 180, 2024-09-19T08:14:14.465 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-09-19T08:14:14.465 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-09-19T08:14:14.466 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-09-19T08:14:14.466 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-09-19T08:14:14.466 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 81781, 2024-09-19T08:14:14.466 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 326, 2024-09-19T08:14:14.466 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-09-19T08:14:14.466 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-09-19T08:14:14.466 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-09-19T08:14:14.467 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-09-19T08:14:14.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.467 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-09-19T08:14:14.467 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-09-19T08:14:14.467 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-09-19T08:14:14.467 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 39, 2024-09-19T08:14:14.467 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 11424, 2024-09-19T08:14:14.467 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 33, 2024-09-19T08:14:14.467 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 39, 2024-09-19T08:14:14.467 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-09-19T08:14:14.467 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 39, 2024-09-19T08:14:14.468 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.207995666, 2024-09-19T08:14:14.468 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.005333222 2024-09-19T08:14:14.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.468 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-09-19T08:14:14.468 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-09-19T08:14:14.468 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 39, 2024-09-19T08:14:14.468 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 61 2024-09-19T08:14:14.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.468 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 39, 2024-09-19T08:14:14.468 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 0, 2024-09-19T08:14:14.468 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 39, 2024-09-19T08:14:14.468 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-09-19T08:14:14.468 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-09-19T08:14:14.469 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 0, 2024-09-19T08:14:14.469 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 10, 2024-09-19T08:14:14.469 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 0, 2024-09-19T08:14:14.469 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 3, 2024-09-19T08:14:14.469 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 7, 2024-09-19T08:14:14.469 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-09-19T08:14:14.469 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-09-19T08:14:14.469 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-09-19T08:14:14.470 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 0, 2024-09-19T08:14:14.470 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 6, 2024-09-19T08:14:14.470 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-09-19T08:14:14.470 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-09-19T08:14:14.470 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-09-19T08:14:14.471 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 0, 2024-09-19T08:14:14.471 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 20, 2024-09-19T08:14:14.471 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-09-19T08:14:14.471 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-09-19T08:14:14.472 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-09-19T08:14:14.472 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-09-19T08:14:14.472 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-09-19T08:14:14.472 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-09-19T08:14:14.472 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-09-19T08:14:14.472 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-09-19T08:14:14.472 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-09-19T08:14:14.472 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-09-19T08:14:14.472 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-09-19T08:14:14.472 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-09-19T08:14:14.473 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-09-19T08:14:14.473 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-09-19T08:14:14.473 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-09-19T08:14:14.473 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-09-19T08:14:14.473 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-09-19T08:14:14.473 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-09-19T08:14:14.473 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-09-19T08:14:14.473 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-09-19T08:14:14.473 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-09-19T08:14:14.473 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-09-19T08:14:14.473 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-09-19T08:14:14.473 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-09-19T08:14:14.474 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-09-19T08:14:14.474 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-09-19T08:14:14.474 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 90, 2024-09-19T08:14:14.474 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-09-19T08:14:14.474 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 12, 2024-09-19T08:14:14.474 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 9, 2024-09-19T08:14:14.474 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 9, 2024-09-19T08:14:14.474 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 0, 2024-09-19T08:14:14.474 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-09-19T08:14:14.474 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 14, 2024-09-19T08:14:14.474 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 0, 2024-09-19T08:14:14.474 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 1 2024-09-19T08:14:14.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.475 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-09-19T08:14:14.475 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 1, 2024-09-19T08:14:14.475 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-09-19T08:14:14.475 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-09-19T08:14:14.475 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-09-19T08:14:14.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.475 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-09-19T08:14:14.475 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 20 2024-09-19T08:14:14.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.476 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-09-19T08:14:14.476 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-09-19T08:14:14.476 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 0, 2024-09-19T08:14:14.476 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-09-19T08:14:14.476 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 0, 2024-09-19T08:14:14.476 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 0, 2024-09-19T08:14:14.476 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 0, 2024-09-19T08:14:14.476 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-09-19T08:14:14.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.476 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-19T08:14:14.476 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:14.476 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-09-19T08:14:14.476 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:14.477 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 868, 2024-09-19T08:14:14.477 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 1497069, 2024-09-19T08:14:14.477 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:14.477 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 868, 2024-09-19T08:14:14.477 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-09-19T08:14:14.477 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-09-19T08:14:14.477 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 868, 2024-09-19T08:14:14.477 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 1497069, 2024-09-19T08:14:14.477 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:14.477 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.478 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.478 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.478 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:14.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.478 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-09-19T08:14:14.478 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:14.478 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-09-19T08:14:14.478 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:14.478 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-09-19T08:14:14.478 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-09-19T08:14:14.478 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:14.478 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:14.478 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 39, 2024-09-19T08:14:14.478 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 11424, 2024-09-19T08:14:14.479 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 39, 2024-09-19T08:14:14.479 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 11424, 2024-09-19T08:14:14.479 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:14.479 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.479 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.479 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.479 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:14.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.479 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-09-19T08:14:14.479 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:14.480 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-09-19T08:14:14.480 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:14.480 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-09-19T08:14:14.480 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-09-19T08:14:14.480 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:14.480 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:14.480 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 39, 2024-09-19T08:14:14.480 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 39, 2024-09-19T08:14:14.480 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 39, 2024-09-19T08:14:14.481 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 39, 2024-09-19T08:14:14.481 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:14.481 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.481 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.481 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.481 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:14.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.481 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-09-19T08:14:14.481 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:14.481 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-09-19T08:14:14.481 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:14.481 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-09-19T08:14:14.481 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-09-19T08:14:14.482 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:14.482 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-09-19T08:14:14.482 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-09-19T08:14:14.482 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-09-19T08:14:14.482 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-09-19T08:14:14.482 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-09-19T08:14:14.482 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:14.482 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.482 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.482 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.482 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:14.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-19T08:14:14.482 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x55a374f30960": { 2024-09-19T08:14:14.483 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-09-19T08:14:14.483 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-09-19T08:14:14.483 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-09-19T08:14:14.483 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 4, 2024-09-19T08:14:14.483 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 1122, 2024-09-19T08:14:14.483 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-09-19T08:14:14.483 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 4, 2024-09-19T08:14:14.483 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-09-19T08:14:14.483 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-09-19T08:14:14.483 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 3, 2024-09-19T08:14:14.483 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 1122, 2024-09-19T08:14:14.483 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-09-19T08:14:14.483 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-09-19T08:14:14.484 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-09-19T08:14:14.484 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-09-19T08:14:14.484 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:14.484 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-19T08:14:14.484 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-19T08:14:15.231 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-09-19T08:14:15.231 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-09-19T08:14:15.231 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-09-19T08:14:15.231 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=14 2024-09-19T08:14:15.231 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=6 2024-09-19T08:14:15.232 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-09-19T08:14:15.232 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-09-19T08:14:15.232 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-19T08:14:15.242 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-19T08:14:15.243 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T08:14:15.243 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-19T08:14:15.264 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-19T08:14:15.264 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-19T08:14:15.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-19T08:14:15.620 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:14:16.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.105+0000 7fd178e86640 1 -- 172.21.15.98:0/3753234870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd174101df0 msgr2=0x7fd1741021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:16.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.105+0000 7fd178e86640 1 --2- 172.21.15.98:0/3753234870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd174101df0 0x7fd1741021d0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009900 tx=0x7fd16c02a410 comp rx=0 tx=0).stop 2024-09-19T08:14:16.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.105+0000 7fd178e86640 1 -- 172.21.15.98:0/3753234870 shutdown_connections 2024-09-19T08:14:16.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.105+0000 7fd178e86640 1 --2- 172.21.15.98:0/3753234870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd174106f20 0x7fd17410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.105+0000 7fd178e86640 1 --2- 172.21.15.98:0/3753234870 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1741027a0 0x7fd1741067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.105+0000 7fd178e86640 1 --2- 172.21.15.98:0/3753234870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd174101df0 0x7fd1741021d0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.105+0000 7fd178e86640 1 -- 172.21.15.98:0/3753234870 >> 172.21.15.98:0/3753234870 conn(0x7fd1740fd8f0 msgr2=0x7fd1740ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.105+0000 7fd178e86640 1 -- 172.21.15.98:0/3753234870 shutdown_connections 2024-09-19T08:14:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.105+0000 7fd178e86640 1 -- 172.21.15.98:0/3753234870 wait complete. 2024-09-19T08:14:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.105+0000 7fd178e86640 1 Processor -- start 2024-09-19T08:14:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd178e86640 1 -- start start 2024-09-19T08:14:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd178e86640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd174101df0 0x7fd174198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd178e86640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1741027a0 0x7fd174198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd178e86640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd174106f20 0x7fd17419cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd178e86640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd174111f80 con 0x7fd174106f20 2024-09-19T08:14:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd178e86640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd174111e00 con 0x7fd1741027a0 2024-09-19T08:14:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd178e86640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd174112100 con 0x7fd174101df0 2024-09-19T08:14:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd172d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd174106f20 0x7fd17419cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd172d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd174106f20 0x7fd17419cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48854/0 (socket says 172.21.15.98:48854) 2024-09-19T08:14:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd17259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd174101df0 0x7fd174198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd172d9d640 1 -- 172.21.15.98:0/1998762670 learned_addr learned my addr 172.21.15.98:0/1998762670 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:14:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd171d9b640 1 --2- 172.21.15.98:0/1998762670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1741027a0 0x7fd174198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd172d9d640 1 -- 172.21.15.98:0/1998762670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd174101df0 msgr2=0x7fd174198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd172d9d640 1 --2- 172.21.15.98:0/1998762670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd174101df0 0x7fd174198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd172d9d640 1 -- 172.21.15.98:0/1998762670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1741027a0 msgr2=0x7fd174198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd172d9d640 1 --2- 172.21.15.98:0/1998762670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1741027a0 0x7fd174198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd172d9d640 1 -- 172.21.15.98:0/1998762670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17419d4d0 con 0x7fd174106f20 2024-09-19T08:14:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.106+0000 7fd172d9d640 1 --2- 172.21.15.98:0/1998762670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd174106f20 0x7fd17419cdf0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fd16400d950 tx=0x7fd16400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.107+0000 7fd16b7fe640 1 -- 172.21.15.98:0/1998762670 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164017030 con 0x7fd174106f20 2024-09-19T08:14:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.107+0000 7fd16b7fe640 1 -- 172.21.15.98:0/1998762670 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd164004020 con 0x7fd174106f20 2024-09-19T08:14:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.107+0000 7fd178e86640 1 -- 172.21.15.98:0/1998762670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd17419d7c0 con 0x7fd174106f20 2024-09-19T08:14:16.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.107+0000 7fd16b7fe640 1 -- 172.21.15.98:0/1998762670 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164003c00 con 0x7fd174106f20 2024-09-19T08:14:16.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.107+0000 7fd178e86640 1 -- 172.21.15.98:0/1998762670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1741a50a0 con 0x7fd174106f20 2024-09-19T08:14:16.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.108+0000 7fd178e86640 1 -- 172.21.15.98:0/1998762670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1741021d0 con 0x7fd174106f20 2024-09-19T08:14:16.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.109+0000 7fd16b7fe640 1 -- 172.21.15.98:0/1998762670 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fd16401a070 con 0x7fd174106f20 2024-09-19T08:14:16.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.110+0000 7fd16b7fe640 1 --2- 172.21.15.98:0/1998762670 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd1600a3430 0x7fd1600a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:16.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.110+0000 7fd17259c640 1 --2- 172.21.15.98:0/1998762670 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd1600a3430 0x7fd1600a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:16.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.110+0000 7fd16b7fe640 1 -- 172.21.15.98:0/1998762670 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8521+0+0 (secure 0 0 0) 0x7fd1640cf440 con 0x7fd174106f20 2024-09-19T08:14:16.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.110+0000 7fd17259c640 1 --2- 172.21.15.98:0/1998762670 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd1600a3430 0x7fd1600a58f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009230 tx=0x7fd16c0058d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:16.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.115+0000 7fd16b7fe640 1 -- 172.21.15.98:0/1998762670 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fd164021070 con 0x7fd174106f20 2024-09-19T08:14:16.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.294+0000 7fd178e86640 1 -- 172.21.15.98:0/1998762670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd17410a100 con 0x7fd174106f20 2024-09-19T08:14:16.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.295+0000 7fd16b7fe640 1 -- 172.21.15.98:0/1998762670 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v91) ==== 81+0+938 (secure 0 0 0) 0x7fd16409be20 con 0x7fd174106f20 2024-09-19T08:14:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1252058007 2024-09-20T08:03:00.753287+0000 2024-09-19T08:14:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/921776066 2024-09-20T08:03:00.753287+0000 2024-09-19T08:14:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/336479555 2024-09-20T08:03:00.753287+0000 2024-09-19T08:14:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/1560445911 2024-09-20T07:54:02.418785+0000 2024-09-19T08:14:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/16410793 2024-09-20T07:54:25.475454+0000 2024-09-19T08:14:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/1560445911 2024-09-20T07:54:02.418785+0000 2024-09-19T08:14:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1497156658 2024-09-20T07:54:02.418785+0000 2024-09-19T08:14:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/470997812 2024-09-20T08:03:00.753287+0000 2024-09-19T08:14:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1238193420 2024-09-20T07:54:25.475454+0000 2024-09-19T08:14:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/470997812 2024-09-20T08:03:00.753287+0000 2024-09-19T08:14:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3838590070 2024-09-20T07:54:25.475454+0000 2024-09-19T08:14:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/16410793 2024-09-20T07:54:25.475454+0000 2024-09-19T08:14:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3842627255 2024-09-20T07:54:25.475454+0000 2024-09-19T08:14:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3866031018 2024-09-20T07:54:02.418785+0000 2024-09-19T08:14:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3194864054 2024-09-20T07:54:02.418785+0000 2024-09-19T08:14:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3067331617 2024-09-20T08:03:00.753287+0000 2024-09-19T08:14:16.297 INFO:teuthology.orchestra.run.smithi098.stderr:listed 16 entries 2024-09-19T08:14:16.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.297+0000 7fd178e86640 1 -- 172.21.15.98:0/1998762670 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd1600a3430 msgr2=0x7fd1600a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:16.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.297+0000 7fd178e86640 1 --2- 172.21.15.98:0/1998762670 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd1600a3430 0x7fd1600a58f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009230 tx=0x7fd16c0058d0 comp rx=0 tx=0).stop 2024-09-19T08:14:16.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.297+0000 7fd178e86640 1 -- 172.21.15.98:0/1998762670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd174106f20 msgr2=0x7fd17419cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:16.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.297+0000 7fd178e86640 1 --2- 172.21.15.98:0/1998762670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd174106f20 0x7fd17419cdf0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fd16400d950 tx=0x7fd16400de10 comp rx=0 tx=0).stop 2024-09-19T08:14:16.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.297+0000 7fd178e86640 1 -- 172.21.15.98:0/1998762670 shutdown_connections 2024-09-19T08:14:16.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.297+0000 7fd178e86640 1 --2- 172.21.15.98:0/1998762670 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fd1600a3430 0x7fd1600a58f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:16.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.297+0000 7fd178e86640 1 --2- 172.21.15.98:0/1998762670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd174106f20 0x7fd17419cdf0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:16.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.297+0000 7fd178e86640 1 --2- 172.21.15.98:0/1998762670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1741027a0 0x7fd174198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:16.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.297+0000 7fd178e86640 1 --2- 172.21.15.98:0/1998762670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd174101df0 0x7fd174198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:16.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.297+0000 7fd178e86640 1 -- 172.21.15.98:0/1998762670 >> 172.21.15.98:0/1998762670 conn(0x7fd1740fd8f0 msgr2=0x7fd1740fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:16.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.297+0000 7fd178e86640 1 -- 172.21.15.98:0/1998762670 shutdown_connections 2024-09-19T08:14:16.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:16.297+0000 7fd178e86640 1 -- 172.21.15.98:0/1998762670 wait complete. 2024-09-19T08:14:17.024 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-19T08:14:17.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T08:14:17.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-19T08:14:17.047 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:16 smithi098 ceph-mon[31333]: pgmap v349: 97 pgs: 97 active+clean; 7.3 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:14:17.047 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:16 smithi098 ceph-mon[31333]: osdmap e91: 12 total, 12 up, 12 in 2024-09-19T08:14:17.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:16 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1998762670' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-19T08:14:17.066 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-09-19T08:14:17.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T08:14:17.066 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-19T08:14:17.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T08:14:17.129 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-09-19T08:14:17.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:16 smithi139 ceph-mon[34874]: pgmap v349: 97 pgs: 97 active+clean; 7.3 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:14:17.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:16 smithi139 ceph-mon[34874]: osdmap e91: 12 total, 12 up, 12 in 2024-09-19T08:14:17.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:16 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1998762670' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-19T08:14:17.185 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-19T08:14:17.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T08:14:17.186 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-19T08:14:17.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T08:14:17.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-09-19T08:14:17.348 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-19T08:14:17.360 INFO:tasks.cephadm:Teardown begin 2024-09-19T08:14:17.360 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-19T08:14:17.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:14:17.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:16 smithi162 ceph-mon[35260]: pgmap v349: 97 pgs: 97 active+clean; 7.3 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-19T08:14:17.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:16 smithi162 ceph-mon[35260]: osdmap e91: 12 total, 12 up, 12 in 2024-09-19T08:14:17.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:16 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1998762670' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-19T08:14:17.608 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:14:18.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:17 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:14:18.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.119+0000 7fb41de3d640 1 -- 172.21.15.98:0/1645870862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb418102570 msgr2=0x7fb4181029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:18.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.119+0000 7fb41de3d640 1 --2- 172.21.15.98:0/1645870862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb418102570 0x7fb4181029b0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009900 tx=0x7fb40c02a480 comp rx=0 tx=0).stop 2024-09-19T08:14:18.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.119+0000 7fb41de3d640 1 -- 172.21.15.98:0/1645870862 shutdown_connections 2024-09-19T08:14:18.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.119+0000 7fb41de3d640 1 --2- 172.21.15.98:0/1645870862 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb418102ef0 0x7fb41810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:18.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.119+0000 7fb41de3d640 1 --2- 172.21.15.98:0/1645870862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb418102570 0x7fb4181029b0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.119+0000 7fb41de3d640 1 --2- 172.21.15.98:0/1645870862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb418100e20 0x7fb418101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.119+0000 7fb41de3d640 1 -- 172.21.15.98:0/1645870862 >> 172.21.15.98:0/1645870862 conn(0x7fb4180fc7d0 msgr2=0x7fb4180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.119+0000 7fb41de3d640 1 -- 172.21.15.98:0/1645870862 shutdown_connections 2024-09-19T08:14:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.119+0000 7fb41de3d640 1 -- 172.21.15.98:0/1645870862 wait complete. 2024-09-19T08:14:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.120+0000 7fb41de3d640 1 Processor -- start 2024-09-19T08:14:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.120+0000 7fb41de3d640 1 -- start start 2024-09-19T08:14:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.120+0000 7fb41de3d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb418100e20 0x7fb418198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.120+0000 7fb41de3d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb418102570 0x7fb418198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.120+0000 7fb41de3d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb418102ef0 0x7fb41819ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.120+0000 7fb41de3d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4181120a0 con 0x7fb418102570 2024-09-19T08:14:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.120+0000 7fb41de3d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb418111f20 con 0x7fb418102ef0 2024-09-19T08:14:18.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.120+0000 7fb41de3d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb418112220 con 0x7fb418100e20 2024-09-19T08:14:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb416ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb418102570 0x7fb418198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb416ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb418102570 0x7fb418198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36612/0 (socket says 172.21.15.98:36612) 2024-09-19T08:14:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb416ffd640 1 -- 172.21.15.98:0/3519397643 learned_addr learned my addr 172.21.15.98:0/3519397643 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:14:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb4177fe640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb418100e20 0x7fb418198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb417fff640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb418102ef0 0x7fb41819ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb416ffd640 1 -- 172.21.15.98:0/3519397643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb418100e20 msgr2=0x7fb418198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb416ffd640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb418100e20 0x7fb418198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb416ffd640 1 -- 172.21.15.98:0/3519397643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb418102ef0 msgr2=0x7fb41819ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb416ffd640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb418102ef0 0x7fb41819ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb416ffd640 1 -- 172.21.15.98:0/3519397643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41819d510 con 0x7fb418102570 2024-09-19T08:14:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb417fff640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb418102ef0 0x7fb41819ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:14:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb416ffd640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb418102570 0x7fb418198aa0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fb40c006010 tx=0x7fb40c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb414ff9640 1 -- 172.21.15.98:0/3519397643 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c03f070 con 0x7fb418102570 2024-09-19T08:14:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb41de3d640 1 -- 172.21.15.98:0/3519397643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb41819d7a0 con 0x7fb418102570 2024-09-19T08:14:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb414ff9640 1 -- 172.21.15.98:0/3519397643 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb40c031c30 con 0x7fb418102570 2024-09-19T08:14:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb414ff9640 1 -- 172.21.15.98:0/3519397643 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c035030 con 0x7fb418102570 2024-09-19T08:14:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.121+0000 7fb41de3d640 1 -- 172.21.15.98:0/3519397643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4181a50e0 con 0x7fb418102570 2024-09-19T08:14:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.122+0000 7fb41de3d640 1 -- 172.21.15.98:0/3519397643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4181021b0 con 0x7fb418102570 2024-09-19T08:14:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.124+0000 7fb414ff9640 1 -- 172.21.15.98:0/3519397643 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fb40c042070 con 0x7fb418102570 2024-09-19T08:14:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.124+0000 7fb414ff9640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb4000a33b0 0x7fb4000a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.124+0000 7fb4177fe640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb4000a33b0 0x7fb4000a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.124+0000 7fb414ff9640 1 -- 172.21.15.98:0/3519397643 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8521+0+0 (secure 0 0 0) 0x7fb40c0f09f0 con 0x7fb418102570 2024-09-19T08:14:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.125+0000 7fb4177fe640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb4000a33b0 0x7fb4000a5870 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb408009bd0 tx=0x7fb408005f90 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.128+0000 7fb414ff9640 1 -- 172.21.15.98:0/3519397643 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb40c0bd3d0 con 0x7fb418102570 2024-09-19T08:14:18.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:17 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:14:18.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.329+0000 7fb41de3d640 1 -- 172.21.15.98:0/3519397643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb418109090 con 0x7fb418102570 2024-09-19T08:14:18.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.329+0000 7fb414ff9640 1 -- 172.21.15.98:0/3519397643 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11983 (secure 0 0 0) 0x7fb40c031de0 con 0x7fb418102570 2024-09-19T08:14:18.330 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:14:18.330 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"btime":"2024-09-19T08:08:26:985720+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:08:26.555658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:14:18.330 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2024-09-19T08:14:18.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.331+0000 7fb41de3d640 1 -- 172.21.15.98:0/3519397643 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb4000a33b0 msgr2=0x7fb4000a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:18.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.331+0000 7fb41de3d640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb4000a33b0 0x7fb4000a5870 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb408009bd0 tx=0x7fb408005f90 comp rx=0 tx=0).stop 2024-09-19T08:14:18.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.331+0000 7fb41de3d640 1 -- 172.21.15.98:0/3519397643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb418102570 msgr2=0x7fb418198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:18.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.331+0000 7fb41de3d640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb418102570 0x7fb418198aa0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fb40c006010 tx=0x7fb40c003c10 comp rx=0 tx=0).stop 2024-09-19T08:14:18.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.332+0000 7fb41de3d640 1 -- 172.21.15.98:0/3519397643 shutdown_connections 2024-09-19T08:14:18.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.332+0000 7fb41de3d640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fb4000a33b0 0x7fb4000a5870 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:18.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.332+0000 7fb41de3d640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb418102ef0 0x7fb41819ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:18.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.332+0000 7fb41de3d640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb418102570 0x7fb418198aa0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:18.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.332+0000 7fb41de3d640 1 --2- 172.21.15.98:0/3519397643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb418100e20 0x7fb418198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:18.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.332+0000 7fb41de3d640 1 -- 172.21.15.98:0/3519397643 >> 172.21.15.98:0/3519397643 conn(0x7fb4180fc7d0 msgr2=0x7fb4180febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:18.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.332+0000 7fb41de3d640 1 -- 172.21.15.98:0/3519397643 shutdown_connections 2024-09-19T08:14:18.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:18.332+0000 7fb41de3d640 1 -- 172.21.15.98:0/3519397643 wait complete. 2024-09-19T08:14:18.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:17 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:14:18.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:18 smithi098 ceph-mon[31333]: pgmap v351: 97 pgs: 97 active+clean; 7.3 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-19T08:14:18.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:14:18.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:14:18.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:18 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:14:18.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:18 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3519397643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:14:19.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T08:14:19.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:18 smithi139 ceph-mon[34874]: pgmap v351: 97 pgs: 97 active+clean; 7.3 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-19T08:14:19.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:14:19.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:14:19.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:18 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:14:19.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:18 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3519397643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:14:19.307 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:14:19.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:18 smithi162 ceph-mon[35260]: pgmap v351: 97 pgs: 97 active+clean; 7.3 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-19T08:14:19.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:14:19.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:14:19.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:18 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' 2024-09-19T08:14:19.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:18 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3519397643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:14:19.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.814+0000 7f7be8286640 1 -- 172.21.15.98:0/2232007954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7be0100a80 msgr2=0x7f7be0100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:19.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.814+0000 7f7be8286640 1 --2- 172.21.15.98:0/2232007954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7be0100a80 0x7f7be0100ee0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc009930 tx=0x7f7bdc02a420 comp rx=0 tx=0).stop 2024-09-19T08:14:19.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.814+0000 7f7be8286640 1 -- 172.21.15.98:0/2232007954 shutdown_connections 2024-09-19T08:14:19.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.814+0000 7f7be8286640 1 --2- 172.21.15.98:0/2232007954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7be0101420 0x7f7be010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:19.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.814+0000 7f7be8286640 1 --2- 172.21.15.98:0/2232007954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7be0100a80 0x7f7be0100ee0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:19.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.814+0000 7f7be8286640 1 --2- 172.21.15.98:0/2232007954 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7be0106a80 0x7f7be0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:19.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.814+0000 7f7be8286640 1 -- 172.21.15.98:0/2232007954 >> 172.21.15.98:0/2232007954 conn(0x7f7be00fc7a0 msgr2=0x7f7be00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.814+0000 7f7be8286640 1 -- 172.21.15.98:0/2232007954 shutdown_connections 2024-09-19T08:14:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.814+0000 7f7be8286640 1 -- 172.21.15.98:0/2232007954 wait complete. 2024-09-19T08:14:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be8286640 1 Processor -- start 2024-09-19T08:14:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be8286640 1 -- start start 2024-09-19T08:14:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be8286640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7be0100a80 0x7f7be019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be8286640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7be0101420 0x7f7be019ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be8286640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7be0106a80 0x7f7be019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be8286640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7be0110ec0 con 0x7f7be0100a80 2024-09-19T08:14:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be8286640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7be0110d40 con 0x7f7be0106a80 2024-09-19T08:14:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be8286640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7be0111040 con 0x7f7be0101420 2024-09-19T08:14:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be6022640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7be0100a80 0x7f7be019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be6022640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7be0100a80 0x7f7be019a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36628/0 (socket says 172.21.15.98:36628) 2024-09-19T08:14:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be6022640 1 -- 172.21.15.98:0/3678287857 learned_addr learned my addr 172.21.15.98:0/3678287857 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:14:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be6823640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7be0106a80 0x7f7be019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be5821640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7be0101420 0x7f7be019ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be6022640 1 -- 172.21.15.98:0/3678287857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7be0101420 msgr2=0x7f7be019ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be6022640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7be0101420 0x7f7be019ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be6022640 1 -- 172.21.15.98:0/3678287857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7be0106a80 msgr2=0x7f7be019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be6022640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7be0106a80 0x7f7be019eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be6022640 1 -- 172.21.15.98:0/3678287857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be019f6d0 con 0x7f7be0100a80 2024-09-19T08:14:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be5821640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7be0101420 0x7f7be019ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T08:14:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.815+0000 7f7be6022640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7be0100a80 0x7f7be019a720 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400b4a0 tx=0x7f7bd400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.816+0000 7f7bd37fe640 1 -- 172.21.15.98:0/3678287857 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4014070 con 0x7f7be0100a80 2024-09-19T08:14:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.816+0000 7f7bd37fe640 1 -- 172.21.15.98:0/3678287857 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bd4010af0 con 0x7f7be0100a80 2024-09-19T08:14:19.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.816+0000 7f7be8286640 1 -- 172.21.15.98:0/3678287857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be019f9c0 con 0x7f7be0100a80 2024-09-19T08:14:19.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.816+0000 7f7bd37fe640 1 -- 172.21.15.98:0/3678287857 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4017d10 con 0x7f7be0100a80 2024-09-19T08:14:19.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.816+0000 7f7be8286640 1 -- 172.21.15.98:0/3678287857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be01a72a0 con 0x7f7be0100a80 2024-09-19T08:14:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.817+0000 7f7be8286640 1 -- 172.21.15.98:0/3678287857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7be0100ee0 con 0x7f7be0100a80 2024-09-19T08:14:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.818+0000 7f7bd37fe640 1 -- 172.21.15.98:0/3678287857 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f7bd4017070 con 0x7f7be0100a80 2024-09-19T08:14:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.819+0000 7f7bd37fe640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7bcc0a33b0 0x7f7bcc0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.819+0000 7f7be5821640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7bcc0a33b0 0x7f7bcc0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.819+0000 7f7bd37fe640 1 -- 172.21.15.98:0/3678287857 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8521+0+0 (secure 0 0 0) 0x7f7bd40cf090 con 0x7f7be0100a80 2024-09-19T08:14:19.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.819+0000 7f7be5821640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7bcc0a33b0 0x7f7bcc0a5870 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc000c00 tx=0x7f7bdc032c70 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:19.823+0000 7f7bd37fe640 1 -- 172.21.15.98:0/3678287857 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f7bd409bb80 con 0x7f7be0100a80 2024-09-19T08:14:20.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:20.024+0000 7f7be8286640 1 -- 172.21.15.98:0/3678287857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7be01a75d0 con 0x7f7be0100a80 2024-09-19T08:14:20.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:20.025+0000 7f7bd37fe640 1 -- 172.21.15.98:0/3678287857 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11983 (secure 0 0 0) 0x7f7bd409b480 con 0x7f7be0100a80 2024-09-19T08:14:20.025 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:14:20.026 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"btime":"2024-09-19T08:08:26:985720+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/85442870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":85442870},{"type":"v1","addr":"172.21.15.98:6835","nonce":85442870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3958359300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3958359300},{"type":"v1","addr":"172.21.15.98:6837","nonce":3958359300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T08:01:11.307169+0000","modified":"2024-09-19T08:08:26.555658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24461,"mds_2":24437,"mds_3":24406,"mds_4":24376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3417988465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3417988465},{"type":"v1","addr":"172.21.15.98:6839","nonce":3417988465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/1339458935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1339458935},{"type":"v1","addr":"172.21.15.98:6841","nonce":1339458935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24376":{"gid":24376,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6833/3959364141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3959364141},{"type":"v1","addr":"172.21.15.162:6833","nonce":3959364141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6835/1149502202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1149502202},{"type":"v1","addr":"172.21.15.162:6835","nonce":1149502202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6833/2267154969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":2267154969},{"type":"v1","addr":"172.21.15.139:6833","nonce":2267154969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3969500353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3969500353},{"type":"v1","addr":"172.21.15.162:6837","nonce":3969500353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/3810709211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":3810709211},{"type":"v1","addr":"172.21.15.162:6839","nonce":3810709211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.139:6835/1275838513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":1275838513},{"type":"v1","addr":"172.21.15.139:6835","nonce":1275838513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.139:6837/1885641254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1885641254},{"type":"v1","addr":"172.21.15.139:6837","nonce":1885641254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.139:6839/1001534713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1001534713},{"type":"v1","addr":"172.21.15.139:6839","nonce":1001534713}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24376,24406,24437,24461,24485]},"id":1}]} 2024-09-19T08:14:20.026 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2024-09-19T08:14:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:20.028+0000 7f7be8286640 1 -- 172.21.15.98:0/3678287857 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7bcc0a33b0 msgr2=0x7f7bcc0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:20.028+0000 7f7be8286640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7bcc0a33b0 0x7f7bcc0a5870 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc000c00 tx=0x7f7bdc032c70 comp rx=0 tx=0).stop 2024-09-19T08:14:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:20.028+0000 7f7be8286640 1 -- 172.21.15.98:0/3678287857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7be0100a80 msgr2=0x7f7be019a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:20.028+0000 7f7be8286640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7be0100a80 0x7f7be019a720 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400b4a0 tx=0x7f7bd400b960 comp rx=0 tx=0).stop 2024-09-19T08:14:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:20.028+0000 7f7be8286640 1 -- 172.21.15.98:0/3678287857 shutdown_connections 2024-09-19T08:14:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:20.028+0000 7f7be8286640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f7bcc0a33b0 0x7f7bcc0a5870 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:20.028+0000 7f7be8286640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7be0106a80 0x7f7be019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:20.028+0000 7f7be8286640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7be0101420 0x7f7be019ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:20.028+0000 7f7be8286640 1 --2- 172.21.15.98:0/3678287857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7be0100a80 0x7f7be019a720 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:20.028+0000 7f7be8286640 1 -- 172.21.15.98:0/3678287857 >> 172.21.15.98:0/3678287857 conn(0x7f7be00fc7a0 msgr2=0x7f7be010c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:20.028+0000 7f7be8286640 1 -- 172.21.15.98:0/3678287857 shutdown_connections 2024-09-19T08:14:20.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:20.028+0000 7f7be8286640 1 -- 172.21.15.98:0/3678287857 wait complete. 2024-09-19T08:14:20.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T08:14:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:14:21.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:20 smithi139 ceph-mon[34874]: pgmap v352: 97 pgs: 97 active+clean; 7.3 MiB data, 649 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 383 B/s wr, 13 op/s 2024-09-19T08:14:21.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:20 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3678287857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:14:21.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:20 smithi098 ceph-mon[31333]: pgmap v352: 97 pgs: 97 active+clean; 7.3 MiB data, 649 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 383 B/s wr, 13 op/s 2024-09-19T08:14:21.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:20 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3678287857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:14:21.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:20 smithi162 ceph-mon[35260]: pgmap v352: 97 pgs: 97 active+clean; 7.3 MiB data, 649 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 383 B/s wr, 13 op/s 2024-09-19T08:14:21.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:20 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3678287857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T08:14:21.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.567+0000 7f5234757640 1 -- 172.21.15.98:0/650097066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522c101de0 msgr2=0x7f522c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:21.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.567+0000 7f5234757640 1 --2- 172.21.15.98:0/650097066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522c101de0 0x7f522c1021c0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f5228009900 tx=0x7f522802a410 comp rx=0 tx=0).stop 2024-09-19T08:14:21.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.567+0000 7f5234757640 1 -- 172.21.15.98:0/650097066 shutdown_connections 2024-09-19T08:14:21.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.567+0000 7f5234757640 1 --2- 172.21.15.98:0/650097066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f522c106f10 0x7f522c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:21.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.567+0000 7f5234757640 1 --2- 172.21.15.98:0/650097066 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f522c102790 0x7f522c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:21.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.567+0000 7f5234757640 1 --2- 172.21.15.98:0/650097066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522c101de0 0x7f522c1021c0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:21.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.567+0000 7f5234757640 1 -- 172.21.15.98:0/650097066 >> 172.21.15.98:0/650097066 conn(0x7f522c0fd8c0 msgr2=0x7f522c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:21.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.567+0000 7f5234757640 1 -- 172.21.15.98:0/650097066 shutdown_connections 2024-09-19T08:14:21.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.567+0000 7f5234757640 1 -- 172.21.15.98:0/650097066 wait complete. 2024-09-19T08:14:21.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.568+0000 7f5234757640 1 Processor -- start 2024-09-19T08:14:21.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.568+0000 7f5234757640 1 -- start start 2024-09-19T08:14:21.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.568+0000 7f5234757640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522c101de0 0x7f522c19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:21.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.568+0000 7f5234757640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f522c102790 0x7f522c19ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:21.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.568+0000 7f5234757640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f522c106f10 0x7f522c19ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:21.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.568+0000 7f5234757640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f522c111f70 con 0x7f522c101de0 2024-09-19T08:14:21.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.568+0000 7f5234757640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f522c111df0 con 0x7f522c106f10 2024-09-19T08:14:21.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.568+0000 7f5234757640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f522c1120f0 con 0x7f522c102790 2024-09-19T08:14:21.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.568+0000 7f52324f3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522c101de0 0x7f522c19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:21.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.568+0000 7f52324f3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522c101de0 0x7f522c19a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36654/0 (socket says 172.21.15.98:36654) 2024-09-19T08:14:21.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.568+0000 7f52324f3640 1 -- 172.21.15.98:0/1077532428 learned_addr learned my addr 172.21.15.98:0/1077532428 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:14:21.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.568+0000 7f5231cf2640 1 --2- 172.21.15.98:0/1077532428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f522c102790 0x7f522c19ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:21.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.568+0000 7f5232cf4640 1 --2- 172.21.15.98:0/1077532428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f522c106f10 0x7f522c19ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:21.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.569+0000 7f52324f3640 1 -- 172.21.15.98:0/1077532428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f522c102790 msgr2=0x7f522c19ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:21.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.569+0000 7f52324f3640 1 --2- 172.21.15.98:0/1077532428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f522c102790 0x7f522c19ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:21.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.569+0000 7f52324f3640 1 -- 172.21.15.98:0/1077532428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f522c106f10 msgr2=0x7f522c19ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:21.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.569+0000 7f52324f3640 1 --2- 172.21.15.98:0/1077532428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f522c106f10 0x7f522c19ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:21.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.569+0000 7f52324f3640 1 -- 172.21.15.98:0/1077532428 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522c19f5e0 con 0x7f522c101de0 2024-09-19T08:14:21.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.569+0000 7f52324f3640 1 --2- 172.21.15.98:0/1077532428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522c101de0 0x7f522c19a630 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f5228006010 tx=0x7f52280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:21.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.569+0000 7f52237fe640 1 -- 172.21.15.98:0/1077532428 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522803f030 con 0x7f522c101de0 2024-09-19T08:14:21.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.569+0000 7f52237fe640 1 -- 172.21.15.98:0/1077532428 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5228003ce0 con 0x7f522c101de0 2024-09-19T08:14:21.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.569+0000 7f52237fe640 1 -- 172.21.15.98:0/1077532428 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5228035070 con 0x7f522c101de0 2024-09-19T08:14:21.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.569+0000 7f5234757640 1 -- 172.21.15.98:0/1077532428 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522c19f870 con 0x7f522c101de0 2024-09-19T08:14:21.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.569+0000 7f5234757640 1 -- 172.21.15.98:0/1077532428 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f522c1a71b0 con 0x7f522c101de0 2024-09-19T08:14:21.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.570+0000 7f5234757640 1 -- 172.21.15.98:0/1077532428 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f522c1021c0 con 0x7f522c101de0 2024-09-19T08:14:21.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.571+0000 7f52237fe640 1 -- 172.21.15.98:0/1077532428 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f5228042030 con 0x7f522c101de0 2024-09-19T08:14:21.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.572+0000 7f52237fe640 1 --2- 172.21.15.98:0/1077532428 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f52180a3430 0x7f52180a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:21.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.572+0000 7f52237fe640 1 -- 172.21.15.98:0/1077532428 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8521+0+0 (secure 0 0 0) 0x7f52280f0540 con 0x7f522c101de0 2024-09-19T08:14:21.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.572+0000 7f5231cf2640 1 --2- 172.21.15.98:0/1077532428 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f52180a3430 0x7f52180a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:21.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.572+0000 7f5231cf2640 1 --2- 172.21.15.98:0/1077532428 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f52180a3430 0x7f52180a58f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f522c0fddc0 tx=0x7f521c00b390 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:21.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.575+0000 7f52237fe640 1 -- 172.21.15.98:0/1077532428 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f52280bc990 con 0x7f522c101de0 2024-09-19T08:14:21.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.744+0000 7f5234757640 1 -- 172.21.15.98:0/1077532428 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f522c1a7380 con 0x7f522c101de0 2024-09-19T08:14:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.745+0000 7f52237fe640 1 -- 172.21.15.98:0/1077532428 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23218 (secure 0 0 0) 0x7f52280c0c90 con 0x7f522c101de0 2024-09-19T08:14:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-19T08:14:21.745 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"1a02525a-765c-11ef-bceb-c7b262605968","created":"2024-09-19T07:53:39.127398+0000","modified":"2024-09-19T08:14:15.883816+0000","last_up_change":"2024-09-19T08:00:22.467777+0000","last_in_change":"2024-09-19T07:59:58.538492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:56:43.773892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T08:01:07.693796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T08:01:09.518028+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":90,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7d01ce6-165d-493d-ae98-d8ad9e5d4134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6803","nonce":1719360533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6805","nonce":1719360533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6809","nonce":1719360533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1719360533},{"type":"v1","addr":"172.21.15.98:6807","nonce":1719360533}]},"public_addr":"172.21.15.98:6803/1719360533","cluster_addr":"172.21.15.98:6805/1719360533","heartbeat_back_addr":"172.21.15.98:6809/1719360533","heartbeat_front_addr":"172.21.15.98:6807/1719360533","state":["exists","up"]},{"osd":1,"uuid":"34724bd5-0738-4473-b087-c09031c1ee2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036962626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036962626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036962626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036962626},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036962626}]},"public_addr":"172.21.15.139:6801/4036962626","cluster_addr":"172.21.15.139:6803/4036962626","heartbeat_back_addr":"172.21.15.139:6807/4036962626","heartbeat_front_addr":"172.21.15.139:6805/4036962626","state":["exists","up"]},{"osd":2,"uuid":"16985ebe-0e32-42ee-bded-b7838e290afc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6801","nonce":2811412891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6803","nonce":2811412891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6807","nonce":2811412891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2811412891},{"type":"v1","addr":"172.21.15.162:6805","nonce":2811412891}]},"public_addr":"172.21.15.162:6801/2811412891","cluster_addr":"172.21.15.162:6803/2811412891","heartbeat_back_addr":"172.21.15.162:6807/2811412891","heartbeat_front_addr":"172.21.15.162:6805/2811412891","state":["exists","up"]},{"osd":3,"uuid":"b140e466-3119-49cb-aec2-ba6ca4d10d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6811","nonce":1383392014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6813","nonce":1383392014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6817","nonce":1383392014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1383392014},{"type":"v1","addr":"172.21.15.98:6815","nonce":1383392014}]},"public_addr":"172.21.15.98:6811/1383392014","cluster_addr":"172.21.15.98:6813/1383392014","heartbeat_back_addr":"172.21.15.98:6817/1383392014","heartbeat_front_addr":"172.21.15.98:6815/1383392014","state":["exists","up"]},{"osd":4,"uuid":"b4412813-df57-4ac6-8fb7-f57b69c74768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6809","nonce":116453520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6811","nonce":116453520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6815","nonce":116453520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":116453520},{"type":"v1","addr":"172.21.15.139:6813","nonce":116453520}]},"public_addr":"172.21.15.139:6809/116453520","cluster_addr":"172.21.15.139:6811/116453520","heartbeat_back_addr":"172.21.15.139:6815/116453520","heartbeat_front_addr":"172.21.15.139:6813/116453520","state":["exists","up"]},{"osd":5,"uuid":"d41fc66a-e8d1-458d-9a01-1a8aa4ec0e3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6809","nonce":3769530297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6811","nonce":3769530297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6815","nonce":3769530297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3769530297},{"type":"v1","addr":"172.21.15.162:6813","nonce":3769530297}]},"public_addr":"172.21.15.162:6809/3769530297","cluster_addr":"172.21.15.162:6811/3769530297","heartbeat_back_addr":"172.21.15.162:6815/3769530297","heartbeat_front_addr":"172.21.15.162:6813/3769530297","state":["exists","up"]},{"osd":6,"uuid":"ba1c1590-6aea-40dc-90c8-e21b88896909","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6819","nonce":692373267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6821","nonce":692373267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6825","nonce":692373267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":692373267},{"type":"v1","addr":"172.21.15.98:6823","nonce":692373267}]},"public_addr":"172.21.15.98:6819/692373267","cluster_addr":"172.21.15.98:6821/692373267","heartbeat_back_addr":"172.21.15.98:6825/692373267","heartbeat_front_addr":"172.21.15.98:6823/692373267","state":["exists","up"]},{"osd":7,"uuid":"eb9093ad-a068-4a63-ab87-0b50522cbc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6817","nonce":1963811650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6819","nonce":1963811650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6823","nonce":1963811650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1963811650},{"type":"v1","addr":"172.21.15.139:6821","nonce":1963811650}]},"public_addr":"172.21.15.139:6817/1963811650","cluster_addr":"172.21.15.139:6819/1963811650","heartbeat_back_addr":"172.21.15.139:6823/1963811650","heartbeat_front_addr":"172.21.15.139:6821/1963811650","state":["exists","up"]},{"osd":8,"uuid":"31ca330a-fc45-466d-81c2-ef4386b3cc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6817","nonce":3607966458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6819","nonce":3607966458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6823","nonce":3607966458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":3607966458},{"type":"v1","addr":"172.21.15.162:6821","nonce":3607966458}]},"public_addr":"172.21.15.162:6817/3607966458","cluster_addr":"172.21.15.162:6819/3607966458","heartbeat_back_addr":"172.21.15.162:6823/3607966458","heartbeat_front_addr":"172.21.15.162:6821/3607966458","state":["exists","up"]},{"osd":9,"uuid":"9c4f216f-aef1-4367-9c58-b913b0c4f7db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6827","nonce":2050391244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6829","nonce":2050391244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6833","nonce":2050391244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2050391244},{"type":"v1","addr":"172.21.15.98:6831","nonce":2050391244}]},"public_addr":"172.21.15.98:6827/2050391244","cluster_addr":"172.21.15.98:6829/2050391244","heartbeat_back_addr":"172.21.15.98:6833/2050391244","heartbeat_front_addr":"172.21.15.98:6831/2050391244","state":["exists","up"]},{"osd":10,"uuid":"03cc7df6-84ae-4d9a-a3d7-74c91e1744e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6825","nonce":2336761639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6827","nonce":2336761639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6831","nonce":2336761639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2336761639},{"type":"v1","addr":"172.21.15.139:6829","nonce":2336761639}]},"public_addr":"172.21.15.139:6825/2336761639","cluster_addr":"172.21.15.139:6827/2336761639","heartbeat_back_addr":"172.21.15.139:6831/2336761639","heartbeat_front_addr":"172.21.15.139:6829/2336761639","state":["exists","up"]},{"osd":11,"uuid":"29e95498-4952-46d4-8fa6-6d8e2b862111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6825","nonce":1231011554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6827","nonce":1231011554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6831","nonce":1231011554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1231011554},{"type":"v1","addr":"172.21.15.162:6829","nonce":1231011554}]},"public_addr":"172.21.15.162:6825/1231011554","cluster_addr":"172.21.15.162:6827/1231011554","heartbeat_back_addr":"172.21.15.162:6831/1231011554","heartbeat_front_addr":"172.21.15.162:6829/1231011554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:55:49.995244+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:15.655543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:56:40.331165+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:04.688881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:29.174966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:57:53.663161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:17.407551+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:58:42.584296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:06.317219+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:31.078625+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:59:55.689269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T08:00:19.659748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1},{"from":7,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1252058007":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/921776066":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/336479555":"2024-09-20T08:03:00.753287+0000","172.21.15.98:6800/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6800/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/1560445911":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/1497156658":"2024-09-20T07:54:02.418785+0000","172.21.15.98:6801/470997812":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/1238193420":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6800/470997812":"2024-09-20T08:03:00.753287+0000","172.21.15.98:0/3838590070":"2024-09-20T07:54:25.475454+0000","172.21.15.98:6801/16410793":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3842627255":"2024-09-20T07:54:25.475454+0000","172.21.15.98:0/3866031018":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3194864054":"2024-09-20T07:54:02.418785+0000","172.21.15.98:0/3067331617":"2024-09-20T08:03:00.753287+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T08:14:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.747+0000 7f5234757640 1 -- 172.21.15.98:0/1077532428 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f52180a3430 msgr2=0x7f52180a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.747+0000 7f5234757640 1 --2- 172.21.15.98:0/1077532428 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f52180a3430 0x7f52180a58f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f522c0fddc0 tx=0x7f521c00b390 comp rx=0 tx=0).stop 2024-09-19T08:14:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.747+0000 7f5234757640 1 -- 172.21.15.98:0/1077532428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522c101de0 msgr2=0x7f522c19a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:21.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.747+0000 7f5234757640 1 --2- 172.21.15.98:0/1077532428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522c101de0 0x7f522c19a630 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f5228006010 tx=0x7f52280042f0 comp rx=0 tx=0).stop 2024-09-19T08:14:21.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.747+0000 7f5234757640 1 -- 172.21.15.98:0/1077532428 shutdown_connections 2024-09-19T08:14:21.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.747+0000 7f5234757640 1 --2- 172.21.15.98:0/1077532428 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f52180a3430 0x7f52180a58f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:21.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.747+0000 7f5234757640 1 --2- 172.21.15.98:0/1077532428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f522c106f10 0x7f522c19ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:21.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.747+0000 7f5234757640 1 --2- 172.21.15.98:0/1077532428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f522c102790 0x7f522c19ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:21.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.747+0000 7f5234757640 1 --2- 172.21.15.98:0/1077532428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522c101de0 0x7f522c19a630 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:21.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.747+0000 7f5234757640 1 -- 172.21.15.98:0/1077532428 >> 172.21.15.98:0/1077532428 conn(0x7f522c0fd8c0 msgr2=0x7f522c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:21.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.748+0000 7f5234757640 1 -- 172.21.15.98:0/1077532428 shutdown_connections 2024-09-19T08:14:21.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:21.748+0000 7f5234757640 1 -- 172.21.15.98:0/1077532428 wait complete. 2024-09-19T08:14:22.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:21 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1077532428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:14:22.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:21 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1077532428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:14:22.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:21 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1077532428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T08:14:22.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs fail cephfs 2024-09-19T08:14:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:14:23.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:22 smithi139 ceph-mon[34874]: pgmap v353: 97 pgs: 97 active+clean; 7.3 MiB data, 649 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-09-19T08:14:23.180 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:22 smithi098 ceph-mon[31333]: pgmap v353: 97 pgs: 97 active+clean; 7.3 MiB data, 649 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-09-19T08:14:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.262+0000 7f3658e71640 1 -- 172.21.15.98:0/543382622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3654100a80 msgr2=0x7f3654100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.262+0000 7f3658e71640 1 --2- 172.21.15.98:0/543382622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3654100a80 0x7f3654100ee0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f363c009930 tx=0x7f363c02a420 comp rx=0 tx=0).stop 2024-09-19T08:14:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.262+0000 7f3658e71640 1 -- 172.21.15.98:0/543382622 shutdown_connections 2024-09-19T08:14:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.262+0000 7f3658e71640 1 --2- 172.21.15.98:0/543382622 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3654101420 0x7f365410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.262+0000 7f3658e71640 1 --2- 172.21.15.98:0/543382622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3654100a80 0x7f3654100ee0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.262+0000 7f3658e71640 1 --2- 172.21.15.98:0/543382622 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3654106a80 0x7f3654106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.262+0000 7f3658e71640 1 -- 172.21.15.98:0/543382622 >> 172.21.15.98:0/543382622 conn(0x7f36540fc7a0 msgr2=0x7f36540febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.262+0000 7f3658e71640 1 -- 172.21.15.98:0/543382622 shutdown_connections 2024-09-19T08:14:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.262+0000 7f3658e71640 1 -- 172.21.15.98:0/543382622 wait complete. 2024-09-19T08:14:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.263+0000 7f3658e71640 1 Processor -- start 2024-09-19T08:14:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.263+0000 7f3658e71640 1 -- start start 2024-09-19T08:14:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.263+0000 7f3658e71640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3654100a80 0x7f3654198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.263+0000 7f3658e71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3654101420 0x7f3654198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.263+0000 7f3658e71640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3654106a80 0x7f365419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.263+0000 7f3658e71640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3654110e80 con 0x7f3654101420 2024-09-19T08:14:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.263+0000 7f3658e71640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3654110d00 con 0x7f3654100a80 2024-09-19T08:14:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.263+0000 7f3658e71640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3654111000 con 0x7f3654106a80 2024-09-19T08:14:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.263+0000 7f3651d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3654101420 0x7f3654198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.263+0000 7f3651d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3654101420 0x7f3654198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36680/0 (socket says 172.21.15.98:36680) 2024-09-19T08:14:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.263+0000 7f3651d9b640 1 -- 172.21.15.98:0/2686473179 learned_addr learned my addr 172.21.15.98:0/2686473179 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:14:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.264+0000 7f365259c640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3654100a80 0x7f3654198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.264+0000 7f3652d9d640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3654106a80 0x7f365419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.264+0000 7f3651d9b640 1 -- 172.21.15.98:0/2686473179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3654106a80 msgr2=0x7f365419cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.264+0000 7f3651d9b640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3654106a80 0x7f365419cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.264+0000 7f3651d9b640 1 -- 172.21.15.98:0/2686473179 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3654100a80 msgr2=0x7f3654198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.264+0000 7f3651d9b640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3654100a80 0x7f3654198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.264+0000 7f3651d9b640 1 -- 172.21.15.98:0/2686473179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f365419d400 con 0x7f3654101420 2024-09-19T08:14:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.264+0000 7f3652d9d640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3654106a80 0x7f365419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:14:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.264+0000 7f3651d9b640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3654101420 0x7f3654198990 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f363c009230 tx=0x7f363c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.264+0000 7f364b7fe640 1 -- 172.21.15.98:0/2686473179 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c03f030 con 0x7f3654101420 2024-09-19T08:14:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.264+0000 7f364b7fe640 1 -- 172.21.15.98:0/2686473179 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f363c003ad0 con 0x7f3654101420 2024-09-19T08:14:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.264+0000 7f3658e71640 1 -- 172.21.15.98:0/2686473179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f365419d690 con 0x7f3654101420 2024-09-19T08:14:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.264+0000 7f364b7fe640 1 -- 172.21.15.98:0/2686473179 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c035070 con 0x7f3654101420 2024-09-19T08:14:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.264+0000 7f3658e71640 1 -- 172.21.15.98:0/2686473179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36541a4fd0 con 0x7f3654101420 2024-09-19T08:14:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.265+0000 7f3658e71640 1 -- 172.21.15.98:0/2686473179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3654100ee0 con 0x7f3654101420 2024-09-19T08:14:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.267+0000 7f364b7fe640 1 -- 172.21.15.98:0/2686473179 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7f363c042030 con 0x7f3654101420 2024-09-19T08:14:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.268+0000 7f364b7fe640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f36400a33b0 0x7f36400a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.268+0000 7f365259c640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f36400a33b0 0x7f36400a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.268+0000 7f364b7fe640 1 -- 172.21.15.98:0/2686473179 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8521+0+0 (secure 0 0 0) 0x7f363c0f0690 con 0x7f3654101420 2024-09-19T08:14:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.268+0000 7f365259c640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f36400a33b0 0x7f36400a5870 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3644005fd0 tx=0x7f364400b390 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:23.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.271+0000 7f364b7fe640 1 -- 172.21.15.98:0/2686473179 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f363c0bd070 con 0x7f3654101420 2024-09-19T08:14:23.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:22 smithi162 ceph-mon[35260]: pgmap v353: 97 pgs: 97 active+clean; 7.3 MiB data, 649 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-09-19T08:14:23.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.473+0000 7f3658e71640 1 -- 172.21.15.98:0/2686473179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f36541a51a0 con 0x7f3654101420 2024-09-19T08:14:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.475+0000 7f364b7fe640 1 -- 172.21.15.98:0/2686473179 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v33) ==== 161+0+0 (secure 0 0 0) 0x7f363c0bc970 con 0x7f3654101420 2024-09-19T08:14:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-19T08:14:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.478+0000 7f3658e71640 1 -- 172.21.15.98:0/2686473179 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f36400a33b0 msgr2=0x7f36400a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.478+0000 7f3658e71640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f36400a33b0 0x7f36400a5870 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3644005fd0 tx=0x7f364400b390 comp rx=0 tx=0).stop 2024-09-19T08:14:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.478+0000 7f3658e71640 1 -- 172.21.15.98:0/2686473179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3654101420 msgr2=0x7f3654198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.478+0000 7f3658e71640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3654101420 0x7f3654198990 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f363c009230 tx=0x7f363c004270 comp rx=0 tx=0).stop 2024-09-19T08:14:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.478+0000 7f3658e71640 1 -- 172.21.15.98:0/2686473179 shutdown_connections 2024-09-19T08:14:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.478+0000 7f3658e71640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7f36400a33b0 0x7f36400a5870 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.478+0000 7f3658e71640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3654106a80 0x7f365419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.478+0000 7f3658e71640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3654101420 0x7f3654198990 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.478+0000 7f3658e71640 1 --2- 172.21.15.98:0/2686473179 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3654100a80 0x7f3654198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.478+0000 7f3658e71640 1 -- 172.21.15.98:0/2686473179 >> 172.21.15.98:0/2686473179 conn(0x7f36540fc7a0 msgr2=0x7f3654104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.478+0000 7f3658e71640 1 -- 172.21.15.98:0/2686473179 shutdown_connections 2024-09-19T08:14:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:23.479+0000 7f3658e71640 1 -- 172.21.15.98:0/2686473179 wait complete. 2024-09-19T08:14:23.794 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a[31329]: 2024-09-19T08:14:23.474+0000 7f55f7d81640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-19T08:14:23.794 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:14:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-a[72355]: ignoring --setuser ceph since I am not root 2024-09-19T08:14:23.794 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:14:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-a[72355]: ignoring --setgroup ceph since I am not root 2024-09-19T08:14:23.794 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:14:23 smithi098 ceph-mds[72361]: -- 172.21.15.98:0/2606852027 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ed0467b860 con 0x55ed04659000 2024-09-19T08:14:23.795 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:14:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-a[72355]: starting mds.a at 2024-09-19T08:14:23.795 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:14:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-g[71945]: ignoring --setuser ceph since I am not root 2024-09-19T08:14:23.795 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:14:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-g[71945]: ignoring --setgroup ceph since I am not root 2024-09-19T08:14:23.795 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:14:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-g[71945]: starting mds.g at 2024-09-19T08:14:23.870 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-i[65901]: ignoring --setuser ceph since I am not root 2024-09-19T08:14:23.870 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-i[65901]: ignoring --setgroup ceph since I am not root 2024-09-19T08:14:23.870 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-mds[65905]: -- 172.21.15.162:0/2474353346 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561f0df81860 con 0x561f0df5f400 2024-09-19T08:14:23.870 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-i[65901]: starting mds.i at 2024-09-19T08:14:23.870 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-c[65598]: ignoring --setuser ceph since I am not root 2024-09-19T08:14:23.871 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-c[65598]: ignoring --setgroup ceph since I am not root 2024-09-19T08:14:23.871 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-mds[65602]: -- 172.21.15.162:0/479489751 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559835447860 con 0x559835425400 2024-09-19T08:14:23.871 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-c[65598]: starting mds.c at 2024-09-19T08:14:23.871 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-f[65294]: ignoring --setuser ceph since I am not root 2024-09-19T08:14:23.871 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-f[65294]: ignoring --setgroup ceph since I am not root 2024-09-19T08:14:23.871 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-mds[65298]: -- 172.21.15.162:0/1036881941 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562c1bcbd860 con 0x562c1bc9a800 2024-09-19T08:14:23.871 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-f[65294]: starting mds.f at 2024-09-19T08:14:23.871 INFO:journalctl@ceph.mds.l.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-l[64992]: ignoring --setuser ceph since I am not root 2024-09-19T08:14:23.872 INFO:journalctl@ceph.mds.l.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-l[64992]: ignoring --setgroup ceph since I am not root 2024-09-19T08:14:23.872 INFO:journalctl@ceph.mds.l.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-mds[64996]: -- 172.21.15.162:0/3891637753 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555c5486bc20 con 0x555c54848800 2024-09-19T08:14:23.872 INFO:journalctl@ceph.mds.l.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-l[64992]: starting mds.l at 2024-09-19T08:14:23.914 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-b[65169]: ignoring --setuser ceph since I am not root 2024-09-19T08:14:23.914 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-b[65169]: ignoring --setgroup ceph since I am not root 2024-09-19T08:14:23.914 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-mds[65173]: -- 172.21.15.139:0/3940689634 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5595a2f9f860 con 0x5595a2f7d400 2024-09-19T08:14:23.915 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-b[65169]: starting mds.b at 2024-09-19T08:14:23.915 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-h[64874]: ignoring --setuser ceph since I am not root 2024-09-19T08:14:23.915 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-h[64874]: ignoring --setgroup ceph since I am not root 2024-09-19T08:14:23.916 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-mds[64878]: -- 172.21.15.139:0/3709579941 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560755443860 con 0x560755420800 2024-09-19T08:14:23.916 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-h[64874]: starting mds.h at 2024-09-19T08:14:23.916 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-e[64574]: ignoring --setuser ceph since I am not root 2024-09-19T08:14:23.916 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-e[64574]: ignoring --setgroup ceph since I am not root 2024-09-19T08:14:23.917 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-mds[64578]: -- 172.21.15.139:0/1056115642 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561f5f81b860 con 0x561f5f7f8800 2024-09-19T08:14:23.917 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-e[64574]: starting mds.e at 2024-09-19T08:14:23.917 INFO:journalctl@ceph.mds.k.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-k[64265]: ignoring --setuser ceph since I am not root 2024-09-19T08:14:23.917 INFO:journalctl@ceph.mds.k.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-k[64265]: ignoring --setgroup ceph since I am not root 2024-09-19T08:14:23.918 INFO:journalctl@ceph.mds.k.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-mds[64269]: -- 172.21.15.139:0/3486420565 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5652188dd860 con 0x5652188ba800 2024-09-19T08:14:23.918 INFO:journalctl@ceph.mds.k.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-k[64265]: starting mds.k at 2024-09-19T08:14:24.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2686473179' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-19T08:14:24.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-mon[34874]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-19T08:14:24.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-mon[34874]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-19T08:14:24.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/2686473179' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-19T08:14:24.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-mon[34874]: osdmap e92: 12 total, 12 up, 12 in 2024-09-19T08:14:24.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:23 smithi139 ceph-mon[34874]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-19T08:14:24.178 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:23 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2686473179' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-19T08:14:24.178 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:23 smithi098 ceph-mon[31333]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-19T08:14:24.178 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:23 smithi098 ceph-mon[31333]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-19T08:14:24.178 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:23 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/2686473179' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-19T08:14:24.178 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:23 smithi098 ceph-mon[31333]: osdmap e92: 12 total, 12 up, 12 in 2024-09-19T08:14:24.178 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:23 smithi098 ceph-mon[31333]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-19T08:14:24.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-09-19T08:14:24.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2686473179' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-19T08:14:24.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-mon[35260]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-19T08:14:24.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-mon[35260]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-19T08:14:24.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/2686473179' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-19T08:14:24.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-mon[35260]: osdmap e92: 12 total, 12 up, 12 in 2024-09-19T08:14:24.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:23 smithi162 ceph-mon[35260]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-19T08:14:24.452 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:14:24.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.930+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1397172659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8106a80 msgr2=0x7fa6e8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.930+0000 7fa6ed31d640 1 --2- 172.21.15.98:0/1397172659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8106a80 0x7fa6e8106e60 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009930 tx=0x7fa6d802a420 comp rx=0 tx=0).stop 2024-09-19T08:14:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.930+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1397172659 shutdown_connections 2024-09-19T08:14:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.930+0000 7fa6ed31d640 1 --2- 172.21.15.98:0/1397172659 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6e8101420 0x7fa6e810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.930+0000 7fa6ed31d640 1 --2- 172.21.15.98:0/1397172659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6e8100a80 0x7fa6e8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.930+0000 7fa6ed31d640 1 --2- 172.21.15.98:0/1397172659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8106a80 0x7fa6e8106e60 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.930+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1397172659 >> 172.21.15.98:0/1397172659 conn(0x7fa6e80fc7a0 msgr2=0x7fa6e80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.930+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1397172659 shutdown_connections 2024-09-19T08:14:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.930+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1397172659 wait complete. 2024-09-19T08:14:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.930+0000 7fa6ed31d640 1 Processor -- start 2024-09-19T08:14:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6ed31d640 1 -- start start 2024-09-19T08:14:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6ed31d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6e8100a80 0x7fa6e8199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6ed31d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6e8101420 0x7fa6e8199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6ed31d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8106a80 0x7fa6e81a4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6ed31d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e8110e40 con 0x7fa6e8106a80 2024-09-19T08:14:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6ed31d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e8110cc0 con 0x7fa6e8101420 2024-09-19T08:14:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6ed31d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e8110fc0 con 0x7fa6e8100a80 2024-09-19T08:14:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6e77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8106a80 0x7fa6e81a4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6e6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6e8100a80 0x7fa6e8199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6e77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8106a80 0x7fa6e81a4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36730/0 (socket says 172.21.15.98:36730) 2024-09-19T08:14:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6e77fe640 1 -- 172.21.15.98:0/1130953635 learned_addr learned my addr 172.21.15.98:0/1130953635 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:14:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6dffff640 1 --2- 172.21.15.98:0/1130953635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6e8101420 0x7fa6e8199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6e77fe640 1 -- 172.21.15.98:0/1130953635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6e8100a80 msgr2=0x7fa6e8199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6e77fe640 1 --2- 172.21.15.98:0/1130953635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6e8100a80 0x7fa6e8199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6e77fe640 1 -- 172.21.15.98:0/1130953635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6e8101420 msgr2=0x7fa6e8199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6e77fe640 1 --2- 172.21.15.98:0/1130953635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6e8101420 0x7fa6e8199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6e77fe640 1 -- 172.21.15.98:0/1130953635 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e81a47c0 con 0x7fa6e8106a80 2024-09-19T08:14:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6e77fe640 1 --2- 172.21.15.98:0/1130953635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8106a80 0x7fa6e81a4280 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fa6e000cce0 tx=0x7fa6e000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.931+0000 7fa6e4ff9640 1 -- 172.21.15.98:0/1130953635 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e000f450 con 0x7fa6e8106a80 2024-09-19T08:14:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.932+0000 7fa6e4ff9640 1 -- 172.21.15.98:0/1130953635 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6e000fad0 con 0x7fa6e8106a80 2024-09-19T08:14:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.932+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1130953635 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6e81a4990 con 0x7fa6e8106a80 2024-09-19T08:14:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.932+0000 7fa6e4ff9640 1 -- 172.21.15.98:0/1130953635 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0018b60 con 0x7fa6e8106a80 2024-09-19T08:14:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.932+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1130953635 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6e81a4e50 con 0x7fa6e8106a80 2024-09-19T08:14:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.932+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1130953635 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6e8100ee0 con 0x7fa6e8106a80 2024-09-19T08:14:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.934+0000 7fa6e4ff9640 1 -- 172.21.15.98:0/1130953635 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fa6e000fc80 con 0x7fa6e8106a80 2024-09-19T08:14:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.934+0000 7fa6e4ff9640 1 --2- 172.21.15.98:0/1130953635 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa6d40a33b0 0x7fa6d40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.934+0000 7fa6e6ffd640 1 --2- 172.21.15.98:0/1130953635 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa6d40a33b0 0x7fa6d40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.934+0000 7fa6e4ff9640 1 -- 172.21.15.98:0/1130953635 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8907+0+0 (secure 0 0 0) 0x7fa6e00d02c0 con 0x7fa6e8106a80 2024-09-19T08:14:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.935+0000 7fa6e6ffd640 1 --2- 172.21.15.98:0/1130953635 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa6d40a33b0 0x7fa6d40a5870 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009900 tx=0x7fa6d80058d0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:24.939+0000 7fa6e4ff9640 1 -- 172.21.15.98:0/1130953635 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fa6e0014030 con 0x7fa6e8106a80 2024-09-19T08:14:25.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:25.142+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1130953635 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa6e81a5140 con 0x7fa6e8106a80 2024-09-19T08:14:25.142 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: pgmap v354: 97 pgs: 97 active+clean; 7.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 818 B/s wr, 11 op/s 2024-09-19T08:14:25.142 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.98:6838/520718679,v1:172.21.15.98:6839/520718679] up:boot 2024-09-19T08:14:25.142 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.98:6840/2107072562,v1:172.21.15.98:6841/2107072562] up:boot 2024-09-19T08:14:25.142 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.162:6833/3688255689,v1:172.21.15.162:6835/3688255689] up:boot 2024-09-19T08:14:25.143 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.139:6832/463596218,v1:172.21.15.139:6833/463596218] up:boot 2024-09-19T08:14:25.143 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.162:6832/707574014,v1:172.21.15.162:6834/707574014] up:boot 2024-09-19T08:14:25.143 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.139:6834/503648545,v1:172.21.15.139:6835/503648545] up:boot 2024-09-19T08:14:25.143 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.162:6836/1906622547,v1:172.21.15.162:6838/1906622547] up:boot 2024-09-19T08:14:25.143 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.162:6837/510066094,v1:172.21.15.162:6839/510066094] up:boot 2024-09-19T08:14:25.143 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.139:6836/3563506699,v1:172.21.15.139:6837/3563506699] up:boot 2024-09-19T08:14:25.143 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: mds.? [v2:172.21.15.139:6838/1126043774,v1:172.21.15.139:6839/1126043774] up:boot 2024-09-19T08:14:25.143 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-19T08:14:25.143 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-19T08:14:25.143 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-19T08:14:25.143 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-19T08:14:25.143 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-19T08:14:25.144 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-19T08:14:25.144 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-19T08:14:25.144 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-19T08:14:25.144 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-19T08:14:25.144 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-19T08:14:25.144 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:24 smithi098 ceph-mon[31333]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-19T08:14:25.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: pgmap v354: 97 pgs: 97 active+clean; 7.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 818 B/s wr, 11 op/s 2024-09-19T08:14:25.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.98:6838/520718679,v1:172.21.15.98:6839/520718679] up:boot 2024-09-19T08:14:25.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.98:6840/2107072562,v1:172.21.15.98:6841/2107072562] up:boot 2024-09-19T08:14:25.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.162:6833/3688255689,v1:172.21.15.162:6835/3688255689] up:boot 2024-09-19T08:14:25.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.139:6832/463596218,v1:172.21.15.139:6833/463596218] up:boot 2024-09-19T08:14:25.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.162:6832/707574014,v1:172.21.15.162:6834/707574014] up:boot 2024-09-19T08:14:25.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.139:6834/503648545,v1:172.21.15.139:6835/503648545] up:boot 2024-09-19T08:14:25.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.162:6836/1906622547,v1:172.21.15.162:6838/1906622547] up:boot 2024-09-19T08:14:25.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.162:6837/510066094,v1:172.21.15.162:6839/510066094] up:boot 2024-09-19T08:14:25.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.139:6836/3563506699,v1:172.21.15.139:6837/3563506699] up:boot 2024-09-19T08:14:25.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: mds.? [v2:172.21.15.139:6838/1126043774,v1:172.21.15.139:6839/1126043774] up:boot 2024-09-19T08:14:25.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-19T08:14:25.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-19T08:14:25.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-19T08:14:25.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-19T08:14:25.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-19T08:14:25.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-19T08:14:25.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-19T08:14:25.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-19T08:14:25.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-19T08:14:25.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-19T08:14:25.179 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:24 smithi139 ceph-mon[34874]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-19T08:14:25.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: pgmap v354: 97 pgs: 97 active+clean; 7.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 818 B/s wr, 11 op/s 2024-09-19T08:14:25.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.98:6838/520718679,v1:172.21.15.98:6839/520718679] up:boot 2024-09-19T08:14:25.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.98:6840/2107072562,v1:172.21.15.98:6841/2107072562] up:boot 2024-09-19T08:14:25.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.162:6833/3688255689,v1:172.21.15.162:6835/3688255689] up:boot 2024-09-19T08:14:25.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.139:6832/463596218,v1:172.21.15.139:6833/463596218] up:boot 2024-09-19T08:14:25.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.162:6832/707574014,v1:172.21.15.162:6834/707574014] up:boot 2024-09-19T08:14:25.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.139:6834/503648545,v1:172.21.15.139:6835/503648545] up:boot 2024-09-19T08:14:25.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.162:6836/1906622547,v1:172.21.15.162:6838/1906622547] up:boot 2024-09-19T08:14:25.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.162:6837/510066094,v1:172.21.15.162:6839/510066094] up:boot 2024-09-19T08:14:25.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.139:6836/3563506699,v1:172.21.15.139:6837/3563506699] up:boot 2024-09-19T08:14:25.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: mds.? [v2:172.21.15.139:6838/1126043774,v1:172.21.15.139:6839/1126043774] up:boot 2024-09-19T08:14:25.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-19T08:14:25.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-19T08:14:25.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-19T08:14:25.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-19T08:14:25.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-19T08:14:25.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-19T08:14:25.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-19T08:14:25.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-19T08:14:25.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-19T08:14:25.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-19T08:14:25.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:24 smithi162 ceph-mon[35260]: from='mgr.15252 172.21.15.98:0/3552295986' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-19T08:14:25.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:25.920+0000 7fa6e4ff9640 1 -- 172.21.15.98:0/1130953635 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) ==== 104+0+0 (secure 0 0 0) 0x7fa6e009cb20 con 0x7fa6e8106a80 2024-09-19T08:14:25.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:25.924+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1130953635 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa6d40a33b0 msgr2=0x7fa6d40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:25.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:25.924+0000 7fa6ed31d640 1 --2- 172.21.15.98:0/1130953635 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa6d40a33b0 0x7fa6d40a5870 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009900 tx=0x7fa6d80058d0 comp rx=0 tx=0).stop 2024-09-19T08:14:25.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:25.924+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1130953635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8106a80 msgr2=0x7fa6e81a4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:25.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:25.924+0000 7fa6ed31d640 1 --2- 172.21.15.98:0/1130953635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8106a80 0x7fa6e81a4280 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fa6e000cce0 tx=0x7fa6e000a620 comp rx=0 tx=0).stop 2024-09-19T08:14:25.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:25.924+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1130953635 shutdown_connections 2024-09-19T08:14:25.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:25.924+0000 7fa6ed31d640 1 --2- 172.21.15.98:0/1130953635 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fa6d40a33b0 0x7fa6d40a5870 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:25.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:25.924+0000 7fa6ed31d640 1 --2- 172.21.15.98:0/1130953635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8106a80 0x7fa6e81a4280 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:25.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:25.924+0000 7fa6ed31d640 1 --2- 172.21.15.98:0/1130953635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6e8101420 0x7fa6e8199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:25.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:25.924+0000 7fa6ed31d640 1 --2- 172.21.15.98:0/1130953635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6e8100a80 0x7fa6e8199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:25.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:25.924+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1130953635 >> 172.21.15.98:0/1130953635 conn(0x7fa6e80fc7a0 msgr2=0x7fa6e80fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:25.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:25.924+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1130953635 shutdown_connections 2024-09-19T08:14:25.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:25.924+0000 7fa6ed31d640 1 -- 172.21.15.98:0/1130953635 wait complete. 2024-09-19T08:14:26.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:25 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1130953635' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-19T08:14:26.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:25 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1130953635' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-19T08:14:26.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:25 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1130953635' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-19T08:14:26.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-19T08:14:26.879 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:14:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:26 smithi139 ceph-mon[34874]: pgmap v356: 97 pgs: 97 active+clean; 7.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 901 B/s wr, 12 op/s 2024-09-19T08:14:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:26 smithi139 ceph-mon[34874]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-19T08:14:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:26 smithi139 ceph-mon[34874]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-19T08:14:27.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:26 smithi139 ceph-mon[34874]: Cluster is now healthy 2024-09-19T08:14:27.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:26 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/1130953635' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-19T08:14:27.178 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:26 smithi139 ceph-mon[34874]: fsmap 12 up:standby 2024-09-19T08:14:27.178 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:26 smithi098 ceph-mon[31333]: pgmap v356: 97 pgs: 97 active+clean; 7.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 901 B/s wr, 12 op/s 2024-09-19T08:14:27.179 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:26 smithi098 ceph-mon[31333]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-19T08:14:27.179 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:26 smithi098 ceph-mon[31333]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-19T08:14:27.179 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:26 smithi098 ceph-mon[31333]: Cluster is now healthy 2024-09-19T08:14:27.179 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:26 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/1130953635' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-19T08:14:27.179 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:26 smithi098 ceph-mon[31333]: fsmap 12 up:standby 2024-09-19T08:14:27.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:26 smithi162 ceph-mon[35260]: pgmap v356: 97 pgs: 97 active+clean; 7.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 901 B/s wr, 12 op/s 2024-09-19T08:14:27.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:26 smithi162 ceph-mon[35260]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-19T08:14:27.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:26 smithi162 ceph-mon[35260]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-19T08:14:27.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:26 smithi162 ceph-mon[35260]: Cluster is now healthy 2024-09-19T08:14:27.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:26 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/1130953635' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-19T08:14:27.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:26 smithi162 ceph-mon[35260]: fsmap 12 up:standby 2024-09-19T08:14:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.376+0000 7fae56943640 1 -- 172.21.15.98:0/2877397667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae50103660 msgr2=0x7fae5010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.376+0000 7fae56943640 1 --2- 172.21.15.98:0/2877397667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae50103660 0x7fae5010bb50 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fae44009900 tx=0x7fae4402a480 comp rx=0 tx=0).stop 2024-09-19T08:14:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.376+0000 7fae56943640 1 -- 172.21.15.98:0/2877397667 shutdown_connections 2024-09-19T08:14:27.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.376+0000 7fae56943640 1 --2- 172.21.15.98:0/2877397667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae5010c170 0x7fae5010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.376+0000 7fae56943640 1 --2- 172.21.15.98:0/2877397667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae50103660 0x7fae5010bb50 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.376+0000 7fae56943640 1 --2- 172.21.15.98:0/2877397667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae50102d40 0x7fae50103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.376+0000 7fae56943640 1 -- 172.21.15.98:0/2877397667 >> 172.21.15.98:0/2877397667 conn(0x7fae500fc7a0 msgr2=0x7fae500febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.377+0000 7fae56943640 1 -- 172.21.15.98:0/2877397667 shutdown_connections 2024-09-19T08:14:27.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.377+0000 7fae56943640 1 -- 172.21.15.98:0/2877397667 wait complete. 2024-09-19T08:14:27.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.377+0000 7fae56943640 1 Processor -- start 2024-09-19T08:14:27.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.377+0000 7fae56943640 1 -- start start 2024-09-19T08:14:27.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.377+0000 7fae56943640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae50102d40 0x7fae501983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:27.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.377+0000 7fae56943640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae50103660 0x7fae50198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:27.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.377+0000 7fae56943640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae5010c170 0x7fae5019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:27.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae56943640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae50111f70 con 0x7fae5010c170 2024-09-19T08:14:27.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae56943640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fae50111df0 con 0x7fae50103660 2024-09-19T08:14:27.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae56943640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fae501120f0 con 0x7fae50102d40 2024-09-19T08:14:27.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae54ee0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae5010c170 0x7fae5019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:27.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae54ee0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae5010c170 0x7fae5019cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50938/0 (socket says 172.21.15.98:50938) 2024-09-19T08:14:27.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae54ee0640 1 -- 172.21.15.98:0/3700414968 learned_addr learned my addr 172.21.15.98:0/3700414968 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:14:27.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae4f7fe640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae50103660 0x7fae50198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:27.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae4ffff640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae50102d40 0x7fae501983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:27.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae54ee0640 1 -- 172.21.15.98:0/3700414968 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae50102d40 msgr2=0x7fae501983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:27.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae54ee0640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae50102d40 0x7fae501983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:27.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae54ee0640 1 -- 172.21.15.98:0/3700414968 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae50103660 msgr2=0x7fae50198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:27.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae54ee0640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae50103660 0x7fae50198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:27.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae54ee0640 1 -- 172.21.15.98:0/3700414968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5019d380 con 0x7fae5010c170 2024-09-19T08:14:27.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae4f7fe640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae50103660 0x7fae50198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:14:27.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae54ee0640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae5010c170 0x7fae5019cca0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fae4800eaa0 tx=0x7fae4800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:27.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae4d7fa640 1 -- 172.21.15.98:0/3700414968 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4800cdb0 con 0x7fae5010c170 2024-09-19T08:14:27.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae4d7fa640 1 -- 172.21.15.98:0/3700414968 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae48005020 con 0x7fae5010c170 2024-09-19T08:14:27.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae56943640 1 -- 172.21.15.98:0/3700414968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae5019d670 con 0x7fae5010c170 2024-09-19T08:14:27.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae4d7fa640 1 -- 172.21.15.98:0/3700414968 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae480116b0 con 0x7fae5010c170 2024-09-19T08:14:27.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.378+0000 7fae56943640 1 -- 172.21.15.98:0/3700414968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae501a4f50 con 0x7fae5010c170 2024-09-19T08:14:27.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.379+0000 7fae56943640 1 -- 172.21.15.98:0/3700414968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae50103120 con 0x7fae5010c170 2024-09-19T08:14:27.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.381+0000 7fae4d7fa640 1 -- 172.21.15.98:0/3700414968 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fae48005560 con 0x7fae5010c170 2024-09-19T08:14:27.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.381+0000 7fae4d7fa640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae380a3430 0x7fae380a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:27.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.381+0000 7fae4ffff640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae380a3430 0x7fae380a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:27.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.381+0000 7fae4d7fa640 1 -- 172.21.15.98:0/3700414968 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8907+0+0 (secure 0 0 0) 0x7fae480d0450 con 0x7fae5010c170 2024-09-19T08:14:27.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.382+0000 7fae4ffff640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae380a3430 0x7fae380a58f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fae4000ba40 tx=0x7fae40005cc0 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:27.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.385+0000 7fae4d7fa640 1 -- 172.21.15.98:0/3700414968 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fae48016030 con 0x7fae5010c170 2024-09-19T08:14:27.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.556+0000 7fae56943640 1 -- 172.21.15.98:0/3700414968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fae50106c50 con 0x7fae5010c170 2024-09-19T08:14:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.927+0000 7fae4d7fa640 1 -- 172.21.15.98:0/3700414968 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v93) ==== 181+0+0 (secure 0 0 0) 0x7fae4809cd30 con 0x7fae5010c170 2024-09-19T08:14:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' removed 2024-09-19T08:14:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.930+0000 7fae56943640 1 -- 172.21.15.98:0/3700414968 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae380a3430 msgr2=0x7fae380a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.930+0000 7fae56943640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae380a3430 0x7fae380a58f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fae4000ba40 tx=0x7fae40005cc0 comp rx=0 tx=0).stop 2024-09-19T08:14:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.930+0000 7fae56943640 1 -- 172.21.15.98:0/3700414968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae5010c170 msgr2=0x7fae5019cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.930+0000 7fae56943640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae5010c170 0x7fae5019cca0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fae4800eaa0 tx=0x7fae4800ef60 comp rx=0 tx=0).stop 2024-09-19T08:14:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.931+0000 7fae56943640 1 -- 172.21.15.98:0/3700414968 shutdown_connections 2024-09-19T08:14:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.931+0000 7fae56943640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fae380a3430 0x7fae380a58f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.931+0000 7fae56943640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae5010c170 0x7fae5019cca0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.931+0000 7fae56943640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae50103660 0x7fae50198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.931+0000 7fae56943640 1 --2- 172.21.15.98:0/3700414968 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae50102d40 0x7fae501983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.931+0000 7fae56943640 1 -- 172.21.15.98:0/3700414968 >> 172.21.15.98:0/3700414968 conn(0x7fae500fc7a0 msgr2=0x7fae500fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.931+0000 7fae56943640 1 -- 172.21.15.98:0/3700414968 shutdown_connections 2024-09-19T08:14:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:27.931+0000 7fae56943640 1 -- 172.21.15.98:0/3700414968 wait complete. 2024-09-19T08:14:28.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:27 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3700414968' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-19T08:14:28.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:27 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3700414968' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-19T08:14:28.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:27 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3700414968' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-19T08:14:28.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid 1a02525a-765c-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-19T08:14:28.938 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T08:14:29.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:28 smithi139 ceph-mon[34874]: pgmap v357: 97 pgs: 97 active+clean; 7.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 716 B/s wr, 8 op/s 2024-09-19T08:14:29.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:28 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/3700414968' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-19T08:14:29.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:28 smithi139 ceph-mon[34874]: osdmap e93: 12 total, 12 up, 12 in 2024-09-19T08:14:29.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:28 smithi098 ceph-mon[31333]: pgmap v357: 97 pgs: 97 active+clean; 7.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 716 B/s wr, 8 op/s 2024-09-19T08:14:29.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:28 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/3700414968' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-19T08:14:29.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:28 smithi098 ceph-mon[31333]: osdmap e93: 12 total, 12 up, 12 in 2024-09-19T08:14:29.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:28 smithi162 ceph-mon[35260]: pgmap v357: 97 pgs: 97 active+clean; 7.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 716 B/s wr, 8 op/s 2024-09-19T08:14:29.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:28 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/3700414968' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-19T08:14:29.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:28 smithi162 ceph-mon[35260]: osdmap e93: 12 total, 12 up, 12 in 2024-09-19T08:14:29.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.391+0000 7fbade4c9640 1 -- 172.21.15.98:0/1508822773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad81037a0 msgr2=0x7fbad810bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:29.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.391+0000 7fbade4c9640 1 --2- 172.21.15.98:0/1508822773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad81037a0 0x7fbad810bc00 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009900 tx=0x7fbac002a410 comp rx=0 tx=0).stop 2024-09-19T08:14:29.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.391+0000 7fbade4c9640 1 -- 172.21.15.98:0/1508822773 shutdown_connections 2024-09-19T08:14:29.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.391+0000 7fbade4c9640 1 --2- 172.21.15.98:0/1508822773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbad810c2f0 0x7fbad810e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:29.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.391+0000 7fbade4c9640 1 --2- 172.21.15.98:0/1508822773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad81037a0 0x7fbad810bc00 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:29.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.391+0000 7fbade4c9640 1 --2- 172.21.15.98:0/1508822773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbad8102df0 0x7fbad81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:29.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.391+0000 7fbade4c9640 1 -- 172.21.15.98:0/1508822773 >> 172.21.15.98:0/1508822773 conn(0x7fbad80fc7d0 msgr2=0x7fbad80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:29.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.391+0000 7fbade4c9640 1 -- 172.21.15.98:0/1508822773 shutdown_connections 2024-09-19T08:14:29.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.391+0000 7fbade4c9640 1 -- 172.21.15.98:0/1508822773 wait complete. 2024-09-19T08:14:29.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.391+0000 7fbade4c9640 1 Processor -- start 2024-09-19T08:14:29.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.391+0000 7fbade4c9640 1 -- start start 2024-09-19T08:14:29.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbade4c9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbad8102df0 0x7fbad8100a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:29.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbade4c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad81037a0 0x7fbad8100f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:29.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbade4c9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbad810c2f0 0x7fbad81014d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:29.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbade4c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbad8112130 con 0x7fbad81037a0 2024-09-19T08:14:29.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbade4c9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbad8111fb0 con 0x7fbad810c2f0 2024-09-19T08:14:29.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbade4c9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbad81122b0 con 0x7fbad8102df0 2024-09-19T08:14:29.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbad77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad81037a0 0x7fbad8100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:29.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbad77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad81037a0 0x7fbad8100f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50952/0 (socket says 172.21.15.98:50952) 2024-09-19T08:14:29.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbad77fe640 1 -- 172.21.15.98:0/416157958 learned_addr learned my addr 172.21.15.98:0/416157958 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-19T08:14:29.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbad7fff640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbad8102df0 0x7fbad8100a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:29.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbadca66640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbad810c2f0 0x7fbad81014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:29.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbad77fe640 1 -- 172.21.15.98:0/416157958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbad8102df0 msgr2=0x7fbad8100a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:29.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbad77fe640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbad8102df0 0x7fbad8100a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:29.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbad77fe640 1 -- 172.21.15.98:0/416157958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbad810c2f0 msgr2=0x7fbad81014d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:29.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbad77fe640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbad810c2f0 0x7fbad81014d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:29.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbad77fe640 1 -- 172.21.15.98:0/416157958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad81a6b30 con 0x7fbad81037a0 2024-09-19T08:14:29.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.392+0000 7fbadca66640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbad810c2f0 0x7fbad81014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T08:14:29.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.393+0000 7fbad77fe640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad81037a0 0x7fbad8100f90 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fbac0006010 tx=0x7fbac00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:29.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.393+0000 7fbad57fa640 1 -- 172.21.15.98:0/416157958 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac003f030 con 0x7fbad81037a0 2024-09-19T08:14:29.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.393+0000 7fbad57fa640 1 -- 172.21.15.98:0/416157958 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbac0003ce0 con 0x7fbad81037a0 2024-09-19T08:14:29.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.393+0000 7fbade4c9640 1 -- 172.21.15.98:0/416157958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad81a6dc0 con 0x7fbad81037a0 2024-09-19T08:14:29.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.393+0000 7fbad57fa640 1 -- 172.21.15.98:0/416157958 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0035070 con 0x7fbad81037a0 2024-09-19T08:14:29.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.393+0000 7fbade4c9640 1 -- 172.21.15.98:0/416157958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad81a7240 con 0x7fbad81037a0 2024-09-19T08:14:29.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.394+0000 7fbade4c9640 1 -- 172.21.15.98:0/416157958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbad81031d0 con 0x7fbad81037a0 2024-09-19T08:14:29.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.395+0000 7fbad57fa640 1 -- 172.21.15.98:0/416157958 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138486+0+0 (secure 0 0 0) 0x7fbac0042030 con 0x7fbad81037a0 2024-09-19T08:14:29.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.396+0000 7fbad57fa640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbac40a33b0 0x7fbac40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T08:14:29.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.396+0000 7fbad7fff640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbac40a33b0 0x7fbac40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T08:14:29.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.396+0000 7fbad57fa640 1 -- 172.21.15.98:0/416157958 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8345+0+0 (secure 0 0 0) 0x7fbac00f02c0 con 0x7fbad81037a0 2024-09-19T08:14:29.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.396+0000 7fbad7fff640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbac40a33b0 0x7fbac40a5870 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009890 tx=0x7fbac8008040 comp rx=0 tx=0).ready entity=mgr.15252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T08:14:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.400+0000 7fbad57fa640 1 -- 172.21.15.98:0/416157958 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fbac00bcd50 con 0x7fbad81037a0 2024-09-19T08:14:29.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.566+0000 7fbade4c9640 1 -- 172.21.15.98:0/416157958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbad81072c0 con 0x7fbad81037a0 2024-09-19T08:14:29.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.930+0000 7fbad57fa640 1 -- 172.21.15.98:0/416157958 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v94) ==== 169+0+0 (secure 0 0 0) 0x7fbac00bc650 con 0x7fbad81037a0 2024-09-19T08:14:29.930 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' removed 2024-09-19T08:14:29.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.933+0000 7fbade4c9640 1 -- 172.21.15.98:0/416157958 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbac40a33b0 msgr2=0x7fbac40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:29.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.933+0000 7fbade4c9640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbac40a33b0 0x7fbac40a5870 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009890 tx=0x7fbac8008040 comp rx=0 tx=0).stop 2024-09-19T08:14:29.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.933+0000 7fbade4c9640 1 -- 172.21.15.98:0/416157958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad81037a0 msgr2=0x7fbad8100f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T08:14:29.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.933+0000 7fbade4c9640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad81037a0 0x7fbad8100f90 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fbac0006010 tx=0x7fbac00042f0 comp rx=0 tx=0).stop 2024-09-19T08:14:29.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.933+0000 7fbade4c9640 1 -- 172.21.15.98:0/416157958 shutdown_connections 2024-09-19T08:14:29.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.933+0000 7fbade4c9640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.98:6800/1961446727,v1:172.21.15.98:6801/1961446727] conn(0x7fbac40a33b0 0x7fbac40a5870 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:29.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.933+0000 7fbade4c9640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbad810c2f0 0x7fbad81014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:29.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.933+0000 7fbade4c9640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad81037a0 0x7fbad8100f90 secure :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fbac0006010 tx=0x7fbac00042f0 comp rx=0 tx=0).stop 2024-09-19T08:14:29.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.933+0000 7fbade4c9640 1 --2- 172.21.15.98:0/416157958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbad8102df0 0x7fbad8100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T08:14:29.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.933+0000 7fbade4c9640 1 -- 172.21.15.98:0/416157958 >> 172.21.15.98:0/416157958 conn(0x7fbad80fc7d0 msgr2=0x7fbad80fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T08:14:29.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.935+0000 7fbade4c9640 1 -- 172.21.15.98:0/416157958 shutdown_connections 2024-09-19T08:14:29.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-19T08:14:29.935+0000 7fbade4c9640 1 -- 172.21.15.98:0/416157958 wait complete. 2024-09-19T08:14:30.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:29 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/416157958' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-19T08:14:30.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:29 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/416157958' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-19T08:14:30.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:29 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/416157958' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-19T08:14:30.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-19T08:14:30.575 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-19T08:14:30.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-19T08:14:30.642 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-19T08:14:30.642 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-19T08:14:30.658 DEBUG:teuthology.orchestra.run.smithi139:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-19T08:14:30.675 DEBUG:teuthology.orchestra.run.smithi162:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-19T08:14:30.700 INFO:tasks.cephadm:Stopping all daemons... 2024-09-19T08:14:30.700 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-19T08:14:30.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mon.a 2024-09-19T08:14:30.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:30 smithi098 systemd[1]: Stopping Ceph mon.a for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:14:31.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:30 smithi098 ceph-mon[31333]: pgmap v359: 65 pgs: 65 active+clean; 577 KiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:14:31.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:30 smithi098 ceph-mon[31333]: from='client.? 172.21.15.98:0/416157958' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-19T08:14:31.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:30 smithi098 ceph-mon[31333]: osdmap e94: 12 total, 12 up, 12 in 2024-09-19T08:14:31.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:31 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a[31329]: 2024-09-19T08:14:31.034+0000 7f55fd58c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:14:31.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:31 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a[31329]: 2024-09-19T08:14:31.034+0000 7f55fd58c640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-19T08:14:31.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:31 smithi098 podman[111647]: 2024-09-19 08:14:31.085631314 +0000 UTC m=+0.193572198 container died 44a037c430ef182d56e33e0bb4c723bc2fcb078a0e5746f498f035526d354886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-19T08:14:31.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:30 smithi162 ceph-mon[35260]: pgmap v359: 65 pgs: 65 active+clean; 577 KiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:14:31.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:30 smithi162 ceph-mon[35260]: from='client.? 172.21.15.98:0/416157958' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-19T08:14:31.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:30 smithi162 ceph-mon[35260]: osdmap e94: 12 total, 12 up, 12 in 2024-09-19T08:14:31.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:30 smithi139 ceph-mon[34874]: pgmap v359: 65 pgs: 65 active+clean; 577 KiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T08:14:31.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:30 smithi139 ceph-mon[34874]: from='client.? 172.21.15.98:0/416157958' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-19T08:14:31.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:30 smithi139 ceph-mon[34874]: osdmap e94: 12 total, 12 up, 12 in 2024-09-19T08:14:31.828 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:31 smithi098 podman[111647]: 2024-09-19 08:14:31.550846719 +0000 UTC m=+0.658787588 container cleanup 44a037c430ef182d56e33e0bb4c723bc2fcb078a0e5746f498f035526d354886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-19T08:14:31.828 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 19 08:14:31 smithi098 bash[111647]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-a 2024-09-19T08:14:31.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mon.a.service' 2024-09-19T08:14:32.035 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:14:32.036 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-19T08:14:32.036 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-19T08:14:32.036 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mon.b 2024-09-19T08:14:32.361 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:32 smithi139 systemd[1]: Stopping Ceph mon.b for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:14:32.661 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:32 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-b[34870]: 2024-09-19T08:14:32.361+0000 7fc1dd995640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:14:32.661 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:32 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-b[34870]: 2024-09-19T08:14:32.361+0000 7fc1dd995640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-19T08:14:32.661 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:32 smithi139 podman[70526]: 2024-09-19 08:14:32.382062535 +0000 UTC m=+0.144377675 container died f39ae79d7bf0919182030b304f2d4208ebe58b4e69db97e62349ef440bc4b26c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=squid-2f599a4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-19T08:14:32.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:32 smithi139 podman[70526]: 2024-09-19 08:14:32.761049686 +0000 UTC m=+0.523364823 container cleanup f39ae79d7bf0919182030b304f2d4208ebe58b4e69db97e62349ef440bc4b26c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-b, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-19T08:14:32.927 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 19 08:14:32 smithi139 bash[70526]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-b 2024-09-19T08:14:33.186 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mon.b.service' 2024-09-19T08:14:33.241 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:14:33.242 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-19T08:14:33.242 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-19T08:14:33.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mon.c 2024-09-19T08:14:33.557 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:33 smithi162 systemd[1]: Stopping Ceph mon.c for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:14:33.869 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:33 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-c[35256]: 2024-09-19T08:14:33.556+0000 7f524613e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:14:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:33 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-c[35256]: 2024-09-19T08:14:33.556+0000 7f524613e640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-19T08:14:33.870 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:33 smithi162 podman[71413]: 2024-09-19 08:14:33.761540301 +0000 UTC m=+0.320028022 container died 500b1c467f929a37e7f3fa083bd2c476bd8dd7588bbeaea2c706ce9f6b545123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-c, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-19T08:14:34.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:34 smithi162 podman[71413]: 2024-09-19 08:14:34.106996689 +0000 UTC m=+0.665484409 container cleanup 500b1c467f929a37e7f3fa083bd2c476bd8dd7588bbeaea2c706ce9f6b545123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-19T08:14:34.370 INFO:journalctl@ceph.mon.c.smithi162.stdout:Sep 19 08:14:34 smithi162 bash[71413]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mon-c 2024-09-19T08:14:34.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mon.c.service' 2024-09-19T08:14:34.577 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:14:34.578 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-19T08:14:34.578 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-19T08:14:34.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mgr.x 2024-09-19T08:14:34.897 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:14:34 smithi098 systemd[1]: Stopping Ceph mgr.x for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:14:35.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:14:34 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x[31559]: 2024-09-19T08:14:34.896+0000 7f868770e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:14:35.182 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:14:34 smithi098 podman[111767]: 2024-09-19 08:14:34.927996587 +0000 UTC m=+0.152205224 container died 9a5fff8af5c4db892866a3a9fce9927bc0ef8e07b39b6dbab212d665658112f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-19T08:14:35.557 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:14:35 smithi098 podman[111767]: 2024-09-19 08:14:35.289433678 +0000 UTC m=+0.513642310 container cleanup 9a5fff8af5c4db892866a3a9fce9927bc0ef8e07b39b6dbab212d665658112f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x, RELEASE=squid-2f599a4, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-19T08:14:35.557 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Sep 19 08:14:35 smithi098 bash[111767]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-x 2024-09-19T08:14:35.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-19T08:14:35.720 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:14:35.721 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-19T08:14:35.721 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-19T08:14:35.721 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mgr.y 2024-09-19T08:14:36.069 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:14:35 smithi139 systemd[1]: Stopping Ceph mgr.y for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:14:36.357 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:14:36 smithi139 podman[70651]: 2024-09-19 08:14:36.069743794 +0000 UTC m=+0.153303938 container died dcdc610cf1f4703ebd0ebb3faa815a9c0b88ff5eb98dc8317aeebdad612973f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-2f599a4, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-19T08:14:36.677 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:14:36 smithi139 podman[70651]: 2024-09-19 08:14:36.457261702 +0000 UTC m=+0.540821845 container cleanup dcdc610cf1f4703ebd0ebb3faa815a9c0b88ff5eb98dc8317aeebdad612973f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:14:36.677 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Sep 19 08:14:36 smithi139 bash[70651]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-y 2024-09-19T08:14:36.875 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-19T08:14:36.909 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:14:36.909 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-19T08:14:36.909 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-09-19T08:14:36.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mgr.z 2024-09-19T08:14:37.239 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:14:36 smithi162 systemd[1]: Stopping Ceph mgr.z for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:14:37.494 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:14:37 smithi162 podman[71532]: 2024-09-19 08:14:37.240258966 +0000 UTC m=+0.131187701 container died cf1f1c2f4f8eb6792292902fe23e0a57a48761e094d4aed064e727fb8082f199 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:14:37.844 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:14:37 smithi162 podman[71532]: 2024-09-19 08:14:37.610803941 +0000 UTC m=+0.501732675 container cleanup cf1f1c2f4f8eb6792292902fe23e0a57a48761e094d4aed064e727fb8082f199 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-19T08:14:37.845 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Sep 19 08:14:37 smithi162 bash[71532]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mgr-z 2024-09-19T08:14:37.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mgr.z.service' 2024-09-19T08:14:38.028 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:14:38.028 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-09-19T08:14:38.028 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-19T08:14:38.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.0 2024-09-19T08:14:38.310 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 08:14:38 smithi098 systemd[1]: Stopping Ceph osd.0 for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:14:38.682 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 08:14:38 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0[45294]: 2024-09-19T08:14:38.310+0000 7f7b3eaa5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:14:38.682 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 08:14:38 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0[45294]: 2024-09-19T08:14:38.310+0000 7f7b3eaa5640 -1 osd.0 94 *** Got signal Terminated *** 2024-09-19T08:14:38.682 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 08:14:38 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0[45294]: 2024-09-19T08:14:38.310+0000 7f7b3eaa5640 -1 osd.0 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-19T08:14:44.020 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 08:14:43 smithi098 podman[111888]: 2024-09-19 08:14:43.709993091 +0000 UTC m=+5.494220153 container died 128437643d5de22f56b41440dcb8b898908b670c68a2b3f358d413d31ae3d0ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux ) 2024-09-19T08:14:44.361 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 08:14:44 smithi098 podman[111888]: 2024-09-19 08:14:44.112245037 +0000 UTC m=+5.896472093 container cleanup 128437643d5de22f56b41440dcb8b898908b670c68a2b3f358d413d31ae3d0ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-19T08:14:44.361 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 08:14:44 smithi098 bash[111888]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0 2024-09-19T08:14:44.672 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 08:14:44 smithi098 podman[111907]: 2024-09-19 08:14:44.362037526 +0000 UTC m=+0.646833630 container remove 128437643d5de22f56b41440dcb8b898908b670c68a2b3f358d413d31ae3d0ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0, org.label-schema.vendor=CentOS, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-19T08:14:44.932 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 08:14:44 smithi098 podman[111992]: 2024-09-19 08:14:44.575579284 +0000 UTC m=+0.015867484 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:14:44.932 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 08:14:44 smithi098 podman[111992]: 2024-09-19 08:14:44.690326388 +0000 UTC m=+0.130614587 container create af79448fc9fb80170f81b52bdd9a12f7fdbe31966c131af86d6ec7cd708d2e75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=) 2024-09-19T08:14:44.932 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 08:14:44 smithi098 podman[111992]: 2024-09-19 08:14:44.917878642 +0000 UTC m=+0.358166843 container init af79448fc9fb80170f81b52bdd9a12f7fdbe31966c131af86d6ec7cd708d2e75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-2f599a4, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-19T08:14:44.932 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 08:14:44 smithi098 podman[111992]: 2024-09-19 08:14:44.92118815 +0000 UTC m=+0.361476351 container start af79448fc9fb80170f81b52bdd9a12f7fdbe31966c131af86d6ec7cd708d2e75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4) 2024-09-19T08:14:45.395 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 08:14:45 smithi098 podman[111992]: 2024-09-19 08:14:45.043065853 +0000 UTC m=+0.483354064 container attach af79448fc9fb80170f81b52bdd9a12f7fdbe31966c131af86d6ec7cd708d2e75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-19T08:14:45.395 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 19 08:14:45 smithi098 podman[111992]: 2024-09-19 08:14:45.107548818 +0000 UTC m=+0.547837026 container died af79448fc9fb80170f81b52bdd9a12f7fdbe31966c131af86d6ec7cd708d2e75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=squid-2f599a4, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-19T08:14:45.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.0.service' 2024-09-19T08:14:45.727 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:14:45.727 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-19T08:14:45.727 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-19T08:14:45.727 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.1 2024-09-19T08:14:45.927 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 08:14:45 smithi139 systemd[1]: Stopping Ceph osd.1 for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:14:46.427 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 08:14:46 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1[39414]: 2024-09-19T08:14:46.052+0000 7f6ccf022640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:14:46.427 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 08:14:46 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1[39414]: 2024-09-19T08:14:46.052+0000 7f6ccf022640 -1 osd.1 94 *** Got signal Terminated *** 2024-09-19T08:14:46.427 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 08:14:46 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1[39414]: 2024-09-19T08:14:46.052+0000 7f6ccf022640 -1 osd.1 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-19T08:14:51.729 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 08:14:51 smithi139 podman[70786]: 2024-09-19 08:14:51.428574419 +0000 UTC m=+5.488251176 container died 1ff1cc20f0b99519ca6a9778f483bba63d8267055bfa9f06b13b98c96ac33b54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, ceph=True) 2024-09-19T08:14:52.095 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 08:14:51 smithi139 podman[70786]: 2024-09-19 08:14:51.81933978 +0000 UTC m=+5.879016525 container cleanup 1ff1cc20f0b99519ca6a9778f483bba63d8267055bfa9f06b13b98c96ac33b54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-19T08:14:52.096 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 08:14:51 smithi139 bash[70786]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1 2024-09-19T08:14:52.403 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 08:14:52 smithi139 podman[70804]: 2024-09-19 08:14:52.095902693 +0000 UTC m=+0.667729860 container remove 1ff1cc20f0b99519ca6a9778f483bba63d8267055bfa9f06b13b98c96ac33b54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-19T08:14:52.673 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 08:14:52 smithi139 podman[70894]: 2024-09-19 08:14:52.306840205 +0000 UTC m=+0.015437381 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:14:52.673 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 08:14:52 smithi139 podman[70894]: 2024-09-19 08:14:52.45509888 +0000 UTC m=+0.163696036 container create 1e4e836eb13d7b8bb25abac0ee6f1a683326bec5fb51d5ce9a6029f80ddac051 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-19T08:14:52.673 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 08:14:52 smithi139 podman[70894]: 2024-09-19 08:14:52.673927675 +0000 UTC m=+0.382524834 container init 1e4e836eb13d7b8bb25abac0ee6f1a683326bec5fb51d5ce9a6029f80ddac051 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-2f599a4, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-19T08:14:52.927 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 08:14:52 smithi139 podman[70894]: 2024-09-19 08:14:52.677411143 +0000 UTC m=+0.386008307 container start 1e4e836eb13d7b8bb25abac0ee6f1a683326bec5fb51d5ce9a6029f80ddac051 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:14:52.927 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 08:14:52 smithi139 podman[70894]: 2024-09-19 08:14:52.74890461 +0000 UTC m=+0.457501772 container attach 1e4e836eb13d7b8bb25abac0ee6f1a683326bec5fb51d5ce9a6029f80ddac051 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-19T08:14:52.928 INFO:journalctl@ceph.osd.1.smithi139.stdout:Sep 19 08:14:52 smithi139 podman[70894]: 2024-09-19 08:14:52.868757911 +0000 UTC m=+0.577355084 container died 1e4e836eb13d7b8bb25abac0ee6f1a683326bec5fb51d5ce9a6029f80ddac051 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, RELEASE=squid-2f599a4, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-19T08:14:53.499 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.1.service' 2024-09-19T08:14:53.539 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:14:53.540 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-19T08:14:53.540 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-19T08:14:53.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.2 2024-09-19T08:14:53.830 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 08:14:53 smithi162 systemd[1]: Stopping Ceph osd.2 for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:14:54.120 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 08:14:53 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2[40270]: 2024-09-19T08:14:53.828+0000 7f8a90e14640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:14:54.120 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 08:14:53 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2[40270]: 2024-09-19T08:14:53.828+0000 7f8a90e14640 -1 osd.2 94 *** Got signal Terminated *** 2024-09-19T08:14:54.120 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 08:14:53 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2[40270]: 2024-09-19T08:14:53.828+0000 7f8a90e14640 -1 osd.2 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-19T08:14:59.596 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 08:14:59 smithi162 podman[71658]: 2024-09-19 08:14:59.326123723 +0000 UTC m=+5.588947338 container died 67d77ca425f01352a3d0f5f58f853f11e76dfd6e032d62bc49b56ca44256bd53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-19T08:14:59.870 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 08:14:59 smithi162 podman[71658]: 2024-09-19 08:14:59.687543657 +0000 UTC m=+5.950367271 container cleanup 67d77ca425f01352a3d0f5f58f853f11e76dfd6e032d62bc49b56ca44256bd53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-2f599a4, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-19T08:14:59.870 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 08:14:59 smithi162 bash[71658]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2 2024-09-19T08:15:00.305 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 08:14:59 smithi162 podman[71687]: 2024-09-19 08:14:59.972308347 +0000 UTC m=+0.639396065 container remove 67d77ca425f01352a3d0f5f58f853f11e76dfd6e032d62bc49b56ca44256bd53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-19T08:15:00.620 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 08:15:00 smithi162 podman[71779]: 2024-09-19 08:15:00.306430366 +0000 UTC m=+0.098822150 container create c89328e40db22db89e32c66ecd0fd7ac8d5ba06171a238f6f82532abfeedf46e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-19T08:15:00.620 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 08:15:00 smithi162 podman[71779]: 2024-09-19 08:15:00.223832093 +0000 UTC m=+0.016223882 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:15:00.620 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 08:15:00 smithi162 podman[71779]: 2024-09-19 08:15:00.55047133 +0000 UTC m=+0.342863105 container init c89328e40db22db89e32c66ecd0fd7ac8d5ba06171a238f6f82532abfeedf46e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-2f599a4, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True) 2024-09-19T08:15:00.620 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 08:15:00 smithi162 podman[71779]: 2024-09-19 08:15:00.553865156 +0000 UTC m=+0.346256933 container start c89328e40db22db89e32c66ecd0fd7ac8d5ba06171a238f6f82532abfeedf46e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-19T08:15:01.036 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 08:15:00 smithi162 podman[71779]: 2024-09-19 08:15:00.625390443 +0000 UTC m=+0.417782221 container attach c89328e40db22db89e32c66ecd0fd7ac8d5ba06171a238f6f82532abfeedf46e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-19T08:15:01.036 INFO:journalctl@ceph.osd.2.smithi162.stdout:Sep 19 08:15:00 smithi162 podman[71779]: 2024-09-19 08:15:00.749283042 +0000 UTC m=+0.541674836 container died c89328e40db22db89e32c66ecd0fd7ac8d5ba06171a238f6f82532abfeedf46e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-2-deactivate, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-19T08:15:01.333 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.2.service' 2024-09-19T08:15:01.374 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:15:01.374 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-19T08:15:01.374 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-19T08:15:01.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.3 2024-09-19T08:15:01.682 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 08:15:01 smithi098 systemd[1]: Stopping Ceph osd.3 for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:15:02.182 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 08:15:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3[51957]: 2024-09-19T08:15:01.698+0000 7ffa356a3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:15:02.182 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 08:15:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3[51957]: 2024-09-19T08:15:01.698+0000 7ffa356a3640 -1 osd.3 94 *** Got signal Terminated *** 2024-09-19T08:15:02.182 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 08:15:01 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3[51957]: 2024-09-19T08:15:01.698+0000 7ffa356a3640 -1 osd.3 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-19T08:15:04.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:04 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:04.464+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:05.370 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:05 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:05.093+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:05.681 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:05 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:05.261+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:05.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:05 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:05.418+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:06.369 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:06 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:06.046+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:06.619 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:06 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:06.369+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:06.682 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:06 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:06.262+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:07.369 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:07 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:07.063+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:07.384 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:07 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:07.261+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:07.384 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 08:15:07 smithi098 podman[112097]: 2024-09-19 08:15:07.134192272 +0000 UTC m=+5.565290192 container died 914ef2286c2e9f1cbf325e5812dda5054da1c2ae1c6a10e8ad1691f0b31133e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-19T08:15:07.681 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 08:15:07 smithi098 podman[112097]: 2024-09-19 08:15:07.56652203 +0000 UTC m=+5.997619940 container cleanup 914ef2286c2e9f1cbf325e5812dda5054da1c2ae1c6a10e8ad1691f0b31133e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-19T08:15:07.681 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 08:15:07 smithi098 bash[112097]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3 2024-09-19T08:15:07.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:07 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:07.411+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:08.112 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 08:15:07 smithi098 podman[112113]: 2024-09-19 08:15:07.793057564 +0000 UTC m=+0.651038821 container remove 914ef2286c2e9f1cbf325e5812dda5054da1c2ae1c6a10e8ad1691f0b31133e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:15:08.112 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:07 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:07.997+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:08.362 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:08 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:08.257+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:08.362 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 08:15:08 smithi098 podman[112201]: 2024-09-19 08:15:08.015224958 +0000 UTC m=+0.015990264 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:15:08.363 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 08:15:08 smithi098 podman[112201]: 2024-09-19 08:15:08.127165103 +0000 UTC m=+0.127930399 container create b43babbd4528f66ff02d9dcb187a24520163842dbd4cf1ac99a9a4d0b603e8e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-19T08:15:08.369 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:08 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:08.096+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:08.682 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 08:15:08 smithi098 podman[112201]: 2024-09-19 08:15:08.363445204 +0000 UTC m=+0.364210504 container init b43babbd4528f66ff02d9dcb187a24520163842dbd4cf1ac99a9a4d0b603e8e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-2f599a4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-19T08:15:08.682 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 08:15:08 smithi098 podman[112201]: 2024-09-19 08:15:08.367500564 +0000 UTC m=+0.368265868 container start b43babbd4528f66ff02d9dcb187a24520163842dbd4cf1ac99a9a4d0b603e8e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-19T08:15:08.682 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 08:15:08 smithi098 podman[112201]: 2024-09-19 08:15:08.45495161 +0000 UTC m=+0.455716933 container attach b43babbd4528f66ff02d9dcb187a24520163842dbd4cf1ac99a9a4d0b603e8e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-2f599a4) 2024-09-19T08:15:08.682 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 19 08:15:08 smithi098 podman[112201]: 2024-09-19 08:15:08.553614112 +0000 UTC m=+0.554379409 container died b43babbd4528f66ff02d9dcb187a24520163842dbd4cf1ac99a9a4d0b603e8e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-19T08:15:08.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:08 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:08.374+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:08.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:08 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:08.508+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:42.323468+0000 front 2024-09-19T08:14:42.323596+0000 (oldest deadline 2024-09-19T08:15:08.223556+0000) 2024-09-19T08:15:09.149 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:09 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:09.039+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:09.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.3.service' 2024-09-19T08:15:09.207 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:15:09.208 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-19T08:15:09.208 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-19T08:15:09.208 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.4 2024-09-19T08:15:09.349 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:09 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:09.066+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:09.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:09 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:09.288+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:09.512 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 08:15:09 smithi139 systemd[1]: Stopping Ceph osd.4 for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:15:09.619 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:09 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:09.349+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:09.620 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:09 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:09.536+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:42.323468+0000 front 2024-09-19T08:14:42.323596+0000 (oldest deadline 2024-09-19T08:15:08.223556+0000) 2024-09-19T08:15:09.927 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 08:15:09 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4[45921]: 2024-09-19T08:15:09.512+0000 7f766d8c2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:15:09.927 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 08:15:09 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4[45921]: 2024-09-19T08:15:09.512+0000 7f766d8c2640 -1 osd.4 94 *** Got signal Terminated *** 2024-09-19T08:15:09.927 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 08:15:09 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4[45921]: 2024-09-19T08:15:09.512+0000 7f766d8c2640 -1 osd.4 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-19T08:15:10.369 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:10 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:10.326+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:10.370 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:10 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:10.113+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:10.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:10 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:10.093+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:10.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:10 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:10.069+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:10.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:10 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:10.245+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:10.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:10 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:10.577+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:42.323468+0000 front 2024-09-19T08:14:42.323596+0000 (oldest deadline 2024-09-19T08:15:08.223556+0000) 2024-09-19T08:15:11.370 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:11 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:11.089+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:11.370 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:11 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:11.318+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:11.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:11 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:11.108+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:11.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:11 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:11.050+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:11.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:11 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:11.279+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:11.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:11 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:11.587+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:42.323468+0000 front 2024-09-19T08:14:42.323596+0000 (oldest deadline 2024-09-19T08:15:08.223556+0000) 2024-09-19T08:15:12.370 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:12 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:12.113+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:12.370 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:12 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:12.295+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:12.370 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:12 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:12.295+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:12.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:12 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:12.073+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:12.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:12 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:12.308+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:12.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:12 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:12.065+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:12.869 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:12 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:12.559+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:42.323468+0000 front 2024-09-19T08:14:42.323596+0000 (oldest deadline 2024-09-19T08:15:08.223556+0000) 2024-09-19T08:15:13.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:13 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:13.081+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:13.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:13 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:13.081+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:13.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:13 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:13.282+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:13.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:13 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:13.032+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:13.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:13 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:13.032+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:13.547 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:13 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:13.145+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:13.547 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:13 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:13.333+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:13.547 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:13 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:13.333+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:13.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:13 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:13.547+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:42.323468+0000 front 2024-09-19T08:14:42.323596+0000 (oldest deadline 2024-09-19T08:15:08.223556+0000) 2024-09-19T08:15:14.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:14 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:14.070+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:14.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:14 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:14.056+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:14.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:14 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:14.056+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:14.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:14 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:14.257+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:14.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:14 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:14.034+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:14.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:14 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:14.034+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:14.515 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:14 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:14.126+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:14.516 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:14 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:14.362+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:14.516 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:14 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:14.362+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:14.869 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:14 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:14.516+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:42.323468+0000 front 2024-09-19T08:14:42.323596+0000 (oldest deadline 2024-09-19T08:15:08.223556+0000) 2024-09-19T08:15:15.288 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 08:15:15 smithi139 podman[70998]: 2024-09-19 08:15:15.016628088 +0000 UTC m=+5.610427404 container died 3ea83d98aff62987a1a6d3f2e69dbdf95277d99f712fd68fd9d93b603a23eb88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4, maintainer=Guillaume Abrioux , RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True) 2024-09-19T08:15:15.288 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:15 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:15.068+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:15.288 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:15 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:15.073+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:15.289 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:15 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:15.073+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:15.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:15 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:15.216+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:15.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:15 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:15.051+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:15.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:15 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:15.051+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:15.488 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:15 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:15.150+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:15.488 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:15 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:15.386+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:15.488 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:15 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:15.386+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:15.621 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 08:15:15 smithi139 podman[70998]: 2024-09-19 08:15:15.395600887 +0000 UTC m=+5.989400199 container cleanup 3ea83d98aff62987a1a6d3f2e69dbdf95277d99f712fd68fd9d93b603a23eb88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-19T08:15:15.621 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 08:15:15 smithi139 bash[70998]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4 2024-09-19T08:15:15.873 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:15 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:15.488+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:42.323468+0000 front 2024-09-19T08:14:42.323596+0000 (oldest deadline 2024-09-19T08:15:08.223556+0000) 2024-09-19T08:15:15.927 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 08:15:15 smithi139 podman[71012]: 2024-09-19 08:15:15.621543026 +0000 UTC m=+0.601334872 container remove 3ea83d98aff62987a1a6d3f2e69dbdf95277d99f712fd68fd9d93b603a23eb88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-2f599a4) 2024-09-19T08:15:16.232 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:16 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:16.049+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:16.232 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:16 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:16.110+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:16.233 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:16 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:16.110+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:16.233 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 08:15:15 smithi139 podman[71104]: 2024-09-19 08:15:15.955644238 +0000 UTC m=+0.111594726 container create 74c3b74d9bd2f3e2b607c76eac6723121d361ac31c3aa6ac96fa74fd931215ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-2f599a4) 2024-09-19T08:15:16.233 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 08:15:15 smithi139 podman[71104]: 2024-09-19 08:15:15.859713599 +0000 UTC m=+0.015664092 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:15:16.233 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 08:15:16 smithi139 podman[71104]: 2024-09-19 08:15:16.149709076 +0000 UTC m=+0.305659561 container init 74c3b74d9bd2f3e2b607c76eac6723121d361ac31c3aa6ac96fa74fd931215ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-19T08:15:16.233 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 08:15:16 smithi139 podman[71104]: 2024-09-19 08:15:16.152973869 +0000 UTC m=+0.308924355 container start 74c3b74d9bd2f3e2b607c76eac6723121d361ac31c3aa6ac96fa74fd931215ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:15:16.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:16 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:16.011+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:16.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:16 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:16.011+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:16.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:16 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:16.190+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:16.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:16 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:16.190+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:16.476 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:16 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:16.191+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:16.476 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:16 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:16.349+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:16.477 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:16 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:16.349+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:16.628 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 08:15:16 smithi139 podman[71104]: 2024-09-19 08:15:16.233033933 +0000 UTC m=+0.388984427 container attach 74c3b74d9bd2f3e2b607c76eac6723121d361ac31c3aa6ac96fa74fd931215ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-19T08:15:16.628 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 19 08:15:16 smithi139 podman[71104]: 2024-09-19 08:15:16.334581084 +0000 UTC m=+0.490531576 container died 74c3b74d9bd2f3e2b607c76eac6723121d361ac31c3aa6ac96fa74fd931215ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-19T08:15:16.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:16 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:16.476+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:42.323468+0000 front 2024-09-19T08:14:42.323596+0000 (oldest deadline 2024-09-19T08:15:08.223556+0000) 2024-09-19T08:15:16.904 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.4.service' 2024-09-19T08:15:16.942 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:15:16.943 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-19T08:15:16.943 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-19T08:15:16.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.5 2024-09-19T08:15:17.177 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:17 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:17.055+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:17.177 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:17 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:17.095+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:17.177 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:17 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:17.095+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:17.254 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:16 smithi162 systemd[1]: Stopping Ceph osd.5 for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:15:17.255 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:17 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:17.190+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:17.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:17 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:17.183+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:17.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:17 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:17.183+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:17.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:17 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:17.013+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:17.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:17 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:17.013+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:17.506 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:17 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:17.376+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:17.506 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:17 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:17.376+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:17.506 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:17 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:17.254+0000 7f6c0575e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:15:17.506 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:17 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:17.254+0000 7f6c0575e640 -1 osd.5 94 *** Got signal Terminated *** 2024-09-19T08:15:17.506 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:17 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:17.254+0000 7f6c0575e640 -1 osd.5 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-19T08:15:17.507 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:17 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:17.506+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:42.323468+0000 front 2024-09-19T08:14:42.323596+0000 (oldest deadline 2024-09-19T08:15:08.223556+0000) 2024-09-19T08:15:17.507 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:17 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:17.506+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:48.224168+0000 front 2024-09-19T08:14:48.224223+0000 (oldest deadline 2024-09-19T08:15:17.024519+0000) 2024-09-19T08:15:18.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:18 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:18.030+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:18.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:18 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:18.068+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:18.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:18 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:18.068+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:18.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:18 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:18.007+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:18.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:18 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:18.007+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:18.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:18 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:18.154+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:18.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:18 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:18.154+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:18.529 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:18 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:18.184+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:18.530 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:18 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:18.418+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:18.530 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:18 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:18.418+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:18.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:18 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:18.529+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:42.323468+0000 front 2024-09-19T08:14:42.323596+0000 (oldest deadline 2024-09-19T08:15:08.223556+0000) 2024-09-19T08:15:18.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:18 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:18.529+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:48.224168+0000 front 2024-09-19T08:14:48.224223+0000 (oldest deadline 2024-09-19T08:15:17.024519+0000) 2024-09-19T08:15:19.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:19 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:19.069+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:19.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:19 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:19.099+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:19.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:19 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:19.099+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:19.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:19 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:19.038+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:19.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:19 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:19.038+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:19.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:19 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:19.112+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:19.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:19 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:19.112+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:19.460 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:19 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:19.169+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:19.869 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:19 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:19.460+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:19.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:19 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:19.460+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:19.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:19 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:19.506+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:42.323468+0000 front 2024-09-19T08:14:42.323596+0000 (oldest deadline 2024-09-19T08:15:08.223556+0000) 2024-09-19T08:15:19.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:19 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:19.506+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:48.224168+0000 front 2024-09-19T08:14:48.224223+0000 (oldest deadline 2024-09-19T08:15:17.024519+0000) 2024-09-19T08:15:19.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:19 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:19.506+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.024954+0000 front 2024-09-19T08:14:57.024979+0000 (oldest deadline 2024-09-19T08:15:19.325065+0000) 2024-09-19T08:15:20.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:20 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:20.080+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:20.428 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:20 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:20.080+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:20.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:20 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:20.104+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:20.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:20 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:20.104+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:20.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:19 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:19.991+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:20.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:19 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:19.991+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:20.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:20 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:20.095+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:20.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:20 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:20.095+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:20.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:20 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:20.095+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:20.464 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:20 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:20.192+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:20.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:20 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:20.464+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:42.323468+0000 front 2024-09-19T08:14:42.323596+0000 (oldest deadline 2024-09-19T08:15:08.223556+0000) 2024-09-19T08:15:20.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:20 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:20.464+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:48.224168+0000 front 2024-09-19T08:14:48.224223+0000 (oldest deadline 2024-09-19T08:15:17.024519+0000) 2024-09-19T08:15:20.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:20 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:20.464+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.024954+0000 front 2024-09-19T08:14:57.024979+0000 (oldest deadline 2024-09-19T08:15:19.325065+0000) 2024-09-19T08:15:20.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:20 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:20.479+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:20.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:20 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:20.479+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:20.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:20 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:20.479+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:21.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:21 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:21.071+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:21.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:21 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:21.071+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:21.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:21 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:21.062+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:21.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:21 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:21.062+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:21.429 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:21 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:21.158+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:21.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:20 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:20.953+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:21.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:20 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:20.953+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:21.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:21 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:21.097+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:21.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:21 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:21.097+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:21.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:21 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:21.097+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:21.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:21 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:21.428+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:42.323468+0000 front 2024-09-19T08:14:42.323596+0000 (oldest deadline 2024-09-19T08:15:08.223556+0000) 2024-09-19T08:15:21.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:21 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:21.428+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:48.224168+0000 front 2024-09-19T08:14:48.224223+0000 (oldest deadline 2024-09-19T08:15:17.024519+0000) 2024-09-19T08:15:21.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:21 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5[46923]: 2024-09-19T08:15:21.428+0000 7f6c01d63640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.024954+0000 front 2024-09-19T08:14:57.024979+0000 (oldest deadline 2024-09-19T08:15:19.325065+0000) 2024-09-19T08:15:21.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:21 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:21.504+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:21.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:21 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:21.504+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:21.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:21 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:21.504+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:22.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:22 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:22.079+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:22.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:22 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:22.079+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:22.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:22 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:22.028+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:22.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:22 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:22.028+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:22.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:22 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:22.028+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:22.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:21 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:21.950+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:22.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:21 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:21.950+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:22.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:21 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:21.950+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.959644+0000 front 2024-09-19T08:14:57.959520+0000 (oldest deadline 2024-09-19T08:15:21.459392+0000) 2024-09-19T08:15:22.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:22 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:22.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:22.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:22 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:22.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:22.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:22 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:22.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:22.507 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:22 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:22.125+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:22.870 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:22 smithi162 podman[71881]: 2024-09-19 08:15:22.737711064 +0000 UTC m=+5.592741774 container died f5628ebfd6be67761e2b6bd5fc700f5ada101390d437f5869cfdece8a41890b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5, RELEASE=squid-2f599a4, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-19T08:15:22.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:22 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:22.508+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:22.871 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:22 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:22.508+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:22.871 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:22 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:22.508+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:23.263 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:23.116+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:23.263 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:23 smithi162 podman[71881]: 2024-09-19 08:15:23.062249923 +0000 UTC m=+5.917280626 container cleanup f5628ebfd6be67761e2b6bd5fc700f5ada101390d437f5869cfdece8a41890b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-19T08:15:23.263 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:23 smithi162 bash[71881]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5 2024-09-19T08:15:23.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:23.091+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:23.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:23.091+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:23.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:23.026+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:23.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:23.026+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:23.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:23.026+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:23.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:22 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:22.941+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:23.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:22 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:22.942+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:23.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:22 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:22.942+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.959644+0000 front 2024-09-19T08:14:57.959520+0000 (oldest deadline 2024-09-19T08:15:21.459392+0000) 2024-09-19T08:15:23.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:23.102+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:23.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:23.102+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:23.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:23.102+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:23.620 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:23 smithi162 podman[71895]: 2024-09-19 08:15:23.263815433 +0000 UTC m=+0.526146062 container remove f5628ebfd6be67761e2b6bd5fc700f5ada101390d437f5869cfdece8a41890b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5, ceph=True, RELEASE=squid-2f599a4, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-19T08:15:23.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:23.475+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:23.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:23.475+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:23.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:23 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:23.475+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:23.950 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:23 smithi162 podman[71984]: 2024-09-19 08:15:23.585490674 +0000 UTC m=+0.015936455 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:15:23.950 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:23 smithi162 podman[71984]: 2024-09-19 08:15:23.714881318 +0000 UTC m=+0.145327092 container create d4d2fa75404216fcbe79edbaaff24e7543da3393879707927dd95f3a2502ca54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-19T08:15:23.950 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:23 smithi162 podman[71984]: 2024-09-19 08:15:23.950957754 +0000 UTC m=+0.381403529 container init d4d2fa75404216fcbe79edbaaff24e7543da3393879707927dd95f3a2502ca54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=squid-2f599a4, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-19T08:15:24.370 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:23 smithi162 podman[71984]: 2024-09-19 08:15:23.954317 +0000 UTC m=+0.384762775 container start d4d2fa75404216fcbe79edbaaff24e7543da3393879707927dd95f3a2502ca54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-19T08:15:24.370 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:24 smithi162 podman[71984]: 2024-09-19 08:15:24.025701348 +0000 UTC m=+0.456147126 container attach d4d2fa75404216fcbe79edbaaff24e7543da3393879707927dd95f3a2502ca54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=squid-2f599a4) 2024-09-19T08:15:24.370 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:24 smithi162 podman[71984]: 2024-09-19 08:15:24.146608568 +0000 UTC m=+0.577054343 container died d4d2fa75404216fcbe79edbaaff24e7543da3393879707927dd95f3a2502ca54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-19T08:15:24.371 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:24 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:24.111+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:24.371 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:24 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:24.111+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:24.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:24 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:24.128+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:24.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:24 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:24.128+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:24.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:23.983+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:24.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:23.983+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:24.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:23 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:23.983+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:24.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:23.956+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:24.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:23.956+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:24.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:23 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:23.956+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.959644+0000 front 2024-09-19T08:14:57.959520+0000 (oldest deadline 2024-09-19T08:15:21.459392+0000) 2024-09-19T08:15:24.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:24.092+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:24.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:24.092+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:24.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:24.092+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:24.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.5.service' 2024-09-19T08:15:24.716 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:24 smithi162 podman[72007]: 2024-09-19 08:15:24.671080428 +0000 UTC m=+0.515360292 container remove d4d2fa75404216fcbe79edbaaff24e7543da3393879707927dd95f3a2502ca54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-2f599a4, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-19T08:15:24.716 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:24 smithi162 systemd[1]: ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.5.service: Deactivated successfully. 2024-09-19T08:15:24.716 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:24 smithi162 systemd[1]: Stopped Ceph osd.5 for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:15:24.716 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 19 08:15:24 smithi162 systemd[1]: ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.5.service: Consumed 35.007s CPU time. 2024-09-19T08:15:24.716 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:24 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:24.456+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:24.716 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:24 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:24.456+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:24.716 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:24 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:24.456+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:24.732 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:15:24.732 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-19T08:15:24.733 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-19T08:15:24.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.6 2024-09-19T08:15:25.044 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:24 smithi098 systemd[1]: Stopping Ceph osd.6 for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:15:25.046 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:24.927+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:25.046 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:24.927+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:25.046 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:24.927+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.959644+0000 front 2024-09-19T08:14:57.959520+0000 (oldest deadline 2024-09-19T08:15:21.459392+0000) 2024-09-19T08:15:25.046 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:25 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:25.044+0000 7f827c3ee640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:15:25.046 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:25 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:25.044+0000 7f827c3ee640 -1 osd.6 94 *** Got signal Terminated *** 2024-09-19T08:15:25.046 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:25 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:25.044+0000 7f827c3ee640 -1 osd.6 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-19T08:15:25.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:25 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:25.130+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:25.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:25 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:25.130+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:25.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:25 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:25.018+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:25.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:25 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:25.018+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:25.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:25 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:25.018+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:25.430 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:25 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:25.140+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:25.430 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:25 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:25.140+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:25.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:25 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:25.076+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:25.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:25 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:25.076+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:25.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:25 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:25.076+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:25.869 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:25 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:25.431+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:25.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:25 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:25.431+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:25.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:25 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:25.431+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:26.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:26 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:26.178+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:26.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:26 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:26.178+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:26.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:25 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:25.998+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:26.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:25 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:25.999+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:26.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:25 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:25.999+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:26.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:25 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:25.936+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:26.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:25 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:25.936+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:26.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:25 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:25.936+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.959644+0000 front 2024-09-19T08:14:57.959520+0000 (oldest deadline 2024-09-19T08:15:21.459392+0000) 2024-09-19T08:15:26.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:26.117+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:26.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:26.117+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:26.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:26.117+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:26.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:26 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:26.151+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:26.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:26 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:26.151+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:26.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:26 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:26.386+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:26.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:26 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:26.386+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:26.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:26 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:26.386+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:27.182 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:27 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:27.090+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:27.182 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:27 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:27.090+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:27.182 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:27 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:27.090+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:27.182 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:26.909+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:27.182 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:26.909+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:27.182 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:26 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:26.909+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.959644+0000 front 2024-09-19T08:14:57.959520+0000 (oldest deadline 2024-09-19T08:15:21.459392+0000) 2024-09-19T08:15:27.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:27.175+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:27.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:27.175+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:27.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:27.010+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:27.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:27.010+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:27.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:27 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:27.010+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:27.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:27 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:27.201+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:27.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:27 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:27.201+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:27.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:27 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:27.344+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:27.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:27 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:27.344+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:27.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:27 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:27.344+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:28.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:28 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:28.164+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:28.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:28 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:28.164+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:28.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:28 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:28.060+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:28.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:28 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:28.060+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:28.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:28 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:28.060+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:28.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:27 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:27.938+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:28.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:27 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:27.938+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:28.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:27 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:27.938+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.959644+0000 front 2024-09-19T08:14:57.959520+0000 (oldest deadline 2024-09-19T08:15:21.459392+0000) 2024-09-19T08:15:28.432 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:27 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:27.938+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.160172+0000 front 2024-09-19T08:15:03.160062+0000 (oldest deadline 2024-09-19T08:15:27.260212+0000) 2024-09-19T08:15:28.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:28 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:28.110+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:28.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:28 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:28.110+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:28.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:28 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:28.110+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:28.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:28 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:28.355+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:28.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:28 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:28.355+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:28.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:28 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:28.355+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:28.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:28 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:28.235+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:28.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:28 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:28.235+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:28.621 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:28 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:28.235+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:29.181 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:28 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:28.920+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:29.182 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:28 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:28.920+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:29.182 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:28 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:28.920+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.959644+0000 front 2024-09-19T08:14:57.959520+0000 (oldest deadline 2024-09-19T08:15:21.459392+0000) 2024-09-19T08:15:29.182 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:28 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:28.920+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.160172+0000 front 2024-09-19T08:15:03.160062+0000 (oldest deadline 2024-09-19T08:15:27.260212+0000) 2024-09-19T08:15:29.182 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:29 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:29.068+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:29.182 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:29 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:29.068+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:29.182 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:29 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:29.068+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:29.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:29 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:29.201+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:29.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:29 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:29.201+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:29.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:29 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:29.032+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:29.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:29 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:29.032+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:29.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:29 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:29.032+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:29.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:29 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:29.332+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:29.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:29 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:29.332+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:29.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:29 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:29.332+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:29.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:29 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:29.247+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:29.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:29 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:29.247+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:29.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:29 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:29.247+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:30.182 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:30 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:30.070+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:30.182 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:30 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:30.070+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:30.182 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:30 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:30.070+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:30.182 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:29 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:29.914+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.457995+0000 front 2024-09-19T08:14:41.458134+0000 (oldest deadline 2024-09-19T08:15:07.358187+0000) 2024-09-19T08:15:30.182 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:29 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:29.914+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:47.358862+0000 front 2024-09-19T08:14:47.358941+0000 (oldest deadline 2024-09-19T08:15:12.658622+0000) 2024-09-19T08:15:30.183 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:29 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:29.914+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.959644+0000 front 2024-09-19T08:14:57.959520+0000 (oldest deadline 2024-09-19T08:15:21.459392+0000) 2024-09-19T08:15:30.183 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:29 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6[58447]: 2024-09-19T08:15:29.914+0000 7f82749eb640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.160172+0000 front 2024-09-19T08:15:03.160062+0000 (oldest deadline 2024-09-19T08:15:27.260212+0000) 2024-09-19T08:15:30.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:30 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:30.240+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:30.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:30 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:30.240+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:30.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:30 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:30.240+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:04.569211+0000 front 2024-09-19T08:15:04.569122+0000 (oldest deadline 2024-09-19T08:15:29.269000+0000) 2024-09-19T08:15:30.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:29 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:29.996+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:30.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:29 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:29.996+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:30.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:29 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:29.996+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:30.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:30 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:30.228+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:30.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:30 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:30.228+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:30.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:30 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:30.228+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:30.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:30 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:30.301+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:30.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:30 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:30.301+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:30.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:30 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:30.301+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:30.754 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:30 smithi098 podman[112303]: 2024-09-19 08:15:30.466618105 +0000 UTC m=+5.536258955 container died ab8411c2601ce30615cdcfd06eecb50e555fbd5939ad1b45410592f74c012e12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-19T08:15:31.077 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:30 smithi098 podman[112303]: 2024-09-19 08:15:30.920155302 +0000 UTC m=+5.989796152 container cleanup ab8411c2601ce30615cdcfd06eecb50e555fbd5939ad1b45410592f74c012e12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-19T08:15:31.077 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:30 smithi098 bash[112303]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6 2024-09-19T08:15:31.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:31 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:31.191+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:31.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:31 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:31.191+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:31.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:31 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:31.191+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:04.569211+0000 front 2024-09-19T08:15:04.569122+0000 (oldest deadline 2024-09-19T08:15:29.269000+0000) 2024-09-19T08:15:31.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:30 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:30.968+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:31.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:30 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:30.968+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:31.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:30 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:30.968+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:31.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:30 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:30.968+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:31.431 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:31 smithi098 podman[112321]: 2024-09-19 08:15:31.196644961 +0000 UTC m=+0.723302611 container remove ab8411c2601ce30615cdcfd06eecb50e555fbd5939ad1b45410592f74c012e12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-19T08:15:31.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:31 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:31.077+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:31.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:31 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:31.077+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:31.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:31 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:31.077+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:31.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:31 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:31.077+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:31.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:31 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:31.245+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:31.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:31 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:31.245+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:31.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:31 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:31.245+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:31.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:31 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:31.348+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:31.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:31 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:31.348+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:31.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:31 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:31.348+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:31.824 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:31 smithi098 podman[112413]: 2024-09-19 08:15:31.448736789 +0000 UTC m=+0.015764806 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:15:31.824 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:31 smithi098 podman[112413]: 2024-09-19 08:15:31.572681838 +0000 UTC m=+0.139709888 container create 01bb9df836e46596c7db5fb8781d59d1c9e7aef66a0d0616e6b10d003d072221 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=squid-2f599a4, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-19T08:15:31.824 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:31 smithi098 podman[112413]: 2024-09-19 08:15:31.825293047 +0000 UTC m=+0.392321057 container init 01bb9df836e46596c7db5fb8781d59d1c9e7aef66a0d0616e6b10d003d072221 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-19T08:15:32.083 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:31 smithi098 podman[112413]: 2024-09-19 08:15:31.828628039 +0000 UTC m=+0.395656045 container start 01bb9df836e46596c7db5fb8781d59d1c9e7aef66a0d0616e6b10d003d072221 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True) 2024-09-19T08:15:32.084 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:31 smithi098 podman[112413]: 2024-09-19 08:15:31.908448935 +0000 UTC m=+0.475476963 container attach 01bb9df836e46596c7db5fb8781d59d1c9e7aef66a0d0616e6b10d003d072221 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-19T08:15:32.084 INFO:journalctl@ceph.osd.6.smithi098.stdout:Sep 19 08:15:32 smithi098 podman[112413]: 2024-09-19 08:15:32.013178022 +0000 UTC m=+0.580206036 container died 01bb9df836e46596c7db5fb8781d59d1c9e7aef66a0d0616e6b10d003d072221 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, ceph=True) 2024-09-19T08:15:32.177 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:31 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:31.920+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:32.177 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:31 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:31.920+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:32.177 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:31 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:31.920+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:32.177 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:31 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:31.920+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:32.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:32 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:32.083+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:32.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:32 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:32.083+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:32.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:32 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:32.083+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:32.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:32 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:32.083+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:32.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.6.service' 2024-09-19T08:15:32.619 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:32 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:32.326+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:32.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:32 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:32.326+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:32.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:32 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:32.326+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:32.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:32 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:32.226+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:32.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:32 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:32.226+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:32.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:32 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:32.226+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:32.638 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:15:32.639 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-19T08:15:32.639 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-19T08:15:32.639 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.7 2024-09-19T08:15:32.667 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:32 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:32.235+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:32.667 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:32 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:32.235+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:32.667 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:32 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:32.235+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:04.569211+0000 front 2024-09-19T08:15:04.569122+0000 (oldest deadline 2024-09-19T08:15:29.269000+0000) 2024-09-19T08:15:32.927 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:32 smithi139 systemd[1]: Stopping Ceph osd.7 for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:15:33.261 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:32 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:32.929+0000 7f3303a01640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:15:33.262 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:32 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:32.929+0000 7f3303a01640 -1 osd.7 94 *** Got signal Terminated *** 2024-09-19T08:15:33.262 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:32 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:32.929+0000 7f3303a01640 -1 osd.7 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-19T08:15:33.262 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:32 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:32.954+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:33.262 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:32 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:32.954+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:33.262 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:32 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:32.954+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:33.262 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:32 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:32.954+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:33.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:33 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:33.042+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:33.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:33 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:33.042+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:33.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:33 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:33.042+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:33.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:33 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:33.042+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:33.619 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:33 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:33.354+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:33.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:33 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:33.354+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:33.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:33 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:33.354+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:33.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:33 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:33.234+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:33.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:33 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:33.234+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:33.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:33 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:33.234+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:33.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:33 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:33.261+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:33.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:33 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:33.261+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:33.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:33 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:33.261+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:04.569211+0000 front 2024-09-19T08:15:04.569122+0000 (oldest deadline 2024-09-19T08:15:29.269000+0000) 2024-09-19T08:15:34.242 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:33 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:33.981+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:34.242 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:33 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:33.981+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:34.242 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:33 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:33.981+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:34.243 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:33 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:33.981+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:34.433 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:34 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:34.083+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:34.433 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:34 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:34.083+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:34.433 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:34 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:34.083+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:34.433 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:34 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:34.083+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:34.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:34 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:34.379+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:34.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:34 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:34.379+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:34.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:34 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:34.379+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:34.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:34 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:34.277+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:34.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:34 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:34.277+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:34.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:34 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:34.277+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:34.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:34 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:34.242+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:34.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:34 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:34.242+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:34.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:34 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:34.242+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:04.569211+0000 front 2024-09-19T08:15:04.569122+0000 (oldest deadline 2024-09-19T08:15:29.269000+0000) 2024-09-19T08:15:35.252 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:34 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:34.967+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:35.252 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:34 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:34.967+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:35.252 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:34 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:34.967+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:35.252 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:34 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:34.967+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:35.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:35 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:35.080+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:35.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:35 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:35.080+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:35.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:35 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:35.080+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:35.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:35 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:35.080+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:35.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:35 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:35.374+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:35.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:35 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:35.374+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:35.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:35 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:35.374+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:35.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:35 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:35.258+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:35.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:35 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:35.258+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:35.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:35 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:35.258+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:35.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:35 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:35.252+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:35.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:35 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:35.252+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:35.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:35 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:35.252+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:04.569211+0000 front 2024-09-19T08:15:04.569122+0000 (oldest deadline 2024-09-19T08:15:29.269000+0000) 2024-09-19T08:15:36.297 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:36 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:36.005+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:36.298 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:36 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:36.005+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:36.298 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:36 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:36.005+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:36.298 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:36 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:36.005+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:36.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:36 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:36.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:36.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:36 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:36.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:36.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:36 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:36.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:36.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:36 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:36.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:36.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:36 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:36.335+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:36.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:36 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:36.335+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:36.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:36 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:36.335+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:36.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:36 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:36.245+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:36.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:36 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:36.245+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:36.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:36 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:36.245+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:36.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:36 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:36.298+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:36.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:36 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:36.298+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:36.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:36 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:36.298+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:04.569211+0000 front 2024-09-19T08:15:04.569122+0000 (oldest deadline 2024-09-19T08:15:29.269000+0000) 2024-09-19T08:15:37.268 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:37 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:37.000+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:37.268 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:37 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:37.000+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:37.269 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:37 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:37.000+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:37.269 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:37 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:37.000+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:37.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:37 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:37.055+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:37.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:37 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:37.055+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:37.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:37 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:37.055+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:37.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:37 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:37.055+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:37.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:37 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:37.269+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:37.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:37 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:37.269+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:37.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:37 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:37.269+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:37.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:37 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:37.373+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:37.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:37 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:37.373+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:37.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:37 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:37.373+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:37.621 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:37 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:37.373+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:37.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:37 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:37.268+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.566644+0000 front 2024-09-19T08:14:50.566764+0000 (oldest deadline 2024-09-19T08:15:13.466869+0000) 2024-09-19T08:15:37.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:37 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:37.268+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:55.168023+0000 front 2024-09-19T08:14:55.167955+0000 (oldest deadline 2024-09-19T08:15:19.867996+0000) 2024-09-19T08:15:37.677 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:37 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-19T08:15:37.268+0000 7f3300006640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:04.569211+0000 front 2024-09-19T08:15:04.569122+0000 (oldest deadline 2024-09-19T08:15:29.269000+0000) 2024-09-19T08:15:38.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:37 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:37.993+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:38.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:37 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:37.993+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:38.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:37 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:37.993+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:38.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:37 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:37.993+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:38.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:37 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:37.993+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:38.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:38 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:38.059+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:38.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:38 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:38.059+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:38.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:38 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:38.059+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:38.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:38 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:38.059+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:38.625 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:38 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:38.261+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:38.638 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:38 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:38.261+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:38.639 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:38 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:38.261+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:38.639 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:38 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:38.413+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:38.639 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:38 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:38.413+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:38.639 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:38 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:38.413+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:38.639 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:38 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:38.413+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:38.747 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:38 smithi139 podman[71207]: 2024-09-19 08:15:38.448724885 +0000 UTC m=+5.612022844 container died 60c68ec52a2adeae91c185bd5f1ab6d424d2653a2c379a7d491c84dd6ef521b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-19T08:15:39.021 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:38 smithi139 podman[71207]: 2024-09-19 08:15:38.845825587 +0000 UTC m=+6.009123545 container cleanup 60c68ec52a2adeae91c185bd5f1ab6d424d2653a2c379a7d491c84dd6ef521b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7, RELEASE=squid-2f599a4, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-19T08:15:39.125 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:38 smithi139 bash[71207]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7 2024-09-19T08:15:39.348 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:39 smithi139 podman[71225]: 2024-09-19 08:15:39.038506519 +0000 UTC m=+0.586997702 container remove 60c68ec52a2adeae91c185bd5f1ab6d424d2653a2c379a7d491c84dd6ef521b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-19T08:15:39.426 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:39 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:39.021+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:39.426 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:39 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:39.021+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:39.426 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:39 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:39.021+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:39.426 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:39 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:39.021+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:39.426 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:39 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:39.021+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:39.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:39 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:39.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:39.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:39 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:39.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:39.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:39 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:39.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:39.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:39 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:39.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:39.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:39 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:39.248+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:39.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:39 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:39.248+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:39.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:39 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:39.248+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:39.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:39 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:39.435+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:39.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:39 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:39.435+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:39.621 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:39 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:39.435+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:39.621 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:39 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:39.435+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:39.649 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:39 smithi139 podman[71313]: 2024-09-19 08:15:39.347574955 +0000 UTC m=+0.092014387 container create 0f6c45a679496b05b9b9116c7d4e4d235e60570f6bbe1cd616bae8fcd8eda80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-19T08:15:39.649 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:39 smithi139 podman[71313]: 2024-09-19 08:15:39.271419824 +0000 UTC m=+0.015859264 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:15:39.649 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:39 smithi139 podman[71313]: 2024-09-19 08:15:39.575328558 +0000 UTC m=+0.319767998 container init 0f6c45a679496b05b9b9116c7d4e4d235e60570f6bbe1cd616bae8fcd8eda80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-19T08:15:39.649 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:39 smithi139 podman[71313]: 2024-09-19 08:15:39.578820907 +0000 UTC m=+0.323260333 container start 0f6c45a679496b05b9b9116c7d4e4d235e60570f6bbe1cd616bae8fcd8eda80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-19T08:15:39.650 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:39 smithi139 podman[71313]: 2024-09-19 08:15:39.649844252 +0000 UTC m=+0.394283697 container attach 0f6c45a679496b05b9b9116c7d4e4d235e60570f6bbe1cd616bae8fcd8eda80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7-deactivate, RELEASE=squid-2f599a4, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-19T08:15:39.927 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 19 08:15:39 smithi139 podman[71313]: 2024-09-19 08:15:39.772995069 +0000 UTC m=+0.517434500 container died 0f6c45a679496b05b9b9116c7d4e4d235e60570f6bbe1cd616bae8fcd8eda80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-7-deactivate, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:15:40.327 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:40 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:40.016+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:40.327 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:40 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:40.016+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:40.327 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:40 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:40.016+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:40.327 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:40 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:40.016+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:40.327 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:40 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:40.016+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:40.355 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.7.service' 2024-09-19T08:15:40.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:40 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:40.035+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:40.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:40 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:40.035+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:40.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:40 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:40.035+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:40.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:40 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:40.035+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:40.435 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:15:40.435 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-19T08:15:40.435 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-19T08:15:40.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.8 2024-09-19T08:15:40.462 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:40 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:40.202+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:40.462 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:40 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:40.202+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:40.463 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:40 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:40.202+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:40.463 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:40 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:40.202+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.302582+0000 front 2024-09-19T08:15:14.302608+0000 (oldest deadline 2024-09-19T08:15:39.602628+0000) 2024-09-19T08:15:40.463 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:40 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:40.440+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:40.463 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:40 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:40.440+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:40.463 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:40 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:40.440+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:40.463 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:40 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:40.440+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:40.755 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:40 smithi162 systemd[1]: Stopping Ceph osd.8 for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:15:41.119 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:40 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:40.753+0000 7f6df7b98640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:15:41.120 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:40 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:40.753+0000 7f6df7b98640 -1 osd.8 94 *** Got signal Terminated *** 2024-09-19T08:15:41.120 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:40 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:40.753+0000 7f6df7b98640 -1 osd.8 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-19T08:15:41.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:41 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:41.014+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:41.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:41 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:41.014+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:41.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:41 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:41.014+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:41.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:41 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:41.014+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:41.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:41 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:41.014+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:41.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:41 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:41.063+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:41.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:41 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:41.063+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:41.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:41 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:41.063+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:41.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:41 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:41.063+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:41.459 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:41 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:41.175+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:41.459 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:41 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:41.175+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:41.459 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:41 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:41.175+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:41.459 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:41 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:41.175+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.302582+0000 front 2024-09-19T08:15:14.302608+0000 (oldest deadline 2024-09-19T08:15:39.602628+0000) 2024-09-19T08:15:41.869 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:41 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:41.458+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:41.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:41 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:41.458+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:41.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:41 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:41.458+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:41.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:41 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:41.458+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:42.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:42 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:42.062+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:42.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:42 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:42.062+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:42.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:42 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:42.062+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:42.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:42 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:42.062+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:42.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:42 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:42.062+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:42.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:42 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:42.031+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:42.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:42 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:42.031+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:42.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:42 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:42.031+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:42.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:42 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:42.031+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:42.463 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:42 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:42.148+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:42.463 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:42 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:42.148+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:42.463 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:42 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:42.148+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:42.463 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:42 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:42.148+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.302582+0000 front 2024-09-19T08:15:14.302608+0000 (oldest deadline 2024-09-19T08:15:39.602628+0000) 2024-09-19T08:15:42.869 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:42 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:42.462+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:42.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:42 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:42.462+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:42.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:42 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:42.462+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:42.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:42 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:42.462+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:43.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:43 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:43.057+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:43.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:43 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:43.057+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:43.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:43 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:43.057+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:43.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:43 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:43.057+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:43.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:43 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:43.057+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:43.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:43 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:43.036+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:43.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:43 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:43.036+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:43.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:43 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:43.036+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:43.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:43 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:43.036+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:43.483 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:43 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:43.133+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:43.483 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:43 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:43.133+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:43.484 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:43 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:43.133+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:43.484 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:43 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:43.133+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.302582+0000 front 2024-09-19T08:15:14.302608+0000 (oldest deadline 2024-09-19T08:15:39.602628+0000) 2024-09-19T08:15:43.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:43 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:43.483+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:43.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:43 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:43.483+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:43.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:43 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:43.483+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:43.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:43 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:43.483+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:44.370 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:44 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:44.103+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:44.370 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:44 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:44.103+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:44.370 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:44 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:44.103+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:44.371 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:44 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:44.103+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.302582+0000 front 2024-09-19T08:15:14.302608+0000 (oldest deadline 2024-09-19T08:15:39.602628+0000) 2024-09-19T08:15:44.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:44 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:44.055+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:44.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:44 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:44.055+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:44.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:44 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:44.055+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:44.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:44 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:44.055+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:44.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:44 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:44.055+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:44.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:44 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:44.055+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:44.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:44 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:44.032+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:44.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:44 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:44.032+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:44.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:44 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:44.032+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:44.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:44 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:44.032+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:44.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:44 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:44.510+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:44.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:44 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:44.510+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:44.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:44 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:44.510+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:44.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:44 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:44.510+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:44.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:44 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:44.510+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:45.329 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:45 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:45.025+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:45.329 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:45 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:45.025+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:45.329 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:45 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:45.025+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:45.329 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:45 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:45.025+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:45.329 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:45 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:45.025+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:45.329 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:45 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:45.025+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:45.370 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:45 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:45.087+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.196669+0000 front 2024-09-19T08:14:41.196701+0000 (oldest deadline 2024-09-19T08:15:04.096763+0000) 2024-09-19T08:15:45.370 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:45 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:45.087+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.999934+0000 front 2024-09-19T08:14:58.000084+0000 (oldest deadline 2024-09-19T08:15:23.900086+0000) 2024-09-19T08:15:45.370 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:45 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:45.087+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:03.900593+0000 front 2024-09-19T08:15:03.900629+0000 (oldest deadline 2024-09-19T08:15:27.400565+0000) 2024-09-19T08:15:45.370 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:45 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:45.087+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.302582+0000 front 2024-09-19T08:15:14.302608+0000 (oldest deadline 2024-09-19T08:15:39.602628+0000) 2024-09-19T08:15:45.370 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:45 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8[53746]: 2024-09-19T08:15:45.087+0000 7f6df419d640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:21.303614+0000 front 2024-09-19T08:15:21.303567+0000 (oldest deadline 2024-09-19T08:15:44.803644+0000) 2024-09-19T08:15:45.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:45 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:45.035+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:45.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:45 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:45.035+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:45.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:45 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:45.035+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:45.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:45 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:45.035+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:45.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:45 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:45.035+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:19.681425+0000 front 2024-09-19T08:15:19.681244+0000 (oldest deadline 2024-09-19T08:15:44.881731+0000) 2024-09-19T08:15:45.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:45 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:45.477+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:45.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:45 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:45.477+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:45.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:45 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:45.477+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:45.871 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:45 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:45.477+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:45.871 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:45 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:45.477+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:46.370 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:46 smithi162 podman[72085]: 2024-09-19 08:15:46.116332771 +0000 UTC m=+5.479665420 container died 99abe786e3cf96291344321fd4a24a2bd40282ecbcf9dc99ec685580c4d388a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS) 2024-09-19T08:15:46.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:46 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:46.049+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:46.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:46 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:46.049+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:46.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:46 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:46.049+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:46.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:46 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:46.049+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:46.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:46 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:46.049+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:46.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:46 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:46.049+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:46.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:46 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:46.062+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:46.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:46 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:46.062+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:46.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:46 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:46.062+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:46.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:46 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:46.062+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:46.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:46 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:46.062+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:19.681425+0000 front 2024-09-19T08:15:19.681244+0000 (oldest deadline 2024-09-19T08:15:44.881731+0000) 2024-09-19T08:15:46.869 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:46 smithi162 podman[72085]: 2024-09-19 08:15:46.44605533 +0000 UTC m=+5.809387968 container cleanup 99abe786e3cf96291344321fd4a24a2bd40282ecbcf9dc99ec685580c4d388a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-19T08:15:46.870 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:46 smithi162 bash[72085]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8 2024-09-19T08:15:46.870 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:46 smithi162 podman[72101]: 2024-09-19 08:15:46.613805296 +0000 UTC m=+0.494797075 container remove 99abe786e3cf96291344321fd4a24a2bd40282ecbcf9dc99ec685580c4d388a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:15:46.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:46 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:46.433+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:46.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:46 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:46.433+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:46.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:46 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:46.433+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:46.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:46 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:46.433+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:46.871 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:46 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:46.433+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:47.282 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:46 smithi162 podman[72190]: 2024-09-19 08:15:46.989596941 +0000 UTC m=+0.089916881 container create 62b2ff44aa8abe6810f8b0794ee3c5758f6f5c8033b1e44d8704daae67ce6bbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, RELEASE=squid-2f599a4) 2024-09-19T08:15:47.283 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:47 smithi162 podman[72190]: 2024-09-19 08:15:46.915382046 +0000 UTC m=+0.015701940 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:15:47.283 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:47 smithi162 podman[72190]: 2024-09-19 08:15:47.200338085 +0000 UTC m=+0.300657978 container init 62b2ff44aa8abe6810f8b0794ee3c5758f6f5c8033b1e44d8704daae67ce6bbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux ) 2024-09-19T08:15:47.283 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:47 smithi162 podman[72190]: 2024-09-19 08:15:47.203589266 +0000 UTC m=+0.303909169 container start 62b2ff44aa8abe6810f8b0794ee3c5758f6f5c8033b1e44d8704daae67ce6bbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=squid-2f599a4) 2024-09-19T08:15:47.283 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:47 smithi162 podman[72190]: 2024-09-19 08:15:47.28348307 +0000 UTC m=+0.383802965 container attach 62b2ff44aa8abe6810f8b0794ee3c5758f6f5c8033b1e44d8704daae67ce6bbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, ceph=True) 2024-09-19T08:15:47.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:47 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:47.003+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:47.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:47 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:47.003+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:47.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:47 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:47.003+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:47.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:47 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:47.003+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:47.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:47 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:47.003+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:47.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:47 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:47.003+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:47.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:47 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:47.094+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:47.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:47 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:47.094+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:47.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:47 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:47.094+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:47.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:47 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:47.094+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:47.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:47 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:47.094+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:19.681425+0000 front 2024-09-19T08:15:19.681244+0000 (oldest deadline 2024-09-19T08:15:44.881731+0000) 2024-09-19T08:15:47.620 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:47 smithi162 podman[72214]: 2024-09-19 08:15:47.425648362 +0000 UTC m=+0.017254928 container died 62b2ff44aa8abe6810f8b0794ee3c5758f6f5c8033b1e44d8704daae67ce6bbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-19T08:15:47.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:47 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:47.408+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:47.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:47 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:47.408+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:47.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:47 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:47.408+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:47.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:47 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:47.408+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:47.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:47 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:47.408+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:47.960 INFO:journalctl@ceph.osd.8.smithi162.stdout:Sep 19 08:15:47 smithi162 podman[72214]: 2024-09-19 08:15:47.945140214 +0000 UTC m=+0.536746783 container remove 62b2ff44aa8abe6810f8b0794ee3c5758f6f5c8033b1e44d8704daae67ce6bbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-19T08:15:47.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.8.service' 2024-09-19T08:15:48.044 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:15:48.044 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-19T08:15:48.045 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-19T08:15:48.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.9 2024-09-19T08:15:48.331 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:48.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:48.331 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:48.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:48.331 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:48.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:48.331 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:48.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:48.331 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:48.053+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:19.681425+0000 front 2024-09-19T08:15:19.681244+0000 (oldest deadline 2024-09-19T08:15:44.881731+0000) 2024-09-19T08:15:48.331 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:48 smithi098 systemd[1]: Stopping Ceph osd.9 for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:15:48.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:48 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:48.017+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:48.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:48 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:48.017+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:48.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:48 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:48.017+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:48.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:48 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:48.017+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:48.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:48 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:48.017+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:48.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:48 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:48.017+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:48.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:48 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:48.424+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:48.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:48 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:48.424+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:48.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:48 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:48.424+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:48.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:48 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:48.424+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:48.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:48 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:48.424+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:48.681 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:48.331+0000 7f8bdbdae640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:15:48.682 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:48.331+0000 7f8bdbdae640 -1 osd.9 94 *** Got signal Terminated *** 2024-09-19T08:15:48.682 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:48 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:48.331+0000 7f8bdbdae640 -1 osd.9 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-19T08:15:49.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:48 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:48.977+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:49.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:48 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:48.977+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:49.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:48 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:48.977+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:49.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:48 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:48.977+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:49.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:48 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:48.977+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:49.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:48 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:48.977+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:49.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:49 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:49.019+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:49.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:49 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:49.019+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:49.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:49 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:49.019+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:49.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:49 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:49.019+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:49.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:49 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:49.019+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:19.681425+0000 front 2024-09-19T08:15:19.681244+0000 (oldest deadline 2024-09-19T08:15:44.881731+0000) 2024-09-19T08:15:49.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:49 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:49.454+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:49.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:49 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:49.454+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:49.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:49 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:49.454+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:49.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:49 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:49.454+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:49.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:49 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:49.454+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:50.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:49 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:49.958+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:50.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:49 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:49.958+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:50.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:49 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:49.958+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:50.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:49 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:49.958+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:50.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:49 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:49.958+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:50.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:49 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:49.958+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:50.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:50 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:50.031+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:50.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:50 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:50.031+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:50.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:50 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:50.031+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:50.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:50 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:50.031+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:50.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:50 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:50.031+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:19.681425+0000 front 2024-09-19T08:15:19.681244+0000 (oldest deadline 2024-09-19T08:15:44.881731+0000) 2024-09-19T08:15:50.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:50 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:50.417+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:50.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:50 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:50.417+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:50.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:50 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:50.417+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:50.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:50 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:50.417+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:50.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:50 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:50.417+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:51.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:50 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:50.940+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:51.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:50 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:50.940+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:51.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:50 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:50.940+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:51.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:50 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:50.940+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:51.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:50 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:50.940+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:51.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:50 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:50.940+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:51.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:51.070+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:51.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:51.070+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:51.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:51.070+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:51.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:51.070+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:51.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:51 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:51.070+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:19.681425+0000 front 2024-09-19T08:15:19.681244+0000 (oldest deadline 2024-09-19T08:15:44.881731+0000) 2024-09-19T08:15:51.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:51 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:51.412+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:51.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:51 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:51.412+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:51.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:51 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:51.412+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:51.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:51 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:51.412+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:51.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:51 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:51.412+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:51.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:51 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:51.412+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:15:52.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:51 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:51.951+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:52.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:51 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:51.951+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:52.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:51 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:51.951+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:52.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:51 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:51.951+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:52.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:51 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:51.951+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:52.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:51 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:51.951+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:52.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:52 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:52.120+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:52.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:52 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:52.120+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:52.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:52 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:52.120+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:52.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:52 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:52.120+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:52.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:52 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:52.120+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:19.681425+0000 front 2024-09-19T08:15:19.681244+0000 (oldest deadline 2024-09-19T08:15:44.881731+0000) 2024-09-19T08:15:52.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:52 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:52.429+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:52.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:52 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:52.429+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:52.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:52 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:52.429+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:52.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:52 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:52.429+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:52.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:52 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:52.429+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:52.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:52 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:52.429+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:15:53.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:52 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:52.986+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:53.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:52 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:52.986+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:53.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:52 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:52.986+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:53.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:52 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:52.986+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:53.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:52 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:52.986+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:53.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:52 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:52.986+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:53.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:52 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:52.986+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:29.913272+0000 front 2024-09-19T08:15:29.913291+0000 (oldest deadline 2024-09-19T08:15:52.213208+0000) 2024-09-19T08:15:53.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:53 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:53.146+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:41.961453+0000 front 2024-09-19T08:14:41.961725+0000 (oldest deadline 2024-09-19T08:15:04.874736+0000) 2024-09-19T08:15:53.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:53 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:53.146+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:50.675860+0000 front 2024-09-19T08:14:50.675882+0000 (oldest deadline 2024-09-19T08:15:15.975991+0000) 2024-09-19T08:15:53.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:53 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:53.146+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.077006+0000 front 2024-09-19T08:14:57.077082+0000 (oldest deadline 2024-09-19T08:15:19.977004+0000) 2024-09-19T08:15:53.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:53 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:53.146+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.679325+0000 front 2024-09-19T08:15:05.679123+0000 (oldest deadline 2024-09-19T08:15:30.979325+0000) 2024-09-19T08:15:53.432 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:53 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9[65318]: 2024-09-19T08:15:53.146+0000 7f8bd43ab640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:19.681425+0000 front 2024-09-19T08:15:19.681244+0000 (oldest deadline 2024-09-19T08:15:44.881731+0000) 2024-09-19T08:15:53.771 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:53 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:53.427+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:53.771 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:53 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:53.427+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:53.771 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:53 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:53.427+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:53.771 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:53 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:53.427+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:53.771 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:53 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:53.427+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:53.772 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:53 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:53.427+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:15:54.097 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:53 smithi098 podman[112518]: 2024-09-19 08:15:53.69538655 +0000 UTC m=+5.457546801 container died 06d5f705b6d50a38c9420f961440417caa746ff6458b57a700348119d9570e7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-19T08:15:54.097 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:54 smithi098 podman[112518]: 2024-09-19 08:15:54.09794568 +0000 UTC m=+5.860105929 container cleanup 06d5f705b6d50a38c9420f961440417caa746ff6458b57a700348119d9570e7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-19T08:15:54.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:53 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:53.953+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:54.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:53 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:53.953+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:54.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:53 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:53.953+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:54.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:53 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:53.953+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:54.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:53 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:53.953+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:54.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:53 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:53.953+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:54.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:53 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:53.953+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:29.913272+0000 front 2024-09-19T08:15:29.913291+0000 (oldest deadline 2024-09-19T08:15:52.213208+0000) 2024-09-19T08:15:54.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:54 smithi098 bash[112518]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9 2024-09-19T08:15:54.431 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:54 smithi098 podman[112535]: 2024-09-19 08:15:54.324721748 +0000 UTC m=+0.624927703 container remove 06d5f705b6d50a38c9420f961440417caa746ff6458b57a700348119d9570e7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-19T08:15:54.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:54 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:54.439+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:54.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:54 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:54.439+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:54.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:54 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:54.439+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:54.871 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:54 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:54.439+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:54.871 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:54 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:54.439+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:54.871 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:54 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:54.439+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:15:54.877 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:54 smithi098 podman[112623]: 2024-09-19 08:15:54.617200884 +0000 UTC m=+0.095201008 container create 000aa0bf7eff6a94aefa8828788087c7ac659320c6b0450e38f3625bd2c15ff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-19T08:15:54.877 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:54 smithi098 podman[112623]: 2024-09-19 08:15:54.537642453 +0000 UTC m=+0.015642592 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:15:54.877 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:54 smithi098 podman[112623]: 2024-09-19 08:15:54.877925788 +0000 UTC m=+0.355925914 container init 000aa0bf7eff6a94aefa8828788087c7ac659320c6b0450e38f3625bd2c15ff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-19T08:15:55.181 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:54 smithi098 podman[112623]: 2024-09-19 08:15:54.881439924 +0000 UTC m=+0.359440056 container start 000aa0bf7eff6a94aefa8828788087c7ac659320c6b0450e38f3625bd2c15ff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9-deactivate, RELEASE=squid-2f599a4, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-19T08:15:55.182 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:54 smithi098 podman[112623]: 2024-09-19 08:15:54.969604173 +0000 UTC m=+0.447604325 container attach 000aa0bf7eff6a94aefa8828788087c7ac659320c6b0450e38f3625bd2c15ff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:15:55.182 INFO:journalctl@ceph.osd.9.smithi098.stdout:Sep 19 08:15:55 smithi098 podman[112623]: 2024-09-19 08:15:55.081031484 +0000 UTC m=+0.559031619 container died 000aa0bf7eff6a94aefa8828788087c7ac659320c6b0450e38f3625bd2c15ff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-2f599a4, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-19T08:15:55.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:54 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:54.962+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:55.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:54 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:54.962+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:55.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:54 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:54.962+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:55.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:54 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:54.962+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:55.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:54 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:54.962+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:55.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:54 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:54.962+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:55.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:54 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:54.962+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:29.913272+0000 front 2024-09-19T08:15:29.913291+0000 (oldest deadline 2024-09-19T08:15:52.213208+0000) 2024-09-19T08:15:55.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.9.service' 2024-09-19T08:15:55.709 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:15:55.710 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-19T08:15:55.710 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-19T08:15:55.710 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.10 2024-09-19T08:15:55.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:55 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:55.450+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:55.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:55 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:55.450+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:55.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:55 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:55.450+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:55.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:55 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:55.450+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:55.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:55 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:55.450+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:55.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:55 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:55.450+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:15:56.005 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:55 smithi139 systemd[1]: Stopping Ceph osd.10 for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:15:56.005 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:55 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:55.916+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:56.005 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:55 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:55.916+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:56.005 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:55 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:55.916+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:56.005 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:55 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:55.916+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:56.005 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:55 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:55.916+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:56.005 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:55 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:55.916+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:56.005 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:55 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:55.916+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:29.913272+0000 front 2024-09-19T08:15:29.913291+0000 (oldest deadline 2024-09-19T08:15:52.213208+0000) 2024-09-19T08:15:56.006 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:56.004+0000 7fabd41cb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:15:56.006 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:56.004+0000 7fabd41cb640 -1 osd.10 94 *** Got signal Terminated *** 2024-09-19T08:15:56.008 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:56.004+0000 7fabd41cb640 -1 osd.10 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-19T08:15:56.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:56 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:56.439+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:56.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:56 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:56.439+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:56.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:56 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:56.439+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:56.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:56 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:56.439+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:56.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:56 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:56.439+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:56.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:56 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:56.439+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:15:57.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:56.956+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:57.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:56.956+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:57.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:56.956+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:57.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:56.956+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:57.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:56.956+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:57.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:56.956+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:57.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:56 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:56.956+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:29.913272+0000 front 2024-09-19T08:15:29.913291+0000 (oldest deadline 2024-09-19T08:15:52.213208+0000) 2024-09-19T08:15:57.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:57 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:57.433+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:57.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:57 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:57.433+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:57.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:57 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:57.433+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:57.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:57 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:57.433+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:57.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:57 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:57.433+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:57.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:57 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:57.433+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:15:58.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:57 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:57.996+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:58.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:57 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:57.996+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:58.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:57 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:57.996+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:58.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:57 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:57.996+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:58.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:57 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:57.996+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:58.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:57 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:57.996+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:58.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:57 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:57.996+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:29.913272+0000 front 2024-09-19T08:15:29.913291+0000 (oldest deadline 2024-09-19T08:15:52.213208+0000) 2024-09-19T08:15:58.735 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:58 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:58.406+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:58.735 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:58 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:58.406+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:58.735 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:58 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:58.406+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:58.736 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:58 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:58.406+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:58.736 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:58 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:58.406+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:58.736 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:58 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:58.406+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:15:59.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:59.017+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:15:59.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:59.017+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:15:59.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:59.017+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:15:59.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:59.017+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:15:59.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:59.017+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:15:59.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:59.017+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:15:59.428 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:15:59 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:15:59.017+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:29.913272+0000 front 2024-09-19T08:15:29.913291+0000 (oldest deadline 2024-09-19T08:15:52.213208+0000) 2024-09-19T08:15:59.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:59.369+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:15:59.675 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:59.369+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:15:59.676 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:59.369+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:15:59.676 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:59.369+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:15:59.676 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:59.369+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:15:59.676 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:15:59 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:15:59.369+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:16:00.427 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:00 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:16:00.001+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:43.306479+0000 front 2024-09-19T08:14:43.306656+0000 (oldest deadline 2024-09-19T08:15:09.206580+0000) 2024-09-19T08:16:00.580 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:00 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:16:00.001+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.207002+0000 front 2024-09-19T08:14:49.206979+0000 (oldest deadline 2024-09-19T08:15:12.707139+0000) 2024-09-19T08:16:00.580 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:00 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:16:00.001+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:57.908699+0000 front 2024-09-19T08:14:57.908601+0000 (oldest deadline 2024-09-19T08:15:21.408472+0000) 2024-09-19T08:16:00.580 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:00 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:16:00.001+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-09-19T08:15:05.509535+0000 front 2024-09-19T08:15:05.509616+0000 (oldest deadline 2024-09-19T08:15:30.209494+0000) 2024-09-19T08:16:00.580 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:00 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:16:00.001+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:14.210766+0000 front 2024-09-19T08:15:14.210803+0000 (oldest deadline 2024-09-19T08:15:37.110861+0000) 2024-09-19T08:16:00.580 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:00 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:16:00.001+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.612032+0000 front 2024-09-19T08:15:20.611862+0000 (oldest deadline 2024-09-19T08:15:43.511875+0000) 2024-09-19T08:16:00.580 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:00 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10[59643]: 2024-09-19T08:16:00.001+0000 7fabcc7c8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:29.913272+0000 front 2024-09-19T08:15:29.913291+0000 (oldest deadline 2024-09-19T08:15:52.213208+0000) 2024-09-19T08:16:00.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:00 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:00.409+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:16:00.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:00 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:00.409+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:16:00.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:00 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:00.409+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:16:00.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:00 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:00.409+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:16:00.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:00 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:00.409+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:16:00.871 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:00 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:00.409+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:16:00.871 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:00 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:00.409+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-09-19T08:15:36.616790+0000 front 2024-09-19T08:15:36.616715+0000 (oldest deadline 2024-09-19T08:16:00.116574+0000) 2024-09-19T08:16:01.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:01 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:01.363+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:16:01.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:01 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:01.363+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:16:01.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:01 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:01.363+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:16:01.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:01 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:01.363+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:16:01.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:01 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:01.363+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:16:01.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:01 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:01.363+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:16:01.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:01 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:01.363+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-09-19T08:15:36.616790+0000 front 2024-09-19T08:15:36.616715+0000 (oldest deadline 2024-09-19T08:16:00.116574+0000) 2024-09-19T08:16:01.677 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:01 smithi139 podman[71427]: 2024-09-19 08:16:01.397510092 +0000 UTC m=+5.487457902 container died 199f7f0b9306f6a3b7e415616ebd0cfec5e4192630018f06ab82e644815bd7e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-19T08:16:02.021 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:01 smithi139 podman[71427]: 2024-09-19 08:16:01.787727531 +0000 UTC m=+5.877675371 container cleanup 199f7f0b9306f6a3b7e415616ebd0cfec5e4192630018f06ab82e644815bd7e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4) 2024-09-19T08:16:02.021 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:01 smithi139 bash[71427]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10 2024-09-19T08:16:02.326 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:02 smithi139 podman[71443]: 2024-09-19 08:16:02.022020749 +0000 UTC m=+0.620830718 container remove 199f7f0b9306f6a3b7e415616ebd0cfec5e4192630018f06ab82e644815bd7e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-19T08:16:02.591 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:02 smithi139 podman[71531]: 2024-09-19 08:16:02.229592055 +0000 UTC m=+0.015687240 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:16:02.591 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:02 smithi139 podman[71531]: 2024-09-19 08:16:02.347804593 +0000 UTC m=+0.133899821 container create f4d1ff5305373930817e004fe8074916c6573a760ac8d9430fb1eaec5755a621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-19T08:16:02.591 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:02 smithi139 podman[71531]: 2024-09-19 08:16:02.591862704 +0000 UTC m=+0.377957888 container init f4d1ff5305373930817e004fe8074916c6573a760ac8d9430fb1eaec5755a621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True) 2024-09-19T08:16:02.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:02 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:02.354+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:16:02.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:02 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:02.354+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:16:02.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:02 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:02.354+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:16:02.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:02 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:02.354+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:16:02.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:02 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:02.354+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:16:02.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:02 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:02.354+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:16:02.621 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:02 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:02.354+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-09-19T08:15:36.616790+0000 front 2024-09-19T08:15:36.616715+0000 (oldest deadline 2024-09-19T08:16:00.116574+0000) 2024-09-19T08:16:02.927 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:02 smithi139 podman[71531]: 2024-09-19 08:16:02.595113434 +0000 UTC m=+0.381208625 container start f4d1ff5305373930817e004fe8074916c6573a760ac8d9430fb1eaec5755a621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10-deactivate, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-19T08:16:02.927 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:02 smithi139 podman[71531]: 2024-09-19 08:16:02.675187232 +0000 UTC m=+0.461282436 container attach f4d1ff5305373930817e004fe8074916c6573a760ac8d9430fb1eaec5755a621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-19T08:16:02.928 INFO:journalctl@ceph.osd.10.smithi139.stdout:Sep 19 08:16:02 smithi139 podman[71531]: 2024-09-19 08:16:02.781168454 +0000 UTC m=+0.567263646 container died f4d1ff5305373930817e004fe8074916c6573a760ac8d9430fb1eaec5755a621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-09-19T08:16:03.434 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.10.service' 2024-09-19T08:16:03.473 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:16:03.473 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-19T08:16:03.473 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-19T08:16:03.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.11 2024-09-19T08:16:03.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:03 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:03.336+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:16:03.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:03 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:03.336+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:16:03.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:03 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:03.336+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:16:03.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:03 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:03.336+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:16:03.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:03 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:03.336+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:16:03.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:03 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:03.336+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:16:03.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:03 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:03.336+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-09-19T08:15:36.616790+0000 front 2024-09-19T08:15:36.616715+0000 (oldest deadline 2024-09-19T08:16:00.116574+0000) 2024-09-19T08:16:03.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:03 smithi162 systemd[1]: Stopping Ceph osd.11 for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:16:04.120 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:03 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:03.770+0000 7f4477f98640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:16:04.120 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:03 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:03.770+0000 7f4477f98640 -1 osd.11 94 *** Got signal Terminated *** 2024-09-19T08:16:04.120 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:03 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:03.770+0000 7f4477f98640 -1 osd.11 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-19T08:16:04.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:04 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:04.363+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:16:04.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:04 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:04.363+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:16:04.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:04 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:04.363+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:16:04.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:04 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:04.363+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:16:04.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:04 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:04.363+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:16:04.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:04 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:04.363+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:16:04.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:04 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:04.363+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-09-19T08:15:36.616790+0000 front 2024-09-19T08:15:36.616715+0000 (oldest deadline 2024-09-19T08:16:00.116574+0000) 2024-09-19T08:16:05.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:05 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:05.407+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:16:05.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:05 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:05.407+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:16:05.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:05 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:05.407+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:16:05.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:05 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:05.407+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:16:05.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:05 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:05.407+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:16:05.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:05 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:05.407+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:16:05.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:05 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:05.407+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-09-19T08:15:36.616790+0000 front 2024-09-19T08:15:36.616715+0000 (oldest deadline 2024-09-19T08:16:00.116574+0000) 2024-09-19T08:16:06.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:06 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:06.360+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:16:06.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:06 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:06.360+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:16:06.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:06 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:06.360+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:16:06.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:06 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:06.360+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:16:06.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:06 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:06.360+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:16:06.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:06 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:06.360+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:16:06.620 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:06 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:06.360+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-09-19T08:15:36.616790+0000 front 2024-09-19T08:15:36.616715+0000 (oldest deadline 2024-09-19T08:16:00.116574+0000) 2024-09-19T08:16:07.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:07 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:07.401+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:16:07.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:07 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:07.401+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:16:07.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:07 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:07.401+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:16:07.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:07 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:07.401+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:16:07.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:07 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:07.401+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:16:07.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:07 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:07.401+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:16:07.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:07 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:07.401+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-09-19T08:15:36.616790+0000 front 2024-09-19T08:15:36.616715+0000 (oldest deadline 2024-09-19T08:16:00.116574+0000) 2024-09-19T08:16:08.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:08 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:08.445+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-19T08:14:40.906678+0000 front 2024-09-19T08:14:40.906698+0000 (oldest deadline 2024-09-19T08:15:04.406681+0000) 2024-09-19T08:16:08.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:08 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:08.445+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-09-19T08:14:49.108050+0000 front 2024-09-19T08:14:49.107923+0000 (oldest deadline 2024-09-19T08:15:12.007935+0000) 2024-09-19T08:16:08.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:08 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:08.445+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-09-19T08:14:56.709230+0000 front 2024-09-19T08:14:56.709156+0000 (oldest deadline 2024-09-19T08:15:20.209370+0000) 2024-09-19T08:16:08.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:08 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:08.445+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-09-19T08:15:12.412193+0000 front 2024-09-19T08:15:12.412309+0000 (oldest deadline 2024-09-19T08:15:36.512192+0000) 2024-09-19T08:16:08.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:08 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:08.445+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-19T08:15:20.414088+0000 front 2024-09-19T08:15:20.414165+0000 (oldest deadline 2024-09-19T08:15:43.914130+0000) 2024-09-19T08:16:08.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:08 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:08.445+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-09-19T08:15:27.815800+0000 front 2024-09-19T08:15:27.815816+0000 (oldest deadline 2024-09-19T08:15:50.715799+0000) 2024-09-19T08:16:08.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:08 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11[60313]: 2024-09-19T08:16:08.445+0000 7f4470595640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-09-19T08:15:36.616790+0000 front 2024-09-19T08:15:36.616715+0000 (oldest deadline 2024-09-19T08:16:00.116574+0000) 2024-09-19T08:16:09.529 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:09 smithi162 podman[72308]: 2024-09-19 08:16:09.272234957 +0000 UTC m=+5.598335423 container died 709b4600fa255c83b72be331c20587f2bb9c062691c2077a3f5976b65dbfd31c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2) 2024-09-19T08:16:09.869 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:09 smithi162 podman[72308]: 2024-09-19 08:16:09.628611831 +0000 UTC m=+5.954712299 container cleanup 709b4600fa255c83b72be331c20587f2bb9c062691c2077a3f5976b65dbfd31c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:16:09.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:09 smithi162 bash[72308]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11 2024-09-19T08:16:10.238 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:09 smithi162 podman[72326]: 2024-09-19 08:16:09.904571036 +0000 UTC m=+0.625588382 container remove 709b4600fa255c83b72be331c20587f2bb9c062691c2077a3f5976b65dbfd31c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-19T08:16:10.515 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:10 smithi162 podman[72413]: 2024-09-19 08:16:10.23853332 +0000 UTC m=+0.096791895 container create 67da03545ab143d1376eb1b63ffeb3d6479fcfde26ca1ceb134da90fc30cf323 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=squid-2f599a4) 2024-09-19T08:16:10.515 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:10 smithi162 podman[72413]: 2024-09-19 08:16:10.157536252 +0000 UTC m=+0.015794827 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T08:16:10.515 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:10 smithi162 podman[72413]: 2024-09-19 08:16:10.440930368 +0000 UTC m=+0.299188944 container init 67da03545ab143d1376eb1b63ffeb3d6479fcfde26ca1ceb134da90fc30cf323 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-19T08:16:10.515 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:10 smithi162 podman[72413]: 2024-09-19 08:16:10.444134817 +0000 UTC m=+0.302393394 container start 67da03545ab143d1376eb1b63ffeb3d6479fcfde26ca1ceb134da90fc30cf323 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-19T08:16:10.515 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:10 smithi162 podman[72413]: 2024-09-19 08:16:10.515607227 +0000 UTC m=+0.373865818 container attach 67da03545ab143d1376eb1b63ffeb3d6479fcfde26ca1ceb134da90fc30cf323 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-19T08:16:10.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:10 smithi162 podman[72413]: 2024-09-19 08:16:10.634100918 +0000 UTC m=+0.492359503 container died 67da03545ab143d1376eb1b63ffeb3d6479fcfde26ca1ceb134da90fc30cf323 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=squid-2f599a4) 2024-09-19T08:16:11.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.11.service' 2024-09-19T08:16:11.190 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:11 smithi162 podman[72436]: 2024-09-19 08:16:11.152146868 +0000 UTC m=+0.512312439 container remove 67da03545ab143d1376eb1b63ffeb3d6479fcfde26ca1ceb134da90fc30cf323 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-19T08:16:11.190 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:11 smithi162 systemd[1]: ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.11.service: Deactivated successfully. 2024-09-19T08:16:11.191 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:11 smithi162 systemd[1]: Stopped Ceph osd.11 for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:16:11.191 INFO:journalctl@ceph.osd.11.smithi162.stdout:Sep 19 08:16:11 smithi162 systemd[1]: ceph-1a02525a-765c-11ef-bceb-c7b262605968@osd.11.service: Consumed 31.991s CPU time. 2024-09-19T08:16:11.205 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:16:11.206 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-19T08:16:11.206 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-09-19T08:16:11.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.a 2024-09-19T08:16:11.508 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:16:11 smithi098 systemd[1]: Stopping Ceph mds.a for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:16:11.931 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:16:11 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-a[72355]: 2024-09-19T08:16:11.507+0000 7f3bde4d7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:16:11.932 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:16:11 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-a[72355]: 2024-09-19T08:16:11.507+0000 7f3bde4d7640 -1 mds.a *** got signal Terminated *** 2024-09-19T08:16:15.791 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:16:15 smithi098 podman[112725]: 2024-09-19 08:16:15.514662745 +0000 UTC m=+4.110817544 container died 760d32eec551ea053ddf0f3ec586a1b0e1330a94bcc58594cebc23faeb0f69cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-19T08:16:16.142 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:16:15 smithi098 podman[112725]: 2024-09-19 08:16:15.957926093 +0000 UTC m=+4.554080884 container cleanup 760d32eec551ea053ddf0f3ec586a1b0e1330a94bcc58594cebc23faeb0f69cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4) 2024-09-19T08:16:16.142 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:16:15 smithi098 bash[112725]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-a 2024-09-19T08:16:16.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.a.service' 2024-09-19T08:16:16.407 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:16:16 smithi098 podman[112737]: 2024-09-19 08:16:16.142789793 +0000 UTC m=+0.621578550 container remove 760d32eec551ea053ddf0f3ec586a1b0e1330a94bcc58594cebc23faeb0f69cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-a, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-19T08:16:16.407 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:16:16 smithi098 systemd[1]: ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.a.service: Deactivated successfully. 2024-09-19T08:16:16.407 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:16:16 smithi098 systemd[1]: Stopped Ceph mds.a for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:16:16.407 INFO:journalctl@ceph.mds.a.smithi098.stdout:Sep 19 08:16:16 smithi098 systemd[1]: ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.a.service: Consumed 2.141s CPU time. 2024-09-19T08:16:16.420 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:16:16.420 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-09-19T08:16:16.420 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-09-19T08:16:16.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.d 2024-09-19T08:16:16.681 INFO:journalctl@ceph.mds.d.smithi098.stdout:Sep 19 08:16:16 smithi098 systemd[1]: Stopping Ceph mds.d for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:16:17.181 INFO:journalctl@ceph.mds.d.smithi098.stdout:Sep 19 08:16:16 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-d[71169]: 2024-09-19T08:16:16.735+0000 7f83787d2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:16:17.182 INFO:journalctl@ceph.mds.d.smithi098.stdout:Sep 19 08:16:16 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-d[71169]: 2024-09-19T08:16:16.735+0000 7f83787d2640 -1 mds.d *** got signal Terminated *** 2024-09-19T08:16:19.345 INFO:journalctl@ceph.mds.d.smithi098.stdout:Sep 19 08:16:19 smithi098 podman[112844]: 2024-09-19 08:16:19.092558657 +0000 UTC m=+2.454557797 container died 0f7fe9582b6a761dbf7f0165c6ce9f01b7c49e2e96dd692aeceac7067cc566a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-2f599a4, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-19T08:16:19.681 INFO:journalctl@ceph.mds.d.smithi098.stdout:Sep 19 08:16:19 smithi098 podman[112844]: 2024-09-19 08:16:19.47077882 +0000 UTC m=+2.832777957 container cleanup 0f7fe9582b6a761dbf7f0165c6ce9f01b7c49e2e96dd692aeceac7067cc566a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, org.label-schema.build-date=20240731) 2024-09-19T08:16:19.682 INFO:journalctl@ceph.mds.d.smithi098.stdout:Sep 19 08:16:19 smithi098 bash[112844]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-d 2024-09-19T08:16:19.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.d.service' 2024-09-19T08:16:19.890 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:16:19.890 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-09-19T08:16:19.890 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-09-19T08:16:19.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.g 2024-09-19T08:16:20.181 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:16:19 smithi098 systemd[1]: Stopping Ceph mds.g for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:16:20.681 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:16:20 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-g[71945]: 2024-09-19T08:16:20.210+0000 7f14e7cdb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:16:20.682 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:16:20 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-g[71945]: 2024-09-19T08:16:20.210+0000 7f14e7cdb640 -1 mds.g *** got signal Terminated *** 2024-09-19T08:16:23.931 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:16:23 smithi098 podman[112966]: 2024-09-19 08:16:23.53839794 +0000 UTC m=+3.429848726 container died 8009a4d16a39985e3a960c3c44b632be07b633058d066ae3052e831972e0a836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-g, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-19T08:16:24.349 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:16:24 smithi098 podman[112966]: 2024-09-19 08:16:24.107550139 +0000 UTC m=+3.999000913 container cleanup 8009a4d16a39985e3a960c3c44b632be07b633058d066ae3052e831972e0a836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-19T08:16:24.349 INFO:journalctl@ceph.mds.g.smithi098.stdout:Sep 19 08:16:24 smithi098 bash[112966]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-g 2024-09-19T08:16:24.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.g.service' 2024-09-19T08:16:24.529 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:16:24.529 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-09-19T08:16:24.529 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-09-19T08:16:24.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.j 2024-09-19T08:16:24.931 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:16:24 smithi098 systemd[1]: Stopping Ceph mds.j for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:16:24.932 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:16:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-j[70773]: 2024-09-19T08:16:24.851+0000 7f32e7732640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:16:24.932 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:16:24 smithi098 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-j[70773]: 2024-09-19T08:16:24.851+0000 7f32e7732640 -1 mds.j *** got signal Terminated *** 2024-09-19T08:16:28.078 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:16:27 smithi098 podman[113088]: 2024-09-19 08:16:27.721084292 +0000 UTC m=+2.973950697 container died d4b63495349adcabf78435929b2fa0ac66ec855ab7b3d1cb60cfc1e84ea8e359 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-19T08:16:28.432 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:16:28 smithi098 podman[113088]: 2024-09-19 08:16:28.178332426 +0000 UTC m=+3.431198817 container cleanup d4b63495349adcabf78435929b2fa0ac66ec855ab7b3d1cb60cfc1e84ea8e359 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-j, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-19T08:16:28.432 INFO:journalctl@ceph.mds.j.smithi098.stdout:Sep 19 08:16:28 smithi098 bash[113088]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-j 2024-09-19T08:16:28.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.j.service' 2024-09-19T08:16:28.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:16:28.634 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-09-19T08:16:28.635 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-09-19T08:16:28.635 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.b 2024-09-19T08:16:28.919 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:16:28 smithi139 systemd[1]: Stopping Ceph mds.b for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:16:29.177 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:16:28 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-b[65169]: 2024-09-19T08:16:28.953+0000 7fae66621640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:16:29.177 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:16:28 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-b[65169]: 2024-09-19T08:16:28.953+0000 7fae66621640 -1 mds.b *** got signal Terminated *** 2024-09-19T08:16:31.902 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:16:31 smithi139 podman[71634]: 2024-09-19 08:16:31.5564275 +0000 UTC m=+2.727359362 container died b05a28ba4813690ab7776b7d62c91b9db00a6b4e867c624fb80e4c7b65d85895 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-19T08:16:32.169 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:16:32 smithi139 podman[71634]: 2024-09-19 08:16:32.001686728 +0000 UTC m=+3.172618582 container cleanup b05a28ba4813690ab7776b7d62c91b9db00a6b4e867c624fb80e4c7b65d85895 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=squid-2f599a4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-19T08:16:32.169 INFO:journalctl@ceph.mds.b.smithi139.stdout:Sep 19 08:16:32 smithi139 bash[71634]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-b 2024-09-19T08:16:32.377 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.b.service' 2024-09-19T08:16:32.412 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:16:32.412 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-09-19T08:16:32.412 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-09-19T08:16:32.413 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.e 2024-09-19T08:16:32.739 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:16:32 smithi139 systemd[1]: Stopping Ceph mds.e for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:16:33.177 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:16:32 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-e[64574]: 2024-09-19T08:16:32.739+0000 7f1aa654e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:16:33.177 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:16:32 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-e[64574]: 2024-09-19T08:16:32.739+0000 7f1aa654e640 -1 mds.e *** got signal Terminated *** 2024-09-19T08:16:35.810 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:16:35 smithi139 podman[71766]: 2024-09-19 08:16:35.554133208 +0000 UTC m=+2.918357884 container died e6725063e11070cf28ffa827f927418fe602e251b6bb2d5158b424ddead3ec0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-19T08:16:36.164 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:16:35 smithi139 podman[71766]: 2024-09-19 08:16:35.905581147 +0000 UTC m=+3.269805823 container cleanup e6725063e11070cf28ffa827f927418fe602e251b6bb2d5158b424ddead3ec0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-e, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-19T08:16:36.164 INFO:journalctl@ceph.mds.e.smithi139.stdout:Sep 19 08:16:35 smithi139 bash[71766]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-e 2024-09-19T08:16:36.338 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.e.service' 2024-09-19T08:16:36.371 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:16:36.372 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-09-19T08:16:36.372 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-09-19T08:16:36.372 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.h 2024-09-19T08:16:36.683 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:16:36 smithi139 systemd[1]: Stopping Ceph mds.h for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:16:37.177 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:16:36 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-h[64874]: 2024-09-19T08:16:36.683+0000 7fde6e50f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:16:37.177 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:16:36 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-h[64874]: 2024-09-19T08:16:36.683+0000 7fde6e50f640 -1 mds.h *** got signal Terminated *** 2024-09-19T08:16:39.927 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:16:39 smithi139 podman[71887]: 2024-09-19 08:16:39.538584134 +0000 UTC m=+2.951777768 container died 438b5988f0ec603879b00e688da8901fa5236e1900c18dcc85baeb628f83c774 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-09-19T08:16:40.267 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:16:40 smithi139 podman[71887]: 2024-09-19 08:16:40.058508933 +0000 UTC m=+3.471702569 container cleanup 438b5988f0ec603879b00e688da8901fa5236e1900c18dcc85baeb628f83c774 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-h, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:16:40.268 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:16:40 smithi139 bash[71887]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-h 2024-09-19T08:16:40.496 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.h.service' 2024-09-19T08:16:40.518 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:16:40 smithi139 podman[71899]: 2024-09-19 08:16:40.268393948 +0000 UTC m=+0.723068305 container remove 438b5988f0ec603879b00e688da8901fa5236e1900c18dcc85baeb628f83c774 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-h, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4) 2024-09-19T08:16:40.518 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:16:40 smithi139 systemd[1]: ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.h.service: Deactivated successfully. 2024-09-19T08:16:40.518 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:16:40 smithi139 systemd[1]: Stopped Ceph mds.h for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:16:40.518 INFO:journalctl@ceph.mds.h.smithi139.stdout:Sep 19 08:16:40 smithi139 systemd[1]: ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.h.service: Consumed 1.936s CPU time. 2024-09-19T08:16:40.528 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:16:40.528 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-09-19T08:16:40.528 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-09-19T08:16:40.529 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.k 2024-09-19T08:16:40.862 INFO:journalctl@ceph.mds.k.smithi139.stdout:Sep 19 08:16:40 smithi139 systemd[1]: Stopping Ceph mds.k for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:16:41.177 INFO:journalctl@ceph.mds.k.smithi139.stdout:Sep 19 08:16:40 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-k[64265]: 2024-09-19T08:16:40.862+0000 7f84d9b2f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:16:41.177 INFO:journalctl@ceph.mds.k.smithi139.stdout:Sep 19 08:16:40 smithi139 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-k[64265]: 2024-09-19T08:16:40.862+0000 7f84d9b2f640 -1 mds.k *** got signal Terminated *** 2024-09-19T08:16:43.844 INFO:journalctl@ceph.mds.k.smithi139.stdout:Sep 19 08:16:43 smithi139 podman[72008]: 2024-09-19 08:16:43.546846546 +0000 UTC m=+2.793177260 container died 6e318dfb50977ac9e925fe5489213dd962196206766c10b0199b4fbbe753f076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-19T08:16:43.845 INFO:journalctl@ceph.mds.k.smithi139.stdout:Sep 19 08:16:43 smithi139 podman[72008]: 2024-09-19 08:16:43.845382247 +0000 UTC m=+3.091712950 container cleanup 6e318dfb50977ac9e925fe5489213dd962196206766c10b0199b4fbbe753f076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-k, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-19T08:16:44.174 INFO:journalctl@ceph.mds.k.smithi139.stdout:Sep 19 08:16:43 smithi139 bash[72008]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-k 2024-09-19T08:16:44.174 INFO:journalctl@ceph.mds.k.smithi139.stdout:Sep 19 08:16:44 smithi139 podman[72020]: 2024-09-19 08:16:44.063050368 +0000 UTC m=+0.514361380 container remove 6e318dfb50977ac9e925fe5489213dd962196206766c10b0199b4fbbe753f076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-k, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-19T08:16:44.184 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.k.service' 2024-09-19T08:16:44.216 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:16:44.217 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-09-19T08:16:44.217 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-09-19T08:16:44.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.c 2024-09-19T08:16:44.499 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:16:44 smithi162 systemd[1]: Stopping Ceph mds.c for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:16:44.869 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:16:44 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-c[65598]: 2024-09-19T08:16:44.499+0000 7ff72cfb3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:16:44.870 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:16:44 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-c[65598]: 2024-09-19T08:16:44.499+0000 7ff72cfb3640 -1 mds.c *** got signal Terminated *** 2024-09-19T08:16:47.839 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:16:47 smithi162 podman[72518]: 2024-09-19 08:16:47.552151166 +0000 UTC m=+3.141885744 container died 60526bf80aba8bd8fd67c5f5e5cdaffedd38a1ffc2c8fe1839f38161091ede4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-c, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-19T08:16:47.839 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:16:47 smithi162 podman[72518]: 2024-09-19 08:16:47.840124637 +0000 UTC m=+3.429859215 container cleanup 60526bf80aba8bd8fd67c5f5e5cdaffedd38a1ffc2c8fe1839f38161091ede4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-c, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-19T08:16:48.119 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:16:47 smithi162 bash[72518]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-c 2024-09-19T08:16:48.120 INFO:journalctl@ceph.mds.c.smithi162.stdout:Sep 19 08:16:48 smithi162 podman[72542]: 2024-09-19 08:16:48.007940732 +0000 UTC m=+0.453242368 container remove 60526bf80aba8bd8fd67c5f5e5cdaffedd38a1ffc2c8fe1839f38161091ede4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True) 2024-09-19T08:16:48.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.c.service' 2024-09-19T08:16:48.307 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:16:48.307 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-09-19T08:16:48.307 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-09-19T08:16:48.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.f 2024-09-19T08:16:48.620 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:16:48 smithi162 systemd[1]: Stopping Ceph mds.f for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:16:49.119 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:16:48 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-f[65294]: 2024-09-19T08:16:48.627+0000 7f4931108640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:16:49.120 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:16:48 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-f[65294]: 2024-09-19T08:16:48.627+0000 7f4931108640 -1 mds.f *** got signal Terminated *** 2024-09-19T08:16:51.873 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:16:51 smithi162 podman[72652]: 2024-09-19 08:16:51.546008801 +0000 UTC m=+3.015183690 container died 8096b91bd9820b2ac1925d31a072992d2aadf4b2740a5f17e174531765473bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-f, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=squid-2f599a4) 2024-09-19T08:16:52.178 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:16:51 smithi162 podman[72652]: 2024-09-19 08:16:51.976743975 +0000 UTC m=+3.445918862 container cleanup 8096b91bd9820b2ac1925d31a072992d2aadf4b2740a5f17e174531765473bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-2f599a4, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-19T08:16:52.178 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:16:51 smithi162 bash[72652]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-f 2024-09-19T08:16:52.409 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.f.service' 2024-09-19T08:16:52.431 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:16:52 smithi162 podman[72663]: 2024-09-19 08:16:52.178111482 +0000 UTC m=+0.629667953 container remove 8096b91bd9820b2ac1925d31a072992d2aadf4b2740a5f17e174531765473bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-f, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-19T08:16:52.431 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:16:52 smithi162 systemd[1]: ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.f.service: Deactivated successfully. 2024-09-19T08:16:52.431 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:16:52 smithi162 systemd[1]: Stopped Ceph mds.f for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:16:52.431 INFO:journalctl@ceph.mds.f.smithi162.stdout:Sep 19 08:16:52 smithi162 systemd[1]: ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.f.service: Consumed 1.925s CPU time. 2024-09-19T08:16:52.445 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:16:52.445 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-09-19T08:16:52.445 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-09-19T08:16:52.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.i 2024-09-19T08:16:52.755 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:16:52 smithi162 systemd[1]: Stopping Ceph mds.i for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:16:53.119 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:16:52 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-i[65901]: 2024-09-19T08:16:52.755+0000 7fadebdda640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:16:53.119 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:16:52 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-i[65901]: 2024-09-19T08:16:52.755+0000 7fadebdda640 -1 mds.i *** got signal Terminated *** 2024-09-19T08:16:55.870 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:16:55 smithi162 podman[72773]: 2024-09-19 08:16:55.540872479 +0000 UTC m=+2.875923515 container died a360c89cc7ff298384ef210724b077d605bd068f9a73a12f8b3dd1f7882aa43f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-i, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-19T08:16:56.355 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:16:56 smithi162 podman[72773]: 2024-09-19 08:16:56.171431997 +0000 UTC m=+3.506483032 container cleanup a360c89cc7ff298384ef210724b077d605bd068f9a73a12f8b3dd1f7882aa43f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-2f599a4, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-19T08:16:56.355 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:16:56 smithi162 bash[72773]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-i 2024-09-19T08:16:56.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.i.service' 2024-09-19T08:16:56.610 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:16:56 smithi162 podman[72786]: 2024-09-19 08:16:56.356011006 +0000 UTC m=+0.810803981 container remove a360c89cc7ff298384ef210724b077d605bd068f9a73a12f8b3dd1f7882aa43f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-19T08:16:56.610 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:16:56 smithi162 systemd[1]: ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.i.service: Deactivated successfully. 2024-09-19T08:16:56.610 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:16:56 smithi162 systemd[1]: Stopped Ceph mds.i for 1a02525a-765c-11ef-bceb-c7b262605968. 2024-09-19T08:16:56.611 INFO:journalctl@ceph.mds.i.smithi162.stdout:Sep 19 08:16:56 smithi162 systemd[1]: ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.i.service: Consumed 2.417s CPU time. 2024-09-19T08:16:56.622 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:16:56.622 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-09-19T08:16:56.622 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-09-19T08:16:56.622 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.l 2024-09-19T08:16:56.870 INFO:journalctl@ceph.mds.l.smithi162.stdout:Sep 19 08:16:56 smithi162 systemd[1]: Stopping Ceph mds.l for 1a02525a-765c-11ef-bceb-c7b262605968... 2024-09-19T08:16:57.370 INFO:journalctl@ceph.mds.l.smithi162.stdout:Sep 19 08:16:56 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-l[64992]: 2024-09-19T08:16:56.966+0000 7f8c09708640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T08:16:57.370 INFO:journalctl@ceph.mds.l.smithi162.stdout:Sep 19 08:16:56 smithi162 ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-l[64992]: 2024-09-19T08:16:56.966+0000 7f8c09708640 -1 mds.l *** got signal Terminated *** 2024-09-19T08:16:59.849 INFO:journalctl@ceph.mds.l.smithi162.stdout:Sep 19 08:16:59 smithi162 podman[72895]: 2024-09-19 08:16:59.547554928 +0000 UTC m=+2.702096378 container died 8d291b9421c5a0cf99f992352cbf9021bcb5ede3b03da3c9d533671f6696308e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-l, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-19T08:16:59.849 INFO:journalctl@ceph.mds.l.smithi162.stdout:Sep 19 08:16:59 smithi162 podman[72895]: 2024-09-19 08:16:59.849878448 +0000 UTC m=+3.004419887 container cleanup 8d291b9421c5a0cf99f992352cbf9021bcb5ede3b03da3c9d533671f6696308e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T08:17:00.120 INFO:journalctl@ceph.mds.l.smithi162.stdout:Sep 19 08:16:59 smithi162 bash[72895]: ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-l 2024-09-19T08:17:00.121 INFO:journalctl@ceph.mds.l.smithi162.stdout:Sep 19 08:17:00 smithi162 podman[72910]: 2024-09-19 08:17:00.084454257 +0000 UTC m=+0.529640547 container remove 8d291b9421c5a0cf99f992352cbf9021bcb5ede3b03da3c9d533671f6696308e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-1a02525a-765c-11ef-bceb-c7b262605968-mds-l, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-19T08:17:00.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1a02525a-765c-11ef-bceb-c7b262605968@mds.l.service' 2024-09-19T08:17:00.260 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-19T08:17:00.261 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-09-19T08:17:00.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1a02525a-765c-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-19T08:17:00.469 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T08:17:02.776 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1a02525a-765c-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-19T08:17:02.979 INFO:teuthology.orchestra.run.smithi139.stdout:Deleting cluster with fsid: 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T08:17:05.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1a02525a-765c-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-19T08:17:05.640 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T08:17:08.130 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-19T08:17:08.164 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-19T08:17:08.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-19T08:17:08.223 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-19T08:17:08.226 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911829/remote/smithi098/crash 2024-09-19T08:17:08.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/crash -- . 2024-09-19T08:17:08.262 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-19T08:17:08.262 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-09-19T08:17:08.264 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911829/remote/smithi139/crash 2024-09-19T08:17:08.265 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/crash -- . 2024-09-19T08:17:08.304 INFO:teuthology.orchestra.run.smithi139.stderr:tar: /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-19T08:17:08.304 INFO:teuthology.orchestra.run.smithi139.stderr:tar: Error is not recoverable: exiting now 2024-09-19T08:17:08.306 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911829/remote/smithi162/crash 2024-09-19T08:17:08.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/crash -- . 2024-09-19T08:17:08.346 INFO:teuthology.orchestra.run.smithi162.stderr:tar: /var/lib/ceph/1a02525a-765c-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-19T08:17:08.346 INFO:teuthology.orchestra.run.smithi162.stderr:tar: Error is not recoverable: exiting now 2024-09-19T08:17:08.347 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-19T08:17:08.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-19T08:17:08.389 INFO:tasks.cephadm:Compressing logs... 2024-09-19T08:17:08.390 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-19T08:17:08.432 DEBUG:teuthology.orchestra.run.smithi139:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-19T08:17:08.435 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-19T08:17:08.456 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-19T08:17:08.456 INFO:teuthology.orchestra.run.smithi098.stderr:: No such file or directory 2024-09-19T08:17:08.456 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.9.log 2024-09-19T08:17:08.456 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-19T08:17:08.457 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.0.log: /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-19T08:17:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.a.log 2024-09-19T08:17:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.audit.log: -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-19T08:17:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-19T08:17:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.g.log 2024-09-19T08:17:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mon.a.log: 84.5% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-19T08:17:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-19T08:17:08.460 INFO:teuthology.orchestra.run.smithi162.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-19T08:17:08.460 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-19T08:17:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-19T08:17:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.i.log 2024-09-19T08:17:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-19T08:17:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.audit.log: /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-19T08:17:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.11.log 2024-09-19T08:17:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mon.c.log: /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-19T08:17:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.g.log: 90.4% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-19T08:17:08.472 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-19T08:17:08.472 INFO:teuthology.orchestra.run.smithi162.stderr: 82.0% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-19T08:17:08.472 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.log 2024-09-19T08:17:08.473 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.l.log 2024-09-19T08:17:08.473 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.log: /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.5.log: 90.5% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-19T08:17:08.473 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.8.log 2024-09-19T08:17:08.473 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.l.log: 87.3% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-19T08:17:08.478 INFO:teuthology.orchestra.run.smithi162.stderr: 90.9% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-19T08:17:08.479 INFO:teuthology.orchestra.run.smithi139.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.10.log 2024-09-19T08:17:08.479 INFO:teuthology.orchestra.run.smithi139.stderr:: No such file or directory 2024-09-19T08:17:08.479 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-19T08:17:08.479 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-19T08:17:08.479 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-19T08:17:08.479 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.b.log 2024-09-19T08:17:08.479 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.k.log 2024-09-19T08:17:08.480 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-volume.log: /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-19T08:17:08.480 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-19T08:17:08.480 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-19T08:17:08.480 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.audit.log: 90.5% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-19T08:17:08.480 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-19T08:17:08.480 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.cephadm.log: 82.0% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-19T08:17:08.481 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.log 2024-09-19T08:17:08.481 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mgr.y.log: 90.9% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-19T08:17:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.log 90.9% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-19T08:17:08.482 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-19T08:17:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.d.log 2024-09-19T08:17:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-19T08:17:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.d.log: 87.3% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-19T08:17:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.j.log 2024-09-19T08:17:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mgr.x.log: 94.6% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.d.log.gz 2024-09-19T08:17:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-19T08:17:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.j.log: 94.5% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.j.log.gz 2024-09-19T08:17:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-19T08:17:08.487 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mgr.z.log 2024-09-19T08:17:08.487 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.c.log 2024-09-19T08:17:08.488 INFO:teuthology.orchestra.run.smithi139.stderr: 91.9% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-19T08:17:08.488 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.h.log 2024-09-19T08:17:08.488 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-19T08:17:08.488 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.h.log: 87.3% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-19T08:17:08.488 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.e.log 2024-09-19T08:17:08.493 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mgr.z.log: 91.9%gzip -5 --verbose -- /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.f.log 2024-09-19T08:17:08.502 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-19T08:17:08.502 INFO:teuthology.orchestra.run.smithi162.stderr: -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mgr.z.log.gz 2024-09-19T08:17:08.503 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-19T08:17:08.512 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.e.log: /var/log/ceph/cephadm.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-19T08:17:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 94.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-19T08:17:08.516 INFO:teuthology.orchestra.run.smithi098.stderr: 94.4% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.a.log.gz 2024-09-19T08:17:08.519 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.f.log: /var/log/ceph/cephadm.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-19T08:17:08.524 INFO:teuthology.orchestra.run.smithi098.stderr: 94.5% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.g.log.gz 2024-09-19T08:17:08.529 INFO:teuthology.orchestra.run.smithi162.stderr: 93.1% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.l.log.gz 2024-09-19T08:17:08.561 INFO:teuthology.orchestra.run.smithi162.stderr: 94.5% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.c.log.gz 2024-09-19T08:17:08.591 INFO:teuthology.orchestra.run.smithi162.stderr: 95.2% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.i.log.gz 2024-09-19T08:17:08.592 INFO:teuthology.orchestra.run.smithi162.stderr: 93.0% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.f.log.gz 2024-09-19T08:17:08.625 INFO:teuthology.orchestra.run.smithi139.stderr: 96.1% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.k.log.gz 2024-09-19T08:17:08.625 INFO:teuthology.orchestra.run.smithi139.stderr: 92.7% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.h.log.gz 2024-09-19T08:17:08.743 INFO:teuthology.orchestra.run.smithi098.stderr: 90.2% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-19T08:17:08.932 INFO:teuthology.orchestra.run.smithi162.stderr: 91.8% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-19T08:17:08.934 INFO:teuthology.orchestra.run.smithi139.stderr: 91.8% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-19T08:17:09.350 INFO:teuthology.orchestra.run.smithi162.stderr: 93.0% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.8.log.gz 2024-09-19T08:17:09.353 INFO:teuthology.orchestra.run.smithi162.stderr: 93.3% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.11.log.gz 2024-09-19T08:17:09.448 INFO:teuthology.orchestra.run.smithi139.stderr: 93.8% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.b.log.gz 2024-09-19T08:17:09.448 INFO:teuthology.orchestra.run.smithi098.stderr: 93.0% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.9.log.gz 2024-09-19T08:17:09.457 INFO:teuthology.orchestra.run.smithi098.stderr: 93.1% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-19T08:17:09.547 INFO:teuthology.orchestra.run.smithi139.stderr: 96.1% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mds.e.log.gz 2024-09-19T08:17:09.605 INFO:teuthology.orchestra.run.smithi162.stderr: 93.1% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-19T08:17:09.631 INFO:teuthology.orchestra.run.smithi098.stderr: 90.4% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-19T08:17:09.651 INFO:teuthology.orchestra.run.smithi098.stderr: 93.2% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-19T08:17:09.680 INFO:teuthology.orchestra.run.smithi162.stderr: 93.0% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-19T08:17:09.682 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-09-19T08:17:09.682 INFO:teuthology.orchestra.run.smithi162.stderr:real 0m1.233s 2024-09-19T08:17:09.682 INFO:teuthology.orchestra.run.smithi162.stderr:user 0m4.743s 2024-09-19T08:17:09.682 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m0.213s 2024-09-19T08:17:09.805 INFO:teuthology.orchestra.run.smithi098.stderr: 93.3% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-19T08:17:09.806 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-19T08:17:09.807 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m1.362s 2024-09-19T08:17:09.807 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m5.879s 2024-09-19T08:17:09.807 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m0.235s 2024-09-19T08:17:09.815 INFO:teuthology.orchestra.run.smithi139.stderr: 93.4% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-19T08:17:09.873 INFO:teuthology.orchestra.run.smithi139.stderr: 93.3% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-19T08:17:09.995 INFO:teuthology.orchestra.run.smithi139.stderr: 93.7% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.10.log.gz 2024-09-19T08:17:10.198 INFO:teuthology.orchestra.run.smithi139.stderr: 94.0% -- replaced with /var/log/ceph/1a02525a-765c-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-19T08:17:10.201 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-09-19T08:17:10.201 INFO:teuthology.orchestra.run.smithi139.stderr:real 0m1.752s 2024-09-19T08:17:10.201 INFO:teuthology.orchestra.run.smithi139.stderr:user 0m8.411s 2024-09-19T08:17:10.201 INFO:teuthology.orchestra.run.smithi139.stderr:sys 0m0.333s 2024-09-19T08:17:10.201 INFO:tasks.cephadm:Archiving logs... 2024-09-19T08:17:10.202 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911829/remote/smithi098/log 2024-09-19T08:17:10.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-19T08:17:11.023 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911829/remote/smithi139/log 2024-09-19T08:17:11.024 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-19T08:17:12.025 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911829/remote/smithi162/log 2024-09-19T08:17:12.026 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-19T08:17:12.790 INFO:tasks.cephadm:Removing cluster... 2024-09-19T08:17:12.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1a02525a-765c-11ef-bceb-c7b262605968 --force 2024-09-19T08:17:12.990 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T08:17:13.244 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1a02525a-765c-11ef-bceb-c7b262605968 --force 2024-09-19T08:17:13.445 INFO:teuthology.orchestra.run.smithi139.stdout:Deleting cluster with fsid: 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T08:17:13.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1a02525a-765c-11ef-bceb-c7b262605968 --force 2024-09-19T08:17:13.911 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: 1a02525a-765c-11ef-bceb-c7b262605968 2024-09-19T08:17:14.178 INFO:tasks.cephadm:Removing cephadm ... 2024-09-19T08:17:14.178 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-19T08:17:14.197 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-19T08:17:14.214 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-19T08:17:14.231 INFO:tasks.cephadm:Teardown complete 2024-09-19T08:17:14.231 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-19T08:17:14.244 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-19T08:17:14.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-19T08:17:14.247 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-19T08:17:14.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-19T08:17:14.370 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-19T08:17:14.381 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-19T08:17:14.381 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-19T08:17:14.385 DEBUG:teuthology.orchestra.run.smithi139:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-19T08:17:14.387 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-19T08:17:14.399 INFO:teuthology.orchestra.run.smithi139.stderr:bash: line 1: ntpq: command not found 2024-09-19T08:17:14.400 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-09-19T08:17:14.401 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-09-19T08:17:14.403 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-19T08:17:14.403 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2024-09-19T08:17:14.403 INFO:teuthology.orchestra.run.smithi139.stdout:^* hv01.front.sepia.ceph.com 3 6 377 50 +1319us[+1299us] +/- 52ms 2024-09-19T08:17:14.403 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 48 -3358us[-3358us] +/- 71ms 2024-09-19T08:17:14.403 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +1622us[+1622us] +/- 71ms 2024-09-19T08:17:14.403 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T08:17:14.405 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-19T08:17:14.405 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-09-19T08:17:14.405 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv01.front.sepia.ceph.com 3 6 377 48 +1304us[+1298us] +/- 52ms 2024-09-19T08:17:14.405 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 49 -3377us[-3383us] +/- 71ms 2024-09-19T08:17:14.405 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 +1595us[+1590us] +/- 71ms 2024-09-19T08:17:14.405 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T08:17:14.407 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-19T08:17:14.407 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-09-19T08:17:14.407 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv01.front.sepia.ceph.com 3 6 377 49 +1292us[+1291us] +/- 52ms 2024-09-19T08:17:14.407 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 50 -3403us[-3404us] +/- 71ms 2024-09-19T08:17:14.408 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +1573us[+1572us] +/- 71ms 2024-09-19T08:17:14.408 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T08:17:14.408 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-19T08:17:14.422 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-19T08:17:14.423 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-19T08:17:14.433 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-19T08:17:14.536 DEBUG:teuthology.orchestra.run.smithi139:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-19T08:17:14.603 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-19T08:17:14.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-19T08:17:14.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T08:17:14.704 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-19T08:17:14.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T08:17:14.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-19T08:17:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T08:17:14.761 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-19T08:17:14.772 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-19T08:17:14.803 INFO:teuthology.task.internal:Duration was 1852.130650 seconds 2024-09-19T08:17:14.803 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-19T08:17:14.818 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-19T08:17:14.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-19T08:17:14.821 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-19T08:17:14.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-19T08:17:14.860 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-19T08:17:14.860 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-19T08:17:14.864 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-19T08:17:15.220 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-19T08:17:15.220 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-09-19T08:17:15.220 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-19T08:17:15.282 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi139.front.sepia.ceph.com 2024-09-19T08:17:15.283 DEBUG:teuthology.orchestra.run.smithi139:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-19T08:17:15.310 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-09-19T08:17:15.310 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-19T08:17:15.341 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-19T08:17:15.342 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-19T08:17:15.345 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-19T08:17:15.353 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-19T08:17:15.409 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-19T08:17:15.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-19T08:17:15.413 DEBUG:teuthology.orchestra.run.smithi139:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-19T08:17:15.420 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-19T08:17:16.671 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-19T08:17:16.684 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-19T08:17:16.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-19T08:17:16.709 DEBUG:teuthology.orchestra.run.smithi139:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-19T08:17:16.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-19T08:17:16.769 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-19T08:17:16.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-19T08:17:16.786 DEBUG:teuthology.orchestra.run.smithi139:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-19T08:17:16.789 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-19T08:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-09-19T08:17:16.813 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = core 2024-09-19T08:17:16.837 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-09-19T08:17:16.865 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-19T08:17:16.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T08:17:16.881 DEBUG:teuthology.orchestra.run.smithi139:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-19T08:17:16.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T08:17:16.923 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-19T08:17:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T08:17:16.940 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-19T08:17:16.951 INFO:teuthology.task.internal:Transferring archived files... 2024-09-19T08:17:16.952 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911829/remote/smithi098 2024-09-19T08:17:16.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-19T08:17:17.014 DEBUG:teuthology.misc:Transferring archived files from smithi139:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911829/remote/smithi139 2024-09-19T08:17:17.014 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-19T08:17:17.054 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911829/remote/smithi162 2024-09-19T08:17:17.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-19T08:17:17.098 INFO:teuthology.task.internal:Removing archive directory... 2024-09-19T08:17:17.098 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-19T08:17:17.101 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-19T08:17:17.103 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-19T08:17:17.149 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-19T08:17:17.161 INFO:teuthology.task.internal:Not uploading archives. 2024-09-19T08:17:17.161 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-19T08:17:17.185 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-19T08:17:17.185 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-19T08:17:17.188 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-19T08:17:17.190 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-19T08:17:17.200 INFO:teuthology.orchestra.run.smithi098.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 19 08:17 /home/ubuntu/cephtest 2024-09-19T08:17:17.204 INFO:teuthology.orchestra.run.smithi139.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 19 08:17 /home/ubuntu/cephtest 2024-09-19T08:17:17.206 INFO:teuthology.orchestra.run.smithi162.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 19 08:17 /home/ubuntu/cephtest 2024-09-19T08:17:17.207 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-19T08:17:17.220 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-19T08:17:17.256 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/iozone}} duration: 1852.1306495666504 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-19T08:17:17.256 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-19T08:17:17.343 INFO:teuthology.run:pass